That is one way to solve it. However it isn't always a good answer. Mutexs - by definition - mean that you stop everyone else from doing something when you are doing it, which means if two threads are trying to do something one will be stopped. It is also very easy to forget to lock/unlock a mutex at the right time, both of which will cause problems. Mutexs also send a message out to other CPUs which means even if no other thread wants the mutex you still slowed your system down. In the real world mutexs become painfully slow when more than a few threads are accessing the same shared data.
There are also atomics which don't lock, and so in some cases are faster that a mutex - but they can also be a lot slower if in some data contention patterns. While these are often faster, there are a lot of useful data structures that cannot be written with atomics.
The correct answer is design everything to avoid the needs for writing shared data as much as possible. Then use atomics and/or mutexs in those final places where data sharing is needed. You always need to benchmark to see what works best.
Right, but if done right you rarely are in a situation two threads want to share the thread, so in practice you are not. However doing this right is difficult.
Yeah, but most of the things that need exclusivity or consistency are relatively rare or very narrow.
You need consitency when you update the socket tables. (Relatively rare, but can be a bottleneck for use cases with frequent open/close and maybe accept)
You need exclusivity when send or recieve on tcp sockets. (Very narrow)
You also need exclusivity when putting packets one of on the NIC's sendqueues. That one's not really rare, better NICs have more queues for less contention though.
For really high volume packet processing, it's helpful to align socket processing so everything is pinned to the same core: nic interrupts, kernel rx, application read and processing, application send, kernel tx, etc. But last I checked OpenBSD doesn't do cpu pinning, so there's that.
