OS & Systems Fundamentals • Processes vs ThreadsMedium⏱️ ~2 min
Scaling Models: Thread Pools vs Event Driven vs Process Pools
The choice of concurrency model determines how your system scales under load. Thread per connection is the simplest mental model: each incoming request gets a dedicated thread. This works well when threads are capped near the number of CPU cores and per request work is bounded. However, it degrades catastrophically when threads far exceed cores. With 10x more threads than cores, you hit scheduler thrash: the OS spends more time context switching than doing useful work. Context switches per second can jump to millions, inflating 99th percentile latency by 5 to 10x.
Event driven architectures solve the high connection count problem. Nginx demonstrates this perfectly: a small number of worker processes (typically one per core) with an event loop per worker can serve 100,000 to 1,000,000 keep alive connections with low memory overhead and minimal context switches. Each worker uses non blocking I/O and a state machine to multiplex thousands of connections. The tradeoff is programming complexity: you must write non blocking code and carefully manage state machines instead of simple blocking I/O.
Process pools offer a middle ground focused on isolation. PostgreSQL's process per connection model provides strong fault isolation: one connection's crash or memory corruption can't affect others. However, at 2,000 connections consuming 5 to 10 MB per backend process, you're looking at 10 to 20 GB just for connection overhead. Production PostgreSQL deployments typically cap connections to a few hundred and rely on connection poolers like PgBouncer to multiplex thousands of application connections onto a smaller pool of database connections.
💡 Key Takeaways
•Thread per connection works well with thread pools sized near CPU core count. MySQL historically used this model with threads (256 KB to 1 MB stack per thread), handling 10,000 connections with far less overhead than PostgreSQL's 10,000 processes.
•Event driven models like Nginx handle 100,000+ connections with just one worker process per core. Memory overhead drops to roughly 10 MB per 10,000 connections versus 1 GB+ for thread per connection with 1 MB stacks.
•Process per connection provides maximum isolation but struggles with scale. PostgreSQL at 2,000 connections consumes 10 to 20 GB just for backend processes. Production deployments use connection pooling to cap actual database connections to a few hundred.
•Scheduler thrash occurs when threads far exceed cores. With 10x oversubscription, context switches per second spike to millions, and 99th percentile latency inflates by 5 to 10x as the OS spends more time switching than executing.
•Microsoft IIS demonstrates hybrid scaling: worker processes for isolation with OS thread pools and async I/O inside each worker. A single worker process handles 100,000+ concurrent connections using IO completion ports with threads sized near core count.
📌 Examples
Nginx: 8 worker processes (one per core) serving 500K concurrent keep-alive connections. Memory: ~500 MB total. Context switches: minimal due to event loop design.
PostgreSQL: 2000 active connections = 2000 backend processes × 8 MB average = 16 GB memory. Solution: PgBouncer pools 10,000 app connections → 200 database connections.
Apache prefork (old model): process per request, limited to few thousand connections. Worker/event MPMs moved to thread pools, enabling 10K+ concurrent connections.