Message Queues & Streaming • Consumer Groups & Load BalancingMedium⏱️ ~3 min
Partition Count as the Scaling Knob: Parallelism vs Ordering Tradeoffs
Partition count is the single most important scaling decision for consumer groups because it defines maximum parallelism, ordering guarantees, and future growth headroom. Each partition is an independent ordered log; records with the same partition key always go to the same partition and are processed in order. This means you can scale horizontally by adding consumers up to the partition count, but you cannot exceed it without increasing partitions.
The tradeoff is between parallelism and ordering. If you need strict global ordering across all events, you are limited to one partition and one active consumer, capping throughput at what a single process can handle (typically 1000 to 10000 records per second depending on record size and processing cost). Most systems relax this by partitioning on entity identifiers: user ID, order ID, device ID. This preserves per entity ordering while allowing parallelism across entities. For example, partition by user ID and you can process millions of users concurrently while each user's events remain ordered.
Provisioning more partitions than current consumers provides growth headroom and finer grained load balancing. A common practice is 2 to 10 times headroom: if you need 20 consumers today, provision 40 to 200 partitions. The upper bound is constrained by broker metadata overhead and rebalancing cost; Kafka clusters commonly run tens of thousands of partitions, but beyond hundreds of thousands you may hit controller bottlenecks. Amazon Kinesis shards enforce 1 MB/s write and 2 MB/s read limits per shard, so throughput scales linearly with shard count.
Increasing partition count midstream is operationally complex. Records already in the topic stay in their original partitions, and new records with the same key may land in different partitions if you change the partitioning function, breaking ordering. This requires coordinated drains, repartitioning jobs, or dual writes during migration. Design for growth upfront by provisioning headroom and choosing partition keys that distribute load evenly.
💡 Key Takeaways
•Maximum active consumers per group equals partition count: 50 partitions support at most 50 concurrent workers; deploying 100 consumers leaves 50 idle and wastes resources.
•Strict global ordering limits you to one partition and one consumer (typically 1000 to 10000 rec/s); per key ordering via partitioning enables horizontal scaling to millions of records per second across entities.
•Provision 2 to 10 times more partitions than current consumers for growth headroom and finer load balancing; 20 consumers today may need 40 to 200 partitions depending on expected growth and rebalancing tolerance.
•Amazon Kinesis shards provide 1 MB/s write and 2 MB/s shared read per shard; enhanced fan out gives each consumer 2 MB/s dedicated read, so a 100 shard stream supports 100 MB/s write and 200 MB/s per consumer read.
•Increasing partitions post launch is operationally risky: existing records stay in old partitions while new records with same keys may hash to different partitions, breaking per key ordering unless you drain and repartition.
•Kafka clusters commonly run tens of thousands of partitions; beyond hundreds of thousands, controller metadata and rebalancing overhead become bottlenecks requiring tuning or architectural changes.
📌 Examples
A payments system partitions by user ID: 100 partitions with 50 consumers processes 500000 payments per second with per user ordering, while one partition would cap at 5000 per second.
Amazon Kinesis stream with 200 shards supports 200 MB/s write and 400 MB/s shared read; with enhanced fan out, each consumer gets 200 MB/s dedicated read bandwidth.
LinkedIn operates Kafka clusters with tens to hundreds of thousands of partitions; sticky assignment and cooperative rebalancing are essential to avoid rebalance storms at this scale.