Partition Count as the Scaling Knob: Parallelism vs Ordering Tradeoffs

Partition Count as the Scaling Ceiling

Partition count defines your maximum parallelism ceiling. Each partition is an independent ordered log: records with the same partition key (a field used to determine which partition receives a record) always route to the same partition, and within that partition, order is preserved. You can scale horizontally by adding consumers up to the partition count, but you cannot exceed it. A topic with 50 partitions supports at most 50 active consumers. If throughput requirements grow to need 100 consumers, you must increase partition count, a potentially disruptive operation on a live system.

The Ordering vs Parallelism Trade-off

Partition count creates a fundamental trade-off between ordering guarantees and parallelism. If your application requires strict global ordering across all events (event A must process before event B regardless of their keys), you are limited to one partition and one active consumer, capping throughput at 1,000-10,000 records/sec depending on processing complexity. Most systems relax this by partitioning on entity identifiers (user ID, order ID, account ID). All events for user 123 go to partition 7 and are processed in order by one consumer; events for user 456 go to partition 12 and are processed in parallel by a different consumer. This preserves per-entity ordering while allowing arbitrarily high throughput.

Sizing for Future Growth

A common practice is provisioning 2-10x headroom beyond current requirements. If you need 20 consumers today, provision 40-200 partitions. The cost of extra partitions is primarily broker metadata overhead: each partition requires memory for index structures and file handles. Most clusters handle 10,000-50,000 partitions comfortably; problems emerge at 100,000+ partitions where rebalance time and controller load become concerns. Err on the side of more partitions because adding them later is complex.

The Danger of Mid-Stream Partition Increases

Increasing partition count on a live topic is operationally risky. Records already in the topic stay in their original partitions. The hash function that maps keys to partitions typically uses hash(key) mod partition_count. When partition count changes, the same key now maps to a different partition. New records for user 123 might go to partition 15 instead of partition 7, while old records remain in partition 7. Two consumers now process events for user 123 concurrently, breaking ordering guarantees. The safe approach: create a new topic with more partitions and migrate producers/consumers, or use a key-to-partition mapping that is stable under partition increases (more complex to implement).

Practical Guidelines

Start with at least 2x-5x your expected consumer count. For high-throughput topics expected to grow significantly, provision 100-500 partitions upfront. For low-throughput topics where parallelism needs are modest, 10-20 partitions suffice. Consider that partition count affects rebalance time: 1000 partitions rebalance faster than 10,000 partitions. Balance growth headroom against operational overhead.