Event Streaming Architecture: Append Only Logs and Partitioned Parallelism

Definition
Event streaming centers on an immutable, append-only log where producers write events to named topics and consumers read sequentially using self-managed offsets. Topics are sharded into partitions to scale throughput linearly, but ordering is guaranteed only within a single partition.
Consumer Offset Control:

Consumers control their read position via offsets, enabling powerful replay and backfill capabilities. If you discover a bug in your analytics pipeline from three days ago, you can reset your consumer offset and reprocess millions of events. This time decoupling means producers and consumers evolve independently. A new analytics team can start consuming historical data without impacting existing consumers or requiring producer changes.

Production Scale:

Modern brokers with NVMe storage and 25 to 100 Gbps networking sustain 200 to 800 MB/s per broker with 1 to 10 KB messages. LinkedIn processes trillions of messages daily across many clusters. Netflix ingests hundreds of billions of events per day with sub-second to low-seconds end-to-end latency. The key enablers are batching, sequential disk I/O, and heavy use of OS page cache rather than application-managed memory.

Operational Complexity:

You must carefully plan partition counts (typically 2 to 3 times max consumer concurrency), choose partition keys that balance load while preserving locality, and monitor metrics like In-Sync Replica (ISR) health and consumer lag distributions. Poor key selection creates hotspots where a single partition becomes a bottleneck, spiking tail latencies from milliseconds to seconds.

💡 Key Takeaways

✓Partitions enable linear scaling: each partition is an independent ordered log. Clusters commonly run tens to hundreds of thousands of partitions with practical limits around 4,000 to 10,000 partitions per broker due to metadata overhead.

✓Offset management enables replay: consumers store their position and can rewind to any historical offset within the retention window (hours to weeks). This supports backfill, A/B test analysis on historical data, and disaster recovery.

✓Throughput is hardware bound, not architecture bound: 200 to 800 MB/s per broker sustained with modern NVMe and networking. Batching reduces per message overhead from microseconds to amortized nanoseconds.

✓Consumer groups divide partitions among members: each partition is assigned to exactly one consumer in a group, preserving order while scaling reads horizontally. Adding a 10th consumer helps only if you have at least 10 partitions.

✓Retention policies decouple storage from processing speed: time based retention (delete after 7 days) suits event transport; log compaction (keep only latest value per key) suits event sourced state, maintaining complete entity history in bounded space.

📌 Interview Tips

1LinkedIn partitions activity stream events by user_id to guarantee per user ordering for feed generation while parallelizing across millions of users. With 10,000 partitions and 100 consumers per group, each consumer handles 100 partitions.

2Netflix uses high partition counts (thousands per topic) to distribute personalization events. Each viewer interaction is keyed by viewer_id, ensuring all events for a viewer land in the same partition for correct session analysis.

3Uber built uReplicator to handle tens of thousands of topics across regions. Each topic's partitions replicate independently with per topic throttling, isolating failures and preventing a single hot topic from saturating cross datacenter bandwidth.

← Back to Kafka/Event Streaming Architecture Overview