CDC Distribution and Partitioning Strategy

The Distribution Problem: You have captured 50,000 change events per second from your database log. Now you need to publish them into a distributed event stream that multiple consumers can read in parallel. But some consumers need strict ordering for a given entity (like all changes to order 12345 in sequence). How do you balance parallelism with ordering guarantees? Partitioning by Entity Key: The solution is partition by primary key or entity ID. Each change event is assigned to a partition based on a hash of its key. All events for order 12345 always land in the same partition, say partition 47 out of 100 total partitions. This preserves per key ordering while allowing horizontal scale. With 100 partitions and 50k writes per second, each partition handles roughly 500 writes per second. Modern distributed logs like Kafka can handle 1,000 to 5,000 writes per second per partition with p99 latency under 100ms. This headroom is critical for handling traffic spikes. Retention for Replay and Recovery: The distributed log retains events for 3 to 7 days, even though the database commit log might only keep 48 hours. This longer retention serves two purposes. First, if a downstream consumer falls behind or needs to backfill, it can replay days of history without touching the OLTP database. Second, it provides time to diagnose and fix consumer bugs without data loss. Durability Through Replication: Each partition is replicated across multiple availability zones. Typically 3 replicas with acknowledgment from at least 2 before considering a write committed. This adds 10 to 30ms to publish latency but ensures no data loss on single node failures. Multiple Consumer Groups: Different downstream systems each run their own consumer group. The search indexing team reads from offset X, the analytics team from offset Y, and the cache invalidation service from offset Z. They progress independently. One slow consumer does not block others because the distributed log decouples them. This architecture is why Netflix can have dozens of teams consuming the same CDC stream for different purposes, each with their own latency requirements. Search might target 1 second freshness, while the data lake batches for 5 minute freshness to optimize storage writes. The partition based distribution makes this flexibility possible.