Message Queues & Streaming • Kafka/Event Streaming ArchitectureMedium⏱️ ~3 min
Replication Factor and Acknowledgment Modes: Durability vs Latency Tradeoffs
Every partition replicates across multiple brokers for durability, with replication factor 3 being the production standard across failure domains (racks or availability zones). One replica is elected leader and handles all reads and writes; followers replicate asynchronously. The set of followers caught up within configurable lag thresholds forms the in sync replica (ISR) set. A healthy partition has ISR equal to replication factor, meaning all replicas are current.
Producers configure acknowledgment semantics to trade latency for safety. Leader only acks return success once the leader writes to its local log, achieving produce latencies around 5 to 10 milliseconds in availability zone but risking data loss if the leader fails before followers replicate. Quorum acks (all ISR must acknowledge) increase latency to 10 to 20 milliseconds in availability zone as the leader waits for follower confirmation, but guarantee durability as long as at least one ISR replica survives. Cross availability zone replication adds 1 to 5 milliseconds; cross region adds 50 to 200 milliseconds driven by wide area network round trip time.
The failure mode is ISR shrink: if followers fall behind due to network issues or overload, they drop out of ISR. When ISR drops below the minimum (often 2 out of 3), writes with quorum acks block entirely, triggering producer timeouts and unavailability. You must monitor ISR count per partition and under replicated partition metrics obsessively. The alternative, unclean leader election (promoting an out of sync replica), can truncate already acknowledged writes and violate durability promises.
In practice, most teams accept quorum acks and size clusters to maintain ISR health under normal load spikes. LinkedIn and Netflix both use replication factor 3 with quorum acks as the default, prioritizing correctness over the last few milliseconds of latency. OpenAI's multi primary design accepts leader only acks in exchange for higher availability, then handles rare duplicates and gaps downstream with idempotency and deduplication logic.
💡 Key Takeaways
•Replication factor 3 across failure domains is the standard: survives single broker or availability zone loss. Higher factors (5) are rare due to cost and write amplification, used only for critical metadata topics.
•ISR health determines availability: when ISR shrinks below minimum in sync replicas setting (typically 2), quorum writes block. Monitor under replicated partitions and ISR delta metrics; alert on any partition with ISR below replication factor for more than 60 seconds.
•Cross availability zone latency is 1 to 5 milliseconds added; cross region is 50 to 200 milliseconds. Many teams run regional clusters and replicate asynchronously across regions to avoid synchronous wide area network latency penalties on every write.
•Unclean leader election trades availability for correctness: promoting an out of sync replica can truncate acknowledged data. Disable unclean election in production for topics requiring durability (financial transactions, orders); accept unavailability until an ISR replica recovers.
•Idempotent producers prevent duplicates on retry: enabling idempotence assigns sequence numbers per producer, allowing brokers to deduplicate retries after transient network failures. This is orthogonal to acks but pairs well with quorum mode for exactly once semantics.
📌 Examples
A financial services platform uses quorum acks with idempotent producers and disables unclean election for payment event topics. They accept 15 to 20 millisecond p99 produce latency in exchange for zero data loss guarantees, even during availability zone failures.
Netflix configures replication factor 3 with quorum acks as default. During a datacenter network partition, ISR shrunk to 1 on some partitions; writes blocked for 90 seconds until network healed and followers rejoined ISR, preventing any data loss.
OpenAI runs multiple primary clusters with leader only acks to maximize availability during primary failovers. They accept rare duplicates when unions of sources overlap and handle deduplication in downstream sinks using idempotency keys.