Message Queues & Streaming • Kafka/Event Streaming ArchitectureHard⏱️ ~3 min
Multi Region Patterns: Active Passive vs Active Active Replication
Multi region event streaming architectures split into two patterns: active passive (single writer region with async disaster recovery replication) and active active (independent regional writers with conflict resolution). Active passive is simpler: one region handles writes, asynchronously replicating to a standby region with 50 to 200 millisecond wide area network lag. Failover is manual with runbooks to promote the standby, typically taking 5 to 30 minutes. This pattern guarantees ordering and avoids conflicts but concentrates risk and increases latency for users far from the primary region.
Active active allows each region to write independently, improving regional autonomy and write latency for local users. Consumers read a union of sources across all primaries, requiring stream processors to handle duplicate events and occasional gaps during failovers. OpenAI built a union of sources connector pattern with watchdog services that detect topology changes (new partitions, primary additions) and auto scale stream jobs. They explicitly accept non exactly once semantics, handling duplicates downstream with idempotency keys and natural deduplication in sinks.
The failure modes differ sharply. Active passive risks extended unavailability if the primary region fails and manual failover is slow or bungled. Active active risks subtle data quality issues: duplicate events from overlapping unions, out of order delivery when primaries have clock skew, and gaps when a primary's replication lags. Cross region network partitions in active active can create split brain scenarios where both regions accept writes for the same key with different values, requiring last write wins with vector clocks or domain specific merge logic.
In practice, choose active passive for correctness critical domains (financial transactions, orders) where global ordering and deterministic replay matter more than regional write latency. Choose active active for availability critical, idempotent workloads (logging, metrics, operational telemetry) where regional autonomy and fast local writes outweigh occasional duplicates. Monitor per source offsets, replication lag, and watermark completeness across all primaries to detect drift and gaps.
💡 Key Takeaways
•Active passive adds 50 to 200 millisecond wide area network latency to replication lag: consumers in the standby region read stale data. Failover requires promoting the standby and redirecting producers, typically taking 5 to 30 minutes with manual runbooks or 1 to 5 minutes with automated orchestration.
•Active active union of sources requires consumers to handle duplicates: use natural idempotency keys (order_id, transaction_id) and deduplicate at the sink. Monitor per source watermarks to detect when one primary falls behind and contributes incomplete data.
•Conflict resolution in active active is domain specific: last write wins with vector clocks works for eventually consistent state; operational transforms (commutative operations like counters) work for aggregations; some domains require manual reconciliation (two regions process the same payment).
•Cross region replication saturates bandwidth quickly at scale: 1,000 events per second at 10 KB each is 10 MB/s or 80 Mbps per topic. With 100 topics and replication factor 2 (bidirectional), you need 16 Gbps sustained wide area network capacity. Cost and throttling are real constraints.
•Uber built uReplicator for tens of thousands of topics: per topic throttling, offset awareness for exactly once regional replay, and failure isolation prevent a single hot topic from saturating cross datacenter links. LinkedIn uses mirroring with automated lag monitoring and circuit breakers to pause replication under backlog, preventing cascade failures.
📌 Examples
A financial services firm uses active passive for payment topics. Writes go to the US primary; Europe reads with 100 to 150 millisecond staleness. During a US region outage, operators manually promote Europe to primary within 10 minutes, accepting a brief write outage to preserve exactly once semantics.
OpenAI runs multiple primary Kafka clusters for high availability. Stream processors consume a union of all primaries, dynamically adding new sources as topology changes. They explicitly accept rare duplicates and gaps, relying on idempotent sinks (upserts with unique constraints) to deduplicate.
Netflix mirrors personalization events across regions with active passive replication. The primary region (US West) handles all writes; other regions replicate for local read latency. During regional failures, they accept degraded user experience (stale recommendations) rather than failing over and risking global ordering loss.