CDC Failure Modes and Recovery at Scale

The Biggest Risk: Falling Behind: Your database produces 50,000 changes per second. Your CDC capture normally processes 55,000 per second. But a network hiccup slows it to 30,000 per second for 2 hours. You now have a backlog of 144 million unprocessed changes. If your database only retains logs for 48 hours, and recovery takes 3 hours, you hit log truncation and permanently lose data. In production, this is the nightmare scenario. You monitor two lag metrics: log sequence number lag (how many transactions behind) and time lag (how many seconds behind). Set database log retention to 3x your maximum expected catch up time. If normal lag is under 1 second and worst case recovery is 6 hours, retain logs for at least 24 hours with alerts at 12 hours. Schema Evolution Breaking Consumers: A developer adds a column to the orders table. The CDC capture includes this new field in change events, serialized with schema version 47. But your search indexing consumer was built for schema version 45 and crashes on unknown fields. Changes pile up unprocessed. Ordering Violations During Failover: Your active CDC capture instance dies. A standby takes over, resuming from the last checkpointed offset at log sequence 1,000,000. But the crashed instance had already published events up to sequence 1,000,050 before dying. The standby replays 1,000,000 to 1,000,050, creating duplicates. Worse, if offset tracking is slightly wrong, you might replay 999,990 to 1,000,050, causing consumers to see events out of order for keys in that range. The solution is idempotent consumers. Each change event includes a unique identifier (table name, primary key, and log sequence number). Consumers track the highest sequence number they have seen per primary key and ignore any event with a lower or equal sequence. This makes replays safe. Network Partitions and Partial Failures: What if CDC capture can reach the database but not the distributed log? It cannot publish changes. Options: buffer in memory (risks out of memory), write to local disk (adds latency and complexity), or block (risks falling behind). Most production systems choose buffering with circuit breakers: if the buffer exceeds a threshold, stop reading from the database log and alarm. Hot Key Edge Case: A single popular entity (viral post, flash sale item) receives 5,000 updates per second while average is 50 per key. That entity's partition becomes a bottleneck. One partition at 5,000 writes per second hits capacity limits, causing lag for all keys in that partition. Some systems detect hot keys and temporarily route them to a separate high throughput partition.