Failure Modes and Advanced Challenges in Log-Based CDC

Consumer Lag and Disk Exhaustion: The most dangerous failure mode is runaway lag. Imagine your CDC connector is processing at 50,000 events per second, keeping up with the database. Then a downstream Kafka broker has a network partition, and event publishing stalls for 2 minutes. During this time, the database continues writing at 50,000 transactions per second, generating 6 million uncommitted log entries. In PostgreSQL with logical replication, these Write Ahead Log (WAL) segments cannot be recycled until they are consumed. If your primary database has 100 GB of available disk and the WAL is growing at 20 MB per second, you have less than 90 minutes before disk fills completely. When disk space hits zero, PostgreSQL refuses new writes, and your application goes down. Production systems prevent this with monitoring and automated remediation. Monitor the delta between current log position and consumer position. Alert when lag exceeds thresholds: warning at 5 minutes or 500 MB, critical at 15 minutes or 2 GB. Remediation can include throttling non critical consumers, scaling out consumer capacity, or in emergencies, temporarily disabling CDC to allow log recycling. Duplicate Events and Idempotence: Log based CDC typically provides at least once delivery. If the CDC connector crashes after reading log position 45000 but before persisting that offset, it restarts from the last saved position, say 44500. It reprocesses 500 entries, emitting duplicate events. Downstream consumers MUST be idempotent. The standard pattern is upserting by primary key. For example, a data warehouse table has columns user_id, email, and updated_at. When processing a CDC event, upsert with a condition: only update if the incoming updated_at is newer than the existing value. This handles both duplicates and out of order delivery. For audit logs or event stores where duplicates are unacceptable, include the source log position in the event and deduplicate based on it. A consumer maintains a table of processed log positions and skips events it has already seen. Schema Evolution Challenges: When you add a NOT NULL column without a default to a table, older CDC events do not have that field. If downstream consumers expect the new schema immediately, they fail. The safe pattern is backward compatible evolution: add columns as nullable, deploy consumers that handle both old and new schemas, wait for all old events to drain (check lag is zero), then optionally make the column NOT NULL. Changing primary keys is even harder. If you change a user's user_id from 123 to 456, CDC emits a DELETE for 123 and an INSERT for 456. Downstream consumers that partition by primary key will route these to different partitions, potentially processing them out of order. Robust systems use a schema registry (like Confluent Schema Registry) to enforce compatibility rules and include both old and new schemas in events during transitions. Cross Region and Multi Source Ordering: In multi region setups with bidirectional replication, changes can originate in different regions and affect the same entity. Suppose user 123's email is updated in Region A at timestamp T1, and their phone is updated in Region B at timestamp T2, where T2 is slightly earlier than T1 due to clock skew. CDC events from both regions arrive at a central consumer, but network delays cause the T2 event to arrive first. Without conflict resolution, the consumer applies updates out of order, ending with stale data. Strategies include last writer wins using a logical clock (like a Lamport timestamp or vector clock), explicit version numbers that increment on each change, or operational transformation that merges concurrent updates. These are complex and often require application level logic, not just CDC infrastructure. Long Running Transactions: A transaction that holds locks for 30 seconds (maybe a large batch update) is invisible to CDC until it commits. When it finally commits, CDC emits all its changes in a burst, potentially thousands of events in a few milliseconds. This can overwhelm downstream systems that expect a steady rate. Production systems handle this with rate limiting or buffering in the CDC connector, or by designing consumers to absorb temporary spikes. Monitoring transaction durations and alerting on long running transactions can also prevent surprises.