CDC Failure Modes and Edge Cases

When Consistency Guarantees Break: Theoretical CDC consistency is clean: transactional capture, ordered delivery, idempotent consumption. Production CDC is messy: log retention windows expire, schemas evolve incompatibly, clocks skew, and poison pill events crash consumers. Understanding these failure modes separates candidates who've built real systems from those who've only read documentation. Log Truncation and the Unrecoverable Gap: Databases don't retain transaction logs forever. PostgreSQL's Write Ahead Log typically keeps 16 MB segments with retention based on replication lag, MySQL binary logs might retain 7 days. If your CDC connector falls behind by more than the retention window, the database deletes log segments you haven't processed yet. The math is unforgiving. A database generating 500 MB per hour of log data with 7 day retention gives you 84 GB of buffer. If a CDC outage lasts 8 days, you've lost 24 hours of changes permanently. The only recovery is a full resnapshot, which for a 10 TB database takes 12 to 18 hours and creates a consistency gap during the cutover. Production systems combat this with aggressive monitoring. They track replication lag in both time (minutes behind) and position (LSN delta). Alerts fire at 15 minute lag, pages fire at 30 minute lag, and runbooks trigger emergency capacity increases or snapshot preparation at 2 hour lag, all well before hitting the 7 day retention limit. Schema Evolution and Breaking Changes: A developer renames a column from user_email to email_address in the source database. CDC events suddenly contain email_address instead of user_email. Consumers expecting the old schema crash or silently drop the field. Without schema governance, this breaks consistency. Consumers that inserted records successfully before the change now fail on every event, stalling the entire partition. Or worse, they succeed but write NULL for the email field, corrupting downstream data. Mature CDC systems use schema registries with compatibility enforcement. Before publishing an event with a new schema version, the producer validates compatibility (backward, forward, or full). Backward compatibility means old consumers can read new events. Forward compatibility means new consumers can read old events. The registry rejects incompatible changes, forcing coordinated rollouts. Poison Pill Events and Stalled Partitions: A malformed event enters the stream: perhaps a bug in the CDC connector serializes a NULL value as an invalid JSON string, or a database trigger writes a 50 MB TEXT field that exceeds consumer memory limits. The first consumer to read this event crashes. It restarts, reads the same event, crashes again. The partition is stuck. With strong ordering guarantees, you cannot skip this event without violating "no data loss." But you also cannot make progress. Production systems handle this with dead letter queues: after N failed attempts (typically 3 to 5), the consumer writes the problematic event to a quarantine topic, logs detailed diagnostics, and continues processing from the next event. Operators manually review quarantined events to fix bugs or data corruption. Clock Skew and Temporal Semantics: CDC ordering is based on commit sequence (LSN), not wall clock time. But downstream systems often use timestamps for windowing or late event handling. If the source database's clock is 10 minutes fast, events arrive with future timestamps. If a fraud detection system uses tumbling windows based on event time, these future timestamped events land in the wrong windows. This isn't a CDC bug, it's a temporal semantics mismatch. Fixing it requires watermarking strategies: downstream systems track the maximum timestamp seen per partition and reject or quarantine events with timestamps more than some threshold (like 5 minutes) ahead of the watermark. Or they switch to processing time semantics, using arrival time instead of event time.