Failure Modes and Edge Cases

When Schema Evolution Goes Wrong: The most insidious failure mode is an incompatible schema change that bypasses registry checks. Imagine compatibility enforcement is accidentally disabled or the wrong subject naming strategy is configured. A producer changes a field from order_amount (string) to order_amount (integer) without adding a new field name. The registry registers this breaking change. Consumers that have not updated their reader schemas now fail deserialization with cryptic type mismatch errors. This gets worse with historical data. A batch job reading three month old events suddenly crashes because it encounters the new incompatible schema mixed with old data. The fix requires either rolling back the schema change (if caught quickly) or deploying updated reader code to all consumers before the producer change, which defeats the purpose of independent evolution. Prevention requires strict governance: enforce compatibility modes (backward transitive at minimum) with monitoring and alerting on registry rejections. Partial Rollout Chaos: Multi region deployments create timing hazards. Region A upgrades producers to schema version 8, but region B consumers still only handle version 6. If compatibility is backward only (not forward), consumers in region B fail when they encounter fields from version 8 that they do not understand. The safe rollout sequence is always N plus 1: first upgrade ALL consumers globally to support both old and new schemas, validate with canary topics and metrics, then upgrade producers. Schema Registry as Single Point of Failure: If the registry becomes unavailable or p99 latency spikes beyond 200 ms, producers attempting to register new schemas will timeout or fail. Steady state traffic continues because clients cache schemas, but deployments requiring new schema versions are blocked. At scale, this is catastrophic. A deployment window for 500 microservices stalls because the registry is down. Mitigation requires multi node registry with synchronous replication and quorum writes (typically 3 or 5 nodes). Monitor registry Queries Per Second (QPS), p99 latency (target under 50 ms), and error rates. Implement cross region disaster recovery with careful schema identifier (ID) coordination to prevent divergence. Some companies run active passive registry clusters with automated failover. Subtle Edge Cases: Union types and nullable fields create tricky evolution paths. Changing a field from union [null, string] to union [null, int] is technically a type change and typically breaks compatibility, even though both are nullable. Another trap: schema retention policies. If Kafka uses log compaction or short retention (7 days) but the registry keeps schemas forever, you accumulate thousands of versions for topics that no longer exist. Implement schema lifecycle governance: deprecate unused schemas, archive old versions, and enforce retention limits per subject.