Failure Modes: Backfills, Hot Keys, and Drift in Denormalized Systems

Schema or logic changes to denormalized projections require backfilling: recomputing derived rows from the source of truth. The scale is daunting. Recomputing 100 billion feed rows at 50,000 rows per second takes roughly 23 days on a single worker. Even with 1,000 way parallelism, it is 33 minutes, but the I/O and Queries Per Second (QPS) burst can overwhelm storage and evict hot data from caches, causing global latency spikes. Production systems throttle backfills to a fixed budget, often capping at 10% of cluster capacity, and process in rolling keyspace windows to minimize cache churn. Meta and Pinterest evolve schemas by dual writing: add new fields to denormalized rows behind a feature flag, write both old and new schemas for a transition window (days to weeks), validate parity on sampled reads, then flip reads to the new schema and stop writing old. Backfill happens incrementally during the transition, spread over days. If logic changes (new ranking features), backfill in small key range batches (1 million rows per job) with rate limits and monitor cache hit rate drop; a 5% hit rate degradation during backfill can breach Service Level Objectives (SLOs) if not controlled. Hot keys are another critical failure mode. A viral post or celebrity activity can create a single key (like a post ID or user ID) accessed millions of times per second, overwhelming a single partition or cache shard. Denormalized counters mitigate by sharding increments: instead of a single counter row, store 64 or 128 shard buckets and route increments randomly; reads sum the shards. This distributes write load but adds read cost. For extremely hot keys, systems cache aggressively at multiple layers (application memory, distributed cache, Content Delivery Network (CDN)) and use approximate counting (HyperLogLog for cardinality, probabilistic data structures) accepting slight inaccuracy (1 to 2% error) for massive throughput gains. Counter drift and delete semantics are ongoing challenges. Approximate counters diverge due to dropped increments (network failures, consumer crashes) or duplicate processing (at least once delivery without perfect deduplication). Without reconciliation, displayed counts can drift by several percent on long lived objects. Periodic jobs (every 10 to 60 minutes) scan and recompute from the event log. Deletes require tombstone propagation: removing an entity means publishing delete events to all derived projections. Missed tombstones create ghost items (deleted posts still in feeds). Ensure delete events are idempotent, retryable, and carry a time to live to prevent infinite retention. Cross region lag exacerbates this: a delete in region A may not propagate to region B for 100 to 300 milliseconds; users in region B see the deleted item briefly, requiring explicit eventual delete indicators or optimistic hiding in the User Interface (UI).

Meta feed backfill strategy: when adding new ranking feature to denormalized feed rows, feature flag enables dual write of old and new features for 2 weeks; backfill runs at 5% of cluster write capacity in 10 million row keyspace batches; cache hit rate monitored and backfill paused if hit rate drops below 92% to prevent latency spikes

Pinterest viral pin hot key: single pin receiving 5 million views in 1 hour would overwhelm single counter partition; counter sharded into 128 buckets with random increment routing keeps per shard writes under 10,000 per second; read path sums 128 shards in parallel adding 5 milliseconds but preventing partition hotspot; approximate count using HyperLogLog considered for future with 1% error margin