Idempotency at Production Scale

The Architecture: At companies like Uber and Netflix, idempotency is designed into every layer of the pipeline, from ingestion to final materialization. The pattern that works at petabyte scale separates concerns: raw ingestion is append only and cheap, while derived tables enforce strict idempotency. Consider an ads analytics platform processing 200,000 events per second at peak. Mobile and web clients send impression and click events into Kafka. The raw topic is append only, accepting duplicates to maximize ingestion throughput. Write latency stays under 10ms at p99 because there's no dedupe or coordination overhead. Layer by Layer: The streaming layer computes real time metrics with 10 second end to end latency at p99. It uses windowed deduplication: maintains a set of seen event_id values for the last 2 hours in memory, sharded by key hash across worker instances. For 200,000 events per second over 2 hours, that's 1.44 billion event IDs to track. Using a compact representation, this fits in 10 to 20 GB of memory distributed across the cluster. The batch layer recomputes daily and hourly aggregates. Jobs are designed to be fully rerunnable. A daily job for campaign metrics computes from the raw append only log and writes results with a composite key of campaign_id plus date. If yesterday's job had a bug, you fix the code and rerun it. The upsert overwrites the bad data. The data warehouse receives cleaned, deduped events via a batch loader that runs hourly. Each load is idempotent because it either replaces entire partitions (for example, DROP PARTITION for hour X, then INSERT new data for hour X) or uses MERGE statements keyed by event_id. Backfills work the same way: to reprocess March 2024, you run the pipeline for those dates and let it overwrite the existing partitions. Feature Store Integration: ML feature stores have tight freshness requirements, often less than 5 minutes for 95% of features. These systems use upserts keyed by user_id plus feature_name plus timestamp. When a pipeline recomputes features for the last hour due to late data, it simply upserts the new values. The inference service always reads the latest value per key.