How Backfill & Reprocessing Work

The Foundation: Immutable Raw Storage The entire strategy depends on one architectural principle: your raw data must be immutable and retained. Companies like LinkedIn and Netflix keep months or years of compressed event logs in object storage (S3, GCS, HDFS). This becomes your source of truth. When you need to backfill or reprocess, you always go back to these raw logs, never to live production databases. Why? Because production databases change. A user record today looks different than it did 6 months ago. But raw event logs capture exactly what happened at that moment in time. The Backfill Workflow: A typical backfill follows this pattern: Real Numbers: At Uber scale, a single day partition might contain 500 GB to 2 TB of raw events. Processing 90 days in parallel across a cluster with 500 workers, each handling 2 partitions per hour, completes in roughly 1 hour of wall clock time. But you need throttling: running at 100 percent cluster capacity starves production jobs. Most teams target 20 to 30 percent of cluster resources for backfills, extending completion to 3 to 5 hours but maintaining production Service Level Agreements (SLAs).