Training Infrastructure & Pipelines • Continuous Training & Model RefreshMedium⏱️ ~3 min
Retraining Strategies: Batch vs Incremental vs Hybrid
Three retraining strategies dominate production systems, each with distinct trade offs. Full batch retrain starts from scratch on a sliding window (last 7 to 28 days), recomputing all weights. This is the safest and most reproducible approach, used by Netflix for nightly homepage personalization retrains and Airbnb for weekly Smart Pricing updates. The cost is high compute (training clusters must handle peak load) and slow reaction (cannot deploy faster than training completes, typically hours to days for large models).
Incremental or online updates continue training from the previous model checkpoint, processing only new data. Meta uses this for embedding layers and calibrators, updating every 15 to 60 minutes to capture trending topics in feeds. The benefit is speed and low cost (only process deltas), but risks accumulate: catastrophic forgetting (model loses older patterns), bias drift (overweights recent data), and harder validation (no clean baseline). Incremental updates work best for stateless components like embeddings or when paired with periodic full retrains.
Hybrid strategies combine both: Uber runs daily full retrains for core ETA models while updating nearline contextual features (traffic conditions, weather) every 5 minutes. This balances stability (full retrain prevents drift accumulation) with responsiveness (nearline features capture real time conditions). The pattern is periodic full retrain (daily to weekly) plus frequent small updates (minutes to hours) for fast moving signals. Cost wise, hybrid is the most expensive but delivers the best accuracy for high value use cases like fraud detection or dynamic pricing.
💡 Key Takeaways
•Full batch retrain on 7 to 28 day sliding windows is safest and most reproducible but costs high compute and cannot react faster than training completes (hours to days for large models like Netflix homepage personalization)
•Incremental updates reduce cost by 10x and enable 15 to 60 minute cadences (Meta embedding updates) but risk catastrophic forgetting, bias drift toward recent data, and harder offline validation without clean baselines
•Hybrid strategies balance stability and responsiveness: Uber runs daily full retrains for core ETA models while updating nearline traffic and weather features every 5 minutes, costing 3x more but improving accuracy by 5 to 8 percent
•Warm start initialization from previous checkpoint speeds convergence by 30 to 50 percent for full retrains and preserves learned patterns, but requires careful learning rate tuning to avoid getting stuck in local minima
•Incremental updates work best for stateless components (embeddings, calibrators) or when paired with weekly full retrains to prevent drift accumulation and catastrophic forgetting over time
📌 Examples
Meta updates ad ranking embeddings incrementally every 15 minutes to capture trending topics, but runs full retrains daily to reset and prevent bias accumulation, maintaining AUC ROC within 0.5 percent of batch trained models
Netflix runs nightly full retrains for recommendation models processing hundreds of millions of interactions, using warm start from previous day to reduce training time from 8 hours to 5 hours