Failure Modes and Edge Cases in Deep Learning Time Series Forecasting

Production time series forecasting systems face several critical failure modes that can silently degrade performance or cause catastrophic errors. Understanding these edge cases is essential for building robust systems that handle real world messiness. Regime shifts and concept drift are the most dangerous. A promotion, supply shock, policy change, or external event like a pandemic can invalidate learned dynamics overnight. Models trained on pre shift data will produce optimistic intervals that miss by multiples. For example, a retail demand model trained through 2019 might predict intervals that are 3x too narrow during 2020 supply disruptions, leading to severe stockouts. Drift detectors must watch both error distributions and covariate distributions. If recent 7 day Weighted Absolute Percentage Error (WAPE) exceeds training WAPE by more than 50%, or Population Stability Index (PSI) on input features crosses 0.15, trigger immediate retraining or failover to adaptive baselines like exponential smoothing that react faster. Intermittent and sparse demand breaks standard losses. Many Store Keeping Units (SKUs) have long zero runs punctuated by occasional spikes. Training with Mean Squared Error (MSE) on these series causes models to predict a constant near zero mean, which gives low training loss but terrible service levels. The solution is intermittent aware losses like Tweedie or zero inflated distributions, hierarchical aggregation to smooth sparsity, and global models with item embeddings that borrow strength. Critically, do not evaluate with Mean Absolute Percentage Error (MAPE), which explodes when actuals are zero. Instead use forecast bias (mean error) and service level metrics like fill rate at P90. Attention scaling limits hit hard at length beyond 512. For context 1024 with batch 32 and 8 heads, a single Transformer attention layer requires roughly 1 gigabyte for attention weight storage. Without sparsity or local attention, serving on Central Processing Unit (CPU) becomes infeasible. Even with Graphics Processing Units (GPUs), Out Of Memory (OOM) errors during training are common. The fix is to cap practical context at 192 to 336, use block or local attention, and downsample distant history using moving averages, keeping only recent steps at full resolution. Exposure bias occurs when training with teacher forcing (feeding ground truth at each step) but serving autoregressively (feeding predictions). The model never sees its own errors during training, so mistakes compound during multi-step generation. This manifests as MAPE doubling from horizon 1 to horizon 24. Use scheduled sampling during training to gradually mix in model predictions, or prefer direct multi-horizon outputs that avoid autoregressive cascading entirely. Data leakage is insidious. Using future information that would not be available at prediction time yields optimistic backtests that collapse in production. Calendar features like day of week are safe, but promotional schedules, weather forecasts, or future covariate values must be restricted to what is truly known ahead. Always validate by simulating the exact information set available at decision time. A common leak is using normalized series statistics computed on the full dataset including future, rather than only on history up to the forecast origin.

Retail stockout from regime shift: COVID demand surge in March 2020 caused models trained on 2019 data to under predict by 200 to 400%, triggering emergency retraining with recent 30 day windows and safety stock overrides at 2x P90

Intermittent SKU failure: Long tail item with 90% zero demand trained with MSE predicts constant 0.5 units, achieving low loss but 60% service level; switching to Tweedie loss and global model with category embedding improves to 85% service level

Transformer OOM in production: Increasing context from 336 to 768 to capture longer seasonality caused 4 GB memory spike per batch, triggering OOM during peak load; fixed by capping at 384 and using dilated convolutions for distant history downsampling