Cold Start Problem: Model Loading and Predictive Warming Strategies

Cold start latency is the dominant constraint for GPU autoscaling in real time inference. The total delay from scaling decision to serving traffic includes node provisioning (60 to 120 seconds for cloud VMs), GPU driver initialization (20 to 40 seconds), container image pull for GPU optimized images (30 to 90 seconds depending on registry proximity), and model weight loading from object storage (100 to 300 seconds for multi gigabyte models). A large language model at 10GB can take 180+ seconds just to load weights into GPU memory, exceeding most p99 latency SLOs. Reactive autoscaling cannot hide this delay. If your autoscaler triggers on queue depth exceeding 50 requests and takes 240 seconds to bring a new replica online, every request during that window experiences degraded latency or timeouts. A sudden traffic spike from 100 to 1000 requests per second causes immediate SLO violations while the system scrambles to add capacity. Production systems address this through multiple strategies operating at different timescales. Predictive autoscaling uses traffic forecasts or reinforcement learning based controllers to anticipate load and pre warm capacity minutes before spikes occur. Google and Meta use historical patterns (like daily/weekly seasonality) plus real time signals to stage GPU nodes and load models proactively. The risk is over provisioning if forecasts drift, requiring cost guardrails and maximum capacity caps. Warm pools maintain a small baseline of ready replicas even during idle periods, trading idle cost (perhaps one replica at $2/hour) for instant response to initial traffic. Scale to zero saves cost but accepts cold start penalties for the first requests. Model loading optimization matters significantly. Techniques include splitting weights into chunks for parallel download, using nearby object storage or content delivery networks to reduce transfer time from 200 seconds to 60 seconds, lazy loading layers progressively as requests arrive, and caching popular models on persistent volumes attached to GPU nodes. Health check grace periods must account for load time: a 180 second grace period prevents Kubernetes from killing pods that are legitimately initializing. Termination grace periods (like 600 seconds) allow in flight inference to drain before shutdown, preventing request failures during scale down.

Production configuration: health check grace period of 180 seconds accommodates model loading; termination grace of 600 seconds allows in flight inference to complete before pod shutdown

Large language model serving uses persistent volumes to cache 10GB model weights on GPU nodes, reducing subsequent pod startup from 180 seconds to 45 seconds by skipping object storage download

Predictive autoscaler at Meta uses reinforcement learning on traffic patterns to pre warm GPU capacity 8 minutes before daily peak, maintaining p99 latency under 150ms target during traffic increase from 500 to 5000 requests per second