Baseline Selection and Windowing Strategy

Choosing the reference baseline is a first order design decision that determines what changes you detect. A static training baseline compares production traffic against the original training data distribution, catching long term model decay and distribution shifts from when the model was built. However, this approach over alerts as products evolve naturally through seasonality, marketing campaigns, and user behavior changes. A rolling baseline compares against recent production data (for example, the last 7 to 28 days), automatically adapting to seasonality and product evolution but risking that you slowly learn in harmful drift without noticing. Most mature systems run both baselines in parallel. Alert if either breaches thresholds, but prioritize differently: static baseline violations indicate potential need for retraining, while rolling baseline violations signal sudden acute incidents requiring immediate investigation. For systems with strong segmentation like geography or device type, stratified baselines per segment prevent false alerts when one region has a legitimate holiday while others operate normally. Windowing determines detection latency and statistical power. Tumbling windows (for example, hourly or daily boundaries) are simpler to implement and cheaper to compute but introduce rigid boundaries that can miss incidents occurring mid window. Sliding windows (for example, 2 hour window moving every 15 minutes) provide smoother detection and faster incident response at higher compute and memory cost due to overlapping aggregations. For a system processing 40,000 requests per second with 0.5% sampling, a 30 minute sliding window with 5 minute steps accumulates roughly 360,000 events per decision, sufficient for stable statistical tests across dozens of features and segments. Window size trades off detection latency against statistical stability. Smaller windows (15 to 30 minutes) catch incidents quickly but need higher sampling rates or traffic volume to reach minimum sample sizes of 1,000 to 5,000 events per feature per segment. Larger windows (2 to 24 hours) provide stable statistics but extend blast radius when incidents occur. Label delay drives window size in practice: if ground truth labels arrive 6 to 12 hours late, there is little value in detecting covariate drift within 10 minutes since you cannot validate whether model performance actually degraded until labels arrive.

Airbnb pricing model uses both a static baseline from training (6 months ago) and a rolling 14 day baseline; static violations trigger retraining evaluation, rolling violations trigger immediate incident response

Netflix recommendation system runs per country stratified baselines; Diwali in India does not trigger false alerts in US traffic, each region compared against its own last 4 weeks

Uber demand prediction uses 15 minute sliding windows (step 3 minutes) for real time surge pricing decisions, providing 12 minute average detection latency for traffic pattern shifts during major events