Training Serving Skew Detection and Prevention

Training serving skew occurs when feature values computed during offline training differ from the same features computed during online inference, even for identical raw inputs. This is one of the most insidious failure modes in production ML because model accuracy metrics during offline evaluation look excellent while live performance degrades by 5 to 20 percent. The root causes are subtle: batch feature computation uses different code paths than real time serving, time zone handling differs between systems, aggregation windows use different boundary semantics, null handling has different defaults, or floating point precision varies between training (often 64 bit) and serving (often 32 bit for speed). Detection requires dual write comparison: computing features through both the batch training pipeline and the online serving pipeline for the same sample of entities, then measuring agreement. For numeric features, define agreement as values within a tolerance (typically 0.1 percent relative error or 0.01 absolute for normalized features). For categorical features, require exact string match after normalization. Production systems sample 1 to 5 percent of traffic, log both batch and online feature values with shared join keys (user_id, timestamp, request_id), then run hourly or daily comparison jobs. At Meta, feed ranking features maintain a parity dashboard showing per feature agreement rate, with alerts when any critical feature drops below 99 percent exact match or 99.5 percent within tolerance match. Prevention strategies start with code reuse. The gold standard is feature computation logic defined once and executed in both batch (Spark, Beam) and streaming (Flink, Kafka Streams) contexts through a feature transformation framework. Netflix uses a shared feature DSL that compiles to both Spark SQL for batch training pipelines and Java for online microservices, ensuring identical semantics. When full code sharing is infeasible, rigorous integration tests become critical: generate synthetic test cases covering edge cases like nulls, boundary timestamps, and extreme values, compute features through both paths, assert bitwise or tolerance based equality. Uber maintains 500 plus test cases per feature category (user features, trip features, geospatial features) that run on every commit. The most subtle skew sources are temporal. A batch job computing "user 7 day purchase count" on date D processes all events with timestamp less than end of day D in the training data time zone (often UTC), while the online system at 14:00 local time on day D plus 7 sees a different event set due to time zone conversion, late arriving events, or event time versus processing time semantics. Mitigation requires time travel testing: replay historical requests through the online serving path and compare to batch computed ground truth. Airbnb pricing model replays 10,000 historical pricing requests daily, measuring that 98.5 percent of features match within tolerance; when this drops below 98 percent, it indicates skew introduction and blocks deployment.

Meta News Feed: feature for "user engagement last 7 days" computed in batch used UTC day boundaries while online used local time then converted to UTC, causing 15 percent skew on users near time zone boundaries; fixed by standardizing to UTC event timestamps in both paths

Uber ETA: batch training computed average_speed using Float64 and Spark aggregations, online used Float32 and hand coded Java; 2 percent of predictions differed by more than 10 percent; unified via shared Scala functions compiled to both Spark and JVM bytecode

Airbnb pricing: late arriving booking events (5 to 10 percent arrive 1 to 3 hours late) included in batch 7 day aggregates but missed in online sliding windows; added event time watermarking with 3 hour lateness tolerance in online path to match batch semantics