What is CI/CD for ML and Why It's Different

Continuous Integration and Continuous Deployment (CI/CD) for machine learning extends traditional software delivery to treat data, features, and models as first class artifacts alongside code. Unlike a typical web service where you deploy stateless binaries, ML systems must manage training datasets (often terabytes), feature pipelines, model binaries (sometimes gigabytes), hyperparameters, and rich metadata that captures lineage.

The critical difference is that models are trained on historical data and validated on offline benchmarks that may not reflect live traffic behavior. A recommendation model might achieve 0.85 precision at k equals 10 on last week's logs, but when deployed it encounters shifted user behavior, missing features due to upstream latency, or numerical differences between training and serving environments. This introduces risks like training serving skew, data drift, and model decay that don't exist in traditional software.

To address these risks, ML CI/CD splits into three distinct loops. Continuous Integration validates code, feature transformations, and lightweight data contracts in under 10 minutes using small fixtures and unit tests. Continuous Training runs on a schedule or drift trigger, trains on production scale data (for example 8 terabytes covering 14 days), and produces a candidate model with captured lineage including commit hash, data snapshot IDs, and computed metrics. Continuous Deployment moves the promoted model through staging, shadow (where it sees live traffic but doesn't affect users), and progressive canary rollouts with automated guardrails that watch latency, error rates, and business metrics like click through rate or fraud detection recall.

Production systems at scale operationalize this rigorously. Uber Michelangelo manages thousands of models with hourly and daily retraining, model registries that track full lineage, and online feature stores delivering features in under 10 milliseconds. Netflix runs shadow evaluations before promotion for personalization models while maintaining p95 inference times of 20 to 40 milliseconds. The pipeline must prove not only functional correctness but also statistical performance against explicit baselines and slice based fairness metrics before any model touches user traffic.

💡 Key Takeaways

•ML CI/CD manages code plus data, features, and model artifacts as versioned, immutable dependencies with full lineage tracking

•Training serving skew is the silent killer: Different transformations, missing value handling, or numerical libraries between training and serving cause metric drops that offline validation misses

•Three loop architecture separates concerns: CI completes in under 10 minutes for fast feedback, continuous training operates on production scale data, CD uses shadow and canary with automated guards

•Shadow deployment surfaces skew and latency issues without user impact by logging predictions on live traffic for a fixed budget like 10 million requests over 2 hours before promoting to canary

•Progressive rollouts start at 1 percent traffic for 30 minutes, then 5 percent for 2 hours, with automated rollback triggering in under 2 minutes if p95 latency exceeds 50ms or business KPIs drop more than 2 percent

•Production scale examples: Uber Michelangelo manages thousands of models with sub 10ms feature fetches, Netflix maintains 20 to 40ms p95 inference for personalization, Google TFX formalizes data validation and skew detection

📌 Examples

Netflix personalization pipeline: Shadow evaluates candidate model on 24 hours of production request logs (replayed offline) to compute incremental click through rate and calibration curves before any live traffic exposure

Uber fraud detection: Model serves 40k requests per second at p95 latency under 50ms, with canary rollout watching tail p99, throughput, feature cache hit rate, and fraud catch rate across traffic slices

Meta learning platforms: Run thousands of training jobs daily, promote through automated policy checks (AUC improvement greater than 0.5 points, calibration slope within 0.02 on top 5 slices), progressive rollouts tied to online experiments

← Back to CI/CD for ML Overview