Production Architecture and Implementation Patterns

Drift detection must run completely off the inference path to preserve latency Service Level Objectives (SLOs). Production models typically target p99 inference latency under 50 to 100 milliseconds; running statistical tests synchronously would violate this budget. The standard pattern is off path logging: the inference service emits feature vectors, predictions, and metadata to a durable log (Kafka, Kinesis, Pub/Sub) with fire and forget semantics adding less than 1 millisecond to request latency. A separate drift detection pipeline consumes this log, aggregates statistics, and runs tests asynchronously. Two layer monitoring architectures balance coverage and cost. Layer 1 runs fast univariate tests (Population Stability Index, Kolmogorov Smirnov, Chi square) on high importance features every 15 to 30 minutes, providing broad coverage at low compute cost. For a ranking service at 40,000 to 80,000 requests per second sampling 0.5% of traffic, computing 10 bin histograms for 60 features across 10 segments on 30 minute windows requires sub 100 millisecond CPU time per segment on commodity cores; an 8 to 12 virtual CPU (vCPU) pool keeps p95 detection latency under 3 minutes. Layer 2 runs expensive multivariate tests (Maximum Mean Discrepancy, adversarial validation) on flagged features or on a rotation schedule, providing deeper investigation without constant overhead. Data structures for online computation enable efficient streaming aggregation. For continuous features, maintain fixed bin histograms (typically 10 to 50 bins) for fast PSI computation, t-digest or Greenwald Khanna (GK) sketches for quantile estimation, and running mean/variance/skewness using Welford online updates. For high dimensional embeddings (768 dimension sentence vectors), project to 32 to 64 dimensions via Principal Component Analysis (PCA) and maintain mean/covariance on the projection; computing Maximum Mean Discrepancy on 50 dimensional projections is 200x faster than full 768 dimensional pairwise distances while capturing 90% plus of variance. Tying detection to automated response requires guardrails to prevent false positive disasters. Practical systems require drift to persist across at least 2 consecutive windows, breach both statistical significance (p less than 0.01) and effect size thresholds (PSI greater than 0.25 OR Wasserstein greater than 0.1 times feature interquartile range), affect at least 10% to 20% of production traffic, and correlate with business metric degradation (Click Through Rate, conversion, fraud rate dropping 2 standard deviations). Only then do they trigger hard responses like auto retraining, feature disabling, or traffic throttling, and even these go through canary validation.

Netflix recommendation pipeline at 50,000 requests per second with 0.2% sampling feeds 100 events per second to drift monitoring; 8 vCPU pool computes histograms and PSI for 60 features times 10 segments in under 3 minutes p95

Meta feed ranking uses t-digest sketches maintaining 100 centroids per feature per segment, providing accurate quantile estimates in 2 kilobytes memory per feature while processing 15 million messages per hour

Uber fraud detection requires drift to persist 6 hours (12 consecutive 30 minute windows), PSI greater than 0.3, affect more than 25% of transactions, AND fraud detection rate to drop 3 standard deviations before triggering auto retraining with shadow evaluation