Failure Modes and Production Reliability

Production image classification systems face diverse failure modes that can cause severe accuracy drops, cost overruns, or service outages if not properly handled. Training serving skew occurs when preprocessing differs between training and inference environments. Using different resize algorithms, crop strategies, or normalization constants can drop accuracy by 2 to 5 percentage points. Teams lock preprocessing as versioned components shipped with model checkpoints and test end to end using real serving code in shadow mode against production traffic to catch discrepancies before rollout. Domain shift and data drift cause accuracy collapse when the production data distribution diverges from training data. A model trained on professional studio product photos then deployed on user generated smartphone content can see recall drop from 85% to 60%. Detection requires monitoring population stability metrics on embedding distributions and label frequencies. When drift exceeds thresholds, for example Kullback Leibler (KL) divergence above 0.15 between training and serving embeddings, teams trigger active learning loops that sample hard examples for human labeling and launch fine tuning jobs to adapt the model. Class imbalance creates severe long tail problems. Common classes with millions of examples dominate the loss function, while rare classes with hundreds of examples achieve near zero F1 scores in production. A category appearing in 0.01% of images effectively becomes invisible to standard training. Solutions include reweighting loss by inverse class frequency, hard example mining that surfaces misclassified instances, and semi supervised learning using pseudo labels on unlabeled data. These techniques can improve rare class recall from under 10% to 60 to 70%. Corrupt or exotic inputs cause decoder crashes and mislabeling. Real production traffic includes truncated JPEGs from interrupted uploads, extreme aspect ratios like 10:1, CMYK color spaces instead of RGB, animated GIF formats, and EXIF rotation metadata that flips or rotates images. Without robust handling, these cause exceptions or silent mislabeling. Production pipelines enforce size and format limits, normalize EXIF orientation, add fallback decoders, and route malformed media to quarantine queues for separate handling. Cache invalidation storms happen when model updates invalidate billions of cached predictions simultaneously, creating thundering herd traffic that overwhelms inference tiers. Teams use versioned cache keys, rate limited background refresh, and gradual rollout to spread load over hours instead of seconds.

Pinterest training serving skew: Training used OpenCV resize, serving used Pillow resize with different interpolation, caused 3.2 point accuracy drop, fixed by shipping OpenCV in serving container

Amazon product classification drift: Model trained pre pandemic on 80% studio photos fails when user uploads shift to 60% smartphone photos during lockdown, KL divergence 0.22 triggers retraining, new model recovers recall to 83%

Google Photos rare class: Wedding category appears in 0.008% of images, standard training yields 2% F1, apply 100x loss weight + pseudo label 500K images with 0.98 confidence, improves F1 to 68%

Meta cache invalidation: Model update invalidates 2 billion cached predictions, naive refresh causes 50,000 QPS spike overwhelming inference tier, gradual rollout at 500 QPS over 48 hours completes refresh without incident