Learn→Computer Vision Systems→Image Preprocessing (Augmentation, Normalization)→5 of 6

Computer Vision Systems • Image Preprocessing (Augmentation, Normalization)Hard⏱️ ~3 min

Common Preprocessing Failure Modes in Production

Training-Serving Skew
The most common preprocessing failure: training and serving use different preprocessing code. Accuracy drops 10-30% with no obvious error. The model runs, predictions come out, but they are wrong.
Common causes: Different image libraries (PIL vs OpenCV). Different resize interpolation methods. Normalization values hardcoded differently in training and serving code. RGB vs BGR channel order mismatch.
Prevention: Share a single preprocessing function between training and serving. Package preprocessing code with the model artifact. Test that training and serving produce identical tensors for the same input image.
Silent Data Corruption
Truncated images: Partial downloads or disk errors produce incomplete JPEG files. Some decoders crash, others return partial images. Neither behavior is correct for production.
Color profile mismatches: Images with embedded ICC profiles decode differently across libraries. A calibrated photo might render with shifted colors.
Detection: Validate image dimensions after loading. Check for unusual pixel value distributions. Log decode failures separately from inference failures.
Augmentation Bugs
Label-breaking transforms: Horizontal flip is safe for object detection but breaks text recognition. Rotation is safe for single objects but breaks multi-object relationships. Verify augmentations preserve label validity.
Distribution shift: Aggressive augmentation can push images out of natural distribution. Over-rotated, over-blurred images may harm performance. Monitor validation accuracy as you add augmentations.
Key Warning: Preprocessing bugs produce no errors - just wrong predictions. Always verify tensor equality between training and serving pipelines with test images.

💡 Key Takeaways

✓Training-serving skew from different preprocessing code causes 10-30% accuracy drops with no errors

✓Share single preprocessing function between training and serving to prevent skew

✓Truncated images and color profile mismatches cause silent data corruption

✓Some augmentations break labels (horizontal flip breaks text, rotation breaks relationships)

📌 Interview Tips

1Interview Tip: Mention training-serving skew as a top production ML bug - it produces no errors, just wrong predictions

2Interview Tip: Describe prevention strategy: package preprocessing code with model, test tensor equality on sample images

← Back to Image Preprocessing (Augmentation, Normalization) Overview