Learn→ML Infrastructure & MLOps→Model Packaging (Docker, ONNX, SavedModel)→3 of 6

ML Infrastructure & MLOps • Model Packaging (Docker, ONNX, SavedModel)Medium⏱️ ~3 min

Docker Containers for Model Serving: Building Lean Inference Images

Model Serving Containers: Docker images containing the model, runtime dependencies, and inference server. Containers provide isolation (no dependency conflicts), reproducibility (same image runs identically everywhere), and scalability (spin up identical replicas on demand).
Why Containers for ML
ML models have complex dependency chains: framework versions, CUDA libraries, system packages, Python environments. Without containers, version conflicts between models are common (Model A needs TensorFlow 2.10, Model B needs 2.14). Containers isolate each model with its own environment. They also enable infrastructure teams to deploy models without understanding ML—they deploy containers, not Python code. The container is the contract between ML and infrastructure.
Base Image Selection
Start with official ML framework images that include CUDA, cuDNN, and framework installation already optimized. Building from scratch (ubuntu base, install CUDA manually) is error-prone and produces larger images. For CPU inference, use slim Python images to minimize size. For GPU inference, use CUDA-enabled base images matching your target GPU architecture. Image size matters: a 10GB image takes minutes to pull, delaying scaling and recovery.
Inference-Only Dependencies
Training images include tools not needed for serving: tensorboard, experiment tracking, data loading utilities. Inference images should contain only what is needed to load the model and run predictions. Audit dependencies ruthlessly: remove training-only packages, use inference-optimized framework builds where available. A TensorFlow training image might be 5GB; an inference image with TF Serving can be under 1GB. Smaller images mean faster deployments and lower storage costs.
Image Layering: Order Dockerfile commands from least to most frequently changing. Base image and framework first (changes rarely), dependencies next (changes sometimes), model files last (changes often). This maximizes layer caching and minimizes rebuild time.

💡 Key Takeaways

✓Containers isolate each model with its own dependency environment

✓Use official ML framework base images rather than building CUDA from scratch

✓Remove training-only dependencies to shrink inference images from 5GB to under 1GB

📌 Interview Tips

1Order Dockerfile: base image first, model files last for layer caching

2TensorFlow training image 5GB vs TF Serving inference image under 1GB

← Back to Model Packaging (Docker, ONNX, SavedModel) Overview