Precision Tradeoffs: FP32 vs FP16 vs INT8

Understanding Precision Levels
FP32 (32-bit float): Full precision, baseline accuracy, no conversion needed. FP16 (16-bit float): Half memory, ~2x faster on modern GPUs with tensor cores, typically <0.5% accuracy loss. INT8 (8-bit integer): Quarter memory of FP32, 2-4x faster than FP16, requires calibration, 0.5-2% accuracy loss typical. The speedups compound with compilation optimizations.
INT8 Calibration
INT8 requires mapping the FP32 value range to 256 discrete levels. Calibration runs the model on representative data, collects activation ranges per layer, and computes optimal scaling factors. Poor calibration (unrepresentative data, too few samples) causes accuracy to collapse. Best practice: use 1000+ samples spanning input diversity. Some layers (final classifier, attention) are sensitive; keep them in FP16 while quantizing others to INT8 (mixed precision).
When Each Precision Makes Sense
FP32: Training, debugging, accuracy-critical production where latency isn"t an issue. FP16: Default for GPU inference; nearly free speedup with minimal risk. INT8: High-throughput serving, edge deployment, cost-sensitive inference. Mixed precision: Best accuracy-speed tradeoff for complex models; sensitive layers stay FP16, bulk of computation in INT8.
⚠️ Gotcha: INT8 speedup requires hardware support (tensor cores, VNNI). On CPUs without VNNI, INT8 may run slower than FP32 due to quantization/dequantization overhead.
Accuracy Validation Protocol
Compare against FP32 baseline on held-out data. Acceptable thresholds: FP16 should be within 0.1% of FP32; INT8 within 0.5-1%. Test edge cases and low-confidence predictions specifically; quantization errors concentrate where activations are small or near decision boundaries.

💡 Key Takeaways

✓FP16: ~2x speedup, <0.5% accuracy loss; INT8: 2-4x over FP16, 0.5-2% accuracy loss typical

✓INT8 calibration needs 1000+ representative samples; poor calibration causes accuracy collapse

✓Mixed precision: sensitive layers (classifier, attention) in FP16, bulk computation in INT8

✓INT8 requires hardware support (tensor cores, VNNI); without it, may run slower than FP32

✓Accuracy thresholds: FP16 within 0.1% of FP32, INT8 within 0.5-1%

📌 Interview Tips

1Explain INT8 calibration process with sample count recommendation (1000+) for practical credibility

2Mention mixed precision strategy for attention layers - shows nuanced understanding

3Warn about hardware requirements for INT8 speedups - demonstrates production awareness

← Back to Model Compilation (TensorRT, ONNX, TVM) Overview