ONNX: The Universal Intermediate Format

Definition
ONNX (Open Neural Network Exchange) is a standard format for representing ML models. It defines a common set of operators and a file format, allowing models trained in PyTorch to run in TensorFlow, or anywhere that reads ONNX.
How ONNX Works
An ONNX file contains a computation graph: nodes (operations like Conv, MatMul, ReLU), edges (tensor connections), and initializers (trained weights). The format is framework-agnostic; it represents what the model computes, not how PyTorch or TensorFlow implemented it. Export: torch.onnx.export(model, sample_input, "model.onnx"). The exporter traces your model on the sample input, recording all operations.
ONNX Runtime
ONNX Runtime is an inference engine that runs ONNX models with optimizations. It applies graph optimizations (constant folding, operation fusion), selects efficient kernel implementations, and supports multiple backends (CPU, CUDA, DirectML, TensorRT). Typical speedups over PyTorch: 1.5-3x. Not as fast as pure TensorRT but works on more hardware and requires less tuning.
Common Export Issues
Dynamic control flow (if statements depending on tensor values) doesn"t export cleanly. The exporter traces one path and bakes it in. Custom operators need explicit ONNX registration. Some operations (like torch.unique) lack ONNX equivalents. Fix: rewrite unsupported ops, use opset version 14+ for better coverage, or fall back to TorchScript for dynamic models.
💡 Tip: Always validate ONNX output matches PyTorch output on sample inputs before deployment. Export can silently produce incorrect graphs.

💡 Key Takeaways

✓ONNX is a framework-agnostic format: nodes (operations), edges (tensors), initializers (weights)

✓Export traces model on sample input, recording operations; dynamic control flow doesn"t export cleanly

✓ONNX Runtime provides 1.5-3x speedup over PyTorch with graph optimizations and multiple backends

✓Common issues: unsupported ops (torch.unique), custom operators need registration, control flow limitations

✓Always validate ONNX output matches original framework output before deployment

📌 Interview Tips

1Mention ONNX opset versions when discussing compatibility - shows awareness of versioning complexities

2Describe tracing limitations (dynamic control flow) as a key export pitfall

3Cite 1.5-3x ONNX Runtime speedup range to set realistic expectations

← Back to Model Compilation (TensorRT, ONNX, TVM) Overview