Learn→ML Model Optimization→Model Pruning (Structured vs Unstructured)→3 of 6

ML Model Optimization • Model Pruning (Structured vs Unstructured)Medium⏱️ ~3 min

Production Implementation: Iterative Pruning and Fine Tuning

Key Insight
Iterative pruning outperforms one-shot pruning. Removing 10% of weights, fine-tuning, then repeating achieves higher accuracy than removing 50% at once. The network adapts gradually rather than experiencing catastrophic forgetting.
The Iterative Pipeline
Start with trained model. In each iteration: compute importance scores, remove lowest 10-20%, fine-tune for 2-5 epochs. Repeat until target sparsity. For 90% sparsity with 10% per iteration, expect 10 pruning rounds. Total fine-tuning time: 20-50 epochs spread across iterations. This sounds expensive but consistently achieves 1-2% better final accuracy than one-shot approaches at the same sparsity.
Layer Sensitivity
Not all layers tolerate pruning equally. Early layers (extracting basic features like edges) and final layers (making predictions) are most sensitive. Middle layers can often be pruned more aggressively. Common pattern: prune 30% from first layer, 60-70% from middle layers, 40% from last layer. Uniform pruning (same ratio everywhere) typically underperforms by 2-3% accuracy.
Lottery Ticket Hypothesis
A pruned network contains a "winning ticket": a subnetwork that, if trained from scratch with original initialization, matches full network accuracy. This suggests optimal sparse structures exist from initialization. Practical implication: you can prune once, reset to initial weights, retrain the sparse network. This sometimes outperforms prune-then-fine-tune but requires storing original weights.
💡 Production Tip: Always save checkpoints before each pruning iteration. If accuracy degrades unexpectedly, roll back one iteration and reduce pruning ratio.

💡 Key Takeaways

✓Iterative pruning (10-20% per round) outperforms one-shot by 1-2% accuracy at same sparsity

✓Each iteration: compute importance, prune 10-20%, fine-tune 2-5 epochs; 90% sparsity needs ~10 rounds

✓Layer sensitivity varies: early and final layers sensitive (30-40% max), middle layers can go 60-70%

✓Uniform pruning across layers underperforms layer-aware pruning by 2-3% accuracy

✓Lottery ticket hypothesis: sparse winning subnetworks exist at initialization and can be retrained

📌 Interview Tips

1Describe iterative vs one-shot trade-off with specific numbers (10 rounds, 1-2% accuracy gain)

2Mention layer sensitivity patterns when discussing pruning strategy - shows practical knowledge

3Reference lottery ticket hypothesis for bonus points - it"s a well-known research result interviewers may ask about

← Back to Model Pruning (Structured vs Unstructured) Overview