ML Model Optimization • Batch Size & Throughput TuningMedium⏱️ ~3 min
Training Batch Size: Memory, Convergence, and Throughput Trade-offs
In training, batch size controls how many examples you group before computing a gradient and updating model weights. This choice directly impacts three critical dimensions: memory usage, convergence behavior, and training throughput measured in examples per second.
Start with the largest batch that fits in GPU memory, typically 256 to 512 examples for models like BERT or ResNet on a single GPU. Larger batches reduce optimizer overhead because you perform fewer weight updates per epoch, and they improve hardware utilization by keeping compute units busy with bigger matrix operations. However, very large batches can converge to sharper minima that sometimes hurt generalization, and they require careful learning rate scaling and warmup schedules to maintain convergence quality.
Small batches of 32 or 64 produce noisy gradients that act as implicit regularization, helping the model escape sharp minima and often improving final validation accuracy. The downside is more optimizer steps, which means more time to reach target accuracy. In practice, teams balance these factors by choosing the largest batch that fits memory, then adjusting learning rate proportionally and using warmup to stabilize early training.
When memory is the constraint, gradient accumulation lets you emulate large effective batch sizes. You run forward and backward passes on micro batches of 16 to 64 examples, accumulate gradients in memory, then update weights after several micro batches. This technique is standard in large scale training on multi GPU or Tensor Processing Unit (TPU) setups. For example, to emulate a batch of 512 with only enough memory for 64, you accumulate gradients over 8 micro batches before the optimizer step. The wall clock time per step increases slightly, but you achieve the convergence benefits and hardware efficiency of the larger effective batch.
💡 Key Takeaways
•Choose the largest batch that fits GPU memory, typically 256 to 512, to maximize hardware utilization and reduce optimizer overhead
•Small batches like 32 or 64 add gradient noise that helps generalization but require more steps to converge, increasing wall clock training time
•Large batches need learning rate scaling and warmup schedules to maintain convergence quality and avoid sharp minima that hurt validation accuracy
•Gradient accumulation emulates large effective batches when memory is constrained by running multiple micro batches of 16 to 64 before updating weights
•The key metric is time to target accuracy, not time per epoch, because larger batches may reach the same validation loss in fewer total hours
•Multi GPU training often uses effective batches of 2,048 to 8,192 with proportional learning rate increases and extended warmup periods
📌 Examples
BERT pretraining at Google: Effective batch size of 8,192 using gradient accumulation over 128 micro batches per GPU across 64 GPUs. Learning rate scaled to 0.004 with 10,000 step warmup.
ResNet ImageNet training: Batch size 256 on single GPU reaches 76.5% top 1 accuracy in 90 epochs. Increasing to 1,024 across 4 GPUs with 4x learning rate reaches same accuracy in 30 epochs, reducing wall time from 29 hours to 8 hours.
GPT model training: Memory limited to batch 16 per GPU. Gradient accumulation over 32 steps emulates batch 512, matching convergence of native batch 512 while fitting in 16GB memory.