ML Model Optimization • Batch Size & Throughput TuningHard⏱️ ~3 min
Batching Failure Modes and Mitigation Strategies
Batching introduces failure modes that do not exist in single item processing. Understanding these edge cases is essential for building reliable production systems. Over batching causes timeouts when batch wait windows or maximum sizes are set too large. Items can exceed Service Level Agreement (SLA) budgets, and in message queues with leases, the lock can expire mid processing. The fix is to renew visibility proactively or reduce batch size so maximum processing time plus network variance fits within the lease.
Head of line blocking occurs when a slow or malformed item holds an entire batch hostage. If you use all or nothing processing, one bad item causes p99 spikes and forces retries of the entire batch. Implement per item retry within the batch, or isolate by sharding batches per key or per model version. Memory issues appear when batch sizes that fit during testing fail in production due to variance in input sizes. A single large input can inflate tensor shape and trigger out of memory (OOM) errors. Add admission control by estimating memory per item and cap batch by memory, not just count. Keep a rescue path to run oversized items alone or on Central Processing Unit (CPU).
Under filled batches during low traffic periods like overnight hours cause throughput to drop and per item cost to rise. Dynamic batching windows expire with very small batches when query rates fall below the natural fill rate. Implement adaptive windows that shrink under low load or use regional pooling to aggregate traffic across geographic zones, increasing batch fill probability.
Partial failure and duplicate processing are common when combining batching with at least once delivery. If a batch partially fails and you retry the whole batch, successful items get processed twice. Design idempotent operations or track per item acknowledgments inside the batch. Ordering violations happen when batching across keys or sessions reorders events. Systems relying on per session order break. Enforce key based partitioning and only batch within a partition when order matters.
💡 Key Takeaways
•Over batching causes timeouts when processing time exceeds visibility timeout or SLA budget, fix by renewing leases or reducing batch size to fit time constraints
•Head of line blocking occurs when one slow item delays the entire batch, causing p99 latency spikes, mitigate with per item retry and isolation by key or version
•Out of memory failures happen when input size variance causes actual batch memory to exceed limits, add admission control based on estimated memory per item
•Under filled batches during low traffic reduce throughput and increase cost, use adaptive windows that shrink to 1 to 2 milliseconds or regional pooling to aggregate traffic
•Partial batch failure with at least once delivery causes duplicate processing, implement idempotency keys or per item acknowledgments to handle retries safely
•Ordering violations occur when batching across sessions or keys reorders events, enforce key based partitioning to preserve order within each key
📌 Examples
Message queue timeout: Batch of 1,000 messages takes 35 seconds to process but visibility timeout is 30 seconds. Locks expire mid batch, causing redelivery and duplicate work. Fix: Reduce batch to 500 or renew visibility every 20 seconds.
Head of line blocking at Uber: One malformed geolocation event causes batch processing to hang for 5 seconds, spiking p99 from 50ms to 5,000ms. Fix: Wrap each item in try/catch, log failures, continue processing remaining items.
OOM in recommendation serving at Netflix: Batch of 64 users with average 50 interactions each fits memory. One user with 5,000 interactions causes batch tensor to exceed 16 GB, crashing server. Fix: Estimate memory per user, cap batch at 12 GB total.
Duplicate charges: Payment processing batch fails after 80 of 100 items succeed. Retry processes all 100 again, charging 80 customers twice. Fix: Store idempotency keys per transaction, check before charging.