DLQ Redrive: Safe Reprocessing and Rate Limiting

Redriving messages from a DLQ back to the main processing path is a high risk operation that can trigger cascading failures if not carefully controlled. The fundamental challenge is that DLQ messages failed for a reason, and bulk reprocessing can overwhelm downstream dependencies that may still be fragile or rate limited. Production teams at Amazon commonly cap redrive throughput to 1 to 5 percent of normal traffic volume, using token bucket or leaky bucket algorithms to enforce limits.

The safe redrive workflow starts with root cause verification using a canary batch of 100 to 1,000 messages. Teams deploy the fix (schema update, dependency configuration, code patch), process the canary, and require greater than 95 percent success rate before proceeding to full redrive. Google Pub/Sub implementations create dedicated rate limited subscriptions for redrive, isolating reprocessing from normal consumer flow and preserving multi tenant isolation. If canary success drops below threshold, the redrive pauses automatically and alerts on call.

Idempotency enforcement becomes critical during redrive because at least once delivery semantics guarantee duplicates. Messages may have partially succeeded before failing (wrote to database but failed external API call), and redriving will replay those operations. Production systems use idempotency keys at business operation granularity: order identifiers, payment identifiers, or request identifiers stored in a bounded retention deduplication store (typically 7 to 30 days). The consumer checks this store before processing and skips if the key already exists with matching version.

Rate limiting must be adaptive to downstream health. If dependency error rates climb during redrive, the system should automatically throttle or pause. Amazon teams implement circuit breakers that monitor error rate over sliding windows (5 errors in 10 requests triggers open state for 60 seconds). Microsoft Azure customers layer multiple safeguards: per message type quotas, dependency specific rate limits, and global throughput caps with manual override requiring manager approval for incident response.

💡 Key Takeaways

•Cap redrive throughput to 1 to 5 percent of normal traffic volume using token bucket or leaky bucket rate limiting to avoid overwhelming dependencies

•Require canary batch verification with 100 to 1,000 messages achieving greater than 95 percent success rate before enabling full redrive

•Google Pub/Sub uses dedicated rate limited subscriptions for redrive, isolating reprocessing from normal flow and preserving multi tenant consumer isolation

•Idempotency keys at business operation level (order ID, payment ID) with bounded 7 to 30 day retention prevent duplicate side effects from at least once delivery

•Adaptive rate limiting monitors downstream error rates during redrive, automatically throttling or pausing when errors exceed thresholds (circuit breaker pattern)

•Amazon Prime Day preparation includes raising circuit breaker thresholds and widening backoff caps from 30 seconds to 2 or 5 minutes to handle 10 to 50 times baseline load

📌 Examples

After fixing schema validation bug, team redrives 50,000 DLQ messages at 500 messages per second (5 percent of 10,000 normal throughput) with circuit breaker monitoring downstream database error rate

E-commerce order processing uses order identifiers as idempotency keys stored in Redis with 14 day TTL; redrive after payment gateway fix skips 20 percent of messages already partially processed

Microsoft Azure customer implements three layer rate limiting: 100 messages per second per message type, 500 total across types, 1,000 global cap requiring VP approval to override during incident

← Back to Dead Letter Queues & Error Handling Overview