Message Queues & Streaming • Delivery Guarantees (At-least-once, Exactly-once)Medium⏱️ ~2 min
Achieving Exactly Once Processing with Idempotency
Exactly once processing is implemented by ensuring consumer side effects are idempotent, meaning the same operation applied multiple times produces the same result. This is typically achieved through unique operation identifiers combined with conditional writes that prevent duplicate effects even when messages are delivered multiple times.
The most common pattern is to generate a deterministic operation identifier (often a hash of business key plus natural identifier) and store it atomically with the side effect. On message receipt, the consumer first checks if this identifier already exists. If found, it returns the recorded outcome without reapplying the effect. If not found, it performs the operation and stores the identifier in the same atomic transaction. Payment systems like Stripe, PayPal, and Amazon Payments use this approach with client provided idempotency keys, typically with a 24 hour time to live (TTL). These systems process millions of requests per day and rely on strongly consistent conditional writes to enforce uniqueness.
The trade off is coordination cost versus duplicate elimination. Dedup stores require memory or storage proportional to queries per second (QPS) times the dedup window. For a system ingesting 50,000 messages per second with a 10 minute window, you must track 30 million identifiers. At roughly 48 bytes per entry (16 byte identifier plus overhead), that is approximately 1.44 GB just for the in memory key set. Durable stores add further overhead and latency (1 to 5 milliseconds locally, 5 to 20 milliseconds cross availability zone).
💡 Key Takeaways
•Generate deterministic operation identifiers (hash of business key plus natural identifier) and store them atomically with side effects using conditional puts (create if not exists) or unique constraints in relational databases.
•Dedup store sizing: For 50,000 messages per second with a 10 minute window (30 million identifiers), at 48 bytes per entry, memory requirement is approximately 1.44 GB. Durable stores add 1 to 20 milliseconds latency depending on locality.
•Payment APIs use idempotency keys with typical 24 hour TTLs to prevent double charging. Retries with the same key return the original result instead of creating duplicate charges, processing millions of requests daily.
•Lost dedup state causes duplicates to slip through. Dedup caches with TTL can evict entries before all duplicates arrive (long pauses, reorders). Right size TTLs and use durable dedup for critical operations.
•In memory least recently used (LRU) dedup offers sub millisecond lookup but bounded windows, suitable for high QPS analytics. Persistent key value stores offer durability with 1 to 5 millisecond local latency.
📌 Examples
Stripe charges API requires clients to provide an idempotency key. When creating a charge, Stripe stores the key and result atomically. If a network failure causes a retry with the same key, Stripe returns the original charge object instead of creating a duplicate charge.
An order processing service hashes order_id concatenated with action_type (e.g., 'charge', 'fulfill') to generate operation identifiers. Before charging a payment method, it checks a DynamoDB table with a unique constraint on the operation identifier. If the identifier exists, it returns the stored charge result.
A stream processor consuming inventory updates uses message offset concatenated with partition as the operation identifier. It writes inventory changes to PostgreSQL in a transaction that also inserts the identifier into a dedup table with a unique constraint, preventing duplicate decrements.