Message Queues & Streaming • Kafka/Event Streaming ArchitectureHard⏱️ ~3 min
Message Delivery Semantics: At Least Once vs Exactly Once Processing
Three Delivery Guarantees:
Event streaming systems offer three delivery guarantees: at most once (fire and forget, rare in practice), at least once (default, accept duplicates), and exactly once (complex coordination). At least once is the workhorse: producers retry on transient failures, and consumers may reprocess events after crashes before committing offsets. This requires downstream idempotency: your consumer must handle duplicate delivery gracefully via versioned upserts, deduplication caches with TTL, or natural idempotency keys.
Exactly Once Mechanisms:
Exactly once processing combines three mechanisms: idempotent producers (sequence numbers prevent duplicate writes), transactional reads and writes (processors read from input topics, update state, write to output topics, and commit offsets atomically), and deterministic reprocessing on failure. The coordinator ensures that either all operations commit together or none do. The cost is significant: transactional overhead reduces throughput by 20 to 50 percent and increases tail latency due to coordination.
When to Use Exactly Once:
Financial systems and billing pipelines are the common exceptions that justify exactly-once complexity, where duplicate charges or payments are unacceptable. For example, upserting to a database with a unique constraint on event ID naturally deduplicates without the overhead of distributed transactions.
Partial Failure Mode:
The key failure mode is partial failure during offset commits: a consumer processes a batch, writes to a sink, but crashes before committing its offset. On restart, it reprocesses the batch. At-least-once semantics deliver duplicates to the sink; exactly-once semantics use transactional commits to make the sink write and offset commit atomic.
"In practice, most high-throughput systems choose at-least-once with idempotent consumers. LinkedIn, Netflix, and Uber all default to at-least-once and design sinks to be idempotent."
💡 Key Takeaways
✓At least once requires idempotency in consumers: use unique constraints, versioned upserts (last write wins with timestamp or version number), or deduplication caches. Most teams find this simpler than transactional exactly once.
✓Exactly once processing reduces throughput by 20 to 50 percent: transactional coordination adds latency and limits parallelism. Brokers must maintain transaction state and enforce isolation, increasing central processing unit and memory overhead.
✓Idempotent producers are a prerequisite for exactly once: sequence numbers per producer session prevent duplicates on network retry, but do not prevent reprocessing after consumer crashes. They eliminate producer side duplicates only.
✓Transactional sinks are required for end to end exactly once: the consumer must write to a sink that supports transactions (Kafka itself, transactional databases, or systems with atomic commit APIs). Writing to eventually consistent stores breaks exactly once guarantees.
✓Financial and billing systems justify exactly once complexity: duplicate charges or payments create customer support burden and regulatory risk. High throughput analytics, logging, and monitoring typically choose at least once with idempotent consumers for 2 times to 3 times better throughput.
📌 Interview Tips
1An e-commerce platform uses exactly once for order placement events to prevent duplicate charges. Each order processing job runs in a transactional context: read order event, update inventory in a database, write fulfillment event to output topic, commit offset atomically. Throughput is 40 percent lower than at least once but duplicate orders are impossible.
2Netflix uses at least once for personalization events with idempotent upserts. Each viewer action event carries a unique event_id; the sink database has a unique constraint on event_id. Duplicate delivery results in a constraint violation and is silently ignored, achieving effective deduplication without transactional overhead.
3A real time analytics pipeline at Uber processes trip events with at least once semantics. Each event includes trip_id and timestamp; the aggregation query uses INSERT ON CONFLICT UPDATE with last write wins logic based on timestamp, naturally handling duplicates without exactly once complexity.