Definition
Event streaming is an architectural pattern that treats continuous, real-time data as an unbounded sequence of immutable events stored in a durable, ordered log that multiple consumers can read at their own pace.
Modern applications generate massive volumes of small events continuously: user clicks, payment transactions, sensor readings, log entries. Traditional batch processing waits 10 to 60 minutes to collect data before processing. But businesses need sub-second fraud detection, real-time recommendations, and live dashboards.
For example, an e-commerce site with 50 million daily active users generates 500,000 user events per second during peak sales. Waiting an hour to detect fraudulent purchases means millions in losses. Batch processing cannot meet this latency requirement.
How Event Streaming Works:
Think of an event stream as an append-only log. Producers (applications, microservices, devices) write events to this log. Each event contains a key, a timestamp, and a payload. The log is durable and replicated for fault tolerance.
Consumers read from the log independently. Consumer A might process events for fraud detection in real time. Consumer B might load the same events into a data warehouse for analytics. Consumer C might update a recommendation engine. They all read the same stream but at different speeds and for different purposes.
Key Properties:
First, decoupling. Producers write events without knowing who will consume them. New consumers can be added without changing producers. Second, replayability. Because the log is durable, consumers can rewind and reprocess historical events. This is powerful for debugging, reprocessing with new logic, or backfilling data. Third, ordering guarantees. Events for the same key (like user ID) are stored in order, enabling stateful processing.
✓ In Practice: Production Kafka clusters at companies like LinkedIn handle trillions of events per day with publish latency under 10ms at the 99th percentile.
✓An event stream is an unbounded, append-only sequence of immutable records stored in a durable log
✓Producers and consumers are decoupled: producers write without knowing consumers, enabling independent scaling
✓Events preserve per-key ordering, allowing stateful processing like aggregations and joins over infinite streams
✓The log is replayable: consumers can rewind to any point in history for reprocessing or debugging
✓Production systems handle millions of events per second with sub 10ms publish latency at p99
1E-commerce site generates 500,000 events per second during peak sales: clicks, cart updates, purchases
2Fraud detection consumer processes payment events with p99 latency under 200ms to block suspicious transactions in real time
3Analytics consumer reads the same event stream to update dashboards within 1 to 5 seconds of actual user activity
4Data warehouse consumer batches events into hourly buckets for long term storage and offline machine learning