Event Time vs Processing Time and Windowing

The Core Challenge: In a distributed system, events do not arrive in perfect order. A user clicks a button at 10:00:00 AM (event time), but due to network latency, device offline state, or processing delays, your streaming system might receive that event at 10:00:45 AM (processing time). If you aggregate events by when they arrive, you miscount events that were delayed. This distinction matters enormously at scale. Consider a mobile app with users in areas with spotty connectivity. Events might be queued locally for minutes or hours before transmission. If you process by arrival time, your hourly active user counts will be wrong. Event Time: Event time is the timestamp embedded in the event itself, representing when something actually happened in the real world. This is usually set by the client or originating service. Event time gives you correct business logic: if you want hourly sales totals, you group by the purchase_timestamp field, not when your server saw the event. The challenge is handling late arrivals and out of order events. You need mechanisms like watermarks (a heuristic that estimates how far event time has progressed) and allowed lateness windows to decide when to finalize results. Processing Time: Processing time is when the streaming system processes the event. It is simpler to implement because it requires no special event time logic and no watermarking. But it produces incorrect results when events arrive late or out of order. Processing time is appropriate when you care about operational metrics tied to the system itself, like events processed per second or system health checks, not business metrics. Windowing Infinite Streams: You cannot compute a sum over an infinite stream without boundaries. Windowing divides the stream into finite chunks. Time based windows (tumbling windows of 5 seconds, sliding windows of 1 minute updated every 10 seconds) let you compute aggregates like counts or averages. Count based windows (every 1000 events) work when event volume is predictable. Session windows (start on first event, close after inactivity timeout) track user sessions.