Failure Modes and Edge Cases in Windowing

Misconfigured Watermarks: The most common failure mode is watermark configuration that does not match actual event arrival patterns. If watermarks are too aggressive (low allowed lateness), late events get dropped or sent to a side channel for late data. This silently undercounts metrics. The impact is systematic, not occasional. If 2 percent of events consistently arrive more than 5 minutes late but your allowed lateness is only 3 minutes, every single aggregate will be off by roughly 2 percent. Volume metrics, revenue calculations, and fraud detection patterns all become subtly wrong. Conversely, if watermarks are too conservative (high allowed lateness), windows stay open far longer than necessary. State size grows, memory pressure increases, checkpoint times expand from seconds to minutes, and recovery latency after failures can violate SLAs. A system designed for 2 minute recovery might take 10 minutes when state bloat causes slow checkpoints. Clock Skew and Future Timestamps: When producers use local device time without synchronization, events can have timestamps in the future relative to server time. A phone with a clock 5 minutes ahead generates events that push watermarks forward prematurely, closing windows before all data arrives. This is particularly nasty because it is intermittent and hard to debug. One misconfigured device among thousands can cause sporadic undercounting. Production systems often clamp event times to a reasonable range around ingestion time (for example, reject or adjust timestamps more than 1 hour in the future or past) or track per source clock skew statistics. Skewed Keys and Hotspots: Session windows keyed by user are vulnerable to extreme skew. Imagine a bot or power user generating 5,000 events per second in a single long session. The stateful operator owning that key becomes a hotspot. While other operators process 1,000 events per second comfortably, this one operator struggles with 5x load. Backpressure propagates upstream, p99 latency spikes from 100ms to 5 seconds, and the operator may run out of memory trying to maintain the massive session state. Mitigation strategies include key splitting (break hot keys into subkeys with a random suffix), adaptive load balancing (detect hot keys and redistribute), or separate treatment of heavy hitters (route them to dedicated high capacity operators). Daylight Savings and Time Zones: If windows are defined on calendar concepts like "per day" instead of fixed 24 hour intervals, daylight savings transitions break expectations. A "daily" window in a region observing daylight savings is 23 hours on spring forward day and 25 hours on fall back day. This causes aggregates to be incomparable across days and can break downstream assumptions. LinkedIn's real time analytics define windows in Coordinated Universal Time (UTC) and handle business reporting logic separately to decouple window boundaries from local time changes. Recovery and Replay Determinism: When a job fails and replays from a log, window state must rebuild deterministically. If the replay logic differs from the original (for example, different random number seeds, nondeterministic aggregation functions), you get inconsistent results. Downstream sinks must also handle replays correctly. If a sink does not support idempotent writes or retractions, replay can double count aggregates. Exactly once semantics require careful coordination between source offsets, stateful operator checkpoints, and sink commits to ensure that replayed events produce identical outputs without duplication.