Database Design • Time-Series Databases (InfluxDB, TimescaleDB)Medium⏱️ ~3 min
Out of Order Data and Late Arrivals: Handling Time Series Reality
Real world time series data rarely arrives in perfect chronological order. Network delays, distributed clock skew, batch processing, and mobile offline sync all cause data points to arrive seconds, minutes, or even hours after their timestamp. Strictly time ordered Time Series Databases (TSDBs) that reject or struggle with out of order writes create production pain, while systems designed for late arrivals maintain stability at the cost of complexity.
The core challenge is that time partitioned storage assumes append only writes within each segment. When a late data point arrives for an already persisted segment, the system must either reject it, buffer it for reingestion, or rewrite the segment incorporating the new point while maintaining sort order and deduplicating. Excessive out of order writes trigger repeated compaction passes that merge overlapping segments, spiking CPU and I/O utilization and degrading query latency.
Production systems use configurable late arrival windows. InfluxDB 3.x accepts writes within a specified lag tolerance (e.g., up to 1 hour late) and maintains a deduplication index to handle the same data point arriving multiple times. Data outside the window gets rejected or routed to a separate repair path. TimescaleDB leverages PostgreSQL flexibility to insert into any time partition with automatic reindexing, though performance degrades if late writes are common enough to constantly split and merge chunks.
The trade-offs are fundamental. Wider acceptance windows increase ingestion complexity and compaction overhead but reduce data loss and client retry logic. Tighter windows simplify storage at the cost of rejecting valid data and requiring sophisticated client buffering. Uber and Tesla likely face this with mobile and vehicle telemetry where connectivity is intermittent. Their solutions involve client side buffering with timestamp preservation, server side deduplication using unique event IDs, and monitoring metrics like late arrival rate and out of order percentage to detect upstream issues. When these metrics spike, it often indicates clock drift, network problems, or misconfigured batch jobs rather than TSDB issues.
💡 Key Takeaways
•Out of order writes force rewriting already persisted time segments to insert late points in correct sort order, triggering expensive compaction passes that spike CPU and I/O and degrade query latency
•InfluxDB 3.x accepts writes within a configurable lag tolerance like 1 hour and uses a deduplication index to handle duplicate arrivals, rejecting or routing data outside the window to separate repair paths
•TimescaleDB allows inserting into any time partition leveraging PostgreSQL flexibility with automatic reindexing, but performance degrades when late writes constantly split and merge chunks
•Wider acceptance windows reduce data loss and simplify clients but increase compaction overhead, while tighter windows simplify storage at the cost of rejecting valid data and requiring sophisticated client retry logic
•Monitoring late arrival rate and out of order percentage helps detect upstream issues like clock drift, network delays, or misconfigured batch processing before they cascade into TSDB instability
📌 Examples
A mobile app with offline mode buffers sensor readings locally with original timestamps. When connectivity returns hours later, it uploads thousands of out of order points. The TSDB accepts writes up to 6 hours late using deduplication by unique device ID plus timestamp.
Tesla vehicles generate telemetry continuously but upload in batches when connected to WiFi. The time series system accepts late writes within 24 hours, uses vehicle ID plus sensor timestamp for deduplication, and tracks out of order percentage to detect GPS clock issues.
A monitoring platform saw compaction CPU usage spike to 80% when a misconfigured batch job sent historical data with old timestamps. They added separate write paths: real time ingestion with 1 hour late tolerance, and backfill ingestion writing directly to cold storage offline.