Tiered Storage and Freshness Management

Tiered storage is the cornerstone of cost effective real-time OLAP, separating hot segments on fast Solid State Drives (SSDs) from warm and cold segments on cheaper object storage while maintaining a unified logical table. Hot data, typically the last 7 to 30 days, lives on local Non-Volatile Memory Express (NVMe) or SSD for sub-millisecond access, supporting strict sub-second p95 SLAs. Warm data (30 to 90 days) moves to object storage like Amazon Simple Storage Service (S3) or Google Cloud Storage (GCS), where queries take multi-second to low tens of seconds due to network fetch latency. Cold data (90+ days) remains in object storage for compliance or deep historical analysis, often routed to a data warehouse for complex joins. The economic impact is dramatic: SSD storage costs $0.10 to $0.20 per Gigabyte (GB) per month versus $0.02 to $0.03 for object storage, a 5x to 10x difference. Keeping 10 Terabytes (TB) hot costs $1000 to $2000 monthly on SSD but $200 to $300 on object storage. Amazon's retail analytics runs this playbook: customer facing dashboards query hot data (last 7 days) in sub-second with SSD, operational reports tolerate 5 to 10 second queries on warm data (last 90 days) from object storage, and analyst deep dives use the warehouse for multi-month ranges. Freshness management defines Service Level Objectives (SLOs) like "99% of events queryable within 10 seconds of creation." The ingestion pipeline tracks end to end lag: producer timestamp to stream arrival, stream processing latency, and OLAP segment commit time. Real-time OLAP uses small, frequently committed segments (every 1 to 5 minutes) for fresh data, then periodically compacts them into larger segments (hundreds of Megabytes (MBs) to low Gigabytes (GBs)) to reduce query overhead. LinkedIn commits segments every few minutes for seconds-level freshness but runs hourly compaction to keep segment counts manageable, avoiding metadata thrashing that would spike query latencies. The failure mode is query performance degradation during backfills or reprocessing. Historical re-ingestion competes with live serving for Central Processing Unit (CPU) and Input/Output (IO), spiking tail latencies. Isolate via separate ingestion clusters or time sliced backfills (process 1 day per hour during low traffic windows). Use versioned segments (v1, v2) for a time range, build v2 in the background, and atomically swap query routing to avoid partial visibility and consistency anomalies.

Uber marketplace analytics: Keeps last 14 days of trip data on SSD (3 TB, $300 to $600 per month) for sub-200 ms queries powering real-time pricing dashboards; moves 15 to 90 day data to S3 (30 TB, $600 to $900 per month) for weekly operational reports tolerating 5 to 10 second latencies

Cloudflare edge analytics: Stores last 30 days of request logs (petabyte scale) on SSD for customer facing dashboards with sub-second SLAs; archives 31 to 365 day data to object storage, accepting 10 to 30 second queries for security investigations and compliance audits

LinkedIn profile view insights: Commits view event segments every 2 minutes for seconds-level freshness in "Who viewed your profile" feature; runs hourly compaction from 1000s of small segments into 100s of optimized segments (500 MB each), preventing segment count explosion that would degrade query performance