Time Based Indexing and Hot Warm Cold Tiering

Time Based Index Partitioning

Systems handling logs, events, metrics, or other time series data partition indexes by time buckets rather than arbitrary sharding. Each day or week gets its own index: logs-2024-01-15, logs-2024-01-16, and so on. This pattern enables targeted queries that touch only relevant time ranges (searching last 24 hours queries one index instead of scanning terabytes of historical data), efficient deletion (dropping old data means deleting entire indexes rather than marking individual documents), and lifecycle management that applies different storage policies based on data age.

Rollover policies automatically create new indexes when the current one hits size or time thresholds. A policy might roll over at 30 GB or 1 day, whichever comes first. This bounds index size regardless of traffic variability. During traffic spikes, more frequent rollovers keep indexes manageable. During quiet periods, time-based triggers prevent stale indexes from growing indefinitely.

Hot Warm Cold Tiering

Production systems assign indexes to tiers based on age and access patterns. The hot tier uses fast storage (SSD or NVMe, solid state drives with no moving parts) with full replica coverage for active writes and low-latency queries. Cost runs high at roughly 500 dollars per TB per month, but p99 (99th percentile) latency stays under 100ms. Hot typically covers 0 to 7 days of data where most queries land.

After the hot window, lifecycle policies transition indexes to the warm tier. Warm uses cheaper spinning disk storage (HDD, hard disk drives) with fewer replicas, accepting higher latency of 200 to 500ms for infrequent queries. Cost drops to roughly 100 dollars per TB per month. Warm typically covers 8 to 90 days. The cold tier uses object storage (distributed file storage accessed over network protocols) at roughly 25 dollars per TB per month for long-term retention, but requires restore operations before querying, taking minutes to hours.

Cost Optimization at Scale

Tiering dramatically reduces costs for large deployments. Consider a system with 10 TB total data: keeping everything on hot tier costs 5,000 dollars per month. If 80% of queries target the last 7 days (2 TB), moving 8 TB to warm saves 3,200 dollars per month with minimal latency impact on typical queries. The key insight is matching storage cost to access frequency: paying premium prices only for data that justifies premium latency.

Streaming ingestion pipelines continuously write events to search databases for real-time querying. Event streaming platforms (like message queues) buffer incoming data, and dedicated indexing workers consume these events and write to the current hot index. This decouples write throughput from query load, enabling systems to ingest billions of events daily while serving interactive dashboards with sub-second latencies.

Over Sharding Anti Pattern

A common failure mode is over-sharding: creating too many small indexes and shards. Each shard consumes heap memory for metadata (roughly 10 to 50 MB per shard), and clusters with thousands of tiny shards exhaust memory managing coordination overhead rather than serving queries. A cluster with 5,000 small indexes at 5 shards each creates 25,000 shards, overwhelming cluster state management.

Prevention requires balancing rollover thresholds. Rolling over too frequently (every hour) during low-traffic periods creates many tiny indexes. Rolling over too infrequently during high-traffic periods creates oversized indexes that slow recovery. Best practice combines size and time triggers: roll over at 30 GB or 1 day, ensuring indexes stay within operational bounds regardless of traffic patterns.