Caching • Cache Patterns (Aside, Through, Back)Hard⏱️ ~3 min
Production Implementation: Multi Level Caches, Observability, and Pattern Composition
Large scale production deployments rarely use a single cache pattern in isolation. The typical stack combines multiple levels and patterns tuned to data characteristics and access frequency. A common architecture is L1 in process cache using cache aside for ultra low latency (sub 100 microsecond access), feeding into L2 distributed read through cache with refresh ahead for popular keys, backed by the source of truth database. Writes often use write around or selective write through: hot entities that users immediately re read get write through treatment for consistency, while bulk or cold writes bypass cache to avoid pollution. Write back is restricted to specific use cases like append only counters or metrics where idempotency guarantees are strong and eventual consistency is acceptable.
Netflix's EVCache exemplifies this layered approach at massive scale. Their client library manages L1 per process cache to shave 100 to 300 microseconds off hot path requests, then queries L2 distributed Memcached based clusters replicated across Availability Zones for durability. Within an AZ, cache hit latency is sub millisecond; cross AZ replication adds 1 to 2 milliseconds. Clusters handle millions of operations per second per region with replication factor of 2 to 3 to survive single AZ failures without read amplification. They use client side sharding and multiget operations to batch requests. TTLs are tuned per data type: seconds to minutes for rapidly changing metadata like recommendations, hours to days for immutable catalog data. Automatic key versioning on schema changes invalidates entire keyspaces without mass deletes.
Observability is critical for operating caches at scale. Track hit ratio globally and per key pattern, with alerts on drops below thresholds (typically 80 to 90 percent for read heavy workloads). Monitor p50, p95, and p99 cache hit latency to detect hotspots, and measure miss penalty (time to fetch from source) to quantify cache value. For write patterns, track write queue lag in write back systems, invalidation success rate, and stampede rate (percentage of requests coalesced). Capacity planning requires sizing memory for target hit ratio with 20 to 30 percent headroom to prevent eviction cascades. Simulate eviction policies with production traces to choose between Least Recently Used (LRU) and Least Frequently Used (LFU). Document consistency SLAs per keyspace: for example, read after write within 200 milliseconds in region, maximum staleness 60 seconds across regions.
💡 Key Takeaways
•Production systems layer L1 in process cache (sub 100 microsecond access) with L2 distributed read through cache, using cache aside and write around or selective write through based on data characteristics
•Netflix EVCache handles millions of operations per second with sub millisecond in AZ latency, cross AZ replication factor 2 to 3, and automatic key versioning on schema changes to avoid stale reads
•Observability must track hit ratio (alert below 80 to 90 percent), p95 p99 hit latency, miss penalty (1 to 5 milliseconds local DB), write queue lag, invalidation success rate, and stampede coalescing rate
•Capacity planning requires 20 to 30 percent memory headroom above working set to prevent eviction cascades; simulate LRU vs LFU with production traces to choose policy
•Document consistency SLAs per keyspace: read after write latency (typically 200 milliseconds in region), maximum staleness (60 seconds), and durability guarantees (write back flush delay 1 to 5 seconds)
•TTL tuning critical: short (seconds to minutes) for dynamic data to bound staleness, long (hours) for immutable data; apply TTL jitter plus or minus 10 to 20 percent to prevent synchronized expiry
📌 Examples
Netflix multi level architecture: L1 per process cache checks in memory map, 100 microsecond access. L2 EVCache hit adds 1 millisecond network hop. Miss penalty to backend service 5 to 20 milliseconds. Client library batches 100 gets into single multiget, amortizing RTT. Automatic key versioning: user:123:v7 invalidates all v6 keys on schema change.
Hit ratio monitoring and alerting: target 90% hit ratio for read heavy keyspace. Alert fires if 5 minute rolling hit ratio drops below 85%, indicating possible stampede, eviction cascade, or TTL misconfiguration. Runbook checks miss penalty spike (DB overload), cache node failures, or recent deployment changing access patterns.
Mixed write pattern by keyspace: For user profile updates (hot, immediate re read), use write through: database.update(profile); cache.set(profile.id, profile, ttl). For analytics events (cold, append only), use write around: database.insert(event); no cache operation. Monitor write amplification per keyspace to validate classification.