Production Scale: Real Numbers and System Choices

At hyperscale, the freshness versus consistency trade off becomes a business decision backed by concrete Service Level Objectives (SLOs). Let's examine how real systems make these choices with actual numbers. Example One: E-Commerce Inventory at 10,000 Writes Per Second An e-commerce platform handles 10,000 order writes per second during peak hours. Product pages get 100,000 reads per second. If you read inventory directly from the primary orders database for consistency, you need a database that can handle 110,000 queries per second with p99 latency under 100ms. This requires expensive hardware and complex connection pooling. Instead, the platform uses a materialized view pattern. Orders write to the primary with 5ms p50 latency. A CDC pipeline streams changes to Kafka within 50ms p50, and a consumer updates a Redis cache within another 100ms. Total freshness: 155ms p50, 700ms p95. The cache serves 100,000 reads per second at 2ms latency. The inconsistency window is explicit: for 700ms after an order, different users might see different stock counts. The business accepts this because overselling 0.1% of items (which they can cancel or backorder) costs less than the infrastructure for synchronous reads. Example Two: Analytics Dashboards with Micro Batching A social media platform needs to show advertisers near real time campaign metrics. Initially, they used batch ETL running every hour, giving 30 to 90 minute freshness. Advertisers complained they could not react to poor performing ads quickly enough. They rebuilt the pipeline with micro batches every 2 minutes. Events are ingested into Kafka, buffered in 2 minute windows, and aggregated into a columnar store. Freshness improved to 2 to 4 minutes p95, satisfying 95% of advertisers. The trade off: they now process some events twice due to late arrivals, requiring idempotent aggregation logic. They also run a nightly reconciliation job to correct any counting errors, prioritizing freshness over perfect consistency in real time views. Example Three: User Facing Features with Read After Write Consistency A messaging app needs users to see their own sent messages immediately (read your writes), but can tolerate other users seeing messages with 500ms to 2 second delay. They implement session affinity: after a user sends a message, their subsequent reads for the next 5 seconds are routed to the primary database. Other users' reads go to replicas that lag by 500ms to 2 seconds. This hybrid approach gives per user consistency where it matters most, while allowing asynchronous replication for scalability. The system handles 50,000 messages per second with p50 write latency of 8ms and p95 read latency of 15ms, compared to 30ms p95 if all reads went to the primary. Monitoring and SLOs: Production systems instrument these trade offs explicitly. Key metrics include replication lag in seconds, age of newest data in each derived store, and rate of consistency anomalies like users not reading their own writes. A typical SLO might be: "95% of profile updates visible to the user within 200ms" and "99% of orders visible in customer support tool within 2 minutes."