What is Compression and Why Does It Matter at Scale?

The Core Problem: At large scale, a single product can generate tens of petabytes of logs monthly and serve millions of requests per second. Without compression, three things become prohibitive: storage costs, replication traffic, and query latency. Think of it this way: If you're storing 200 terabytes of raw event data per day and replicating it three times for durability, that's 600 TB daily just for one pipeline. Over a year, you're paying for storage and network capacity for 219 petabytes. How Compression Helps: By shrinking data before storage or transmission, you reduce all three bottlenecks. A 4x compression ratio turns that 219 petabytes into roughly 55 petabytes, a massive cost difference. The Three Key Metrics: Every compression algorithm is measured by three dimensions. First, compression ratio: original size divided by compressed size, typically ranging from 2x to 10x in practice. A 1 GB file compressed to 250 MB has a 4x ratio. Second, compression speed: measured in megabytes per second (MB/s) that one CPU core can compress. Third, decompression speed: how fast you can expand the data back, also in MB/s per core. The fundamental insight is that compression lets you move the bottleneck. If storage and network capacity grow slower than your data volume, compression becomes essential infrastructure, not optional optimization.