Production Compression in Data Pipelines: Layered Optimization

The Multi Stage Reality: In production systems, compression isn't a single decision. It's applied at multiple stages of the data flow, each with different optimization goals. Understanding where to compress and with which codec at each layer is critical to achieving both performance and cost efficiency. Consider a large scale logging pipeline that processes 5 million events per second, about 5 GB/s raw. Data flows through four stages: producer, message bus, storage, and analytics query engine. Each stage compresses differently. Stage One: Producer to Message Bus: At this layer, primary concerns are throughput and p99 latency. If each producer instance can spare only 20 to 30 percent CPU for compression, you choose fast codecs like Snappy or LZ4. These compress at hundreds of MB/s per core with ratios around 2x, keeping per request latency under 1 millisecond at p99. This halves network traffic between producers and brokers, potentially reducing link requirements from 40 Gbit to 20 Gbit. Stage Two: Long Term Storage: For object storage similar to S3, optimization shifts toward cost per terabyte and scan efficiency. A trillion events daily at 200 bytes each equals roughly 200 TB raw. With zlib or Zstd at 3x to 4x ratio, you store only 50 to 70 TB. When replicated three times and stored for a year, the cost delta compared to Snappy at 2x becomes massive. Stage Three: Analytics Query Layer: Engines like Presto or Spark SQL read compressed blocks from columnar formats, then decompress in memory. The key metric here is effective scan throughput: IO bandwidth divided by compression ratio, minus CPU cost. If storage reads at 10 GB/s and Zstd achieves 4x ratio, you effectively scan 40 GB/s of logical data, assuming decompression stays under 30 to 40 percent CPU. Edge and API Layer: User facing services compress JSON or HTML to reduce bandwidth and improve page load time. Static assets might use zlib level 6 (compressed once, decompressed millions of times), while API responses use Zstd at mid levels to keep server side latency below 5 milliseconds at p99. The pattern is clear: compress fast and light where latency is critical, compress aggressively where data is written once and read many times, and match codec characteristics to workload demands at each pipeline stage.