Understanding Encoding Strategies: Dictionary, RLE, and Delta

The Core Problem: Large analytical systems need to scan terabytes of data to answer dashboard queries within 1 to 3 seconds. The bottleneck is not CPU but I/O and memory bandwidth. Reading fewer bytes from disk or object storage directly translates to faster queries and lower costs. Three Pattern Exploiting Approaches: Dictionary Encoding targets repeated values, especially in categorical columns. Instead of storing "United States" 500 million times as a string, you store it once in a dictionary and reference it with a small integer like 7. This works because string comparisons are expensive and strings take more space, while integer comparisons are cache friendly and take fixed space. Run Length Encoding targets consecutive sequences of identical values. Instead of storing A, A, A, A, you store (A, 4). This is powerful when data is sorted by a column like status or country, creating long runs of the same value naturally. Delta Encoding targets ordered numeric values that change gradually. Instead of storing 1000000, 1000010, 1000013, you store base 1000000 and deltas [0, 10, 3]. These smaller numbers fit in fewer bits and compress better. This pattern dominates time series and monotonically increasing IDs like timestamp or order_id. These encodings are not just academic. They are how petabyte scale warehouses make queries fast enough to render dashboard charts in under 1 second while keeping storage costs manageable.