How Dictionary and RLE Encoding Work at Scale

The Implementation Mechanics: These encodings are applied per column chunk or page, not globally. A typical pattern in Parquet or Snowflake breaks each column into pages of 64 KB to 1 MB uncompressed. For each page, the system profiles the data, decides which encoding to use, then encodes and optionally compresses it. Dictionary Encoding Process: The encoder makes a first pass over the page to collect distinct values up to a size limit. Values go into a hash table. Once the dictionary is built, it gets serialized. A second pass replaces each value with its dictionary index. The index bit width adapts to dictionary size: 8 bits for up to 256 distinct values, 16 bits for up to 65,536. These indices are bit packed to avoid wasting bytes. RLE Hybrid Scheme: RLE is often combined with bit packing. The encoder scans the sequence (often dictionary indices) and detects runs. For runs longer than a threshold (typically 4 to 8 values), it emits an RLE segment containing the value and run length. For scattered values, it emits a bit packed block of raw values. Query Time Optimization: Query engines exploit these encodings without always fully decoding. For dictionary encoding, filters like country = 'US' are translated into dictionary_id IN {7}, and the scan checks integer equality. Aggregations like COUNT DISTINCT operate on dictionary IDs and dictionary size instead of values. For RLE, a filter like status = 'ACTIVE' can skip entire runs of other statuses with a single comparison. At larger scale, systems implement adaptive encoding selection. During ingestion, the engine samples part of each column, evaluates candidate encodings by estimating compressed sizes, then picks the best. This decision can be revisited during later compaction when more data is available.