How ORC Stripe Architecture Works

The Core Mechanism: ORC organizes data into stripes, which are self contained units typically 64 MB to 256 MB of uncompressed data. Within each stripe, ORC divides rows into smaller row groups of 10,000 to 20,000 rows. The key insight is that both stripe level and row group level statistics enable multi level pruning. When you write data to ORC, the writer buffers incoming records until it accumulates enough rows to fill a stripe. For each column, it analyzes the data characteristics and selects an encoding strategy. A string column with only 100 distinct values across 5 million rows gets dictionary encoding: the 100 unique strings are stored once, and each row stores a small integer reference. An integer column with long runs of repeated values gets run length encoding, storing just the value and count. Statistics at Multiple Granularities: ORC computes and stores statistics at two levels. First, stripe level statistics cover all rows in the stripe: minimum value, maximum value, total count, and null count for each column. Second, row group statistics provide the same metrics for each 10,000 to 20,000 row segment within the stripe. Real Performance Numbers: At Meta, ORC optimizations on a 600 million row dataset reduced single column query wall time by 3.5 to 4 times compared to older readers. CPU time dropped by about 4 times. Scaling to a dataset 10,000 times larger, the improvements held: 3.5 to 4.5 times faster wall time and 4.5 to 6.5 times lower CPU usage. The combination of column pruning, predicate pushdown, and optimized encoding delivered these gains. Some workloads with aggressive filtering saw up to 30 times effective speedup because the engine avoided decoding most data entirely.