What is ORC Format?

The Problem It Solves: Imagine you have a table with 10 billion rows and 200 columns stored in a traditional row oriented format. When an analyst queries for just 3 columns, the system still reads all 200 columns from disk, wasting enormous amounts of input/output (I/O), network bandwidth, and CPU time. Worse, row oriented formats compress poorly because adjacent data has mixed types and patterns. How ORC Works: ORC flips the storage model. Instead of storing rows together, it stores each column separately in large chunks called stripes. A typical stripe contains 64 MB to 256 MB of uncompressed data. Within each stripe, ORC groups rows into smaller units of 10,000 to 20,000 rows each. Because each column contains homogeneous data, ORC applies specialized compression per column. String columns with low cardinality get dictionary encoding. Integer columns with repeated values get run length encoding. This achieves much better compression ratios than generic row compression. The Key Innovation: ORC stores rich statistics at multiple levels: minimum, maximum, count, and null count for each column in every stripe and row group. Query engines use these statistics to skip entire stripes that cannot possibly match filter conditions. This is called predicate pushdown. When you read only 3 columns from a 200 column table, you avoid reading 98.5% of the data. Combined with stripe skipping from predicate pushdown, queries can avoid 80 to 95 percent of disk reads entirely.