ORC Failure Modes and Edge Cases

When ORC Optimizations Break Down: ORC relies heavily on data characteristics and correct metadata. When these assumptions are violated, performance degrades catastrophically. Understanding failure modes is critical for system design interviews because it shows you know when NOT to use ORC and how to mitigate problems. High Cardinality and Random Distribution: Predicate pushdown depends on meaningful minimum and maximum statistics. Consider a column storing UUIDs, random hashes, or cryptographic identifiers. Every stripe contains nearly the full range of possible values because they are uniformly distributed. When a query filters on such a column, min and max checks cannot eliminate any stripes. Meta explicitly noted this limitation: predicate pushdown is ineffective for high cardinality, well distributed identifier columns. The system still pays the cost of computing and storing statistics for every stripe and row group, but gains almost no pruning benefit. For such workloads, the primary value comes from column pruning and lazy reads, not predicate pushdown. Small File and Small Stripe Syndrome: Streaming jobs that flush ORC files every 30 seconds or on every 10,000 rows create thousands of tiny files, each with 1 to 5 MB stripes. This causes multiple problems. First, metadata overhead dominates. Each file has a footer describing schema and stripe locations. Reading 10,000 small files means 10,000 footer reads and 10,000 file open operations. Second, query engines schedule one task per file or per stripe. With 10,000 files, the scheduler must launch 10,000 tasks. Task startup overhead, network round trips to fetch data, and result merging dominate actual computation time. p99 query latencies can degrade from 5 seconds to 60 seconds even though total data volume is only a few gigabytes. The fix is aggressive compaction. Periodically rewrite many small files into fewer large files, targeting 128 MB to 512 MB per file and 64 MB to 256 MB stripes. This reduces file count by 100 to 1000 times and restores performance. Corrupted or Inconsistent Metadata: ORC stores statistics in the file footer. If a writer bug or schema evolution process produces incorrect statistics, query engines may over prune or under prune. Over pruning is catastrophic: the engine skips stripes that actually contain matching rows, causing missing results. This is a silent correctness failure. Under pruning is less severe but expensive. The engine reads unnecessary stripes, wasting I/O and CPU. In mature systems, new writer implementations are validated by running dual reads: write data with both old and new writers, then run thousands of queries comparing checksums and row counts to detect discrepancies. Write Heavy and Low Latency Requirements: ORC is fundamentally designed for append mostly, batch oriented workloads. If you have a write heavy system with over 80 percent writes, or you need sub second ingestion to query latency, ORC is the wrong choice. Buffering rows to fill stripes adds latency. Computing statistics and encoding adds CPU cost. For such workloads, you need a hybrid architecture. Write to a fast store optimized for writes: a row oriented database, key value store, or in memory cache. Periodically, perhaps every 5 to 60 minutes, compact the fast store into ORC for historical analytics. This gives you low latency for recent data and efficient analytics for older data. Bloom Filter Overuse: Some ORC implementations support bloom filters for faster point lookups. A bloom filter can quickly answer whether a value definitely does NOT exist in a stripe, enabling additional pruning beyond min and max. However, bloom filters consume memory and CPU. For high cardinality columns with millions of distinct values per stripe, bloom filters may be large (megabytes) and offer minimal pruning benefit. The decision is: what is your false positive rate tolerance, and what is your query pattern? For point lookups on medium cardinality columns (1000 to 100,000 distinct values per stripe), bloom filters can reduce scanned data by 50 to 90 percent. For range scans or very high cardinality, they add overhead without benefit.