What is File-Level Partitioning?

The Core Problem: Imagine you store 500 TB of event data in a data lake as Parquet files. Without partitioning, a query filtering for yesterday's data must scan through files from every single day, which could mean reading hundreds of terabytes unnecessarily. At cloud query costs, this becomes prohibitively expensive and slow. Partitioning solves this by creating a directory structure that mirrors your filter criteria. Instead of one flat directory with millions of files, you organize data like event_date=2024-12-25/region=US/file001.parquet. The query engine examines this directory structure and metadata to determine which partitions match your filters before reading any data. How It Works: When data arrives, the ingestion system writes it to directories corresponding to partition key values. For a ride sharing app ingesting 5 million events per minute, events are written to paths like dt=2024-12-25/city=SF/part-0001.parquet. The partition keys (dt and city) become physical directories in object storage. When you query "average trip price in SF last 7 days", the planner looks at your WHERE clause, identifies that you need 7 specific dates and 1 city, then generates a list of exactly those directories. This is called partition pruning. Instead of scanning 365 days times 100 cities worth of data, you scan 7 times 1, a 5,000x reduction in files examined. This is horizontal partitioning at the file system level. Each partition contains the same schema, just different subsets of rows based on partition key values. Unlike database partitioning where the system manages physical layout, file-level partitioning requires you to design the directory structure and choose partition keys carefully.