Production Architecture and Scale

The Full Pipeline: At companies operating at tens of billions of events per day, AQP is not a standalone feature. It is integrated into the entire analytics stack, from ingestion through query execution to user interface. Ingestion and Maintenance: Raw events land in a data lake or warehouse, often in Parquet or ORC columnar format. Nightly batch jobs create or refresh multiple sample tables at 0.1%, 1%, and 10% rates. These samples are partitioned and distributed identically to the base table to preserve join keys and locality. For example, if the main table is partitioned by date and sharded by user_id, the samples maintain the same scheme. In parallel, streaming ingestion jobs maintain sketch structures in a real time Online Analytical Processing (OLAP) store. As each event arrives, the system updates approximate counters: HyperLogLog sketches for distinct users per feature flag, quantile sketches for latency percentiles per Application Programming Interface (API), Count Min Sketches for top queries. These sketches are stored as time series, typically per hour or per day, in columnar stores optimized for fast retrieval. Query Routing and Optimization: When an analyst issues a query through a Business Intelligence (BI) tool, a query router classifies it. The rules might be: if the query is exploratory (not feeding a regulated report), dominated by large scans with aggregate functions, and does not have highly selective filters, then rewrite it to use samples or sketches. For example, "count distinct users by country over last 90 days" can be served from per day, per country HyperLogLog sketches. The engine reads 90 days × 200 countries = 18,000 tiny sketches (perhaps 36 MB total), merges them using HLL union operations, and returns results in 100 to 500ms at p95 latency, even at 10,000 queries per second (QPS) globally. Online Aggregation: Some advanced systems implement progressive refinement. The engine starts scanning and produces a running estimate with confidence intervals that shrink over time. The user interface might show an initial answer after 200ms with ±20% error, then refine it to ±5% after 1 second, and ±2% after 3 seconds. Users can stop early if they see the metric clearly exceeds or falls below a threshold. This is particularly valuable during incidents when engineers need quick directional answers. Operational Reality: At Meta scale with 50 to 100 billion events per day retained for 1 to 2 years, you have multi petabyte warehouses. Thousands of analysts and data scientists run exploratory queries per hour expecting p50 latency under 1 second and p95 under 3 seconds. Without AQP, you would need to massively overprovision clusters or accept degraded user experience.