ETL and ELT at Scale: The Real Numbers

The Volume Challenge: At companies like Netflix or Amazon, you are not moving megabytes. Daily ingestion can hit multiple terabytes across hundreds of data sources: mobile app events generating 5,000 to 50,000 events per second, transactional databases with hundreds of writes per second per shard, microservice logs, and third party data feeds. ETL at Production Scale: Classic ETL architectures batch process data hourly or daily. Sources write Change Data Capture (CDC) streams or periodic dumps into staging. A processing engine like Spark pulls data, applies business rules including data type normalization, Personally Identifiable Information (PII) masking, and reference data joins, then writes dimension and fact tables into the warehouse. Storage stays relatively compact because you only keep curated datasets. Instead of 500 TB of raw plus curated, you might store just 50 TB of modeled data. Query performance is strong because schemas and indexes are designed upfront for known access patterns. The cost: end-to-end latency from event occurrence to queryable metric hits p99 of 1 to 4 hours. This works for finance reporting or executive dashboards that refresh once or twice daily, but fails for real time product analytics. ELT at Production Scale: Modern cloud warehouses like Snowflake, BigQuery, Databricks, and Redshift enable ELT by separating storage from compute. Raw data lands directly via streaming or micro batch. Clickstream events appear in raw tables within seconds. Storage grows to hundreds of terabytes or petabytes because you preserve all raw fields. Transformations run inside the warehouse on elastic compute. You can schedule sessionization and denormalization every 5 to 15 minutes for near real time analytics. For heavy backfills, you scale compute clusters up to finish in 30 minutes instead of 6 hours, then scale down immediately to control cost. The Hybrid Reality: Production systems rarely pick one pattern exclusively. You use ETL for highly regulated datasets where only clean data can enter the analytics environment. In parallel, you use ELT for behavioral data where analysts, data scientists, and machine learning engineers benefit from raw history access. Orchestrators coordinate dependencies so downstream dashboards with 99.9 percent availability always read from validated tables.