Production Scale Memory Patterns

The Real World Problem: A company like Uber runs Spark pipelines processing petabytes per day with Service Level Agreements (SLAs) demanding 99.5% availability and p95 stage completion under a few minutes. Their nightly ETL might process 20 TB of event logs across 200 executors (32 GB heap, 8 cores each), transforming raw data into analytics tables within a 90 minute window. Memory management isn't academic here; it's the difference between meeting SLA and paging the on call engineer at 3 AM. Early pipeline stages run narrow transformations: filtering, mapping, parsing JSON. These create billions of short lived objects. If young generation is undersized, minor garbage collection fires every few seconds. Each pause is 50 to 200 milliseconds. Across 2000 concurrent tasks, this aggregates into minutes of wasted time, dropping throughput from 5 million records per second to 3 million. The Shuffle Scaling Problem: Later stages involve multiway joins. A 2 TB fact table joins a 200 GB dimension table. Should you broadcast the dimension? The driver must collect, serialize, and distribute it. If driver memory is 16 GB and the serialized broadcast hits 15 GB, you're on the edge of out of memory. Executors must hold the full broadcast in memory, competing with execution and storage. One company had entire clusters destabilized when a dimension table unexpectedly grew from 2 GB to 15 GB due to new product categories, causing cascading driver crashes. If you don't broadcast, Spark falls back to shuffle join, generating several terabytes of intermediate shuffle data. Execution memory controls how much fits in memory versus spilling to disk. Insufficient execution memory causes excessive spilling: tasks that should complete in 500 milliseconds take 30 seconds as they thrash between memory and disk. The p99 task latency explodes, pushing job completion past SLA. Caching Trade Offs: Suppose 10 downstream jobs reuse a filtered 500 GB subset of the fact table. Without caching, each job scans 2 TB from cloud storage at 5 GB per second cluster throughput: 400 seconds per job, 4000 seconds total. Caching saves 4000 seconds but consumes 500 GB of storage memory across the cluster. If that pushes a heavy shuffle over its memory limit, causing spilling and adding 10 minutes to job time, you've traded scan time for spill overhead. Companies like Databricks report that most severe production incidents are memory related: unexpected data growth, skewed keys causing one task to process 50 GB while others handle 1 GB, or new joins that weren't tested at full scale. The same job code that works fine on 1 TB fails on 10 TB not from CPU limits but because memory footprints scale super linearly with skew. Observability and Alerts: Production teams instrument Spark heavily. They track per executor heap usage, spill bytes per stage, cache hit rates, and garbage collection pause distributions. Alert thresholds trigger when 5% of tasks spill over 1 GB each, or when garbage collection pause time exceeds 10% of executor runtime. When input size or schema changes push memory patterns out of safe bounds, teams catch it before jobs fail repeatedly during critical processing windows.