Failure Modes and Edge Cases

Executor Out of Memory Cascade: The most visible failure is executor out of memory during a wide shuffle or join. Symptoms: many tasks fail repeatedly on the same executor with out of memory errors. Root causes fall into three categories. First, data skew: one partition receives 50 GB while others get 1 GB. That task exhausts executor memory even though average memory per task looks fine. Second, misconfigured unified memory leaves too little room for execution, causing aggressive spilling that eventually overflows. Third, user code materializes large collections, perhaps a 10 GB list built incrementally in a User Defined Function, which bypasses Spark's memory management entirely. At production scale, a single hot partition can kill every executor that attempts it. Spark retries the task on different executors, but if the data itself is the problem, all executors fail. Eventually the job aborts after exhausting retry attempts. This is why companies invest heavily in skew detection and mitigation: salting keys, two phase aggregations, or custom partitioners. Garbage Collection Storms: When heaps are large and object creation rate is high, executors can enter garbage collection storms. Normal state: executors spend 2% of time in garbage collection, 98% running tasks. Under memory pressure, this shifts to 15% garbage collection, slowing throughput noticeably. In a garbage collection storm, executors spend 40% or more of time in garbage collection: multiple second pauses every few seconds. Tasks that should take 2 seconds take 10 seconds. Cluster throughput collapses. In batch jobs, this inflates p99 task durations and causes cascading backpressure on shuffle services and storage. In streaming jobs with sub minute microbatches, long pauses miss processing deadlines. A streaming pipeline targeting p95 end to end latency of 10 seconds cannot tolerate executors disappearing for 5 to 10 seconds under garbage collection. The result is missed Service Level Objectives (SLOs) and potential data loss if the system cannot keep up with incoming event rate. Storage vs Execution Contention: Subtle issues arise from storage and execution contention. Suppose many datasets are cached uncompressed, and storage memory grows until it hits its fraction of unified memory (say, 50%). A heavy shuffle starts, needing more than its initial 50% execution share. Execution can evict cached blocks, but only until storage shrinks to its reserved minimum. If that's insufficient, execution spills to disk. The job still succeeds, but runtime unexpectedly doubles or triples. Worse, if user code also maintains large in memory caches (outside Spark's management), total memory pressure pushes the executor over the edge into out of memory. This failure is hard to diagnose because metrics show storage and execution memory individually within bounds, but the sum exceeds available heap. Broadcast Edge Cases: Broadcast variables create unique failure modes. The driver must collect the entire dataset, serialize it, and then distribute it to all executors. If a dimension table expected at 2 GB grows to 15 GB, the driver may out of memory during collection. Even if the driver survives, executors must hold the full 15 GB in memory. On a 32 GB heap executor running 8 tasks concurrently, this leaves only 17 GB divided among 8 tasks: about 2 GB per task. If the shuffle also needs memory, tasks start failing. One company experienced cascading failures when an oversized broadcast caused repeated driver out of memory and job retries. Each retry attempted the same oversized broadcast, failing again. The retry loop destabilized the cluster for hours until the root cause was identified and the broadcast disabled.