How Memory Regions Work Together

The Borrowing Mechanism: When a Spark executor starts, it calculates unified memory as a fraction of available heap after reserving space. By default, 60% of heap minus reserved becomes unified memory, split roughly 50/50 between execution and storage initially. But here's where it gets interesting: these boundaries are soft, not hard. Imagine an executor with 16 GB heap. After 300 MB reserved and user memory allocation, perhaps 9 GB becomes unified memory: 4.5 GB for execution, 4.5 GB for storage. Now a heavy shuffle stage starts, needing 7 GB to hold hash tables for a join. Execution can borrow up to the full unified memory, evicting cached blocks from storage as needed. Storage shrinks but retains its guaranteed minimum region. The Eviction Rules: Execution can evict storage blocks aggressively when it needs memory. Evicted blocks either spill to disk (if persisted with disk fallback) or get recomputed when accessed again. However, storage cannot evict execution data. This asymmetry protects active computation from being interrupted by caching operations. If storage tries to grow beyond its share while execution is active, it simply cannot, preventing cache operations from destabilizing running tasks. Serialization's Hidden Impact: Object layout profoundly affects how much data fits in memory. Consider a DataFrame with 100 million rows of simple records. Using default Java serialization, each object has header overhead, pointer indirection, and padding. The raw data might be 8 GB, but in memory objects could consume 24 GB: a 3x multiplier. Serialized storage using formats like Kryo or the Tungsten binary format can pack that same data into 10 GB, cutting memory usage by more than half. The tradeoff is CPU time for serialization and deserialization on access. But at large scale, reducing memory pressure and garbage collection overhead often improves end to end performance despite the CPU cost. Garbage Collection Interaction: Spark workloads create millions of temporary objects per executor: intermediate results, deserialized rows, hash map entries. Poor memory management floods the JVM with objects that promote from young generation to old generation. When old generation fills, a full garbage collection pause can freeze the executor for seconds, during which no tasks progress. At 200 executors, if even 10% hit simultaneous multi second pauses, cluster throughput collapses.