How YARN Schedules and Allocates Resources

The Two Level Scheduling Model: YARN uses an elegant separation of concerns. The global ResourceManager handles cluster wide fairness and capacity allocation, while each job's ApplicationMaster makes fine grained scheduling decisions within its allocated resources. This division lets frameworks like Spark implement their own task placement logic while YARN enforces inter application policies. When you submit a job, the ResourceManager first finds space to launch an ApplicationMaster container, typically 1 to 2 GB and 1 CPU core. Once running, the ApplicationMaster requests containers for the actual work: for a Spark job, these are executor containers. Each request specifies resources like 4 vcores and 16 GB, plus optional preferences such as node locality hints to run near Hadoop Distributed File System (HDFS) blocks. Scheduler Types and Queue Management: YARN offers multiple scheduler implementations. The Capacity Scheduler divides cluster resources into queues with guaranteed minimum capacities and configurable maximum limits. For example, you might configure a production ETL queue at 40 percent minimum, data science at 30 percent, and ad hoc at 20 percent, with remaining capacity shared elastically. The Fair Scheduler aims to equalize resource distribution among active users or applications over time. If one user submitted jobs early and consumed most resources, newly arriving jobs from other users gradually receive more allocation as the scheduler rebalances. Both schedulers support preemption: when a high priority queue needs resources but they're held by lower priority jobs, the scheduler can kill those containers to free space. Data Locality Optimization: YARN's killer feature for Hadoop workloads is data locality awareness. When requesting containers, the ApplicationMaster can specify node local preferences: I want this container on node47 because the HDFS block I need to process is stored there. The scheduler tries to honor these hints, falling back to rack local, then any node if local placement isn't possible within a timeout. This matters tremendously for performance. Reading an HDFS block locally from the same machine's disk takes around 100 MB per second. Reading it over the network from another rack adds latency and consumes network bandwidth, dropping throughput to perhaps 50 MB per second while creating congestion. For a job processing terabytes, the difference between 80 percent node locality and 40 percent can mean doubling runtime from 15 to 30 minutes. Resource Isolation with Cgroups: YARN relies on Linux control groups (cgroups) for enforcing resource limits. Each container runs in its own cgroup with hard limits on CPU shares and memory. If a container tries to exceed its memory allocation, the kernel's Out Of Memory (OOM) killer terminates it immediately. CPU is typically enforced as shares rather than hard limits, allowing containers to burst if idle capacity exists, but guaranteeing their minimum allocation under contention.