MapReduce Execution Model: Master Worker Architecture

How Jobs Actually Execute: When you submit a MapReduce job, you provide the input location (for example, a directory in Hadoop Distributed File System containing 50 TB of logs), your map and reduce functions, and the number of reducers (for example, 2,000). A central master node (called JobTracker in Hadoop) orchestrates everything. The master queries the distributed file system to see where data blocks are physically stored. If your 50 TB input is split into 400,000 blocks of 128 MB each, the master creates 400,000 map tasks. Here's the critical optimization: each map task is scheduled on a worker node that already has a local copy of its input block. This data locality means 90 to 95% of map input is read from local disk at 100 to 150 MB/s instead of over the network at 10 to 20 MB/s. The Three Phase Lifecycle: Execution happens in distinct phases. During the map phase, workers read input blocks, run your map function on each record, and write intermediate key value pairs to local disk. These pairs are partitioned (usually by hashing the key) into R partitions, where R is the number of reducers. On a 2,000 node cluster with 5 map slots per node, you can run 10,000 concurrent map tasks. The 400,000 total maps complete in waves over perhaps 15 to 20 minutes for a 50 TB job. Each map writes its intermediate data (sorted by key within each partition) to local disk and reports completion to the master. The shuffle phase begins while maps are still running. Each of the 2,000 reducers pulls its partition from all 400,000 completed map tasks over the network. If intermediate data totals 30 TB (after map side compression and combining), each reducer fetches about 15 GB from 400,000 sources. Reducers use multiple parallel fetch threads and write incoming data to disk, performing external merge sorts to combine sorted segments. Once all maps finish and shuffle completes, the reduce phase runs. Each reducer has a fully sorted stream of key value pairs. It iterates through this stream, grouping consecutive records with the same key, and invokes your reduce function with each key and its value iterator. Final output is written directly to distributed storage with atomic commits. Fault Tolerance Through Re-execution: Workers send heartbeats to the master every 30 to 60 seconds. If a worker stops responding, the master marks it dead and reschedules all its tasks. Map tasks on failed nodes must be re-run even if they completed, because their intermediate output was on local disk that's now lost. Reduce tasks only need re-running if they were in progress when the worker died. This simple retry mechanism works because map and reduce are deterministic and pure.