MapReduce Failure Modes and Edge Cases

The Straggler Problem: MapReduce jobs don't finish until the last task completes. In a job with 100,000 map tasks, if 99,999 finish in 30 seconds but one takes 10 minutes due to a slow disk or CPU contention, the entire job waits. This is the straggler problem, and it's pervasive at scale. Speculative execution mitigates this by launching duplicate tasks for slow runners. When the master detects a task running 50% slower than median, it starts a backup copy. Whichever finishes first wins. This works well for hardware issues: a flaky disk or overloaded CPU. But it doubles resource usage for those tasks and can create hot spots if many stragglers exist simultaneously. Data Skew and Hot Keys: The worst case for MapReduce is extreme key skew. Imagine counting hashtag mentions on Twitter. If #WorldCup appears in 500 million tweets and other hashtags appear in thousands, the default hash partitioner sends all 500 million #WorldCup records to one reducer. That reducer takes hours while others finish in minutes. Solutions require custom logic. Key salting splits the hot key: instead of (#WorldCup, count), emit (#WorldCup_0, count), (#WorldCup_1, count), and so on, distributing work across reducers, then run a second MapReduce job to combine the salted results. Alternatively, use a two stage approach: first job aggregates per mapper (a combiner on steroids), second job does final aggregation. These patterns add complexity but are necessary for real production workloads with power law distributions. Side Effects and Non-Idempotent Operations: MapReduce assumes map and reduce functions are pure: same input always produces same output, with no external side effects. In practice, developers sometimes write to external databases, send notifications, or call APIs from map or reduce code. This breaks when tasks are retried. If a map task writes a record to an external database, then fails and is retried, you get duplicate writes. If speculative execution runs two copies, both might succeed and write duplicates. The solution is to make all writes go through the framework controlled output with atomic commits, or design operations to be idempotent (for example, using upserts with unique keys instead of inserts). Small Files and Scheduling Overhead: If your input consists of 1 million files of 1 MB each instead of 8,000 files of 128 MB, the master creates 1 million map tasks. Task scheduling overhead (launching JVMs, reporting status, coordinating) dominates actual computation. The master can become a bottleneck just tracking task state. The fix is input consolidation. Use a preprocessing job to merge small files into larger blocks, or configure custom input formats that combine multiple small files per map task. Many production pipelines include a nightly compaction job specifically to merge small files generated by streaming ingest. Network Partitions and Shuffle Failures: During shuffle, reducers fetch data from thousands of map task outputs over the network. If cross rack bandwidth is saturated (for example, 10 TB shuffle on a cluster with 50 GB/s aggregate bandwidth takes 200 seconds minimum), tasks time out and retry, extending p99 latency from 20 minutes to over an hour. Heavy MapReduce jobs are sometimes scheduled during off-peak windows (for example, overnight) to avoid contention with online services. Alternatively, organizations deploy separate analytics clusters isolated from production traffic, trading cost for predictable performance.