What is MapReduce?

The Core Problem: Imagine you need to process 50 terabytes of log files from yesterday to count how many times each user visited your website. On a single machine reading at 100 MB/s, this takes 5.8 days. Even if you had 1,000 machines, coordinating them with traditional programming (managing which machine processes what, handling failures, collecting results) requires extensive distributed systems expertise. MapReduce solves this by constraining your logic to two simple functions that operate on key value pairs: map and reduce. How the Two Functions Work: The map function takes an input record and emits zero or more intermediate key value pairs. For our user visit example, map reads a log line and emits (user_id, 1). This function runs independently on each input record across thousands of machines. The reduce function receives a key and all values associated with that key, then aggregates them into a final output. For our example, reduce receives (user_id, [1, 1, 1, ...]) and sums the values to emit (user_id, total_visits). The Key Constraint: Both functions must be deterministic and side effect free. This lets the framework parallelize safely, retry failed tasks, and run speculative duplicates of slow tasks, all without risking incorrect results. You focus on business logic; the framework handles distribution, fault tolerance, and data movement.