Data Processing Patterns • MapReduce & Batch ProcessingMedium⏱️ ~3 min
Cloud Era MapReduce: Disaggregated Storage and Elastic Compute
Traditional MapReduce colocated compute and storage for data locality: mappers ran on the same machines that stored input blocks, minimizing network reads. Cloud deployments flip this model by separating durable object storage (Amazon S3, Google Cloud Storage) from ephemeral compute clusters. Workers spin up on demand, read from remote object stores, shuffle across instance networking, write results back to object storage, and terminate when done. This sacrifices some locality and adds per operation latency but unlocks major operational benefits.
The economics are compelling: transient clusters with autoscaling allow you to size compute precisely to workload demand rather than maintaining always on infrastructure. Preemptible or spot instances cut compute costs by 60 to 90 percent compared to on demand pricing. MapReduce's deterministic retry semantics absorb spot instance loss gracefully: when a node disappears mid task, the framework simply reschedules the work elsewhere. Amazon EMR style deployments commonly run hundreds to thousands of ephemeral workers, scaling job waves independently and relying on object storage for durable input, output, and often intermediate stage persistence in multi job pipelines.
The trade-offs are concrete: object store read latency is 5 to 50 milliseconds per operation versus sub millisecond for local disk, requiring larger read ahead buffers and parallel multipart fetches to saturate bandwidth. Network costs can add up when shuffling terabytes across availability zones. But for workloads that tolerate batch latency and need to process sporadic or unpredictable data volumes, the ability to spin up a 1000 node cluster in minutes, process a petabyte scale backfill, and tear down immediately provides flexibility that colocated clusters cannot match. The pattern works best when job orchestration treats compute as stateless and transient while data lake storage provides the durable system of record.
💡 Key Takeaways
•Disaggregated architecture separates durable object storage from stateless ephemeral compute, trading data locality for elasticity and cost savings
•Spot or preemptible instances reduce compute costs by 60 to 90 percent; MapReduce retry semantics absorb instance loss without manual intervention
•Object store read latency (5 to 50 milliseconds) is 5x to 50x higher than local disk, requiring read ahead buffering and parallel fetches to saturate bandwidth
•Amazon EMR pattern: hundreds to thousands of transient workers autoscale per job wave, with S3 as durable data lake for inputs, outputs, and intermediate stages
•Network transfer costs matter: shuffling 100 terabytes across availability zones can add thousands of dollars, so compress aggressively and consider zone placement
•Best for sporadic or unpredictable workloads: spin up 1000 nodes for a petabyte backfill, run for hours, tear down immediately rather than paying for idle capacity
📌 Examples
Netflix video encoding: ephemeral Hadoop clusters on AWS process petabytes of raw video, writing compressed outputs to S3. Clusters scale from zero to thousands of nodes based on upload queue depth.
Retail demand forecasting: nightly jobs spin up EMR clusters to join billions of transaction records from S3, train models, write predictions back to S3, and terminate within a scheduled window.
Log aggregation pipeline: ingest streaming logs to S3 hourly, trigger batch MapReduce job to compact and index, output to serving data store, shut down cluster until next hour.