Micro-batching Architecture in Production Systems

The Full System View: In production, micro-batching sits between raw event ingestion and serving layers, orchestrating a pipeline that transforms streaming data into actionable insights. Consider a large scale ad platform processing hundreds of thousands of events per second. Mobile and web clients emit impression and click events to a message broker like Kafka, which buffers several minutes of traffic across partitions. The micro-batch engine periodically pulls events from Kafka, say every 5 seconds or every 50,000 messages, forming one micro-batch per partition. This dual trigger strategy (time or size, whichever comes first) prevents both excessive latency and oversized batches. Processing Flow: For each micro-batch, the engine applies a series of transformations. First, join events with campaign metadata from a key-value store. Second, compute per-campaign aggregates like click-through rates and conversion metrics. Third, update machine learning features for bidding models. Finally, write results to multiple sinks simultaneously. These sinks serve different purposes. A real-time analytics store (like Druid or Pinot) powers dashboards showing campaign performance with 10 to 15 second freshness. A feature store serves online models that need updated user or campaign features. A columnar warehouse (like Parquet in S3 or Delta tables) archives data for later offline analysis. Control Plane and Observability: The system includes several supporting components. A scheduler orchestrates micro-batch jobs and manages resource allocation. A metadata layer stores source offsets, checkpoints, and schema versions for each batch. Monitoring tracks critical metrics: batch processing time, backlog size (number of queued batches), and per-batch failure rates. If processing time approaches the batch interval, you have a problem. With a 5 second interval, if batches start taking 4.5 to 5 seconds, you're running at capacity. Any spike will create backlog. Autoscaling policies monitor this ratio and add executors when processing time exceeds 50 percent of the interval for several consecutive batches. Real World Examples: Netflix has documented using micro-batch pipelines for near real-time recommendations, recomputing candidate sets every few seconds rather than per event. Databricks promotes micro-batch streaming on Delta tables, where each micro-batch becomes an atomic transaction. Uber processes trip and driver events with single digit second latencies to compute surge multipliers and estimated time of arrival features. The goal is maintaining 99.9 percent availability via retries and replication, while keeping end-to-end p99 latency within Service Level Objective (SLO), typically under 30 seconds for analytics workloads and under 10 seconds for user-facing features.