Advanced Pipeline Patterns and Observability

Fan Out and Fan In Patterns: Beyond linear pipelines, production systems use fan out and fan in topologies. Fan out means one stage writes to multiple downstream stages. For example, a normalization stage outputs to both a real time analytics pipeline and a batch data lake pipeline. Each consumer reads at its own pace without blocking the other. Fan in means multiple upstream stages feed one downstream stage. A machine learning feature pipeline might consume from event streams, database change logs, and external API feeds, joining them into unified feature vectors. The challenge is synchronization: how do you ensure all inputs for a given entity (like user_id 12345) arrive before processing? Typically, use keyed state: buffer inputs by key until you have received from all sources or a timeout expires. Lambda and Kappa Architectures: Two advanced patterns address the batch versus streaming dilemma. Lambda architecture runs both batch and streaming pipelines in parallel. The batch layer reprocesses all historical data daily or weekly, producing accurate results. The streaming layer processes recent data with lower latency but potential approximations. A serving layer merges both: queries hit the batch view for old data and the streaming view for recent data. The trade off: you maintain two codebases doing similar logic. Kappa architecture simplifies this: use only streaming, but make the stream replayable. To reprocess historical data, reset the stream consumer to an old offset and replay. This requires your streaming logic to handle both real time and batch speeds, which is harder to implement but reduces duplication. End to End Observability: Production pipelines require comprehensive observability. Instrument each stage to emit metrics: throughput (messages per second), latency (p50, p90, p99), error rate, and queue depth. Use a time series database to store and visualize these metrics. Set alerts: if p99 latency exceeds SLO for 5 minutes, page on call. If error rate exceeds 1 percent, page immediately. Distributed tracing is critical for debugging. Assign each event a unique trace_id. As the event flows through stages, each stage logs with that trace_id. When investigating a slow request, query traces to see: Stage 1 took 20ms, Stage 2 took 150ms, Stage 3 took 600ms. Stage 3 is the bottleneck. Implement data quality monitoring. Track schema changes: if Stage 2 suddenly starts outputting null values for a required field, alert. Track data volume: if Stage 3 outputs 50 percent fewer records than yesterday, alert. This catches silent failures where the pipeline runs but produces garbage. Testing and Validation: Unit test each stage independently with mock inputs. Integration test multi stage segments by running on a test cluster with sampled production data. For critical pipelines, implement shadow mode: run new logic alongside old logic, compare outputs, and alert on differences exceeding a threshold before switching traffic. Data validation is as important as code tests. Check invariants: row counts should not drop by more than 10 percent between stages unless filtering is expected. Check distributions: if average order value suddenly doubles, halt the pipeline and investigate. Automated validation catches bugs that unit tests miss, especially around schema evolution and edge cases in production data. Schema Evolution: As pipelines evolve, schemas change. New fields are added, old fields are deprecated. Without careful management, this breaks consumers. Use schema registries: centralize schema definitions and enforce compatibility rules. Forward compatibility allows old code to read new data by ignoring unknown fields. Backward compatibility allows new code to read old data by providing defaults for missing fields. Enforce these rules at schema registration time, failing incompatible changes.