Trade-Offs: Tight vs Loose Coupling

The Fundamental Trade-Off: Cross pipeline dependency management is fundamentally about trading coupling versus autonomy, and correctness versus latency. The right choice depends on your organizational structure, scale, and SLA requirements. Tight Coupling: The Monolithic Approach A single massive DAG contains all related tasks, even across organizational boundaries. Team A's ingestion tasks, Team B's normalization, and Team C's analytics all live in one DAG definition. This gives strong guarantees: the orchestrator can visualize the entire dependency graph, optimize scheduling across all tasks, and immediately see failure propagation paths. The problem appears at organizational scale. Ownership becomes tangled. A change to Team A's ingestion logic requires coordinating releases with Teams B and C. Testing becomes complex because you cannot test in isolation. At FAANG scale with hundreds of teams, this monolithic approach creates deployment bottlenecks and breaks team autonomy. One team's bug can block dozens of other teams. Loose Coupling: Independent Pipelines Each team owns separate DAGs with explicit external dependencies. Team A publishes a completion event or updates metadata when their pipeline finishes. Team B declares a dependency on that signal and waits for it. Teams can deploy independently, test in isolation, and own their release cadence. The trade-off is you now need robust shared contracts. What format are events? What constitutes "data ready"? What versioning scheme ensures compatibility? Debugging becomes multi system: when Team C's analytics fail, you trace back through metadata and logs across Teams A and B. You need strong governance to prevent contract breakage. Vertical Consistency vs Horizontal Scalability A strongly consistent central orchestrator with a single metadata database simplifies reasoning. All dependency decisions happen in one place with ACID (Atomicity, Consistency, Isolation, Durability) guarantees. However, this can limit global throughput. A single Postgres instance might handle 10,000 writes per second before becoming a bottleneck. More distributed approaches use independent schedulers per region or team, coordinated through pub/sub. This scales horizontally to handle 100,000+ events per second across multiple systems. The cost is eventual consistency: a dependency might be satisfied in one region before another sees the event. You need careful design around idempotency and conflict resolution. When to Choose Each Approach: Use tight coupling (single DAG) when you have a small team (under 10 people), tightly related steps with no organizational boundaries, or need atomic rollback across all stages. For example, a financial reporting pipeline where all steps must succeed or fail together. Use loose coupling (separate pipelines) when you have multiple teams with different release cycles, need independent scaling of pipeline components, or have cross organizational boundaries. This is the pattern at companies operating at petabyte scale with hundreds of teams like Uber, Netflix, and LinkedIn.