DAG Orchestration Failure Modes

Control Plane Outage: The most severe failure is scheduler or metadata database downtime during the batch window. If your control plane fails for 30 minutes starting at 2:00 AM and your daily batch runs from 1:00 AM to 6:00 AM, the impact cascades. Scheduled tasks don't start. Running tasks continue but their heartbeats are not recorded. When the control plane recovers, tasks that finished during the outage might be marked as failed or "zombie" (running but orphaned). At Airbnb scale with thousands of concurrent tasks, recovery requires reconciling state: querying actual task status from logs, identifying which tasks need rerunning, and reconstructing dependency chains. Manual intervention often takes 1 to 3 hours, causing widespread SLA misses. Dependency Misconfiguration: This is subtle and dangerous because DAGs appear healthy. Consider a metrics publishing workflow. The correct dependency chain is: load raw data, compute aggregates, publish metrics. If you accidentally omit the edge from "compute aggregates" to "publish metrics", both tasks run in parallel. Publish metrics executes first using stale aggregates from yesterday. The DAG shows all green checkmarks, but your dashboard displays incorrect numbers with fresh timestamps, misleading business decisions. This happens in practice when teams modify DAGs incrementally and test individual tasks but not the full dependency graph. Code review catches obvious errors but subtle timing assumptions slip through. Backfill Catastrophes: Your pipeline was down for 3 days due to an upstream API outage. Now you must backfill. Each daily run processes 2 TB and takes 3 hours. Do you run 3 days sequentially (9 hours total, missing today's SLA) or in parallel (6 TB and potentially exceeding cluster capacity)? Most teams choose parallel backfill and discover that 3x load exhausts memory on Spark clusters, causing out-of-memory errors and failed runs. The correct approach requires resource-aware backfill: run 2 days in parallel (4 TB, within capacity) then the third day, but this logic must be built into your orchestrator or handled with manual scheduling. Idempotency becomes critical here. If a backfill run partially completes (wrote 800 GB of 2 TB before crashing), rerunning must either overwrite cleanly or skip already written partitions. Non-idempotent tasks create data duplication or corruption during backfills. Dynamic DAG Edge Cases: Prefect-style dynamic flows introduce runtime surprises. Imagine a flow that discovers customer IDs from a database and spawns one task per customer. Normally you have 100 customers and 100 tasks. A new enterprise customer onboards with 10,000 sub accounts. Your flow now generates 10,100 tasks in a single run. The worker pool, sized for 500 concurrent tasks, is overwhelmed. The control plane tries to enqueue 10,100 tasks, exhausting queue capacity. The metadata database sees 50,000 state writes (10,100 tasks times 5 state changes each) in 30 minutes instead of the usual 2,500, causing write latency to spike from 5ms to 200ms and creating a cascading slowdown across all DAGs. Preventing this requires defensive coding: capping generated task counts, pre-flight checking data cardinality, and failing fast with clear errors rather than attempting impossible workloads. Monitoring Blind Spots: Alerting on DAG failure is table stakes. The dangerous blind spot is alerting only on failure and not on duration anomalies. Consider 50 critical DAGs normally completing in 1 hour. Due to cluster resource contention, p99 duration increases to 1.5 hours but all tasks still succeed. Your alerts stay silent because nothing failed. Meanwhile, downstream reporting misses SLAs. Users see stale data at 9:00 AM when they expect fresh results. This causes business impact despite zero task failures. Robust monitoring requires duration percentile tracking and alerting on regressions: p99 duration exceeds baseline by 25% for 3 consecutive days triggers investigation.