Failure Modes: Upstream Delays and Cascading SLA Violations

The Ripple Effect Problem: The most common and dangerous failure mode in cross pipeline systems is upstream delay rippling through many downstream pipelines, causing cascading SLA violations. A single bottleneck can create a domino effect that impacts dozens of dependent jobs. Consider a real scenario. Pipeline A ingests clickstream data with expected runtime of 1 hour, finishing at 1:00 AM. Pipeline B normalizes that data, taking 20 minutes, finishing at 1:20 AM. Pipelines C, D, E, and F depend on B, each taking 30 to 60 minutes, all expected to finish by 3:00 AM for morning business reports. One night, traffic spikes 3x due to a viral event. Pipeline A now takes 3 hours, finishing at 3:00 AM instead of 1:00 AM. Pipeline B starts late and finishes at 3:20 AM. Now pipelines C through F all start 2 hours late and finish between 4:00 AM and 5:00 AM, completely missing their 3:00 AM SLA. Business stakeholders see stale data and trust in the platform erodes. The Invisible Data Quality Failure: A more subtle and dangerous failure mode is when upstream produces partial or corrupted data but does not fail visibly. Suppose an ingestion job has a bug that causes it to write only 50% of expected rows, but the job still exits with success code because it did not crash. Or a schema change upstream breaks an assumption downstream, but data validation is weak. Downstream pipelines happily run and produce incorrect metrics. Revenue reports show half the actual numbers. Recommendation features train on biased samples. These bugs can go undetected for days or weeks because the pipelines show green status in monitoring. Handling Delays with Explicit States: Robust systems do not just mark partitions as SUCCESS or FAILED. They use explicit LATE or DELAYED states in metadata. When pipeline A misses its expected completion time by some threshold (for example, 30 minutes), metadata is updated to status=LATE. Downstream pipelines B and C can then make policy decisions: First, skip and proceed with degraded inputs if the data is optional. Second, wait up to a timeout threshold then fail explicitly to prevent invalid results. Third, send alerts to on call engineers while continuing to wait. Fourth, fall back to previous day's data if business logic allows. This requires configuration at the dependency level: wait_policy=TIMEOUT, max_wait_minutes=60, fallback_partition=PREVIOUS_DAY. Without explicit policies, you get unpredictable behavior where some jobs timeout arbitrarily and others wait forever. Backfills and Version Aware Dependencies: Backfilling is particularly tricky. Suppose you discover a bug affecting the last 7 days. You fix pipeline A and rerun it for those dates. How do dependent pipelines B and C detect that newer versions of data are available? Without version tracking, you risk mixing old and new data. Pipeline B might have already run for Dec 20 using buggy data from A. When A reruns Dec 20, B does not automatically reprocess unless you manually trigger it. At scale with hundreds of dependent jobs, manual triggering is error prone. Solution: partition plus version keys. Metadata tracks dataset=events, partition=2025-12-20, version=2. When A backfills, it writes version 2. Dependencies are configured to always use latest version: required_version=LATEST. The orchestrator detects the new version and automatically schedules dependent backfills for the affected date range. Cross Platform and Cyclic Dependencies: Another edge case appears with cross platform or cross environment dependencies. For example, a dbt project owned by Team A in one environment is used by Team B's Airflow DAG in another environment. Authentication, network access, and package versioning issues create failures like "permission denied" when cloning private repositories at runtime. Some teams solve this by packaging shared code or models and deploying them with the orchestrator runtime rather than fetching dynamically. The failure mode is pipelines running with stale shared code if deployment coordination is weak. Cyclic dependencies can creep in over time. Pipeline A depends on aggregated metrics computed by Pipeline C, which itself depends on transformations from Pipeline B, which depends on A. In a single DAG this is rejected as a cycle. Across separate pipelines, it is hidden and causes deadlocks or infinite retries. Governance and dependency analysis tools are needed to detect and prevent such topologies.