Failure Modes and Edge Cases in dbt Workflows

Silent Data Corruption: The most dangerous failure in a dbt workflow is when models run successfully from an infrastructure perspective but produce wrong results. The scheduler reports success, tests pass if they are too weak, and incorrect data propagates downstream. In a system with 500 models, a single wrong aggregation in a core intermediate model can corrupt hundreds of dashboards and dozens of downstream models. Real world example: an incorrect revenue calculation that double counts refunds runs for 6 hours before detection. During that time, executives make decisions on inflated revenue metrics, customer success teams use wrong churn numbers, and finance reports incorrect figures. The blast radius extends to every model and dashboard referencing that core revenue logic. Schema Drift from Sources: Upstream ingestion pipelines evolve independently. When a source system adds, renames, or removes columns, staging models may fail or worse, silently produce partial results. The trade off is between strict contracts and resilience. Failing fast on any schema mismatch improves correctness but reduces availability. If your ingestion adds a new column, should dbt models fail or pass it through? At 10x scale with dozens of source systems, you must have explicit conventions. Some teams version their staging models and run schema change detection in CI. Others use permissive patterns that allow new columns but alert on removals. The choice depends on how often schemas change and how critical freshness is. Performance Degradation and Cost Spikes: A model that normally scans 100 GB and completes in 2 minutes suddenly starts scanning 1 TB after a join logic change. Runtime jumps to 20 minutes, warehouse costs spike by 10x, and downstream models miss freshness SLOs. This cascades: models that depend on the slow one queue up, and the entire pipeline that normally finishes in 15 minutes at p95 now takes 45 minutes at p95. Mitigation requires monitoring runtime distributions. Alert when p95 runtime for any model exceeds its baseline by 2x. Catch regressions in staging before they hit prod. Some teams run cost estimation in CI, failing builds if query costs exceed thresholds. Backfill and Partial Run Challenges: When you need to recompute months of historical data after a logic fix, naive full refreshes can scan tens of terabytes and cost thousands of dollars. For a fact table with 10 billion rows spanning 2 years, a full rebuild might take 8 hours and scan 5 TB per run. Incremental models help, but backfilling requires careful partition aware logic to avoid reprocessing everything. Concurrency is another edge case. Running multiple overlapping dbt jobs that write to the same tables creates race conditions. On warehouses with strict concurrency limits, you might hit lock contention or serialization errors. Teams solve this by time boxing runs (only one prod run per hour window) or using separate target schemas for concurrent dev work.