Failure Modes and Edge Cases

When Profiling Systems Break: Data profiling is itself a complex distributed system, and understanding its failure modes is critical for building reliable data platforms. These failures often cascade, turning data quality issues into production incidents. Stale Statistics Misleading Optimizers: In rapidly changing datasets, statistics go stale quickly. A query optimizer using yesterday's cardinality estimates may choose a nested loop join expecting 1,000 matching rows, but the table grew 100x overnight. The query that ran in 2 seconds now takes 5 minutes, timing out and failing dashboards. This is particularly painful during product launches or traffic spikes. Statistics collected during normal 10,000 Queries Per Second (QPS) traffic become wrong when QPS jumps to 50,000. Systems like BigQuery and Snowflake update stats adaptively, but extreme volatility outpaces updates. The mitigation is either profiling more frequently during known volatile periods (expensive) or using robust query plans that degrade gracefully with wrong estimates (complex). Profiling Jobs Saturating Production: Heavyweight profiling can become a Denial of Service (DoS) attack on your own infrastructure. A nightly job profiling 50 TB at peak business hours increases read IOPS by 40 percent, throttling Extract Transform Load (ETL) pipelines and pushing their Service Level Agreement (SLA) from 2 hours to 4 hours. The failure cascades: delayed ETL means stale dashboards, which triggers manual investigations consuming engineering time, which delays feature work. The root cause was not planning for profiling's resource needs. The fix is capacity isolation: dedicated compute pools for profiling, off peak scheduling, or rate limiting I/O to stay below impact thresholds. Semantically Wrong Rules Creating Alert Fatigue: Data quality rules can be logically incorrect, eroding trust in monitoring. A rule enforcing age between 0 and 120 fires constantly on test environments using synthetic ages like 999. A rule requiring non null country is valid for registered users but breaks guest checkout flows where country is collected after order creation. After enough false positives, teams ignore alerts. Then a real data quality incident (payment amounts suddenly all zero due to a code bug) gets missed because the alert is just another noisy ping. The solution is aligning rules with business semantics: separate validation for production versus test data, context aware rules per user journey, and ruthless pruning of low signal alerts. Skewed Distributions Hiding Segmented Issues: Extreme skew masks problems. In a catalog with one product representing 60 percent of orders, overall statistics remain stable even when 20 smaller products have bugs. Average order value, null rates, and distinct customer counts all look normal because the dominant product dilutes the signal. This is especially dangerous in ML pipelines. A model trained on skewed data appears to have good aggregate metrics (90 percent accuracy) but fails completely on minority segments (10 percent accuracy for rare categories). Profiling must be stratified: compute statistics per meaningful cohort (region, product category, user segment) to catch localized degradation. The Cold Start Problem: New datasets have no historical baseline. Profiling a brand new table produces statistics, but are they good? You have no 7 day moving average to compare against. Early alerts are pure guesses, leading to either too many false positives or missed real issues. The workaround is bootstrapping from similar tables, using domain knowledge to set initial bounds, then iteratively tightening thresholds as you collect history. For the first 30 days, alerts might be informational only, requiring human judgment before taking action.