Failure Modes and Edge Cases

Concept Drift vs Real Anomalies: The most common failure mode is confusing legitimate business changes with data quality issues. A successful marketing campaign doubles traffic for three weeks. Row counts jump from 5 million to 10 million daily. A naive detector flags every run as anomalous because values exceed historical ranges. If your team manually suppresses these alerts, you create alert fatigue where engineers start ignoring notifications. Real issues get buried in false positives. Eventually the model learns the new baseline, but during the transition period (often 7 to 14 days), detection is unreliable. AWS Glue mitigates this by allowing users to exclude confirmed anomalies from training data, but that requires human judgment for every major shift. Seasonality and Calendar Effects: Retail traffic around Black Friday or Singles Day can be 10x normal volume. If your detector has not seen enough yearly cycles (ideally 2+ years of history), it treats this as an extreme anomaly. Similarly, workday versus weekend patterns confuse detectors operating at daily granularity. Monday might see 6 million events while Sunday sees 3 million. Without calendar aware features, a Sunday following a Saturday maintenance window (also low volume) might not trigger alerts even though Sunday data is actually missing. Backfill and Reprocessing: When you rerun a monthly job with corrected logic, row counts and distributions can change drastically compared to the buggy version. If detection compares only to the last run, it treats corrected data as anomalous. For example, fixing a join that was dropping 2 million rows suddenly increases output from 8 million to 10 million rows. The detector flags this as a 25% anomaly. One strategy is to scope detection to production forward fills only, explicitly disabling checks for historical backfills. Alternatively, compute separate baselines for backfill windows by comparing the reprocessed data to other backfills of the same time period, not to recent production runs. Partial Failures and Distribution Shifts: An upstream service silently stops sending events for one region due to a configuration change. Total row count drops only 5% (from 20 million to 19 million), within normal daily variance. If you monitor only global row count, you miss the issue entirely. However, the regional composition changed drastically. US events dropped from 8 million to 100, while other regions stayed normal. A robust design monitors dimensions like region, platform, or client version, detecting distribution shifts even when totals look normal. The challenge is that monitoring every dimension creates thousands of metrics. Small volume segments (a new app version with 1,000 daily users) have high random noise, producing spurious alerts. Meta Failures: The anomaly detection system itself can fail. A streaming detector falls behind due to processing bottleneck, operating on data with 1 hour lag instead of the usual 2 minute latency. It misses time sensitive anomalies entirely. Mature platforms treat data quality jobs as first class workloads with their own SLAs, health checks, and monitoring. You need alerts on alert system health: detection lag exceeding 5 minutes, profiler jobs failing, metric store unavailability.