Failure Modes and Edge Cases in Production

Unsupervised anomaly detection fails in predictable ways that must be anticipated in production systems. Contamination in training data is a primary risk. Autoencoders trained on data with 2 to 5 percent undetected fraud learn to reconstruct those patterns, which raises the threshold required to catch them and drives up false negatives. Isolation Forest is more robust but heavy contamination above 10 percent biases path lengths and reduces separation. Teams mitigate this by training on periods known to be clean, using active learning to remove flagged anomalies before retraining, and monitoring precision on sampled labels. Concept drift and seasonality shift distributions over time. Changes in customer mix, product launches, holidays, or traffic spikes make previously normal patterns look anomalous. Isolation Forest trained on a two week historic window may mark changed normals as anomalies, spiking false positives. Autoencoders see rising reconstruction error as the manifold drifts. Without retraining or context features like time of day, false positives can jump by 3x to 5x during known events like Black Friday. Solutions include windowing training data to recent periods, adding seasonal features, and using sliding window retraining daily or hourly. AWS CloudWatch retrains per metric models continuously on recent windows to adapt to drift. Clustered anomalies from fraud campaigns create dense groups of similar attacks. Isolation Forest may require more splits to isolate them, which reduces anomaly scores and causes misses. Autoencoders can still detect them if they lie off the learned manifold, but if similar campaigns contaminated training, they get reconstructed well. Adversarial behavior compounds this. Attackers mimic normal feature distributions to evade detectors. Isolation Forest is hard to game globally but can be circumvented by crafting inputs near high density regions. Autoencoders can be manipulated by inputs that follow learned correlations but represent harmful behavior. Ensembles with orthogonal features and periodic adversarial testing reduce this risk. High cardinality sparse features like one hot expansions with tens of thousands of categories make both methods brittle. Random splits on sparse columns can isolate points too quickly, inflating anomaly scores. Autoencoders may collapse to trivial identity along sparse axes. Dimensionality reduction or target encoding is required. Another edge case is rare but important events like emergency spikes, maintenance windows, or product experiments. If the system automatically triggers mitigation, it can suppress real traffic. Safety valves include whitelisting known events, requiring human acknowledgment for high impact actions, rate limiting automated responses, and maintaining a false positive budget per unit time. Score calibration per segment using quantiles ensures uneven false positive rates do not create alert fatigue. Lack of calibration leads to noisy alerts and eventual distrust by operations teams.

Stripe contamination handling: Active learning removes top 0.1% flagged events from training data before autoencoder retraining, reducing false negatives by 25%

PayPal Black Friday: False positive rate jumps from 0.5% to 12% without seasonal features, adding day of week and hour of day reduces to 1.5%

Uber adversarial evasion: Attackers craft transactions near high density regions to evade Isolation Forest, ensemble with autoencoder catches 60% of evasions

Amazon one hot explosion: 50,000 category features cause Isolation Forest to isolate too quickly, target encoding reduces false positives by 40%

AWS CloudWatch maintenance: Whitelisting known deployment windows prevents 90% of false alarms during scheduled maintenance events