Failure Modes: Concept Drift, Adversarial Attacks, and Cold Start

Supervised anomaly detection systems fail in predictable ways. Recognizing these failure modes and building mitigation into the architecture is what separates experimental models from production grade systems. The three most critical failures are concept drift from adversaries, misleading metrics that hide poor performance, and cold start on new segments without history. Concept drift is inevitable in adversarial domains. Fraudsters probe model edges and adapt when patterns get blocked. Features with high importance decay quickly after deployment. A device fingerprint feature that provides 30% lift in the first week may drop to 10% lift after a month as fraudsters rotate devices. Payment processors expect weekly drift on top features. Mitigation includes model refresh cadences of 1 to 2 weeks, drift monitors on feature and score distributions (alert on Population Stability Index over 0.2), and automatic threshold retuning based on recent cost curves. Stripe retrains the primary fraud model every 7 days and runs 3 challenger models in shadow to detect when refresh is needed earlier. Misleading metrics kill projects. ROC AUC can show 0.95 while precision at the operating point is under 20%. This happens because ROC AUC averages performance across all thresholds, including irrelevant low threshold regions where you would never operate. One team shipped a model with 0.96 ROC AUC that produced 18% precision at 60% recall, far below the 80% precision target. Always monitor Precision Recall AUC, precision at K (precision in the top 1% of scores), expected cost on validation, and action specific metrics like review precision. Track these metrics separately for different merchant segments and time windows because aggregates hide pockets of failure. Cold start appears whenever you encounter new entities without history. New merchants, devices, payment methods, or geographic markets lack the behavioral features that drive model performance. A global fraud model trained on US and Europe fails when deployed to a new country in Asia where fraud patterns differ. New device fingerprints have no velocity features. Mitigation includes conservative defaults (route to review), external signals like consortium risk scores and IP reputation feeds, and rule based fallbacks. After 100 to 1,000 events accumulate for a new segment, switch from rules to the learned model. Amazon uses separate models for new seller accounts (first 30 days) versus established sellers because feature distributions and fraud rates differ by 5x to 10x.

Stripe fraud model: device fingerprint feature drops from 30% to 10% importance over 4 weeks as fraudsters rotate devices, triggers weekly retraining and feature engineering

PayPal shipped model with 0.96 ROC AUC but 18% precision at 60% recall operating point, failed to meet 80% precision SLA, reverted to previous model

Amazon new seller cold start: first 30 days use rule based scoring with 90% review rate, after 1,000 transactions switch to learned model, reduces review to 5%

Uber geographic expansion: US trained safety model failed in India with 30% drop in recall, retrained on local data with region specific features like vehicle type and payment method