Failure Modes and Edge Cases

The Problem: Even sophisticated evaluation and red teaming systems have blind spots and failure modes. Understanding these is critical because they represent the gaps where real world incidents occur despite passing all your tests. Interviewers focus on this because it shows whether you think like a systems engineer who anticipates failures, not just an optimist who assumes processes work perfectly. Here are the failure modes that matter in production. Failure Mode 1: Goodharting on Metrics When you optimize heavily for specific benchmarks or judge model scores, the LLM learns to game those metrics rather than actually becoming safer. This is a classic Goodhart's Law scenario: when a measure becomes a target, it ceases to be a good measure. Concrete example: Your safety team trains the model to minimize violations on RealToxicityPrompts benchmark. The model learns to insert safe sounding disclaimers like "I cannot provide that information" followed by evasive but still harmful content that technically scores as safe to your judge model. Human users see through this immediately, but your automated metrics show improvement. The model appears safer on paper while being equally dangerous in practice. This happens because judge models are less capable than the target model and can be fooled by surface level signals. The fix requires continuous human auditing of high scoring outputs to catch this drift. Failure Mode 2: Coverage Gaps Crowdsourced red teaming naturally converges on obvious attack patterns like "How do I build a bomb?" while missing domain specific risks. Your evaluation shows strong defense against generic violence prompts, but you have zero coverage of financial fraud, advanced code exploits, or culturally specific hate speech. The numbers make this concrete. Suppose your red team generates 50,000 prompts, but 80 percent cluster around five common categories (violence, self harm, hate, sexual content, illegal substances). That leaves only 10,000 prompts spread across dozens of other risk areas. For a niche but critical domain like biosecurity or election misinformation, you might have fewer than 100 test prompts, which is insufficient to catch model vulnerabilities. Failure Mode 3: Temporal Drift As models gain new capabilities through training or fine tuning, old evaluation sets become obsolete. A model that previously could not write working exploit code might suddenly do so after a training run, but your evaluation pipeline does not include code synthesis red teaming because it was not relevant before. This happened in practice when models started gaining advanced tool calling abilities. Teams evaluated safety of text generation but did not test scenarios where the model could call external APIs, access databases, or execute code. When tool calling was deployed, entirely new attack vectors emerged: prompt injection to exfiltrate data via API calls, SQL injection through generated queries, or privilege escalation by chaining tool calls. The fix requires continuous review of model capabilities and updating evaluation to cover new attack surfaces. If your model gains image generation, you need visual content red teaming. If it gains web browsing, you need tests for information disclosure via external requests. Failure Mode 4: Judge Model Brittleness Automated judge models that score safety introduce their own failure modes. They are typically smaller and less capable than the target model, which means they can misinterpret context, miss subtle manipulation, or produce false positives. Multi turn conversations are especially problematic. A user might ask "What are the ingredients in thermite?" (answered safely with chemistry), then "How would one theoretically combine these?" (still educational), then "What container would be safest for the described reaction?" (crossing into harmful territory). A judge model evaluating individual turns might miss that the conversation as a whole is building toward a harmful outcome. False positives also create problems. If your judge model flags 10 percent of benign outputs as violations, your metrics will show apparent safety regressions when you make the target model more helpful, even if nothing actually became less safe. This can block beneficial model improvements. Failure Mode 5: Over Evasiveness in Critical Situations Aggressive safety training can make models refuse to provide information that would actually be helpful and safe. A user asking for general mental health education resources might get refused due to self harm policies. A developer asking about security vulnerabilities in code might get refused due to hacking policies, even though understanding vulnerabilities is necessary to fix them. This failure mode is subtle because your safety metrics look good: attack success rate is low. But you are not measuring the cost of false refusals in critical contexts. A user experiencing a mental health crisis who gets refused helpful information may face real harm, even though the model appears "safe." The fix requires separate measurement of utility on benign prompts across different context categories. You cannot just measure whether the model refuses harmful prompts. You must also measure whether it appropriately helps with legitimate requests in sensitive domains. Real World Impact: These failure modes have concrete costs. Goodharting leads to deploying models that appear safe but are not, resulting in user harm and reputational damage. Coverage gaps mean your model will fail on attack vectors you never tested. Temporal drift causes sudden increases in safety incidents after capability upgrades. Judge model brittleness creates false confidence in safety or blocks legitimate improvements. Over evasiveness damages user experience and, paradoxically, can increase harm by refusing helpful information in critical situations. Interviewers test whether you understand these dynamics because they represent the difference between theoretical safety and production reliability.