The Evaluation Pipeline Architecture

The Full System: At companies like OpenAI, Anthropic, or Google, LLM evaluation is not a one time test before launch. It is a continuously running service tightly integrated into the training and deployment pipeline. Understanding this architecture is critical because it shows how evaluation scales from thousands of prompts in research to millions in production. Here is how the pieces fit together. Component 1: Policy Repository This is version controlled documentation defining harm categories like self harm, hate speech, sexual content with minors, malware, fraud, and personal data disclosure. Each category has severity levels, edge case guidelines, and example prompts. Product teams treat policy changes as backward incompatible events requiring full model re evaluation. Think of this as your safety contract. Component 2: Prompt Generator Instead of static test sets, you maintain parameterized templates and automated generators. Hand written templates cover classic jailbreak patterns like roleplay attacks ("pretend you are..."), third person requests, or multi step manipulations. Model assisted generators take a harm category and propose candidate prompts designed to bypass guardrails. For a 70B parameter model release, you might generate 1 million to 10 million evaluation prompts covering all risk areas. Component 3: Evaluation Runner This internal service selects model variants, runs prompts at scale using batch inference across thousands of Graphics Processing Units (GPUs), and logs all inputs, outputs, and metadata. For cost efficiency, teams often start with smaller proxy models (7B or 13B parameters) to triage ideas before scaling to the full model. Target throughput is typically 5,000 to 20,000 queries per second (QPS) with p95 latency around 300 to 800 milliseconds per prompt. Component 4: Scoring Layer Outputs flow to safety classifiers, toxicity models, and LLM judge models that answer questions like "Does this violate self harm policy?" Judge models can score thousands of prompts per second, enabling continuous wide coverage. A sample of outputs (typically high severity or ambiguous cases) goes to human raters for ground truth calibration. Scores aggregate into metrics like attack success rate per category and refusal rate on benign prompts. Component 5: Release Gate Continuous Integration (CI) and model deployment pipelines query the metrics store and enforce policies. For example, block promotion if critical category regression exceeds threshold, or require manual review for non critical regressions. Dashboards show trend lines across model versions, helping teams understand whether safety is improving or degrading with each training run.