Production NER Implementation: Training, Serving, and Monitoring

Building production grade NER requires careful orchestration of data pipelines, model architecture, serving infrastructure, and continuous monitoring. Each component has specific design patterns that separate experimental systems from those serving billions of inferences. Data quality determines ceiling performance. Start with a clear annotation guide that defines entity types, boundary rules, and handling of ambiguous cases. Measure inter annotator agreement using Cohen's kappa; target above 0.8 on a stratified sample to ensure consistent labels. For domain specific work, 50 thousand to 200 thousand labeled sentences yield strong results, but quality matters more than quantity. Augment with weak supervision: use pattern matchers and distant supervision from knowledge bases to generate noisy pre training labels on millions of sentences, then clean a smaller high quality validation set of 5 to 10 thousand examples for final evaluation. This hybrid approach reduces annotation cost by 5 to 10 times while maintaining model quality. Model architecture follows a standard pattern for maximum accuracy. Fine tune a pre trained transformer encoder (BERT, RoBERTa, or domain specific variants) with a token classification head. Optionally add a Conditional Random Field (CRF) decoder on top to enforce legal label transitions (prevent "I ORG" without a preceding "B ORG"). The CRF adds 1 to 2 milliseconds of latency but improves F1 by 1 to 2 points by eliminating invalid sequences. For latency critical paths, distill the teacher model into a smaller student (6 layers instead of 12), quantize weights to int8, and export to an optimized runtime like ONNX or TensorRT. Distillation and quantization together reduce latency by 2 to 4 times with only 1 to 3 point F1 loss, making the difference between 15 millisecond and 4 millisecond p95 latencies on CPU. Serving patterns diverge by workload. Online inference deploys stateless microservices behind load balancers, keeping batch size at 1 for predictable latency. Enable dynamic batching with a 2 to 5 millisecond batching window that accumulates up to 8 requests during traffic spikes, improving GPU utilization without sacrificing p95 latency guarantees. Co locate an in memory feature store or gazetteer service that boosts predictions with dynamic entity lists updated every few minutes. Cache frequent inputs using a Least Recently Used (LRU) cache with Time To Live (TTL) of 5 to 60 minutes; hot query caches in consumer search often reach above 80 percent hit rates, collapsing average latency to under 2 milliseconds. For offline batch processing, partition documents by identifier and process in parallel with large batch sizes (64 to 256 sequences) to saturate GPU memory bandwidth. Use distributed frameworks like Apache Beam or Spark to schedule work across GPU clusters, with checkpointing every few million documents to enable restarts without reprocessing. Monitor job progress and throughput (tokens per second per GPU), and autoscale clusters based on queue depth to meet Service Level Agreements (SLAs) for data freshness. Post processing and entity linking close the loop. Normalize dates and monetary values to standard formats, expand common abbreviations ("Corp." to "Corporation"), validate organization suffixes against allowed lists, and map surface strings to canonical identifiers using a resolver service. The resolver embeds entity mentions, retrieves top candidate ids from a vector index, then reranks using context features. Maintain an override list for policy sensitive entities that takes effect within minutes without model retraining, allowing compliance teams to add or remove surface forms instantly. Monitoring is continuous. Track entity level precision and recall by type on a rolling labeled sample of 1 to 5 thousand examples refreshed weekly. Monitor distribution drift in token lengths, casing, character sets, and entity type frequencies. Track latency percentiles (p50, p95, p99), throughput (requests per second), error rates, and cache hit rates. Set explicit Service Level Objectives (SLOs): maintain p95 latency under 10 milliseconds for online systems, p99 under 20 milliseconds. Alert when PII recall drops below 95 percent on the canary set or when organization precision falls below 90 percent. Use shadow deployments to test new models against production traffic before cutover, and implement progressive rollouts (1 percent, 5 percent, 25 percent, 100 percent) with automatic rollback if metrics regress beyond thresholds. Keep a fallback rule based extraction mode that activates when the model service is degraded, ensuring critical redaction and compliance features continue operating with conservative precision even during outages.

Google production NER: Fine tuned RoBERTa with CRF achieves 92% F1, distilled to 6 layer model serves 10k QPS per GPU with 4ms p95 latency, monitored with hourly F1 checks per entity type

Amazon product extraction: Offline batch processes 500M titles nightly with 128 batch size on 50 GPU cluster, checkpointing every 10M documents, completing in 6 hours

Meta content moderation: Real time NER caches 150k hot queries with 85% hit rate, reducing average latency from 8ms to 1.5ms, with automatic fallback to regex rules during model outages

Microsoft PII redaction: Weekly retrained model deployed via shadow mode at 1% traffic for 24 hours, promoted to 100% only after maintaining >95% recall on canary set for 3 days