Production Failure Modes: Drift, Truncation, and Domain Mismatch

Embedding drift occurs when models are retrained or data distributions shift, rotating the vector space. Old indices built with previous embeddings no longer align with new query embeddings. If you deploy a new SBERT model for query encoding without rebuilding the document index, retrieval quality can collapse. Cosine similarities become meaningless because the two vector sets occupy different geometric spaces. The solution is versioned embedding spaces with dual write during migration: compute both old and new embeddings, build a shadow index, validate recall and quality metrics, then cut traffic atomically. At Google and Meta scale, backfilling billions of embeddings takes days to weeks, requiring careful orchestration. Tokenization and truncation cause subtle failures. BERT models truncate inputs at max length, typically 512 tokens. Long documents lose tail content, which may contain critical information. Unicode normalization and punctuation handling can drift between offline embedding pipelines and online query processing, producing vectors that should match but diverge slightly. These mismatches accumulate and degrade recall. Use consistent tokenizer versions and normalization across offline and online paths. For long documents, chunk into overlapping segments, compute embeddings per chunk, and aggregate via pooling or max similarity. Monitor vector norm distributions and cosine histograms to detect processing inconsistencies. Domain mismatch is severe when general pretrained models encounter specialized vocabulary. BERT trained on Wikipedia and news performs poorly on clinical notes, legal contracts, or scientific papers where jargon dominates. Domain specific terms cluster incorrectly, causing false positives in retrieval. For example, medical abbreviations might map to unrelated general words. Fine tuning on in-domain sentence pairs or adding domain adapters improves alignment by 10 to 30 percent on specialized benchmarks. Collect representative query and document pairs from production logs, label similarity, and fine tune with contrastive objectives. Approximate nearest neighbor indices introduce recall loss. Aggressive quantization and small probe counts reduce latency but miss true neighbors. Monitor recall on a held out gold set. If recall drops below 0.85, retrieval quality degrades noticeably in user metrics. Build safeguards by blending vector results with keyword search using BM25. Popularity bias in graph embeddings causes recommender systems to overfit to head items, starving new or niche content. Use exploration strategies and debiasing techniques during training. Adversarial inputs and privacy leaks through vector similarity are emerging concerns: screen inputs, apply filters, and enforce access controls on vector stores.

Search system deployed new SBERT model without rebuilding index: recall dropped from 0.91 to 0.63, required 2 week backfill to recompute 100 million document embeddings with new model

Medical document retrieval using general BERT: clinical abbreviations like CHF and MI clustered with unrelated common words, fine tuning on 50K medical sentence pairs improved Mean Average Precision (MAP) from 0.72 to 0.88

Long legal contract search: truncation at 512 tokens missed clauses in tail sections, chunking into 256 token overlapping segments with max similarity aggregation recovered 12 percent recall

Recommendation system with graph embeddings: 80 percent of traffic went to top 5 percent of items, added exploration with epsilon greedy (10 percent random) and collected data to improve cold start item embeddings