Failure Modes and Edge Cases in Production Semantic Search

Production semantic search systems face failure modes that are not obvious from offline evaluation. Understanding these edge cases is critical for building robust systems and is a key differentiator in system design interviews. Embedding drift occurs when you upgrade your embedding model. The new model produces vectors in a different space, and old and new vectors are not comparable. Mixing them in one index collapses recall because distances are no longer meaningful. For example, upgrading from a 256 dimensional model to a 768 dimensional model, or even retraining the same architecture, changes the embedding space. The mitigation is dual indexing during migration: shadow write embeddings for both old and new models, run both indices in parallel, and gradually shift traffic after backfilling the corpus. This requires careful planning and 2 times storage during the transition. Poor score calibration is subtle but damaging. Cosine similarity and inner product produce scores that are not calibrated across different queries. A score of 0.8 on one query might indicate high relevance, while 0.8 on another query is the best available but still marginal. Applying a global threshold like "return results only if cosine above 0.7" leads to inconsistent behavior: some queries return nothing (false negatives), others return spam (false positives). Mitigate by normalizing per query, for example by standardizing scores using the mean and standard deviation of the candidate distribution, or by learning a query specific threshold model. Provide an abstain path: if the top result is below a learned threshold, fall back to keyword search rather than returning poor results. Filter leakage happens because ANN structures are not inherently filter aware. If you apply filters after ANN retrieval (for example, "language equals English" or "tenant equals customer A"), you might discard 90 percent of the nearest neighbors and return only a few items, hurting the user experience. Solutions include building separate indices per major filter dimension (for example, per language or per tenant), using a prefilter with an inverted index to get candidates before ANN, or using filter bitmaps to constrain the graph or partition search. Each approach has overhead: separate indices multiply memory, prefiltering adds latency, and bitmap integration requires custom index logic. Head tail skew in training data degrades long tail recall. Very frequent entities dominate the training set for IVF centroids or quantizer codebooks, so the index optimizes for the head. Rare or niche queries suffer lower recall. For example, training k means on raw data might create 10 percent of centroids around celebrity names and only a few centroids for obscure technical topics. Mitigate by reweighting or stratifying training samples, and monitor recall by popularity decile in offline evaluation. If long tail recall drops below 80 percent while head recall is 95 percent, you have a skew problem. Multilingual and cross domain queries are failure modes for monolingual encoders. If you index documents in multiple languages with a single language encoder, queries in language A will not match documents in language B, even if they are semantically identical. The solution is either a multilingual encoder (like multilingual sentence transformers) or separate indices per language with language detection at query time. Similarly, mixing very different domains (for example, e commerce products and technical documentation) in one index with one encoder can lead to poor embeddings for each domain. Domain specific fine tuning or separate indices often work better. Adversarial or out of distribution inputs are a real risk. Vague queries like "things" or "stuff" collapse to generic neighbors that are not useful. Very short queries (one or two words) produce dense clusters, making it hard to distinguish relevance. Embeddings can also leak personally identifiable information (PII) through nearest neighbor memorization: if a document contains sensitive data, a crafted query can retrieve it even if access controls should block it. Mitigations include minimum distance thresholds to abstain on vague queries, hybrid retrieval with must have keyword terms, query expansion to add context, and enforcing content governance and access checks before returning results. Freshness gaps and replication lag are operational failures. If your online index supports eventually consistent inserts, recent items might not appear for seconds to minutes. For recency sensitive queries like news search or social feeds, this hurts relevance. Monitor replication lag and expose a best effort recent queue for high priority new items. Numeric instability can also cause subtle bugs. Large vector dimensions with float16 quantization can lead to underflow in norm calculations, producing inaccurate cosine scores. HNSW with very high efSearch can spike CPU and cause tail latency violations. Set query budgets and rate limits to protect the cluster.

Google upgrades embedding model from 256d to 384d: runs dual indices for 2 weeks, shadow writes both embeddings, backfills 10 billion documents over 5 days, then cuts traffic to new index after A/B test shows 3 percent relevance improvement

Elasticsearch hybrid filter: prefilter documents with "language:en" using inverted index to get 1 million candidates, then run HNSW ANN on that subset to get top 100, avoiding filter leakage and maintaining recall

Pinterest monitors embedding drift by tracking p50 and p95 of nearest neighbor distances over time; detects 15 percent increase in average distance after data distribution shift, triggers model retraining

News search uses per query score calibration: for each query, compute z score of candidates using mean and std of top 200 scores, then threshold at z score above negative 1.5 to abstain on poor matches