What Is Embedding Quality Evaluation?

Definition
Embedding quality evaluation measures how well vector representations capture the similarity relationships needed for your downstream task—whether search, recommendations, or classification.
WHY EVALUATION MATTERS
Embeddings can look reasonable in visualization tools but fail in production. Two items might be close in embedding space but completely unrelated for your use case. Evaluation catches these failures before they affect users.
The core question: do similar items (as defined by your labels, clicks, or purchases) have similar embeddings? If the correlation is weak, your retrieval will surface irrelevant results regardless of how sophisticated your ANN index is.
INTRINSIC VS EXTRINSIC METRICS
Intrinsic metrics: Measure embedding properties directly. Clustering quality, alignment with known similarity labels, distance statistics. Fast to compute, useful for debugging. Examples: silhouette score, alignment@k.
Extrinsic metrics: Measure performance on actual downstream tasks. Retrieval recall, classification accuracy, recommendation CTR. Slower to compute, but directly measures what you care about. Examples: NDCG@10, recall@100.
Rule of thumb: intrinsic metrics for fast iteration during development, extrinsic metrics for final decisions and production monitoring.
KEY METRICS FOR RETRIEVAL
Recall@K: What fraction of true relevant items appear in top K results? recall@100 = 0.90 means 90% of relevant items are in the first 100 candidates. Critical for two-stage retrieval where Stage 2 cannot fix Stage 1 misses.
NDCG@K: Measures ranking quality—are relevant items ranked at the top? Accounts for position: item at rank 1 matters more than rank 100. NDCG@10 = 0.85 is good; below 0.7 indicates ranking problems.
💡 Key Insight: Always evaluate on held-out data that represents your production distribution. Training-set performance is misleading—embeddings memorize training examples.

💡 Key Takeaways

✓Intrinsic metrics measure embedding properties directly; extrinsic measure downstream task performance

✓Recall@K: fraction of relevant items in top K results—critical for two-stage retrieval

✓NDCG@K: ranking quality accounting for position—NDCG@10 above 0.7 is acceptable

📌 Interview Tips

1Interview Tip: Explain intrinsic vs extrinsic—intrinsic for debugging, extrinsic for production decisions.

2Interview Tip: Describe why recall@K matters for two-stage retrieval—Stage 2 cannot fix Stage 1 misses.

← Back to Embedding Quality Evaluation Overview