Embeddings & Similarity Search • Dimensionality Reduction (PCA, UMAP)Medium⏱️ ~2 min
PCA vs UMAP: Choosing the Right Technique
The choice between PCA and UMAP is driven by the structure you need to preserve, inference latency budget, and scale. PCA preserves global linear structure and keeps distance relationships approximately consistent for linear manifolds, making it the default choice for retrieval systems where a consistent distance metric is critical. UMAP preserves local neighborhoods well and often reveals cluster structure, making it ideal for visualization and exploratory clustering, but it can distort global distances and relative cluster spacing.
From a latency perspective, PCA is a single matrix multiply taking roughly 100,000 operations for 768D to 128D, feasible at 50,000 QPS across a small CPU cluster. UMAP transform of a new point requires neighbor search and iterative updates, with per point latency of 10 to 100 milliseconds, which is unacceptable in most online request paths. For determinism, PCA is stable and components remain consistent under small data changes, while UMAP has stochastic optimization producing different layouts across runs even with fixed seeds.
In terms of scalability, PCA handles tens or hundreds of millions of points using randomized or incremental methods with streaming computation. UMAP scales to a few million points well with approximate neighbors, but beyond that, graph construction becomes the bottleneck with memory requirements of 2.4 to 4.8 GB for 5M points at 30 neighbors. The interpretability also differs: PCA components are linear combinations of original features that can be inspected or used in model cards, while UMAP embeddings are coordinates without direct interpretability.
💡 Key Takeaways
•PCA for online serving: deterministic matrix multiply with sub millisecond latency scales to 50,000 QPS, preserves global distances for retrieval metrics
•UMAP for offline analysis: stochastic non linear method with 10 to 100ms per point latency reveals local cluster structure for visualization and audits
•Scalability ceiling: PCA handles 100M+ points with streaming methods, UMAP limited to 1 to 5M points before graph construction memory (2.4 to 4.8 GB for 5M) becomes prohibitive
•Structure tradeoff: PCA may collapse non linear manifolds but maintains metric consistency; UMAP preserves local neighborhoods but distorts global cluster spacing
•Interpretability: PCA components are inspectable linear combinations of features; UMAP coordinates have no direct feature interpretation
•Alternatives exist: autoencoders for non linear explicit transforms, random projections for fast Johnson Lindenstrauss guarantees, learned transforms in feature extractors
📌 Examples
Google product search: PCA reduces 768D to 128D in embedding service, applies transform before ANN index lookup, serves 40K QPS at 7ms p95 latency
Meta content moderation: UMAP maps 2M post embeddings offline to 2D for weekly drift dashboards, analysts inspect dense clusters for policy violations
Spotify recommendation: PCA for online candidate retrieval (100ms p95 budget), UMAP for offline music map used by curators to audit genre coverage
Trade space: PCA at 128D gives 98% recall@10 at 7ms; UMAP visualization shows 15 genre clusters but takes 2 hours to compute for 5M songs