Embeddings & Similarity Search • Dimensionality Reduction (PCA, UMAP)Medium⏱️ ~2 min
Principal Component Analysis (PCA) for Online Systems
Principal Component Analysis (PCA) is a linear dimensionality reduction method that finds orthogonal directions, called principal components, that capture maximum variance in the data and projects points onto the top k components. It can be computed by eigen decomposition of the covariance matrix or by Singular Value Decomposition (SVD). PCA is deterministic, fast, and yields an explicit linear transform that is cheap to apply at inference time.
The key advantage for production systems is that PCA is a stateless projection, just a single matrix multiply per vector. For projecting 768 dimensions to 128, that is roughly 98,000 multiply add operations per vector. At 50,000 queries per second, this translates to about 5 billion operations per second, which is feasible across a small CPU cluster or a single low end GPU. A 768 to 128 projection adds roughly 100,000 floating point operations per vector, and at 10,000 QPS per instance, this is about 1 billion floating point operations per second of overhead.
PCA preserves global linear structure and keeps distance relationships approximately consistent for data that lies on linear manifolds. It works best when the original high dimensional data has linear correlations. Many companies apply PCA before compression in ANN systems, using a linear decorrelation transform followed by product quantization or scalar quantization to improve quantization error. In practice, projecting to 256 or 384 dimensions then applying 8 bit quantization per sub vector yields 10x memory reduction with under 2 percent recall drop in large catalogs.
💡 Key Takeaways
•PCA is deterministic and produces a stateless linear transform (matrix multiply) that takes about 100,000 floating point operations for 768D to 128D projection
•Inference latency is predictable and low: single matrix multiply fits in online request paths serving 10,000 to 50,000 QPS per instance
•Preserves global linear structure and pairwise distances for approximately linear manifolds, making it suitable for retrieval systems requiring consistent distance metrics
•Synergizes with product quantization: applying PCA before quantization improves quantization error and achieves 10x memory reduction with under 2 percent recall drop
•Scales to tens or hundreds of millions of points using randomized SVD or incremental methods with streaming computation over data lakes
•Requires careful preprocessing: fit only on training data to avoid leakage, standardize features to prevent high variance dimensions from dominating components
📌 Examples
Embedding service pattern: Apply PCA transform (center, multiply, renormalize) before writing to vector store, reducing network payload from 3KB to 512 bytes
Google and Meta scale: Use randomized SVD on 100M vectors with 768D stored in data lake, compute top 128 components with multiple streaming passes
Index compression: 256D vectors quantized with 16 sub vectors at 8 bits per code = 16 bytes per vector, 1.6GB for 100M items in RAM resident index
Python sklearn: pca = PCA(n_components=128); pca.fit(train_embeddings); reduced = pca.transform(new_embeddings)