Recommendation Systems • Collaborative Filtering (Matrix Factorization)Hard⏱️ ~3 min
Production Serving Architecture and Latency Budgets
Serving Matrix Factorization recommendations at scale requires a carefully orchestrated pipeline that fits within strict latency budgets. A typical production request for a homepage or feed must complete in 100 to 200 milliseconds end to end, and candidate retrieval from Matrix Factorization is just the first stage. The serving path has three critical steps: fetch or compute the user embedding, perform approximate nearest neighbor (ANN) search over item embeddings, and pass the top candidates to a downstream ranker.
The item embeddings and biases live in a low latency store (in memory cache or key value store) with an ANN index built on top. For 100 million items at 64 dimensions, this is approximately 25.6 GB of raw embeddings plus index overhead (commonly 1.5x to 3x the raw size depending on the ANN algorithm). Sharding across a small cluster keeps lookup latency under 1 to 10 milliseconds. User embeddings are either precomputed and cached (for batch updated users) or computed on the fly by aggregating recent interactions (for real time freshness). On the fly computation might average the item vectors of the user's last 50 plays or run a few Stochastic Gradient Descent (SGD) steps, taking 1 to 5 milliseconds.
Once you have the user vector, the ANN search retrieves the top 500 to 1000 candidate items in 5 to 10 milliseconds with tunable recall (commonly 85% to 95% recall means you find 85% to 95% of the true top K within the search budget). These candidates feed a heavier ranking model that incorporates rich features, context, and business logic, consuming the remaining 50 to 100 milliseconds of the latency budget. The key production insight: Matrix Factorization is optimized for fast, scalable candidate generation (high throughput, low latency, simple scoring), not for the final ranking decision.
Freshness is the toughest operational challenge. Item embeddings are typically rebuilt daily or hourly in batch jobs. User embeddings can be updated more frequently (real time or nearline) but must stay consistent with the item index version. Versioning and synchronized snapshots prevent serving inconsistencies where a user vector computed against yesterday's item factors is scored against today's ANN index, causing score drift or relevance drops.
💡 Key Takeaways
•Typical latency budget: 1 to 5ms for user vector fetch or computation, 5 to 10ms for ANN search over 100M items, 50 to 100ms for downstream ranking. Total end to end: 100 to 200ms
•Item embeddings stored in memory with ANN index. For 100M items at 64 dims: 25.6 GB raw embeddings plus 1.5x to 3x index overhead (40 to 75 GB total). Shard across cluster for sub 10ms lookups
•ANN search trades recall for latency. At 90% recall you find 90% of true top K items in 5 to 10ms. Exact search would take seconds (O(100M) dot products). Tuning recall vs latency is critical
•User embeddings can be precomputed and cached (batch updated) or computed on the fly from recent interactions (real time freshness). On the fly: average last 50 item vectors or run 3 to 5 SGD steps in under 5ms
•Freshness requires versioned embeddings and synchronized ANN rebuilds. User vector from old item factors scored against new ANN index causes drift. Schedule coordinated snapshots and dual read during transitions
•Throughput targets: Tens of thousands of queries per second (QPS) per region. Memory, network bandwidth, and cache hit rates dominate cost at this scale
📌 Examples
Spotify candidate retrieval: User plays a song, system computes user vector from last 50 plays (3ms), performs ANN search over 100M track embeddings with 90% recall (7ms), returns 500 candidates to ranking model. Ranking consumes remaining 80ms of 100ms budget
Netflix two stage pipeline: Matrix Factorization generates 1000 candidates per user in 10ms. These feed a deep neural network ranker with context features (time, device, browse history) that takes 60ms. Final slate of 20-30 titles shown to user