Production Implementation: Training Pipelines and Serving Architecture for Learning to Rank

Building a production learning to rank system requires careful orchestration of data logging, feature engineering, model training, and low latency serving. The pipeline starts with comprehensive logging of every query, the candidate set shown, the final positions, user interactions like clicks and dwell time, and downstream conversions, all timestamped and linked to user context. This log data is the foundation for training and must capture enough detail to reconstruct the ranking decision and user response. At scale, this means billions of rows per day with sub millisecond logging overhead per request. Feature engineering is the next critical step. Maintain a feature store that serves query features like language and intent, item features like popularity and quality scores, and cross features computed on demand such as BM25 text match and embedding similarity. Precompute slow item features offline and update them hourly or daily in batch jobs. Compute expensive cross features online only for the candidate set, not the entire catalog. For example, computing embedding similarity between a query and 500 candidates takes 2 to 5 milliseconds if embeddings are cached, but computing it for millions of items would take seconds. Normalize features to ensure stability across traffic segments and time, and version features so that training and serving use consistent definitions. Training uses time based splits to avoid leakage: train on data up to day T, validate on day T plus 1, and test on day T plus 2. This prevents the model from seeing future information and ensures metrics reflect real world performance. For pairwise models, sample 10 to 50 pairs per query per epoch, focusing on pairs with high NDCG delta. For listwise models, weight examples by query traffic to emphasize high volume queries that drive most user impact. Training datasets for large systems include millions of queries and hundreds of millions to billions of pairs. Distributed training on 20 to 50 machines trains a gradient boosted tree ensemble with 200 trees in 4 to 8 hours. Neural listwise models require more compute, often training on GPUs or TPUs for 12 to 24 hours with careful regularization to avoid overfitting list context. Serving architecture uses a cascade. Retrieval returns 5,000 to 50,000 candidates from an inverted index or approximate nearest neighbor search in 5 to 30 milliseconds. A lightweight ranker scores these with 50 to 100 fast features, narrowing to 300 to 800 items in another 10 milliseconds. The final learning to rank model scores this set using 200 to 1,000 features within a 10 to 30 millisecond budget. For gradient boosted trees, this is 50 to 60 microseconds per item on CPU, so 500 items take 25 milliseconds. Neural models are slower at 200 to 500 microseconds per item, so they may score only the top 50 to 100 items or run on GPU. Fallback and monitoring are essential. If the feature service times out, degrade gracefully to a simpler ranker that uses only cached features. If the model service is down, fall back to a rule based ranker or a cached model. Roll out new models with shadow traffic first, comparing score distributions and top k overlap with the production model. Then ramp with A/B tests, monitoring NDCG, click through rate, conversion rate, and latency percentiles. Watch for anomalies like sudden drops in diversity or coverage, which can indicate training bugs or data drift. Retrain frequently: daily for fast moving catalogs like e commerce or news, weekly for slower domains like job search. At LinkedIn, the feed ranking model retrains every 8 hours on fresh engagement data, catching trends and seasonal shifts quickly while maintaining stable performance.

Airbnb scoring pipeline: retrieval returns 8,000 listings in 20ms, lightweight ranker with 60 features narrows to 400 in 10ms, final GBDT with 420 features scores in 22ms, total latency 52ms with NDCG@10 of 0.78.

Google uses a neural listwise model on TPUs for the final re-ranking stage, scoring the top 100 candidates in 30ms p99 with 800 features after tree based rankers narrow from millions of pages.

Pinterest feature store serves 200 precomputed item features (popularity, quality) with 5ms p95 latency and computes 150 cross features (embedding similarity, interaction history) online for 600 candidates in 8ms.