Position Bias and Label Debiasing Techniques

Users click items shown at the top of results far more often than identical items ranked lower. This position bias corrupts training labels. If you rank items by historical Click Through Rate (CTR) without correction, high CTR reflects past exposure rather than true quality. The ranker locks in the existing order, preventing better items from ever rising. This creates a feedback loop where popular items stay popular simply because they were shown first. Position bias is multiplicative. An item at position 1 might receive 10x more clicks than the same item at position 5, even with identical relevance. A naive ranker trained on these labels learns that items previously ranked high are good and items ranked low are bad, regardless of intrinsic quality. New items with zero exposure history start with zero CTR and never get a chance. The system ossifies around whatever the initial ranking happened to be. Inverse Propensity Weighting (IPW) corrects this by estimating how much each position inflates click probability. Allocate 1 to 5 percent of traffic to exploration, where item positions are randomized within reasonable constraints. Log the displayed position and the probability of showing the item at that position, called the propensity. During training, weight each labeled example by the inverse of its propensity. An item clicked at position 10, where propensity is low, receives much higher weight than an item clicked at position 1, where propensity is high. This rebalancing recovers unbiased relevance estimates. Alternatively, use counterfactual learning objectives that directly model position effects. Encode the displayed position as a feature during both training and serving. Train with data that includes varied positions from exploration traffic. At inference time, score candidates as if they were all at position 1 or score them without position features entirely. This teaches the model to separate intrinsic relevance from positional effects. Google and YouTube rely heavily on exploration traffic and propensity logging, maintaining detailed position based click curves updated continuously from randomized experiments. The tradeoff is short term metrics versus long term quality. Exploration traffic shows suboptimal rankings to some users, costing 1 to 3 percent immediate engagement. However, without exploration, the ranker cannot discover better items or adapt to changing preferences. Systems that skip debiasing see new items and tail content systematically underranked, reducing diversity and user satisfaction over weeks. The 1 to 5 percent exploration cost pays for continuous learning and prevents model stagnation.

YouTube exploration randomizes 3 to 5 percent of video positions in recommendations, logs propensities per position, and applies IPW during training to prevent viral videos from monopolizing top slots purely due to exposure

Google Search runs continuous interleaved experiments where candidate rankings are shuffled within constraints, building propensity models that estimate position 1 receives 8x to 12x more clicks than position 5 for identical relevance

Airbnb search found that new listings with zero booking history were systematically ranked below position 20, getting almost no exposure. Adding 2 percent exploration traffic and IPW increased new listing bookings by 15 percent within two weeks