Label Delay and Two Stage Learning

THE LABEL DELAY PROBLEM
Most ML systems face a gap between prediction and ground truth. A fraud model predicts at transaction time, but fraud confirmation arrives 30-90 days later after investigation. A recommendation model predicts clicks, but conversions happen days or weeks later.
This delay creates a blind spot. You cannot measure true model performance until labels arrive. By then, the model may have been wrong for weeks. In fast-changing domains, the model that was wrong for 30 days has already caused significant damage.
Delay varies by domain: click labels arrive in seconds, conversion labels in days, fraud labels in weeks, churn labels in months. Design monitoring strategies around your specific label latency.
TWO-STAGE LEARNING
Stage 1 - Proxy model: Train on fast-arriving labels (clicks, views, engagement). Update frequently. Provides responsiveness to drift.
Stage 2 - Outcome model: Train on delayed labels (conversions, fraud confirmation). Update less frequently. Provides accuracy on true outcomes.
Combine stages: proxy model provides immediate signal; outcome model corrects for proxy-outcome mismatch. Weight toward proxy model for recent predictions (labels not yet available), shift toward outcome model as labels arrive.
HANDLING PARTIAL LABELS
Label imputation: For unlabeled recent data, use model predictions as pseudo-labels. Risky: if the model is wrong, you reinforce errors. Use only when confident in model stability.
Semi-supervised learning: Train on both labeled (old) and unlabeled (recent) data. Consistency regularization encourages similar predictions for similar inputs regardless of label availability.
Active labeling: Prioritize labeling for samples where the model is uncertain or drift is detected. Targeted labeling provides maximum information per label.
⚠️ Key Trade-off: Two-stage learning adds complexity. Simple domains may not need it. Use when label delay exceeds drift rate—if concepts change faster than labels arrive, proxy models are essential.

💡 Key Takeaways

✓Label delay varies: clicks (seconds), conversions (days), fraud (weeks), churn (months)—design monitoring around your latency

✓Two-stage learning: proxy model on fast labels for responsiveness, outcome model on delayed labels for accuracy

✓Partial label strategies: imputation (risky), semi-supervised learning, active labeling for uncertain samples

📌 Interview Tips

1Interview Tip: Explain the label delay problem with concrete timelines for your domain.

2Interview Tip: Describe two-stage learning architecture—when to weight proxy vs outcome model.

← Back to Concept Drift & Model Decay Overview