Metric Ladders and Mediation Chains

METRIC LADDERS EXPLAINED
A metric ladder is a hierarchy connecting technical metrics to business outcomes. At the bottom: model metrics (accuracy, AUC, latency). Middle: product metrics (CTR, engagement time, conversion rate). Top: business metrics (revenue, profit, customer lifetime value).
Each rung in the ladder has a transfer function describing how improvements at one level affect the next. A 10% improvement in model AUC might yield 5% improvement in CTR, which yields 2% improvement in revenue. These transfer functions are estimated from historical data and validated through experiments.
MEDIATION CHAINS
Mediation analysis decomposes the total effect of a model change into direct and indirect paths. Direct effect: the model change affects business metrics directly. Indirect effect: the model change affects intermediate metrics, which then affect business metrics.
Example: A faster model (reduced latency) improves engagement (users see more recommendations per session), which improves purchases. The direct effect of latency on purchases is small. The indirect effect through engagement is large. Understanding mediation helps you identify which intermediate metrics matter most.
BUILDING THE LADDER
Step 1: Identify all metrics from model to business. Map dependencies explicitly. Which model metrics affect which product metrics?
Step 2: Estimate transfer functions. Use historical data to compute correlations. A/B tests provide causal estimates when available.
Step 3: Validate the chain. When you improve a model metric, does the expected improvement propagate? If not, your transfer functions are wrong or confounders exist.
SENSITIVITY ANALYSIS
Not all model metrics have equal business impact. Sensitivity analysis identifies which model improvements yield the largest business gains. If improving latency from 50ms to 40ms yields 10x more revenue impact than improving AUC from 0.85 to 0.87, focus on latency.
⚠️ Key Trade-off: Metric ladders are approximations. They work well for incremental changes but break down for large changes where non-linearities dominate.

💡 Key Takeaways

✓Metric ladder: model metrics (AUC) → product metrics (CTR) → business metrics (revenue), each with transfer functions

✓Mediation analysis separates direct effects from indirect effects through intermediate metrics

✓Sensitivity analysis identifies which model improvements yield largest business gains—focus there first

📌 Interview Tips

1Interview Tip: Draw a metric ladder showing how model improvements propagate to business outcomes.

2Interview Tip: Explain mediation—latency improvement affects engagement, which affects purchases (indirect > direct).

← Back to Business Metrics Correlation Overview