SystemOverflow
Learn→Privacy & Fairness in ML→Bias Detection & Mitigation→2 of 6
Privacy & Fairness in ML • Bias Detection & MitigationMedium⏱️ ~3 min

Fairness Metrics: Group, Individual, and Calibration Parity

Group Fairness Metrics

Group fairness asks: do different demographic groups receive similar treatment in aggregate? Demographic parity: Positive prediction rates should be equal across groups. If 40% of Group A gets approved, 40% of Group B should too. Equalized odds: True positive rate and false positive rate should be equal across groups. If 90% of qualified Group A members get approved, 90% of qualified Group B members should too. Equal opportunity: A relaxed version requiring only equal true positive rates. These metrics treat groups as monolithic, ignoring individual variation.

Individual Fairness Metrics

Individual fairness asks: are similar individuals treated similarly? Two applicants with identical qualifications should receive identical predictions regardless of group membership. The challenge: defining similarity. What features determine similarity? If zip code is included, and zip codes correlate with race, you embed bias in your similarity definition. Counterfactual fairness: would the prediction change if only the protected attribute changed? For the same person, changing their gender should not change the prediction. Implementation requires causal models to identify which features depend on protected attributes.

Calibration Parity

Calibration asks: when the model predicts 80% probability, does the event occur 80% of the time for all groups? A well-calibrated model saying "80% chance of loan default" should see 80% default rate for both men and women. Miscalibration is common: models often overpredict risk for minority groups (predicting 80% default when actual rate is 50%). Recalibration fits separate probability mappings per group, adjusting predictions post-hoc. Check calibration with reliability diagrams plotting predicted versus actual probabilities per group.

⚠️ Key Trade-off: These metrics are mathematically incompatible when base rates differ. If Group A has 60% qualification rate and Group B has 40%, you cannot achieve demographic parity and equalized odds simultaneously. Choose based on your fairness philosophy.
💡 Key Takeaways
✓Group fairness: demographic parity (equal rates), equalized odds (equal TPR/FPR), equal opportunity (equal TPR only)
✓Individual fairness: similar individuals should receive similar predictions regardless of group
✓Counterfactual fairness: changing only protected attribute should not change prediction
✓Calibration parity: 80% predicted probability should mean 80% actual rate for all groups
✓These metrics are mathematically incompatible when base rates differ between groups
📌 Interview Tips
1Explain three types of group fairness with concrete approval rate examples
2Mention counterfactual fairness requires causal models to identify dependent features
← Back to Bias Detection & Mitigation Overview
Fairness Metrics: Group, Individual, and Calibration Parity | Bias Detection & Mitigation - System Overflow