Algorithmic Fairness Assessment for Clinical Prediction Models

Post-hoc fairness auditing toolkit for clinical prediction models. Unlike in-processing approaches that modify model training, this package evaluates existing models by computing group-wise fairness metrics (demographic parity, equalized odds, predictive parity, calibration disparity), visualizing disparities across protected attributes, and performing threshold-based mitigation. Supports intersectional analysis across multiple attributes and generates audit reports useful for fairness-oriented auditing in clinical AI settings. Methods described in Obermeyer et al. (2019) and Hardt, Price, and Srebro (2016) .

Overview

clinicalfair is a post-hoc fairness auditing toolkit for clinical prediction models. It evaluates existing models by computing group-wise fairness metrics, visualizing disparities, and performing threshold-based mitigation -- motivated by regulatory expectations for transparency in clinical AI.

• Metrics: demographic parity, equalized odds, predictive parity, AUC, Brier score -- with optional bootstrap confidence intervals
• Visualization: disparity plots, ROC by group, calibration by group
• Mitigation: group-specific threshold optimization with configurable grid resolution (default 0.01)
• Intersectional: cross-tabulated analysis (race x sex x age) with automatic small-group filtering
• Reporting: four-fifths rule violation detection

Figure 1 | Fairness audit with threshold mitigation. (a) Group-wise selection rate, TPR, and FPR before and after equalized-odds threshold optimization. Mitigation reduces cross-group TPR disparity while maintaining acceptable accuracy. (b) ROC curves by racial group showing differential model performance. Data: COMPAS-style simulated recidivism predictions. Methods: Hardt et al. (2016); Obermeyer et al. (2019).

Why clinicalfair?

Existing R packages approach fairness from different angles:

clinicalfair is designed for the clinical audit use case: a clinician or regulator receives a trained model and needs to answer "is this model fair across patient subgroups?" in one function call, with actionable output.

# One-line audit
fairness_report(fairness_data(predictions, labels, race))

Installation

# From GitHub:
devtools::install_github("CuiweiG/clinicalfair")

# After CRAN acceptance:
install.packages("clinicalfair")

Quick start

library(clinicalfair)
data(compas_sim)

# Create fairness evaluation object
fd <- fairness_data(
    predictions = compas_sim$risk_score,
    labels = compas_sim$recidivism,
    protected_attr = compas_sim$race
)

# Compute metrics
fairness_metrics(fd)

# With bootstrap confidence intervals
fairness_metrics(fd, ci = TRUE, n_boot = 2000)

# Generate audit report
fairness_report(fd)

# Mitigate via threshold optimization
threshold_optimize(fd, objective = "equalized_odds")

Figure 2 | Calibration curves by racial group. Each point represents a decile bin; point size proportional to sample count. Deviation from the diagonal indicates miscalibration. Differential calibration across groups is a key fairness concern identified by Obermeyer et al. (2019) Science 366:447.

Functions

Key references

• Obermeyer Z et al. (2019). Dissecting racial bias in an algorithm. Science 366:447. doi:10.1126/science.aax2342
• Hardt M et al. (2016). Equality of Opportunity in Supervised Learning. NeurIPS.
• FDA (2021). Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) Action Plan.

License

MIT