Differential Item Functioning for AI-Scored Assessments

Detects and quantifies differential item functioning (DIF) in AI-scored educational and psychological assessments. Provides a fully self-contained robust DIF engine (M-estimation via iteratively re-weighted least squares with the bi-square loss), cross-checked against the reference implementation in 'robustDIF', alongside the Differential AI Scoring Bias (DASB) test, which detects item-level scoring shifts that differ across subgroups when comparing human and AI scoring conditions. Supports independent and paired scoring designs, including a bootstrap estimator of the cross-condition covariance that paired designs require, a one-call workflow from raw item scores through 'mirt' calibration, robust linking of the cross-condition contrast, bootstrap standard errors, multiplicity control, simulation of paired human and AI item scores, anchor weight diagnostics, and an AI-effect classification framework. Methods follow Halpin (2024) .

aiDIF: Differential Item Functioning for AI-Scored Assessments

When AI systems score essays, short answers, or structured responses, a fairness question follows: does the scoring engine shift item difficulties differently for different groups?

aiDIF separates three quantities:

• DIF under human scoring — the group contrast in one condition
• DIF under AI scoring — the same contrast in the other
• Differential AI Scoring Bias (DASB) — the change in that contrast

An item can carry large DIF under both conditions and have a DASB of zero: the AI reproduced what the humans were doing. That distinction is the point of the package.

Installation

install.packages("aiDIF")

# development version
# remotes::install_github("subirhait/aiDIF")

Quick start

library(aiDIF)

eg  <- make_aidif_eg()
mod <- fit_aidif(eg$human, eg$ai, metric = "linked", adjust = "BH")

print(mod)
summary(mod)

plot(mod, type = "dif_forest")
plot(mod, type = "dasb")
plot(mod, type = "weights")
plot(mod, type = "rho")

From raw item scores (paired design)

When the same responses are scored by humans and by an AI engine, pass the two score matrices (same rows, same items) and the group vector:

fit <- fit_aidif_responses(human_scores, ai_scores, group,
                           groups = c("reference", "focal"),
                           n_boot = 200,
                           se = "bootstrap", cores = 4)
fit$scoring_bias

This calibrates both conditions in mirt (2PL or graded), estimates the cross-condition covariance by a within-group bootstrap, and runs the DASB test with the paired variance. It defaults to metric = "linked", which guards against an AI engine that changes item slopes when the groups differ in mean trait. simulate_aidif_responses() generates paired scores with known DIF and DASB for method studies.

Relation to robustDIF

The within-condition robust DIF step implements Halpin's (2024) robust scaling estimator, which is also available in robustDIF. compare_robustdif() shows the two implementations agree to machine precision. aiDIF adds the cross-condition layer: the DASB contrast, its identification under separate calibrations, paired-design inference, and the response-level workflow.

After calibration with mirt

human_in <- as_aidif(human_mod, groups = c("reference", "focal"))
ai_in    <- as_aidif(ai_mod,    groups = c("reference", "focal"))

fit <- fit_aidif(human_in, ai_in)

Identification

DASB is a difference of differences across four calibrations. If each carries its own arbitrary metric origin, the constants contribute a common offset to every item's DASB, confounded with a uniform effect.

• metric = "common" assumes linking was established upstream.
• metric = "linked" estimates the offset with the same bi-square M-estimator used for within-condition scaling, assuming most items are free of scoring-condition-by-group effects.

Do not subtract intercepts from separately normalised scales without one of these.

Scoring designs

• design = "independent" (default) sums the four marginal variances.
• design = "paired" accepts cross-condition covariance via cross_cov when the same responses were scored twice.

Because the cross-condition covariance is normally positive, the independence variance is an upper bound for paired data: the default is conservative, not anti-conservative.

Scope

aiDIF operates after IRT calibration. It works from item parameter estimates and their asymptotic covariance matrices and never sees respondent-level responses. It does not perform calibration, and it does not replace mirt, difR, or lordif for general DIF work.

Citation

citation("aiDIF")

The robust scaling method is due to Halpin, P. F. (2024). Differential item functioning via robust scaling. Psychometrika, 89(3), 796–821. doi:10.1007/s11336-024-09957-6

License

GPL (>= 3)