aiEvalR
Statistical and Psychometric Evaluation of AI Systems
Evaluates artificial intelligence (AI) systems as measurement instruments using psychometric methods. Provides multi-facet generalizability theory (G-study and D-study) via 'lme4', reliability via the intraclass correlation coefficient (ICC), calibration via the expected calibration error (ECE) and Brier score, robustness stress testing, and group disparity diagnostics. Item-level differential item functioning (DIF) based on item response theory (IRT) is delegated to the 'aiDIF' package. Methods follow Cronbach, Gleser, Nanda and Rajaratnam (1972,
aiEvalR
Statistical and Psychometric Evaluation of AI Systems
aiEvalR evaluates AI systems the way psychometricians evaluate tests —
as measurement instruments with reliability, fairness, calibration, and
error structure — rather than as black boxes scored on accuracy alone.
Modules
| Module | What it does |
|---|---|
| Psychometric quality | The methodological core. Multi-facet generalizability theory (case × prompt × model × run) via lme4, with a G-study/D-study workflow. Prompts become "items," AI outputs become "item responses." |
| Reliability | Test-retest (ICC), internal consistency (alpha), bootstrap intervals. |
| Robustness | Stress testing, prompt sensitivity, adversarial stability. |
| Fairness | Group disparity + equalized odds; integrates with (does not duplicate) the aiDIF package for item-level IRT-based AI-scoring bias. |
| Calibration | ECE, Brier score, calibration curves, input-validated. |
| Hallucination | Aggregation of adjudicated verdicts; lexical-overlap baseline. |
| Dashboard | Descriptive multi-module profile (not a validated composite score). |
Quick examples
Generalizability (the core)
library(aiEvalR)
# long-format: one row per case x prompt x model x run
g <- ai_generalizability(data, score = "score", case = "case",
facets = c("prompt", "model", "run"))
print(g) # G-study variance components
# D-study: project dependability for a specific measurement procedure
ai_dstudy(g, n_prompt = 3, n_model = 2, n_run = 1)
Fairness (Module 2) — integrates with aiDIF
# outcome-level group disparity (this package)
ai_group_disparity(outcome, group, predicted_class = decisions)
# item-level IRT-based AI-scoring bias -> dispatches to aiDIF if installed
ai_fairness(outcome, group, human_mle = h, ai_mle = a)
aiEvalR's fairness functions and the causalfragility-lab/aiDIF
package answer different questions: aiEvalR checks whether overall
outcomes/decisions differ by group; aiDIF checks whether an AI scoring
engine shifts specific item difficulties differently by group (via
paired human/AI IRT calibration and a Differential AI Scoring Bias
test). Use both, not one instead of the other. aiDIF is on CRAN
(install.packages("aiDIF")).
Naming notes
Some functions carry deliberately precise names to avoid overclaiming what they measure:
| Function | Rationale |
|---|---|
lexical_overlap() |
Measures token overlap (Jaccard); named to make clear it does NOT detect negation/contradiction or factual consistency. |
empirical_output_interval() |
Describes the observed spread of repeated outputs; named to avoid implying calibrated prediction-interval coverage guarantees. |
ai_group_disparity() |
A between-group difference is a disparity statistic, not automatically "bias" without further context. |
ai_dif_mh() |
A basic Mantel-Haenszel screening helper; named to distinguish it from the fuller IRT-based aiDIF package. |
Installation
devtools::install_github("causalfragility-lab/aiEvalR")
Citation
Hait, S. (2026). aiEvalR: Statistical and Psychometric Evaluation of AI Systems. R package version 0.1.0.
Generalizability-theory foundation:
Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The Dependability of Behavioral Measurements. Wiley.
Brennan, R. L. (2001). Generalizability Theory. Springer.