Score Comparability for Translated and Adapted Exams

An adaptation-comparability workflow for small, lower-scoring and unbalanced language groups where standard differential item functioning (DIF) tools (Magis, Beland, Tuerlinckx and De Boeck, 2010, ) break down. Calibrates the Rasch model in each language group, links the groups robustly through the densest cluster of items rather than assuming DIF cancels out (on anchor selection see Kopf, Zeileis and Strobl, 2015, ), detects small-sample DIF with an empirical-Bayes spike-and-slab model and local false discovery rates (Efron, 2004, ), quantifies whether item-level DIF accumulates into different pass rates, explains DIF by item features to give translators actionable guidance, and drafts a comparability report.

transDIF

Does a score mean the same thing in both languages?

A comparability workflow for translated and adapted exams, built for the conditions where standard DIF tools break down: a small translated-language group (50–200), a group that differs in ability, and DIF that runs mostly in one direction, so anchors themselves are biased.

library(transDIF)

sim <- td_simulate(n_focal = 150, seed = 1)          # or your 0/1 matrix + group
cal <- td_calibrate(sim$responses, sim$group)        # Rasch MML per language group
dif <- td_dif(cal)                                   # robust linking + small-sample DIF
imp <- td_impact(dif, cut = 36)                      # does DIF change pass rates?
fea <- td_features(dif, sim$features)                # which features predict DIF?
cat(td_report(dif, imp, fea, c("English", "French")))

Installation

From CRAN:

install.packages("transDIF")

Development version from GitHub:

install.packages("pak")
pak::pak("edidatasolutions/transDIF")

Method

• Calibration. Rasch MML in each group, with SEs from the exact marginal Hessian. DIF uses relative SEs: the scale-location uncertainty shared by all items belongs to the linking shift, and is added to its SE there.
• Robust linking. Item differences are modeled as d_i ~ N(c + delta_i, se_i^2). The shift c (the ability difference) is the precision-weighted mode of the d_i: the center of the densest cluster of items. It assumes that DIF-free items form the largest cluster, not that DIF cancels out. Its SE is a sandwich estimate plus the scales' location uncertainty.
• Ambiguity warning. If a second cluster of items is nearly as dense, the data cannot say which cluster is DIF-free. td_dif then warns, and the report shows both candidate shifts.
• Small-sample DIF. Given c, a spike-and-slab mixture (a DIF-free majority with negligible tau0, plus a directional DIF slab) gives each item a posterior DIF probability, a shrunken DIF estimate and a local-FDR flag.
• Aggregate impact. The group's pass-rate change and the raw-score shift at the cut, from exact score distributions. Intervals propagate both linking and DIF uncertainty.
• Feature explanation. Random-effects meta-regression of DIF on item features, giving concrete guidance for translators.
• Baselines for comparison: mean linking, iterative purification, Mantel–Haenszel with purification and BH.

Validation (known truth, 100 replications per size)

60 items; reference n = 2,000; translated group 0.5 logits lower; ~35% of items carry adaptation features that make them mostly harder in translation.

DIF detection (items with DIF > 0.2 logits, target FDR 10%):

EB has the most power at every size and keeps its false discovery proportion below the 10% target. With 50 translated-language candidates, no method finds more than a few percent of DIF items, so treat flags there as candidates for expert review.

DIF effect estimation (RMSE, logits), EB shrinkage vs raw differences on DIF-free items: 0.105 vs 0.363 (n = 50), 0.092 vs 0.260 (n = 100), 0.083 vs 0.188 (n = 200). Genuinely large DIF is shrunk somewhat toward the slab mean.

Linking (true shift 0.5):

Mean linking is biased by directional DIF, and more data does not fix that. The weighted mode is essentially unbiased and has the lowest RMSE at every size. In a broader benchmark (180 data sets: balanced, directional and heavy DIF at n = 50/100/200; dev/bench_link.R) it had the lowest overall RMSE of nine estimators (0.114, vs 0.119 for purification, 0.137 for mean linking and 0.164 for the joint mixture used in earlier development). Its 90% intervals cover the true shift 89–90% of the time.

Impact: 90% intervals for the pass-rate change cover the truth in 94% of replications at n = 150 (true mean change -3.0 points, estimated -2.2).

Features: idiom, cultural referent, units and vocabulary effects are all recovered (true 0.60 / 0.50 / -0.40 / 0.35; mean estimates 0.60 / 0.48 / -0.38 / 0.35), with 95% CI coverage of 92–97%.

When the linking is not identified. Linking assumes DIF-free items form the largest cluster. On short tests with pervasive DIF this can fail: in real cross-national data (14 FIMS mathematics items, Australia vs Japan), the linkings disagreed by 0.3 logits and the estimated impact of DIF on the pass rate ranged from -8.0 points to -0.6 (interval including zero). td_sensitivity() reports this dependence; see the vignette.

Status

Done: td_simulate, td_calibrate, td_dif, td_mh, td_impact, td_features, td_report, td_sensitivity (since 0.2.0). Next: uniform vs non-uniform DIF (2PL), polytomous items, and validation on real French–English data.

Getting help and contributing

Questions and bug reports: https://github.com/edidatasolutions/transDIF/issues. See CONTRIBUTING.md for how to report problems, get help, or contribute code.