Statistical Inference for Language Model Evaluations

Treats language model evaluations as statistical experiments and supplies the inference they require. Provides central limit theorem and cluster-robust standard errors for evaluation scores, paired and unpaired model comparisons, variance decomposition when several responses are drawn per question, control-variate variance reduction, multiplicity adjustment across benchmark suites, and power and minimum detectable effect calculations for planning evaluations, following Miller (2024) . For evaluations scored by a model judge, implements agreement statistics against a human gold standard and prediction-powered inference (Angelopoulos et al. 2023) with the power-tuned estimator of Angelopoulos, Bates and Jordan (2023) , so a small set of human labels debiases a large set of judge scores. Leaderboards are supported through bootstrap rank intervals and Bradley-Terry ratings (Bradley and Terry 1952) . Accepts scores from any evaluation harness.

evaluatellm

Language model evaluations are experiments, and experiments have standard errors. Almost no published evaluation reports one. evaluatellm supplies the inference: standard errors that respect how questions were sampled, model comparisons that use the pairing you already have, power calculations that tell you whether an evaluation can answer its question before you run it, and prediction-powered estimators that let a small set of human labels correct a large set of model-judge scores.

It is a pure computation package. It runs no evaluations and calls no APIs: give it scores from any harness and it gives you the statistics.

Installation

install.packages("evaluatellm")

Development version:

# install.packages("pak")
pak::pak("charlescoverdale/evaluatellm")

The problem in one number

Reading comprehension evaluations ask several questions about each passage. Those questions are not independent draws, so the usual standard error is wrong. Across 1,500 simulated evaluations of 30 passages with 10 questions each, nominal 95 per cent intervals actually covered the true value:

One evaluation in four was reporting an interval that did not contain the answer.

Usage

library(evaluatellm)

e <- as_eval(results, score = correct, item = q, model = model, cluster = passage)

ev_score(e, "new")
#> Evaluation score: new
#>
#>   Estimate   0.6950
#>   Std. error 0.0401
#>   95% CI     [0.6144, 0.7756]
#>
#>   Items      400
#>   Clusters   50
#>   Design eff 3.03 (SE is 1.74x the independent estimate)
#>   Cluster-robust standard error, t on 49 df.

Comparing two models uses the pairing automatically:

ev_paired(e, model_a = "new", model_b = "old")
#> Paired comparison: new vs old
#>
#>   new  0.6950
#>   old  0.6350
#>
#>   Difference   0.0600  [-0.0035, 0.1235] (p = 0.063)
#>   Std. error   0.0316  (cluster-robust, 50 clusters)
#>   t = 1.899 on 49 df

Run the same comparison without the cluster column and it reports 0.0600 [0.0000, 0.1200], p = 0.050. The naive analysis clears the bar and the honest one does not. Note also that clustering inflates the standard error on the level by 1.74 times but the difference by much less, because pairing has already removed the passage difficulty both models faced.

Before you run an evaluation

ev_mde(n_items = 500, p_a = 0.72, p_b = 0.70, correlation = 0.7,
       icc = 0.25, cluster_size = 8)
#>   MDE            0.0552

Five hundred clustered questions cannot detect a two point gain. Reporting a null result from that evaluation says nothing about the models. Report the minimum detectable effect alongside it, or run ev_power() first and buy enough questions.

Model judges

A judge that is biased stays biased however many items it grades. Measure it, then correct it:

ev_judge_agreement(judge, human)     # accuracy, Cohen's kappa, McNemar bias test
ev_judge_debias(judge, gold)         # prediction-powered estimate
ev_judge_power(n_total = 20000, correlation = 0.8, target_se = 0.01)

ev_judge_debias() takes judge scores for every item and human labels for a random subset, marked NA elsewhere. It returns an estimate of what a full human evaluation would have found, with valid intervals, and reports how many human labels the result is worth:

#>   Your 250 human labels carry the precision of 400.

The estimator is tuned so it is never less precise than using the human labels alone, so there is no downside to including a weak judge.

Function reference

Validation

Every analytic result is checked against an independent implementation or a simulation in the test suite:

• ev_score(), ev_paired() and ev_unpaired() reproduce stats::t.test() to machine precision, including Welch degrees of freedom.
• ev_cluster() reproduces sandwich::vcovCL(type = "HC0", cadjust = TRUE) exactly.
• Cohen's kappa uses the Fleiss, Cohen and Everitt (1969) asymptotic variance, checked against a bootstrap.
• ev_resample() recovers a known between and within variance decomposition.
• ev_judge_debias() is checked over 300 replications for unbiasedness, standard error accuracy and interval coverage.
• ev_power() agrees with stats::power.t.test() to within the difference between normal and t quantiles.

Limitations

Intervals are the normal approximation. Following Miller (2024), the standard errors and confidence intervals are the CLT-based Wald form, computed on the t distribution. That is the right default for a benchmark of a few hundred items, and it has two known failure points at the edges.

The first is a slice where every item passes or every item fails. The standard error is then exactly zero and the interval collapses to a point, which looks like certainty but is not: 50 out of 50 correct gives a Wilson interval of about [0.93, 1.00], not [1.00, 1.00]. The second is a handful of clusters, where the t multiplier is large enough to push the interval outside [0, 1], which is not a range a pass rate can occupy. Both cases now warn. Treat the interval as uninformative rather than clipping it, and if you need a bound at the boundary, use an exact method such as Wilson or Clopper-Pearson.

Cluster-robust inference needs enough clusters. The asymptotics are in the number of clusters, not the number of items. Below roughly 30 clusters the standard errors are biased downward, and ev_cluster() reports the cluster count so you can judge that. Very few clusters trigger the interval warning above.

Prediction-powered inference assumes the labelled subset is random. If the human-labelled items were chosen because they looked hard, or interesting, or were the first 200 in the file, ev_judge_debias() will produce a confident answer to the wrong question. The guarantee comes from the sampling design, which the package cannot check.

Judge agreement is not judge accuracy. High kappa between a model judge and a human means they agree, including where they are both wrong in the same direction. It is a necessary condition for trusting a judge, not a sufficient one.

The package does not run evaluations. It takes scores that already exist. Anything about prompting, sampling temperature, or harness configuration is out of scope and upstream of everything here.

Working with other tools

evaluatellm consumes scores, so it sits downstream of whatever produced them. The vitals package runs evaluations in R and its logs pass straight into as_eval(); scores exported from Inspect, lm-eval-harness or a bespoke pipeline work the same way.

References

Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. https://doi.org/10.48550/arXiv.2411.00640

Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., and Zrnic, T. (2023). Prediction-powered inference. Science. https://doi.org/10.1126/science.adi6000

Angelopoulos, A. N., Bates, S., and Jordan, M. I. (2023). PPI++: Efficient Prediction-Powered Inference. https://doi.org/10.48550/arXiv.2311.01453

Bradley, R. A. and Terry, M. E. (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika. https://doi.org/10.2307/2334029

Fleiss, J. L., Cohen, J., and Everitt, B. S. (1969). Large sample standard errors of kappa and weighted kappa. Psychological Bulletin.

Licence

MIT