Evaluation Metrics for Spaced Repetition Schedulers

Calibration and discrimination metrics for spaced-repetition memory models. Provides the sample-weighted binned root mean squared error (RMSE(bins)) used to rank schedulers in the open spaced repetition benchmark, together with log loss, the area under the ROC curve, and calibration curves.

srsbench

Evaluation metrics for spaced-repetition schedulers in R: the sample-weighted binned RMSE (RMSE(bins)) used to rank algorithms in the open spaced repetition benchmark, together with log loss, the area under the ROC curve, and calibration curves. The metrics take plain vectors of predicted recall probabilities and observed outcomes, so they can benchmark any scheduler's predictions.

Install

# install.packages("remotes")
remotes::install_github("chrislongros/srsbench")

Use

Each metric takes one row per review as parallel vectors: the predicted recall probability p, the observed outcome y (1 recalled, 0 forgotten), and -- for rmse_bins() -- the interval in days, the review number, and the lapse count.

library(srsbench)

p            <- c(0.95, 0.86, 0.72, 0.60, 0.91)
y            <- c(1,    1,    0,    1,    1)
elapsed_days <- c(1,    6,    20,   4,    15)
i            <- c(2,    3,    4,    2,    3)
lapse        <- c(0,    0,    1,    0,    0)

rmse_bins(p, y, elapsed_days, i, lapse)   # calibration, lower is better
log_loss(p, y)                            # cross-entropy, lower is better
srs_auc(p, y)                             # discrimination, higher is better
calibration_bins(p, y)                    # predicted vs observed by bin

To benchmark two schedulers, evaluate each one's predictions on the same reviews and read off the numbers -- lower rmse_bins/log_loss, higher srs_auc wins.

Getting the predictions

srsbench benchmarks predictions; it does not generate them or read Anki files. Any algorithm that predicts a recall probability per review can be benchmarked directly: FSRS (via the rfsrs package), Ebisu, half-life regression, DASH, ACT-R style activation models, neural models, or a plain logistic regression. Interval schedulers such as SM-2, Leitner or Anki's default do not provide probabilities.

Produce p by running the algorithm over a review log with a time-based train/test split so the predictions are out-of-sample. Read Anki collections with ankiR. srsbench then evaluates the output (p, y).

Metrics

• rmse_bins() -- sample-weighted binned RMSE. Reviews are grouped along three log-quantised axes (interval, review number, lapse count) and the mean predicted recall is compared with the observed recall rate in each bin.
• log_loss() -- binary cross-entropy.
• srs_auc() -- rank-based area under the ROC curve.
• calibration_bins() -- predicted probability versus observed recall in equal-width bins, for calibration plots.

Attribution

RMSE(bins) is the metric defined by the open spaced repetition benchmark; rmse_bins() is an independent R implementation of its rmse_matrix definition. The log loss, ROC AUC and calibration metrics are standard.

License

MIT.