Efficient Sequential Testing with Evidence Ratios

An implementation of sequential testing that uses evidence ratios computed from the weights of a set of models. These weights correspond either to Akaike weights computed from the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC) and following Burnham & Anderson (2004, ) recommendations, or to pseudo-BMA weights computed from the WAIC or the LOO-IC of models fitted with 'brms' and following Yao et al. (2017, ).

CRAN status Build Status

The ESTER package implements sequential testing based on evidence ratios computed from the Akaike weights of a set of models. These weights are being computed using either the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC).


To install the latest version from CRAN you can use


Or the development version can be installed with

if (!require("devtools") ) install.packages("devtools")
devtools::install_github("lnalborczyk/ESTER", dependencies = TRUE)

Different questions

  1. Simulation. Given an expected effect size and a given sample size, what evolution of evidence ratios should I reasonnably expect ?

  2. Observed data. When to stop recruiting participants ?


The simER function runs a simulated study in which we compare two independant groups, for various effect size (cohensd) and sample size (nmax). The nmin argument serves to specify from which participant we want to start doing sequential testing (we usually recommend to avoid nmin < 10). We can define a boundary at which we would like to stop the sequential testing, as well as how many simulations we want to evaluate (nsims).

simER(cohensd = 0.8, nmin = 20, nmax = 100, boundary = 10, nsims = 100, ic = bic)

Observed data

On the other hand (and perhaps more interestingly), ESTER can be used to do sequential testing on your own data. You can study the evolution of sequential ERs using the seqER function.

mod1 <- lm(mpg ~ cyl, mtcars)
mod2 <- lm(mpg ~ cyl + disp, mtcars)
seqER(ic = bic, mod1, mod2, nmin = 10)

In addition, seqER allows you to study the behavior of sequential ERs computed on your own data, along with sequential ERs computed on permutation samples. This feature might be useful to study to what extent the evolution of evidence ratios you observed on the original sample is dependent to the order of the observations.

seqER(ic = bic, mod1, mod2, nmin = 10, nsims = 10)

More detailed information can be found in the main vignette, available online here, or by typing vignette("ESTER") in the console.


ESTER version 0.2.0

Major updates

  • deprecating distER.R (will be removed in the next release)
  • merging seqER.R and seqERboot.R into seqER.R, by adding a "nsims" argument in seqER.R
  • deprecating seqERboot.R (will be removed in the next release)
  • simER.R: parallelisation and update of the plot method
  • introducing the analysER.R function which allow to analyse the results of simulations ran with simER.R

Minor updates

  • doc updates
  • adding pseudo-BMA weights computations for brmsfit models in ictab.R
  • seqER.R now accepts brmsfit models
  • adding a "blind" argument to seqER.R to conduct triple blind analyses, with a "boundary" argument at which the sequential testing is stopped

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.


0.2.0 by Ladislas Nalborczyk, a year ago


Report a bug at https://github.com/lnalborczyk/ESTER/issues

Browse source code at https://github.com/cran/ESTER

Authors: Ladislas Nalborczyk [aut, cre]

Documentation:   PDF Manual  

MIT + file LICENSE license

Imports brms, lme4, dplyr, magrittr, tidyr, ggplot2, rlang, foreach, doParallel, cowplot

Suggests knitr, rmarkdown

See at CRAN