Comparison of Algorithms with Iterative Sample Size Estimation

Functions for performing experimental comparisons of algorithms using adequate sample sizes for power and accuracy. Implements the methodology originally presented in Campelo and Takahashi (2019) for the comparison of two algorithms, and later generalised in Campelo and Wanner (Submitted, 2019) .

Felipe Campelo ([email protected]) and Fernanda Takahashi ([email protected])
Operations Research and Complex Systems Laboratory - ORCS Lab
Universidade Federal de Minas Gerais
Belo Horizonte, Brazil

Implementation of R package CAISEr, with routines for automatically determining the sample size needed for performing comparative experiments with algorithms.

To install the most up-to-date version directly from Github, simply type:


The most recent CRAN release of the package is also available for installation directly from the R prompt, using:


For instructions and examples of use, please take a look at the vignette Adapting Algorithms for CAISEr, and at the package documentation, particularly functions run_experiment() and run_nreps2().

Please send any bug reports, questions, suggestions, chocolate (to Fernanda) or beers (to Felipe - we can always hope!) directly to the package authors listed at the top of this document.



CAISEr 0.3.3

  • fixed problem with printing version in the vignette.

CAISEr 0.3.2

  • fixed rare bug in calc_se() that resulted in NaN if two vectors with the same sample mean and same sample variance were passed as arguments.

CAISEr 0.3.1

  • Added function to consolidate partial results saved to file (consolidate.partial.results())
  • Minor improvements to saving partial results to file: users can now select arbitrary directory for saving

CAISEr 0.3

  • run_experiment() can now be run in parallel using multiple cores.
  • run_experiment() and calc_nreps2() can now save results to files.

CAISEr 0.2.4

  • run_experiment() now forces the use of all available instances if power >= 1.

CAISEr 0.2.3

  • Improved plot and summary functions for CAISErPowercurve objects.
  • Added options to calc_power_curve() to determine the range of effect sizes to consider.

CAISEr 0.2.2

  • Added new example and use case to calc_nreps2()

CAISEr 0.2.1

  • Minor fixes, particularly in printing function.

CAISEr 0.2.0

  • Initial release on CRAN.

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.


1.0.16 by Felipe Campelo, 2 years ago

Report a bug at

Browse source code at

Authors: Felipe Campelo [aut, cre] , Fernanda Takahashi [ctb] , Elizabeth Wanner [ctb]

Documentation:   PDF Manual  

GPL-2 license

Imports assertthat, parallel, pbmcapply, ggplot2, gridExtra

Suggests MOEADr, smoof, knitr, rmarkdown, car, dplyr, pkgdown

See at CRAN