pROC
Display and Analyze ROC Curves
Tools for visualizing, smoothing and comparing receiver operating characteristic (ROC curves). (Partial) area under the curve (AUC) can be compared with statistical tests based on U-statistics or bootstrap. Confidence intervals can be computed for (p)AUC or ROC curves.
An R package to display and analyze ROC curves.
For more information, see:
- Xavier Robin, Natacha Turck, Alexandre Hainard, et al. (2011) “pROC: an open-source package for R and S+ to analyze and compare ROC curves”. BMC Bioinformatics, 7, 77. DOI: 10.1186/1471-2105-12-77
- The official web page on ExPaSy
- The CRAN page
- My blog
- The FAQ
Stable
The latest stable version is best installed from the CRAN:
install.packages("pROC")
Help
Once the library is loaded with library(pROC), you can get help on pROC by typing ?pROC.
Getting started
If you don't want to read the manual first, try the following:
library(pROC)data(aSAH)Basic ROC / AUC analysis
roc(aSAH$outcome, aSAH$s100b)roc(outcome ~ s100b, aSAH)Smoothing
roc(outcome ~ s100b, aSAH, smooth=TRUE) more options, CI and plotting
roc1 <- roc(aSAH$outcome, aSAH$s100b, percent=TRUE, # arguments for auc partial.auc=c(100, 90), partial.auc.correct=TRUE, partial.auc.focus="sens", # arguments for ci ci=TRUE, boot.n=100, ci.alpha=0.9, stratified=FALSE, # arguments for plot plot=TRUE, auc.polygon=TRUE, max.auc.polygon=TRUE, grid=TRUE, print.auc=TRUE, show.thres=TRUE) # Add to an existing plot. Beware of 'percent' specification! roc2 <- roc(aSAH$outcome, aSAH$wfns, plot=TRUE, add=TRUE, percent=roc1$percent) Coordinates of the curve
coords(roc1, "best", ret=c("threshold", "specificity", "1-npv"))coords(roc2, "local maximas", ret=c("threshold", "sens", "spec", "ppv", "npv"))Confidence intervals
# Of the AUCci(roc2) # Of the curvesens.ci <- ci.se(roc1, specificities=seq(0, 100, 5))plot(sens.ci, type="shape", col="lightblue")plot(sens.ci, type="bars") # need to re-add roc2 over the shapeplot(roc2, add=TRUE) # CI of thresholdsplot(ci.thresholds(roc2))Comparisons
# Test on the whole AUC roc.test(roc1, roc2, reuse.auc=FALSE) # Test on a portion of the whole AUC roc.test(roc1, roc2, reuse.auc=FALSE, partial.auc=c(100, 90), partial.auc.focus="se", partial.auc.correct=TRUE) # With modified bootstrap parameters roc.test(roc1, roc2, reuse.auc=FALSE, partial.auc=c(100, 90), partial.auc.correct=TRUE, boot.n=1000, boot.stratified=FALSE)Sample size
# Two ROC curves power.roc.test(roc1, roc2, reuse.auc=FALSE) power.roc.test(roc1, roc2, power=0.9, reuse.auc=FALSE) # One ROC curve power.roc.test(auc=0.8, ncases=41, ncontrols=72) power.roc.test(auc=0.8, power=0.9) power.roc.test(auc=0.8, ncases=41, ncontrols=72, sig.level=0.01) power.roc.test(ncases=41, ncontrols=72, power=0.9)Development
Installing the development version
Download the source code from git, unzip it if necessary, and then type R CMD INSTALL pROC. Alternatively, you can use the devtools package by Hadley Wickham to automate the process (make sure you follow the full instructions to get started):
if (! requireNamespace("devtools")) install.packages("devtools")devtools::install_github("xrobin/pROC")Check
To run all automated tests, including slow tests:
cd .. # Run from parent directory
VERSION=$(grep Version pROC/DESCRIPTION | sed "s/.\+ //")
R CMD build pROC
RUN_SLOW_TESTS=true R CMD check pROC_$VERSION.tar.gz
vdiffr
The vdiffr package is used for visual tests of plots.
To run all the test cases (incl. slow ones) from the command line:
run_slow_tests <- TRUEvdiffr::manage_cases()To run the checks upon R CMD check, set environment variable NOT_CRAN=1:
NOT_CRAN=1 RUN_SLOW_TESTS=true R CMD check pROC_$VERSION.tar.gz
Release steps
- Get new version to release:
VERSION=$(grep Version pROC/DESCRIPTION | sed "s/.\+ //") && echo $VERSION - Build & check package:
R CMD build pROC && R CMD check --as-cran pROC_$VERSION.tar.gz - Check with slow tests:
NOT_CRAN=1 RUN_SLOW_TESTS=true R CMD check pROC_$VERSION.tar.gz - Check with R-devel:
rhub::check_with_rdevel() - Chec reverse dependencies:
devtools::revdep_check(libpath = rappdirs::user_cache_dir("revdep_lib"), srcpath = rappdirs::user_cache_dir("revdep_src")) - Update
VersionandDateinDESCRIPTION - Update version and date in
NEWS - Create a tag:
git tag v$VERSION - Submit to CRAN
News
1.14.0 (2019-03-12)
- The 'multiclass.roc' function now accepts multivariate decision values (code contributed by Matthias Döring)
- 'ggroc' supports multiple aesthetics
- Make 'ggplot2' dependency optional
- Suggested packages can be installed interactively when required
- Passing both 'cases' and 'controls' or 'response' and 'predictor' arguments is now an error
- Many small bug fixes
1.13.0 (2018-09-23)
- 'roc' now returns 'NaN' when predictor contains infinite values (issue #30)
- Better handling of near-ties near +-Infinity and 0
- 'ggroc' supports aes="group" to allow curves with identical aesthetics
1.12.1 (2018-05-06)
- Fix a regression causing the allocation of a very large vector (issue #29)
1.12.0 (2018-05-05)
- Fix bug that crashed DeLong calculations when predictor had near-ties close to the floating point precision limit that were rounded back to a predictor value (issue #25)
- Fix bug that crashed 'ci.auc' and 'var' if 'direction' was ">" and 'percent=TRUE' (issue #25)
- Fix bug causing 'ci' to return 'NaN' values with 'method="delong"' when cases or controls had a single observation (issue #27)
- Fix 'power.roc.curve' failed with curves having 'percent=TRUE'
- Fix 'ci(..., of="coords")' returned the 'ci' function instead of the CI
- C++ code now check for user interrupts regularly with Rcpp::checkUserInterrupt()
- Better error message for 'ci.coords' attempting to return 'threshold'
- New algorithm = 5 (used by default) chooses the algorithm based on the number of thresholds to avoid worst case with algorithm = 3
1.11.0 (2018-03-24)
- Added argument 'legacy.axes' to 'ggroc'
- Fix NOTE about 'apparent S3 methods exported but not registered' in 'R CMD check'
1.10.0 (2017-06-10)
- Basic ggplot2 support (one and multiple ROC curves)
- Implement 'precision' and 'recall' for 'coords'
- Fix: properly handle NAs in cases when passing cases/controls to 'roc' (thanks Thomas König for the report)
- Fix various minor bugs detected with new unit tests
1.9.1 (2017-02-05)
- Fix: 'subset' and 'na.action' arguments now handled properly in 'roc.formula' (thanks Terry Therneau for the report)
- Added policies to handle the case where a ROC curve has multiple "best" threshold in 'ci' (thanks Nicola Toschi for the report)
- Support 'xlim' and 'ylim' gracefully in 'plot.roc'
- Improved validation of input class 'levels' and 'direction'; messages when auto-detecting, use 'quiet' to turn on
- Removed extraneous 'name' attribute on the 'p.value' (thanks Paweł Kleka for the report)
- Faster DeLong algorithm (code contributed by Stefan Siegert)
1.8 (2015-05-04)
- NAMESPACE now properly exports the methods as S3 methods.
- Now uses and works with 'requireNamespace'
- Add ability to supply two ordered factors with identical levels as control / cases
- Deprecate dangerous use of a matrix as response / predictor with a warning
- Forward 'best.method' and 'best.weights' arguments to 'coords' in 'ci.thresholds'
- Fix CITATION file as per CRAN request
- pAUC correction is undefined for partial AUC below the diagonal (result < 0.5) and now returns NA (with a warning). Thanks Vincenzo Lagani for the report.
1.7.3 (2014-06-14)
- Fixed AUC of binomial-smoothed ROC off by 100^2 (thanks Bao-Li Chang for the report)
- Fix print of logcondens-smoothed ROC
1.7.2 (2014-04-05)
- Fixed bug where 'ci.coords' with 'x="best"' would fail if one or more resampled ROC curve had multiple "best" thresholds
- Fixed bug in 'ci.coords': passing more than one value in 'x' now works
- Fixed typo in documentation of 'direction' argument to 'roc' (thanks Le Kang for the report)
- Add a warning when computing statistics of ROC curve with AUC = 1
- Require latest version of Rcpp to avoid weird errors (thanks Tom Liptrot for the report)
1.7.1 (2014-02-20)
- Close SOCK cluster on Windows with parallel=TRUE
- Fixed really use algorithm 1 when microbenchmark fails
1.7 (2014-02-19)
- Faster algorithm for DeLong 'roc.test', 'power.roc.test', 'ci.auc', 'var' and 'cov' function (no large matrix allocation)
- Handling Math and Operations correctly on 'auc' and 'ci' objects (see '?groupGeneric.pROC')
- The 'formula' for 'roc.formula' can now provide several predictors and a list of ROC curves will be returned
- Fixed documentation of 'ci.coords' with examples
- Fixed binormal AUC computed with triangulation despite the claim in the documentation
- Fixed unstated requirement on Rcpp >= 0.10.5
1.6.0.1 (2013-12-28)
- Removed erroneous error message displayed when predictors and responses were not vectors
1.6 (2013-12-26)
- New 'power.roc.test' function for sample size and power computations
- New 'cov' and 'var' functions supports new "obuchowski" method
- New 'ci.coords' function to compute CI of arbitrary coords
- 'coords' accepts new 'ret' value "1-accuracy"
- Introducing various algorithms to compute sensitivities and specificites, with a more vectorized code or Rcpp. See 'algorithm' in ?roc for more details on the trade-offs of the different methods.
- Faster algorithm for DeLong 'roc.test', 'ci', 'var' and 'cov' function (thanks Kazuki Yoshida).
- 'are.paired' now also checks for identical 'levels'
- Fixed a warning generated in the examples
- Fixed several bugs related with 'smooth.roc' curves
- Additional input data sanity checks
- Now requires R >= 2.13 (in fact, since 1.5.1, thanks Emmanuel Curis for the report)
- Progress bars now defaults to text on Macs where 'tcltk' seems broken (thanks Gerard Smits for the report)
1.5.4 (2012-08-31)
- Running less smooth.roc examples with logcondens because they take too much time (requested by Uwe Ligges)
1.5.3 (2012-08-31)
- AUC specification was lost when roc.test, cov or var was passed an 'auc' object.
- Correct computation of "accuracy" in 'coords' (thanks to Kosuke Yoshihara for the report)
1.5.1 (2012-03-09)
- Faster loading of the package (thanks to Prof Brian Ripley and Glenn Lawyer for the report)
1.5 (2011-12-11)
- New 'cov' and 'var' functions
- 'coords' accepts new 'ret' values: "accuracy", "tn", "tp", "fn", "fp", "npv", "ppv", "1-specificity", "1-sensitivity", "1-npv", "1-ppv", "npe" and "ppe"
- New 'legacy.axes' argument to 'plot' 1-specificity rather than specificity
- New 'axes' argument to turn off the plotting of the axis
- New 'logcondens' and 'logcondens.smooth' (Univariate Log-Concave Density Estimation) smoothing methods
- New function 'has.partial.auc' to determine if an AUC is full or partial
- New argument 'drop' for 'coords'
- 'auc' and 'multiclass.auc' objects now also have secondary class 'numeric'
- Updated load call
- Delong's CI reversed in ROC curves with direction=">"
- Delong's CI AUC returned values > 1 or < 0 in some rare cases
- Minor improvements in documentation
1.4.4 (2011-08-09)
- Fixed alternative for one-tailed tests (thanks to Lisa Koch for the report)
- Removed COPYING file to fix a warning in r-devel.
1.4.3 (2011-03-17)
- Updated citation
1.4.2 (2011-03-03)
- Fixed bootstrap 'roc.test' generating NAs when 'smooth.roc's were used with 'reuse.auc=FALSE' (thanks to Buddy for the report)
- Documented a warning that was missing in roc.test
- Updated citation
1.4.1 (2011-01-27)
- 'venkatraman''s test for unpaired ROC curves
1.4 (2011-01-21)
- 'smooth' does not apply on 'ordered' factors anymore
- Multi-class AUC support
- Can choose how 'best' thresold is determined ('best.method' and 'best.weights' in 'coords' and 'print.thres.best.method' and 'print.thres.best.weights' in 'plot.roc')
- Minor fixes in documentation
- 'print' now prints the response instead of "response" and more informative data in 'htest's
- Bootstrap with 'ci.auc' consumes much less memory
- Unpaired bootstrap and DeLong's test
- Specificity and sensitivity tests (in 'roc.test')
1.3.2 (2010-08-24)
- 'print.auc' printed incorrect CI in 'plot.roc' (thanks to Alexander B. Leichtle for the report)
- Failed to detect local maximas in 'coords' when 2 or less points were selected
- Don't consider ROC extremities (+-Inf at 1.0/0.0 SE<->SP) as local maximas
1.3.1 (2010-08-18)
- Sensitivity and specificity were inverted in coords when results were reported as list
- Faster checks with '\dontrun{}' in 'roc.test'
1.3 (2010-08-13)
- '...' not passed correctly in 'plot.ci.se' with 'type="bars"' resulting in an error
- CI is not re-computed by default in 'smooth.roc'. You can still turn it on with 'reuse.ci=TRUE'
- New function 'are.paired'
- Local maximas could be incorrectly detected in 'coords' (and 'plot.roc') with 'predictor's containing more than 2 levels.
- New method 'venkatraman' for 'roc.test'
- MASS and tcltk packages are now only suggested instead of required
1.2.1 (2010-05-11)
- Handle 'method' arguments for 'smooth.roc' and 'ci.auc' separately in 'roc.default'
- Added 'auc.polygon.' and 'max.auc.polygon.' arguments for 'polygon' in 'plot.roc'
1.2 (2010-05-09)
- Added DeLong method in 'ci.auc'
- Return value of 'ci.auc' does not contain an 'aucs' item anymore
- Put most examples with bootstrap within '\dontrun{}' blocks for faster (but less useful) checks execution
1.1 (2010-05-05)
- Added 'lines.roc' functions for ROC
- Added 'type' argument for both 'lines.roc' and 'plot.roc'
- Added 'print.auc.col' argument to 'plot.roc'
- Fixed a warning in 'roc.test.default' when the class of 'predictor1' had several elements
- Fixed an encoding failure during the checks on MacOS X
1.0.1 (2010-04-28)
- Reduced examples execution time. Added low 'boot.n' in the slowest examples and 'reuse.auc' and 'reuse.ci' arguments in smooth.roc.roc
1.0 (2010-04-27)
- First public release