51

pROC

Display and Analyze ROC Curves

Tools for visualizing, smoothing and comparing receiver operating characteristic (ROC curves). (Partial) area under the curve (AUC) can be compared with statistical tests based on U-statistics or bootstrap. Confidence intervals can be computed for (p)AUC or ROC curves.

Build Status Codecov coverage CRAN Version Downloads

An R package to display and analyze ROC curves.

For more information, see:

  1. Xavier Robin, Natacha Turck, Alexandre Hainard, et al. (2011) “pROC: an open-source package for R and S+ to analyze and compare ROC curves”. BMC Bioinformatics, 7, 77. DOI: 10.1186/1471-2105-12-77
  2. The official web page on ExPaSy
  3. The CRAN page
  4. My blog
  5. The FAQ

Stable

The latest stable version is best installed from the CRAN:

install.packages("pROC")

Help

Once the library is loaded with library(pROC), you can get help on pROC by typing ?pROC.

Getting started

If you don't want to read the manual first, try the following:

library(pROC)
data(aSAH)

Basic ROC / AUC analysis

roc(aSAH$outcome, aSAH$s100b)
roc(outcome ~ s100b, aSAH)

Smoothing

roc(outcome ~ s100b, aSAH, smooth=TRUE) 

more options, CI and plotting

roc1 <- roc(aSAH$outcome,
            aSAH$s100b, percent=TRUE,
            # arguments for auc
            partial.auc=c(100, 90), partial.auc.correct=TRUE,
            partial.auc.focus="sens",
            # arguments for ci
            ci=TRUE, boot.n=100, ci.alpha=0.9, stratified=FALSE,
            # arguments for plot
            plot=TRUE, auc.polygon=TRUE, max.auc.polygon=TRUE, grid=TRUE,
            print.auc=TRUE, show.thres=TRUE)
 
    # Add to an existing plot. Beware of 'percent' specification!
    roc2 <- roc(aSAH$outcome, aSAH$wfns,
            plot=TRUE, add=TRUE, percent=roc1$percent)        

Coordinates of the curve

coords(roc1, "best", ret=c("threshold", "specificity", "1-npv"))
coords(roc2, "local maximas", ret=c("threshold", "sens", "spec", "ppv", "npv"))

Confidence intervals

# Of the AUC
ci(roc2)
 
# Of the curve
sens.ci <- ci.se(roc1, specificities=seq(0, 100, 5))
plot(sens.ci, type="shape", col="lightblue")
plot(sens.ci, type="bars")
 
# need to re-add roc2 over the shape
plot(roc2, add=TRUE)
 
# CI of thresholds
plot(ci.thresholds(roc2))

Comparisons

    # Test on the whole AUC
    roc.test(roc1, roc2, reuse.auc=FALSE)
 
    # Test on a portion of the whole AUC
    roc.test(roc1, roc2, reuse.auc=FALSE, partial.auc=c(100, 90),
             partial.auc.focus="se", partial.auc.correct=TRUE)
 
    # With modified bootstrap parameters
    roc.test(roc1, roc2, reuse.auc=FALSE, partial.auc=c(100, 90),
             partial.auc.correct=TRUE, boot.n=1000, boot.stratified=FALSE)

Sample size

    # Two ROC curves
    power.roc.test(roc1, roc2, reuse.auc=FALSE)
    power.roc.test(roc1, roc2, power=0.9, reuse.auc=FALSE)
 
    # One ROC curve
    power.roc.test(auc=0.8, ncases=41, ncontrols=72)
    power.roc.test(auc=0.8, power=0.9)
    power.roc.test(auc=0.8, ncases=41, ncontrols=72, sig.level=0.01)
    power.roc.test(ncases=41, ncontrols=72, power=0.9)

Development

Installing the development version

Download the source code from git, unzip it if necessary, and then type R CMD INSTALL pROC. Alternatively, you can use the devtools package by Hadley Wickham to automate the process (make sure you follow the full instructions to get started):

if (! requireNamespace("devtools")) install.packages("devtools")
devtools::install_github("xrobin/pROC")

Check

To run all automated tests, including slow tests:

cd .. # Run from parent directory
VERSION=$(grep Version pROC/DESCRIPTION | sed "s/.\+ //")
R CMD build pROC
RUN_SLOW_TESTS=true R CMD check pROC_$VERSION.tar.gz

Release steps

  1. Build & check package: R CMD build pROC && R CMD check --as-cran pROC_1.12.0.tar.gz
  2. Check with slow tests: RUN_SLOW_TESTS=true R CMD check pROC_1.12.0.tar.gz
  3. Check with R-devel: rhub::check_with_rdevel()
  4. Chec reverse dependencies: devtools::revdep_check(libpath = rappdirs::user_cache_dir("revdep_lib"), srcpath = rappdirs::user_cache_dir("revdep_src"))
  5. Update Version and Date in DESCRIPTION
  6. Update version and date in NEWS
  7. Create a tag: git tag v1.12.0
  8. Submit to CRAN

News

1.13.9 (future)

  • Turn several warnings into errors

1.13.0 (2018-09-23)

  • 'roc' now returns 'NaN' when predictor contains infinite values (issue #30)
  • Better handling of near-ties near +-Infinity and 0
  • 'ggroc' supports aes="group" to allow curves with identical aesthetics

1.12.1 (2018-05-06)

  • Fix a regression causing the allocation of a very large vector (issue #29)

1.12.0 (2018-05-05)

  • Fix bug that crashed DeLong calculations when predictor had near-ties close to the floating point precision limit that were rounded back to a predictor value (issue #25)
  • Fix bug that crashed 'ci.auc' and 'var' if 'direction' was ">" and 'percent=TRUE' (issue #25)
  • Fix bug causing 'ci' to return 'NaN' values with 'method="delong"' when cases or controls had a single observation (issue #27)
  • Fix 'power.roc.curve' failed with curves having 'percent=TRUE'
  • Fix 'ci(..., of="coords")' returned the 'ci' function instead of the CI
  • C++ code now check for user interrupts regularly with Rcpp::checkUserInterrupt()
  • Better error message for 'ci.coords' attempting to return 'threshold'
  • New algorithm = 5 (used by default) chooses the algorithm based on the number of thresholds to avoid worst case with algorithm = 3

1.11.0 (2018-03-24)

  • Added argument 'legacy.axes' to 'ggroc'
  • Fix NOTE about 'apparent S3 methods exported but not registered' in 'R CMD check'

1.10.0 (2017-06-10)

  • Basic ggplot2 support (one and multiple ROC curves)
  • Implement 'precision' and 'recall' for 'coords'
  • Fix: properly handle NAs in cases when passing cases/controls to 'roc' (thanks Thomas König for the report)
  • Fix various minor bugs detected with new unit tests

1.9.1 (2017-02-05)

  • Fix: 'subset' and 'na.action' arguments now handled properly in 'roc.formula' (thanks Terry Therneau for the report)
  • Added policies to handle the case where a ROC curve has multiple "best" threshold in 'ci' (thanks Nicola Toschi for the report)
  • Support 'xlim' and 'ylim' gracefully in 'plot.roc'
  • Improved validation of input class 'levels' and 'direction'; messages when auto-detecting, use 'quiet' to turn on
  • Removed extraneous 'name' attribute on the 'p.value' (thanks Paweł Kleka for the report)
  • Faster DeLong algorithm (code contributed Stefan Siegert)

1.8 (2015-05-04)

  • NAMESPACE now properly exports the methods as S3 methods.
  • Now uses and works with 'requireNamespace'
  • Add ability to supply two ordered factors with identical levels as control / cases
  • Deprecate dangerous use of a matrix as response / predictor with a warning
  • Forward 'best.method' and 'best.weights' arguments to 'coords' in 'ci.thresholds'
  • Fix CITATION file as per CRAN request
  • pAUC correction is undefined for partial AUC below the diagonal (result < 0.5) and now returns NA (with a warning). Thanks Vincenzo Lagani for the report.

1.7.3 (2014-06-14)

  • Fixed AUC of binomial-smoothed ROC off by 100^2 (thanks Bao-Li Chang for the report)
  • Fix print of logcondens-smoothed ROC

1.7.2 (2014-04-05)

  • Fixed bug where 'ci.coords' with 'x="best"' would fail if one or more resampled ROC curve had multiple "best" thresholds
  • Fixed bug in 'ci.coords': passing more than one value in 'x' now works
  • Fixed typo in documentation of 'direction' argument to 'roc' (thanks Le Kang for the report)
  • Add a warning when computing statistics of ROC curve with AUC = 1
  • Require latest version of Rcpp to avoid weird errors (thanks Tom Liptrot for the report)

1.7.1 (2014-02-20)

  • Close SOCK cluster on Windows with parallel=TRUE
  • Fixed really use algorithm 1 when microbenchmark fails

1.7 (2014-02-19)

  • Faster algorithm for DeLong 'roc.test', 'power.roc.test', 'ci.auc', 'var' and 'cov' function (no large matrix allocation)
  • Handling Math and Operations correctly on 'auc' and 'ci' objects (see '?groupGeneric.pROC')
  • The 'formula' for 'roc.formula' can now provide several predictors and a list of ROC curves will be returned
  • Fixed documentation of 'ci.coords' with examples
  • Fixed binormal AUC computed with triangulation despite the claim in the documentation
  • Fixed unstated requirement on Rcpp >= 0.10.5

1.6.0.1 (2013-12-28)

  • Removed erroneous error message displayed when predictors and responses were not vectors

1.6 (2013-12-26)

  • New 'power.roc.test' function for sample size and power computations
  • New 'cov' and 'var' functions supports new "obuchowski" method
  • New 'ci.coords' function to compute CI of arbitrary coords
  • 'coords' accepts new 'ret' value "1-accuracy"
  • Introducing various algorithms to compute sensitivities and specificites, with a more vectorized code or Rcpp. See 'algorithm' in ?roc for more details on the trade-offs of the different methods.
  • Faster algorithm for DeLong 'roc.test', 'ci', 'var' and 'cov' function (thanks Kazuki Yoshida).
  • 'are.paired' now also checks for identical 'levels'
  • Fixed a warning generated in the examples
  • Fixed several bugs related with 'smooth.roc' curves
  • Additional input data sanity checks
  • Now requires R >= 2.13 (in fact, since 1.5.1, thanks Emmanuel Curis for the report)
  • Progress bars now defaults to text on Macs where 'tcltk' seems broken (thanks Gerard Smits for the report)

1.5.4 (2012-08-31)

  • Running less smooth.roc examples with logcondens because they take too much time (requested by Uwe Ligges)

1.5.3 (2012-08-31)

  • AUC specification was lost when roc.test, cov or var was passed an 'auc' object.
  • Correct computation of "accuracy" in 'coords' (thanks to Kosuke Yoshihara for the report)

1.5.1 (2012-03-09)

  • Faster loading of the package (thanks to Prof Brian Ripley and Glenn Lawyer for the report)

1.5 (2011-12-11)

  • New 'cov' and 'var' functions
  • 'coords' accepts new 'ret' values: "accuracy", "tn", "tp", "fn", "fp", "npv", "ppv", "1-specificity", "1-sensitivity", "1-npv", "1-ppv", "npe" and "ppe"
  • New 'legacy.axes' argument to 'plot' 1-specificity rather than specificity
  • New 'axes' argument to turn off the plotting of the axis
  • New 'logcondens' and 'logcondens.smooth' (Univariate Log-Concave Density Estimation) smoothing methods
  • New function 'has.partial.auc' to determine if an AUC is full or partial
  • New argument 'drop' for 'coords'
  • 'auc' and 'multiclass.auc' objects now also have secondary class 'numeric'
  • Updated load call
  • Delong's CI reversed in ROC curves with direction=">"
  • Delong's CI AUC returned values > 1 or < 0 in some rare cases
  • Minor improvements in documentation

1.4.4 (2011-08-09)

  • Fixed alternative for one-tailed tests (thanks to Lisa Koch for the report)
  • Removed COPYING file to fix a warning in r-devel.

1.4.3 (2011-03-17)

  • Updated citation

1.4.2 (2011-03-03)

  • Fixed bootstrap 'roc.test' generating NAs when 'smooth.roc's were used with 'reuse.auc=FALSE' (thanks to Buddy for the report)
  • Documented a warning that was missing in roc.test
  • Updated citation

1.4.1 (2011-01-27)

  • 'venkatraman''s test for unpaired ROC curves

1.4 (2011-01-21)

  • 'smooth' does not apply on 'ordered' factors anymore
  • Multi-class AUC support
  • Can choose how 'best' thresold is determined ('best.method' and 'best.weights' in 'coords' and 'print.thres.best.method' and 'print.thres.best.weights' in 'plot.roc')
  • Minor fixes in documentation
  • 'print' now prints the response instead of "response" and more informative data in 'htest's
  • Bootstrap with 'ci.auc' consumes much less memory
  • Unpaired bootstrap and DeLong's test
  • Specificity and sensitivity tests (in 'roc.test')

1.3.2 (2010-08-24)

  • 'print.auc' printed incorrect CI in 'plot.roc' (thanks to Alexander B. Leichtle for the report)
  • Failed to detect local maximas in 'coords' when 2 or less points were selected
  • Don't consider ROC extremities (+-Inf at 1.0/0.0 SE<->SP) as local maximas

1.3.1 (2010-08-18)

  • Sensitivity and specificity were inverted in coords when results were reported as list
  • Faster checks with '\dontrun{}' in 'roc.test'

1.3 (2010-08-13)

  • '...' not passed correctly in 'plot.ci.se' with 'type="bars"' resulting in an error
  • CI is not re-computed by default in 'smooth.roc'. You can still turn it on with 'reuse.ci=TRUE'
  • New function 'are.paired'
  • Local maximas could be incorrectly detected in 'coords' (and 'plot.roc') with 'predictor's containing more than 2 levels.
  • New method 'venkatraman' for 'roc.test'
  • MASS and tcltk packages are now only suggested instead of required

1.2.1 (2010-05-11)

  • Handle 'method' arguments for 'smooth.roc' and 'ci.auc' separately in 'roc.default'
  • Added 'auc.polygon.' and 'max.auc.polygon.' arguments for 'polygon' in 'plot.roc'

1.2 (2010-05-09)

  • Added DeLong method in 'ci.auc'
  • Return value of 'ci.auc' does not contain an 'aucs' item anymore
  • Put most examples with bootstrap within '\dontrun{}' blocks for faster (but less useful) checks execution

1.1 (2010-05-05)

  • Added 'lines.roc' functions for ROC
  • Added 'type' argument for both 'lines.roc' and 'plot.roc'
  • Added 'print.auc.col' argument to 'plot.roc'
  • Fixed a warning in 'roc.test.default' when the class of 'predictor1' had several elements
  • Fixed an encoding failure during the checks on MacOS X

1.0.1 (2010-04-28)

  • Reduced examples execution time. Added low 'boot.n' in the slowest examples and 'reuse.auc' and 'reuse.ci' arguments in smooth.roc.roc

1.0 (2010-04-27)

  • First public release

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.
install.packages("pROC")

1.13.0 by Xavier Robin, 5 months ago

http://expasy.org/tools/pROC/

Report a bug at https://github.com/xrobin/pROC/issues

Browse source code at https://github.com/cran/pROC

Authors: Xavier Robin [cre, aut] , Natacha Turck [aut] , Alexandre Hainard [aut] , Natalia Tiberti [aut] , Frédérique Lisacek [aut] , Jean-Charles Sanchez [aut] , Markus Müller [aut] , Stefan Siegert [ctb] (Fast DeLong code)

Documentation:   PDF Manual  

GPL (>= 3) license

Imports plyr, utils, methods, Rcpp, ggplot2

Suggests microbenchmark, tcltk, MASS, logcondens, doParallel, testthat, vdiffr

Linking to Rcpp

Imported by BioPET, Biocomb, BootValidation, CalibratR, EFS, FAMILY, FFTrees, GmAMisc, Immigrate, LANDD, LEGIT, LogisticDx, SCGLR, ThresholdROC, TrendInTrend, biomod2, biospear, blkbox, chemmodlab, ebmc, elo, finalfit, gren, lilikoi, mcca, metamisc, mlDNA, nestfs, nonet, quantable, r4lineups, randomUniformForest, reportROC, sambia, stepPenal, tpAUC, wevid, yardstick.

Depended on by FRESA.CAD, RatingScaleReduction, RcmdrPlugin.ROC, bimixt, roccv.

Suggested by Causata, OSTSC, RcmdrPlugin.EZR, WeightedROC, aplore3, arsenal, bst, caret, caretEnsemble, dtree, eclust, ensemblepp, fscaret, kernDeepStackNet, mldr, palasso, prioritylasso, riskRegression, sjstats, uavRst, waffect.

See at CRAN

R links

R homepage

Download R

Mailing lists

R documentation

R manuals

R FAQs

The R Journal

CRAN links

CRAN homepage

CRAN repository policy

Submit a package

METACRAN stuff

About METACRAN

At github

Report a bug