Calculate Accurate Precision-Recall and ROC (Receiver Operator Characteristics) Curves

Accurate calculations and visualization of precision-recall and ROC (Receiver Operator Characteristics) curves. Saito and Rehmsmeier (2015) .

Precrec

precrec calculates and plots ROC and precision-recall curves for binary classifiers. It is built for the case where the two curves disagree: on an imbalanced dataset a ROC curve can look excellent while the precision-recall curve shows the classifier is not usable. All the main calculations are implemented in C++ through Rcpp.

Quick example

library(precrec)

# Load a test dataset
data(P10N10)

# Calculate ROC and Precision-Recall curves
sscurves <- evalmod(scores = P10N10$scores, labels = P10N10$labels)

# The ggplot2 package is required
library(ggplot2)

# Show ROC and Precision-Recall plots
autoplot(sscurves)

ROC and precision-recall curves of the P10N10 test dataset, drawn side by side

auc(sscurves) gives the areas, and as.data.frame(sscurves) gives the curve points.

Documentation

Everything is on the package website, in short pages:

Get started – the five-minute tour. Also available offline with vignette("introduction", package = "precrec").
How-to – one page per task: preparing data, several test sets, cross-validation, more than two classes, large datasets.
Compare and decide – which model to ship, at which threshold, and how sure you can be.
Metrics – what each available metric means and when it misleads.
Plots – every plot the package draws, and how to change it.
Reference – the help page of every function, each with examples. Also help(package = "precrec").

Why precrec

Accurate curves. Non-linear interpolation, elongation to the y axis where the first point is undefined, and score-wise thresholds instead of fixed bins. Joining raw precision-recall points with straight lines – what most tools do – overestimates the area.

Fast. Curves over a large dataset take seconds. mode = "aucroc" computes the ROC area from the U statistic without building the curve at all.

Many metrics. Fourteen per-cutoff metrics by default and seventeen more on request, plus AUC, partial AUC, the precision-recall break-even point, and the probability-based Brier score, RMSE and log loss. See the metrics overview.

Several models and several test sets. Averaged curves with confidence bands, cross-validation folds, and confidence intervals of the AUC.

More than two classes. One-vs-rest decomposition, with per-class and macro-averaged areas.

Partial curves. Partial AUCs for any x or y range, standardized so ranges of different widths can be compared.

Installation

Install the release version from CRAN:

install.packages("precrec")

Or the development version from GitHub, which needs a working compiler (Rtools on Windows, Xcode on macOS, the usual build tools on Linux):

# install.packages("devtools")
devtools::install_github("evalclass/precrec")

Functions

Function	Description
`evalmod`	Main function to calculate evaluation metrics
`mmdata`	Reformat input data for performance evaluation calculation
`join_scores`	Join scores of multiple models into a list
`join_labels`	Join observed labels of multiple test datasets into a list
`create_sim_samples`	Create random samples for simulations
`format_nfold`	Create n-fold cross validation dataset from data frame
`prob_metrics`	Calculate the Brier score, the RMSE and the log loss
`prob_metrics_ci`	Calculate CIs of the Brier score, the RMSE and the log loss
`metric_curve`	Draw one evaluation metric against another
`metric_table`	Get every evaluation metric at every cutoff
`best_cutoff`	Pick the cutoff that optimizes one evaluation metric
`prbe`	Find the precision-recall break-even point
`average_precision`	Calculate the step estimator of the PRC area
`classification_report`	Report precision, recall and F-score per class at a threshold
`auc_boot`	Bootstrap the AUC of a single test set
`auc_delong`	Analytic standard error of the ROC AUC (DeLong)
`auc_diff`	Compare the AUCs of two models on the same resamples

Ten S3 generics work on the objects evalmod and metric_curve return.

S3 generic	Package	Description
`print`	base	Print the calculation results and the summary of the test data
`as.data.frame`	base	Convert a precrec object to a data frame
`as.data.table`	data.table	Convert a precrec object to a data.table
`plot`	graphics	Plot performance evaluation metrics
`autoplot`	ggplot2	Plot performance evaluation metrics with ggplot2
`fortify`	ggplot2	Prepare a data frame for ggplot2
`auc`	precrec	Make a data frame with AUC scores
`part`	precrec	Calculate partial curves and partial AUC scores
`pauc`	precrec	Make a data frame with pAUC scores
`auc_ci`	precrec	Calculate confidence intervals of AUC scores

Citation

Precrec: fast and accurate precision-recall and ROC curve calculations in R

Takaya Saito; Marc Rehmsmeier

Bioinformatics 2017; 33 (1): 145-147.

doi: 10.1093/bioinformatics/btw570

External links

Classifier evaluation with imbalanced datasets - our web site that contains several pages with useful tips for performance evaluation on binary classifiers.
The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets - our paper that summarized potential pitfalls of ROC plots with imbalanced datasets and advantages of using precision-recall plots instead.