Testing Workbench for Precision-Recall Curves

A testing workbench for evaluating precision-recall curves under various conditions.

The aim of the prcbench package is to provide a testing workbench for evaluating precision-recall curves under various conditions. It contains integrated interfaces for the following five tools. It also contains predefined test data sets.

Dependencies

AUCCalculator requires a Java runtime (>= 6).

Bioconductor libraries

PerfMeas requires Bioconductor libraries. To automatically install the dependencies, add a Bioconductor repository to the repository list as:

## Include a Bioconductor repository
setRepositories(ind = 1:2)

Installation

• Install the release version of prcbench from CRAN with install.packages("prcbench").

• Alternatively, you can install a development version of prcbench from our GitHub repository. To install it:

  1. Make sure you have a working development environment.

     • Windows: Install Rtools (available on the CRAN website).
     • Mac: Install Xcode from the Mac App Store.
     • Linux: Install a compiler and various development libraries (details vary across different flavors of Linux).

  2. Install devtools from CRAN with install.packages("devtools").

  3. Install prcbench from the GitHub repository with devtools::install_github("takayasaito/prcbench").

Potential installation issues

Bioconductor libraries

You can manually install the dependencies from Bioconductor if install.packages fails to access the Bioconductor repository.

## try http:// if https:// URLs are not supported
source("https://bioconductor.org/biocLite.R")
biocLite("limma")
biocLite("graph")
biocLite("RBGL")

rJava

Some OSs require further configuration for rJava.

Use:

Sys.setenv(JAVA_HOME = "<path to JRE>")

or

#!/bin/bash

export JAVA_HOME = "<path to JRE>"
R CMD javareconf

microbenchmark

microbenchmark does not work on some OSs. prcbench uses system.time when microbenchmark is not available.

Documentation

• Introduction to prcbench - a package vignette that contains the descriptions of the functions with several useful examples. View the vignette with vignette("introduction", package = "prcbench") in R. The HTML version is also available on the GitPages.

• Help pages - all the functions including the S3 generics have their own help pages with plenty of examples. View the main help page with help(package = "prcbench") in R. The HTML version is also available on the GitPages.

Examples

Following two examples show the basic usage of prcbench functions.

Benchmarking

The run_benchmark function outputs the result of microbenchmark for specified tools.

## Load library
library(prcbench)
 
## Run microbenchmark for aut5 on b10
testset <- create_testset("bench", "b10")
toolset <- create_toolset(set_names = "auc5")
res <- run_benchmark(testset, toolset)
 
## Use knitr::kable to show the result in a table format
knitr::kable(res$tab, digits = 2)

Evaluation of precision-recall curves

The run_evalcurve function evaluates precision-recall curves with predefined test datasets. The autoplot shows a plot with the result of the run_evalcurve function.

## ggplot2 is necessary to use autoplot
library(ggplot2)
 
## Plot base points and the result of precrec on c1, c2, and c3 test sets
testset <- create_testset("curve", c("c1", "c2", "c3"))
toolset <- create_toolset("precrec")
scores1 <- run_evalcurve(testset, toolset)
autoplot(scores1)

## Plot the results of PerfMeas and PRROC on c1, c2, and c3 test sets
toolset <- create_toolset(c("PerfMeas", "PRROC"))
scores2 <- run_evalcurve(testset, toolset)
autoplot(scores2, base_plot = FALSE)

Citation

Precrec: fast and accurate precision-recall and ROC curve calculations in R

Takaya Saito; Marc Rehmsmeier

Bioinformatics 2016;

doi: 10.1093/bioinformatics/btw570

External links

• Classifier evaluation with imbalanced datasets - our web site that contains several pages with useful tips for performance evaluation on binary classifiers.

• The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets - our paper that summarized potential pitfalls of ROC plots with imbalanced datasets and advantages of using precision-recall plots instead.