Testing Workbench for Precision-Recall Curves

A testing workbench for evaluating precision-recall curves under various conditions.


The aim of the prcbench package is to provide a testing workbench for evaluating precision-recall curves under various conditions. It contains integrated interfaces for the following five tools. It also contains predefined test data sets.

ToolLink
ROCRTool web site, CRAN
AUCCalculatorTool web site
PerfMeasCRAN
PRROCCRAN
precrecTool web site, CRAN

AUCCalculator requires a Java runtime (>= 6).

PerfMeas requires Bioconductor libraries. To automatically install the dependencies, add a Bioconductor repository to the repository list as:

## Include a Bioconductor repository
setRepositories(ind = 1:2)
  • Install the release version of prcbench from CRAN with install.packages("prcbench").

  • Alternatively, you can install a development version of prcbench from our GitHub repository. To install it:

    1. Make sure you have a working development environment.

      • Windows: Install Rtools (available on the CRAN website).
      • Mac: Install Xcode from the Mac App Store.
      • Linux: Install a compiler and various development libraries (details vary across different flavors of Linux).
    2. Install devtools from CRAN with install.packages("devtools").

    3. Install prcbench from the GitHub repository with devtools::install_github("takayasaito/prcbench").

You can manually install the dependencies from Bioconductor if install.packages fails to access the Bioconductor repository.

## try http:// if https:// URLs are not supported
source("https://bioconductor.org/biocLite.R")
biocLite("limma")
biocLite("graph")
biocLite("RBGL")

Some OSs require further configuration for rJava.

Use:

Sys.setenv(JAVA_HOME = "<path to JRE>")

or

#!/bin/bash

export JAVA_HOME = "<path to JRE>"
R CMD javareconf

microbenchmark does not work on some OSs. prcbench uses system.time when microbenchmark is not available.

  • Introduction to prcbench - a package vignette that contains the descriptions of the functions with several useful examples. View the vignette with vignette("introduction", package = "prcbench") in R. The HTML version is also available on the GitPages.

  • Help pages - all the functions including the S3 generics have their own help pages with plenty of examples. View the main help page with help(package = "prcbench") in R. The HTML version is also available on the GitPages.

Following two examples show the basic usage of prcbench functions.

The run_benchmark function outputs the result of microbenchmark for specified tools.

## Load library
library(prcbench)
 
## Run microbenchmark for aut5 on b10
testset <- create_testset("bench", "b10")
toolset <- create_toolset(set_names = "auc5")
res <- run_benchmark(testset, toolset)
 
## Use knitr::kable to show the result in a table format
knitr::kable(res$tab, digits = 2)
testsettoolsettoolnameminlqmeanmedianuqmaxneval
b10auc5ROCR2.222.2820.532.282.4493.435
b10auc5AUCCalculator2.182.855.473.145.0714.105
b10auc5PerfMeas0.090.0978.950.110.11394.335
b10auc5PRROC2.502.864.152.862.989.555
b10auc5precrec6.266.3112.026.346.4334.755

The run_evalcurve function evaluates precision-recall curves with predefined test datasets. The autoplot shows a plot with the result of the run_evalcurve function.

## ggplot2 is necessary to use autoplot
library(ggplot2)
 
## Plot base points and the result of precrec on c1, c2, and c3 test sets
testset <- create_testset("curve", c("c1", "c2", "c3"))
toolset <- create_toolset("precrec")
scores1 <- run_evalcurve(testset, toolset)
autoplot(scores1)

## Plot the results of PerfMeas and PRROC on c1, c2, and c3 test sets
toolset <- create_toolset(c("PerfMeas", "PRROC"))
scores2 <- run_evalcurve(testset, toolset)
autoplot(scores2, base_plot = FALSE)

Precrec: fast and accurate precision-recall and ROC curve calculations in R

Takaya Saito; Marc Rehmsmeier

Bioinformatics 2016;

doi: 10.1093/bioinformatics/btw570

News

precrec 0.6.2

  • Create github pages with pkgdown

precrec 0.5.2

  • Update README

  • Update wrapper functions so that precrec works when PerfMeas is not available

prcbench 0.5

  • Change predifined C3 data

  • Update AppVeyor config for rJava

prcbench 0.4

  • Enhance create_usrtool

    • x and y values can be specified as precalculated precision and recall
  • Add a new test set

    • C4
  • Add test categories to curve evaluation test result

  • Improve graph options

prcbench 0.3

  • Improve the testing enviroment

    • unit tests
    • codecov
  • Change Java version

    • 1.7 -> 1.6

prcbench 0.2

  • Fix microbenchmark

    • Change from 'Imports' to 'Suggests'
    • Use sytem time when microbenchmark is unavailable
  • Improve several documents

    • help files (.Rd)
    • package vignette
    • README

prcbench 0.1

  • The first release version of prcbench

  • The package offers four main functions

    • Common tool interface for multiple tools
    • Common test data interface for benchmarking and curve evaluation
    • Benchmarcking of tools that generate Precision-Recall curves
    • Evaluation of Precision-Recall curves
  • The package contains predefined interfaces of the following five tool

    • ROCR
    • AUCCalculator
    • PerfMeas
    • PRROC
    • precrec

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.

install.packages("prcbench")

0.7.3 by Takaya Saito, a month ago


http://takayasaito.github.io/prcbench/, https://github.com/takayasaito/prcbench


Report a bug at https://github.com/takayasaito/prcbench/issues


Browse source code at https://github.com/cran/prcbench


Authors: Takaya Saito [aut, cre], Marc Rehmsmeier [aut]


Documentation:   PDF Manual  


GPL-3 license


Imports ROCR, PRROC, precrec, rJava, R6, assertthat, grid, gridExtra, graphics, ggplot2, methods, memoise

Suggests microbenchmark, PerfMeas, testthat, knitr, rmarkdown


See at CRAN