Artificial Intelligence Systems and Observer Performance

Analyzing the performance of artificial intelligence (AI) systems/algorithms characterized by a "search-and-report" strategy. While historically observer performance has dealt with measuring radiologists' performance in search tasks – i.e., searching for lesions in medical images and reporting them - the software described here applies equally to any task involving searching for and reporting arbitrary targets in images. The package can be used to analyze the performance of AI systems, compare AI performance to a group of human readers or optimize the reporting threshold of an AI system. In addition to performing conventional receiver operating characteristic (ROC) analysis (localization information ignored), the software also performs free-response receiver operating characteristic (FROC) analysis, where lesion localization information is integral to the analyzed data. A book using the software has been published: Chakraborty DP: Observer Performance Methods for Diagnostic Imaging - Foundations, Modeling, and Applications with R-Based Examples, Taylor-Francis LLC; 2017. An online update of this book is at <>. Illustrations of the software (vignettes) are at <>. Supported data collection paradigms are the ROC, FROC and the location ROC (LROC). ROC data consists of single ratings per images, where a rating is the perceived confidence level that the image is that of a diseased patient. An ROC curve is a plot of true positive fraction vs. false positive fraction. FROC data consists of a variable number (zero or more) of mark-rating pairs per image, where a mark is the location of a reported suspicious region and the rating is the confidence level that it is a real lesion. LROC data consists of a rating and a location of the most suspicious region, for every image. Four models of observer performance, and curve-fitting software, are implemented: the binormal model (BM), the contaminated binormal model (CBM), the correlated contaminated binormal model (CORCBM), and the radiological search model (RSM). Unlike the binormal model, CBM, CORCBM and RSM predict "proper" ROC curves that do not inappropriately cross the chance diagonal. Additionally, RSM parameters are related to search performance (not measured in conventional ROC analysis) and classification performance. Search performance refers to finding lesions, i.e., true positives, while simultaneously not finding false positive locations. Classification performance measures the ability to distinguish between true and false positive locations. Knowing these separate performances allows principled optimization of reader or AI system performance. RJafroc supersedes Windows JAFROC (jackknife alternative FROC) software V4.2.1, <>. Package functions are organized as follows. Data file related function names are preceded by "Df", curve fitting functions by "Fit", included data sets by "dataset", plotting functions by "Plot", significance testing functions by "St", sample size related functions by "Ss", data simulation functions by "Simulate" and utility functions by "Util". Implemented are figures of merit (FOMs) for quantifying performance and functions for visualizing empirical or fitted operating characteristics: e.g., ROC, FROC, alternative FROC (AFROC) and weighted AFROC (wAFROC) curves. For fully crossed study designs significance testing of reader-averaged FOM differences between modalities is implemented via either Dorfman-Berbaum-Metz or the Obuchowski-Rockette methods. Also implemented is single treatment analysis, which allows comparison of performance of a group of radiologists to a specified value, or comparison of AI to a group of radiologists interpreting the same cases. Crossed-modality analysis is implemented wherein there are two crossed treatment factors and the aim is to determined performance in each treatment factor averaged over all levels of the second factor. Sample size estimation tools are provided for ROC and FROC studies; these use estimates of the relevant variances from a pilot study to predict required numbers of readers and cases in a pivotal study to achieve the desired power. Utility and data file manipulation functions allow data to be read in any of the currently used input formats, including Excel, and the results of the analysis can be viewed in text or Excel output files. The methods are illustrated with several included datasets from the author's collaborations. This version corrects bugs, simplifies usage of the software and updates the dataset structure. All changes are noted in NEWS.


Update History


RJafroc 1.0.2

Minor bug

StSignificanceTestingCadVsRadiologists was not working for different numbers of readers. As noted by Alejandro, the number of readers was hard coded. Fixed this and extended DfExtractDataset to include LROC datasets. Removed function SsFROCPowerGivenJK: FROC power is implemented in Online Appendix Chapter 19 (see email exchange with Kota Aoyagi)


RJafroc 1.0.1

Minor bug

Package was not installing on Solaris - overloading errors. Changed sqrt(2) in RsmFuncs.cpp to sqrt(2.0). However, Solaris is incompatible with ggplot2; so will recommend that Solaris version not be distributed on CRAN.

Sorry, but I'm not sure what's different between the CRAN Solaris machine and R-hub's Solaris machine. You could prepare a new package submission for CRAN with the caveat that, since you do not have access to a Solaris machine, your fix is speculative and may yet fail to compile on the CRAN Solaris machine. The CRAN Repository Policy ( also states: Package authors should make all reasonable efforts to provide cross-platform portable code. Packages will not normally be accepted that do not run on at least two of the major R platforms. Cases for Windows-only packages will be considered, but CRAN may not be the most appropriate place to host them. So you could in theory argue your case that your package does not support Solaris, and request that CRAN not distribute your package on that platform. But given that the issue you're bumping to is (not) documented explicitly in the R manuals, I'm not sure how much success you would have.


RJafroc 1.0.0

Major changes

Renamed functions for better organization; Removed shiny GUI interface Support for LROC datasets and cross-modality datasets CAD vs. radiologist analysis, both single modality and dual modality

RJafroc 0.1.1

Bug fixes

  • A critcal error in the p value calculation that gave incorrect p value (possibly exceeding one) when the first modality performed better than 2nd has been fixed. Thanks to Lucy D'Agostino McGowan for pointing out the error and the fix. This error, which does not occur in Windows version of JAFROC V 4.2.1, was not noticed as in all example files the 2nd modality performed better.

RJafroc 0.1.0

Major change

  • A "shiny" based GUI has been added, accessed by the function RJafrocGui(). This allows a user only interested in analyzing a data file to access the underlying code in a "user friendly way. The GUI is similar in functionality to that of Windows JAFROC 4.2.1 software.

Minor bug and aesthetic fixes

  • For the curve plotting functions, legend position and direction are automatically decided if they are not explicityly specified.
  • The the output number of significant digits for statistical power in power table has been set to 3.
  • Variance and covariance calculation error for ROI data has been fixed.
  • A bug in the JAFROC data reading function that caused an error when encountering non-numeric values has been fixed.
  • Floating point ratings are rounded to 6 significant digits when saving a dataset in JAFROC format.
  • A bug in the plotting routine that affected plots for a single rating FROC dataset has been fixed.
  • A bug in the plotting of AFROC curves for a dataset containing only non-diseased cases has been fixed.

RJafroc 0.0.1

Original version posted to CRAN website

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.


2.0.1 by Dev Chakraborty, a year ago

Browse source code at

Authors: Dev Chakraborty [cre, aut, cph] , Peter Phillips [aut] , Xuetong Zhai [aut]

Documentation:   PDF Manual  

GPL-3 license

Imports bbmle, binom, dplyr, ggplot2, mvtnorm, numDeriv, openxlsx, readxl, Rcpp, stats, stringr, tools, utils

Linking to Rcpp

See at CRAN