Routines for Performing Empirical Calibration of Observational Study Estimates

Routines for performing empirical calibration of observational study estimates. By using a set of negative control hypotheses we can estimate the empirical null distribution of a particular observational study setup. This empirical null distribution can be used to compute a calibrated p-value, which reflects the probability of observing an estimated effect size when the null hypothesis is true taking both random and systematic error into account. A similar approach can be used to calibrate confidence intervals, using both negative and positive controls.


EmpiricalCalibration

Build Status CRAN_Status_Badge CRAN_Status_Badge

EmpiricalCalibration is part of the OHDSI Methods Library.

Introduction

This R package contains routines for performing empirical calibration of observational study estimates. By using a set of negative control hypotheses we can estimate the empirical null distribution of a particular observational study setup. This empirical null distribution can be used to compute a calibrated p-value, which reflects the probability of observing an estimated effect size when the null hypothesis is true taking both random and systematic error into account, as described in the paper Interpreting observational studies: why empirical calibration is needed to correct p-values.

Also supported is empirical calibration of confidence intervals, based on the results for a set of negative and positive controls, as described in the paper Empirical confidence interval calibration for population-level effect estimation studies in observational healthcare data.

Features

  • Estimate the empirical null distribution given the effect estimates of a set of negative controls.
  • Estimate the calibrated p-value of a given hypothesis given the estimated empirical null distribution.
  • Estimate a systematic error distribution given the effect estimates for a set of negative and positive controls.
  • Estimate the calibrated confidence interval for a given estimate given the systematic error distribution.
  • Produce various plots for evaluating the empirical calibration.
  • Contains the data sets from the papers for illustration.

Screenshots and examples

Calibration effect plot
data(sccs) #Load one of the included data sets
negatives <- sccs[sccs$groundTruth == 0,] #Select the negative controls
null <- fitNull(logRr = negatives$logRr, seLogRr = negatives$seLogRr) #Fit the null distribution
positive <- sccs[sccs$groundTruth == 1,]  #Select the positive control
plotCalibrationEffect(logRrNegatives = negatives$logRr,
                      seLogRrNegatives = negatives$seLogRr,
                      logRrPositives = positive$logRr,
                      seLogRrPositives = positive$seLogRr,
                      null = null)
 
#Compute the calibrated p-value:
calibrateP(null = null, logRr = positive$logRr, seLogRr = positive$seLogRr) #Compute calibrated p-value
[1] 0.8390598

Technology

This is a pure R package.

System requirements

Requires R (version 3.1.0 or newer).

Installation

In R, use the following commands to install the latest stable version from CRAN:

install.packages("EmpiricalCalibration")

To install the latest development version directly from GitHub, use:

install.packages("devtools")
library(devtools)
install_github("ohdsi/EmpiricalCalibration")

User Documentation

Support

License

EmpiricalCalibration is licensed under Apache License 2.0

Development

This package has been developed in RStudio.

Development status

This package is ready for use.

Acknowledgements

Martijn Schuemie is the author of this package.

News

EmpiricalCalibration 1.4.0

NEW FEATURES

  • Added plot showing effec of confidence interval calibration, similar to p-value plot.

BUG FIXES

  • Fixed 'unknown aesthetics' warning when calling plotTrueAndObserved.

EmpiricalCalibration 1.3.6

NEW FEATURES

  • Added plots showing expected type 1 error given an estimated empirical null distribution.

  • Closed form solution for RR vs SE plot for faster computation (especially when plotting credible intervals).

BUG FIXES

  • Several improvements of the robustness of fitting systematic error models.

EmpiricalCalibration 1.3.1

NEW FEATURES

  • Confidence interval calibration model StdDev transformed to log scale to prevent negative StdDev.

  • Confidence interval calibration cross-validation now allows specification of leave-out groups.

  • various new plots for evaluating confidence interval calibration.

  • Added confidence interval calibration vignette.

  • Added example data for confidence interval calibration

BUG FIXES

  • None

EmpiricalCalibration 1.2.0

NEW FEATURES

  • Ability to add credible intervals to calibration effect plot

  • Plot CI calibration (using leave-one-out cross-validation)

BUG FIXES

  • Fixed vignette name in index

  • Removed coverage plot (moved to MethodEvaluation package)

EmpiricalCalibration 1.1.0

Changes: initial submission to CRAN

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.

install.packages("EmpiricalCalibration")

2.0.0 by Martijn Schuemie, 4 months ago


https://ohdsi.github.io/EmpiricalCalibration, https://github.com/OHDSI/EmpiricalCalibration


Report a bug at https://github.com/OHDSI/EmpiricalCalibration/issues


Browse source code at https://github.com/cran/EmpiricalCalibration


Authors: Martijn Schuemie [aut, cre] , Marc Suchard [aut]


Documentation:   PDF Manual  


Apache License 2.0 license


Imports ggplot2, gridExtra, methods

Suggests knitr, rmarkdown


See at CRAN