Routines for performing empirical calibration of observational study estimates. By using a set of negative control hypotheses we can estimate the empirical null distribution of a particular observational study setup. This empirical null distribution can be used to compute a calibrated p-value, which reflects the probability of observing an estimated effect size when the null hypothesis is true taking both random and systematic error into account. A similar approach can be used to calibrate confidence intervals, using both negative and positive controls.
EmpiricalCalibration is part of the OHDSI Methods Library.
This R package contains routines for performing empirical calibration of observational study estimates. By using a set of negative control hypotheses we can estimate the empirical null distribution of a particular observational study setup. This empirical null distribution can be used to compute a calibrated p-value, which reflects the probability of observing an estimated effect size when the null hypothesis is true taking both random and systematic error into account, as described in the paper Interpreting observational studies: why empirical calibration is needed to correct p-values.
Also supported is empirical calibration of confidence intervals, based on the results for a set of negative and positive controls, as described in the paper Empirical confidence interval calibration for population-level effect estimation studies in observational healthcare data.
data(sccs) #Load one of the included data setsnegatives <- sccs[sccs$groundTruth == 0,] #Select the negative controlsnull <- fitNull(logRr = negatives$logRr, seLogRr = negatives$seLogRr) #Fit the null distributionpositive <- sccs[sccs$groundTruth == 1,] #Select the positive controlplotCalibrationEffect(logRrNegatives = negatives$logRr, seLogRrNegatives = negatives$seLogRr, logRrPositives = positive$logRr, seLogRrPositives = positive$seLogRr, null = null) #Compute the calibrated p-value:calibrateP(null = null, logRr = positive$logRr, seLogRr = positive$seLogRr) #Compute calibrated p-value[1] 0.8390598
This is a pure R package.
Requires R (version 3.1.0 or newer).
In R, use the following commands to install the latest stable version from CRAN:
install.packages("EmpiricalCalibration")
To install the latest development version directly from GitHub, use:
install.packages("devtools")library(devtools)install_github("ohdsi/EmpiricalCalibration")
EmpiricalCalibration is licensed under Apache License 2.0
This package has been developed in RStudio.
This package is ready for use.
Martijn Schuemie is the author of this package.
NEW FEATURES
BUG FIXES
NEW FEATURES
Added plots showing expected type 1 error given an estimated empirical null distribution.
Closed form solution for RR vs SE plot for faster computation (especially when plotting credible intervals).
BUG FIXES
NEW FEATURES
Confidence interval calibration model StdDev transformed to log scale to prevent negative StdDev.
Confidence interval calibration cross-validation now allows specification of leave-out groups.
various new plots for evaluating confidence interval calibration.
Added confidence interval calibration vignette.
Added example data for confidence interval calibration
BUG FIXES
NEW FEATURES
Ability to add credible intervals to calibration effect plot
Plot CI calibration (using leave-one-out cross-validation)
BUG FIXES
Fixed vignette name in index
Removed coverage plot (moved to MethodEvaluation package)
Changes: initial submission to CRAN