Develop Clinical Prediction Models Using the Common Data Model

A user friendly way to create patient level prediction models using the Observational Medical Outcomes Partnership Common Data Model. Given a cohort of interest and an outcome of interest, the package can use data in the Common Data Model to build a large set of features. These features can then be used to fit a predictive model with a number of machine learning algorithms. This is further described in Reps (2017) .

PatientLevelPrediction

PatientLevelPrediction is part of HADES.

Introduction

PatientLevelPrediction is an R package for building and validating patient-level predictive models using data in the OMOP Common Data Model format.

Reps JM, Schuemie MJ, Suchard MA, Ryan PB, Rijnbeek PR. Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data. J Am Med Inform Assoc. 2018;25(8):969-975.

The figure below illustrates the prediction problem we address. Among a population at risk, we aim to predict which patients at a defined moment in time (t = 0) will experience some outcome during a time-at-risk. Prediction is done using only information about the patients in an observation window prior to that moment in time.

To define a prediction problem we have to define t=0 by a Target Cohort (T), the outcome we like to predict by an outcome cohort (O), and the time-at-risk (TAR). Furthermore, we have to make design choices for the model we like to develop, and determine the observational datasets to perform internal and external validation. This conceptual framework works for all type of prediction problems, for example those presented below (T=green, O=red).

Features

• Takes one or more target cohorts (Ts) and one or more outcome cohorts (Os) and develops and validates models for all T and O combinations.
• Allows for multiple prediction design options.
• Extracts the necessary data from a database in OMOP Common Data Model format for multiple covariate settings.
• Uses a large set of covariates including for example all drugs, diagnoses, procedures, as well as age, comorbidity indexes, and custom covariates.
• Allows you to add custom covariates or cohort covariates.
• Includes a large number of state-of-the-art machine learning algorithms that can be used to develop predictive models, including Regularized logistic regression, Random forest, Gradient boosting machines, Decision tree, Naive Bayes, Neural network, AdaBoost and Support vector machines.
• Allows you to add custom algorithms.
• Allows you to add custom feature engineering
• Allows you to add custom under/over sampling (or any other sampling) [note: based on existing research this is not recommended]
• Contains functionality to externally validate models.
• Includes functions to plot and explore model performance (ROC + Calibration).
• Build ensemble models using EnsemblePatientLevelPrediction.
• Build Deep Learning models using DeepPatientLevelPrediction.
• Generates learning curves.
• Includes a shiny app to interactively view and explore results.
• In the shiny app you can create a html file document (report or protocol) containing all the study results.

Screenshots

Calibration Plot	ROC Plot

Technology

PatientLevelPrediction is an R package, with some functions using python through reticulate.

System Requirements

Requires R (version 4.1 or higher). Installation on Windows requires RTools. Some optional functionality requires Java or Python.

Some machine learning algorithms require Python. Reticulate will manage Python automatically using uv (https://docs.astral.sh/uv/getting-started/). If you're in an offline environment you can manage Python in any way you want and set the RETICULATE_PYTHON environment variable to the Python binary. The Python environment must include the packages needed by the selected model, including scikit-learn.

Getting Started

• To install the package please read the Package Installation guide

• Have a look at the video below for an extensive demo of the package.

Please read the main vignette for the package:

• Building Single Patient-Level Predictive Models

In addition we have created vignettes that describe advanced functionality in more detail:

• Building Multiple Patient-Level Predictive Models
• Adding Custom Machine Learning Algorithms
• Building Deep Learning Models
• Building Ensemble Models
• Creating Learning Curves

Package function reference: Reference

User Documentation

Documentation can be found on the package website.

Support

• Developer questions/comments/feedback: OHDSI Forum
• We use the GitHub issue tracker for all bugs/issues/enhancements

Contributing

Read here how you can contribute to this package.

License

PatientLevelPrediction is licensed under Apache License 2.0

Development

PatientLevelPrediction is being developed in R Studio.

Acknowledgements

• The package is maintained by Egill Fridgeirsson and Jenna Reps and has been developed with major contributions from Peter Rijnbeek, Martijn Schuemie, Patrick Ryan, and Marc Suchard.
• We like to thank the following persons for their contributions to the package: Seng Chan You, Ross Williams, Henrik John, Xiaoyong Pan, James Wiggins, Alexandros Rekkas
• This project is supported in part through the National Science Foundation grant IIS 1251151.