Tidy Drift Detection for Monitored Machine Learning Models

Detects concept drift and data drift in streams produced by deployed machine learning models, using a tidy interface that composes with the 'tidymodels' ecosystem. Detectors are specified, fitted on a baseline period, and advanced over new batches of observations, returning tibbles annotated with warning and drift flags. A catalogue of 22 sequential drift detectors is provided. Error-based methods include the Drift Detection Method (DDM) of Gama et al. (2004) , the Early Drift Detection Method (EDDM) of Baena-Garcia et al. (2006), the Hoeffding's inequality based Drift Detection Methods (HDDM) of Frias-Blanco et al. (2015) , and the Exponentially Weighted Moving Average (EWMA) chart of Ross et al. (2012) . Distribution-based methods include Adaptive Windowing (ADWIN) of Bifet and Gavalda (2007) , Kolmogorov-Smirnov Windowing (KSWIN) of Raab et al. (2020) , and the Page-Hinkley test of Page (1954) .

deriva

Read this in other languages: Português

deriva detects concept drift and data drift in streams produced by deployed machine learning models, through a tidy interface that composes naturally with the tidymodels ecosystem. Detectors are specified, fitted on a baseline period, and advanced over new batches of observations, returning tibbles annotated with warning and drift flags.

A machine learning model trained on historical data implicitly assumes the data-generating process stays stable over time. When that assumption breaks — user behaviour shifts, a sensor drifts out of calibration, the market changes — predictions degrade silently, with no obvious error raised. deriva watches a stream of per-observation signals (typically prediction errors) and flags the moment the underlying distribution changed.

The package ships a catalogue of 22 sequential drift detectors, covering both error-based methods (DDM, EDDM, HDDM, EWMA, ...) and distribution-based methods (ADWIN, KSWIN, Page-Hinkley, ...).

Installation

Install the released version from CRAN:

install.packages("deriva")

Or the development version from GitHub:

# install.packages("remotes")
remotes::install_github("bonijoao/deriva")

Quick start

library(deriva)

# Simulate a stream: 500 stable observations, then 500 with a higher error rate
stream <- sim_drift_stream(
  n_pre = 500, n_post = 500,
  p_pre = 0.05, p_post = 0.30,
  seed = 2
)

result <- detect_drift(stream, .col = error, method = "ddm")

# Where was drift flagged?
subset(result, .drift)
#> # A tibble: 1 × 5
#>       t error drift_true .warning .drift
#>   <int> <int> <lgl>      <lgl>    <lgl>
#> 1   542     1 TRUE       FALSE    TRUE

deriva correctly flags the change shortly after observation 500, the true drift point — with no false drift detections in the 500 stable observations before it.

The deriva interface

deriva follows the same three-verb pattern as tidymodels: specify → fit → advance.

drift_detector("ddm") |>           # specify: an inert spec, no computation yet
  fit(baseline, signal = error) |> # fit: learn the reference (baseline) level
  advance(new_batch)               # advance: update state, flag drift, keep history

The fitted object is immutable — advance() returns a new object with the updated engine state and the annotated batch appended to the history; the original is left untouched, so a stream can be replayed or forked freely.

Supplementary verbs, following the broom/tidymodels convention, make it straightforward to inspect results at any point:

• augment() — the full annotated history as a tibble
• tidy() — the detected drift points
• glance() — a one-row summary
• autoplot() — a ready-made plot of the signal with warning/drift markers

Bridging from tidymodels

add_prediction_error() converts the output of a tidymodels augment() call (which holds truth and estimate columns) into an .error column that drift detectors can consume directly — the absolute error for regression, a 0/1 mismatch indicator for classification.

monitoring_data <- model |>
  augment(new_data = production_data) |>
  add_prediction_error(truth = y)

fit(drift_detector("page_hinkley"), monitoring_data, signal = .error)

Available methods

Use drift_detector("<method>") to inspect the default hyperparameters for any method.

See vignette("deriva") for a complete walkthrough.

License

MIT © deriva authors — see LICENSE.