Unified Predictive Modeling Toolkit

A unified toolkit for binary and multiclass classification modelling that chains preprocessing, feature selection, model training and tuning, evaluation, visualization, and SHAP interpretability behind one set of objects, while keeping a strict separation between the data that fits a model and the data that judges it. Models, tasks, resampling, and tuning are 'mlr3' objects, multiple imputation is pooled with Rubin's rules, and results are exported in a layout that can be reopened outside R.


uniml

English | 简体中文

R-CMD-check

uniml is a unified predictive modeling toolkit for binary and multiclass classification. It chains preprocessing, feature selection, model training and tuning, evaluation, visualization, and SHAP interpretability behind one small set of objects, while keeping a strict separation between the data that fits a model and the data that judges it.

The package is organized as a stack of layers. Every layer can be used on its own, and run_workflow() wires them together.

Layer Entry points Purpose
L0 data uniml_spec(), as_uniml_data(), make_split() Declare the task, roles, and splits
L1 task type list_task_types(), task_type_spec() Extensible per-task behaviour
L2 preprocessing fit_preprocessor(), impute_data(), scale_numeric(), encode_categorical(), balance_classes() Imputation, scaling, encoding, balancing
L3 selection select_features(), select_lasso(), select_boruta(), select_rf_importance(), select_rfe(), select_univariate(), select_multivariate(), combine_selections(), remove_collinear() Feature selection and collinearity handling
L4 modeling make_task(), resolve_learners(), train_model(), tune_model() mlr3 learners, tuning, cross validation
L5 statistics calc_metrics(), calc_ci(), calibration_metrics(), multiclass_calibration(), calibrate_probabilities(), delong_test(), hl_test(), calc_nri_idi(), dca(), find_threshold() Metrics with confidence intervals, calibration numbers, tests, decision curves
L6 results assemble_result(), rank_models(), export_results(), save_result(), read_result() Ranking, on-disk export, and a result you can hand over
L7 plots plot_roc(), plot_pr_curve(), plot_calibration(), plot_dca(), plot_confusion(), theme_uniml() Performance figures
L8 SHAP explain_shap(), plot_shap_importance(), plot_shap_beeswarm(), plot_shap_dependence(), plot_shap_waterfall(), extract_thresholds() Interpretability
L9 workflow run_workflow(), predict_uniml() End-to-end run and reuse of fitted state

Why uniml

uniml does not replace mlr3 or tidymodels. It is opinionated orchestration for one job: producing a binary risk model together with a report you can defend.

Compared with using mlr3 directly. mlr3 gives you learners, tasks, resampling, and tuning, and it lets you assemble them in any order. Order is the part that goes wrong: fit a scaler or a feature selector on the full data frame and every cross-validated number becomes optimistic. uniml fixes the order once — fit on training data, apply to held-out data, never look back — turns the remaining choices into an explicit leakage_policy object instead of an accident of script layout, and adds the reporting layer mlr3 leaves to you: bootstrap intervals, calibration fitted on out-of-fold predictions, DeLong and Hosmer-Lemeshow tests, NRI/IDI, decision curves, Rubin pooling across imputations, and a TRIPOD+AI self-audit.

Compared with tidymodels. recipes, workflows, and tune cover the modelling mechanics well. uniml targets the clinical-prediction reporting path — the statistics and the audit trail — and is built on the mlr3 learner ecosystem, so everything in it is an mlr3 learner you can also use directly.

What neither gives you out of the box. Strict Rubin's rules applied across the whole pipeline rather than to a single metric, an export layout that rebuilds every figure outside R, and failures recorded in result$meta$failures instead of silently dropped.

When not to use it. If you need a model and an AUC and nothing else, plain mlr3 is simpler. If you need multiclass, survival, or deep learning, this package is not it yet.

Scope and limitations

  • Two task types: binary and multiclass. run_workflow() is binary-only: its prediction table, thresholds, calibrators, and post-model tests assume a single probability column, so a multiclass specification stops with a message pointing at vignette("uniml-multiclass"). Multiclass is supported at the layer level — specification, data, task, learners, tuning, metrics, calibration, and plots.
  • Some binary methods have no multiclass counterpart. Decision curves, NRI/IDI, DeLong, and Hosmer-Lemeshow need one positive class. They are refused for multiclass rather than silently given a convention. Calibration is available, under two stated conventions: top-label ECE/MCE and class-wise ECE (multiclass_calibration()). Recalibration is available too, as temperature scaling (calibrate_probabilities()): one fitted scalar, which cannot change the predicted class. The stronger Dirichlet calibration is not, because its K^2 + K parameters are not identifiable on the sample size this package targets.
  • Survival is out of scope. It is a separate package with its own data model (time and status, censoring) and is not implemented here.
  • The example cohorts are synthetic. Both files in inst/extdata come from data-raw/simulate-cvd-mortality.R, and the 6.5% target event rate is a simulation parameter. They make the examples runnable and reproducible. They are not a patient cohort, and numbers computed on them say nothing about clinical performance or about how the methods behave on real data.
  • Tabular data frames only. Numeric, factor, character, and logical columns. No images, text, longitudinal or time-varying covariates, censoring, or competing risks.
  • Missing data is assumed missing at random. Imputation is median_mode, mice, missForest, or missRanger; missingness that depends on the unobserved value is not modelled.
  • No fairness or subgroup analysis. tripod_ai_check() reports those items as missing; writing them is your job.
  • A fixed learner registry, plus anything you inject. Seven defaults and four optional keys. Other mlr3 learners work, but they are not tuned unless a search space is supplied.
  • Not a deployment service. predict_uniml() scores a data frame in R. There is no REST endpoint, no model registry, and no drift monitoring.

Installation

The package is not on CRAN yet. Install it from GitHub:

# install.packages("remotes")
remotes::install_github("zhengyangszy/uniml")

Or from a local checkout, after R CMD build uniml:

install.packages("uniml_0.1.0.tar.gz", repos = NULL, type = "source")

Quick start

library(uniml)

development <- read.csv(
  system.file("extdata", "uniml-cvd-mortality.csv", package = "uniml")
)
new_admissions <- read.csv(
  system.file("extdata", "uniml-cvd-mortality-new.csv", package = "uniml")
)

spec <- uniml_spec(
  task_type = "binary",
  outcome = "died",
  positive = "yes",
  id = "id"
)
spec <- set_preprocess(spec, impute = "median_mode", scale = "zscore", encode = "onehot")
spec <- set_feature_selection(spec, methods = c("univariate", "lasso"), combine = "intersect")
spec <- set_learners(spec, c("lr", "cart"), tune = FALSE)
spec <- set_tuning(spec, folds = 5, evals = 10)
spec <- set_evaluation(
  spec,
  ci = "bootstrap",
  calibration = "platt",
  select_on = "validation"
)

split <- make_split(
  as_uniml_data(development, spec),
  prop = c(0.8, 0.2),
  stratify = TRUE
)
result <- run_workflow(
  spec,
  train = split$train$data,
  validation = split$validation$data,
  test = new_admissions
)
result

Predictions for new data reuse the fitted preprocessing, selection, and models; they never refit anything:

predict_uniml(result, newdata = new_admissions, model = "best")

Example cohorts

inst/extdata contains two synthetic cardiovascular cohorts:

File Rows Purpose
uniml-cvd-mortality.csv 2000 Development cohort; split into training and validation
uniml-cvd-mortality-new.csv 2000 Independent new admissions; external test cohort

Both cohorts target an in-hospital mortality probability of 6.5%. Their observed event rates differ slightly because the binary outcomes are sampled. The new-admission cohort is generated independently from the development cohort and has non-overlapping identifiers.

Available models

Every model in uniml is an mlr3 Learner, every data set becomes an mlr3 Task, and tuning and resampling use mlr3 objects directly. The package adds a thin layer on top instead of hiding mlr3:

  • make_task() builds a TaskClassif from your data frame and specification.
  • resolve_learners() returns mlr3 learners plus their search spaces, which you can also tune yourself with mlr3tuning.
  • run_workflow() stores the fitted learners in result$models[[key]]$learner, so anything that accepts an mlr3 learner keeps working.
  • as_uniml_binary() converts an mlr3 PredictionClassif into the object used by the evaluation and plotting helpers, and plot_roc(), plot_calibration(), and plot_confusion() accept an mlr3 prediction directly.

The registry provides seven stable default binary-classification learners:

Key mlr3 learner
lr classif.cv_glmnet
rf classif.ranger
xgb classif.xgboost
nnet classif.nnet
knn classif.kknn
cart classif.rpart
nb classif.naive_bayes

It also provides an optional set of common learners that can be requested by key but are not installed or tuned by default. These entries use their mlr3 defaults unless an explicit search space is supplied:

Key mlr3 learner Optional package
log_reg classif.log_reg stats
svm classif.svm e1071
lda classif.lda MASS
qda classif.qda MASS

The package is not limited to those keys. Any installed mlr3 classification learner can be passed to set_learners(). Learners outside the registry use their mlr3 defaults and are not tuned unless a search space is supplied.

spec <- set_learners(
  spec,
  c("lr", "classif.log_reg", "classif.svm", "classif.lda"),
  tune = TRUE
)

This is also how you use learners from packages that are not on CRAN, such as LightGBM. Install and load the providing package first, then pass the mlr3 learner key:

# remotes::install_github("mlr-org/mlr3extralearners")
library(mlr3extralearners)
spec <- set_learners(spec, "classif.lightgbm", tune = FALSE)

The retired lgb, randomForest, and gbm shortcuts now fail with a message that points at the mlr3 learner key to use instead; nothing else changes.

For a custom search space, register the learner explicitly:

register_learner(
  "svm_custom",
  mlr3::lrn("classif.svm", predict_type = "prob", probability = TRUE),
  paradox::ps(
    cost = paradox::p_dbl(0.1, 10),
    gamma = paradox::p_dbl(0.001, 1)
  )
)
spec <- set_learners(spec, "svm_custom", tune = TRUE)

Step-level arguments

Every stage accepts an explicit argument list, and unknown names are rejected instead of being silently stored:

spec <- set_preprocess(
  spec,
  impute = "mice",
  impute_args = list(m = 5, max_iter = 10),
  scale_args = list(vars = c("age", "bmi")),
  encode_args = list(max_levels = 20)
)
spec <- set_feature_selection(
  spec,
  methods = c("lasso", "boruta"),
  args = list(
    lasso = list(nfolds = 5, alpha = 0.8),
    boruta = list(tentative = TRUE)
  )
)
spec <- set_tuning(
  spec,
  folds = 5,
  repeats = 5,
  terminator = "seconds",
  seconds = 600
)
spec <- set_evaluation(
  spec,
  calibration = "platt",
  calibration_args = list(folds = 5, bins = 10, fit = "out_of_fold")
)

Leakage policy

Validation and test data never take part in imputation, scaling, encoding, selection, balancing, tuning, threshold selection, or calibration. Fitting happens on the training data only, and the resulting parameters are applied to the held-out data. set_leakage_policy() controls whether selection, balancing, and tuning are re-run inside each cross-validation fold (strict_in), left outside the folds (strict_out), or configured step by step (custom).

Mode Steps re-run inside each fold Use it when
strict_in imputation, scaling, encoding, selection, balance, tuning you want the least optimistic cross-validation estimate and can pay for the run time
strict_out none the steps are expensive (Boruta, RFE, tuning) and the data set is large
custom selection, balance, tuning you want to name the trade-off yourself
spec <- set_leakage_policy(
  spec,
  mode = "custom",
  in_cv = c("selection", "balance")
)

Two cautions:

  • strict_out fits selection, balancing, and tuning once on the whole training set. Those steps have then seen every training row, so per-fold scores are optimistic for them. Cross-validation stays useful for comparing models; do not report it as an unbiased absolute estimate.
  • Whatever the mode, validation, test, and external are never used for fitting, tuning, calibration, or threshold selection. strict_out relaxes what happens inside the training set, not outside it.

Data sets and selection

run_workflow() takes up to four data sets. test and external are both optional:

run_workflow(spec, train, validation,
             test = internal_holdout,      # optional
             external = new_admissions)    # optional

validation is used for model comparison and selection; test and external are reporting-only and never take part in fitting, tuning, calibration, threshold selection, or ranking. Requesting select_on = "test" is allowed but warns that the test performance becomes optimistic.

Cross-validation and calibration

run_workflow() reports per-fold cross-validation rows and a summary; repeats > 1 instantiates repeated cross-validation, and the summary averages folds within a repeat before summarising across repeats. Calibration is off by default (calibration = "none"); when requested, the calibrator is fitted on out-of-fold training predictions and both raw and calibrated probabilities are kept. Classification thresholds come from the same out-of-fold predictions. Failed or downgraded steps are never silent: they are listed in result$meta$failures.

With multiple imputation (impute = "mice", m > 1) the workflow fits each completed data set, pools metric estimates and confidence intervals with strict Rubin's rules (pool_mi(), Barnard-Rubin degrees of freedom), and runs cross-validation once per imputation before pooling. Prediction probabilities are averaged across imputations ("pooled predictions") and are labelled separately from the Rubin-pooled metric table.

Methodological conventions

The choices that are not obvious are deliberate:

  • Balancing is off by default (balance = "none"). Resampling a risk model changes what the predicted probability means (van den Goorbergh et al., 2022), so it is opt-in.
  • Multiple imputation is pooled with strict Rubin's rules — T = U + (1 + 1/m)B, Barnard-Rubin degrees of freedom, a t interval, and the interval clamped to the metric's domain.
  • Calibration and thresholds are fitted on out-of-fold training predictions, never on validation or test data.
  • Nothing fails silently. A step that is skipped or degraded is recorded in result$meta$failures and raises a warning when the workflow returns.

Reporting

NRI/IDI are computed against reference_model (default: the model with the best training AUC) with bootstrap intervals, and written to tests/nri_idi.csv. SHAP zero-crossing thresholds are opt-in:

spec <- set_explain(
  spec,
  model = "best",
  plots = c("importance", "beeswarm", "dependence", "waterfall"),
  thresholds = list(targets = c("age", "bmi"))
)

Plots accept theme, title, subtitle, and caption, and palette can be a ggsci name, a colour, or a colour vector. Export size and format are controlled with export_results(..., plot_args = list(dpi = 600, device = "png")).

Reusing exported results

Everything needed to redraw or re-tabulate is exported as plain CSV plus one RDS: predictions/predictions.csv (model, set, raw and calibrated probabilities, truth) is enough to rebuild ROC/calibration/DCA/confusion figures with plot_roc()/plot_calibration()/plot_dca()/plot_confusion(), performance/thresholds.csv stores the raw and calibrated thresholds per model, and shap/values_matrix_*.csv plus shap/feature_values_*.csv allow custom SHAP figures. result.rds contains the complete fitted state (models, preprocessor, calibrators) for predict_uniml() on new data.

Vignettes

vignette("uniml-atomic", package = "uniml")
vignette("uniml-workflow", package = "uniml")
vignette("uniml-shap", package = "uniml")
vignette("uniml-realdata", package = "uniml")
vignette("uniml-multiclass", package = "uniml")

uniml-realdata runs the whole workflow on pROC::aSAH, a published cohort of 113 patients with subarachnoid haemorrhage, and shows what the reporting layer does when the data is small: the outcome is a collapse of gos6, so that column has to be excluded; ratios whose bootstrap distribution is degenerate come back as NA with a reason; and a Hosmer-Lemeshow statistic that cannot be defined says so instead of returning NaN.

uniml-multiclass covers the three-class path: the metric conventions, the k-by-k confusion table, one-vs-rest ROC curves, and the methods that are deliberately not implemented for more than two classes.

Development status

uniml is at version 0.2.0. The binary and multiclass task types are implemented; survival is registered as a placeholder only. GitHub Actions checks the package on Windows, macOS, and Linux, and tests/testthat/test-consistency.R pins the metric layer to independent implementations (mlr3measures, pROC, PRROC, stats::glm(), stats::isoreg(), ResourceSelection). See NEWS.md for changes.

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.