Threshold-Free Class-Specific Comparison of Binary Classifiers

Implements the U-smile methodology for threshold-free, class-specific comparison of probabilistic binary classifiers. The package quantifies prediction improvement and worsening separately for non-events and events using the Brier alteration (BA), relative Brier (RB), improvement proportion (I) coefficients, and relative likelihood ratio (rLR) coefficients, and provides U-smile, prediction improvement-worsening, receiver operating characteristic, and precision-recall plots. The original U-smile framework is described in Kubiak et al. (2024) , its three-level extension for imbalanced binary classification in Wieckowska et al. (2025) , and the likelihood-based extension in Wieckowska and Guzik (2026) .

Usmile

Overview

Universal Smile Layout for Explanation (Usmile)

Usmile is an R package for threshold-free, class-specific evaluation and comparison of probabilistic binary classifiers.

The package implements the U-smile methodology, which evaluates changes in predicted probabilities separately for:

• non-events with improved prediction;
• non-events with worsened prediction;
• events with worsened prediction;
• events with improved prediction.

The U-smile plot visualizes these four subclasses in a fixed order. The resulting shape provides an immediate graphical assessment of whether a new model improves or worsens prediction for the non-event and event classes.

The package supports the following U-smile coefficients:

• BA: absolute average change in prediction;
• RB: relative change in prediction error;
• I: proportion of individuals with changed prediction;
• rLR: relative likelihood-ratio measure of prediction improvement.

The methodology is intended for probabilistic binary classifiers and does not require selecting a classification threshold.

Main functionality

Usmile provides functions for:

• comparing two fitted probabilistic binary classification models;
• comparing vectors of predicted probabilities;
• calculating class- and subclass-specific U-smile coefficients;
• visualizing prediction improvement and worsening;
• generating U-smile, PIW, ROC, precision-recall, and calibration plots;
• applying externally fitted probability calibration models;
• analysing training and external test datasets.

The core functions include:

Installation

CRAN

After the package is published on CRAN, it can be installed with:

install.packages("Usmile")

Development version

The current development version can be installed from GitHub:

# install.packages("remotes")
remotes::install_github("bbwieckowska/Usmile")

Quick start

The following example compares a reference logistic regression model with a larger model containing additional predictors.

library(Usmile)

data("heart_disease_train")
data("heart_disease_test")

heart_disease_train$disease <- as.factor(
  heart_disease_train$disease
)

heart_disease_test$disease <- as.factor(
  heart_disease_test$disease
)

reference_model <- stats::glm(
  disease ~ 1,
  data = heart_disease_train,
  family = stats::binomial()
)

new_model <- stats::glm(
  disease ~ ill_high_asym + age + cp,
  data = heart_disease_train,
  family = stats::binomial()
)

Evaluation on the training dataset

results_train <- UScalc_mdl(
  ref_model = reference_model,
  new_model = new_model,
  y_coef = "rLR",
  testing = FALSE
)

results_train$results

USplot(
  plot_data = results_train$plot_data,
  y_coef = "rLR",
  net = TRUE,
  crit = 2
)

Evaluation on an external test dataset

results_test <- UScalc_mdl(
  ref_model = reference_model,
  new_model = new_model,
  y_coef = "rLR",
  dataset = heart_disease_test,
  testing = TRUE
)

results_test$results

USplot(
  plot_data = results_test$plot_data,
  y_coef = "rLR",
  net = TRUE,
  crit = 2
)

Comparison from predicted probabilities

The U-smile coefficients can also be calculated without passing fitted model objects.

The input data frame must contain:

• y: observed binary outcome;
• p_ref: probability predicted by the reference model;
• p: probability predicted by the new model.

prediction_data <- data.frame(
  y = c(0, 0, 1, 1),
  p_ref = c(0.10, 0.35, 0.60, 0.85),
  p = c(0.05, 0.40, 0.55, 0.92)
)

results_raw <- UScalc_raw(
  raw_data = prediction_data,
  y_coef = "rLR",
  n_vars_diff = 1
)

USplot(
  plot_data = results_raw,
  y_coef = "rLR",
  net = TRUE,
  crit = 2
)

This workflow is model-agnostic because it requires only observed outcomes and predicted probabilities.

Probability calibration

Calibration must be fitted independently of the dataset used for final model evaluation.

calibration_outcomes <- c(0, 0, 0, 1, 1, 1)

calibration_predictions <- c(
  0.10,
  0.25,
  0.40,
  0.55,
  0.75,
  0.90
)

# Fit the calibrator on an independent calibration dataset
calibrator <- USfit_calibrator(
  y = calibration_outcomes,
  p = calibration_predictions,
  method = "logistic"
)

# Example predictions from a separate test dataset
test_predictions <- c(
  0.15,
  0.35,
  0.65,
  0.85
)

calibrated_probabilities <- USapply_calibrator(
  object = calibrator,
  p = test_predictions
)

calibrated_probabilities

Available calibration methods are:

• "logistic";
• "intercept";
• "isotonic".

The calibration model should not be estimated using the final test dataset.

Shiny application

A point-and-click implementation of the U-smile workflow is available as a companion Shiny application:

https://barbarawieckowska.shinyapps.io/ShinyApp/

The Shiny application is intended for interactive analyses, demonstrations, and educational use.

Documentation

Function-level documentation is available directly in R:

help(package = "Usmile")
?UScalc_mdl
?UScalc_raw
?USplot

The package citation can be displayed with:

citation("Usmile")

A PDF reference manual can be generated from the package source with:

devtools::build_manual()

Methodological background

The original U-smile framework, including the BA, RB, and I coefficients and the prediction improvement-worsening matrix, was introduced in:

Kubiak KB, Więckowska B, Jodłowska-Siewert E, Guzik P (2024). Visualising and quantifying the usefulness of new predictors stratified by outcome class: The U-smile method. PLOS ONE, 19(5), e0303276.
https://doi.org/10.1371/journal.pone.0303276

The three-level U-smile approach and its evaluation under class imbalance were described in:

Więckowska B, Kubiak KB, Guzik P (2025). Evaluating the three-level approach of the U-smile method for imbalanced binary classification. PLOS ONE, 20(4), e0321661.
https://doi.org/10.1371/journal.pone.0321661

The likelihood-based extension of the methodology and the rLR coefficient were described in:

Więckowska B, Guzik P (2026). Usmile likelihood evaluation provides robust threshold free assessment of binary classification models for balanced and imbalanced datasets. Scientific Reports, 16, 10000.
https://doi.org/10.1038/s41598-026-40545-z

Citation

When using the package, cite both the software version and the publication describing the methodological component used in the analysis.

The recommended citations can be obtained with:

citation("Usmile")

Reproducibility

For reproducible analyses, report at least:

• the Usmile package version;
• the R version;
• the coefficient used: BA, RB, I, or rLR;
• whether results were calculated on training or test data;
• the definitions of the reference and new models;
• any probability calibration procedure;
• the random seed for stochastic modelling algorithms.

Package and R versions can be recorded with:

packageVersion("Usmile")
sessionInfo()

Contributing

Bug reports and feature requests can be submitted through the GitHub issue tracker:

https://github.com/bbwieckowska/Usmile/issues

Contributions should preserve compatibility with the documented statistical definitions of the U-smile coefficients. Changes affecting numerical results should include appropriate tests and documentation.

License

Usmile is released under the MIT License. See the LICENSE file for details.