Machine Learning Prediction Auditing and Model Card Reporting

Audits predictions from fitted machine learning models without requiring retraining or access to the fitted model object. It supports binary classification, multiclass classification, and regression, providing performance metrics with bootstrap confidence intervals, calibration and threshold analyses, residual diagnostics, subgroup comparisons, user-defined acceptance criteria, risk warnings, and self-contained reports. For binary-classification and regression models, the package can also assemble and render structured model cards documenting intended use, evaluation data, performance, subgroup results, assumptions, and limitations. The model-card reporting approach is described by Mitchell et al. (2019) "Model Cards for Model Reporting" .

modelcardr 0.4.0

modelcardr audits predictions from a model that has already run. It does not retrain the model and does not need the fitted model object.

It supports:

• binary classification
• multiclass classification
• regression

Install

Open modelcardr.Rproj in RStudio and choose Build > Install and Restart, or install the built source package:

install.packages("modelcardr.tar.gz", repos = NULL, type = "source")
library(modelcardr)

Binary classification

audit <- audit_predictions(
  truth = actual_class,
  probability = positive_class_probability,
  positive = "yes",
  bootstrap = 500
)
render_audit(audit, "binary-audit.html")

The audit includes classification metrics, bootstrap intervals, confusion matrix, threshold trade-offs, ROC and precision-recall curves, probability calibration, optional subgroup checks, and optional probability drift.

Multiclass classification

The probability matrix must contain one named column per class.

audit <- audit_predictions(
  truth = actual_class,
  probability = class_probability_matrix,
  task = "multiclass_classification",
  bootstrap = 500
)
render_audit(audit, "multiclass-audit.html")

The audit includes overall metrics, macro and weighted F1, minimum class recall, multiclass log loss and Brier score, one-vs-rest AUC, confusion matrix, class-level metrics, top-label calibration, and subgroup comparisons.

Regression

audit <- audit_predictions(
  truth = actual_value,
  estimate = predicted_value,
  task = "regression",
  bootstrap = 500
)
render_audit(audit, "regression-audit.html")

The audit includes MAE, RMSE, MSE, R-squared, bias, MAPE, SMAPE, correlation, bootstrap intervals, actual-versus-predicted diagnostics, residual analysis, and subgroup error comparisons.

Automatic task detection

task = "auto" is the default. Character and factor outcomes are treated as classification. Numeric truth and numeric estimates are treated as regression. For numeric class codes, set task explicitly.

Pass/fail requirements

rules <- audit_requirements(
  min_accuracy = 0.80,
  min_macro_f1 = 0.75,
  max_expected_calibration_error = 0.05
)

audit <- audit_predictions(
  truth = actual_class,
  probability = class_probability_matrix,
  task = "multiclass_classification",
  requirements = rules
)

audit$overall_status

Regression rules include max_rmse, max_mae, min_r_squared, max_absolute_bias, max_mape, and max_smape.

A requirement that does not apply to the selected task is marked INCONCLUSIVE, rather than silently ignored.