Tidy Characterizations of Model Performance

Tidy tools for quantifying how well model fits to a data set such as confusion matrices, class probability curve summaries, and regression metrics (e.g., RMSE).

yardstick is a package to estimate how well models are working using tidy data principals. The package webpage is https://topepo.github.io/yardstick/ for more information.

For example, suppose you create a classification model and predict a data set. You might have data that look like this:

library(yardstick)
library(dplyr)
 
head(two_class_example)

##    truth  Class1   Class2 predicted
## 1 Class2 0.00359 0.996411    Class2
## 2 Class1 0.67862 0.321379    Class1
## 3 Class2 0.11089 0.889106    Class2
## 4 Class1 0.73516 0.264838    Class1
## 5 Class2 0.01624 0.983760    Class2
## 6 Class1 0.99928 0.000725    Class1

You can use a dplyr-like syntax to compute common performance characteristics of the model and get them back in a data frame:

metrics(two_class_example, truth, predicted)

## # A tibble: 1 x 1
##   accuracy
##      <dbl>
## 1    0.838

# or 
 
two_class_example %>% roc_auc(truth, Class1)

## [1] 0.939

Quasiquotation can also be used:

# probability columns:
lvl <- levels(two_class_example$truth)
 
two_class_example %>% mnLogLoss(truth, !! lvl)

## [1] -0.328

Installation

To install the package:

install.packages("yardstick")
 
## for development version:
require("devtools")
install_github("topepo/yardstick")