Classification and Regression Tests

Provides wrapper functions for running classification and regression tests using different machine learning techniques, such as Random Forests and decision trees. The package provides standardized methods for preparing data to suit the algorithm's needs, training a model, making predictions, and evaluating results. Also, some functions are provided to run multiple instances of a test.


The aim of Classification and Regression Tests is to make the process of running classifier and regression tests easier. Each classifier and regression technique available in R, such as Adaboost, Random Forests, or Classification and Regression Trees, seems to have slightly different requirements for the data, such as whether NAs are allowed. Furthermore, each technique can have (slightly) different parameters for training a model and making predictions.

This package provides a single API for making predictions based on a regression model or classification model, that makes sure the data is cleaned in such a way that the technique can process it with a minimal amount of problems. The results of a test are presented in a consistent way.

Basically, the aim of crtests is to make running a classifier or regression test as simple as providing data, selecting a technique. Several techniques are supported and verified 'out of the box': Random Forests for regression and classification through the randomForest package; CART for regression and classification through the rpart package; Adaboost for regression through the gbm package and Adaboost for classification through the boosting function from the adabag package; linear regression through lm.

crtests splits the classifier/regression testing process into multiple steps, each of which can be adapted to other techniques with requirements of their own. See the "extending" vignette to see how crtests can be made to work with your favorite technique, if it does not work out of the box.

crtests can be used both for single tests, where a single sample of data is used as a training set, and for multiple tests, where multiple samples of training data are created, and a test is run on each. The latter supports cross validation.

library(crtests)
library(randomForest)
#> randomForest 4.6-12
#> Type rfNews() to see new features/changes/bug fixes.
library(caret)
#> Loading required package: lattice
#> Loading required package: ggplot2
#> 
#> Attaching package: 'ggplot2'
#> The following object is masked from 'package:randomForest':
#> 
#>     margin
 
data(iris)
# A classification test
test <- createtest(data = iris, 
                   dependent = "Species",
                   problem = "classification",
                   method = "randomForest",
                   name = "An example classification test",
                   train_index = sample(150, 100)
                   )
runtest(test)
#> Classification Test Evaluation: An example classification test
#> 
#> Test attributes:
#>                                                
#>                    Method : randomForest       
#>        Dependent variable : Species            
#>       Percentage held out : 33.33333% (50 rows)
#>        Total rows in data : 150                
#>       Data transformation : None               
#> 
#> Performance measures & statistics:
#>                                                 
#>                  Accuracy : 0.98                
#>                    95% CI : 0.8935305, 0.9994938
#>       No information rate : 0.46                
#>  P-value (accuracy > NIR) : 8.200386e-16        
#>    McNemar's test P-value : NaN
 
# A regression test
test <- createtest(data = iris,
                   dependent = "Sepal.Width",
                   problem = "regression",
                   method = "randomForest",
                   name = "An example regression test",
                   train_index = sample(150,100)
                   )
runtest(test)
#> Regression Test Evaluation: An example regression test
#> 
#> Test attributes:
#>                                                      
#>                          Method : randomForest       
#>              Dependent variable : Sepal.Width        
#>             Percentage held out : 33.33333% (50 rows)
#>              Total rows in data : 150                
#>             Data transformation : None               
#> 
#> Performance measures & statistics:
#>                                             
#>                      Mean error : -0.1257484
#>             Mean absolute error : 0.2032545 
#>               Mean square error : 0.07694968
#>  Mean absolute percentage error : 6.02664   
#>          Root mean square error : 0.2773981
library(crtests)
library(randomForest)
library(rpart)
library(caret)
library(stringr)
 
# A classification multitest
summary(
  multitest(data = iris,
            dependent = "Species",
            problem = "classification",
            method = "randomForest",
            name = "An example classification multitest",
            iterations = 10,
            cross_validation = TRUE,
            preserve_distribution = TRUE
           )
)
#> Classification Multiple Test Evaluation: An example classification multitest 
#> 
#> Test attributes:
#> 
#> General:
#>                                                         
#>                    Method: randomForest                 
#>        Dependent variable: Species                      
#>       Data transformation: identity                     
#>           Sampling method: 10-fold cross validation with
#>                            preservation of class        
#>                            distribution                 
#> 
#> Summary of attributes per test iteration:
#> 
#>                            Min. 1st Qu. Median Mean 3rd Qu. Max.
#>              Rows held out   30      30     30   30      30   30
#>         Total rows in data  150     150    150  150     150  150
#> 
#> Performance measures:
#> 
#>                                Min.  1st Qu.   Median     Mean  3rd Qu.
#>                   Accuracy 9.00e-01 9.00e-01 9.67e-01 9.50e-01 9.92e-01
#>      Lower bound of 95% CI 7.35e-01 7.35e-01 8.28e-01 8.07e-01 8.70e-01
#>      Upper bound of 95% CI 9.79e-01 9.79e-01 9.99e-01 9.91e-01 1.00e+00
#>        No information rate 3.33e-01 3.33e-01 3.33e-01 3.33e-01 3.33e-01
#>   P-value (accuracy > NIR) 4.86e-15 7.77e-14 2.96e-13 6.67e-11 1.66e-10
#>     McNemar's test P-value       NA       NA       NA      NaN       NA
#>                                Max.
#>                   Accuracy 1.00e+00
#>      Lower bound of 95% CI 8.84e-01
#>      Upper bound of 95% CI 1.00e+00
#>        No information rate 3.33e-01
#>   P-value (accuracy > NIR) 1.66e-10
#>     McNemar's test P-value       NA
 
# A regression multitest
summary(
  multitest(data = iris,
            dependent = "Sepal.Width",
            problem = "regression",
            method = "rpart",
            name = "An example regression multitest",
            iterations = 15,
            cross_validation = FALSE
           )
)
#> Regression Multiple Test Evaluation: An example regression multitest 
#> 
#> Test attributes:
#> 
#> General:
#>                                                   
#>                          Method: rpart            
#>              Dependent variable: Sepal.Width      
#>             Data transformation: identity         
#>                 Sampling method: 15 random samples
#> 
#> Summary of attributes per test iteration:
#> 
#>                                  Min. 1st Qu. Median Mean 3rd Qu. Max.
#>                    Rows held out   10      10     10   10      10   10
#>               Total rows in data  150     150    150  150     150  150
#> 
#> Performance measures:
#> 
#>                                     Min. 1st Qu.   Median    Mean 3rd Qu.
#>                       Mean error -0.1597 -0.0881 0.000791 0.00278  0.0403
#>              Mean absolute error  0.1760  0.2044 0.214800 0.23850  0.2638
#>                Mean square error  0.0458  0.0656 0.084580 0.09507  0.1072
#>   Mean absolute percentage error  5.6550  6.7110 7.674000 8.01900  9.3140
#>           Root mean square error  0.2141  0.2560 0.290800 0.30220  0.3273
#>                                    Max.
#>                       Mean error  0.188
#>              Mean absolute error  0.354
#>                Mean square error  0.223
#>   Mean absolute percentage error 11.840
#>           Root mean square error  0.472
  1. Install the release version of devtools from CRAN: install.packages(devtools)
  2. Install crtests from GitHub: devtools::install_github("sjoerdvds/crtests")

News

crtests 0.2.1

  • First version of crtests
  • Tested to work with randomForest, rpart, boosting, gbm, and lm

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.

install.packages("crtests")

0.2.1 by Sjoerd van der Spoel, 9 months ago


Browse source code at https://github.com/cran/crtests


Authors: Sjoerd van der Spoel [aut, cre]


Documentation:   PDF Manual  


GPL-3 | file LICENSE license


Imports caret, plyr, stringr

Suggests testthat, knitr, e1071, randomForest, rpart, adabag, rmarkdown


See at CRAN