sortinghat is a classification framework to streamline the evaluation of classifiers (classification models and algorithms) and seeks to determine the best classifiers on a variety of simulated and benchmark data sets. Several error-rate estimators are included to evaluate the performance of a classifier. This package is intended to complement the well-known 'caret' package.
sortinghat package is a framework in R to streamline the evaluation of
classifiers (classification models and algorithms) and seeks to determine the
best classifiers on a variety of simulated and benchmark data sets with a
collection of benchmark metrics.
You can install the stable version on CRAN:
install.packages('sortinghat', dependencies = TRUE)
If you prefer to download the latest version, instead type:
A primary goal of
sortinghat is to enable rapid benchmarking across a variety of
classification scenarios. To achieve this, we provide a large selection of both
real and simulated data sets collected from the literature and around the
sortinghat, researchers can quickly replicate findings within the
literature as well as rapidly prototype new classifiers.
The list of real and simulated data sets will continue to grow. Contributions are greatly appreciated as pull requests.
Benchmark data sets are useful for evaluating and comparing classifiers...
(Work in Progress: Version 0.2 will include a collection of benchmark data sets)
In addition to benchmark data sets,
sortinghat provide a large collection of
data-generating models for simulations based on studies in the literature. Thus
far, we have added multivariate simulation models based on the following family
Moreover, data can be generated based on the well-known configurations from:
The simulated data sets listed above can be generated via the
Classifier superiority is often determined by classification error rate (1 - accuracy). To assess classification efficacy, we utilize the following error-rate estimators:
Each of these error rates can be accessed via the
errorest function, which
acts as a wrapper around the error-rate estimators listed above.
Simulated data sets and configurations are each available in functions
simdata function is a wrapper around each of
?simdata for a list of all the available simulated data sets and
the implementation details.
Several error-rate estimators are available, including cross-validation, .632,
.632+, and others. The name of each estimator's function is prefaced with
errorest is a wrapper function around the error-rate
estimators implemented. See
?errorest for a list of all available error-rate
estimators and the implementation details.
cv_partition: Partitions data for cross-validation.
partition_data: Randomly partitions data sets into training and test data
sets with a specified percentage in each.
which_min: Determines the index (location) of the minimum element in a
vector. Breaks ties in a variety of ways -- in particular, at random. This
function is intended to replace the base
cov_intraclass: Constructs a p-dimensional intraclass covariance matrix.
cov_autocorrelation: Constructs a p-dimensional covariance matrix with an
cov_block_autocorrelation: Constructs a p-dimensional block-diagonal
covariance matrix with autocorrelated blocks. Based on Guo, Hastie, and