Datasets and Functions to Benchmark Neural Network Packages

Datasets and functions to benchmark (convergence, speed, ease of use) R packages dedicated to regression with neural networks (no classification in this version). The templates for the tested packages are available in the R, R Markdown and HTML formats at < https://github.com/pkR-pkR/NNbenchmarkTemplates> and < https://theairbend3r.github.io/NNbenchmarkWeb/index.html>. The submitted article to the R-Journal can be read at < https://www.inmodelia.com/gsoc2020.html>.


NNbenchmark

NNbenchmark was created during the Google Summer of Code, 2019 as a part of The R Project for Statistical Computing, to verify the convergence of the training algorithms provided in 69 Neural Network R packages available on CRAN to date. Neural networks must be trained with second order algorithms and not with first order algorithms as many packages seem to do.

The purpose of this project is to verify the quality of the training algorithms in R packages that provide neural network of perceptron type (one input layer, one normalized layer, one hidden layer with nonlinear activation function usually tanh(), one normalized layer, one output output layer) for regression purpose i.e. NN(X1, ..., Xn) = E[Y].

Packages Tested

This GSoC project will conduct a comprehensive survey of all packages that have the “neural network” keyword in thepackage title or in the package description. There are currently 69 packages on CRAN with this keyword.

Packages
AMORE ANN2 appnn autoencoder automl BNN
brnn Buddle CaDENCE cld2 cld3 condmixt
DALEX2 DamiaNN deepnet deepNN DNMF elmNNrcpp
ELMR EnsembleBase evclass gamlss.add gcForest GMDH
GMDH2 GMDHreg grnn h2o hybridEnsemble isingLenzMC
keras kerasformula kerasR leabRa learNN LilRhino
monmlp neural neuralnet NeuralNetTools NlinTS nnet
nnetpredint nnfor onnx OptimClassifier OSTSC pnn
polyreg predictoR qrnn QuantumOps quarrint radiant.model
rasclass rcane rminer rmn RSNNS ruta
simpleNeural snnr softmaxreg Sojourn.Data spnn TeachNet
tensorflow tfestimators trackdem TrafficBDE validann

Evaluation Criteria


The algorithms were tested on 12 regression datasets (both univariate and multivariate) of varying complexity.

The score for each package was based on the following parameters:

  • Documentation (0-3 stars or a binary value of 0/1)
  • UtilFunction (0-3 stars or a binary value of 0/1)
  • ConvergenceQuality (0-3 stars based on percentile method)
  • ConvergenceTime (0-3 stars based on percentile method)

To obtain the final rating, we take a weighted average of these 4 columns (giving more weight to ConvergenceQuality and ConvergenceTime).

Authors

Selected Students:

  • Akshaj Verma
  • Salsabila Mahdi

Mentors:

  • Patrice Kiener
  • Christophe Dutang
  • John C. Nash

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.

install.packages("NNbenchmark")

3.2.0 by Patrice Kiener, 5 years ago


https://github.com/pkR-pkR/NNbenchmark


Browse source code at https://github.com/cran/NNbenchmark


Authors: Patrice Kiener [aut, cre] , Christophe Dutang [aut] , Salsabila Mahdi [aut] , Akshaj Verma [aut] , Yifu Yan [ctb]


Documentation:   PDF Manual  


GPL-2 license


Imports R6, pkgload

Suggests brnn, validann


See at CRAN