Estimation of Entropy, Mutual Information and Related Quantities

Implements various estimators of entropy for discrete random variables, including the shrinkage estimator by Hausser and Strimmer (2009), the maximum likelihood and the Millow-Madow estimator, various Bayesian estimators, and the Chao-Shen estimator. It also offers an R interface to the NSB estimator. Furthermore, the package provides functions for estimating the Kullback-Leibler divergence, the chi-squared divergence, mutual information, and the chi-squared divergence of independence. It also computes the G statistic and the chi-squared statistic and corresponding p-values. Furthermore, there are functions for discretizing continuous random variables.


             Release History of "entropy" Package

         CHANGES IN `entropy' PACKAGE VERSION 1.2.1
  • change of maintainer email address.

           CHANGES IN `entropy' PACKAGE VERSION 1.2.0
  • new functions KL.plugin(), KL.empirical(), Kl.Dirichlet(), and KL.shrink() for computing the Kullback-Leibler divergence.

  • new functions chi2.plugin(), chi2.empirical(), chi2.Dirichlet(), and chi2.shrink() for computing the chi-squared statistic.

  • new functions chi2indep.plugin(), chi2indep.empirical(), chi2indep.Dirichlet(), and chi2indep.shrink() for computing the chi-squared statistic of independence.

  • new discretize2d() function for simple binning of a pair of random variables.

  • in all shrinkage functions the target option has been removed (and the default uniform target is now always used).

           CHANGES IN `entropy' PACKAGE VERSION 1.1.9
  • new discretize() function for simple binning of a continuous random variable.

  • now requires R >= 2.15.1.

           CHANGES IN `entropy' PACKAGE VERSION 1.1.8
  • License file removed.

  • Dependencies updated.

  • the functions freqs.shrink(), entropy.shrink(), mi.shrink(), entropy(), and freqs() now allow to specify the shrinkage intensity.

           CHANGES IN `entropy' PACKAGE VERSION 1.1.7
  • NAMESPACE file added

           CHANGES IN `entropy' PACKAGE VERSION 1.1.6
  • small in documentation file fixed (for R 2.13.0 release)

  • added in documentation that the default unit for the entropy is "nats" (natural units), for "bits" one needs to set unit="log2".

           CHANGES IN `entropy' PACKAGE VERSION 1.1.5
  • small bug in documentation file fixed (for R 2.11.0 release)

           CHANGES IN `entropy' PACKAGE VERSION 1.1.4
  • reference to publication (Hausser and Strimmer, 2009) added

           CHANGES IN `entropy' PACKAGE VERSION 1.1.3
  • small changes to update documentation

           CHANGES IN `entropy' PACKAGE VERSION 1.1.2
  • verbose messages for shrinkage entropy are now more informative and in the same style as those from corpcor

  • man pages slightly updated

            CHANGES IN `entropy' PACKAGE VERSION 1.1.1
  • bug fix: freqs.shrink() now also works for sample sizes 1 and 0

  • freqs.shrink(), entropy.shrink() and mi.shrink() now return the estimated shrinkage intensity as attribute (lambda.freqs).

            CHANGES IN `entropy' PACKAGE VERSION 1.1.0
  • several new functions added for estimating mutual information: mi.plugin(), mi.empirical(), mi.Dirichlet() and mi.shrink().

            CHANGES IN `entropy' PACKAGE VERSION 1.0.0

This package implements the following estimators of entropy:

  • the James-Stein shrinkage estimator by Hausser and Strimmer,
  • the maximum likelihood estimator,
  • the Miller-Madow estimator,
  • various Bayesian estimators (based on Dirichlet-multinomial model),
  • and the Chen-Shao estimator.

It also provides an R interface to the

  • Nemenman-Shafee-Bialek (NSB) estimator.

This is the first public release (2 September 2008).

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.