Goodness of Fit Noise Analysis Using Monte Carlo Techniques

Goodness-of-fit metrics, such as R-Squared, RMSE, etc., share a sensitivity to noise, dependent on the degrees of freedom. Some metrics, such as R-Squared, decrease with increasing dof and some, such as RMSE, increase with increasing dof. This package calculates the noise baseline (ceiling) by random sampling, calculating the metric’s value for each sample and counting the number of samples below a desired level, 95% by default. If one’s measure is above (below) the calculation corresponding to the desired level, then the measurement is distinguishable from noise. In addition, the ratio of the measurement to the calculated level provides a way to compare measurements of different degrees of freedom.

When measuring the goodness of fit (gof), one must be cognizant of noise. Several metrics exist to measure gof, such as R-Squared, RMSE, Kolmogorov-Smirnov, etc., but none give an understanding of the noise associated with a particular measurement. This package (gofMC) provides the estimate of the noise threshold for a given gof metric, desired noise level, noise distribution and degrees of freedom. For instance, an R-squared measurement using 5 degrees of freedom, 95% noise level, and normal noise distribution, has a threshold of 0.77. Any R-Squared measurement below this value for the same conditions is below the threshold, is at least 95% likely to have been effected by noise and is thus not distinguishable from noise. We make the measurement using a monte carlo technique where we calculate the gof metric using many random samples, then find the level at which 95% (or what ever level the user desires) of the measurements fall below. Because this is a calculation-intensive technique, some of the functions will run slowly if the number of degrees of freedom and the number of samples is large. Lastly, we can use the ratio of fit metric to noise level to define a new quantity (Fit Equivalent) that can be used for comparing two measures of different degrees of freedom.

Reference manual

install.packages("gofMC")

1.1.2 by Joseph G Kreke, 3 years ago

Browse source code at https://github.com/cran/gofMC

Authors: Joseph G Kreke , PhD; Harris , Inc. Sangeet Khemlani , PhD; Naval Research Laboratory. Greg Trafton , PhD; Naval Research Laboratory.

Documentation:   PDF Manual