Measures of Uncertainty for Model Selection

Following the common types of measures of uncertainty for parameter estimation, two measures of uncertainty were proposed for model selection, see Liu, Li and Jiang (2020) . The first measure is a kind of model confidence set that relates to the variation of model selection, called Mac. The second measure focuses on error of model selection, called LogP. They are all computed via bootstrapping. This package provides functions to compute these two measures. Furthermore, a similar model confidence set adapted from Bayesian Model Averaging can also be computed using this package.

maclogp: Measures of Uncertainty for Model Selection

The goal of maclogp is to compute measures of uncertainty for a model selection method based on an information criterion. Two measures were proposed by Liu, et.al. The first measure is a kind of model confidence set that measures the variation of model selection, called MAC. The second measure focuses on error of model selection, called LogP. Another similar model confidence set adapted from Bayesian Model Averaging can also be computed using this package.

Installation

You can install the released version of maclogp from github with:

devtools::install_github("YuanyuanLi96/maclogp")

Example

This is a basic example which shows you how to solve a common problem:

library(maclogp)
set.seed(0)
n= 100
B=100
p=5
x = matrix(rnorm(n*p, mean=0, sd=1), n, p)
true_b = c(1:3, rep(0,p-3))
y = x%*% true_b+rnorm(n)
alpha=c(0.1,0.05,0.01)
data=list(x=x,y=y)
models=Models_gen(1:p)
result=MAC(models, data, B, alpha)#default selection criterion is "BIC".
plot_MAC(models, alpha, result$con_sets, p)

#> [[1]]
#>      [,1] [,2] [,3]  [,4]  [,5]
#> [1,] TRUE TRUE TRUE FALSE FALSE
#> 
#> [[2]]
#>      [,1] [,2] [,3]  [,4]  [,5]
#> [1,] TRUE TRUE TRUE FALSE FALSE
#> [2,] TRUE TRUE TRUE  TRUE FALSE
#> 
#> [[3]]
#>      [,1] [,2] [,3]  [,4]  [,5]
#> [1,] TRUE TRUE TRUE FALSE FALSE
#> [2,] TRUE TRUE TRUE  TRUE FALSE
#> [3,] TRUE TRUE TRUE FALSE  TRUE

References

Liu, X., Li, Y. & Jiang, J. Simple measures of uncertainty for model selection. TEST (2020). https://doi.org/10.1007/s11749-020-00737-9.