Many regularized procedures produce
sparse solution and therefore are sometimes used for variable selection in
linear regression. It has been showed that regularized procedures are more stable than subset selection. Such procedures include LASSO, SCAD, and adaptive
LASSO, to name just a few. However, their performance depends crucially on the
tuning parameter selection.
For the purpose of prediction, popular methods for
the tuning parameter selection include Cp, cross-validation, and generalized
cross-validation. For the purpose of variable selection, the most popular
method for the tuning parameter selection is BIC. The selection consistencies of BIC for some regularized procedures have been shown. (Here the selection
consistency means that the probability of selecting the data generating model
is tending to one when the sample size goes to infinity, assuming that the data
generating model is a subset of the full model.)