Model selection for regression on a random design

Yannick Baraud

ESAIM: Probability and Statistics (2010)

  • Volume: 6, page 127-146
  • ISSN: 1292-8100

Abstract

top
We consider the problem of estimating an unknown regression function when the design is random with values in ℝ k . Our estimation procedure is based on model selection and does not rely on any prior information on the target function. We start with a collection of linear functional spaces and build, on a data selected space among this collection, the least-squares estimator. We study the performance of an estimator which is obtained by modifying this least-squares estimator on a set of small probability. For the so-defined estimator, we establish nonasymptotic risk bounds that can be related to oracle inequalities. As a consequence of these, we show that our estimator possesses adaptive properties in the minimax sense over large families of Besov balls Bα,l,∞(R) with R>0, l ≥ 1 and α > α1 where α1 is a positive number satisfying 1/l - 1/2 ≤ α1 < 1/l. We also study the particular case where the regression function is additive and then obtain an additive estimator which converges at the same rate as it does when k=1.


How to cite

top

Baraud, Yannick. "Model selection for regression on a random design." ESAIM: Probability and Statistics 6 (2010): 127-146. <http://eudml.org/doc/104283>.

@article{Baraud2010,
abstract = { We consider the problem of estimating an unknown regression function when the design is random with values in $\mathbb\{R\}^k$. Our estimation procedure is based on model selection and does not rely on any prior information on the target function. We start with a collection of linear functional spaces and build, on a data selected space among this collection, the least-squares estimator. We study the performance of an estimator which is obtained by modifying this least-squares estimator on a set of small probability. For the so-defined estimator, we establish nonasymptotic risk bounds that can be related to oracle inequalities. As a consequence of these, we show that our estimator possesses adaptive properties in the minimax sense over large families of Besov balls Bα,l,∞(R) with R>0, l ≥ 1 and α > α1 where α1 is a positive number satisfying 1/l - 1/2 ≤ α1 < 1/l. We also study the particular case where the regression function is additive and then obtain an additive estimator which converges at the same rate as it does when k=1.
},
author = {Baraud, Yannick},
journal = {ESAIM: Probability and Statistics},
keywords = {Nonparametric regression; least-squares estimators; penalized criteria; minimax rates; Besov spaces; model selection; adaptive estimation.; least-squares estimators; model selection; adaptive estimation},
language = {eng},
month = {3},
pages = {127-146},
publisher = {EDP Sciences},
title = {Model selection for regression on a random design},
url = {http://eudml.org/doc/104283},
volume = {6},
year = {2010},
}

TY - JOUR
AU - Baraud, Yannick
TI - Model selection for regression on a random design
JO - ESAIM: Probability and Statistics
DA - 2010/3//
PB - EDP Sciences
VL - 6
SP - 127
EP - 146
AB - We consider the problem of estimating an unknown regression function when the design is random with values in $\mathbb{R}^k$. Our estimation procedure is based on model selection and does not rely on any prior information on the target function. We start with a collection of linear functional spaces and build, on a data selected space among this collection, the least-squares estimator. We study the performance of an estimator which is obtained by modifying this least-squares estimator on a set of small probability. For the so-defined estimator, we establish nonasymptotic risk bounds that can be related to oracle inequalities. As a consequence of these, we show that our estimator possesses adaptive properties in the minimax sense over large families of Besov balls Bα,l,∞(R) with R>0, l ≥ 1 and α > α1 where α1 is a positive number satisfying 1/l - 1/2 ≤ α1 < 1/l. We also study the particular case where the regression function is additive and then obtain an additive estimator which converges at the same rate as it does when k=1.

LA - eng
KW - Nonparametric regression; least-squares estimators; penalized criteria; minimax rates; Besov spaces; model selection; adaptive estimation.; least-squares estimators; model selection; adaptive estimation
UR - http://eudml.org/doc/104283
ER -

References

top
  1. Y. Baraud, Model selection for regression on a fixed design. Probab. Theory Related Fields117 (2000) 467-493.  
  2. A. Barron, L. Birgé and P. Massart, Risk bounds for model selection via penalization. Probab. Theory Related Fields113 (1999) 301-413.  
  3. A.R. Barron and T.M. Cover, Minimum complexity density estimation. IEEE Trans. Inform. Theory37 (1991) 1738.  
  4. L. Birgé and P. Massart, An adaptive compression algorithm in Besov spaces. Constr. Approx.16 (2000) 1-36.  
  5. L. Birgé and P. Massart, Minimum contrast estimators on sieves: Exponential bounds and rates of convergence. Bernoulli4 (1998) 329-375.  
  6. L. Birgé and P. Massart, Gaussian model selection. JEMS3 (2001) 203-268.  
  7. L. Birgé and Massart, A generalized Cp criterion for Gaussian model selection, Technical Report. University Paris 6, PMA-647 (2001).  
  8. L. Birgé and Y. Rozenholc, How many bins should be put in a regular histogram, Technical Report. University Paris 6, PMA-721 (2002).  
  9. O. Catoni, Statistical learning theory and stochastic optimization, in École d'été de probabilités de Saint-Flour. Springer (2001).  
  10. A. Cohen, I. Daubechies and P. Vial, Wavelet and fast wavelet transform on an interval. Appl. Comp. Harmon. Anal.1 (1993) 54-81.  
  11. I. Daubechies, Ten lectures on wavelets. SIAM: Philadelphia (1992).  
  12. R.A. DeVore and G.G. Lorentz, Constructive approximation. Springer-Verlag, Berlin (1993).  
  13. D.L. Donoho and I.M. Johnstone, Ideal spatial adaptation via wavelet shrinkage. Biometrika81 (1994) 425-455.  
  14. D.L. Donoho and I.M. Johnstone, Minimax estimation via wavelet shrinkage. Ann. Statist.26 (1998) 879-921.  
  15. M. Kohler, Inequalities for uniform deviations of averages from expectations with applications to nonparametric regression. J. Statist. Plann. Inference89 (2000) 1-23.  
  16. M. Kohler, Nonparametric regression function estimation using interaction least square splines and complexity regularization. Metrika47 (1998) 147-163.  
  17. A.P. Korostelev and A.B. Tsybakov, Minimax theory of image reconstruction. Springer-Verlag, New York NY, Lecture Notes in Statis. (1993).  
  18. C.J. Stone, Additive regression and other nonparametric models. Ann. Statist.13 (1985) 689-705.  
  19. M. Wegkamp, Model selection in non-parametric regression, Preprint. Yale University (2000).  
  20. Y. Yang, Model selection for nonparametric regression. Statist. Sinica9 (1999) 475-499.  
  21. Y. Yang, Combining different procedures for adaptive regression. J. Multivariate Anal.74 (2000) 135-161.  
  22. Y. Yang and A. Barron, Information-Theoretic determination of minimax rates of convergence. Ann. Statist.27 (1999) 1564-1599.  

NotesEmbed ?

top

You must be logged in to post comments.

To embed these notes on your page include the following JavaScript code on your page where you want the notes to appear.

Only the controls for the widget will be shown in your chosen language. Notes will be shown in their authored language.

Tells the widget how many notes to show per page. You can cycle through additional notes using the next and previous controls.

    
                

                
Note: Best practice suggests putting the JavaScript code just before the closing </body> tag.