Model selection and estimation of a component in additive regression

Xavier Gendre

ESAIM: Probability and Statistics (2014)

  • Volume: 18, page 77-116
  • ISSN: 1292-8100

Abstract

top
Let Y ∈ ℝn be a random vector with mean s and covariance matrix σ2PntPn where Pn is some known n × n-matrix. We construct a statistical procedure to estimate s as well as under moment condition on Y or Gaussian hypothesis. Both cases are developed for known or unknown σ2. Our approach is free from any prior assumption on s and is based on non-asymptotic model selection methods. Given some linear spaces collection {Sm, m ∈ ℳ}, we consider, for any m ∈ ℳ, the least-squares estimator ŝm of s in Sm. Considering a penalty function that is not linear in the dimensions of the Sm’s, we select some m̂ ∈ ℳ in order to get an estimator ŝm̂ with a quadratic risk as close as possible to the minimal one among the risks of the ŝm’s. Non-asymptotic oracle-type inequalities and minimax convergence rates are proved for ŝm̂. A special attention is given to the estimation of a non-parametric component in additive models. Finally, we carry out a simulation study in order to illustrate the performances of our estimators in practice.

How to cite

top

Gendre, Xavier. "Model selection and estimation of a component in additive regression." ESAIM: Probability and Statistics 18 (2014): 77-116. <http://eudml.org/doc/274383>.

@article{Gendre2014,
abstract = {Let Y ∈ ℝn be a random vector with mean s and covariance matrix σ2PntPn where Pn is some known n × n-matrix. We construct a statistical procedure to estimate s as well as under moment condition on Y or Gaussian hypothesis. Both cases are developed for known or unknown σ2. Our approach is free from any prior assumption on s and is based on non-asymptotic model selection methods. Given some linear spaces collection \{Sm, m ∈ ℳ\}, we consider, for any m ∈ ℳ, the least-squares estimator ŝm of s in Sm. Considering a penalty function that is not linear in the dimensions of the Sm’s, we select some m̂ ∈ ℳ in order to get an estimator ŝm̂ with a quadratic risk as close as possible to the minimal one among the risks of the ŝm’s. Non-asymptotic oracle-type inequalities and minimax convergence rates are proved for ŝm̂. A special attention is given to the estimation of a non-parametric component in additive models. Finally, we carry out a simulation study in order to illustrate the performances of our estimators in practice.},
author = {Gendre, Xavier},
journal = {ESAIM: Probability and Statistics},
keywords = {model selection; nonparametric regression; penalized criterion; oracle inequality; correlated data; additive regression; minimax rate},
language = {eng},
pages = {77-116},
publisher = {EDP-Sciences},
title = {Model selection and estimation of a component in additive regression},
url = {http://eudml.org/doc/274383},
volume = {18},
year = {2014},
}

TY - JOUR
AU - Gendre, Xavier
TI - Model selection and estimation of a component in additive regression
JO - ESAIM: Probability and Statistics
PY - 2014
PB - EDP-Sciences
VL - 18
SP - 77
EP - 116
AB - Let Y ∈ ℝn be a random vector with mean s and covariance matrix σ2PntPn where Pn is some known n × n-matrix. We construct a statistical procedure to estimate s as well as under moment condition on Y or Gaussian hypothesis. Both cases are developed for known or unknown σ2. Our approach is free from any prior assumption on s and is based on non-asymptotic model selection methods. Given some linear spaces collection {Sm, m ∈ ℳ}, we consider, for any m ∈ ℳ, the least-squares estimator ŝm of s in Sm. Considering a penalty function that is not linear in the dimensions of the Sm’s, we select some m̂ ∈ ℳ in order to get an estimator ŝm̂ with a quadratic risk as close as possible to the minimal one among the risks of the ŝm’s. Non-asymptotic oracle-type inequalities and minimax convergence rates are proved for ŝm̂. A special attention is given to the estimation of a non-parametric component in additive models. Finally, we carry out a simulation study in order to illustrate the performances of our estimators in practice.
LA - eng
KW - model selection; nonparametric regression; penalized criterion; oracle inequality; correlated data; additive regression; minimax rate
UR - http://eudml.org/doc/274383
ER -

References

top
  1. [1] H. Akaike, Statistical predictor identification. Ann. Inst. Stat. Math.22 (1970) 203–217. Zbl0259.62076MR286233
  2. [2] S. Arlot, Choosing a penalty for model selection in heteroscedastic regression. Preprint arXiv:0812.3141v2 (2010). 
  3. [3] S. Arlot and P. Massart, Data-driven calibration of penalties for least-squares regression. J. Machine Learn. Research10 (2009) 245–279. 
  4. [4] Y. Baraud, Model selection for regression on a fixed design. Probab. Theory Related Fields117 (2000) 467–493. Zbl0997.62027MR1777129
  5. [5] Y. Baraud, Model selection for regression on a random design. ESAIM: Probab. Statist. 6 (2002) 127–146. Zbl1059.62038MR1918295
  6. [6] Y. Baraud, F. Comte and G. Viennet, Adaptive estimation in autoregression or β-mixing regression via model selection. Ann. Stat.29 (2001) 839–875. Zbl1012.62034MR1865343
  7. [7] L. Birgé and P. Massart, From model selection to adaptive estimation. Festschrift for Lucien Lecam: Research Papers in Probab. Stat. (1997) 55–87. Zbl0920.62042MR1462939
  8. [8] L. Birgé and P. Massart, Minimum contrast estimators on sieves: exponential bounds and rates of convergence. Bernoulli4 (1998) 329–375. Zbl0954.62033MR1653272
  9. [9] L. Birgé and P. Massart, An adaptive compression algorithm in Besov spaces. Constr. Approx.16 (2000) 1–36. Zbl1004.41006MR1848840
  10. [10] L. Birgé and P. Massart. Gaussian model selection. J. Europ. Math. Soc.3 (2001) 203–268. Zbl1037.62001MR1848946
  11. [11] L. Birgé and P. Massart, Minimal penalties for gaussian model selection. Probab. Theory Related Fields138 (2007) 33–73. Zbl1112.62082MR2288064
  12. [12] L. Breiman and J.H. Friedman, Estimating optimal transformations for multiple regression and correlations (with discussion). J. Amer. Stat. Assoc.80 (1985) 580–619. Zbl0594.62044MR803258
  13. [13] E. Brunel and F. Comte, Adaptive nonparametric regression estimation in presence of right censoring. Math. Methods Stat.15 (2006) 233–255. MR2278288
  14. [14] E. Brunel and F. Comte, Model selection for additive regression models in the presence of censoring, chapt. 1 in “Mathematical Methods in Survival Analysis, Reliability and Quality of Life”, Wiley (2008) 17–31. MR2730701
  15. [15] A. Buja, T.J. Hastie and R.J. Tibshirani, Linear smoothers and additive models (with discussion). Ann. Stat.17 (1989) 453–555. Zbl0689.62029MR994249
  16. [16] F. Comte and Y. Rozenholc, Adaptive estimation of mean and volatility functions in (auto-)regressive models. Stoch. Process. Appl.97 (2002) 111–145. Zbl1064.62046MR1870963
  17. [17] X. Gendre, Simultaneous estimation of the mean and the variance in heteroscedastic gaussian regression. Electron. J. Stat.2 (2008) 1345–1372. Zbl1320.62092MR2471290
  18. [18] W. Härdle, M. Müller, S. Sperlich and A. Werwatz. Nonparametric and Semiparametric Models. Springer (2004). Zbl1059.62032MR2061786
  19. [19] T.J. Hastie and R.J. Tibshirani, Generalized additive models. Chapman and Hall (1990). Zbl0747.62061MR1082147
  20. [20] R.A. Horn and C.R. Johnson, Matrix analysis. Cambridge University Press (1990). Zbl0704.15002MR1084815
  21. [21] B. Laurent, J.M. Loubes and C. Marteau, Testing inverse problems: a direct or an indirect problem? J. Stat. Plann. Inference141 (2011) 1849–1861. Zbl05862619MR2763215
  22. [22] B. Laurent and P. Massart, Adaptive estimation of a quadratic functional by model selection. Ann. Stat.28 (2000) 1302–1338. Zbl1105.62328MR1805785
  23. [23] W. Leontief, Introduction to a theory of the internal structure of functional relationships. Econometrica15 (1947) 361–373. Zbl0030.17105MR25144
  24. [24] O. Linton and J.P. Nielsen, A kernel method of estimating structured nonparametric regression based on marginal integration. Biometrika82 (1995) 93–101. Zbl0823.62036MR1332841
  25. [25] C.L. Mallows, Some comments on cp. Technometrics 15 (1973) 661–675,. Zbl0269.62061
  26. [26] E. Mammen, O. Linton and J.P. Nielsen, The existence and asymptotic properties of a backfitting projection algorithm under weak conditions. Ann. Stat.27 (1999) 1443–1490. Zbl0986.62028MR1742496
  27. [27] P. Massart, Concentration inequalities and model selection, in vol. 1896 of Lect. Notes Math. Lectures from the 33rd Summer School on Probability Theory held in Saint-Flour, July 6-23 (2003). Springer, Berlin (2007). Zbl1170.60006MR2319879
  28. [28] A.D.R. McQuarrie and C.L. Tsai, Regression and times series model selection. River Edge, NJ (1998). Zbl0907.62095MR1641582
  29. [29] L. Meier, S. van de Geer and P. Bühlmann, High-dimensional additive modeling. Ann. Stat.37 (2009) 3779–3821. Zbl05644256MR2572443
  30. [30] J. Opsomer and D. Ruppert, Fitting a bivariate additive model by local polynomial regression. Ann. Stat.25 (1997) 186–211. Zbl0869.62026MR1429922
  31. [31] V.V. Petrov, Limit theorems of probability theory: sequences of independent random variables. Oxford Studies Probab. 4 (1995). Zbl0826.60001MR1353441
  32. [32] P.D. Ravikumar, H. Liu, J.D. Lafferty and L.A. Wasserman, Sparse additive models. J. Royal Statist. Soc.71 (2009) 1009–1030. MR2750255
  33. [33] S. Robin, F. Rodolphe and S. Schbath, DNA, Words and Models. Cambridge University Press (2005). Zbl1185.92047MR2214200
  34. [34] D. Ruppert and M.P. Wand, Multivariate locally weighted least squares regression. Ann. Stat.22 (1994) 1346–1370. Zbl0821.62020MR1311979
  35. [35] H. Scheffé, The analysis of variance. Wiley-Interscience (1959). Zbl0998.62500
  36. [36] E. Severance-Lossin and S. Sperlich, Estimation of derivatives for additive separable models. Statististics33 (1999) 241–265. Zbl0938.62044MR1750253
  37. [37] C.J. Stone, Additive regression and other nonparametric models. Ann. Stat.14 (1985) 590–606. Zbl0603.62050MR790566
  38. [38] D. Tjøstheim and B. Auestad, Nonparametric identification of nonlinear time series: Selecting significant lags. J. Amer. Stat. Assoc.89 (1994) 1410–1430. Zbl0813.62037MR1310231
  39. [39] B. von Bahr and C.G. Esseen, Inequalities for the rth absolute moment of a sum of random variables 1 ≤ r ≤ 2 . Ann. Math. Stat. 36 (1965) 299–303. Zbl0134.36902MR170407

NotesEmbed ?

top

You must be logged in to post comments.

To embed these notes on your page include the following JavaScript code on your page where you want the notes to appear.

Only the controls for the widget will be shown in your chosen language. Notes will be shown in their authored language.

Tells the widget how many notes to show per page. You can cycle through additional notes using the next and previous controls.

    
                

                
Note: Best practice suggests putting the JavaScript code just before the closing </body> tag.