Risk bounds for mixture density estimation

Alexander Rakhlin; Dmitry Panchenko; Sayan Mukherjee

ESAIM: Probability and Statistics (2005)

  • Volume: 9, page 220-229
  • ISSN: 1292-8100

Abstract

top
In this paper we focus on the problem of estimating a bounded density using a finite combination of densities from a given class. We consider the Maximum Likelihood Estimator (MLE) and the greedy procedure described by Li and Barron (1999) under the additional assumption of boundedness of densities. We prove an O ( 1 n ) bound on the estimation error which does not depend on the number of densities in the estimated combination. Under the boundedness assumption, this improves the bound of Li and Barron by removing the log n factor and also generalizes it to the base classes with converging Dudley integral.

How to cite

top

Rakhlin, Alexander, Panchenko, Dmitry, and Mukherjee, Sayan. "Risk bounds for mixture density estimation." ESAIM: Probability and Statistics 9 (2005): 220-229. <http://eudml.org/doc/246104>.

@article{Rakhlin2005,
abstract = {In this paper we focus on the problem of estimating a bounded density using a finite combination of densities from a given class. We consider the Maximum Likelihood Estimator (MLE) and the greedy procedure described by Li and Barron (1999) under the additional assumption of boundedness of densities. We prove an $O(\frac\{1\}\{\sqrt\{n\}\})$ bound on the estimation error which does not depend on the number of densities in the estimated combination. Under the boundedness assumption, this improves the bound of Li and Barron by removing the $\log n$ factor and also generalizes it to the base classes with converging Dudley integral.},
author = {Rakhlin, Alexander, Panchenko, Dmitry, Mukherjee, Sayan},
journal = {ESAIM: Probability and Statistics},
keywords = {mixture density estimation; maximum likelihood; Rademacher processes; Mixture density estimation},
language = {eng},
pages = {220-229},
publisher = {EDP-Sciences},
title = {Risk bounds for mixture density estimation},
url = {http://eudml.org/doc/246104},
volume = {9},
year = {2005},
}

TY - JOUR
AU - Rakhlin, Alexander
AU - Panchenko, Dmitry
AU - Mukherjee, Sayan
TI - Risk bounds for mixture density estimation
JO - ESAIM: Probability and Statistics
PY - 2005
PB - EDP-Sciences
VL - 9
SP - 220
EP - 229
AB - In this paper we focus on the problem of estimating a bounded density using a finite combination of densities from a given class. We consider the Maximum Likelihood Estimator (MLE) and the greedy procedure described by Li and Barron (1999) under the additional assumption of boundedness of densities. We prove an $O(\frac{1}{\sqrt{n}})$ bound on the estimation error which does not depend on the number of densities in the estimated combination. Under the boundedness assumption, this improves the bound of Li and Barron by removing the $\log n$ factor and also generalizes it to the base classes with converging Dudley integral.
LA - eng
KW - mixture density estimation; maximum likelihood; Rademacher processes; Mixture density estimation
UR - http://eudml.org/doc/246104
ER -

References

top
  1. [1] A.R. Barron, Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans. Inform. Theory 39 (1993) 930–945. Zbl0818.68126
  2. [2] A.R. Barron, Approximation and estimation bounds for artificial neural networks. Machine Learning 14 (1994) 115–133. Zbl0818.68127
  3. [3] L. Birgé and P. Massart, Rates of convergence for minimum contrast estimators. Probab. Theory Related Fields 97 (1993) 113–150. Zbl0805.62037
  4. [4] R.M. Dudley, Uniform Central Limit Theorems. Cambridge University Press (1999). Zbl0951.60033MR1720712
  5. [5] L.K. Jones, A simple lemma on greedy approximation in Hilbert space and convergence rates for Projection Pursuit Regression and neural network training. Ann. Stat. 20 (1992) 608–613. Zbl0746.62060
  6. [6] M. Ledoux and M. Talagrand, Probability in Banach Spaces. Springer-Verlag, New York (1991). Zbl0748.60004MR1102015
  7. [7] J. Li and A. Barron, Mixture density estimation, in Advances in Neural information processings systems 12, S.A. Solla, T.K. Leen and K.-R. Muller Ed. San Mateo, CA. Morgan Kaufmann Publishers (1999). 
  8. [8] J. Li, Estimation of Mixture Models. Ph.D. Thesis, The Department of Statistics. Yale University (1999). 
  9. [9] C. McDiarmid, On the method of bounded differences. Surveys in Combinatorics (1989) 148–188. Zbl0712.05012
  10. [10] S. Mendelson, On the size of convex hulls of small sets. J. Machine Learning Research 2 (2001) 1–18. Zbl1008.68107
  11. [11] P. Niyogi and F. Girosi, Generalization bounds for function approximation from scattered noisy data. Adv. Comput. Math. 10 (1999) 51–80. Zbl1053.65506
  12. [12] S.A. van de Geer, Rates of convergence for the maximum likelihood estimator in mixture models. Nonparametric Statistics 6 (1996) 293–310. Zbl0872.62039
  13. [13] S.A. van de Geer, Empirical Processes in M-Estimation. Cambridge University Press (2000). Zbl1179.62073
  14. [14] A.W. van der Vaart and J.A. Wellner, Weak Convergence and Empirical Processes with Applications to Statistics. Springer-Verlag, New York (1996). Zbl0862.60002MR1385671
  15. [15] W.H. Wong and X. Shen, Probability inequalities for likelihood ratios and convergence rates for sieve mles. Ann. Stat. 23 (1995) 339–362. Zbl0829.62002

NotesEmbed ?

top

You must be logged in to post comments.

To embed these notes on your page include the following JavaScript code on your page where you want the notes to appear.

Only the controls for the widget will be shown in your chosen language. Notes will be shown in their authored language.

Tells the widget how many notes to show per page. You can cycle through additional notes using the next and previous controls.

    
                

                
Note: Best practice suggests putting the JavaScript code just before the closing </body> tag.