Mixture decompositions of exponential families using a decomposition of their sample spaces

Guido F. Montúfar

Kybernetika (2013)

  • Volume: 49, Issue: 1, page 23-39
  • ISSN: 0023-5954

Abstract

top
We study the problem of finding the smallest m such that every element of an exponential family can be written as a mixture of m elements of another exponential family. We propose an approach based on coverings and packings of the face lattice of the corresponding convex support polytopes and results from coding theory. We show that m = q N - 1 is the smallest number for which any distribution of N q -ary variables can be written as mixture of m independent q -ary variables. Furthermore, we show that any distribution of N binary variables is a mixture of m = 2 N - ( k + 1 ) ( 1 + 1 / ( 2 k - 1 ) ) elements of the k -interaction exponential family.

How to cite

top

Montúfar, Guido F.. "Mixture decompositions of exponential families using a decomposition of their sample spaces." Kybernetika 49.1 (2013): 23-39. <http://eudml.org/doc/252551>.

@article{Montúfar2013,
abstract = {We study the problem of finding the smallest $m$ such that every element of an exponential family can be written as a mixture of $m$ elements of another exponential family. We propose an approach based on coverings and packings of the face lattice of the corresponding convex support polytopes and results from coding theory. We show that $m=q^\{N-1\}$ is the smallest number for which any distribution of $N$$q$-ary variables can be written as mixture of $m$ independent $q$-ary variables. Furthermore, we show that any distribution of $N$ binary variables is a mixture of $m = 2^\{N-(k+1)\}(1+ 1/(2^k-1))$ elements of the $k$-interaction exponential family.},
author = {Montúfar, Guido F.},
journal = {Kybernetika},
keywords = {mixture model; non-negative tensor rank; perfect code; marginal polytope; mixture model; non-negative tensor rank; perfect code; marginal polytope},
language = {eng},
number = {1},
pages = {23-39},
publisher = {Institute of Information Theory and Automation AS CR},
title = {Mixture decompositions of exponential families using a decomposition of their sample spaces},
url = {http://eudml.org/doc/252551},
volume = {49},
year = {2013},
}

TY - JOUR
AU - Montúfar, Guido F.
TI - Mixture decompositions of exponential families using a decomposition of their sample spaces
JO - Kybernetika
PY - 2013
PB - Institute of Information Theory and Automation AS CR
VL - 49
IS - 1
SP - 23
EP - 39
AB - We study the problem of finding the smallest $m$ such that every element of an exponential family can be written as a mixture of $m$ elements of another exponential family. We propose an approach based on coverings and packings of the face lattice of the corresponding convex support polytopes and results from coding theory. We show that $m=q^{N-1}$ is the smallest number for which any distribution of $N$$q$-ary variables can be written as mixture of $m$ independent $q$-ary variables. Furthermore, we show that any distribution of $N$ binary variables is a mixture of $m = 2^{N-(k+1)}(1+ 1/(2^k-1))$ elements of the $k$-interaction exponential family.
LA - eng
KW - mixture model; non-negative tensor rank; perfect code; marginal polytope; mixture model; non-negative tensor rank; perfect code; marginal polytope
UR - http://eudml.org/doc/252551
ER -

References

top
  1. Amari, S., Information geometry on hierarchical decomposition of stochastic interactions., IEEE Trans. Inform. Theory 47 (1999), 1701-1711. 
  2. Amari, S., Nagaoka, H., Methods of information geometry, Vol. 191., Oxford University Press, 2000. Translations of mathematical monographs. MR1800071
  3. Ay, N., Knauf, A., Maximizing multi-information., Kybernetika 42 (2006), 517-538. Zbl1249.82011MR2283503
  4. Ay, N., Montúfar, G. F., Rauh, J., Selection criteria for neuromanifolds of stochastic dynamics., In: Advances in Cognitive Neurodynamics (III). Springer, 2011. 
  5. Bishop, C. M., Pattern Recognition and Machine Learning (Information Science and Statistics)., Springer-Verlag, New York 2006. MR2247587
  6. Bocci, C., Chiantini, L., On the identifiability of binary segre products., J. Algebraic Geom. 5 (2011). 
  7. Brown, L., Fundamentals of Statistical Exponential Families: With Applications in Statistical Decision Theory., Institute of Mathematical Statistics, Hayworth 1986. Zbl0685.62002MR0882001
  8. Catalisano, M. V., Geramita, A. V., Gimigliano, A., 10.1090/S1056-3911-10-00537-0, J. Algebraic Geom. 20 (2011), 295-327. MR2762993DOI10.1090/S1056-3911-10-00537-0
  9. Diaconis, P., 10.1007/BF00486116, Synthese 36 (1977), 271-281. Zbl0397.60005MR0517222DOI10.1007/BF00486116
  10. Efron, B., 10.1214/aos/1176344130, Ann. Statist. 6 (1978), 2, 362-376. Zbl0436.62027MR0471152DOI10.1214/aos/1176344130
  11. Cohen, S. L. G., Honkala, I., Lobstein, A., Covering Codes., Elsevier, 1997. Zbl0874.94001
  12. Gale, D., Neighborly and cyclic polytopes., In: Convexity: Proc. Seventh Symposium in Pure Mathematics of the American Mathematical Society 1961, pp. 225-233. Zbl0137.41801MR0152944
  13. Gawrilow, E., Joswig, M., Polymake: a framework for analyzing convex polytopes., In: Polytopes - Combinatorics and Computation (G. Kalai and G. M. Ziegler, eds.), Birkhäuser 2000, pp. 43-74. Zbl0960.68182MR1785292
  14. Geiger, D., Meek, C., Sturmfels, B., 10.1214/009053606000000263, Ann. Statist. 34 (2006), 1463-1492. Zbl1104.60007MR2278364DOI10.1214/009053606000000263
  15. Gilbert, E., A comparison of signalling alphabets., Bell System Techn. J. 31 (1052), 504-522. 
  16. Gilula, Z., 10.1093/biomet/66.2.339, Biometrika 66 (1979), 2, 339-344. Zbl0411.62003MR0548203DOI10.1093/biomet/66.2.339
  17. Grünbaum, B., Convex Polytopes. Second edition., Springer-Verlag, New York 2003. MR1976856
  18. Henk, M., Richter-Gebert, J., Ziegler, G. M., Basic Properties of Convex Polytopes., CRC Press, Boca Raton 1997. Zbl0911.52007MR1730169
  19. Hoşten, S., Sullivant, S., 10.1006/jcta.2002.3301, J. Combin. Theory Ser. A 100 (2002), 2, 277-301. Zbl1044.62065MR1940337DOI10.1006/jcta.2002.3301
  20. Kahle, T., Neighborliness of marginal polytopes., Contrib. Algebra Geometry 51 (2010), 45-56. Zbl1238.60012MR2650476
  21. Kahle, T., Ay, N., Support sets of distributions with given interaction structure., In: Proc. WUPES'06, 2006. 
  22. Kahle, T., Wenzel, W., Ay, N., Hierarchical models, marginal polytopes, and linear codes., Kybernetika 45 (2009), 189-208. Zbl1167.94340MR2518148
  23. Kalai, G., Some aspects of the combinatorial theory of convex polytopes., 1993. Zbl0804.52006MR1322063
  24. Kingman, J. F. C., 10.1214/aop/1176995566, Ann. Probab. 6 (1978), 2, 183-197. Zbl0374.60064MR0494344DOI10.1214/aop/1176995566
  25. Lindsay, B. G., Mixture models: theory, geometry, and applications., NSF-CBMS Regional Conference Series in Probability and Statistics. Institute of Mathematical Statistics, 1995. Zbl1163.62326
  26. McLachlan, G., Peel, D., Finite Mixture Models., Wiley Series in Probability and Statistics: Applied Probability and Statistics. Wiley, 2000. Zbl0963.62061MR1789474
  27. Montúfar, G. F., Ay, N., 10.1162/NECO_a_00113, Neural Comput. 23 (2011), 5, 1306-1319. MR2814846DOI10.1162/NECO_a_00113
  28. Montúfar, G. F., Rauh, J., Ay, N., Expressive power and approximation errors of restricted Boltzmann machines., In: Advances in Neural Information Processing Systems 24 (J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger, eds.), MIT Press, 2011, pp. 415-423. 
  29. Rauh, J., Finding the Maximizers of the Information Divergence from an Exponential Family., Ph. D. Thesis, Universität Leipzig, 2011. MR2817016
  30. Rauh, J., Kahle, T., Ay, N., 10.1016/j.ijar.2011.01.013, Internat. J. Approximate Reasoning 52 (2011), 5, 613-626. MR2787021DOI10.1016/j.ijar.2011.01.013
  31. Settimi, R., Smith, J. Q., On the geometry of Bayesian graphical models with hidden variables., In: Proc. Fourteenth conference on Uncertainty in artificial intelligence, UAI'98, Morgan Kaufmann Publishers 1998, pp. 472-479. 
  32. Shemer., I., 10.1007/BF02761235, Israel J. Math. 43 (1982), 291-311. Zbl1223.52005MR0693351DOI10.1007/BF02761235
  33. Titterington, D., Smith, A. F. M., Makov, U. E., Statistical Analysis of Finite Mixture Distributions., John Wiley and Sons, 1985. Zbl0646.62013MR0838090
  34. Varshamov, R., Estimate of the number of signals in error correcting codes., Dokl. Akad. Nauk SSSR 117 (1957), 739-741. 

NotesEmbed ?

top

You must be logged in to post comments.

To embed these notes on your page include the following JavaScript code on your page where you want the notes to appear.

Only the controls for the widget will be shown in your chosen language. Notes will be shown in their authored language.

Tells the widget how many notes to show per page. You can cycle through additional notes using the next and previous controls.

    
                

                
Note: Best practice suggests putting the JavaScript code just before the closing </body> tag.