Optimally approximating exponential families

Johannes Rauh

Kybernetika (2013)

  • Volume: 49, Issue: 2, page 199-215
  • ISSN: 0023-5954

Abstract

top
This article studies exponential families ℰ on finite sets such that the information divergence D ( P ∥ ℰ ) of an arbitrary probability distribution from ℰ is bounded by some constant D > 0 . A particular class of low-dimensional exponential families that have low values of D can be obtained from partitions of the state space. The main results concern optimality properties of these partition exponential families. The case where D = log ( 2 ) is studied in detail. This case is special, because if D < log ( 2 ) , then ℰ contains all probability measures with full support.

How to cite

top

Rauh, Johannes. "Optimally approximating exponential families." Kybernetika 49.2 (2013): 199-215. <http://eudml.org/doc/260703>.

@article{Rauh2013,
abstract = {This article studies exponential families $\mathcal \{E\}$ on finite sets such that the information divergence $D(P\Vert \mathcal \{E\})$ of an arbitrary probability distribution from $\mathcal \{E\}$ is bounded by some constant $D>0$. A particular class of low-dimensional exponential families that have low values of $D$ can be obtained from partitions of the state space. The main results concern optimality properties of these partition exponential families. The case where $D=\log (2)$ is studied in detail. This case is special, because if $D<\log (2)$, then $\mathcal \{E\}$ contains all probability measures with full support.},
author = {Rauh, Johannes},
journal = {Kybernetika},
keywords = {exponential family; information divergence; exponential family; information divergence},
language = {eng},
number = {2},
pages = {199-215},
publisher = {Institute of Information Theory and Automation AS CR},
title = {Optimally approximating exponential families},
url = {http://eudml.org/doc/260703},
volume = {49},
year = {2013},
}

TY - JOUR
AU - Rauh, Johannes
TI - Optimally approximating exponential families
JO - Kybernetika
PY - 2013
PB - Institute of Information Theory and Automation AS CR
VL - 49
IS - 2
SP - 199
EP - 215
AB - This article studies exponential families $\mathcal {E}$ on finite sets such that the information divergence $D(P\Vert \mathcal {E})$ of an arbitrary probability distribution from $\mathcal {E}$ is bounded by some constant $D>0$. A particular class of low-dimensional exponential families that have low values of $D$ can be obtained from partitions of the state space. The main results concern optimality properties of these partition exponential families. The case where $D=\log (2)$ is studied in detail. This case is special, because if $D<\log (2)$, then $\mathcal {E}$ contains all probability measures with full support.
LA - eng
KW - exponential family; information divergence; exponential family; information divergence
UR - http://eudml.org/doc/260703
ER -

References

top
  1. Ay, N., 10.1214/aop/1020107773, Ann. Probab. 30 (2002), 416-436. Zbl1010.62007MR1894113DOI10.1214/aop/1020107773
  2. Ay, N., 10.1162/089976602760805368, Neural Computat. 14 (2002), 2959-2980. Zbl1079.68582DOI10.1162/089976602760805368
  3. Brown, L., Fundamentals of Statistical Exponential Families: With Applications in Statistical Decision Theory., Institute of Mathematical Statistics, Hayworth 1986. Zbl0685.62002MR0882001
  4. Cover, T., Thomas, J., Elements of Information Theory. First edition., Wiley, 1991. MR1122806
  5. Csiszár, I., Shields, P., Information Theory and Statistics: A Tutorial. First edition., Foundations and Trends in Communications and Information Theory. Now Publishers, 2004. 
  6. Csiszár, I., Matúš, F., 10.1007/s00440-007-0084-z, Probab. Theory Rel. Fields 141 (2008), 213-246. MR2372970DOI10.1007/s00440-007-0084-z
  7. Pietra, S. Della, Pietra, V. Della, Lafferty, J., 10.1109/34.588021, IEEE Trans. Pattern Analysis and Machine Intelligence 19 (1997), 380-393. DOI10.1109/34.588021
  8. Drton, M., Sturmfels, B., Sullivant, S., Lectures on algebraic statistics., In: Oberwolfach Seminars 39, Birkhäuser, Basel 2009. Zbl1166.13001MR2723140
  9. Geiger, D., Meek, C., Sturmfels, B., 10.1214/009053606000000263, Ann. Statist. 34 (2006), 5, 1463-1492. Zbl1104.60007MR2278364DOI10.1214/009053606000000263
  10. Jaynes, E. T., 10.1103/PhysRev.106.620, Phys. Rev. 106 (1957), 4, 620-630. Zbl0084.43701MR0087305DOI10.1103/PhysRev.106.620
  11. Juríček, J., Maximization of information divergence from multinomial distributions., Acta Univ. Carolin. 52 (2011), 1, 27-35. MR2808291
  12. Lauritzen, S. L., Graphical Models. First edition., Oxford Statistical Science Series, Oxford University Press, 1996. MR1419991
  13. Linsker, R., 10.1109/2.36, IEEE Computer 21 (1988), 105-117. DOI10.1109/2.36
  14. Matúš, F., Ay, N., On maximization of the information divergence from an exponential family., In: Proc. WUPES'03, University of Economics, Prague 2003, pp. 199-204. 
  15. Matúš, F., Rauh, J., Maximization of the information divergence from an exponential family and criticality., In: 2011 IEEE International Symposium on Information Theory Proceedings (ISIT2011), 2011. 
  16. Montúfar, G., Rauh, J., Ay, N., Expressive power and approximation errors of Restricted Boltzmann Machines., In: NIPS, 2011. 
  17. Oxley, J., Matroid Theory. First edition., Oxford University Press, New York 1992. MR1207587
  18. Rauh, J., Finding the Maximizers of the Information Divergence from an Exponential Family., Ph.D. Dissertation, Universität Leipzig, 2011. MR2817016
  19. Rauh, J., 10.1109/TIT.2011.2136230, IEEE Trans. Inform. Theory 57 (2011), 6, 3236-3247. MR2817016DOI10.1109/TIT.2011.2136230
  20. Rauh, J., Kahle, T., Ay, N., 10.1016/j.ijar.2011.01.013, Internat. J. Approx. Reasoning 52 (2011), 5, 613-626. MR2787021DOI10.1016/j.ijar.2011.01.013
  21. Zhu, S. C., Wu, Y. N., Mumford, D., 10.1162/neco.1997.9.8.1627, Neural Computation 9 (1997), 1627-1660. DOI10.1162/neco.1997.9.8.1627

NotesEmbed ?

top

You must be logged in to post comments.

To embed these notes on your page include the following JavaScript code on your page where you want the notes to appear.

Only the controls for the widget will be shown in your chosen language. Notes will be shown in their authored language.

Tells the widget how many notes to show per page. You can cycle through additional notes using the next and previous controls.

    
                

                
Note: Best practice suggests putting the JavaScript code just before the closing </body> tag.