Maximizing multi–information

Nihat Ay; Andreas Knauf

Kybernetika (2006)

  • Volume: 42, Issue: 5, page 517-538
  • ISSN: 0023-5954

Abstract

top
Stochastic interdependence of a probability distribution on a product space is measured by its Kullback–Leibler distance from the exponential family of product distributions (called multi-information). Here we investigate low-dimensional exponential families that contain the maximizers of stochastic interdependence in their closure. Based on a detailed description of the structure of probability distributions with globally maximal multi-information we obtain our main result: The exponential family of pure pair-interactions contains all global maximizers of the multi-information in its closure.

How to cite

top

Ay, Nihat, and Knauf, Andreas. "Maximizing multi–information." Kybernetika 42.5 (2006): 517-538. <http://eudml.org/doc/33822>.

@article{Ay2006,
abstract = {Stochastic interdependence of a probability distribution on a product space is measured by its Kullback–Leibler distance from the exponential family of product distributions (called multi-information). Here we investigate low-dimensional exponential families that contain the maximizers of stochastic interdependence in their closure. Based on a detailed description of the structure of probability distributions with globally maximal multi-information we obtain our main result: The exponential family of pure pair-interactions contains all global maximizers of the multi-information in its closure.},
author = {Ay, Nihat, Knauf, Andreas},
journal = {Kybernetika},
keywords = {multi-information; exponential family; relative entropy; pair- interaction; infomax principle; Boltzmann machine; neural networks; multi-information; relative entropy; pair-interaction; infomax principle; Boltzmann machine; neural networks},
language = {eng},
number = {5},
pages = {517-538},
publisher = {Institute of Information Theory and Automation AS CR},
title = {Maximizing multi–information},
url = {http://eudml.org/doc/33822},
volume = {42},
year = {2006},
}

TY - JOUR
AU - Ay, Nihat
AU - Knauf, Andreas
TI - Maximizing multi–information
JO - Kybernetika
PY - 2006
PB - Institute of Information Theory and Automation AS CR
VL - 42
IS - 5
SP - 517
EP - 538
AB - Stochastic interdependence of a probability distribution on a product space is measured by its Kullback–Leibler distance from the exponential family of product distributions (called multi-information). Here we investigate low-dimensional exponential families that contain the maximizers of stochastic interdependence in their closure. Based on a detailed description of the structure of probability distributions with globally maximal multi-information we obtain our main result: The exponential family of pure pair-interactions contains all global maximizers of the multi-information in its closure.
LA - eng
KW - multi-information; exponential family; relative entropy; pair- interaction; infomax principle; Boltzmann machine; neural networks; multi-information; relative entropy; pair-interaction; infomax principle; Boltzmann machine; neural networks
UR - http://eudml.org/doc/33822
ER -

References

top
  1. Aarts E., Korst J., Simulated Annealing and Boltzmann Machines, Wiley, New York 1989 Zbl0674.90059MR0983115
  2. Ackley D. H., Hinton G. E., Sejnowski T. J., 10.1207/s15516709cog0901_7, Cognitive Science 9 (1985), 147–169 (1985) DOI10.1207/s15516709cog0901_7
  3. Aigner M., Combinatorial Theory, Classics in Mathematics, Springer–Verlag, Berlin 1997 MR1434477
  4. Amari S., 10.1109/18.930911, IEEE Trans. Inform. Theory 47 (2001), 1701–1711 Zbl0997.94009MR1842511DOI10.1109/18.930911
  5. Amari S., Kurata, K., Nagaoka H., 10.1109/72.125867, IEEE Trans. Neural Networks 3 (1992), 2, 260–271 (1992) DOI10.1109/72.125867
  6. Ay N., 10.1214/aop/1020107773, Ann. Probab. 30 (2002), 416–436 Zbl1010.62007MR1894113DOI10.1214/aop/1020107773
  7. Ay N., 10.1162/089976602760805368, Neural Computation 14 (2002), 2959–2980 Zbl1079.68582DOI10.1162/089976602760805368
  8. Linsker R., 10.1109/2.36, IEEE Computer 21 (1988), 105–117 (1988) DOI10.1109/2.36
  9. Matúš F., Ay N., On maximization of the information divergence from an exponential family, In: Proc. WUPES’03 (J. Vejnarová, ed.), University of Economics, Prague 2003, pp. 199–204 
  10. Shannon C. E., 10.1002/j.1538-7305.1948.tb01338.x, Bell System Tech. J. 27 (1948), 379–423, 623–656 (1948) Zbl1154.94303MR0026286DOI10.1002/j.1538-7305.1948.tb01338.x
  11. Tononi G., Sporns, O., Edelman G. M., 10.1073/pnas.91.11.5033, Proc. Nat. Acad. Sci. U. S. A. 91 (1994), 5033–5037 (1994) DOI10.1073/pnas.91.11.5033

Citations in EuDML Documents

top
  1. František Matúš, Optimality conditions for maximizers of the information divergence from an exponential family
  2. Thomas Kahle, Walter Wenzel, Nihat Ay, Hierarchical models, marginal polytopes, and linear codes
  3. Thomas Merkh, Guido F. Montúfar, Factorized mutual information maximization
  4. Guido F. Montúfar, Mixture decompositions of exponential families using a decomposition of their sample spaces
  5. Johannes Rauh, František Matúš, Maximizing the Bregman divergence from a Bregman family

NotesEmbed ?

top

You must be logged in to post comments.

To embed these notes on your page include the following JavaScript code on your page where you want the notes to appear.

Only the controls for the widget will be shown in your chosen language. Notes will be shown in their authored language.

Tells the widget how many notes to show per page. You can cycle through additional notes using the next and previous controls.

    
                

                
Note: Best practice suggests putting the JavaScript code just before the closing </body> tag.