Mixture of experts architectures for neural networks as a special case of conditional expectation formula

Jiří Grim

Kybernetika (1998)

  • Volume: 34, Issue: 4, page [417]-422
  • ISSN: 0023-5954

Abstract

top
Recently a new interesting architecture of neural networks called “mixture of experts” has been proposed as a tool of real multivariate approximation or prediction. We show that the underlying problem is closely related to approximating the joint probability density of involved variables by finite mixture. Particularly, assuming normal mixtures, we can explicitly write the conditional expectation formula which can be interpreted as a mixture-of- experts network. In this way the related optimization problem can be reduced to standard estimation of normal mixtures by means of EM algorithm. The resulting prediction is optimal in the sense of minimum dispersion if the assumed mixture model is true. It is shown that some of the recently published results can be obtained by specifying the normal components of mixtures in a special form.

How to cite

top

Grim, Jiří. "Mixture of experts architectures for neural networks as a special case of conditional expectation formula." Kybernetika 34.4 (1998): [417]-422. <http://eudml.org/doc/33371>.

@article{Grim1998,
abstract = {Recently a new interesting architecture of neural networks called “mixture of experts” has been proposed as a tool of real multivariate approximation or prediction. We show that the underlying problem is closely related to approximating the joint probability density of involved variables by finite mixture. Particularly, assuming normal mixtures, we can explicitly write the conditional expectation formula which can be interpreted as a mixture-of- experts network. In this way the related optimization problem can be reduced to standard estimation of normal mixtures by means of EM algorithm. The resulting prediction is optimal in the sense of minimum dispersion if the assumed mixture model is true. It is shown that some of the recently published results can be obtained by specifying the normal components of mixtures in a special form.},
author = {Grim, Jiří},
journal = {Kybernetika},
keywords = {neural networks; mixtures; multivariate approximation; prediction; neural networks; mixtures; multivariate approximation; prediction},
language = {eng},
number = {4},
pages = {[417]-422},
publisher = {Institute of Information Theory and Automation AS CR},
title = {Mixture of experts architectures for neural networks as a special case of conditional expectation formula},
url = {http://eudml.org/doc/33371},
volume = {34},
year = {1998},
}

TY - JOUR
AU - Grim, Jiří
TI - Mixture of experts architectures for neural networks as a special case of conditional expectation formula
JO - Kybernetika
PY - 1998
PB - Institute of Information Theory and Automation AS CR
VL - 34
IS - 4
SP - [417]
EP - 422
AB - Recently a new interesting architecture of neural networks called “mixture of experts” has been proposed as a tool of real multivariate approximation or prediction. We show that the underlying problem is closely related to approximating the joint probability density of involved variables by finite mixture. Particularly, assuming normal mixtures, we can explicitly write the conditional expectation formula which can be interpreted as a mixture-of- experts network. In this way the related optimization problem can be reduced to standard estimation of normal mixtures by means of EM algorithm. The resulting prediction is optimal in the sense of minimum dispersion if the assumed mixture model is true. It is shown that some of the recently published results can be obtained by specifying the normal components of mixtures in a special form.
LA - eng
KW - neural networks; mixtures; multivariate approximation; prediction; neural networks; mixtures; multivariate approximation; prediction
UR - http://eudml.org/doc/33371
ER -

References

top
  1. Dempster A. P., Laird N. M., Rubin D. B., Maximum likelihood from incomplete data via the EM algorithm, J. Roy. Statist. Soc. ser. B 39 (1977), 1–38 (1977) Zbl0364.62022MR0501537
  2. Grim J., On numerical evaluation of maximum–likelihood estimates for finite mixtures of distributions, Kybernetika 18 (1982), 3, 173–190 (1982) Zbl0489.62028MR0680154
  3. Grim J., Maximum likelihood design of layered neural networks, In: IEEE Proceedings of the 13th International Conference on Pattern Recognition, IEEE Press 1996, pp. 85–89 (1996) 
  4. Grim J., Design of multilayer neural networks by information preserving transforms, In: Proc. 3rd Systems Science European Congress (E. Pessa, M. B. Penna and A. Montesanto, eds.), Edizzioni Kappa, Roma 1996, pp. 977–982 (1996) 
  5. Jacobs R. A., Jordan M. I., Nowlan S. J., Hinton G. E., 10.1162/neco.1991.3.1.79, Neural Comp. 3 (1991), 79–87 (1991) DOI10.1162/neco.1991.3.1.79
  6. Jordan M. I., Jacobs R. A., 10.1162/neco.1994.6.2.181, Neural Comp. 6 (1994), 181–214 (1994) DOI10.1162/neco.1994.6.2.181
  7. Chen, Ke, Xie, Dahong, Chi, Huisheng, 10.1109/72.536325, IEEE Trans. Neural Networks 7 (1996), 1309–1313 (1996) DOI10.1109/72.536325
  8. Ramamurti V., Ghosh J., Structural adaptation in mixtures of experts, In: IEEE Proceedings of the 13th International Conference on Pattern Recognition, IEEE Press, 1996, pp. 704–708 (1996) 
  9. Titterington D. M., Smith A. F. M., Makov U. E., Statistical Analysis of Finite Mixture Distributions, John Wiley & Sons, Chichester – Singapore – New York 1985 Zbl0646.62013MR0838090
  10. Vajda I., Theory of Statistical Inference and Information, Kluwer, Boston 1992 Zbl0711.62002
  11. Wu C. F. J., 10.1214/aos/1176346060, Ann. Statist. 11 (1983), 95–103 (1983) Zbl0517.62035MR0684867DOI10.1214/aos/1176346060
  12. Xu L., Jordan M. I., 10.1162/neco.1996.8.1.129, Neural Comp. 8 (1996), 129–151 (1996) DOI10.1162/neco.1996.8.1.129
  13. Xu L., Jordan M. I., Hinton G. E., A modified gating network for the mixtures of experts architecture, In: Proc. WCNN’94, San Diego 1994, Vol. 2, pp. 405–410 (1994) 

NotesEmbed ?

top

You must be logged in to post comments.

To embed these notes on your page include the following JavaScript code on your page where you want the notes to appear.

Only the controls for the widget will be shown in your chosen language. Notes will be shown in their authored language.

Tells the widget how many notes to show per page. You can cycle through additional notes using the next and previous controls.

    
                

                
Note: Best practice suggests putting the JavaScript code just before the closing </body> tag.