On adaptive control of a partially observed Markov chain

Giovanni Di Masi; Łukasz Stettner

Applicationes Mathematicae (1994)

  • Volume: 22, Issue: 2, page 165-180
  • ISSN: 1233-7234

Abstract

top
A control problem for a partially observable Markov chain depending on a parameter with long run average cost is studied. Using uniform ergodicity arguments it is shown that, for values of the parameter varying in a compact set, it is possible to consider only a finite number of nearly optimal controls based on the values of actually computable approximate filters. This leads to an algorithm that guarantees nearly selfoptimizing properties without identifiability conditions. The algorithm is based on probing control, whose cost is additionally assumed to be periodically observable.

How to cite

top

Di Masi, Giovanni, and Stettner, Łukasz. "On adaptive control of a partially observed Markov chain." Applicationes Mathematicae 22.2 (1994): 165-180. <http://eudml.org/doc/219089>.

@article{DiMasi1994,
abstract = {A control problem for a partially observable Markov chain depending on a parameter with long run average cost is studied. Using uniform ergodicity arguments it is shown that, for values of the parameter varying in a compact set, it is possible to consider only a finite number of nearly optimal controls based on the values of actually computable approximate filters. This leads to an algorithm that guarantees nearly selfoptimizing properties without identifiability conditions. The algorithm is based on probing control, whose cost is additionally assumed to be periodically observable.},
author = {Di Masi, Giovanni, Stettner, Łukasz},
journal = {Applicationes Mathematicae},
keywords = {uniform ergodicity; long run average cost; filtering process; adaptive control; approximate filter; partially observed systems; partially observable Markov chain depending on a parameter; optimal controls; approximate filters},
language = {eng},
number = {2},
pages = {165-180},
title = {On adaptive control of a partially observed Markov chain},
url = {http://eudml.org/doc/219089},
volume = {22},
year = {1994},
}

TY - JOUR
AU - Di Masi, Giovanni
AU - Stettner, Łukasz
TI - On adaptive control of a partially observed Markov chain
JO - Applicationes Mathematicae
PY - 1994
VL - 22
IS - 2
SP - 165
EP - 180
AB - A control problem for a partially observable Markov chain depending on a parameter with long run average cost is studied. Using uniform ergodicity arguments it is shown that, for values of the parameter varying in a compact set, it is possible to consider only a finite number of nearly optimal controls based on the values of actually computable approximate filters. This leads to an algorithm that guarantees nearly selfoptimizing properties without identifiability conditions. The algorithm is based on probing control, whose cost is additionally assumed to be periodically observable.
LA - eng
KW - uniform ergodicity; long run average cost; filtering process; adaptive control; approximate filter; partially observed systems; partially observable Markov chain depending on a parameter; optimal controls; approximate filters
UR - http://eudml.org/doc/219089
ER -

References

top
  1. [1] A. Arapostathis and S. I. Marcus, Analysis of an identification algorithm arising in the adaptive estimation of Markov chains, Math. Control Signals Systems 3 (1990), 1-29. Zbl0685.93063
  2. [2] V. V. Baranov, A recursive algorithm in Markovian decision processes, Cybernetics 18 (1982), 499-506. Zbl0517.90089
  3. [3] D. P. Bertsekas, Dynamic Programming and Stochastic Control, Academic Press, New York, 1976. 
  4. [4] J. L. Doob, Stochastic Processes, Wiley, New York, 1953. Zbl0053.26802
  5. [5] W. Feller, An Introduction to Probability Theory and Its Applications II, Wiley, New York, 1971. Zbl0219.60003
  6. [6] E. Fernández-Gaucherand, A. Arapostathis and S. I. Marcus, On the adaptive control of a partially observable Markov decision process, in: Proc. 27th IEEE Conf. on Decision and Control, 1988, 1204-1210. 
  7. [7] E. Fernández-Gaucherand, A. Arapostathis and S. I. Marcus, On the adaptive control of a partially observable binary Markov decision process, in: Advances in Computing and Control, W. A. Porter, S. C. Kak and J. L. Aravena (eds.), Lecture Notes in Control and Inform. Sci. 130, Springer, New York, 1989, 217-228. Zbl0712.93063
  8. [8] L. G. Gubenko and E. S. Shtatland, On discrete-time Markov decision processes, Theory Probab. Math. Statist. 7 (1975), 47-61. 
  9. [9] O. Hernández-Lerma, Adaptive Markov Control Processes, Springer, New York, 1989. 
  10. [10] O. Hernández-Lerma and S. I. Marcus, Adaptive control of Markov processes with incomplete state information and unknown parameters, J. Optim. Theory Appl. 52 (1987), 227-241. Zbl0585.90090
  11. [11] O. Hernández-Lerma and S. I. Marcus, Nonparametric adaptive control of discrete-time partially observable stochastic systems, J. Math. Anal. Appl. 137 (1989), 312-334. Zbl0675.93055
  12. [12] A. H. Jazwinski, Stochastic Processes and Filtering Theory, Academic Press, New York, 1970. Zbl0203.50101
  13. [13] N. W. Kartashov, Criteria for uniform ergodicity and strong stability of Markov chains in general state space, Theory Probab. Math. Statist. 30 (1985), 71-89. Zbl0586.60058
  14. [14] P. R. Kumar and P. Varaiya, Stochastic Systems: Estimation, Identification and Adaptive Control, Prentice-Hall, Englewood Cliffs, 1986. Zbl0706.93057
  15. [15] H. J. Kushner and H. Huang, Approximation and limit results for nonlinear filters with wide bandwidth observation noise, Stochastics 16 (1986), 65-96. Zbl0595.60046
  16. [16] G. E. Monahan, A survey of partially observable Markov decision processes: theory, models and algorithms, Management Sci. 28 (1982), 1-16. Zbl0486.90084
  17. [17] W. J. Runggaldier and Ł. Stettner, Nearly optimal controls for stochastic ergodic problems with partial observation, SIAM J. Control Optim. 31 (1993), 180-218. Zbl0770.93092
  18. [18] Ł. Stettner, On nearly self-optimizing strategies for a discrete-time uniformly ergodic adaptive model, J. Appl. Math. Optim. 27 (1993), 161-177. Zbl0769.93084

NotesEmbed ?

top

You must be logged in to post comments.

To embed these notes on your page include the following JavaScript code on your page where you want the notes to appear.

Only the controls for the widget will be shown in your chosen language. Notes will be shown in their authored language.

Tells the widget how many notes to show per page. You can cycle through additional notes using the next and previous controls.

    
                

                
Note: Best practice suggests putting the JavaScript code just before the closing </body> tag.