A recursive self-tuning control scheme for finite Markov chains is proposed wherein the unknown parameter is estimated by a stochastic approximation scheme for maximizing the log-likelihood function and the control is obtained via a relative value iteration algorithm. The analysis uses the asymptotic o.d.e.s associated with these.
The problem of minimizing the ergodic or time-averaged cost for a controlled diffusion with partial observations can be recast as an equivalent control problem for the associated nonlinear filter. In analogy with the completely observed case, one may seek the value function for this problem as the vanishing discount limit of value functions for the associated discounted cost problems. This passage is justified here for the scalar case under a stability hypothesis, leading in particular to a "martingale"...
Page 1