In this paper a problem of consumption and investment is presented as a model of a discounted Markov decision process with discrete-time. In this problem, it is assumed that the wealth is affected by a production function. This assumption gives the investor a chance to increase his wealth before the investment. For the solution of the problem there is established a suitable version of the Euler Equation (EE) which characterizes its optimal policy completely, that is, there are provided conditions...

We consider a stochastic approach in order to define an equilibrium model for a traffic-network problem. In particular, we assume a markovian behaviour of the users in their movements throughout the zones of the traffic area. This assumption turns out to be effective at least in the context of urban traffic, where, in general, the users tend to travel by choosing the path they find more convenient and not necessarily depending on the already travelled part. The developed model is a homogeneous Markov...

We consider a stochastic approach in order to define an equilibrium model for a traffic-network problem. In particular, we assume a Markovian behaviour of the users in their movements throughout the zones of the traffic area. This assumption turns out to be effective at least in the context of urban traffic, where, in general, the users tend to travel by choosing the path they find more convenient and not necessarily depending on the already travelled part. The developed model is a homogeneous...

The present paper studies the approximate value iteration (AVI) algorithm for the average cost criterion with bounded costs and Borel spaces. It is shown the convergence of the algorithm and provided a performance bound assuming that the model satisfies a standard continuity-compactness assumption and a uniform ergodicity condition. This is done for the class of approximation procedures that can be represented by linear positive operators which give exact representation of constant functions and...

We show how the use of a parallel between the ordinary (+, X) and the (max, +) algebras, Maslov measures that exploit this parallel, and more specifically their specialization to probabilities and the corresponding cost measures of Quadrat, offer a completely parallel treatment of stochastic and minimax control of disturbed nonlinear discrete time systems with partial information. This paper is based upon, and improves, the discrete time part of the earlier paper [9].

In a Discounted Markov Decision Process (DMDP) with finite action sets the Value Iteration Algorithm, under suitable conditions, leads to an optimal policy in a finite number of steps. Determining an upper bound on the necessary number of steps till gaining convergence is an issue of great theoretical and practical interest as it would provide a computationally feasible stopping rule for value iteration as an algorithm for finding an optimal policy. In this paper we find such a bound depending only...

We offer the quantitative estimation of stability of risk-sensitive cost optimization in the problem of optimal stopping of Markov chain on a Borel space $X$. It is supposed that the transition probability $p(·|x)$, $x\in X$ is approximated by the transition probability $\tilde{p}(·|x)$, $x\in X$, and that the stopping rule ${\tilde{f}}_{*}$ , which is optimal for the process with the transition probability $\tilde{p}$ is applied to the process with the transition probability $p$. We give an upper bound (expressed in term of the total variation distance: ${sup}_{x\in X}\parallel p(·|x)-\tilde{p}(·|x)\parallel )$ for...

We study the adaptive control problem for discrete-time Markov control processes with Borel state and action spaces and possibly unbounded one-stage costs. The processes are given by recurrent equations $x_{t+1}=F(x_t,a_t,{\xi }_t),t=0,1,...$ with i.i.d. ${\Re }^k$-valued random vectors ${\xi }_t$ whose density $\rho$ is unknown. Assuming observability of ${\xi }_t$ we propose the procedure of statistical estimation of $\rho$ that allows us to prove discounted asymptotic optimality of two types of adaptive policies used early for the processes with bounded costs.

The article presents an extension of the theory of standard Markov decision processes on discrete spaces and with the average cost as the objective function which permits to take into account a fuzzy average cost of a trapezoidal type. In this context, the fuzzy optimal control problem is considered with respect to two cases: the max-order of the fuzzy numbers and the average ranking order of the trapezoidal fuzzy numbers. Each of these cases extends the standard optimal control problem, and for...

This work concerns controlled Markov chains with finite state space and compact action sets. The decision maker is risk-averse with constant risk-sensitivity, and the performance of a control policy is measured by the long-run average cost criterion. Under standard continuity-compactness conditions, it is shown that the (possibly non-constant) optimal value function is characterized by a system of optimality equations which allows to obtain an optimal stationary policy. Also, it is shown that the...

We address a queueing control problem considering service times and conversion times following normal distributions. We formulate the multi-server queueing control problem by constructing a semi-Markov decision process (SMDP) model. The mechanism of state transitions is developed through mathematical derivation of the transition probabilities and transition times. We also study the property of the queueing control system and show that optimizing the objective function of the addressed queueing control...

This paper deals with a certain class of unbounded optimization problems. The optimization problems taken into account depend on a parameter. Firstly, there are established conditions which permit to guarantee the continuity with respect to the parameter of the minimum of the optimization problems under consideration, and the upper semicontinuity of the multifunction which applies each parameter into its set of minimizers. Besides, with the additional condition of uniqueness of the minimizer, its...

In this paper we give a new set of verifiable conditions for the existence of average optimal stationary policies in discrete-time Markov decision processes with Borel spaces and unbounded reward/cost functions. More precisely, we provide another set of conditions, which only consists of a Lyapunov-type condition and the common continuity-compactness conditions. These conditions are imposed on the primitive data of the model of Markov decision processes and thus easy to verify. We also give two...