Discrete-time Markov control processes with recursive discount rates

Yofre H. García; Juan González-Hernández

Kybernetika (2016)

  • Volume: 52, Issue: 3, page 403-426
  • ISSN: 0023-5954

Abstract

top
This work analyzes a discrete-time Markov Control Model (MCM) on Borel spaces when the performance index is the expected total discounted cost. This criterion admits unbounded costs. It is assumed that the discount rate in any period is obtained by using recursive functions and a known initial discount rate. The classic dynamic programming method for finite-horizon case is verified. Under slight conditions, the existence of deterministic non-stationary optimal policies for infinite-horizon case is proven. Also, to find deterministic non-stationary ϵ - optimal policies, the value-iteration method is used. To illustrate an example of recursive functions that generate discount rates, we consider the expected values of stochastic processes, which are solutions of certain class of Stochastic Differential Equations (SDE) between consecutive periods, when the initial condition is the previous discount rate. Finally, the consumption-investment problem and the discount linear-quadratic problem are presented as examples; in both cases, the discount rates are obtained using a SDE, similar to the Vasicek short-rate model.

How to cite

top

García, Yofre H., and González-Hernández, Juan. "Discrete-time Markov control processes with recursive discount rates." Kybernetika 52.3 (2016): 403-426. <http://eudml.org/doc/281537>.

@article{García2016,
abstract = {This work analyzes a discrete-time Markov Control Model (MCM) on Borel spaces when the performance index is the expected total discounted cost. This criterion admits unbounded costs. It is assumed that the discount rate in any period is obtained by using recursive functions and a known initial discount rate. The classic dynamic programming method for finite-horizon case is verified. Under slight conditions, the existence of deterministic non-stationary optimal policies for infinite-horizon case is proven. Also, to find deterministic non-stationary $\epsilon -$optimal policies, the value-iteration method is used. To illustrate an example of recursive functions that generate discount rates, we consider the expected values of stochastic processes, which are solutions of certain class of Stochastic Differential Equations (SDE) between consecutive periods, when the initial condition is the previous discount rate. Finally, the consumption-investment problem and the discount linear-quadratic problem are presented as examples; in both cases, the discount rates are obtained using a SDE, similar to the Vasicek short-rate model.},
author = {García, Yofre H., González-Hernández, Juan},
journal = {Kybernetika},
keywords = {dynamic programming method; optimal stochastic control},
language = {eng},
number = {3},
pages = {403-426},
publisher = {Institute of Information Theory and Automation AS CR},
title = {Discrete-time Markov control processes with recursive discount rates},
url = {http://eudml.org/doc/281537},
volume = {52},
year = {2016},
}

TY - JOUR
AU - García, Yofre H.
AU - González-Hernández, Juan
TI - Discrete-time Markov control processes with recursive discount rates
JO - Kybernetika
PY - 2016
PB - Institute of Information Theory and Automation AS CR
VL - 52
IS - 3
SP - 403
EP - 426
AB - This work analyzes a discrete-time Markov Control Model (MCM) on Borel spaces when the performance index is the expected total discounted cost. This criterion admits unbounded costs. It is assumed that the discount rate in any period is obtained by using recursive functions and a known initial discount rate. The classic dynamic programming method for finite-horizon case is verified. Under slight conditions, the existence of deterministic non-stationary optimal policies for infinite-horizon case is proven. Also, to find deterministic non-stationary $\epsilon -$optimal policies, the value-iteration method is used. To illustrate an example of recursive functions that generate discount rates, we consider the expected values of stochastic processes, which are solutions of certain class of Stochastic Differential Equations (SDE) between consecutive periods, when the initial condition is the previous discount rate. Finally, the consumption-investment problem and the discount linear-quadratic problem are presented as examples; in both cases, the discount rates are obtained using a SDE, similar to the Vasicek short-rate model.
LA - eng
KW - dynamic programming method; optimal stochastic control
UR - http://eudml.org/doc/281537
ER -

References

top
  1. Arnold, L., 10.1002/zamm.19770570413, John Wiley and Sons, New York 1973. Zbl0278.60039MR0443083DOI10.1002/zamm.19770570413
  2. Ash, R., Doléans-Dade, C., Probability and Measure Theory., Academic Press, San Diego 2000. Zbl0944.60004MR1810041
  3. Bellman, R., Dynamic Programming., Princeton Univ. Press, New Jersey 1957. Zbl1205.90002MR0090477
  4. Bertsekas, D., Shreve, S., Stochastic Optimal Control: The Discrete Time Case., Athena Scientific, Massachusetts 1996. Zbl0633.93001MR0511544
  5. Brigo, D., Mercurio, F., 10.1007/978-3-662-04553-4, Springer-Verlag, New York 2001. Zbl1109.91023MR1846525DOI10.1007/978-3-662-04553-4
  6. Black, F., Karasinski, P., 10.2469/faj.v47.n4.52, Financ. Anal. J. 47 (1991), 4, 52-59. DOI10.2469/faj.v47.n4.52
  7. Carmon, Y., Shwartz, A., 10.1016/j.orl.2008.10.005, Oper. Res. Lett. 37 (2009), 51-55. Zbl1154.90610MR2488083DOI10.1016/j.orl.2008.10.005
  8. Vecchia, E. Della, Marco, S. Di, Vidal, F., Dynamic programming for variable discounted Markov decision problems., In: Jornadas Argentinas de Informática e Investigación Operativa (43JAIIO) - XII Simposio Argentino de Investigación Operativa (SIO), Buenos Aires, 2014, pp. 50-62. 
  9. Cox, J., Ingersoll, J., Ross, S., 10.2307/1911242, Econometrica 53 (1985), 385-407. Zbl1274.91447MR0785475DOI10.2307/1911242
  10. Dothan, U., 10.1016/0304-405x(78)90020-x, J. Financ. Econ. 6 (1978), 59-69. DOI10.1016/0304-405x(78)90020-x
  11. Feinberg, E., Shwartz, A., 10.1287/moor.19.1.152, J. Finan. Econ. 19 (1994), 152-168. Zbl0803.90123MR1290017DOI10.1287/moor.19.1.152
  12. González-Hernández, J., López-Martínez, R., Pérez-Hernández, J., 10.1007/s00186-006-0092-2, Math. Method Oper. Res. 65 (2006), 1, 27-44. Zbl1126.90075MR2302022DOI10.1007/s00186-006-0092-2
  13. González-Hernández, J., López-Martínez, R., Minjarez-Sosa, A., Adaptive policies for stochastic systems under a randomized discounted cost criterion., Bol. Soc. Mat. Mex. 14 (2008), 3, 149-163. Zbl1201.93130MR2667162
  14. González-Hernández, J., López-Martínez, R., Minjarez-Sosa, A., Approximation, estimation and control of stochastic systems under a randomized discounted cost criterion., Kybernetika 45 (2008), 5, 737-754. Zbl1190.93105MR2599109
  15. Guo, X., Hernández-Del-Valle, A., Hernández-Lerma, O., 10.3166/ejc.18.528-538, Eur. J. Control 15 (2012), 7, 528-538. MR3086896DOI10.3166/ejc.18.528-538
  16. Hernández-Lerma, O., Lasserre, J. B., 10.1007/978-1-4612-0729-0, Springer-Verlag, New York 1996. Zbl0840.93001MR1363487DOI10.1007/978-1-4612-0729-0
  17. Minjarez-Sosa, J., 10.1007/s11750-015-0360-5, TOP 23 (2015), 3, 743-772. MR3407674DOI10.1007/s11750-015-0360-5
  18. Hinderer, K., 10.1007/978-3-642-46229-0, In: Lecture Notes Operations Research (M. Bechmann and H. Künzi, eds.), Springer-Verlag 33, Zürich 1970. MR0267890DOI10.1007/978-3-642-46229-0
  19. Ho, T., Lee, S., 10.1111/j.1540-6261.1986.tb02528.x, J. Financ. 41 (1986), 1011-1029. DOI10.1111/j.1540-6261.1986.tb02528.x
  20. Hull, J., Options, Futures and other Derivatives. Sixth edition., Prentice Hall, New Jersey 2006. 
  21. Hull, J., White, A., 10.1093/rfs/3.4.573, Rev. Financ. Stud. 3 (1990), 573-592. DOI10.1093/rfs/3.4.573
  22. Mercurio, F., Moraleda, J., 10.1080/13518470151141440, Eur. J. Finance 7 (2001), 93-116. DOI10.1080/13518470151141440
  23. Rendleman, R., Bartter, B., 10.2307/2979016, J. Financ. Quant. Anal. 15 (1980), 11-24. DOI10.2307/2979016
  24. Rieder, U., 10.1007/bf01168566, Manuscripta Math. 24 (1978), 115-131. Zbl0385.28005MR0493590DOI10.1007/bf01168566
  25. Schäl, M., 10.1007/bf00532612, Z. Wahrscheinlichkeit 32 (1975), 179-196. Zbl0316.90080MR0378841DOI10.1007/bf00532612
  26. Vasicek, O., 10.1016/0304-405x(77)90016-2, J. Financ. Econ. 5 (1977), 177-188. DOI10.1016/0304-405x(77)90016-2
  27. Wei, Q., X, X. Guo, 10.1016/j.orl.2011.06.014, Oper. Res. Lett. 39 (2011), 369-374. Zbl1235.90178MR2835530DOI10.1016/j.orl.2011.06.014
  28. Ye, L., Guo, X., 10.1007/s10440-012-9669-3, Acta Appl. Math. 121 (2012), 1, 5-27. Zbl1281.90082MR2966962DOI10.1007/s10440-012-9669-3
  29. Zhang, Y., 10.1007/s11750-011-0186-8, TOP 21 (2013), 2, 378-408. Zbl1273.90235MR3068494DOI10.1007/s11750-011-0186-8

NotesEmbed ?

top

You must be logged in to post comments.

To embed these notes on your page include the following JavaScript code on your page where you want the notes to appear.

Only the controls for the widget will be shown in your chosen language. Notes will be shown in their authored language.

Tells the widget how many notes to show per page. You can cycle through additional notes using the next and previous controls.

    
                

                
Note: Best practice suggests putting the JavaScript code just before the closing </body> tag.