A review of deterministic and stochastic approximation methods for function minimization

 Access to full text

 Full (PDF)

1. Ablow C. M., G. Brigham (1955), An analog solution of programming problems, OR 3, 388 to 394. MR0073323
2. Abrham J. (1958), Přibližná metoda pro nelinearní programování, Čas. pěst. mat. 83, 425-439. MR0101503
3. Abrham J. (1961), An Approximate Method for Convex Programming, Econometrica 29, 700 to 703. MR0138507
4. Abrham J. (1962), The Multiplex Method and its Application to Concave Programming, CZM 12, 325-345. MR0145988
5. Ackoff R. L. (1961), Progress in Operations Research, Vol. I. John Wiley & Sons. Inc., New York - London. MR0121239
6. Akaike H. (1959), On successive transformation of probability distribution and its application to the analysis of the optimum gradient method, Ann. Inst. Statist. Math. Tokyo 11, 1 - 16. MR0107973
7. Altman M. (1964), Stationary points in non-linear programming, Bull. Acad. Polon. Sci. Ser. Sci. Math. Astronom. Phys. 12, 29-35. MR0164810
8. Archer D. H. (1960), An optimising control for the chemical process industries, Britisch Chemical Engineering, 88 -94. 
9. Arnoff E. L., S. S. Sengupta (1961), Mathematical Programming, Viz sb. Ackoff (1961), 105-210. (1961) MR0121245
10. Arrow K. J., L. Hurwicz (1951), A gradient method for approximation of saddle points and conditionned maxima, Rep. 253 RAND Corporation, June 13. 
11. Arrow K. J., L. Hurwicz (1956), Reduction of constrained maxima to saddle-point problems, Proc. 3rd Berkeley Symp. Math., Statist. Probability, vol. V, 1 - 20, Un. Calif. Press. MR0084938
12. Arrow K. J., L. Hurwicz H. Uzawa (1958), Studies in Linear and Non-linear Programming, Stanford University Press, Stanford, California. MR0108399
13. Beale E. M. L. (1955), On optimising a convex function subject to linear inequalities, JRSS B 17, 173-184. MR0089101
14. Beale E. M. L. (1958), On an iterative method for finding a local minimum of a function of more than one variable, Statist. Tech. Research Group Techn. Report No. 25, Princeton Univ., Princeton, N. Y. 
15. Beale E. M. L. (1959), On quadratic programming, Naval Research Logistic Quart. 6, 227 - 243. MR0113722
16. Bellman R. (1957), Dynamic Programming, Princeton University Press, Princeton, N. Y. MR0090477
17. Block H. D. (1957), On stochastic approximation, ONR Report, Dept. of Mathematics. Cornell University, Ithaca, New York, 13 pp. 
18. Blum J. R. (1954a), Approximation methods which converge with probability one, AMS 25, 382 to 386. MR0062399
19. Blum J. R. (1954b), Multidimensional stochastic approximation methods, AMS 25, 463 - 483. MR0065092
20. Blum J. R. (1958), A note on stochastic approximation, Proc. Am. Math. Soc., 9, 404-407. MR0098426
21. Both A. D. (1949), An application of the method of steepest descent to the solution of systems of non-linear simultaneous equations, Quar. J. Mech. Appl. Math. 2, 460 -468. MR0035119
22. Box G. E. P., K. B. Wilson (1951), On the experimental attainment of optimum conditions, JRSS B 13, 1-45. MR0046009
23. Box G. E. P. (1954), The exploration and exploitation of response surfaces: some general considerations and examples, Biometrics 10, 16-60. 
24. Box G. E. P., P. V. Youle (1955), The exploration and exploitation of response surfaces: an example of the link between the fitted surface and the basic mechanism of the system, Biometrics 11, 287-323. 
25. Box G. E. P. (1957), Evolutionary operation: a method for increasing industrial productivity, Applied Statistics 6, 81 - 101. 
26. Box G. E. P., J. S. Hunter (1959), Condensed calculations for evolutionary operation programs, Technometrics, 77-95. MR0107950
27. Brooks S. H. (1959), A comparison of maximum seeking methods, J. Oper. Research Soc. 7, 430-457. 
28. Brownlee K. A. J. L. Hodges, Jr., M. Rosenblatt (1963), The Up-and-Down method with small samples, JASA 48, 262-277. MR0055644
29. Burkholder D. L. (1956), On a certain class of stochastic approximation procedures, AMS 27, 1044-1059. MR0085653
30. Carroll Ch. W. (1961), The created response surface technique for non-linear restrained systems, OR 9, 169-185. MR0129020
31. Cauchy A. (1847), Méthode générale pour la résolution des systèms d'equations simultanées, Compte Rendu, Ac. Sci. Paris, 25, 536-538. 
32. Cheney E. W., A. A. Goldstein (1959), Newton's method of convex programming and Tchebycheff approximation, Numerische Math. 1, 253 - 268. MR0109430
33. Chung K. L. (1954), On a stochastic approximation method, AMS 25, 463 - 483. MR0064365
34. Crockett J. B., H. Chernoff (1955), Gradient methods of maximization, Pacif. J. Math. 5, 33-50. MR0075676
35. Curry H. B. (1944), The method of steepest descent for non-linear minimization problems, Quarterly Appl. Math. 2, 258-261. MR0010667
36. DeLand E. C. (1959), Continuous programming methods on an analog computer, Rand Corpor. Paper No. P-1815, Sept. 
37. Derman C. (1956), An application of Chung's lemma to the Kiefer-Wolfowitz stochastic approximation procedure, AMS 27, 532-536. MR0078595
38. Derman C. (1956), Stochastic approximation, AMS 27, 879-886. MR0088133
39. Derman C. (1957), Non-parametric Up-and-Down experimentations, AMS 28, 795 - 798. MR0090956
40. Derman C., J. Sacks (1959), On Dvoretzky's stochastic approximation theorem, AMS 30, 601-606. MR0107893
41. Dixon W. J., A. M. Mood (1948), A method for obtaining and analyzing sensitivity data, JASA 43, 109-126. 
42. Driml M., O Hanš (1960), On experience theory problems, PRTR2, 93 - 111. MR0142189
43. Driml M., O. Hanš (1960), Continuous stochastic approximation, PRTR2, 113-122. MR0130702
44. Driml M., J. Nedoma (1960), Stochastic approximation for continuous random processes, PRTR2, 145-158. MR0126877
45. Dupač V. (1957), O Kiefer-Wolfowitzově aproximační metodě, Čas. pěst. mat. 82, 47 - 75. MR0089556
46. Dupač V. (1958), Notes on stochastic approximation methods, CZM 8 (83), 139-149. MR0096301
47. Dupač V. (1965a), A dynamic stochastic approximation method, V tisku v AMS 36. MR0193724
48. Dupač V. (1965b), A dynamical stochastic approximation method, Referát na 4. pražské konferenci o teorii informace, statistických rozhodovacích funkcích a náhodných procesech. 
49. Dvoretzky A. (1956), On stochastic approximation, Proc. Third. Berkeley Symp. Math. Statistics and Probability, Vol. I, 39-55. University of California Press. MR0084911
50. Fabian V. (1960), Stochastic approximation methods, CZM 10 (85), 123-159. MR0113253
51. Fabian V. (1961), Stochastická aproximační metoda pro hledání optimálních podmínek v experimentální práci a v samoadaptivních systémech, Aplikace matematiky, 6, 162-183. MR0131896
52. Fabian V. (1962), Blokové logické schema automatického optimalizátoru, Aplikace matematiky 7, 426-440. 
53. Fabian V. (1964), A new one-dimensional stochastic approximation method for finding a local minimum of a function, PRTR3, 85-105. MR0169317
54. Fabian V. (1965), Stochastic approximation of constrained minima, Referát na 4. pražské konferenci o teorii informace, statistických rozhodovacích funkcích a náhodných procesech. 
55. Fabius J. (1959), Stochastic approximation, Statistics Neerlandica 13, 445-452. MR0132597
56. Фаддеев Д. К. B. H. Фаддеевa (1963), Вычислительные методы линейной алгебры, Москва. 
57. Farkas J. (1902), Über die Theorie der einfachen Ungleichungen, Zeit. Reine Angew. Math. 124, 1-24. 
58. Farrell R. H. (1962), Bounded length confidence intervals for the zero of a regression funkction, AMS 33, 237-247. MR0137231
59. Фельдбаум А. А. (1958), Автоматический оптимизатор, Автоматика и телемeхaника 14, 731-743. 
60. Fiacco A. V., G. P. McCormick (1964a), The sequential unconstrained minimization technique for nonlinear programming, a primal-dual method, Management Science 10, 360 - 366. MR1058438
61. Fiacco A. V., G. P. McCormick (1964b), Computational algorithm for the sequential unconstrained minimization technique for nonlinear programming, Management Science 10, 601-617. 
62. Finkel R. W. (1959), The method of resultant descents for the minimization of an arbitrary function, Space Technology Lab. Tech. Report, Los Angeles. 
63. Fletcher R., M. J. D. Powell (1963/4), A rapidly convergent descent method for minimization, Comp. J. 6, 163 - 168. MR0152116
64. Forsythe A. I., G. E. Forsythe (1952), Punched-cards experiments with accelerated gradient method for linear equations, NBS Report 1643, National Bureau of Standards, Los Angeles. 
65. Forsythe G. E., T. S. Motzkin (1951), Assymptotic properties of the optimum gradient method, Bull. Amer. Math. Soc. 57, 73. 
66. Forsythe G. E. (1953), Solving linear algebraic expressions can be interesting, Bull. Amer. Math. Soc. 59, 299-329. MR0056372
67. Forsythe G. E., W. R. Wasow (1959), Finite-difference methods for partial difference equations, New York, Wiley & Sons. 
68. Фридман B. M. (1962), О сходимости методов типа найскорейшего спуска, Усп. мат. наук 17, 201-204. MR0140180
69. Friedman M., L. J. Savage (1947), Planning experiments seeking maxima, Select. Techn. of Statist. Analysis. McGraw-Hill, New York. 
70. Friedman S.. (1963), On stochastic approximations, AMS 34, 343 - 346. MR0144379
71. Frish R. (1957), The multiplex method for linear programming, Sankhya 18, 329-362. MR0093430
72. Frish R. (1958), The multiplex method for linear programming, Memorandum fra Socialekonomisk Institut, Universitetet Oslo 12. 
73. Gardner L. A., Jr. (1963), Stochastic approximation and its application to problems of prediction and control synthesis, Internat. Symp. Nonlinear Differential Equations and Nonlinear Mech., Colorado Springs, New York -London, Acad. Press, 241 - 258. MR0148200
74. Gardner L. A., Jr. (1964), Adaptive predictors, PRTR3. MR0166896
75. Germansky B. (1934), Notiz über die Lösung von Extemaufgaben mittels Iteration, Z. Ang. Math. Mech. 14, 187-187. 
76. Гладышев Е. Г., O стохастической аппроксимации, Теория вероятностей и прим. 10, 297-300. Zbl1170.81475
77. Glass H., L. Cooper (1965), Sequential search: a method for solving constrained optimization problems, JACM 12, 71-82. 
78. Goldstein A. A. (1962), Cauchy's method of minimization, Numer. Math. 4, 146-150. MR0141222
79. Goldstein A. A., Ward Cheney (1958), A finite algorithm for the solutions of consistent linear equations and inequalities and for the Tchebycheff approximation of inconsistent linear equations, Pacific J. Math. 8, 415-427. MR0101505
80. Griffith R. E., R. A. Stewart (1961), A nonlinear programming technique for the optimization of continuous processing systems, Management Science 7, 379-392. MR0134472
81. Guttman L., R. Guttman (1959), An illustration of the use of stochastic approximation methods, Biometrics 15, 551 - 559. 
82. Hanš O., A. Špaček (1960), Random fixed point approximation by differentiable trajectories, PRTR2, 203-213. MR0124084
83. Hartley H. O., R. R. Hocking (1963), Convex programming by tangential approximation, Management Science 9, 600-612. MR0154748
84. Hestens M. R. (1951), Iterative methods for solving linear equations, NAML Report 52-9, Nat. Bureau of Standards, Los Angeles, 19 pp. (19 p) 
85. Ho Yu Chi (1962), On the stochastic approximation method and optimal filtering theory, Journal of math. analysis and applications 6, 152-154. MR0144440
86. Hodges J. L., Jr., E. L. Lehmann (1956), Two approximation to the Robbins Monro process, Proc. 3rd Berkeley Symp. on Math. Statistics and Prob., vol. 1, 95-104. MR0084915
87. Hooke R., T. A. Jeeves (1961), Direct search solution of numerical and statistical problems, JACM 8, 212- 
88. Hotelling H. (1941), Experimental determination of the maximum of a function, AMS 12, 20-46. MR0003521
89. Householder A. S., F. L. Bauer (1960), Certain iterative methods for solving linear systems, Numerische Mathematik 2, 55-59. MR0116464
90. Иванов B. B. (1962), Oб алгоритмах быстрого спуска, ДAH CCCP 143, 775-778. MR0148218
91. John F. (1964), Extremum problems with inequalities as subsidiary conditions, „Studies and Essays", Courant anniversary volume, pp. 187-204. MR0030135
92. Kantorovič L. V. (1945), On an effective method of solving extremal problems for quadratic functional, C. R. de l'Ac. Sci. de l'URSS XLVIII, no. 7, 455-459. MR0016528
93. Канторович Л. B. (1945), 0б oднom методе решения экстремальных задач для квадратных функциохалов, ДАН CCCP 48, 483-487. 
94. Канторович Л. B. (1947), О методе найскорейшего спуска, ДAH CCCP 56, 233-236 MR0023126
95. Канторович Л. B. (1948), Функциональный анализ и прокладная математика, Усп. мат. наук 3, 89-185. MR0027947
96. Kelley J. E. (1960), The cutting-plane method for solving convex programs, JSIAM 8, 703-712. MR0118538
97. Kerner M. (1933), Die Differentiale in der allegemeinen Analysis, Ann. of Math. 34, 546- 572. MR1503123
98. Kerstukos A., R. I. van Nice (1958), Optimizing process operations, Automation 5. 
99. Kesten H. (1958), Accelerated stochastic approximation, AMS 29, 41-59. MR0093851
100. Kiefer J., J. Wolfowitz (1952), Stochastic estimation of the maximum of a regression function, AMS 23, 462-466. MR0050243
101. Kiefer J. (1953), Sequential minimax search for a maximum, Proc. Amer. Math. Soc. 4, 502-506. MR0055639
102. Kiefer J. (1957), Optimum sequential search and approximation methods under minimum regularity assumptions, JSIAM 5, 105-136. MR0092326
103. Killianpur G. (1954), A note on the Robbins-Monro stochastic approximation method, AMS 25. MR0062400
104. Kitagawa T. (1959), Successive processes of statistical controls (2), Met. Coll. Sci. Univ. Kyoto, Ser. A, Math. 13, 1-16. MR0105184
105. Kizner W. (1964), A numerical method for finding solutions of nonlinear equations, JSIAM 12, 424-428. MR0168115
106. Klingman W. R., D. M. Himmelblau (1964), Nonlinear programming with the aid of a multiplegradient summation technique, JACM 11, 400-415. MR0175635
107. Kose T. (1956), Solutions of saddle value problems by differential equations, Econometrica 24, 59-70. MR0077854
108. Красулина Т. П. (1962), 3амечания о некотопых процесах стохастической апроксимации, Теория вероят к прим. 7, 113-118. 
109. Krolak P., L. Cooper (1963), An extension of Fibonaccian search to several variables, CACM 6, 639-641. MR0154749
110. Kuhn H. W., A. W. Tucker (1950), Nonlinear programming, Proc. 2nd Berkeley Symp. on Mathem. Statist. and Probability, Univ. of Calif. Pres., 481-492. MR0047303
111. Lanczos C. (1952), Solution of systems of linear equations by minimized iterations, J. Res. Nat. Bureau of Standards 49, 33-53. MR0051583
112. Levenberg K. (1944), A method for the solution of certain non-linear problems in least squares, Qu. Appl. Math. 2, 164-168. MR0010666
113. Leżański T. (1963), Über das Minimumproblem für Funktionale in Banachschen Räumen, Math. Annalen 152, 271-274. MR0158253
114. Maňas M. (1965), Nelineární programování, Ek. mat. obzor 1, 3-34. 
115. Mangasarian O. I. (1962), Duality in nonlinear programming, Qu. Appl. Math. 20, 300-302. MR0141530
116. Mangasarian O. I. (1963), Equivalence in nonlinear programming, Naval. Res. Logist. Quart. 10, 299-306. MR0164813
117. Manne A. S. (1953), Concave programming for gasoline blends, Rand. Corp. Paper. No. P-383. 
118. Marquardt D. W. (1963), An algorithm for least-squares estimation of nonlinear parameters, JSIAM 11, 431-441. MR0153071
119. Mises R. von, H. Pollaczek-Geiringer (1929), Praktische Verfahren der Gleichungsauflösung, Z. Angew. Math. Mech. 9, 58-77. 
120. Nashed M. Z. (1964), The convergence of the method of steepest descent for nonlinear equations with variational or quasi-variational operators, J. Math. Mech. 13, 765-794. MR0166638
121. Nashed M. Z. (1965), On general iterative methods of the solutions of a class of nonlinear operator equations, Math. of Computation 19, 14-24. MR0179906
122. Nelder J. A., R. Mead (1965), A simplex method for function minimization, Comp. J. 7, 308-313. 
123. Newman D. J. (1960), Locating the maximum on a unimodal surface, Presented at the 18th National Meeting of Operations Res. Soc., October 1960, neuveřejněno, viz Spang (1962). (1960) 
124. Newman D. J. (1965), Location of the maximum on unimodal surfaces, JACM 12, 395-398. MR0182129
125. Pietrzykowski T. (1961), On an iteration method for maximizing a concave function on a convex set, Práce ZAM, A 13. MR0134346
126. Pietrzykowski T. (1962), On a method of approximative finding conditional maximums, Algorytmy 1, 9-15. MR0186777
127. Powell M. J. D. (1964), An efficient method for finding the mininum of a function of several variables without calculating the derivatives, Computer Journal 7, 155-162. MR0187376
128. Powell M. J. D. (1965), A method for minimizing a sum of squares of non-linear functions without calculating derivatives, Comp. J. 7, 303 - 307. MR0172456
129. Reeves C. M. (1964), Algorithm 238, conjugate gradient method, CACM 7, 481. 
130. Remez E. (1934), Sur un procédé convergent d'approximations succesives pour determiner les polynomes d'approximation, C. R. Ac. Sci. Paris 198, 2063-2065. (198,) 
131. Robbins H. S. Monro (1951), A stochastic approximation method, AMS 22, 400-407. MR0042668
132. Rosen J. B. (1960), The gradient projection method for nonlinear programming. I. Linear constrainsts, JSIAM 8, 181-217. MR0112750
133. Rosen J. B. (1961), The gradient projection method for nonlinear programming, Part. II. Nonlinear constraints. JSIAM 9, 514-532. MR0135991
134. Rosenbloom P. C. (1956), The method of steepest descent, Proc. of Symp. in Applied Math. vol. 6, pp. 127-176. MR0079339
135. Rosenbrock H. H. (1960), An automatic method for finding the largest or least value of a function, Computer Journal 3, 175-184. MR0136042
136. Saaty T. L., Bram J. (1964), Nonlinear mathematics, McGraw-Hill Book Co., New York. MR0164812
137. Sacks J. (1958), Asymptotic distributions of stochastic approximatons, AMS 29, 373-405. MR0098427
138. Sakrison D. J. (1962a), Design of filters for non-mean-square error performance criteria by means of a continuous adjustment procedure, Quarterly Progress Report No. 66, 189-201 and No. 67, 119-126, Research Lab. of Electronics. 
139. Sakrison D. J. (1962b), Application of stochastic approximation methods to system optimization, Techn. Rep. 391, July 10, Massachusetts Inst. Technology, Res. Lab. Electronics. Cambridge, Massachusetts. 
140. Sakrison D. J. (1963), Iterative design of optimum filters for non-mean-square error performance criteria, Trans. of the IEEE Professional Group on Information Theory 9, 161-167. 
141. Sakrison D. J. (1964), A continuous Kiefer-Wolfowitz procedure for random processes, AMS 35, 590-599. MR0161445
142. Schmetterer L. (1953), Bemerkungen zum Verfahren der stochastischen Iteration, Österreichisches Ingenieur-Archiv VII, 111-117. MR0055640
143. Schmetterer L. (1954a), Sur l'approximation stochastique, Bull. Inst. Internat. Statist. 24. 203 to 206. MR0068790
144. Schmetterer L. (1954b), Zum Sequentialverfahren von Robbins und Monro, Monatshefte für Mathematik 58, 33-37. MR0061328
145. Schmetterer L. (1958), Sur l'iteration stochastique, Colloques Internationaux du centre national de la recherch scientifique. 87, 55-63. MR0107882
146. Schmetterer L. (1961), Stochastic approximation, Proc. 4th Berkeley Sympos. Math. Statist. Probab. 1, 587-609. MR0137268
147. Schröder E. (1870), Über unendlich viele Algorithmen zur Auflösung der Gleichungen, Math. Ann. 2, 317-365. MR1509664
148. Shah B. V., Buehler R. J., Kempthorne O. (1964), Some algorithms for minimizing a function of several variables, JSIAM 12, 74-92. MR0165655
149. Spang H. A. III (1962), A review of minimization techniques for nonlinear functions, SIAM Review 4, 343-365. MR0145642
150. Стаховский Р. И. (1958), Двухканальный автоматический оптимизатор, Автоматика и телемеханика, 14, 744-756. 
151. Stiefel E. (1952), Über einige Methoden der Relaxationsrechnung, Zeit. Angew. Math. Phys. 3, 1-33. MR0047409
152. Stiefel E. (1955), Relaxation Methoden bester Strategie zur Lösung linearer Gleichungssysteme, Comm. Math. Helv. 29, 157-159. MR0071119
153. Ваинбепг М. М. (1960), О сходимости метода найскорейшего спуска для нелнейных уравнений, ДАН СССР, 130, 9-12. 
154. Vosiková M. (1964), Asymptotické vlastnosti Kiefer-Wolfowitzovy aproximační metody, Dipl. práce, Mat.-fyz. fak. KU. 
155. Wells M. (1965), Algorithm 251, Function minimization [E 4], CACM 8, 169-170. 
156. Wetherill G. B. (1963), Sequential estimation of quantal response curves (with discussion), JRS. B 25, 1-48. 
157. Witzgall C. (1960), Gradient-projection methods for linear programming, Princeton Un. and IBM Corp. Rep. No. 2, 
158. Wolfe P. (1960), Accelerating the cutting-plane method for nonlinear programming, Rand. Corp. Paper No. P-2010. MR0134354
159. Wolfe P. (1961), Accelerating the cutting-plane method for nonlinear programming, JSIAM 9, 481-488. MR0134354
160. Wolfe P. (1962), Recent developments in nonlinear programming, Advance in computers, vol. 3, Acad. Press. New York, 155-187. MR0146002
161. Wolfowitz (1952), On the stochastic approximation method of Robbins and Monro, . AMS 23, 457-561. MR0050242
162. Wolfowitz (1956), On stochastic approximation methods, AMS 27, 1151 - 1155. MR0086437
163. Zoutendijk G. (1959), Maximizing a function in a convex region, JRSS B 21, 338 - 355. MR0122578
164. Zoutendijk G. (1960), Methods of feasible directions, Elsevier Publ. Co., Amsterdam-London -New York -Princeton. MR0129119

1. Juraj Hrivňák, Jeden typ jednoparametrových nelineárnych číslicových regulačných obvodov