Multi-step-length gradient iterative method for separable nonlinear least squares problems

Hai-Rong Cui; Jing Lin; Jian-Nan Su

Kybernetika (2024)

  • Issue: 2, page 197-209
  • ISSN: 0023-5954

Abstract

top
Separable nonlinear least squares (SNLLS) problems are critical in various research and application fields, such as image restoration, machine learning, and system identification. Solving such problems presents a challenge due to their nonlinearity. The traditional gradient iterative algorithm often zigzags towards the optimal solution and is sensitive to the initial guesses of unknown parameters. In this paper, we improve the convergence rate of the traditional gradient method by implementing a multi-step-length gradient iterative algorithm. Moreover, we incorporate the variable projection (VP) strategy, taking advantage of the separable structure observed in SNLLS problems. We propose a multi-step-length gradient iterative-based VP (Mul-GI-VP) method to solve such nonlinear optimization problems. Our simulation results verify the feasibility and high efficiency of the proposed algorithm.

How to cite

top

Cui, Hai-Rong, Lin, Jing, and Su, Jian-Nan. "Multi-step-length gradient iterative method for separable nonlinear least squares problems." Kybernetika (2024): 197-209. <http://eudml.org/doc/299351>.

@article{Cui2024,
abstract = {Separable nonlinear least squares (SNLLS) problems are critical in various research and application fields, such as image restoration, machine learning, and system identification. Solving such problems presents a challenge due to their nonlinearity. The traditional gradient iterative algorithm often zigzags towards the optimal solution and is sensitive to the initial guesses of unknown parameters. In this paper, we improve the convergence rate of the traditional gradient method by implementing a multi-step-length gradient iterative algorithm. Moreover, we incorporate the variable projection (VP) strategy, taking advantage of the separable structure observed in SNLLS problems. We propose a multi-step-length gradient iterative-based VP (Mul-GI-VP) method to solve such nonlinear optimization problems. Our simulation results verify the feasibility and high efficiency of the proposed algorithm.},
author = {Cui, Hai-Rong, Lin, Jing, Su, Jian-Nan},
journal = {Kybernetika},
keywords = {separable nonlinear least squares; multi-step-length gradient iterative method; variable projection algorithm; image restoration},
language = {eng},
number = {2},
pages = {197-209},
publisher = {Institute of Information Theory and Automation AS CR},
title = {Multi-step-length gradient iterative method for separable nonlinear least squares problems},
url = {http://eudml.org/doc/299351},
year = {2024},
}

TY - JOUR
AU - Cui, Hai-Rong
AU - Lin, Jing
AU - Su, Jian-Nan
TI - Multi-step-length gradient iterative method for separable nonlinear least squares problems
JO - Kybernetika
PY - 2024
PB - Institute of Information Theory and Automation AS CR
IS - 2
SP - 197
EP - 209
AB - Separable nonlinear least squares (SNLLS) problems are critical in various research and application fields, such as image restoration, machine learning, and system identification. Solving such problems presents a challenge due to their nonlinearity. The traditional gradient iterative algorithm often zigzags towards the optimal solution and is sensitive to the initial guesses of unknown parameters. In this paper, we improve the convergence rate of the traditional gradient method by implementing a multi-step-length gradient iterative algorithm. Moreover, we incorporate the variable projection (VP) strategy, taking advantage of the separable structure observed in SNLLS problems. We propose a multi-step-length gradient iterative-based VP (Mul-GI-VP) method to solve such nonlinear optimization problems. Our simulation results verify the feasibility and high efficiency of the proposed algorithm.
LA - eng
KW - separable nonlinear least squares; multi-step-length gradient iterative method; variable projection algorithm; image restoration
UR - http://eudml.org/doc/299351
ER -

References

top
  1. Aravkin, A. Y., Drusvyatskiy, D., Leeuwen, T. van, , IEEE Trans. Automat. Control 63 (2017), 7, 2131-2138. MR3820213DOI
  2. Chen, G. Y., Gan, M., Chen, C. P., Chen, L., 10.1109/TIM.2018.2826798, IEEE Trans. Instrument. Measur. 67 (2018), 11, 2518-2525. DOI10.1109/TIM.2018.2826798
  3. Chen, G. Y., Gan, M., Chen, C. P., Li, H. X., , IEEE Trans. Cybernet. 51 (2021), 2, 614-623. DOI
  4. Chen, G. Y., Wang, S. Q., Gan, M., Chen, C. P., , IEEE Trans. Imagw Process. 30 (2021), 2, 1207-1218. MR4206220DOI
  5. Chen, J., Ding, F., , J. Vibration Control 17 (2011), 9, 1281-1286. MR2859041DOI
  6. Chen, J., Ding, F., Liu, Y., Zhu, Q., , Systems Control Lett. 115 (2018), 15-21. MR3786116DOI
  7. Chung, J., G, J., Nagy, , SIAM J. Scientif. Comput. 31 (2010), 6, 4654-4674. MR2594997DOI
  8. Cornelio, A., Piccolomini, E. L., Nagy, J. G., , Numer. Algorithms 65 (2014) 1, 23-42. MR3147677DOI
  9. Ding, F., , IET Control Theory Appl. 7 (2013), 1, 68-79. MR3088190DOI
  10. Ding, F., Liu, G., Liu, X. P., , IEEE Trans. Automat. Control 55 (2010), 8, 1976-1981. MR2681302DOI
  11. Ding, F., Liu, Y., Bao, B., , Proc. Inst. Mechanic. Engrs, Part I: J. Systems Control Engrg. 226 (2012), 1, 43-55. DOI
  12. Erichson, N. B., Zheng, P., Manohar, K., Brunton, S. L., Kutz, J. N., Aravkin, A. Y., , SIAM J. Appl. Math. 80 (2020), 2, 977-1002. MR4091178DOI
  13. Gan, M., Chen, C. P., Chen, G. Y., Chen, L., , IEEE Trans. Cybernet. 48 (2018), 10, 2866-2874. DOI
  14. Gan, M., Guan, Y., Chen, G. Y., Chen, C. P., Recursive variable projection algorithm for a class of separable nonlinear models., IEEE Trans. Neural Netw. Learn. Syst. (2020). MR4332261
  15. Gan, M., Li, H. X., Peng, H., , IEEE Trans. Cybernet. 45 (2014), 3, 462-471. DOI
  16. Gan, M., Peng, H., , Appl. Soft Comput. 12 (2012), 1, 174-181. DOI
  17. Golub, G., Pereyra, V., , Inverse Problems 19 (2003), R1. MR1991786DOI
  18. Golub, G. .H, Pereyr, V., , SIAM J. Numer. Snal. 10 (1973), 2, 413-432. MR0336980DOI
  19. Haggan, V., Ozaki, T., , Biometrika 68 (1981), 1, 189-196. MR0614955DOI
  20. Hansen, P. Ch., Nagy, J. G., O'leary, D. P., Deblurring images: matrices, spectra, and filtering., SIAM 2006. MR2271138
  21. Hussu, A, , Int. J. Control 15 (1972), 1, 79-82. DOI
  22. Kaufman, L., , BIT Numer. Math. 15 (1975), 1, 49-57. MR0501738DOI
  23. Li, J., Zheng, Y., Lin, Z., , Syst. Control Lett. 66 (2014), 104-110. MR3179836DOI
  24. Li, M., Abubakar, A., Gao, F., Habashy, T. M, , IEEE Trans. Antennas Propag. 64 (2015), 1, :332-335. MR3455550DOI
  25. Liu, Y., Ding, F., Shi, Y., , Automatica 50 (2014), 3, 962-970. MR3173998DOI
  26. Okatani, T., Deguchi, K., , Int. J. Comput. Vision 72 (2007), 3, 329-337. DOI
  27. R, M., Osborne, Smyth, G. K., , SIAM J. Scient. Comput. 16 (1995), 1, :119-138. MR1311681DOI
  28. Peng, H., Ozaki, T., Toyoda, Y., Shioya, H., Nakano, K., Haggan-Ozaki, V., Mori, M., , Control Engrg. Practice 12 (2004), 2, 191-203. DOI
  29. Ruhe, A., Wedin, Pe. A., , SIAM Rev. 22 (1980), 3, 318-337. MR0584380DOI
  30. Sjoberg, J., Viberg, M., Separable non-linear least-squares minimization-possible improvements for neural net fitting., In: Neural Networks for Signal Processing VII. Proc. 1997 IEEE Signal Processing Society Workshop, IEEE 1997, pp. 345-354. 
  31. Stathopoulos, G., Korda, M., Jones, C. N., , IEEE Trans. Automat. Control. 62 (2016), 4, 1752-1767. MR3636331DOI
  32. Yang, H., Gao, J., Wu, Z., , IEEE Signal Process. Lett. 15 (2008), 289-292. DOI
  33. Yu, D., Chen, C. P., Xu, H., , IEEE Trans. Systems Man Cybernet.: Systems 52 (2021), 4, 2262-2274. DOI
  34. Yu, D., Xu, H., Chen, C. P., Bai, W., Wang, Z., Dynamic coverage control based on k-means., IEEE Trans. Industr. Electron. 2021. 
  35. Zeng, J., He, T., Wang, M., , Systems Control Lett. 107 (2017), 36-43. MR3692336DOI

NotesEmbed ?

top

You must be logged in to post comments.

To embed these notes on your page include the following JavaScript code on your page where you want the notes to appear.

Only the controls for the widget will be shown in your chosen language. Notes will be shown in their authored language.

Tells the widget how many notes to show per page. You can cycle through additional notes using the next and previous controls.

    
                

                
Note: Best practice suggests putting the JavaScript code just before the closing </body> tag.