Exploring the impact of post-training rounding in regression models

Jan Kalina

Applications of Mathematics (2024)

  • Volume: 69, Issue: 2, page 257-271
  • ISSN: 0862-7940

Abstract

top
Post-training rounding, also known as quantization, of estimated parameters stands as a widely adopted technique for mitigating energy consumption and latency in machine learning models. This theoretical endeavor delves into the examination of the impact of rounding estimated parameters in key regression methods within the realms of statistics and machine learning. The proposed approach allows for the perturbation of parameters through an additive error with values within a specified interval. This method is elucidated through its application to linear regression and is subsequently extended to encompass radial basis function networks, multilayer perceptrons, regularization networks, and logistic regression, maintaining a consistent approach throughout.

How to cite

top

Kalina, Jan. "Exploring the impact of post-training rounding in regression models." Applications of Mathematics 69.2 (2024): 257-271. <http://eudml.org/doc/299255>.

@article{Kalina2024,
abstract = {Post-training rounding, also known as quantization, of estimated parameters stands as a widely adopted technique for mitigating energy consumption and latency in machine learning models. This theoretical endeavor delves into the examination of the impact of rounding estimated parameters in key regression methods within the realms of statistics and machine learning. The proposed approach allows for the perturbation of parameters through an additive error with values within a specified interval. This method is elucidated through its application to linear regression and is subsequently extended to encompass radial basis function networks, multilayer perceptrons, regularization networks, and logistic regression, maintaining a consistent approach throughout.},
author = {Kalina, Jan},
journal = {Applications of Mathematics},
keywords = {supervised learning; trained model; perturbations; effect of rounding; low-precision arithmetic},
language = {eng},
number = {2},
pages = {257-271},
publisher = {Institute of Mathematics, Academy of Sciences of the Czech Republic},
title = {Exploring the impact of post-training rounding in regression models},
url = {http://eudml.org/doc/299255},
volume = {69},
year = {2024},
}

TY - JOUR
AU - Kalina, Jan
TI - Exploring the impact of post-training rounding in regression models
JO - Applications of Mathematics
PY - 2024
PB - Institute of Mathematics, Academy of Sciences of the Czech Republic
VL - 69
IS - 2
SP - 257
EP - 271
AB - Post-training rounding, also known as quantization, of estimated parameters stands as a widely adopted technique for mitigating energy consumption and latency in machine learning models. This theoretical endeavor delves into the examination of the impact of rounding estimated parameters in key regression methods within the realms of statistics and machine learning. The proposed approach allows for the perturbation of parameters through an additive error with values within a specified interval. This method is elucidated through its application to linear regression and is subsequently extended to encompass radial basis function networks, multilayer perceptrons, regularization networks, and logistic regression, maintaining a consistent approach throughout.
LA - eng
KW - supervised learning; trained model; perturbations; effect of rounding; low-precision arithmetic
UR - http://eudml.org/doc/299255
ER -

References

top
  1. Agresti, A., Foundations of Linear and Generalized Linear Models, Wiley Series in Probability and Statistics. John Wiley & Sons, Hoboken (2015). (2015) Zbl1309.62001MR3308143
  2. Blokdyk, G., Artificial Neural Network: A Complete Guide, 5STARCooks, Toronto (2021). (2021) 
  3. Carroll, R. J., Ruppert, D., Stefanski, L. A., Crainiceanu, C. M., 10.1201/9781420010138, Monographs on Statistics and Applied Probability 105. Chapman & Hall/CRC, Boca Raton (2006). (2006) Zbl1119.62063MR2243417DOI10.1201/9781420010138
  4. Croci, M., Fasi, M., Higham, N. J., Mary, T., Mikaitis, M., 10.1098/rsos.211631, R. Soc. Open Sci. 9 (2022), Article ID 211631, 25 pages. (2022) DOI10.1098/rsos.211631
  5. Egrioglu, E., Bas, E., Karahasan, O., 10.1007/s41066-022-00345-y, Granul. Comput. 8 (2023), 491-501. (2023) DOI10.1007/s41066-022-00345-y
  6. Fasi, M., Higham, N. J., Mikaitis, M., Pranesh, S., 10.7717/peerj-cs.330, PeerJ Computer Sci. 7 (2021), Article ID e330, 19 pages. (2021) DOI10.7717/peerj-cs.330
  7. Gao, F., Li, B., Chen, L., Shang, Z., Wei, X., He, C., 10.1016/j.ultras.2020.106344, Ultrasonics 112 (2021), Article ID 106344, 8 pages. (2021) DOI10.1016/j.ultras.2020.106344
  8. Greene, W. H., Econometric Analysis, Pearson Education, Harlow (2018). (2018) 
  9. Hastie, T., Tibshirani, R., Wainwright, R., 10.1201/b18401, Monographs on Statistics and Applied Probability 143. CRC Press, Boca Raton (2015). (2015) Zbl1319.68003MR3616141DOI10.1201/b18401
  10. Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., Peste, A., Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks, J. Mach. Learn. Res. 22 (2021), Article ID 241, 124 pages. (2021) Zbl07626756MR4329820
  11. Kalina, J., Tichavský, J., 10.2478/msr-2020-0002, Measurement Sci. Rev. 20 (2020), 6-14. (2020) DOI10.2478/msr-2020-0002
  12. Kalina, J., Vidnerová, P., Soukup, L., 10.1007/978-981-99-2074-7_125, Handbook of Metrology and Applications Springer, Singapore (2023), 2355-2376. (2023) DOI10.1007/978-981-99-2074-7_125
  13. Louizos, C., Reisser, M., Blankevoort, T., Gavves, E., Welling, M., 10.48550/arXiv.1810.01875, Available at https://arxiv.org/abs/1810.01875 (2018), 14 pages. (2018) DOI10.48550/arXiv.1810.01875
  14. Maddox, W. J., Potapczynski, A., Wilson, A. G., Low-precision arithmetic for fast Gaussian processes, Proc. Mach. Learn. Res. 180 (2022), 1306-1316. (2022) 
  15. Nagel, M., Fournarakis, M., Amjad, R. A., Bondarenko, Y., Baalen, M. van, Blankevoort, T., 10.48550/arXiv.2106.08295, Available at https://arxiv.org/abs/2106.08295 (2021), 27 pages. (2021) DOI10.48550/arXiv.2106.08295
  16. Park, J.-H., Kim, K.-M., Lee, S., 10.1145/3524066, ACM Trans. Embedded Comput. Syst. 21 (2022), Article ID 60, 22 pages. (2022) DOI10.1145/3524066
  17. Pillonetto, G., 10.1016/j.automatica.2018.03.065, Automatica 93 (2018), 321-332. (2018) Zbl1400.93316MR3810919DOI10.1016/j.automatica.2018.03.065
  18. Riazoshams, H., Midi, H., Ghilagaber, G., 10.1002/9781119010463, John Wiley & Sons, Hoboken (2019). (2019) Zbl1407.62022MR3839600DOI10.1002/9781119010463
  19. Saleh, A. K. M. E., Picek, J., Kalina, J., 10.1007/s00184-010-0328-2, Metrika 75 (2012), 311-328. (2012) Zbl1239.62081MR2909549DOI10.1007/s00184-010-0328-2
  20. Seghouane, A.-K., Shokouhi, N., 10.1109/TCYB.2019.2951811, IEEE Trans. Cybernetics 51 (2021), 2847-2856. (2021) DOI10.1109/TCYB.2019.2951811
  21. Shultz, K. S., Whitney, D., Zickar, M. J., 10.4324/9781315869834, Routledge, New York (2020). (2020) DOI10.4324/9781315869834
  22. Šíma, J., Vidnerová, P., Mrázek, V., 10.1007/978-3-031-44204-9_16, Artificial Neural Networks and Machine Learning -- ICANN 2023 Lecture Notes in Computer Science 14263. Springer, Cham (2023), 186-198. (2023) MR4776700DOI10.1007/978-3-031-44204-9_16
  23. Smucler, E., Yohai, V. J., 10.1016/j.csda.2017.02.002, Comput. Stat. Data Anal. 111 (2017), 116-130. (2017) Zbl1464.62164MR3630222DOI10.1016/j.csda.2017.02.002
  24. Sze, V., Chen, Y.-H., Yang, T.-J., Emer, J. S., 10.1109/JPROC.2017.2761740, Proc. IEEE 105 2017 (2295-2329). (2017) MR3784727DOI10.1109/JPROC.2017.2761740
  25. Víšek, J.Á., Consistency of the least weighted squares under heteroscedasticity, Kybernetika 47 (2011), 179-206. (2011) Zbl1220.62064MR2828572
  26. Wang, N., Choi, J., Brand, D., Chen, C.-Y., Gopalakrishnan, K., 10.5555/3327757.3327866, NIPS'18: Proceedings of the 32nd International Conference on Neural Information Processing Systems Curran Associates, New York (2018), 7686-7695. (2018) DOI10.5555/3327757.3327866
  27. Yan, W. Q., 10.1007/978-981-99-4823-9, Texts in Computer Science. Springer, Singapore (2023). (2023) Zbl7783714MR4660076DOI10.1007/978-981-99-4823-9
  28. Yu, J., Anitescu, M., 10.1007/s10107-019-01421-z, Math. Program. 185 (2021), 37-76 9999DOI99999 10.1007/s10107-019-01421-z . (2021) Zbl1458.62158MR4201708DOI10.1007/s10107-019-01421-z
  29. Zhang, R., Wilson, A. G., Sa, C. De, Low-precision stochastic gradient Langevin dynamics, Proc. Mach. Learn. Res. 162 (2022), 26624-26644. (2022) 

NotesEmbed ?

top

You must be logged in to post comments.

To embed these notes on your page include the following JavaScript code on your page where you want the notes to appear.

Only the controls for the widget will be shown in your chosen language. Notes will be shown in their authored language.

Tells the widget how many notes to show per page. You can cycle through additional notes using the next and previous controls.

    
                

                
Note: Best practice suggests putting the JavaScript code just before the closing </body> tag.