Exponential expressivity of ReLU k neural networks on Gevrey classes with point singularities

Joost A. A. Opschoor; Christoph Schwab

Applications of Mathematics (2024)

  • Volume: 69, Issue: 5, page 695-724
  • ISSN: 0862-7940

Abstract

top
We analyze deep Neural Network emulation rates of smooth functions with point singularities in bounded, polytopal domains D ⊂ ℝ d , d = 2 , 3 . We prove exponential emulation rates in Sobolev spaces in terms of the number of neurons and in terms of the number of nonzero coefficients for Gevrey-regular solution classes defined in terms of weighted Sobolev scales in D , comprising the countably-normed spaces of I. M. Babuška and B. Q. Guo. As intermediate result, we prove that continuous, piecewise polynomial high order (“ p -version”) finite elements with elementwise polynomial degree p ∈ ℕ on arbitrary, regular, simplicial partitions of polyhedral domains D ⊂ ℝ d , d ≥ 2 , can be exactly emulated by neural networks combining ReLU and ReLU 2 activations. On shape-regular, simplicial partitions of polytopal domains D , both the number of neurons and the number of nonzero parameters are proportional to the number of degrees of freedom of the h p finite element space of I. M. Babuška and B. Q. Guo.

How to cite

top

Opschoor, Joost A. A., and Schwab, Christoph. "Exponential expressivity of ${\rm ReLU}^k$ neural networks on Gevrey classes with point singularities." Applications of Mathematics 69.5 (2024): 695-724. <http://eudml.org/doc/299319>.

@article{Opschoor2024,
abstract = {We analyze deep Neural Network emulation rates of smooth functions with point singularities in bounded, polytopal domains $\{\rm D\} \subset \mathbb \{R\}^d$, $d=2,3$. We prove exponential emulation rates in Sobolev spaces in terms of the number of neurons and in terms of the number of nonzero coefficients for Gevrey-regular solution classes defined in terms of weighted Sobolev scales in $\{\rm D\}$, comprising the countably-normed spaces of I. M. Babuška and B. Q. Guo. As intermediate result, we prove that continuous, piecewise polynomial high order (“$p$-version”) finite elements with elementwise polynomial degree $p\in \mathbb \{N\} $ on arbitrary, regular, simplicial partitions of polyhedral domains $\{\rm D\} \subset \mathbb \{R\}^d$, $d\ge 2$, can be exactly emulated by neural networks combining ReLU and ReLU$^2$ activations. On shape-regular, simplicial partitions of polytopal domains $\{\rm D\}$, both the number of neurons and the number of nonzero parameters are proportional to the number of degrees of freedom of the $hp$ finite element space of I. M. Babuška and B. Q. Guo.},
author = {Opschoor, Joost A. A., Schwab, Christoph},
journal = {Applications of Mathematics},
keywords = {neural network; $hp$-finite element method; singularities; Gevrey regularity; exponential convergence},
language = {eng},
number = {5},
pages = {695-724},
publisher = {Institute of Mathematics, Academy of Sciences of the Czech Republic},
title = {Exponential expressivity of $\{\rm ReLU\}^k$ neural networks on Gevrey classes with point singularities},
url = {http://eudml.org/doc/299319},
volume = {69},
year = {2024},
}

TY - JOUR
AU - Opschoor, Joost A. A.
AU - Schwab, Christoph
TI - Exponential expressivity of ${\rm ReLU}^k$ neural networks on Gevrey classes with point singularities
JO - Applications of Mathematics
PY - 2024
PB - Institute of Mathematics, Academy of Sciences of the Czech Republic
VL - 69
IS - 5
SP - 695
EP - 724
AB - We analyze deep Neural Network emulation rates of smooth functions with point singularities in bounded, polytopal domains ${\rm D} \subset \mathbb {R}^d$, $d=2,3$. We prove exponential emulation rates in Sobolev spaces in terms of the number of neurons and in terms of the number of nonzero coefficients for Gevrey-regular solution classes defined in terms of weighted Sobolev scales in ${\rm D}$, comprising the countably-normed spaces of I. M. Babuška and B. Q. Guo. As intermediate result, we prove that continuous, piecewise polynomial high order (“$p$-version”) finite elements with elementwise polynomial degree $p\in \mathbb {N} $ on arbitrary, regular, simplicial partitions of polyhedral domains ${\rm D} \subset \mathbb {R}^d$, $d\ge 2$, can be exactly emulated by neural networks combining ReLU and ReLU$^2$ activations. On shape-regular, simplicial partitions of polytopal domains ${\rm D}$, both the number of neurons and the number of nonzero parameters are proportional to the number of degrees of freedom of the $hp$ finite element space of I. M. Babuška and B. Q. Guo.
LA - eng
KW - neural network; $hp$-finite element method; singularities; Gevrey regularity; exponential convergence
UR - http://eudml.org/doc/299319
ER -

References

top
  1. Arora, R., Basu, A., Mianjy, P., Mukherjee, A., 10.48550/arXiv.1611.01491, Available at https://arxiv.org/abs/1611.01491 (2018), 17 pages. (2018) DOI10.48550/arXiv.1611.01491
  2. Babuška, I., Guo, B. Q., 10.1137/0519014, SIAM J. Math. Anal. 19 (1988), 172-203. (1988) Zbl0647.35021MR0924554DOI10.1137/0519014
  3. Babuška, I., Guo, B. Q., 10.1137/0725048, SIAM J. Numer. Anal. 25 (1988), 837-861. (1988) Zbl0655.65124MR0954788DOI10.1137/0725048
  4. Babuška, I., Guo, B. Q., 10.1137/0520054, SIAM J. Math. Anal. 20 (1989), 763-781. (1989) Zbl0706.35028MR1000721DOI10.1137/0520054
  5. Babuška, I., Guo, B. Q., 10.1016/0045-7825(95)00946-9, Comput. Methods Appl. Mech. Eng. 133 (1996), 319-346. (1996) Zbl0882.65096MR1399640DOI10.1016/0045-7825(95)00946-9
  6. Banjai, L., Melenk, J. M., Schwab, C., 10.1007/s00211-022-01329-5, Numer. Math. 153 (2023), 1-47. (2023) Zbl1511.65117MR4530200DOI10.1007/s00211-022-01329-5
  7. Banjai, L., Melenk, J. M., Schwab, C., 10.1093/imanum/drac070, IMA J. Numer. Anal. 43 (2023), 3282-3325. (2023) Zbl07800835MR4673336DOI10.1093/imanum/drac070
  8. Bao, G., Hu, G., Liu, D., 10.1016/j.jcp.2012.04.002, J. Comput. Phys. 231 (2012), 4967-4979. (2012) Zbl1245.65125DOI10.1016/j.jcp.2012.04.002
  9. Bolley, P., Dauge, M., Camus, J., 10.1080/03605308508820383, Commun. Partial Differ. Equations 10 (1985), 391-431 French. (1985) Zbl0573.35024MR0784683DOI10.1080/03605308508820383
  10. Cancès, E., Chakir, R., Maday, Y., 10.1007/s10915-010-9358-1, J. Sci. Comput. 45 (2010), 90-117. (2010) Zbl1203.65237MR2679792DOI10.1007/s10915-010-9358-1
  11. Chemin, J.-Y., 10.1093/oso/9780198503972.001.0001, Oxford Lecture Series in Mathematics and Its Applications 14. Clarendon Press, New York (1998). (1998) Zbl0927.76002MR1688875DOI10.1093/oso/9780198503972.001.0001
  12. Chen, Q., Babuška, I., 10.1016/0045-7825(96)01051-1, Comput. Methods Appl. Mech. Eng. 137 (1996), 89-94. (1996) Zbl0877.65004MR1420062DOI10.1016/0045-7825(96)01051-1
  13. Costabel, M., Dauge, M., Nicaise, S., 10.1051/m2an:1999155, M2AN, Math. Model. Numer. Anal. 33 (1999), 627-649. (1999) Zbl0937.78003MR1713241DOI10.1051/m2an:1999155
  14. Costabel, M., Dauge, M., Nicaise, S., 10.1142/S0218202512500157, Math. Models Methods Appl. Sci. 22 (2012), Article ID 1250015, 63 pages. (2012) Zbl1257.35056MR2928103DOI10.1142/S0218202512500157
  15. Dubiner, M., 10.1007/BF01060030, J. Sci. Comput. 6 (1991), 345-390. (1991) Zbl0742.76059MR1154903DOI10.1007/BF01060030
  16. Elbrächter, D., Grohs, P., Jentzen, A., Schwab, C., 10.1007/s00365-021-09541-6, Constr. Approx. 55 (2022), 3-71. (2022) Zbl1500.35009MR4376559DOI10.1007/s00365-021-09541-6
  17. Ern, A., Guermond, J.-L., 10.1007/978-3-030-56341-7, Texts in Applied Mathematics 72. Springer, Cham (2021). (2021) Zbl1476.65003MR4242224DOI10.1007/978-3-030-56341-7
  18. Faustmann, M., Marcati, C., Melenk, J. M., Schwab, C., 10.1137/22M152493X, SIAM J. Numer. Anal. 61 (2023), 2601-2622. (2023) Zbl1533.65224MR4667264DOI10.1137/22M152493X
  19. Feischl, M., Schwab, C., 10.1007/s00211-019-01085-z, Numer. Math. 144 (2020), 323-346. (2020) Zbl1433.65290MR4057425DOI10.1007/s00211-019-01085-z
  20. Fournais, S., Hoffmann-Ostenhof, M., Hoffmann-Ostenhof, T., Sørensen, T. Ø., 10.1088/1751-8113/42/31/315208, J. Phys. A. Math. Theor. 42 (2009), Article ID 315208, 11 pages. (2009) Zbl1181.35282MR2521310DOI10.1088/1751-8113/42/31/315208
  21. Guo, B., The h - p Version of Finite Element Method in Two Dimensions. Mathematical Theory and Computational Experience: Ph.D. Thesis, University of Maryland, College Park (1985). (1985) MR2634330
  22. Guo, B. Q., Babuška, I., 10.1006/aama.1993.1016, Adv. Appl. Math. 14 (1993), 307-347. (1993) Zbl0790.35028MR1228743DOI10.1006/aama.1993.1016
  23. Guo, B., Babuška, I., 10.1017/S0308210500023520, Proc. R. Soc. Edinb., Sect. A 127 (1997), 77-126. (1997) Zbl0874.35019MR1433086DOI10.1017/S0308210500023520
  24. Guo, B., Babuška, I., 10.1017/S0308210500029899, Proc. R. Soc. Edinb., Sect. A 127 (1997), 517-545. (1997) Zbl0884.35022MR1453280DOI10.1017/S0308210500029899
  25. Guo, B., Schwab, C., 10.1016/j.cam.2005.02.018, J. Comput. Appl. Math. 190 (2006), 487-519. (2006) Zbl1121.35098MR2209521DOI10.1016/j.cam.2005.02.018
  26. He, J., Li, L., Xu, J., Zheng, C., 10.4208/jcm.1901-m2018-0160, J. Comput. Math. 38 (2020), 502-527. (2020) Zbl1463.68072MR4087799DOI10.4208/jcm.1901-m2018-0160
  27. He, J., Xu, J., 10.48550/arXiv.2312.14276, Available at https://arxiv.org/abs/2312.14276 (2024), 29 pages. (2024) DOI10.48550/arXiv.2312.14276
  28. He, Y., Marcati, C., Schwab, C., 10.1137/22M1527428, SIAM J. Math. Anal. 56 (2024), 2488-2520. (2024) Zbl07849793MR4719953DOI10.1137/22M1527428
  29. Herrmann, L., Opschoor, J. A. A., Schwab, C., 10.1007/s10915-021-01718-2, J. Sci. Comput. 90 (2022), Article ID 75, 37 pages. (2022) Zbl1500.41002MR4362469DOI10.1007/s10915-021-01718-2
  30. Hesthaven, J. S., 10.1137/S003614299630587X, SIAM J. Numer. Anal. 35 (1998), 655-676. (1998) Zbl0933.41004MR1618874DOI10.1137/S003614299630587X
  31. Holm, B., Wihler, T. P., 10.1007/s00211-017-0918-2, Numer. Math. 138 (2018), 767-799. (2018) Zbl1453.65118MR3767700DOI10.1007/s00211-017-0918-2
  32. Li, B., Tang, S., Yu, H., 10.4208/cicp.OA-2019-0168, Commun. Comput. Phys. 27 (2020), 379-411. (2020) Zbl1474.65031MR4040947DOI10.4208/cicp.OA-2019-0168
  33. Longo, M., Opschoor, J. A. A., Disch, N., Schwab, C., Zech, J., 10.1016/j.neunet.2023.06.008, Neural Netw. 165 (2023), 721-739. (2023) Zbl1532.65083DOI10.1016/j.neunet.2023.06.008
  34. Maday, Y., Marcati, C., 10.1142/S0218202519500295, Math. Models Methods Appl. Sci. 29 (2019), 1585-1617. (2019) Zbl1431.65209MR3986800DOI10.1142/S0218202519500295
  35. Marcati, C., Opschoor, J. A. A., Petersen, P. C., Schwab, C., 10.1007/s10208-022-09565-9, Found. Comput. Math. 23 (2023), 1043-1127. (2023) Zbl1519.35253MR4603228DOI10.1007/s10208-022-09565-9
  36. Marcati, C., Schwab, C., 10.1137/19M1247334, SIAM J. Math. Anal. 52 (2020), 2945-2968. (2020) Zbl1447.35252MR4113068DOI10.1137/19M1247334
  37. Melenk, J. M., 10.1007/b84212, Lecture Notes in Mathematics 1796. Springer, Berlin (2002). (2002) Zbl1021.65055MR1939620DOI10.1007/b84212
  38. Melenk, J. M., Schwab, C., 10.1137/S0036142997317602, SIAM J. Numer. Anal. 35 (1998), 1520-1557. (1998) Zbl0972.65093MR1626030DOI10.1137/S0036142997317602
  39. Opschoor, J. A. A., 10.3929/ethz-b-000614671, ETH, Zürich (2023). (2023) DOI10.3929/ethz-b-000614671
  40. Opschoor, J. A. A., Petersen, P. C., Schwab, C., 10.1142/S0219530519410136, Anal. Appl., Singap. 18 (2020), 715-770. (2020) Zbl1452.65354MR4131037DOI10.1142/S0219530519410136
  41. Opschoor, J. A. A., Schwab, C., 10.1016/j.camwa.2024.06.008, Comput. Math. Appl. 169 (2024), 142-162. (2024) Zbl7894795MR4768195DOI10.1016/j.camwa.2024.06.008
  42. Opschoor, J. A. A., Schwab, C., Xenophontos, C., 10.48550/arXiv.2401.06656, Available at https://arxiv.org/abs/2401.06656 (2024), 39 pages. (2024) DOI10.48550/arXiv.2401.06656
  43. Opschoor, J. A. A., Schwab, C., Zech, J., 10.1007/s00365-021-09542-5, Constr. Approx. 55 (2022), 537-582. (2022) Zbl1500.41008MR4376568DOI10.1007/s00365-021-09542-5
  44. Petersen, P., Voigtlaender, F., 10.1016/j.neunet.2018.08.019, Neural Netw. 108 (2018), 296-330. (2018) Zbl1434.68516DOI10.1016/j.neunet.2018.08.019
  45. Samarskii, A. A., Galaktionov, V. A., Kurdyumov, S. P., Mikhailov, A. P., 10.1515/9783110889864.535, de Gruyter Expositions in Mathematics 19. Walter De Gruyter, Berlin (1995). (1995) Zbl1020.35001MR1330922DOI10.1515/9783110889864.535
  46. Schwab, C., p - and h p -Finite Element Methods: Theory and Applications in Solid and Fluid Mechanics, Numerical Mathematics and Scientific Computation. Clarendon Press, Oxford (1998). (1998) Zbl0910.73003MR1695813
  47. Szabó, B., Babuška, I., Finite Element Analysis, John Wiley & Sons, New York (1991). (1991) Zbl0792.73003MR1164869
  48. Yang, Y., Wu, Y., Yang, H., Xiang, Y., 10.48550/arXiv.2310.10766, Available at https://arxiv.org/abs/2310.10766 (2023), 32 pages. (2023) DOI10.48550/arXiv.2310.10766
  49. Yarotsky, D., 10.1016/j.neunet.2017.07.002, Neural Netw. 94 (2017), 103-114. (2017) Zbl1429.68260DOI10.1016/j.neunet.2017.07.002

NotesEmbed ?

top

You must be logged in to post comments.

To embed these notes on your page include the following JavaScript code on your page where you want the notes to appear.

Only the controls for the widget will be shown in your chosen language. Notes will be shown in their authored language.

Tells the widget how many notes to show per page. You can cycle through additional notes using the next and previous controls.

    
                

                
Note: Best practice suggests putting the JavaScript code just before the closing </body> tag.