Abstract
Learning neural networks has long been known to be difficult. One of the causes of such difficulties is thought to be the equilibrium points caused by the symmetry between the weights of the neural network. Such an equilibrium point is known to delay neural network training. However, neural networks have been widely used in recent years largely because of the development of methods that make learning easier. One such technique is batch normalization, which is empirically known to speed up learning. Therefore, if the equilibrium point due to symmetry truly affects the neural network learning, and batch normalization speeds up the learning, batch normalization should help escape from such equilibrium points. Therefore, we analyze whether batch normalization helps escape from such equilibrium points by a method called statistical mechanical analysis. By examining the eigenvalue of the Hessian matrix of the generalization error at the equilibrium point, we find that batch normalization delays escape from poor equilibrium points. This contradicts the empirically known finding of speeding up learning, and we discuss why we obtained this result.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Amari, S.: Natural gradient works efficiently in learning. Neural Comput. 10(2), 251–276 (1998)
Amari, S., Ozeki, T., Karakida, R., Yoshida, Y., Okada, M.: Dynamics of learning in MLP: natural gradient and singularity revisited. Neural Comput. 30(1), 1–33 (2018)
Amari, S., Park, H., Ozeki, T.: Singularities affect dynamics of learning in neuromanifolds. Neural Comput. 18(5), 1007–1065 (2006)
Arora, S., Li, Z., Lyu, K.: Theorical analysis of auto rate-tuning by batch normalization. arXiv preprint arXiv:1812.03981 (2018)
Arpit, D., et al.: A closer look at memorization in deep networks. In: Proceedings of the 34th International Conference on Machine Learning (2017)
Biehl, M., Schwarze, H.: Learning by on-line gradient descent. J. Phys. A: Math. Gen. 28(3), 643 (1995)
Bjorck, J., Gomes, G., Selman, B., Weinberger, K.Q.: Understanding batch normalization. In: Advances in Neural Information Processing Systems, vol. 31 (2018)
Chaudhari, P., Soatto, S.: Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. In: 6th International Conference on Learning Representations (2018)
Cousseau, F., Ozeki, T., Amari, S.: Dynamics of learning in multilayer perceptrons near singularities. IEEE Trans. Neural Netw. 19(8), 1313–1328 (2008)
Fukumizu, K.: A regularity condition of the information matrix of a multilayer perception network. Neural Netw. 9(5), 871–879 (1996)
Fukumizu, K., Amari, S.: Local minima and plateaus in hierarchical structures of multilayer perceptrons. Neural Netw. 13, 317–327 (2000)
Goldt, S., Advani, M.S., Saxe, A.M., Krzakala, F., Zdeborova, L.: Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Goldt, S., Mezard, M., Krzakala, F., Zdeborova, L.: Modelling the influence of data structure on learning in neural networks: the hidden manifold model. arXiv preprint arXiv:1909.11500 (2019)
Gunasekar, S., Woodworth, B.E., Bhojanapalli, S., Neyshabur, B., Srebro, N.: Implicit regularization in matrix factorization. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. In: Advances in Neural Information Processing Systems, vol. 28 (2015)
Hassibi, B., Stork, D.G.: Second order derivatives for network pruning: optimal brain surgeon. In: Advances in Neural Information Processing Systems, vol. 6 (1993)
Hochreiter, S., Schmidhuber, J.: Flat minima. Neural Comput. 9(1), 1–42 (1997)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd International Conference on Machine Learning (2015)
Jastrzebski, S., et al.: Three factors influencing minima in SGD. arXiv preprint arXiv:1711.04623 (2017)
Karakida, R., Akaho, S., Amari, S.: The normalization method for alleviating pathological sharpness in wide neural networks. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Karakida, R., Akaho, S., Amari, S.: Universal statistics of fisher information in deep neural networks: mean field approach. In: Chaudhuri, K., Sugiyama, M. (eds.) Proceedings of Machine Learning Research, 16–18 April 2019, vol. 89, pp. 1032–1041. PMLR (2019)
Keskar, N.S., Mudigere, D., Nocedal, J., Smelyanskiy, M., Tang, P.T.P.: On large-batch training for deep learning: generalization gap and sharp minima. In: 5th International Conference on Learning Representations (2017)
Kohler, J., Daneshmand, H., Lucchi, A., Zhou, M., Neymeyr, K., Hofmann, T.: Exponential convergence rates for batch normalization: the power of length-direction decoupling in non-convex optimization. arXiv preprint arXiv:1805.10694 (2018)
LeCun, Y., Denker, J.S., Solla, S.A.: Optimal brain damage. In: Advances in Neural Information Processing Systems, vol. 3 (1990)
Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 (2016)
Luo, P., Wang, X., Shao, W., Peng, Z.: Towards understanding regularization in batch normalization. arXiv preprint arXiv:1809.00846 (2018)
Mandt, S., Hoffman, M., Blei, D.: A variational analysis of stochastic gradient algorithms. In: Proceedings of the 33nd International Conference on Machine Learning (2016)
Neyshabur, B., Tomioka, R., Salakhutdinov, R., Srebro, N.: Geometry of optimization and implicit regularization in deep learning. arXiv preprint arXiv:1705.03071 (2017)
Neyshabur, B., Tomioka, R., Srebro, N.: In search of the real inductive bias: on the role of implicit regularization in deep learning. In: 3rd International Conference on Learning Representations (2015)
Riegler, P., Biehl, M.: On-line backpropagation in two-layered neural networks. J. Phys. A 28, L507–L513 (1995)
Saad, D., Solla, S.A.: Dynamics of on-line gradient descent learning for multilayer neural networks. In: Advances in Neural Information Processing Systems, vol. 8 (1995)
Saad, D., Solla, S.A.: Exact solution for on-line learning in multilayer neural networks. Phys. Rev. Lett. 74(41), 4337–4340 (1995)
Saad, D., Solla, S.A.: On-line learning in soft committee machines. Phys. Rev. E 52(4), 4225–4243 (1995)
Santurkar, S., Tsipras, D., Ilyas, A., Mardy, A.: How does batch normalization help optimization? arXiv preprint arXiv:1805.11604 (2018)
Schwarze, H.: Learning a rule in a multilayer neural network. J. Phys. A 26, 5781–5794 (1993)
Seung, H.S., Somopolinsky, H., Tishby, N.: Statistical mechanics of learning from examples. Phys. Rev. A 45(8), 6056–6091 (1992)
Watanabe, S.: Algebraic geometrical methods for hierarchical learning machines. Neural Netw. 14(8), 1049–1060 (2001)
Watanabe, S., Amari, S.: Learning coefficients of layered models when the true distribution mismatches the singularities. Neural Comput. 15(5), 1011–1033 (2003)
Wei, H., Amari, S.: Dynamics of learning near singularities in radial basis function networks. Neural Netw. 21(7), 989–1005 (2008)
Wei, H., Zhang, J., Cousseau, F., Ozeki, T., Amari, S.: Dynamics of learning in multilayer perceptrons near singularities. Neural Comput. 20(3), 813–842 (2008)
West, A.H.L., Saad, D., Nabney, I.T.: The learning dynamics of a universal approximator. In: Advances in Neural Information Processing Systems, vol. 9 (1996)
Yoshida, Y., Karakida, R., Okada, M., Amari, S.: Statistical mechanical analysis of learning dynamics of two-layer perceptron with multiple output units. J. Phys. A 52(18), 184002 (2019)
Yoshida, Y., Okada, M.: Data-dependence of plateau phenomenon in learning with neural network – statistical mechanical analysis. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Takagi, S., Yoshida, Y., Okada, M. (2020). The Effect of Batch Normalization in the Symmetric Phase. In: Farkaš, I., Masulli, P., Wermter, S. (eds) Artificial Neural Networks and Machine Learning – ICANN 2020. ICANN 2020. Lecture Notes in Computer Science(), vol 12397. Springer, Cham. https://doi.org/10.1007/978-3-030-61616-8_19
Download citation
DOI: https://doi.org/10.1007/978-3-030-61616-8_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-61615-1
Online ISBN: 978-3-030-61616-8
eBook Packages: Computer ScienceComputer Science (R0)