A Geometric Interpretation of Stochastic Gradient Descent

We start with a review of the main steps of the Deep Learning algorithm, together with some historical remarks. We then concentrate on the key ingredient, stochastic gradient descent (SGD), whose geometric significance appears elusive and was modelled using the SDE Fokker Planck by Chaudhari and Soatto. We then study a deterministic model in which the trajectories of our dynamical systems are described via geodesics of a family of metrics arising from the diffusion matrix (natural gradient method). These metrics encode information about the highly non-isotropic gradient noise in SGD. This is a joint work with S. Soatto (UCLA, Amazon) and P. Chaudhari (U. Penn.)