Unveiling the Hessian's Connection to the Decision Boundary

Understanding why some neural network minima generalize better than others is a fundamental challenge in deep learning. To analyse this question, we bridge two perspectives: the analysis of the geometric complexity of decision boundaries in input space and the spectral properties of the Hessian of the training loss in parameter space. We show that the top eigenvectors of the Hessian encode the decision boundary, with the number of spectral outliers correlating with its complexity, a finding consistent across datasets and architectures. This insight leads to a formulation of a proxy generalization measure based on alignment between training gradients and Hessian eigenvectors. Additionally, as the measure is blind to simplicity bias, we develop a novel margin estimation technique that, in combination with the generalization measure, helps analyse the generalisation capabilities of neural networks trained on toy and real datasets.