Theory explains double descent in neural networks using path regularization

A new theoretical framework for multilayer neural networks offers fresh insights into why training dynamics behave differently across model sizes. The work centers on path regularization, an approach to constraining neural network training that appears to outperform traditional weight decay methods.

What the research addresses#

The researchers have developed what they call a near-complete nonasymptotic generalization theory—essentially a set of mathematical bounds that predict how well trained networks will perform on unseen data without assuming infinite model size. This matters because most existing theory relies on asymptotic analysis, which assumes parameters grow arbitrarily large.

A key distinction: the new theory does not require the loss function to be bounded, a standard assumption in the field that limits practical applicability. The work also removes requirements that network width, depth, or other properties must scale to infinity.

Core findings and implications#

The team provides an explicit error upper bound for multilayer networks with certain activation functions and Lipschitz-continuous losses. Their framework takes into account both approximation error—how well a network can theoretically fit a function—and generalization error, the gap between training and test performance.

Notably, the bounds exhibit the double descent phenomenon, a counterintuitive behavior where performance first gets worse as model capacity increases, then improves again at even larger scales. This characteristic aligns the theory with real deep learning practice, setting it apart from previous nonasymptotic results that fail to capture this effect.

The work also claims near-minimax optimality for regression tasks with ReLU activations, suggesting the bounds are tight. The researchers indicate their framework may eventually explain neural network scaling laws.

Future directions#

The authors note that proving matching lower bounds—demonstrating these bounds are as tight as theoretically possible—could reveal the true mechanism underlying double descent. This represents a significant open question they plan to address.