New Framework Brings Mathematical Guarantees to Quantized Neural Network Training

Training neural networks with extremely reduced numerical precision has long posed a theoretical puzzle: the discrete parameter spaces and non-convex loss surfaces create obstacles that existing methods struggle to navigate. A new approach published on arXiv offers a principled solution with provable mathematical guarantees.

The Core Challenge#

Quantization—reducing neural network parameters to just 1 or 2 bits—promises major efficiency gains for deployment. Yet the underlying optimization problem becomes intractable: standard training methods treat parameters as continuous values that can be infinitesimally adjusted, while quantized networks require discrete jumps. The researchers identified a fundamental geometric problem: even in heavily overparameterized networks, the best performing solutions can remain mathematically isolated from one another, unreachable by gradient-based exploration.

A New Formulation#

The team developed an Exact Quadratic Constrained Binary Optimization (QCBO) framework that reformulates the quantization problem as a convex optimization problem with a completely positive structure. By carefully mapping finite-depth architectures with bounded activation states into this framework, they achieve a mathematical formulation that preserves the global optimum with zero approximation error. This contrasts with typical relaxation methods that introduce unavoidable gaps.

To handle realistic dataset scales, they introduced Decomposed Lower-Bound Optimization (DLBO), which reduces the optimization workload from processing entire datasets at once to solving smaller single-sample subproblems. The method constructs a hierarchy of bounds inspired by Hamiltonian locality, allowing practitioners to trade problem complexity against solution tightness.

Experimental Validation#

The researchers demonstrated their approach on a coherent Ising machine—hardware specifically designed to solve binary optimization problems. Their system achieved 94.95% accuracy on a binary classification task (distinguishing coats from sandals in Fashion-MNIST) at 1.1-bit precision. Multi-class experiments on variants of Fashion-MNIST, Wine, and Digits datasets confirmed the method scales to more complex scenarios. The results suggest the approach resists the typical collapse in accuracy that occurs when compressing networks to extreme precision levels.