AI
New Framework Makes Neural Network Pruning Scale-Invariant Using Projective Geometry
Researchers introduce IPPRO, a pruning method that overcomes the fundamental flaw of magnitude-based filter selection by embedding networks in projective space.
A fundamental problem undermines current neural network pruning approaches: filter magnitude is an unreliable measure of importance because identical filters can appear drastically different in importance simply due to rescaling. Researchers have now proposed IPPRO, a scale-invariant pruning framework designed to overcome this limitation.
The Core Problem#
Most structured pruning methods rely on filter magnitude as a proxy for determining which components to remove. However, this approach breaks down because of scale invariance—mathematically identical filters operating at different scales receive wildly different importance scores, making the selection arbitrary and inconsistent.
How IPPRO Works#
The new method uses projective geometry to reframe the pruning problem. By embedding filters into real projective space (ℝP^N), IPPRO resolves the mathematical singularity at the origin, placing all filters at equal angular distance from the zero filter. This creates a geometric foundation where importance is measured by something more reliable than magnitude.
The framework introduces PROscore, which captures functional importance by measuring how much a filter would collapse toward zero under a single gradient step (directional collapse). The researchers further connect PROscore to exact L₀ relaxation, demonstrating that this one-shot criterion can reliably forecast multi-step pruning dynamics.
Strong Experimental Results#
Testing across diverse architectures—CNNs (ResNet), Vision Transformers (DeiT), and large language models (LLaMA)—IPPRO consistently outperforms existing methods. The gains are particularly pronounced under high compression rates and in scenarios where networks cannot be fine-tuned after pruning, suggesting the method captures something fundamental about filter utility.
Why It Matters#
IPPRO establishes a robust, architecture-agnostic approach to neural network compression grounded in principled mathematical foundations rather than ad-hoc heuristics. This could make model compression more reliable and efficient across diverse applications.
