Researchers uncover new non-Siamese layers for permutation-equivariant neural networks

Machine learning researchers have re-examined how neural networks can be built to respect permutation symmetries—situations where the order of input elements shouldn't matter—using a pure mathematical lens rather than conventional parameter-sharing tricks.

A New Angle on Equivariance#

The team approached the problem through irreducible representations and Schur's lemma, mathematical tools from group theory. This framework yielded cleaner derivations for well-known models like DeepSets, 2-IGN graph networks, and Deep Weight Space (DWS) architectures. The DWS derivation, in particular, turns out to be substantially simpler than prior treatments.

Expanding Beyond Siamese Networks#

The researchers then tackled a harder problem: unaligned symmetric sets, where equivariance to wreath products of groups is needed. Earlier work had imposed tight restrictions that forced nearly all equivariant layers into a Siamese (twin-network) structure. The new analysis shows this constraint was unnecessary—there exist many additional non-Siamese equivariant layers that prior frameworks had missed.

Practical Gains#

Testing these newly-identified layers on real tasks showed tangible benefits:

• Graph anomaly detection improved with the additional design flexibility
• Weight space alignment (aligning neural network parameters across different models) performed better
• Learning Wasserstein distance functions became more efficient

The work opens up a wider design space for practitioners building permutation-invariant and equivariant models, moving beyond the limited Siamese bottleneck that constrained earlier approaches.