AI
Researchers rethink vision networks: new FFNet design rivals transformers with simple convolutions
A new architecture called FFNet challenges the dominance of transformer attention mechanisms by showing that feed-forward layers can be reimagined as efficient token mixers using only convolutional operations.
Computer vision researchers have long treated self-attention as the linchpin of modern neural networks, but a new paper from arXiv suggests the overlooked component—the feed-forward layer—may be equally crucial to performance.
Rethinking the Feed-Forward Layer#
The feed-forward network (FFN), present in virtually every transformer and deep learning model, has historically played a supporting role in the shadow of attention mechanisms. Yet recent research indicates that feed-forward layers function similarly to key-value memory systems. This observation prompted a team to ask whether the real source of transformer power lies not in attention per se, but in the broader query-key-value framework that both components share.
To test this hypothesis, the researchers developed a concept called FFNification: converting self-attention into an efficient token mixer built entirely from large-kernel convolutions, while maintaining the core query-key-value structure. Instead of softmax (used in traditional attention), the new design employs GELU activation.
The FFNet Architecture#
The resulting approach, termed Fast-Forward Networks (FFNet), combines two key modules: an "FFNified attention" mechanism that learns to detect locally distributed spatial patterns like a memory system, paired with convolutional blocks that operate across different spatial dimensions. Despite relying only on straightforward components—no complex specializations—FFNet achieves competitive performance across multiple domains with notable efficiency improvements.
A Generalized Framework#
Building on these findings, the team proposes MetaMixer: a flexible mixer architecture that leaves the specific operations within the query-key-value framework open to design, rather than prescribing them in advance. This generalization suggests that the query-key-value paradigm itself may be more fundamental to competitive vision models than the particular mechanisms (attention, convolution, or otherwise) used to realize it.
