Plain Transformers Emerge as Effective Graph Neural Networks Without Added Complexity

Transformers have become the foundation of modern machine learning, powering systems across language, vision, and now potentially graph-based learning. A new research paper suggests that rather than building specialized graph Transformers with complex modifications, plain Transformers can handle graph tasks just as effectively—if researchers make just three straightforward architectural adjustments.

The Problem with Specialized Graph Transformers#

Most attempts to adapt Transformers for graph learning have introduced substantial architectural departures from the original design. These variants often layer in message-passing mechanisms borrowed from graph neural networks or implement intricate attention modifications. Such deviations make it harder for graph learners to benefit from training improvements and innovations developed for standard Transformers in language and vision domains. The field has grown increasingly fragmented, with each specialized variant requiring its own optimization and tuning strategies.

A Simpler Path Forward#

The proposed approach, called Powerful Plain Graph Transformers (PPGT), maintains the fundamental Transformer structure while incorporating three minimal changes:

• Simplified L₂ attention — replacing the standard scaled-dot-product attention with a measure that captures magnitude closeness between tokens rather than cosine similarity
• Adaptive root-mean-square normalization — a technique that retains information about token magnitudes, which standard normalization discards
• MLP-based graph positional encoding — a lightweight stem that encodes structural position information into the model

Performance and Expressivity#

The research demonstrates that PPGT achieves strong empirical expressivity on standard graph benchmarks, holding its own against more elaborate alternatives such as subgraph-based graph neural networks and higher-order message-passing approaches. The theoretical expressivity of the design aligns with what's observed in practice, suggesting the modifications target the right architectural bottlenecks.

Why This Matters#

This finding reinforces the versatility of the plain Transformer design and opens the door to a genuinely unified backbone that works across language, vision, and graph modalities. Rather than maintaining separate, specialized architectures for each domain, practitioners could rely on a single core framework with domain-specific preprocessing and minor parameter adjustments—making it easier to transfer improvements across fields and accelerate progress in all of them.