LibMoE offers open-source benchmarking for Mixture of Experts in large language models

Mixture of Experts (MoE) architectures have emerged as a critical technology for building more capable language models—powering systems like GPT-OSS, DeepSeek-V3, Llama-4, and Gemini-2.5. Yet the field faces a major obstacle: the extreme computational expense of training and evaluating these systems has left MoE research largely inaccessible to academic teams and smaller organizations.

Introducing LibMoE#

Researchers have released LibMoE, an open-source unified framework designed to lower the barrier to entry for MoE research. The library supports multiple training approaches, including both from-scratch pretraining and sparse-upcycling (where existing models are modified to use MoE structures). Beyond simply providing implementations, LibMoE includes analytical tools for understanding how routing mechanisms and expert utilization evolve during training.

Three areas of investigation#

Using the framework, the team conducted a wide-ranging analysis across three dimensions:

• Routing dynamics: How experts are selected, whether routing remains stable over time, and what patterns emerge in how routing entropy reveals task specialization and expert diversity
• Router initialization: How subtle differences in how the routing mechanism is initialized affect load balancing and early expert utilization patterns
• Training regimes: How the routing behavior and stability differ when models are trained from scratch versus when existing models are adapted to use sparse routing

Impact#

By standardizing evaluation methods and providing accessible tools, LibMoE aims to enable more researchers to study MoE systems systematically. The framework and comprehensive analysis are available on GitHub and intended to guide future work on scaling and optimizing these increasingly important architectures.