AI
New Benchmark Reveals How Far AI Still Lags in Generating Production-Ready Simulink Models
SimuVerity tests whether AI agents can build functional engineering models, not just syntactically correct ones—and exposes why structural similarity isn't enough.
Researchers have introduced SimuVerity, a rigorous testing framework that evaluates how well AI agents can generate Simulink models—the block-diagram software widely used in control systems, signal processing, and embedded systems engineering. The benchmark reveals a sobering gap between what existing tools measure and what engineers actually need.
The Problem with Today's Benchmarks#
Most current Simulink benchmarks assess whether generated models compile and run, or how closely they visually resemble a target design. These metrics miss the crucial question: do the models satisfy their engineering specifications? A model can execute without errors while failing to meet accuracy requirements, stability constraints, or robustness across real-world operating conditions.
How SimuVerity Works#
The new benchmark comprises 101 text-to-code tasks spanning ten engineering domains. Each task pairs a natural-language specification with executable system profiles that ground the actual engineering requirements. Rather than relying on a single reference model, SimuVerity includes four families of native simulation scenarios that test the generated models under diverse conditions.
Evaluation proceeds in layers:
- Qualification stage: Does the model compile, execute, and meet basic engineering criteria?
- Performance scoring: Across six dimensions—accuracy, output quality, mechanistic fidelity, control and causal integrity, robustness across operating domains, and dynamic response characteristics.
The Results#
The research team tested six agent systems against SimuVerity. The best-performing system achieved an overall score of only 42.86 out of 100—a stark reminder that generating production-ready engineering models remains an unsolved problem. The findings split into two revealing insights:
- Structural similarity is misleading: Models that look similar to reference designs often fail on engineering grounds.
- Multi-dimensional gaps: Capability bottlenecks emerge both in producing models that qualify in the first place, and in satisfying the full range of performance metrics afterward.
Notably, some high-scoring models still exhibited severe visual-layout disorder, suggesting that appearance and functional correctness can diverge sharply.
