Researchers Release Major Dataset for Testing AI-Generated System Design Diagrams

A research team has unveiled SEMAADB, a substantial resource aimed at measuring how well large language models can generate coherent sets of system engineering diagrams. The dataset contains 3,000 engineering scenarios paired with 15,000 diagrams formatted in SysML (Systems Modeling Language), a standard used by engineers to document complex systems.

The Challenge of Multi-Diagram Consistency#

Systems engineers typically work with multiple interconnected diagrams—each capturing a different view of the same system—and must ensure these views remain logically consistent. When language models generate diagrams automatically, they can produce syntactically correct output, but maintaining consistency across related diagrams poses a more difficult challenge that has not been systematically measured until now. Existing benchmarks and datasets do not directly evaluate this multi-diagram coherence at scale.

Dataset Design and Validation#

Each scenario in SEMAADB contains five linked SysML views: Requirement, Block Definition, Activity, State Machine, and Sequence. Every diagram has been checked for both consistency and valid rendering. The researchers manually verified a subset of 100 contexts to establish a gold-standard test set, ensuring the benchmark reflects real-world quality standards.

Performance on Two Key Tasks#

The team evaluated three language models on two practical problems. In diagram repair—where models fix semantic errors in flawed diagrams—the best-performing system corrected 64.3% of issues. For cross-diagram updates, where a single change must propagate correctly across related diagrams, the leading model achieved an F1 score of 80.7%.

What the Results Reveal#

The findings show an important split in model capabilities: fixing surface-level syntax errors is nearly solved, but identifying and resolving semantic mistakes and maintaining consistency across connected diagrams remain substantially harder problems. SEMAADB thus serves both as a large-scale resource for training and as a measurement tool for tracking progress on these real engineering challenges.