New benchmark tests AI's ability to generate software architectures from requirements

Software architecture represents the foundational design blueprint for any system, defining structural choices that ripple through implementation and affect overall quality. Yet the automated generation of architecture designs from requirement documents has remained largely unexplored, with few specialized benchmarks or evaluation frameworks available.

A new research effort aims to fill this gap with R2ABench, a purpose-built benchmark for evaluating requirement-to-architecture (R2A) reasoning tasks. The work provides empirical insights into where current AI systems succeed and where they fall short in this critical design activity.

Dataset and Evaluation Framework#

The R2ABench dataset encompasses 68 software projects drawn from both academic settings and open-source repositories on GitHub. Each project entry supplies a formal software requirements specification (SRS), a reference architecture diagram rendered in PlantUML format, and the underlying requirement artifacts for ground-truth comparison.

To assess generated architectures, the researchers developed a multi-layered evaluation methodology that operates across three dimensions:

• Syntax and readability validation — ensuring diagrams are valid and parseable
• Structural graph analysis — examining whether component relationships match expected patterns
• Semantic scoring — measuring whether generated designs capture requirement coverage and create traceable links between requirements and architecture choices

Key Findings on Current LLM Performance#

The empirical evaluation of state-of-the-art language models and agent systems reveals a nuanced performance profile. These systems generally produce syntactically correct and human-readable architecture diagrams. However, structural accuracy breaks down when handling relationship modeling between components.

A striking asymmetry emerges: models identify system components with relative confidence, but consistently underperform when recovering the connections (edges) between those components. The most common failure mode is edge hallucination—incorrectly inventing relationships that do not exist. Beyond structural issues, generated architectures often fail to maintain traceability back to the original requirements, suggesting a disconnect between what the model draws and what the requirements actually specify.