New benchmark measures how well AI models convert written instructions into software diagrams

A gap in LLM evaluation#

Large language models have become increasingly adept at translating natural language descriptions into structured technical formats—a skill particularly valuable in software development. However, assessing whether these models actually produce correct diagrams has been challenging due to the absence of standardized benchmarks and evaluation methods. Researchers have now introduced a new tool to fill this gap.

What MermaidSeqBench does#

The benchmark, called MermaidSeqBench, contains 132 carefully curated samples designed to evaluate how well LLMs generate Mermaid sequence diagrams (visual tools commonly used in software engineering to document system interactions) from text descriptions. The dataset was assembled using a three-part approach: human experts created and verified initial examples, automated systems expanded these with variations, and rule-based techniques further diversified the test cases.

Evaluation approach#

Rather than relying on simple yes-or-no correctness, the benchmark employs an LLM-as-a-judge model to measure performance across multiple dimensions:

• Syntax correctness: Does the generated diagram follow proper Mermaid formatting?
• Activation handling: Are entity interactions properly sequenced?
• Error handling: Does the diagram gracefully represent exception scenarios?
• Practical usability: Can the output actually be used in real workflows?

Key findings#

Early testing across multiple state-of-the-art LLMs revealed substantial performance variations between models and different evaluation approaches, highlighting that the ability to generate structured diagrams remains inconsistent across the current generation of language models.

Why this matters#

MermaidSeqBench establishes a foundation for rigorously measuring LLM capabilities in structured generation tasks. By enabling systematic and reproducible evaluation, the benchmark helps clarify where models excel and where they struggle, ultimately advancing how AI systems are assessed on technical document generation.