Iterative Feedback Strengthens LLM Diagram Generation from Natural Language

A team of researchers has investigated how large language models can be improved at converting natural-language descriptions into structured diagrams, specifically focusing on activity diagrams used in software and process modeling. Their findings suggest that adding a loop of evaluation and refinement—where the generated diagram is checked for errors and then revised—substantially boosts both the structural correctness and semantic fidelity of the output.

The LADEX Pipeline and Two Types of Validation#

The researchers developed LADEX (LLM-based Activity Diagram Extractor), a system that translates process descriptions into activity diagrams through a multi-stage pipeline. The approach addresses two key challenges: ensuring the diagram follows formal well-formedness rules (structural correctness) and verifying it accurately reflects the original text's intent (semantic alignment).

LADEX can perform structural validation in two ways: either through algorithm-based rule checking or by asking another LLM call to identify problems. Semantic checks rely on LLM evaluation, since these require understanding meaning. By testing five different configurations of the system, the researchers isolated the effects of each component.

Testing Against Real-World Benchmarks#

The team evaluated their system using two datasets: a public collection and proprietary data from Ciena, an industry collaborator. To measure success, they employed two independent assessment methods—one based on tracing execution paths through the diagram and another using LLM-driven matching—and measured both correctness (whether all essential diagram elements were present) and completeness (how much of the target diagram was captured).

Key Findings: When Critique Helps Most#

The experiments revealed three main insights. First, the critique-refine cycle demonstrably improved the quality of generated diagrams across multiple metrics compared to single-pass generation. Second, LLM-based semantic validation played a meaningful role in refinement. Most tellingly, algorithmic checks proved superior to LLM-based checks for enforcing structural rules, with algorithmically-validated diagrams achieving consistent well-formedness while LLM-validated ones frequently retained errors. Both evaluation methods yielded comparable results, suggesting the findings are robust.