New framework evaluates how well AI models can generate business process diagrams

A team of computer scientists has created a systematic way to evaluate whether large language models can reliably generate Business Process Model and Notation (BPMN) diagrams—a critical skill for organizations that need to document and automate their workflows.

The Challenge of Automating Process Modeling#

Creating BPMN models is traditionally labor-intensive work requiring both specialist knowledge of business operations and familiarity with formal modeling standards. Advances in LLMs have opened the door to generating these diagrams directly from plain language descriptions, promising to speed up what is usually a time-consuming manual process. But until now, there has been no rigorous way to assess whether LLM-generated models are actually any good.

Introducing BEF4LLM#

The researchers developed BEF4LLM, a novel evaluation framework that examines model quality from four distinct angles:

• Syntactic quality — whether the diagram follows BPMN grammar rules correctly
• Pragmatic quality — whether it matches what the user intended to describe
• Semantic quality — whether it captures the true business logic and meaning
• Validity — overall soundness and usability of the model

Key Findings#

Testing open-source LLMs against outputs from human modeling experts revealed a nuanced picture. Language models demonstrated strong capabilities in syntactic and pragmatic dimensions, performing competitively with professionals. However, humans consistently outperformed AI systems when it came to semantic accuracy—understanding and faithfully representing the deeper business logic embedded in process descriptions.

The performance gaps, while meaningful, were smaller than expected, suggesting LLMs have significant potential for practical deployment despite these current limitations.

Why It Matters#

The framework provides both immediate insights into LLM strengths and weaknesses for process modeling, and guidance for future development. As organizations explore automating diagram creation, understanding exactly where these models excel and where they stumble is essential for deciding when and how to trust them with real business workflows.