LLMs show promise but inconsistency as knowledge graph builders, new research finds

Researchers have put seven leading large language models to the test to assess their ability to automatically build knowledge graphs—structured representations of information linking facts and entities. The results offer a mixed picture: while modern LLMs generally extract accurate, document-faithful statements with minimal hallucination, their performance across different stages of graph construction varies considerably.

The research approach#

The team introduced a framework called Detail-to-Abstract Hierarchical Knowledge Graph (D2A-HKG) that breaks knowledge graph construction into three sequential stages: an initial extraction phase, a splitting phase to organize information, and an abstraction phase to consolidate and generalize the extracted facts. They then evaluated the resulting graphs on both their semantic quality (how meaningful and accurate the relationships are) and their structural properties (how well-organized the graph is).

The evaluation used a specialized dataset called CMW-Lit, drawn from published research articles focused on children's mental well-being. This dataset presented a particularly rigorous test case because the source material contains diverse types of evidence, multiple interconnected factors, and relationships described with statistical nuance and qualification—realistic characteristics that make graph construction challenging.

What the models actually accomplished#

The leading language models tested demonstrated a notable ability to stay faithful to source documents while extracting relationship triples (subject-predicate-object statements). Instances of fabricated or unsupported information remained surprisingly limited. However, the analysis also uncovered substantial differences in how each model approached different phases of the task, suggesting that LLMs may struggle with consistency when asked to progressively refine and abstract extracted information.

Why this matters#

The findings provide insight into where modern LLMs excel and where they encounter difficulties in a practical, knowledge-intensive application. The researchers have released both the CMW-Lit dataset and the knowledge graphs they generated during the study, positioning them as useful resources for future research and as starting points for domain experts to refine and validate before deployment in real-world applications.