GPTKB v1.5 Builds 100-Million-Triple Knowledge Base from GPT-4 to Map LLM Factual Knowledge

A new demonstration paper introduces GPTKB v1.5, a knowledge base containing 100 million triples of structured information extracted from GPT-4 to provide unprecedented visibility into what facts language models actually know.

What GPTKB v1.5 Is#

The 100-million-triple knowledge base was built for approximately $14,000 using a technique called massive-recursive LLM knowledge materialization. The system queries GPT-4 repeatedly and systematically to extract facts, then organizes them into an interlinked database that researchers and developers can explore.

How It Works#

The GPTKB platform enables three distinct ways to investigate LLM knowledge:

• Link traversal: Users can browse factual relationships by navigating from one entity to another, similar to following hyperlinks through a web of interconnected information
• Structured querying: The knowledge base supports SPARQL queries, a formal language for database searches, allowing researchers to ask complex questions across the entire dataset
• Comparative analysis: The system facilitates side-by-side examination of where language models excel and where they fall short in their factual understanding

Why This Matters#

Language models remain somewhat opaque in terms of their actual factual knowledge. While they produce text that often seems knowledgeable, understanding precisely what information they've internalized—and how reliably they can retrieve it—has been difficult to assess systematically. By materializing this knowledge into a queryable database, researchers gain tools for rigorous statistical analysis and methodical exploration of model capabilities and limitations.

The methodology also opens new possibilities for automated knowledge base construction, where LLMs themselves become sources for populating large structured databases of facts.