New Framework Helps LLMs Extract Knowledge Without Manual Prompts

Extracting structured knowledge from raw text without predefined categories remains a thorny challenge for large language models. While LLMs excel at in-context learning through carefully designed prompts, they struggle to adapt automatically when faced with new domains and shifting linguistic contexts—and developers typically don't have the ground-truth labels needed to spot and fix errors.

The Core Problem#

Current approaches to open-domain relational triplet extraction rely on static, manually written prompts that don't adjust for different input domains. This rigidity leads to two interconnected issues: the model generates unsupported triplets that don't match the source text, and without gold-standard annotations, it becomes nearly impossible to identify and correct these failures. Additionally, when LLMs generate relation names freely, they produce non-standard variations that fracture knowledge graph consistency.

A Self-Improving Solution#

Researchers have introduced Knowledge Restoration-driven Prompt Optimization (KRPO), a framework that adapts prompts automatically without requiring labeled training data. The approach works by converting extracted triplets back into natural-language statements, then comparing those reconstructions against the original source text to measure semantic alignment. This intrinsic feedback—derived purely from the model's own outputs—becomes the signal for refinement.

The system then translates this feedback into actionable guidance that optimizes prompts iteratively across batches of inputs. Separately, KRPO adds a Memory-augmented Relation Canonicalizer component that normalizes the free-form relation strings the model produces, mapping them to a dynamically updated schema to enforce consistency across the knowledge graph.

Demonstrated Impact#

Testing on three open-domain triplet extraction benchmarks with multiple LLM architectures showed KRPO achieved the highest average F1 scores across different evaluation settings, validating both the prompt adaptation and relation canonicalization strategies.