AI
New benchmark dataset helps AI extract structured knowledge from full NLP research papers
Researchers release SciNLP, the first annotated dataset enabling machines to identify entities and relationships across entire scientific publications.
Automatically pulling structured information from academic papers remains challenging for machine learning models, particularly when documents span multiple sections and pages. A new benchmark dataset called SciNLP aims to improve this capability by providing carefully labeled examples of entities and their relationships throughout complete NLP research articles.
What SciNLP offers#
The dataset contains 60 full-text NLP research papers that have been manually annotated with entity and relationship labels. Across these documents, researchers identified and tagged 6,429 entities and 1,649 relations. This marks a significant departure from prior work in the field, as most existing datasets focus only on specific sections of papers—such as abstracts or introductions—rather than entire documents. The full-text approach captures the complexity and richness of how concepts are distributed and discussed throughout academic publications.
Validation and performance insights#
To test SciNLP's utility, the team benchmarked several state-of-the-art extraction models against it. Their experiments revealed that different models perform with varying degrees of success depending on paper length and complexity. When compared head-to-head with existing datasets, SciNLP delivered notable performance gains for certain baseline models, suggesting it provides a sufficiently distinct and valuable training signal for improving extraction systems.
Building knowledge graphs from papers#
The researchers used models trained on SciNLP to automatically construct a knowledge graph specific to the NLP domain. The resulting graph exhibits an average node degree of 3.3 per entity, indicating that it captures dense semantic connections and structured topological relationships. This richness in the knowledge representation has practical value for downstream applications that depend on organized, navigable scientific knowledge.
The dataset is now publicly available for researchers interested in developing and evaluating information extraction systems on academic texts. View the paper on arXiv.
