New Platform Automates Linguistic Annotation Tasks Using AI Models

Researchers have long struggled with the labor-intensive process of manually annotating text for linguistic analysis. A new tool called LinguistAgent aims to ease this burden by automating the annotation of complex language phenomena using a combination of large language models and reflective review mechanisms.

The Challenge#

Data labeling represents a persistent constraint in language research and the social sciences, particularly when tasks require nuanced human judgment—such as identifying metaphors or other figurative language. While large language models show genuine capability for these kinds of language tasks, there remains a considerable distance between what they can theoretically do and what researchers can realistically apply to their work.

How LinguistAgent Works#

The system operates through a dual-model design: an initial annotator that performs the tagging, paired with an optional reviewer that evaluates and refines those tags. This two-stage process mimics the human peer-review workflow. The platform lets researchers compare results across three distinct methodologies:

• Prompt Engineering (including zero-shot, few-shot, and chain-of-thought variants)
• Retrieval-Augmented Generation (combining retrieval and generation)
• Fine-tuning (customizing models for specific tasks)

Validation and Results#

The researchers tested LinguistAgent by recreating a published metaphor-identification study. The system evaluated its output in real time at the token level, measuring performance using standard metrics like F1 score and Cohen's kappa against human-annotated reference data. This approach provides concrete benchmarks for assessing how well the automated system matches human judgment.

Open Research#

The application and underlying code have been made publicly available, enabling other researchers to build on the work and adapt the platform for their own annotation needs.