New method breaks down annotation disagreement into analyzable layers

When humans label data for machine learning tasks, they often disagree—and figuring out why can be surprisingly difficult. A paper on arXiv proposes a new approach to disentangle these disagreements by breaking annotation tasks into separate conceptual layers, making it easier to spot patterns in where disagreement actually occurs.

How the approach works#

The method decomposes an annotation task into distinct levels, then examines inter-annotator disagreement within each level independently. The researchers describe two complementary strategies:

• Theory-driven decomposition: A researcher uses domain knowledge to manually design layers based on what they already understand about the annotation task
• Exploratory decomposition: The system automatically generates and presents many possible decompositions, allowing researchers to discover latent structures they might not have anticipated

Demonstration and impact#

The team tested both strategies using a dataset built for narrative analysis, showing how the approach can validate hypotheses about disagreement sources and expose hidden relationships in annotation schemes. Rather than treating disagreement as a monolithic problem, the framework reveals which specific aspects of a task drive confusion between annotators.

Looking ahead#

The authors outline ways to extend and adapt their approach for different purposes, suggesting the method could become useful beyond the initial application to narrative analysis.