Skip to content
MAR
Sign inRequest access
AI

Research proposes framework to improve crowdsourced data labeling by vetting labelers with expert input

A new method combines crowdsourced labels with strategically placed expert validation to identify which crowd workers are reliable and which data points are genuinely difficult to label.

MrBarisCMrBarisC@barisMarsel TeamCreator
CrawledarXiv cs.AI

Crowdsourced labeling has become essential for preparing large datasets, but a persistent problem undermines its usefulness: determining which crowd workers produce accurate annotations and which do not. When too many unreliable labelers participate, label quality can collapse abruptly in what researchers call a "phase transition."

The Challenge with Crowd Labor#

Building high-quality labeled datasets at scale requires tremendous effort and expense. Crowdsourcing has made this feasible, yet it introduces a difficult quality-control problem. Workers vary widely in reliability—some are diligent, others careless or deliberately malicious. When the proportion of poor-quality labelers climbs past a certain threshold, the overall accuracy of aggregated labels can drop sharply and nonlinearly, making it hard to salvage the data.

A New Approach: Expert-Validated Crowd Labeling#

Researchers have introduced a framework called Expert Label Injected Crowd Estimation (ELICE) to address this instability. The method works by strategically inserting expert-labeled examples into the crowd-labeling workflow. These verified labels serve two purposes: they help measure the competence of individual crowd workers and identify which instances in the dataset are inherently difficult to classify correctly.

By combining this intelligence about labeler ability and instance difficulty with standard aggregation techniques, ELICE can delay or prevent the phase transition that typically degrades label quality. The framework also offers variants that further improve resilience.

Practical Impact#

Empirical testing demonstrates that ELICE outperforms existing methods. The research also establishes a quantifiable lower bound on how many expert-labeled examples are needed to reliably estimate worker quality and dataset characteristics while achieving better overall label accuracy. This gives practitioners concrete guidance on resource allocation.

Category: Human-Computer Interaction
Source: arXiv:1607.02174