Researchers achieve 90% accuracy in identifying text varieties across 16 languages

Linguists and machine learning researchers have developed a system that automatically identifies the register—the contextual style and purpose of written text—across the open web in 16 languages simultaneously. The work reveals both the potential and the inherent limitations of teaching machines to distinguish between text varieties like news articles, forum discussions, and academic papers.

Building a multilingual taxonomy#

The team created the Multilingual CORE corpora, a dataset of over 72,000 documents labeled according to a hierarchical taxonomy of 25 distinct registers designed to capture the full diversity of web content. Rather than treating register identification as a simple binary or few-category task, the researchers embraced the complexity of real-world text, where documents often blend multiple styles and purposes.

Their best-performing model achieved an average F1 score of 79% across all 16 languages using multi-label classification—a significant result given the sophistication of the classification scheme. This performance matched or surpassed previous studies that relied on simpler, less granular register categories.

The ceiling effect and the ambiguity problem#

What became apparent during experimentation was a consistent performance plateau across different model architectures and configurations. Counterintuitively, this suggested the limitation wasn't computational but intrinsic to the data itself. When the researchers removed documents with uncertain or conflicting labels through a data pruning process, performance jumped above 90% F1.

This finding points to a fundamental insight: the challenge isn't that models struggle to classify registers, but that many web documents genuinely defy clear categorization. Text that mixes news reporting with forum discussion, or academic content with informal commentary, resists unambiguous labeling—a problem no algorithm can fully overcome.

Multilingual strengths and language-specific quirks#

The study uncovered an important advantage of training on multiple languages simultaneously: multilingual models outperformed language-specific variants, especially for languages with smaller training datasets. However, zero-shot performance—applying the model to entirely new, unseen languages—dropped by an average of 7 percentage points (ranging from 3% to 8% depending on the language).