Direct Audio Analysis Detects Abusive Speech in Indic Languages Without Transcription

Identifying hateful and abusive speech from audio remains challenging, especially when transcription quality is poor and tone—a key marker of abuse—gets lost in text. A new research effort demonstrates that analyzing audio directly, rather than converting it to text first, can effectively flag abusive speech across a range of Indic languages with minimal labeled training data.

The Problem with Text-Based Detection#

Traditional abuse detection relies on transcribing speech to text, then applying classification systems. This approach has critical limitations. For languages with weak speech recognition infrastructure, transcription errors compound the challenge. Additionally, the emotional weight of speech—the anger, mockery, or menace in someone's voice—vanishes once words are extracted and printed. This is especially problematic because abuse often lives in tone and delivery as much as in the words themselves.

How CLAP Models Work#

The research centers on CLAP, a model trained to learn shared representations between sound and language. By operating in a joint audio-language space, CLAP captures both the linguistic content and acoustic qualities that make abuse recognizable. The team evaluated their approach across ten Indic languages using the ADIMA dataset, testing whether CLAP's learned representations could flag abusive content directly from voice.

Key Findings#

A lightweight classifier built on top of CLAP's existing audio representations—without retraining or adapting the underlying model—performed within one to three percentage points of a fully supervised system trained on substantial labeled data. This demonstrates that CLAP's off-the-shelf representations already contain strong signals for detecting abuse across languages. Even without any fine-tuning, the approach vastly outperformed zero-shot prompting methods that rely on no labeled examples at all.

Minimal Adaptation Shows Mixed Results#

When researchers added adaptation using just a handful of labeled examples per language, the benefits were modest and inconsistent. Depending on the language, small amounts of additional training sometimes helped, sometimes hurt, and results varied unpredictably. This suggests that CLAP's baseline representations are already well-suited to the task and that language-specific fine-tuning may be unnecessary in practice.