New dataset and methods help AI recognize emotions expressed by sign language users

Detecting emotional states from sign language videos presents distinct technical hurdles that differ from analyzing spoken language. Researchers working on this problem have now published findings that address two major obstacles: distinguishing between grammatical and affective facial movements, and overcoming the limited availability of training data.

The core challenges#

Sign language relies heavily on facial expressions to convey both grammatical information and emotional content, making it difficult for machine learning models to separate the two. Additionally, the scarcity of annotated video datasets for sign language training compounds the problem, as most emotion recognition work has focused on spoken language or text.

New datasets and approach#

The team introduced eJSL, a newly created benchmark containing 1,092 video clips of Japanese Sign Language signers expressing 78 distinct utterances across seven different emotional states. They also leveraged BOBSL, an existing large British Sign Language dataset with accompanying subtitles, to expand their research scope across languages.

Their approach incorporated textual emotion signals from spoken language models to help compensate for limited sign language data—demonstrating that information from text-based emotion recognition can improve performance on video data. The researchers also found that careful selection of temporal segments within videos significantly impacts accuracy, and that incorporating hand motion patterns as a feature enhances the model's ability to recognize emotions in signers.

Results#

The resulting system outperformed baseline approaches based on large language models trained on spoken language, establishing a more robust starting point for future work in this specialized domain.