Transformer Model Recognizes Unseen Sign Language with Few Examples

Recognizing sign language from video presents a dual challenge: inferring three-dimensional structure from flat 2D footage while capturing the inherently dynamic, spatial nature of movement over time. A research team has tackled this problem with a Transformer-based contrastive learning model that learns representations of body keypoint sequences in ways that enable the system to recognize signs it has never encountered before.

The Core Problem#

Sign language has an ever-expanding vocabulary in real-world use, making traditional closed-set classification—where all possible classes must be known and labeled during training—impractical. Every time new signs enter common usage, the conventional approach requires fresh labeled data and complete model retraining. The new method sidesteps this limitation by learning general representations that support direct comparison between sign examples, unlocking one-shot and few-shot recognition.

How It Works#

The model leverages contrastive learning, which trains the system to recognize similar signs as related and dissimilar ones as distinct, without explicit labels for every possible variation. By working with body keypoint sequences extracted from monocular video, the approach avoids the need for expensive 3D capture systems while still learning meaningful structure.

Performance Results#

Testing on the LSA64 dataset, the researchers trained on 48 classes and evaluated the model's ability to classify 16 completely held-out sign categories. With as few as eight reference examples per unseen sign, the model achieved 88.4% accuracy. Performance improved consistently as the training set grew and more support examples became available for new signs.