New AI framework ensures sign language videos match word order and meaning

Researchers have developed a new generative AI system designed to automatically convert written text into sign language video, addressing persistent challenges that plague existing approaches. The framework, called SIGNER, tackles two core problems that have limited the quality of automated sign language generation: maintaining correct word order (lexical ordering) and ensuring each gesture accurately represents its intended meaning (semantic accuracy).

The core challenge#

Sign language differs fundamentally from spoken or written language in how meaning is conveyed. Rather than linear word sequences, sign language relies on glosses—word-like units that correspond to specific hand gestures, body positions, and movements. When generating sign video from text, the system must:

• Preserve the logical sequence of these gesture units to avoid scrambling the intended message
• Ensure each individual gesture faithfully represents the gloss it should embody

Most existing text-to-sign systems apply conditioning information uniformly across the entire generation process, a strategy that blurs the connection between when a gloss should appear and which part of the video should express it. This temporal misalignment leads to garbled lexical order and ambiguous signs.

How SIGNER works#

The new framework introduces time-resolved conditioning, which grounds each gloss in a specific moment during video generation. The process unfolds in two key steps:

• Temporal-gloss condition: The system first estimates both the sequence of glosses needed and how long each one should last, then maps these semantic meanings across the temporal dimension of the video
• Local temporal fusion (LTF): During the generation process, conditioning information is applied only within a narrow time window around each gloss—preventing the system from confusing where different signs belong

This localized approach ensures that each gloss remains semantically distinct and appears in the correct position relative to others.

Results and impact#

Testing on two established sign language benchmarks—Phoenix-2014T and CSL-Daily—showed that SIGNER achieves state-of-the-art performance. Analysis of motion smoothness in the generated videos further confirmed the approach's effectiveness. By maintaining temporal grounding throughout the generation process, the system produces sign sequences that follow proper grammar and convey clear, unambiguous meaning.