New Benchmark Tackles Machine Translation Challenges in Chinese Social Media Slang

Translating Chinese social media text poses an underappreciated challenge: the language is constantly shifting with platform-specific slang, made-up words, and distinctive writing styles that don't appear in traditional parallel datasets. Existing machine translation systems struggle with this informal terrain, yet most benchmarks still rely on formal, edited text.

The Problem#

Researchers have identified two major obstacles blocking progress on real-world Chinese social media translation. First, data scarcity stems from the need for high-quality bilingual training data—annotators must be fluent in both the source and target languages and conversant with contemporary platform slang and stylistic quirks on both sides. Second, metric limitations mean that standard evaluation tools like COMET often miss the mark when assessing whether a translation preserves not just meaning but tone, style, and the flavor of informal expressions.

Introducing CSM-MTBench#

To address these gaps, a team of researchers has created CSM-MTBench, a curated benchmark covering five Chinese-foreign language pairs. The dataset splits into two distinct categories:

• Fun Posts: Dense with slang, neologisms, and contextual richness
• Social Snippets: Brief, emotion-laden messages where tone and style are paramount

Evaluation Innovations#

The benchmark employs custom assessment strategies tailored to each subset. For Fun Posts, researchers measure how successfully models translate platform-specific slang and newly coined terms. For Social Snippets, they blend embedding-based metrics with LLM-as-a-judge approaches to gauge whether the translation preserves the original's emotional register and stylistic character.

Findings#

Testing across over 20 translation models revealed substantial differences in how systems navigate the intersection of semantic accuracy and social-media-specific informal style. The results underscore that current MT systems still have significant work to do in handling real-world Chinese social media at scale.