AI
New Benchmark Reveals How AI Models Handle Mixed Chinese-English Technical Language
Researchers create ChiEngMixBench to test whether language models match how professional communities actually blend native and English terms in expert writing.
Professional communities working across languages often develop their own conventions about which technical terms to keep in English and which to translate. A new benchmark called ChiEngMixBench examines how well large language models understand and reproduce these mixing patterns in Chinese AI and computer science discourse.
The Benchmark's Design#
Researchers constructed ChiEngMixBench from real technical discussions, assembling 1,706 source-derived examples that cover 1,344 different technical terms. The dataset includes a carefully controlled subset of 1,167 pairs where the Chinese framing and syntax stay identical while only the terminology choice—English retention versus Chinese translation—varies. This design isolates the actual terminology decision from surrounding linguistic factors.
The evaluation combines two approaches: paired likelihood comparisons measure which form the model prefers, while a reference-profile diagnostic examines how well open-ended responses match observed community practice.
What the Testing Revealed#
Across nine open-weight models, a consistent pattern emerged: Chinese translations of technical terms received higher probability scores than their English equivalents on most pairs. This reveals a notable mismatch—models gravitate toward Chinese equivalents even when the source-attested, real-world usage in these professional communities favors keeping English terminology.
The researchers found that specialized terms showed only a small directional preference for English retention, and this effect proved fragile when controlling for word frequency and length or after applying statistical corrections for multiple comparisons.
Practical Implications#
Human evaluation showed that how well models conform to the community's actual terminology profile matters when evaluating multilingual outputs, though conformity alone does not reliably predict whether people find the overall response quality acceptable. The work highlights a gap between what language models learn and what professional communities actually practice in their day-to-day technical communication.
ChiEngMixBench provides researchers with a reusable framework for testing community-specific multilingual conventions, with clear diagnostic boundaries that help isolate terminology choices from other aspects of language production.
