New benchmark tests how well AI language models learn rare programming languages

Large language models typically perform well on mainstream programming languages like Python and JavaScript, but they often stumble when asked to work with less common tongues that lack abundant training data. A new research effort aims to fill this gap by introducing CangjieBench, a benchmark designed specifically to measure how well various LLMs can handle Cangjie, a representative low-resource general-purpose programming language.

What CangjieBench covers#

The benchmark contains 248 manually curated test cases, each carefully adapted from established coding evaluation suites—specifically HumanEval and ClassEval. These samples span two main task categories: converting natural language descriptions into working code (Text-to-Code) and translating code between representations (Code-to-Code). The researchers ensured the benchmark is contamination-free, meaning the test data was not inadvertently included in LLM training sets.

Comparing different approaches#

The evaluation examined how multiple LLMs performed across four distinct strategies. Direct Generation, where models simply produce code from a prompt, showed weak results. More promising was Syntax-Constrained Generation, which guided models toward grammatically correct output—this method offered the best balance between accuracy and efficiency.

Researchers also tested Retrieval-Augmented Generation (RAG), which enriches prompts with relevant code examples, and an Agent approach that allows models to iteratively refine their solutions. While the Agent method achieved the highest accuracy rates, it came at a significant cost in computational tokens consumed.

Key findings on language transfer#

One striking discovery emerged around code translation tasks: models often performed worse when translating from an existing language into Cangjie compared to generating code directly from text instructions. This pattern suggests a negative transfer phenomenon, where models become overly dependent on patterns from the source language rather than learning Cangjie's unique structure.

Researchers hope these findings will illuminate how LLMs generalize to unfamiliar and data-scarce programming languages, offering guidance for improving model performance in low-resource domains.