New benchmark unlocks AI-powered code repair for HarmonyOS's ArkTS language

A new research framework aims to accelerate automated code repair for ArkTS, the statically typed TypeScript variant that powers development on Huawei's HarmonyOS mobile ecosystem. As large language models increasingly reshape software development, the lack of specialized evaluation tools for ArkTS has left developers without rigorous metrics to assess repair automation—a gap that ArkEval now addresses.

The problem: Tools missing for a growing ecosystem#

HarmonyOS has become an increasingly important mobile platform, yet the development toolkit for ArkTS lags behind its peers. Researchers identified an urgent need: while LLMs have demonstrated strong code generation capabilities, no high-quality benchmark existed to evaluate their ability to repair ArkTS code specifically. This has made it difficult to measure progress and compare approaches in this low-resource language domain.

Building the benchmark through large-scale mining#

The team constructed ArkEval by extracting real-world issues from Huawei's official repository, which contains over 400 standalone ArkTS applications. Using a rigorous multi-stage filtering process, they narrowed thousands of potential cases down to 502 reproducible, well-documented issues. To ensure these problems could be reliably evaluated, they pioneered an LLM-based test generation approach featuring automated voting across multiple models, including Claude, to validate test cases and prevent ambiguity.

Evaluating state-of-the-art models#

The researchers benchmarked four leading LLMs against their dataset, employing a retrieval-augmented repair workflow that grounds repair attempts in relevant code context. The results reveal both capabilities and limitations of current models when applied to ArkTS—providing a baseline for future research and development in this emerging technical area.

Why it matters#

The introduction of ArkEval establishes the first standardized evaluation infrastructure for ArkTS automated repair. This paves the way for researchers and tool builders to measure and improve repair automation, ultimately accelerating development velocity for the HarmonyOS ecosystem and demonstrating how domain-specific benchmarks can unlock progress in underserved language domains.