How Large Language Models Perform on China's High-School Entrance Exam

Researchers have created GAOKAO-Bench, a comprehensive evaluation framework to measure how well large language models perform on one of the world's most challenging educational assessments: China's GAOKAO national college entrance examination.

What the Benchmark Tests#

The benchmark draws directly from actual GAOKAO questions, covering both objective multiple-choice items and open-ended subjective questions that require longer, more nuanced responses. By using real exam materials, the researchers sought to create a more realistic measure of how these AI systems handle complex, knowledge-intensive tasks compared to traditional NLP benchmarks. The evaluation methodology mirrors human testing conditions, employing zero-shot settings where the models receive no prior examples to learn from.

Key Findings#

The study found that advanced language models—including GPT-4 and ChatGPT—achieved scores competitive with human performance on the GAOKAO. However, performance was far from uniform. Different subjects revealed substantial disparities, with the models excelling in some areas while struggling in others. This uneven performance suggests that LLMs have domain-specific weaknesses, particularly in subjects requiring specialized knowledge or reasoning beyond general language patterns.

Assessing Subjective Responses#

A notable aspect of the research involved using LLMs themselves to grade the subjective portions of the exam, a task normally requiring human judgment. When these machine-generated scores were compared against human evaluations, they demonstrated moderate consistency—suggesting that models can partially replicate human assessment criteria, though not perfectly.