Agentic AI
New benchmark reveals large language model agents struggle with Earth observation tasks
A coding benchmark of over 400 questions shows current AI systems achieve only 40% accuracy on satellite data analysis—well below what's needed to automate environmental monitoring.
Researchers have created a large-scale evaluation framework to test whether large language model (LLM) agents can reliably handle Earth observation tasks—a critical capability for automating environmental monitoring, disaster response, and climate research.
The UnivEARTH benchmark#
The study introduces UnivEARTH, a coding benchmark containing 408 yes/no questions derived from NASA Earth Observatory articles. The questions span seven distinct topics and draw on data from over 15 different satellite instruments and sources. By using the Google Earth Engine API as a tool, researchers evaluated how well modern LLM agents could formulate and execute queries to answer these questions in a zero-shot setting—meaning the models received no prior training examples for this specific task.
Current performance gaps#
The results expose significant limitations in today's AI capabilities. Leading models including Claude-4.5-Sonnet, Gemini-2.5-Pro, and GPT-5 achieved only 40% accuracy on the benchmark when tested without any error-correction framework. More troublingly, the code generated by these systems failed to run properly over 44% of the time—a fundamental issue that prevents meaningful answers from being produced at all.
Improvements with error correction#
The researchers explored whether incorporating a Reflexion framework—a technique that allows LLM agents to review and correct their own errors—could improve outcomes. Using this approach, model accuracy improved to approximately 60%. While this represents a meaningful gain, it remains only slightly better than random chance on a yes/no question format, indicating the models still lack robust understanding of Earth observation workflows.
Methodology insights#
The analysis also examined how different implementation choices affect LLM performance. Providing models with documentation alongside API tools yielded modest benefits, as did choosing Python over the JavaScript API variant. These findings suggest that while presentation matters, the core challenge lies deeper—in the models' fundamental ability to reason about satellite data and geospatial analysis.
