Withdrawn GUI Agent Framework Proposed Dual-System Reasoning for Automation Tasks

A recent paper submitted to arXiv proposed CogniGUI, a framework designed to improve how AI agents automate tasks on graphical user interfaces by mimicking human cognitive processes. The work has since been withdrawn by its author.

The Problem With Current GUI Agents#

Existing AI systems that automate digital tasks through computer vision and language models face significant gaps. They often rely on trial-and-error rather than deliberate reasoning, and lack the ability to learn and refine their strategies from repeated interactions. Traditional evaluation methods—measuring single-step accuracy—fail to capture the complexity of real-world GUI navigation, where agents must make sequences of decisions across multiple applications and handle changing interface states.

Proposed Approach: Dual-System Design#

The CogniGUI framework drew inspiration from Kahneman's Dual Process Theory, which contrasts quick intuitive thinking with slower deliberative reasoning. The approach combined two core components:

• Omni parser engine: A visual semantic analyzer that rapidly identifies and hierarchically parses actionable GUI elements
• Group-based Relative Policy Optimization (GRPO) agent: A system that evaluates multiple interaction paths using relative rewards to favor efficient, minimal-step solutions

This dual structure aimed to enable iterative cycles of exploration, learning, and mastery, allowing agents to improve strategies based on accumulated experience.

New Benchmark: ScreenSeek#

The authors introduced ScreenSeek, a benchmark designed to test agent generalization and adaptability across challenges often missing from existing evaluations. These include multi-application navigation, dynamic state transitions, and cross-interface coherence.

Note#

The paper was submitted in June 2025 and withdrawn in July 2026, with only a brief version note available.