OpenAI's chief argues AI agents should mimic human computer use—not force software redesign

OpenAI president Greg Brockman recently made a case for rethinking how AI agents interact with software. Instead of engineering specialized integrations and middleware, he suggests agents should operate like humans—using keyboards, mice, and screens directly.

Speaking on the a16z podcast, Brockman described the current state of AI tooling as a form of costly retrofitting. "We're kind of retooling the world," he said, pointing to the proliferation of MCP servers, command-line interfaces, and API connectors built specifically to bridge AI systems with applications. These integrations, while functional, weren't designed with human workflows in mind.

The case for human-like interaction#

The core insight is straightforward: if AI models can learn to interact with graphical interfaces the same way users do, developers would no longer need to maintain purpose-built connectors for every application. Brockman framed it as a philosophical preference: "The computer should be there to empower you."

OpenAI has been exploring this direction for nearly a decade. During an internal planning session in 2015, the team discussed what it would mean to use reinforcement learning directly against the computer interface—treating screen pixels, keyboard input, and mouse clicks as the environment for AI agents to navigate. This approach would theoretically unlock access to virtually any software without custom engineering.

The practical appeal#

Beyond simplicity, Brockman pointed out the sheer volume of manual work that could be eliminated. Much of professional life involves repetitive clicking, menu navigation, and data entry—tasks that general-purpose computer use could automate without specialized integrations.

Brockman also cited computer use capabilities as one reason he believes OpenAI's latest model GPT-6 Astra represents meaningful progress toward AGI.

Connectors remain entrenched#

Despite Brockman's vision, the broader industry hasn't abandoned structured integrations. AWS recently expanded its Bedrock AgentCore service with a managed consent portal for connecting agents to tools like GitHub and Slack. Similarly, OpenAI's own plugin ecosystem—and its newer Codex Chrome extension for browser automation—remain in active use because direct API integrations continue to offer reliable service access without the overhead of interface navigation.