GitHub Copilot Gains Desktop Control—But Not as a First Resort

GitHub has rolled out computer-use capabilities to Copilot CLI and its desktop application, allowing the AI to interact with graphical software on macOS and Windows. The new feature can read interface content, click buttons, type text, scroll, and drag—even within legacy applications that lack APIs or command-line access.

The company showed off the capability by automating an expense report in Safari during its launch demonstration but noted potential uses ranging from extracting data from older software to updating presentations and shuffling information between programs. Users can trigger the feature either from the terminal or through GitHub's Copilot app, which launched earlier in the year as a direct competitor to Anthropic's Claude Code and OpenAI's Codex.

A Deliberate Second Choice#

GitHub's framing of computer use is notably cautious. The company explicitly recommends leaning on alternatives—APIs, MCP servers, terminal commands, filesystem operations, or specialized browser tools—wherever viable. These structured approaches deliver cleaner data and more reliable outcomes than visual automation.

This stance contrasts with OpenAI president Greg Brockman's recent argument that agents using human-facing interfaces could spare the industry from maintaining countless software connectors. GitHub seems to view desktop interaction as a fallback rather than an ideal approach.

How It Works#

Under the hood, computer use runs as a bundled MCP server plugin, operating within local sessions by parsing the system's accessibility tree and capturing screenshots when needed for visual context. On macOS, users must grant both Accessibility and Screen Recording permissions to enable the feature.

Developers activate it with the /computer on command in Copilot CLI or toggle it in the desktop app's settings. Permission handling lets users allow access for a single session, save approvals for future use, or block specific applications. Approvals sync between the CLI and desktop app on the same machine, though removing an app from the allowlist only affects future sessions—running sessions keep their permissions intact.

Enterprise Safeguards and Known Limits#

Organizations can block computer use entirely or restrict whether developers can skip approval prompts, with these settings enforced across Copilot CLI, the desktop app, and VS Code. Preview features remain opt-in even under GitHub's default-enablement policy for business and enterprise accounts.