For a long time, I assumed serious agent integration required a structured interface.
API, CLI, SDK, then MCP.
Codex Computer Use changed that assumption.
An agent can now look at the same interface as a user, operate it, observe the resulting state, and continue.
GPT-5.6 Sol was already strong enough for us to start using this seriously. GPT-6 Astra raised the baseline further.
OpenAI reports Astra at 72.6% vs. 65.7% for Sol on OSWorld 2.0 — specifically v2026.08.08, offline set, partial score — and 92.7% vs. 76.9% on ScreenSpot-Pro, under the no-tools setting.
These are benchmark results, not end-to-end success rates for arbitrary enterprise workflows, but the improvement is substantial.
More importantly, OpenAI now explicitly describes frontend QA, installing and testing software, and troubleshooting visible problems as Computer Use workloads.
The important change is not that AI can click buttons.
The GUI has become a practical agent interface.
The integration path changed
With MCP, the protocol itself is usually not the difficult part.
The difficult part is designing what sits behind it.
Application
↓
API / CLI
↓
agent-oriented operations
↓
MCP
↓
Agent
For an agent to operate complex software reliably, we often have to design the right verbs, schemas, permissions, side effects, errors, and recovery behavior.
That remains the right architecture for operations requiring determinism, throughput, explicit contracts, or strong auditability.
Computer Use does not replace MCP.
It changes how much of that integration surface has to exist before an agent can do useful work.
Another path is now practical:
Existing GUI
↓
observe
↓
click / type / navigate
↓
observe again
↓
Agent
The current Computer Use implementation can visually inspect and operate graphical applications, including workflows where CLI or structured integrations are not sufficient.
In the Codex desktop environment, Computer Use is exposed through the plugin/tool system.
So this is not really MCP versus Computer Use.
Computer Use reduces the need to design every domain-specific operation upfront.
It does not remove the need to engineer execution boundaries and verify outcomes.
The integration cost moved
This is the part I find most interesting.
Previously, much of the engineering cost looked like this:
design API
design CLI
design agent verbs
wrap with MCP
Computer Use moves part of that work toward:
control identity
control environment
control permissions
observe execution
verify outcomes
The integration cost did not disappear.
It moved.
For a specialized desktop tool with a weak API, building twenty MCP operations may no longer be the best first step.
We can let the agent operate the existing interface, observe the real workflow, and identify which operations are frequent, dangerous, slow, or difficult to verify.
Then those operations can become structured tools.
GUI
↓
real usage
↓
identify critical operations
↓
API / CLI / MCP
The GUI becomes a compatibility layer.
Structured interfaces remain both an optimization layer and a place to enforce explicit contracts and policy.
The GUI is not an authorization boundary
This distinction becomes critical once the agent can operate the same interface as a human.
Computer Use changes how an agent reaches an operation. It does not define what authorizes that operation.
If an AI can see a human approval button, that does not mean the AI should be allowed to press it.
If credentials, raw data, privileged controls, or connected accounts are visible in the environment, they may also become part of what the agent can observe or operate.
This is not a sufficient security model:
Agent
↓
GUI
↓
whatever the logged-in user can do
For important actions, GUI operations and structured tool calls should converge on the same authorization, validation, approval, and audit boundaries.
Agent
↓
controlled execution environment
↓
GUI / structured tools
↓
policy + validation
↓
system of record
A visual interface is an execution surface.
It should not become the source of authority.
We use Computer Use for QA
A lot of Computer Use demos naturally focus on video editing, design software, spreadsheets, and other professional tools.
Our main use case is more mundane:
software testing.
We already use Playwright heavily.
We also use agent-based evals, generated test data, and fuzzing to explore cases that are difficult to maintain as hand-written fixtures.
Computer Use adds another execution path.
I do not see these as a linear pipeline where every test eventually reaches Computer Use.
They are different ways of exercising the same system, ideally converging on shared outcome checks.
scenario
│
┌───────────┼───────────┐
│ │ │
Playwright Agent tools Computer Use
│ │ │
└───────────┼───────────┘
│
shared outcomes
and invariants
│
reproduce / reduce
│
regression test
Playwright remains better when we know exactly what contract we want to verify.
Computer Use is useful when we want to exercise the product closer to how a user actually experiences it.
Instead of encoding a fixed sequence of selectors and assertions, we can give the agent a task such as:
Open the application.
Complete onboarding.
Load this generated dataset.
Perform the primary workflow.
Investigate anything visibly wrong.
Preserve evidence of failures.
That exposes a different class of problems:
- native dialogs
- broken focus
- hidden or overlapping controls
- unexpected application state
- simulator behavior
- cross-application workflows
- failures visible to a user but awkward to express as DOM assertions
OpenAI explicitly positions Computer Use for desktop application testing, GUI-only bugs, and applications that are not adequately covered by command-line or structured integrations.
We also combine this with generated scenarios.
Rich or unusual data can be produced automatically and then exercised through the actual application rather than only through APIs.
But I do not want the model to become the oracle.
If the requirement is that no duplicate object can be created, code should verify that.
If a user must never gain a particular permission, the authorization layer should verify that.
The agent explores.
Deterministic checks decide facts.
Humans decide where judgment is actually required.
Human review remains valuable for ambiguous product behavior, UX quality, and release decisions.
It does not mean a human needs to inspect every successful run.
In practice, Computer Use has reduced a significant amount of the work before human review:
- setup
- navigation
- reproduction
- retries
- exploration
- evidence collection
That is already useful to us today.
Codex App Server changes the product boundary
The next interesting part is that the Codex harness itself is embeddable.
Codex App Server exposes the Codex harness to external clients through a bidirectional JSON-RPC-style interface.
The embedding application can manage threads and turns, receive agent events, handle approvals, and observe tool activity.
Conceptually:
Our application
│
▼
Codex App Server
│
▼
Codex harness
│
├── shell / files
├── MCP servers
├── skills
└── agent loop
This matters because App Server exposes much more than raw model inference.
Codex core manages threads and persistence, executes tools, wires MCP servers and skills into the agent loop, and can pause execution while the embedding client handles an approval request.
But there is an important boundary here.
App Server is not a remote screenshot-and-click API for Computer Use.
In the ChatGPT desktop app, Computer Use is installed as a plugin and exposes a Computer-use MCP server and a Computer Use skill that Codex can use. App Server separately exposes the surrounding Codex harness.
App Server exposes the surrounding Codex harness.
Those are related building blocks, but they are not the same thing.
The existence of App Server plus the desktop Computer Use capability does not by itself guarantee that the official Computer Use implementation can be packaged and independently deployed inside every arbitrary App Server environment.
For that reason, I separate three things:
officially exposed capability
↓
what we have connected and validated
inside controlled environments
↓
architectures that become possible
from those building blocks
We are actively implementing the second category.
The architectural pattern we care about is:
Our product
↓
Codex / agent runtime
↓
controlled desktop environment
↓
application being operated
This lets computer operation become part of an application architecture rather than something that only happens on a developer's everyday desktop.
Sandbox the desktop, not just the shell
This becomes especially important for enterprise systems.
The Codex harness already has strong concepts around sandboxed tool execution, permissions, MCP integrations, and approval flows.
Computer Use expands the execution surface.
OpenAI explicitly warns that Computer Use can affect application and system state outside the project workspace.
So protecting the project directory is not enough.
For the systems we are building, the isolation boundary needs to include the desktop environment itself.
isolated environment
│
├── dedicated OS/session identity
├── dedicated browser profile
├── restricted applications
├── restricted network egress
├── controlled filesystem
├── controlled clipboard
├── controlled shared folders
├── limited accounts
└── disposable state
A VM alone is not a security model.
If an isolated VM is logged into production with privileged credentials, production side effects are still possible.
The screen must be isolated.
The identity must be limited.
The network must be constrained.
The applications, browser sessions, connected accounts, shared folders, and credentials available inside the environment must all be intentional.
Important side effects should still pass through explicit policy boundaries.
This is what makes Computer Use interesting for enterprise deployment.
Some organizations cannot deploy a general-purpose AI desktop application to employee endpoints.
Others need dedicated identities, isolated networks, specific audit controls, or access to highly specialized software that only exists in a controlled environment.
The useful model is not:
AI controls employee laptop
It is:
AI controls purpose-built environment
We are implementing this pattern in our own application architecture.
Specialized software is where this gets interesting
The long tail of enterprise software is full of difficult integrations:
- legacy desktop applications
- internal tools
- engineering software
- industry-specific applications
- vendor tools with incomplete APIs
Historically, making those systems agent-accessible meant building an adapter first.
Computer Use changes the economics.
If a human can operate the application visually, an agent can increasingly attempt the same workflow before a complete structured integration exists.
That lets us validate the workflow first.
Then we can progressively move important operations into deterministic tools.
Computer Use:
open project
navigate UI
inspect state
MCP:
fetch structured data
perform validated mutation
commit privileged change
Computer Use:
verify rendered result
I expect this hybrid architecture to become increasingly common.
The question is not:
MCP or Computer Use?
It is:
Which parts of this workflow deserve a structured contract?
The boundary changed
I still want structured interfaces.
For payments, production changes, privileged administration, bulk operations, and anything requiring strong transactional guarantees, explicit tool contracts matter.
Computer Use does not make those requirements disappear.
But I no longer think every application needs a complete agent-native integration before an agent can start doing useful work.
That changes the development order.
Before:
design integration
↓
expose tools
↓
agent can work
Now:
agent uses existing interface
↓
observe real workflow
↓
identify critical boundaries
↓
structure where necessary
For us, this is already changing software testing.
It is also changing how we think about bringing agents into specialized enterprise software.
The engineering work did not disappear.
It moved from:
How do we expose every operation?
to:
What environment can the agent operate, what is it authorized to do, and how do we prove the result?
That is a much more important shift than better mouse automation.
I still think important APIs will increasingly be redesigned for agents.
I just no longer think agents have to wait for that redesign.
Top comments (0)