I’m Building a Model-Agnostic Agent OS—How Would You Break This Architecture?

I’m building a personal Agent OS to help me develop software with AI without handing over control of my repositories.

Some parts are already running across my projects. Others—such as switching between model providers, automating different trust levels, and sharing approved knowledge across repositories—are still being designed and tested.

I currently use Codex as the coding agent, but I don’t want the whole system to depend on one provider. The same workflow should eventually work with Claude, Kimi, DeepSeek, MiniMax, or any other capable coding agent.

The coding agent should be replaceable. The rules around it should not be.

I’m sharing the architecture now because I’d rather discover its weak points through honest feedback than after building too much around the wrong assumptions.

The basic idea

My rule is simple:

Give AI room to do the work, but not permission to redefine the work.

I still decide:

• What the goal is
• Which repositories the agent can access
• Which changes need approval
• What can reach production
• Which lessons are trustworthy enough to reuse

The agent can inspect, plan, write code, run tests, and propose changes. It just has to do that inside a clearly defined boundary.

1. A request becomes a Task Contract

Suppose I give the system this instruction:

Improve the authentication system.

That is far too vague to trigger an autonomous coding run.

What does “improve” mean? Which files can change? Can the agent modify the database? What tests must pass? Is it allowed to replace a dependency?

Before any code is touched, the Agent OS turns the request into a bounded Task Contract.

The contract can define:

• The objective
• Allowed repositories
• Allowed files or directories
• Areas that must not be changed
• Expected outputs
• Required tests
• Risk level
• Time or compute limits
• Approval requirements
• Completion criteria

This contract becomes the boundary of the run.

If the coding agent discovers something that requires work outside that boundary, it should stop and ask for a decision. It should not quietly enlarge its own assignment.

That sounds restrictive, but I see it as the condition that makes useful autonomy possible.

2. The Execution Governor enforces the boundary

The Agent OS plans and coordinates the work. I don’t want it to hand an AI model unrestricted access to my development environment.

That is why I’m introducing a separate component: the Execution Governor.

Its job is to:

• Validate the Task Contract
• Choose the appropriate coding agent
• Enforce repository and tool permissions
• Apply time and resource limits
• Record what happens during the run
• Stop unsafe or out-of-scope work
• Recover interrupted runs
• Send completed work into verification

I’m treating this as a boundary between intent and execution.

It also prevents the architecture from becoming too dependent on one model. A contract could be routed to Codex today and a different coding agent tomorrow without changing the governance process around it.

At least, that is the idea. One thing I’m still questioning is whether the Execution Governor is genuinely useful or simply another orchestration layer with a more impressive name.

3. Coding happens in isolation

The chosen coding agent works inside a temporary, isolated environment.

It can inspect the permitted code, make changes, run tests, and collect evidence. It cannot directly change production or silently push work into the main branch.

A run should produce something concrete:

• A patch or branch
• Test results
• Static-analysis results
• A list of changed files
• Assumptions made during the task
• Known risks or unresolved questions
• A proposed pull request

The sandbox can disappear when the run finishes.

The evidence should remain.

4. Confidence is not proof

Coding agents are very good at sounding confident—even when the code is wrong.

Because of that, I don’t want the system to accept an agent’s summary as proof that a task succeeded.

If the agent says, “All tests pass,” I want evidence that the tests actually ran.

Depending on the project, verification might include:

• Unit and integration tests
• Type checking
• Linting
• Security checks
• Database migration validation
• Production builds
• Browser or end-to-end tests
• Human review

The repository and CI pipeline remain the source of truth. A polished explanation is helpful, but it cannot replace a verifiable result.

5. More risk means more human control

Not every code change deserves the same level of supervision.

A documentation correction and an authentication migration should not follow the same approval process. I’m therefore experimenting with three trust tiers.

Tier 1: Autonomous

This tier is for predictable, low-risk maintenance work, such as:

• Small dependency updates
• Formatting fixes
• Documentation corrections
• Clearly bounded mechanical changes

These tasks could potentially run and merge automatically—but only when every required check passes.

I’m not yet convinced that automatic merging is worth the risk, even here.

Tier 2: Hybrid

This tier covers bounded changes to application logic.

The system can start the task and prepare the pull request automatically, but a human must review it before anything merges.

Tier 3: Strictly governed

This tier is reserved for sensitive work, including:

• Architectural changes
• Authentication or authorization changes
• Sensitive database migrations
• Security-critical code
• Infrastructure or production configuration

These tasks need human approval before work begins and again before the result can merge.

The difficult part is not defining the tiers. It is classifying tasks reliably enough to trust them.

6. Memory has to be earned

I want the system to get better over time, especially when the same patterns appear across several projects.

But I don’t want it to remember everything.

A failed attempt, a temporary workaround, or an agent’s unverified assumption should not quietly become permanent knowledge.

My current rule is that a lesson becomes reusable only after:

1. The implementation passes its required checks.
2. The change successfully lands in the repository.
3. The outcome is reviewed.
4. A human approves the extracted lesson.

Approved lessons are then given a scope:

• Project-specific
• Stack-specific
• Portfolio-wide

A lesson from one repository should not automatically affect another repository. Cross-project reuse requires an explicit decision.

This is meant to stop one incorrect assumption from spreading through every project that happens to use the same technology.

7. More context is not always better

It is tempting to give an AI agent everything the system has ever learned.

In practice, that can create noisy prompts, conflicting instructions, and outdated assumptions.

Before a task starts, the Agent OS should retrieve only the relevant, verified knowledge needed for that particular task.

The coding agent gets the context it needs—not the entire history of every previous run.

I expect this to keep the agent more focused, but context selection creates another problem: how do I know that the system has not excluded one crucial lesson?

The complete flow

The workflow currently looks like this:

1. A human defines the goal.
2. The Agent OS creates a bounded Task Contract.
3. The Execution Governor validates and routes it.
4. A coding agent works inside an isolated environment.
5. Automated checks and human review verify the result.
6. Approved code lands in the project repository.
7. The system may extract a lesson from the completed work.
8. A human decides whether that lesson becomes reusable knowledge.

Each repository remains independently governed, even if orchestration and reusable capabilities are managed centrally.

The parts I’m still unsure about

Some of this architecture is based on problems I have already encountered. Some of it may be unnecessary machinery added to prevent problems that will never become serious.

These are the questions I’m wrestling with:

• Is the Execution Governor a useful boundary or just another orchestration layer?
• How should a Task Contract handle legitimate discoveries made during execution?
• What evidence is enough before a result can honestly be called “proven”?
• Can low-risk changes ever be classified reliably enough for automatic merging?
• How do I prevent human approvals from becoming meaningless rubber stamps?
• Should the system extract lessons automatically for later approval, or should lessons be written manually?
• How should it resolve conflicting lessons from different projects?
• Could selective context hide something the agent genuinely needs?
• What important failure mode have I missed?

I’d especially appreciate feedback from people working with coding agents, CI/CD, developer platforms, sandboxed execution, or multi-agent systems.

Where would this architecture break in real use? What feels over-engineered? What important safeguard is missing?

---

Disclosure: I used AI tools to help edit this article and create its visual presentation. The architecture, implementation experience, and design decisions are my own.