Code Nomi Nomi

Posted on Aug 30

Beyond the Model: Does the Architecture Around an AI Agent Matter?

#agentaichallenge #ai

Ref
Cisco Systems, Inc.
Router Architecture and IOS Internals
Cisco Systems, Inc., 2001.

David A. Patterson & John L. Hennessy
Computer Organization and Design: The Hardware/Software Interface
Chapter 5 — The Processor: Datapath and Control.

We often discuss AI systems as if the model were the entire system.

A model is benchmarked.

A model is compared to another model.

A larger model is assumed to be more capable.

But an AI agent is not just a model.

It is a model operating inside an environment.

That raises a question I have been exploring with two systems I have built: Alice and the Control Tower.

How much of an agent's operational capability comes from the model itself, and how much comes from the architecture surrounding it?

I don't have the final answer.

I have a hypothesis.

And I am now testing it.

The hypothesis

Consider two systems using exactly the same model.

System A
User
↓
Qwen3 32B
↓
Answer
System B
User
↓
Control / Routing
↓
Memory
↓
Tools
↓
Circuits
↓
Policies
↓
Qwen3 32B
↓
Observation
↓
Result

The underlying model has not changed.

The environment has.

My hypothesis is that the second system can convert the model's capabilities into more useful and reliable operational behaviour.

Not because the model became more intelligent.

Because it has a better system around it.

Why I started thinking about this

The architecture behind Alice contains several mechanisms that are normally treated as separate concerns:

routing;
memory;
caching;
tools;
circuits;
operational state;
local knowledge;
control mechanisms.

The Control Tower takes this idea further.

It provides a living representation of an environment containing agents, services, circuits, decisions, infrastructure and enterprise capabilities.

This led me to a different way of thinking about an AI agent.

Instead of:

Model → Tool → Result

I started thinking about:

Model
↓
Control
↓
Routing
↓
Context
↓
Tool / Circuit
↓
Execution
↓
Observation
↓
Memory

The model becomes a component of a larger computational system.

There is an interesting precedent in computing

This isn't an analogy I invented from scratch.

Computer architecture has long separated computation from control.

A processor contains a datapath, but also a control unit that determines which operations and paths are activated.

In the MIPS architecture described in Computer Organization and Design, different execution paths are selected depending on the decoded instruction, with a finite-state control mechanism determining subsequent actions.

Networking provides another useful analogy.

Cisco's architecture material explicitly separates:

Control Plane
Management Plane
Data Plane

The control plane maintains structures such as forwarding tables, while the data plane uses those structures to actually process and forward packets.

This leads to an interesting question:

Could agentic systems benefit from a similar separation between reasoning, control, routing and execution?

Alice is my first experiment

Alice was built around a Qwen model.

But the model is not operating alone.

The environment includes mechanisms for routing requests, accessing local knowledge, using cached results and interacting with operational circuits.

For example, one of the interfaces I built can route a request to a local SQLite cache instead of invoking the LLM when an appropriate result already exists.

That means:

Request
↓
Router
├── Cache hit → Result
│
└── Cache miss → LLM

The model doesn't need to reason about something that the system already knows.

This has obvious consequences for latency, cost and potentially reliability.

But it also raises a larger question:

Where does the capability actually come from?

The Control Tower provides another environment

The Control Tower introduces a larger operational context.

An agent can interact with:

a living map;
operational circuits;
enterprise systems;
memory and knowledge;
tools;
policies;
decisions;
observability mechanisms.

Some sensitive actions can become explicit decisions rather than direct agent actions.

The architecture therefore becomes closer to:

                AI AGENT
                   │
                   ▼
            ┌─────────────┐
            │   CONTROL   │
            │    TOWER    │
            └──────┬──────┘
                   │
      ┌────────────┼────────────┐
      ▼            ▼            ▼
   Memory       Circuits      Tools
      │            │            │
      └────────────┼────────────┘
                   ▼
            Decision / Policy
                   │
                   ▼
              Execution
                   │
                   ▼
              Observation

The important part is that the environment itself becomes structured.

The experiment

I am now connecting Qwen3 32B directly to the Control Tower.

The objective is deliberately simple:

Keep the environment constant and change the model environment.

The first comparison will be:

Qwen3 32B
vs
Qwen3 32B + Control Tower

Then, where useful:

Qwen3 32B + Control Tower
vs
Qwen3 32B + Alice

The same tasks should be given to each configuration.

No moving goalposts.

No changing the system just to make one model look better.

What will be measured?

Not just whether the final answer looks good.

I want to measure:

Task performance
correct diagnosis;
correct tool selection;
architecture understanding;
correct circuit selection;
successful completion.
Reliability
hallucinations;
invalid actions;
policy violations;
incorrect assumptions.
Efficiency
latency;
tokens;
number of LLM calls;
cache hits;
cost.
Operational behaviour
ability to navigate the environment;
ability to use existing knowledge;
ability to reason through circuits;
ability to respect the control mechanisms.
Then comes the more interesting experiment: ablation

Suppose the complete system performs better.

That still doesn't tell us why.

So the next step is to remove components.

Qwen
↓

Tools ↓
Memory ↓
Routing ↓
Circuits ↓
Control

Each configuration can be measured independently.

The objective is to determine whether individual architectural components contribute measurable improvements.

That would be much more interesting than simply saying:

"Alice works well."

What would success actually mean?

It would not prove that a smaller model has become more intelligent.

It would suggest something more specific:

The capabilities of an agent cannot be evaluated independently from the environment in which the model operates.

A model's intrinsic reasoning capability remains a property of the model.

But its operational capability may depend substantially on:

what information it can access;
how that information is structured;
how actions are routed;
what memory exists outside the model;
which tools are available;
how execution is constrained;
how feedback returns to the system.

In other words:

Model capability ≠ System capability

That distinction may become increasingly important as agentic systems move from chat interfaces into real operational environments.

And there is a possibility I find particularly interesting

If the experiment shows that a relatively modest model performs significantly better inside a structured control environment, we may have evidence for a simple but important principle:

Don't only make the model bigger. Improve the machine around the model.

Better routing.

Better context.

Better memory.

Better tools.

Better state management.

Better control.

Better feedback.

The model remains important.

But it is no longer the entire architecture.

This is still a hypothesis

I am deliberately not claiming that the Control Tower makes Qwen better.