AI agents are getting very good at doing things.
They can search databases, call APIs, modify tickets, draft code, update records, trigger workflows, and interact with production systems.
And that changes the engineering problem.
When an LLM only generates text, a bad answer is usually just that: a bad answer.
When an LLM can take an action, a bad answer can become a bad state change.
So the most important question in agent architecture is no longer:
Can the model figure out what to do?
It is:
Who decides whether the model should actually be allowed to do it?
Those are two very different responsibilities.
And I think one of the most useful principles for production AI agents is surprisingly simple:
Use the model to reason. Don’t automatically give it authority to execute.
The architecture that works beautifully in demos
A lot of agent demos reduce to something like this:
User → LLM → Tool → Action
The model receives a request.
It reasons about what should happen.
It selects a tool.
It generates the parameters.
The tool executes.
That is an incredibly productive abstraction.
It is also a risky one when the tool can affect something real.
The same probabilistic system is effectively doing two jobs:
- deciding what it believes should happen;
- authorizing that thing to happen.
You can try to fix this with prompting:
Always ask for confirmation before making important changes.
But that is still an instruction.
It is not a security boundary.
The difference becomes clearer when you compare the two architectures.
%%{init: {'theme':'base','themeVariables': {
'primaryTextColor':'#111827',
'secondaryTextColor':'#111827',
'tertiaryTextColor':'#111827',
'textColor':'#111827',
'edgeLabelBackground':'#FFFFFF',
'lineColor':'#4B5563'
}}}%%
flowchart LR
subgraph BAD["❌ Demo-Style Agent"]
direction LR
A["User"] --> B["🧠 LLM"]
B --> C["🔧 Tool"]
C --> D["💥 Real-World Action"]
end
subgraph GOOD["✅ Production-Oriented Agent"]
direction LR
E["User"] --> F["🔎 Evidence"]
F --> G["🧠 LLM"]
G --> H["🔍 Review"]
H --> I["🛡️ Code Gates"]
I --> J["👤 Approval"]
J --> K["🔐 Tool"]
K --> L["✅ Action"]
end
classDef bad fill:#FEE2E2,stroke:#DC2626,stroke-width:2px,color:#111827;
classDef good fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#111827;
classDef ai fill:#EDE9FE,stroke:#7C3AED,stroke-width:2px,color:#111827;
classDef guard fill:#DBEAFE,stroke:#2563EB,stroke-width:2px,color:#111827;
class A,B,C,D bad;
class E,F,J,K,L good;
class G,H ai;
class I guard;The second design has more moving parts.
That is intentional.
Because the system is separating:
- reasoning
- validation
- authorization
- execution
Those should not always belong to the same component.
1. Don’t use an LLM when deterministic code is enough
One of the easiest mistakes in AI engineering is using the model simply because the model is available.
Suppose incoming tasks fall into three broad categories:
Known mechanical condition
↓
Deterministic workflow
Needs interpretation
↓
AI investigation
High-risk or ambiguous
↓
Human review
If the routing decision can be made reliably in code, make it in code.
For example:
def classify(task):
if task.has_known_failure_signal:
return "deterministic"
if task.needs_investigation:
return "ai_investigation"
return "human_review"
The interesting part here is the default:
human_review
Not:
let_the_model_guess
LLMs are extremely valuable when a problem genuinely requires interpretation.
They do not need to become the control plane for everything around them.
This has practical benefits too:
- lower latency
- lower cost
- easier debugging
- predictable behavior
- deterministic regression testing
Use intelligence where intelligence is actually required.
2. Make reasoning structured
If another system component needs to inspect the model's output, don't make that component parse a paragraph.
Instead of asking the model to generate:
I believe the likely root cause is...
return something closer to:
{
"root_cause": "...",
"severity": "medium",
"missing_information": [],
"recommended_actions": [],
"citations": []
}
Schema-constrained output changes how the rest of the application can interact with the model.
Now downstream code can make checks such as:
risk_ok = diagnosis.severity in {"low", "medium"}
citations_present = bool(diagnosis.citations)
The model is no longer merely producing prose.
It is generating typed data consumed by a larger system.
That distinction becomes increasingly important as agent workflows become more complex.
3. Grounding does not automatically mean relevance
RAG introduces another subtle problem.
Suppose an LLM cites document:
issue-1842
Your application verifies:
citation_id in retrieved_documents
Great.
The citation is real.
But that only proves the model cited something retrieval returned.
It does not prove retrieval returned something useful.
Imagine the query concerns a concurrency bug, but the vector search returns three vaguely related caching incidents.
All three documents are real.
All three IDs are valid.
The LLM can still build an extremely confident, beautifully cited, completely wrong explanation from them.
So a stronger check may look more like:
groundedness_ok = all(
citation_id in retrieved_ids
and relevance_score[citation_id] >= MIN_RELEVANCE_SCORE
for citation_id in diagnosis.citations
)
Now the system checks two different properties:
Does the source exist?
↓
Provenance
Is the source sufficiently relevant?
↓
Retrieval quality
These are not the same thing.
That leads to a broader lesson:
“The model cited a real source” and “the model cited evidence that supports its claim” are different guarantees.
A RAG system can be perfectly citation-valid and still be badly grounded.
4. A second LLM can review the first — but it shouldn’t necessarily control the gate
A common agent pattern now looks like this:
LLM A
↓
Generate answer
LLM B
↓
Evaluate answer
"PASS"
↓
Proceed
This is already better than trusting one generation blindly.
But it still leaves an interesting question:
Why should another probabilistic model have the final authority?
A stronger architecture separates critique from enforcement.
LLM A
↓
Generate proposal
LLM B
↓
Critique proposal
Code
↓
Apply enforceable conditions
For example:
groundedness_ok = ...
risk_ok = ...
permission_ok = ...
approved = (
groundedness_ok
and risk_ok
and permission_ok
)
The reviewer model can still produce something very valuable:
This diagnosis appears weak because the cited evidence does
not fully support the proposed root cause...
That explanation is useful to a human.
But the system does not need to parse:
APPROVE
from the model's response and treat that string as authority.
The distinction is simple:
Let the model explain. Let deterministic systems enforce.
This becomes especially important for conditions like:
- permission scope
- severity thresholds
- resource ownership
- allowed operations
- schema validation
- rate limits
- approval state
These are usually better represented as explicit program state than as natural-language judgment.
5. Put safety checks inside the execution boundary
Imagine your workflow graph contains:
review → approval → execute
Everything looks safe.
But six months later someone refactors the graph.
A shortcut gets introduced:
review → execute
If approval existed only as orchestration logic, you just removed the safety control by changing one edge.
A stronger design puts the check inside the function that performs the mutation.
def execute(state):
if not state.get("approved"):
raise PermissionError(
"Execution requires explicit approval."
)
perform_action()
Now you have two protections.
The graph says:
You should not reach execute yet.
The execution boundary says:
Even if you reach me, I refuse to run.
That is defense in depth.
And this idea generalizes far beyond AI.
Security-sensitive properties should ideally be enforced as close as possible to the resource being protected.
6. Human approval should be a real execution pause
Many systems technically have a human approval screen.
But underneath, the implementation is surprisingly fragile.
Maybe the workflow state exists only in memory.
Maybe the process is just waiting.
Maybe the exact action gets regenerated after approval.
A stronger human-in-the-loop design looks like this:
Agent proposes action
↓
Workflow suspends
↓
[minutes / hours / days]
↓
Human approves
↓
The exact approved action executes
This creates an infrastructure requirement that is easy to miss:
the state of the paused workflow must survive independently of the application process.
If the application container disappears, the approval state must not disappear with it.
Conceptually:
Agent Runtime
↓
Checkpoint
↓
Persistent Storage
That lets the process restart completely while the workflow remains resumable.
This matters in real deployments because:
- containers restart
- instances scale down
- deployments replace running processes
- infrastructure fails
A human approval system that only works while one Python process stays alive is not really durable human approval.
7. Execute exactly what was approved
There is another subtle detail here.
Suppose the agent presents this to the user:
I propose posting comment X.
The human approves it.
Then the application asks the LLM:
Generate the final comment.
That creates a new output.
The human never approved the new output.
Instead, approval should usually bind to a concrete proposed action:
proposed_action = build_action(state)
approved = wait_for_human(proposed_action)
if approved:
execute(proposed_action)
No regeneration.
No reinterpretation.
No second chance for model variance.
The artifact the human reviews should be the artifact that crosses the mutation boundary.
8. Think of an agent as layers of trust
Once you combine these ideas, the architecture starts looking less like a chatbot with tools and more like a proper software system.
%%{init: {'theme':'base','themeVariables': {
'primaryTextColor':'#111827',
'secondaryTextColor':'#111827',
'tertiaryTextColor':'#111827',
'textColor':'#111827',
'edgeLabelBackground':'#FFFFFF',
'lineColor':'#4B5563'
}}}%%
flowchart TD
A["📥 User Request / Event"] --> B["🔎 Gather Evidence"]
B --> C{"🧭 Deterministic Classification"}
C -->|"Known / Mechanical"| D["⚙️ Deterministic Path"]
C -->|"Needs Investigation"| E["🧠 LLM Reasoning"]
C -->|"Ambiguous / High Risk"| H["👤 Human Review"]
E --> F["🔍 Independent LLM Review"]
F --> G{"🛡️ Code-Enforced Gates"}
G -->|"Grounded ✓<br/>Risk ✓<br/>Permission ✓"| I["⏸️ Human Approval"]
G -->|"Any Gate Fails"| H
D --> I
I -->|"Approved"| J["🚀 Execute Action"]
I -->|"Rejected"| K["🛑 Stop"]
J --> L["🌐 External System"]
classDef input fill:#F3F4F6,stroke:#4B5563,stroke-width:2px,color:#111827;
classDef deterministic fill:#DBEAFE,stroke:#2563EB,stroke-width:2px,color:#111827;
classDef ai fill:#EDE9FE,stroke:#7C3AED,stroke-width:2px,color:#111827;
classDef gate fill:#FFEDD5,stroke:#EA580C,stroke-width:2px,color:#111827;
classDef human fill:#FEE2E2,stroke:#DC2626,stroke-width:2px,color:#111827;
classDef execute fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#111827;
class A input;
class B,C,D deterministic;
class E,F ai;
class G gate;
class H,I,K human;
class J,L execute;Each component has a different responsibility.
Evidence layer
Answers:
What do we actually know?
LLM layer
Answers:
Given the available evidence, what might this mean?
Review layer
Answers:
What might be wrong with that reasoning?
Deterministic gate
Answers:
Are the machine-enforceable conditions satisfied?
Human approval
Answers:
Do we actually want this action to happen?
Execution boundary
Answers:
Is this exact operation authorized right now?
These are different questions.
Trying to answer all of them with one LLM call creates unnecessary coupling.
9. Think of LLMs as probabilistic components inside deterministic systems
This is probably the mental model I find most useful.
An agent doesn't need to be either:
fully deterministic
or:
fully AI-controlled
The system can deliberately alternate between probabilistic and deterministic stages.
%%{init: {'theme':'base','themeVariables': {
'primaryTextColor':'#111827',
'secondaryTextColor':'#111827',
'tertiaryTextColor':'#111827',
'textColor':'#111827',
'edgeLabelBackground':'#FFFFFF',
'lineColor':'#4B5563'
}}}%%
flowchart LR
A["🧠 LLM<br/>Reason"] --> B["📋 Proposed Action"]
B --> C["🔍 Independent Review"]
C --> D{"🛡️ Deterministic Gates"}
D -->|"PASS"| E["👤 Human Approval"]
D -->|"FAIL"| F["🚨 Escalate"]
E -->|"Approve"| G["🔐 Execution Boundary"]
E -->|"Reject"| H["🛑 Stop"]
G --> I["⚡ Tool / API"]
subgraph Intelligence["Probabilistic Layer"]
A
B
C
end
subgraph Control["Deterministic Control Layer"]
D
G
end
subgraph Authority["Human Authority"]
E
F
H
end
classDef model fill:#EDE9FE,stroke:#7C3AED,stroke-width:2px,color:#111827;
classDef control fill:#DBEAFE,stroke:#2563EB,stroke-width:2px,color:#111827;
classDef human fill:#FFEDD5,stroke:#EA580C,stroke-width:2px,color:#111827;
classDef action fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#111827;
class A,B,C model;
class D,G control;
class E,F,H human;
class I action;The probabilistic layer is allowed to be flexible.
The control layer is not.
That is a useful distinction.
10. Evaluate capability and safety differently
Agent evaluation becomes much clearer once you stop treating every metric the same way.
Consider:
How often does the model correctly diagnose the issue?
Maybe the answer is:
82%
Then you improve retrieval.
87%
Then improve the model.
91%
That's normal.
This is a capability evaluation.
Now consider:
Does the execution function reject requests without approval?
The acceptable score is:
100%
Not:
97%
Not:
99.7%
Why?
Because these tests measure fundamentally different things.
One asks:
How intelligent is the system?
The other asks:
Can a safety invariant ever be violated?
A model-quality test may reasonably be statistical.
A permission boundary should usually be deterministic.
So it is useful to maintain separate evaluation categories.
Capability evals
Examples:
- diagnosis correctness
- answer quality
- citation relevance
- summarization quality
- tool-selection accuracy
These may improve gradually.
Safety regression tests
Examples:
- unauthorized execution rejected
- high-risk action always escalates
- invalid tool parameters blocked
- missing approval prevents writes
- permission scope enforced
These should generally have a much harder threshold.
One bypassed safety gate isn't something you average away.
11. Failure modes become easier to reason about
Once the architecture is separated, failures become much easier to locate.
Suppose an incorrect action was proposed.
You can ask:
Was the evidence bad?
Was retrieval irrelevant?
Did the reasoning fail?
Did the reviewer miss it?
Did a deterministic gate fail?
Was the human shown the wrong artifact?
Did execution violate authorization?
Those are diagnosable boundaries.
Compare that with:
The agent did something weird.
Modularity isn't only about clean architecture.
It dramatically improves observability.
12. This is where AI engineering starts looking like ordinary systems engineering
The first wave of agent development focused heavily on:
- prompts
- tool calling
- function schemas
- reasoning loops
Those are still important.
But once agents affect real systems, the harder questions start looking familiar.
Security
Who is allowed to perform this action?
Distributed systems
Where does workflow state live if a process disappears?
Reliability
What happens after retries, partial failures, and timeouts?
Observability
Can I reconstruct why an action was proposed?
API design
What is the actual mutation boundary?
Testing
Which behaviors can tolerate probabilistic failure, and which absolutely cannot?
Human-computer interaction
What exactly is the human approving?
In other words:
Building reliable AI agents eventually becomes software engineering again.
The LLM is an extraordinarily powerful component.
But it is still a component.
A simple design checklist
When building an agent that can make real changes, these are the questions I now find most useful.
Reasoning
- Does this decision genuinely require an LLM?
- Can deterministic routing handle part of it?
- Is the model output structured?
Grounding
- Are citations validated?
- Is retrieval relevance measured?
- What happens when no sufficiently relevant evidence exists?
Review
- Is the evaluator independent from the generator?
- Does the reviewer provide critique or actual authority?
- Which conditions can be checked deterministically?
Authorization
- Is approval stored explicitly?
- Does the execution function verify it itself?
- Are high-risk operations handled differently?
Human-in-the-loop
- Does execution actually pause?
- Can the pause survive process restarts?
- Does the exact artifact approved by the human get executed?
Evaluation
- Which tests measure capability?
- Which tests protect invariants?
- Which failures are acceptable statistically?
- Which failures should never occur?
The bigger lesson
The interesting question in AI engineering is slowly changing.
It used to be:
How do I make an LLM call a tool?
Now it is increasingly:
How do I build a trustworthy system around a component that is intentionally probabilistic?
That requires more than prompting.
It requires architecture.
It requires deciding where intelligence belongs and where guarantees belong.
It requires treating authorization differently from reasoning.
And it requires accepting that sometimes the best component for an AI system is...
ordinary code.
So if I had to reduce the whole architecture to one principle, it would be this:
Use AI for what requires intelligence. Use code for what requires guarantees.
And don't confuse the two.
Top comments (0)