DEV Community: juan gonzalez

The Cost of Trusting the Wrong Claim in AI Systems

juan gonzalez — Sun, 04 Oct 2026 08:04:26 +0000

# Don’t Trust the Agent’s “Done”: Verify the System State

juan gonzalez — Thu, 01 Oct 2026 15:15:07 +0000

AI agents are increasingly allowed to do real work.
They deploy applications.
They restart services.
They write to databases.
They call APIs.
They trigger automation chains.
And when they finish, they usually return something like:

“Done.”
The problem is that a successful tool call is not necessarily proof that the expected real-world state exists.
That distinction becomes increasingly important as AI systems receive more execution authority.

Execution and verification are different jobs

Suppose an agent performs a deployment.
The agent reports:

Deployment successful.
Version 4.7.2 is running.

There are several things that could have happened:

the deployment command returned exit code 0;
an API accepted the request;
the orchestration layer reported success;
the agent interpreted the response correctly. But the claim we actually care about may be:

Production is currently serving version 4.7.2.

Those are not the same statement.
The first group describes execution events.
The second describes the resulting system state.
A system should not automatically be allowed to prove the second merely by reporting the first.

Treat the agent’s result as a claim

This is the model I use in SCC Runner:

Claim

Authority
↓
Evidence
↓
Verdict

The agent says:

The service is running.

That becomes a claim.
Now we ask:
What source has authority over that property?
For a systemd service, that might be the service mnager.
For example:

ActiveState=active

For a database operation, the database itself may be the authority.
For a deployment, the runtime environment may be more relevant than the deployment tool that initiated the change.
For an external transaction, the external API or ledger may be the relevant authority.
The important idea is simple:
Evidence should come from a source capable of observing the property being claimed.

Why another LLM is not enough

One common pattern is:

Agent A performs the task
↓
Agent B reviews Agent A

That may improve reasoning quality.
But it does not necessarily provide independent evidence.
If Agent A says:

“The database record exists.”
and Agent B only reviews Agent A’s explanation, Agent B still does not know whether the record actually exists.
Both models may be reasoning about the same self-report.
For claims about external reality, verification needs access to the relevant external authority.

Agent
↓
Claim
Database / Runtime / API / Git / Logs
↓
Evidence

The distinction is not about which model is smarter.
It is about where the evidence comes from.

“False” and “not observable” are different states

There is another important distinction.
Suppose we need to verify:

Service X is running.

But the authority cannot be reached.
That does not prove:

Service X is stopped.
``
It means something else:

text
NOT EVALUABLE
AUTHORITY UNAVAILABLE

Verification systems should preserve this distinction.
Otherwise temporary observability failures become false claims, or worse, unavailable evidence becomes accidental confirmation.
Explicit uncertainty is safer than invented certainty.
## From verification to execution gates
Once a claim can be verified reliably, the result can eventually participate in execution decisions.
Conceptually:

text
Claim
↓
Authority
↓
Evidence
↓
Verdict
↓
PASS / BLOCK

For example:

text
Claim:
Deployment is healthy.
Evidence:
expected version present
health endpoint reachable
required instances ready
Verdict:
CONFIRMED
Decision:
PASS

Or:

text
Claim:
Deployment is healthy.
Evidence:
required state cannot be established
Verdict:
NOT EVALUABLE
Decision:
BLOCK

That second behavior is what makes fail-closed verification interesting.
But I would not start there.
## Start in shadow mode
A new verification system should not receive production authority immediately.
A safer adoption path is:

text
SHADOW
↓
ADVISORY
↓
ENFORCED

### Shadow
Observe claims without influencing execution.
Compare the evidence with the checks humans already perform.
### Advisory
Produce evidence and verdicts.
Humans or existing systems still make the final decision.
### Enforced
Selected verification results become execution conditions.
Only after repeated observation should the verifier gain that authority.
Verification software should not ask to be trusted.
**It should demonstrate that it deserves more authority.**
## The goal is not another monitoring dashboard
Monitoring systems answer questions such as:

text
CPU = 42%
requests = 12,431
error rate = 0.3%

Those metrics can be extremely useful.
But verification asks a different question:
> Does the available evidence actually establish the claim required for this decision?
A log entry saying:

text
deployment completed

may be useful evidence.
But it may not be sufficient evidence for:

text
the new version is healthy in production

The claim determines what evidence is relevant.
## Where this becomes useful
This pattern is particularly useful for:
- AI agents with tool access;
- deployment systems;
- autonomous remediation;
- data pipelines;
- workflow automation;
- infrastructure operations;
- multi-agent systems;
- tool chains where one tool calls another tool.
The more layers between the agent and the final system state, the more dangerous it becomes to equate:

text
tool returned success

with:

text
desired outcome exists

## The simplest useful implementation
You do not need to begin with a large verification framework.
Start with one boring claim.
For example:

text
Claim:
Service nginx is active.
Authority:
systemd
Observed:
ActiveState=active
Verdict:
CONFIRMED

Run that verification next to your existing manual process.
Compare the results.
Repeat.
That is a much stronger foundation than beginning by giving a new system permission to block production
## Build systems that can prove what they did
As AI systems become capable of taking more actions, I think we need to separate two questions:

text
What did the agent say happened?
What can the system actually prove happened?



Those questions are often treated as if they were identical.
They are not.
That is the problem I am working on with SCC Runner.
The full canonical article is here:
https://taiwildlab.com/articles/scc-runner-dont-just-take-your-systems-word-for-it/
SCC Runner is currently available in **Free Early Access**:
https://scc.taiwildlab.com/signup

Why an Evolutionary System Needs Memory

juan gonzalez — Wed, 26 Aug 2026 10:07:27 +0000

I used to think an evolutionary system needed two things:
variation and selection.
Something changes. The system observes the result. Better variants survive.
That sounds reasonable — until you try to build one.
While working on an evolutionary trading system, we ran into a more basic problem:
If the system cannot reliably remember what changed, why it changed, and what happened afterward, it cannot prove that it is improving.
It can only prove that it is changing.
That distinction matters.
A system can modify parameters, promote agents, remove weak strategies, or generate new variants. From the outside, all of that looks like evolution.
But later you may want to ask:
Why was this agent promoted?
What evidence justified the decision?
What was the previous state?
What happened after the change?
Was the new state actually better?
If those answers cannot be reconstructed, the system has activity, but not real evolutionary memory.
Memory is more than storage
More logs are not enough.
More metrics are not enough.
Useful evolutionary memory needs structure.
A meaningful change should leave behind something like:
previous state → evidence → decision → change → resulting state
That chain gives us provenance.
It tells us not only what the system is now, but how it got there.
This is similar to software development.
A source file tells you what the code looks like today.
Git tells you how it became that way.
An evolutionary system needs the same kind of lineage.
Sometimes evolution means doing nothing
There is another consequence.
An evolutionary system should not change simply because it has permission to change.
If no candidate satisfies the evidence threshold, the correct action may be:
no change.
That is not failure.
It is discipline.
A system that always changes produces novelty.
A system that remembers, evaluates, and sometimes refuses to change has a chance to produce learning.
So my original equation:
Evolution = variation + selection
has changed.
I would now write:
Evolution = variation + selection + verifiable memory
Because without memory, selection cannot accumulate knowledge.
And without accumulated knowledge, a system may keep changing while repeatedly rediscovering the same mistakes.
The better question is no longer:
Can the system change itself?
It is:
Can the system explain, with evidence, why its current state exists?
If it can, then we may be getting closer to a system that actually evolves.

I shipped a fix. The system recovered. The fix wasn't why.

juan gonzalez — Sat, 08 Aug 2026 19:02:51 +0000

An autonomous system of mine lost access to its execution environment and went into a crash loop. I read the logs, formed a theory, wrote a commit, deployed it. It came back up. I closed the incident and moved on.

A few days later I checked it properly, because not taking a system's word for itself is the work I do — and that has to include when the system is mine.

Here is what the check looked like.

The claim: a code change restored the system's access to its execution environment.

The source that could settle it: not my commit message, and not my memory of that evening. The operational log, the provider's own error code, and the diff of what the commit actually modified.

The test: compare the failure class in the log against the mechanism the commit changed. If they're the same mechanism, the claim holds. If they aren't, it can't hold, no matter how convincing the timing was.

The result: the incident was an authentication failure. My commit corrected clock synchronisation — a real bug, in a different failure class entirely. The two were never connected. Something else brought the system back, most likely an environment change I made around the same time and didn't record.

Verdict: FALSIFIED.

The limit: I still can't show which environment variable changed. Key rotation fits the evidence. It is not demonstrated, and I'm not going to write it down as if it were. The gap is part of the finding.

The shape of the reasoning

The uncomfortable part isn't being wrong about a cause. It's the shape of the reasoning, because it's the shape most of us use:

I deployed X. The problem stopped. Therefore X fixed it.

That holds up exactly as long as nobody checks. In most systems nobody does, because there's nothing forcing the question. The incident closed. The graph went green. The next thing was already on fire.

It gets worse with autonomous systems, and I think this part is under-discussed. Classic software failed loudly — an exception, a non-zero exit, a stack trace. Agents and pipelines fail quietly and keep reporting success. The path that executes and the path that reports are usually the same path. An agent says "done" because the command returned, not because the file exists. A dashboard says the traffic is human because the dashboard counts it that way.

In that architecture, the absence of errors tells you nothing at all.

What I do about it

I take one specific claim a system makes about itself and check it against a source the system can't write to.

Not an audit of the organisation. Not an implementation of the fix. One claim.

Four possible verdicts: confirmed, falsified, partially confirmed, not assessable. The last one is a real outcome, not a failure of the check. If a claim can't be tested, what you've found is a hole in your observability — and a system that can't demonstrate what it claims today won't be able to demonstrate it on the day it breaks either.

Three evidence levels, stated openly in every report: direct (read-only access), reproduced (you run the query, I read the output), declared (a statement, which doesn't stand on its own).

Some claims I check with no access at all, because the surface is already public — response headers, DNS, what an endpoint actually returns, what a downloadable artefact actually contains.

If you run one of these

If you operate an agent, a RAG pipeline, or an automation, and there's one sentence about it you'd be uncomfortable defending under questioning — that sentence is the interesting one.

I'm running a few of these free right now while I build the public record. You get the full report either way, including when the verdict is boring.

taiwildlab.com — Juan Gonzalez, TaiwildLab

A Shadow Decision System Found a Semantic Boundary Bug in 14 Minutes

juan gonzalez — Sun, 12 Jul 2026 11:18:00 +0000

I am running an experimental autonomous trading architecture called Darwin. During an audit of its agent-ranking and sizing authorities, I built a read-only shadow producer to compare a candidate constraint system against the existing runtime behavior.

The candidate had no operational authority:

no position or capital writes;
no lifecycle mutations;
no promotions or reproduction;
no writes to the operational sizing ledger;
separate persistence for shadow cycles, events and comparisons.

The first complete diagnostic cycle ran on July 11 at 18:22.

Results:

agents_evaluated: 25
matches: 8
mismatches: 17
max_absolute_delta: 0.5

Every mismatch had the same multiplicative shape:

expected candidate
0.500 1.000
0.425 0.850
0.350 0.700
0.275 0.550
0.200 0.400

This immediately suggested that one path applied a 0.5 constraint that the other did not.

The useful design decision was persisting the composition of each resolved multiplier, not only the final values.

The comparison data showed the semantic divergence:

expected_sources:
[reduce_50, anticonvergence_penalty]

candidate_sources:
[anticonvergence_penalty]

The discrepancy was therefore not primarily numerical. It was a disagreement about authority.
The legacy expected-value translator applied a 50% reduction whenever coherence was below 0.40.
The approved architecture defined two different domains:

0.30 <= coherence < 0.40
sizing authority
multiplier *= 0.5

coherence < 0.30
severe lifecycle condition
outside sizing authority

The candidate producer respected the new boundary. The dual-run oracle still encoded the legacy interpretation.
The bug was in the expected-value mechanism, not in the candidate.
The correction was committed at 18:36. The same condition was executed again at 18:37:

agents_evaluated: 25
matches: 25
mismatches: 0
max_absolute_delta: 0.0

The failed cycle was retained.
The operational databases remained unchanged, and the operational sizing ledger contained zero new events.

What made the diagnosis fast?
A shadow comparator that stores only:

expected_multiplier
candidate_multiplier
absolute_delta

can raise an alert.
A comparator that also stores:

expected_sources
candidate_sources
dominant_constraint
rule_version
input_snapshot

can produce a diagnosis.
That changes the debugging question from:
Where did the multiplication go wrong?
to:
Which rule was present on one decision path and absent on the other?
The second audit: residual authority
After correcting the mismatch, I opened a formal 48-hour observation window.
The population was supposed to remain fixed. Reproduction had already been disabled through:
REPRODUCTION_ACTIVE=false
However, an hourly cron entry still executed auto_promote.py.
No promotion occurred during the initial observation period, but the authority to mutate agent stages remained active.
The process was disabled, and the formal window was restarted from the first successful shadow cycle after the freeze.
This produced a useful distinction:
Absence of mutation is not absence of mutation authority.
For controlled observation, capabilities matter more than labels such as “reproduction disabled” or “read-only.”
General design principles
From this incident, I would extract five reusable rules:
Persist decision composition, not only resolved values.
Treat the expected-value oracle as a hypothesis, not as truth.
Separate permission to calculate from permission to enforce.
Preserve failed cycles as evidence.
Audit every component that retains authority over the observed state.

The shadow did not merely detect that two functions differed.

It made the semantic boundary visible.

La confianza no es una propiedad del agente. Es una propiedad de la arquitectura.

juan gonzalez — Tue, 07 Jul 2026 17:31:15 +0000

I Run 20+ Autonomous Trading Agents. Here Are the 5 Memory Failures That Almost Killed the System.

juan gonzalez — Sat, 11 Apr 2026 10:39:29 +0000