DEV Community: juan gonzalez

I shipped a fix. The system recovered. The fix wasn't why.

juan gonzalez — Sat, 08 Aug 2026 19:02:51 +0000

An autonomous system of mine lost access to its execution environment and went into a crash loop. I read the logs, formed a theory, wrote a commit, deployed it. It came back up. I closed the incident and moved on.

A few days later I checked it properly, because not taking a system's word for itself is the work I do — and that has to include when the system is mine.

Here is what the check looked like.

The claim: a code change restored the system's access to its execution environment.

The source that could settle it: not my commit message, and not my memory of that evening. The operational log, the provider's own error code, and the diff of what the commit actually modified.

The test: compare the failure class in the log against the mechanism the commit changed. If they're the same mechanism, the claim holds. If they aren't, it can't hold, no matter how convincing the timing was.

The result: the incident was an authentication failure. My commit corrected clock synchronisation — a real bug, in a different failure class entirely. The two were never connected. Something else brought the system back, most likely an environment change I made around the same time and didn't record.

Verdict: FALSIFIED.

The limit: I still can't show which environment variable changed. Key rotation fits the evidence. It is not demonstrated, and I'm not going to write it down as if it were. The gap is part of the finding.

The shape of the reasoning

The uncomfortable part isn't being wrong about a cause. It's the shape of the reasoning, because it's the shape most of us use:

I deployed X. The problem stopped. Therefore X fixed it.

That holds up exactly as long as nobody checks. In most systems nobody does, because there's nothing forcing the question. The incident closed. The graph went green. The next thing was already on fire.

It gets worse with autonomous systems, and I think this part is under-discussed. Classic software failed loudly — an exception, a non-zero exit, a stack trace. Agents and pipelines fail quietly and keep reporting success. The path that executes and the path that reports are usually the same path. An agent says "done" because the command returned, not because the file exists. A dashboard says the traffic is human because the dashboard counts it that way.

In that architecture, the absence of errors tells you nothing at all.

What I do about it

I take one specific claim a system makes about itself and check it against a source the system can't write to.

Not an audit of the organisation. Not an implementation of the fix. One claim.

Four possible verdicts: confirmed, falsified, partially confirmed, not assessable. The last one is a real outcome, not a failure of the check. If a claim can't be tested, what you've found is a hole in your observability — and a system that can't demonstrate what it claims today won't be able to demonstrate it on the day it breaks either.

Three evidence levels, stated openly in every report: direct (read-only access), reproduced (you run the query, I read the output), declared (a statement, which doesn't stand on its own).

Some claims I check with no access at all, because the surface is already public — response headers, DNS, what an endpoint actually returns, what a downloadable artefact actually contains.

If you run one of these

If you operate an agent, a RAG pipeline, or an automation, and there's one sentence about it you'd be uncomfortable defending under questioning — that sentence is the interesting one.

I'm running a few of these free right now while I build the public record. You get the full report either way, including when the verdict is boring.

taiwildlab.com — Juan Gonzalez, TaiwildLab

A Shadow Decision System Found a Semantic Boundary Bug in 14 Minutes

juan gonzalez — Sun, 12 Jul 2026 11:18:00 +0000

I am running an experimental autonomous trading architecture called Darwin. During an audit of its agent-ranking and sizing authorities, I built a read-only shadow producer to compare a candidate constraint system against the existing runtime behavior.

The candidate had no operational authority:

no position or capital writes;
no lifecycle mutations;
no promotions or reproduction;
no writes to the operational sizing ledger;
separate persistence for shadow cycles, events and comparisons.

The first complete diagnostic cycle ran on July 11 at 18:22.

Results:

agents_evaluated: 25
matches: 8
mismatches: 17
max_absolute_delta: 0.5

Every mismatch had the same multiplicative shape:

expected candidate
0.500 1.000
0.425 0.850
0.350 0.700
0.275 0.550
0.200 0.400

This immediately suggested that one path applied a 0.5 constraint that the other did not.

The useful design decision was persisting the composition of each resolved multiplier, not only the final values.

The comparison data showed the semantic divergence:

expected_sources:
[reduce_50, anticonvergence_penalty]

candidate_sources:
[anticonvergence_penalty]

The discrepancy was therefore not primarily numerical. It was a disagreement about authority.
The legacy expected-value translator applied a 50% reduction whenever coherence was below 0.40.
The approved architecture defined two different domains:

0.30 <= coherence < 0.40
sizing authority
multiplier *= 0.5

coherence < 0.30
severe lifecycle condition
outside sizing authority

The candidate producer respected the new boundary. The dual-run oracle still encoded the legacy interpretation.
The bug was in the expected-value mechanism, not in the candidate.
The correction was committed at 18:36. The same condition was executed again at 18:37:

agents_evaluated: 25
matches: 25
mismatches: 0
max_absolute_delta: 0.0

The failed cycle was retained.
The operational databases remained unchanged, and the operational sizing ledger contained zero new events.

What made the diagnosis fast?
A shadow comparator that stores only:

expected_multiplier
candidate_multiplier
absolute_delta

can raise an alert.
A comparator that also stores:

expected_sources
candidate_sources
dominant_constraint
rule_version
input_snapshot

can produce a diagnosis.
That changes the debugging question from:
Where did the multiplication go wrong?
to:
Which rule was present on one decision path and absent on the other?
The second audit: residual authority
After correcting the mismatch, I opened a formal 48-hour observation window.
The population was supposed to remain fixed. Reproduction had already been disabled through:
REPRODUCTION_ACTIVE=false
However, an hourly cron entry still executed auto_promote.py.
No promotion occurred during the initial observation period, but the authority to mutate agent stages remained active.
The process was disabled, and the formal window was restarted from the first successful shadow cycle after the freeze.
This produced a useful distinction:
Absence of mutation is not absence of mutation authority.
For controlled observation, capabilities matter more than labels such as “reproduction disabled” or “read-only.”
General design principles
From this incident, I would extract five reusable rules:
Persist decision composition, not only resolved values.
Treat the expected-value oracle as a hypothesis, not as truth.
Separate permission to calculate from permission to enforce.
Preserve failed cycles as evidence.
Audit every component that retains authority over the observed state.

The shadow did not merely detect that two functions differed.

It made the semantic boundary visible.

La confianza no es una propiedad del agente. Es una propiedad de la arquitectura.

juan gonzalez — Tue, 07 Jul 2026 17:31:15 +0000

I Run 20+ Autonomous Trading Agents. Here Are the 5 Memory Failures That Almost Killed the System.

juan gonzalez — Sat, 11 Apr 2026 10:39:29 +0000