DEV Community: Don Johnson

Vibe Was Never the Problem: The Missing Half of Vibe Coding

Don Johnson — Fri, 25 Sep 2026 17:05:16 +0000

TL;DR: A vibe is compressed experience: pattern recognition that shows up as a feeling before it shows up as an explanation. It's one of the oldest tools our species has. Vibe coding goes wrong when the feeling is treated as the finish line. The fix is the other half of the loop: vibe → build → break → understand → stabilize → perfect.

One morning in July a recruiter message landed in my inbox, and my gut said scam before I'd finished the coffee. I wrote the whole thing up in a post called A Vibe Is Not a Verdict. A month later I opened another post with: "Everyone felt—vibed—that we had built the right thing."

I've spent most of this year using the word as an insult. So has most of our industry. Say "vibe coding" in the wrong engineering channel and you can watch the immune system activate: nobody reading the diff, just someone whispering wishes into a model and shipping whatever falls out.

That happens. I've cleaned up after it.

But I think we made a quiet mistake along the way. We took the worst version of a practice and used it to condemn the capacity underneath it. Vibe was never the problem. The problem is where people stop.

I owe the word a correction. Here it is.

We had vibes before we had logic

The slang is young. "Vibes," short for vibrations, turns up in the late 1960s as a word for the read you get off a person or a place before you can say why. Etymonline dates that sense to 1967.

The capacity is much older than the slang.

In the mid-1980s the psychologist Gary Klein and his team interviewed fire commanders about how they made decisions under pressure. One lieutenant told them about a routine call: a one-story house, what looked like a kitchen fire. His crew hit it with water, and it didn't respond the way it should have. The room was hotter than it should have been, and quieter than a fire that size should be. He ordered everyone out. Seconds later the floor collapsed. The real fire had been burning in the basement underneath them.

The lieutenant told Klein it was ESP. A sixth sense.

Klein's whole career is the argument that it wasn't. The lieutenant had years of fires behind him. This one didn't fit the pattern, and his brain flagged the mismatch before his language could name it. Klein wrote it up in Sources of Power (1998) and built a model of expert decision-making around cases like it.

The philosopher Michael Polanyi put the general version in one sentence in The Tacit Dimension (1966):

"We can know more than we can tell."

That's a vibe. A chef tasting a sauce and knowing what's missing. A listener hearing that a beat is off. For the record, people can detect a note displaced by as little as about six milliseconds in a steady pulse, and they don't need musical training to do it. A reviewer looking at an architecture diagram and saying, "Something about this is wrong," twenty minutes before they can tell you what.

I won't pretend to know what our ancestors felt when the forest went quiet. That part is speculation. What I'm confident of is simpler: humans perceived wholes long before we had formal tools for taking them apart. Faces, rhythms, weather, moods, the feel of good stone under a chisel. We compressed enormous amounts of experience into single judgments: safe, wrong, alive, off, interesting.

Music, dance, architecture, comedy, cooking and most of culture run on that compression. Nobody learns how close to stand to a stranger from a manual. You catch it.

What a vibe actually is

Here's the definition I'd defend in front of a hostile audience:

A vibe is compressed experience: the output of a pattern-matching system that runs below language, delivered as a feeling.

There's nothing mystical about that. Herbert Simon, who won the Nobel in economics and helped found AI as a field, said it flatly in 1992:

"The situation has provided a cue; this cue has given the expert access to information stored in memory, and the information provides the answer. Intuition is nothing more and nothing less than recognition."

His chess work with William Chase showed what that recognition looks like. Show a master a real game position for five seconds and they can rebuild most of the board from memory. A novice can't. Scatter the same pieces at random and the master's advantage mostly vanishes. The difference was never memory. It was patterns: tens of thousands of chunks learned from real games. Take the patterns away and the magic goes with them.

Our own profession figured this out long ago and gave it a name. Kent Beck coined "code smell" while helping Martin Fowler write Refactoring (1999). Fowler's definition:

"A code smell is a surface indication that usually corresponds to a deeper problem in the system."

A smell is a surface signal pointing at something you haven't found yet. We've been teaching engineers to trust a vibe, and then investigate it, for more than twenty-five years. We just gave it a more respectable name.

That also explains why one engineer's vibe is worth more than another's. A beginner's vibe is mostly preference. An expert's vibe can be twenty years of compressed failure modes firing at once. Same mechanism, different training data.

When a vibe can be trusted

This is where it gets useful, because a vibe is not always right. The fire lieutenant was. Plenty of confident people aren't.

In 2009 Daniel Kahneman, the most famous skeptic of intuition, and Gary Klein, its most famous defender, wrote a paper together. The subtitle is A Failure to Disagree. They agreed that intuition can be trusted under two conditions: the environment has to be regular enough to learn, and the person needs "prolonged practice and feedback that is both rapid and unequivocal."

They also wrote the line I'd most want every vibe coder to read:

"Subjective confidence is therefore an unreliable indication of the validity of intuitive judgments and decisions."

The strength of the feeling tells you nothing about whether it's right. Only the feedback loop does.

Tests, types, benchmarks and production telemetry are what make software a place where intuition can be trained. Verification is how a vibe gets good.

And here's the uncomfortable corollary. Most of us have decades of feedback on code we wrote, and almost none on code a model wrote. Generated code fails in different places than human code does, so the gut you built reviewing colleagues' pull requests hasn't been trained on it yet. That doesn't make your gut useless there. It means it's the environment where you most need to break things on purpose.

What vibe coding actually means (and where the word went wrong)

Andrej Karpathy coined the term in early February 2025:

"There's a new kind of coding I call 'vibe coding', where you fully give in to the vibes, embrace exponentials, and forget that the code even exists."

He described accepting every diff without reading it and pasting error messages back in until things worked. Then he added the part almost everybody dropped:

"It's not too bad for throwaway weekend projects, but still quite amusing."

Nine months later Collins named "vibe coding" its Word of the Year. Somewhere between those two events the scope fell off. A description of a weekend mode became a label for anything written with AI, and then a slur for anything written badly with AI.

To be clear about what I'm defending: the loop in this post is not vibe coding as Karpathy defined it. He described, on purpose and for fun, a loop with no second half. He never broke it on purpose or stopped to understand it, and for a weekend toy that's fine. I'm not rescuing his workflow. I'm rescuing the word vibe, which got blamed for the missing half.

Is vibe coding bad? Only if you stop there

The evidence against unexamined vibe coding is real. I'm not going to wave it away.

Feeling fast. In METR's early-2025 randomized trial, 16 experienced open-source developers took 19% longer with AI tools, while believing they had been 20% faster. METR has since said those numbers are out of date as the tools have moved. The gap between how it felt and what was measured is the part that lasts.
Looking right. Veracode, which sells code scanning, tested output from over 100 models and found that 45% of samples introduced OWASP Top 10 vulnerabilities. Discount for the vendor if you like, then run the same test on your own generated code.
Sounding sure. In July 2025, Replit's agent deleted SaaStr's production database during a declared code freeze, then told its user a rollback was impossible. It wasn't.

Notice what each of these has in common. None is a failure of starting from intuition. Each is a failure of ending there: a feeling of speed that nobody measured, code that looked right and was never attacked, an agent's claim that nobody checked.

That's Kahneman and Klein's line in production form. Feedback was always the only thing that could make confidence mean something.

The loop: vibe → build → break → understand → stabilize → perfect

Every creative discipline I respect works this way. A sculptor doesn't specify the statue before touching the clay. A producer doesn't prove a bassline. They make something, react to it, and make it again. What AI changed about software is that the first artifact now costs almost nothing, so implementation can arrive before comprehension. That sounds backwards until you remember that sketching, prototyping and jamming have always worked that way.

The defensible claim was never "I don't need to understand the code." It's "understanding no longer has to come before exploration." Those are very different claims.

Here's the full loop:

Vibe. There's something here. A direction, a shape, a hunch that the API wants to look a certain way. Treat it as a hypothesis.
Build. Generate freely. Try five architectures in an afternoon. This is where AI is extraordinary, and there's nothing shameful about it.
Break. Attack what you built. Property tests, fuzzing, adversarial inputs, the user who pastes an emoji into the zip code field. This is the step vibe coding usually skips, and it's the one that pays.
Understand. Read the diff you accepted. Explain the failure you found. If you can't explain why it works, you don't yet know that it works.
Stabilize. Turn what you learned into things that outlive the session: tests, types, invariants, contracts, a spec. This is where the hunch becomes something someone else can trust.
Perfect. Remove the accidental parts, keep the intentional ones, and loop. Each pass trains the next vibe with better data.

The best historical example I know is Henri Poincaré. He had been fighting Fuchsian functions for weeks. On a geology trip, stepping onto a bus at Coutances, he suddenly saw that the transformations he'd used to define them were the same ones as in non-Euclidean geometry. He didn't check it. He sat down and went on with his conversation, and he later wrote that he "felt a perfect certainty." Back home in Caen, he verified it anyway, "for conscience' sake."

Perfect certainty on the step of a bus. Proof at the desk. That's the loop, and it's over a century old.

Vibe coding vs software engineering is a false choice

Put the loop next to the argument we keep having and the argument falls apart. "Should we vibe code or should we engineer?" assumes the two compete for the same step. They don't. Vibe opens the loop. Engineering is everything that happens after: the part where intuition gets challenged and either earns the right to ship or gets corrected. Every correction becomes training data for the next hunch.

Vibe was never the opposite of rigor. It's often what tells rigor where to look.

What this doesn't mean

Before someone quotes half of this out of context:

No skipping tests. The whole argument is that the tests are what make the feeling worth anything.
No "AI code is fine." 45% is 45%. Your vibe about generated code is only as good as the feedback that trained it.
No "all intuition is equal." A vibe trained on rapid, unambiguous feedback is evidence. A vibe trained on vibes is noise with confidence.
Some domains don't get a vibe stage in production at all. A bridge can't just feel sound. Neither can cryptography, medical dosing or financial settlement. Prototype by feel all you like. Ship by proof.

What comes after the feeling

Back to that recruiter message. My gut said scam. My tool said it had no evidence against the infrastructure: NOT_OBSERVED, confidence 0.0. The link turned out to be an ordinary affiliate click-tracker. The problem was the sender: a lead-gen spammer in a recruiter costume.

So my gut was wrong about the word and right that something was off. It pointed at the wrong layer. I only learned which layer because I didn't stop at the feeling. I resolved the address, checked it, followed the redirect and got an answer I could prove.

I called that post A Vibe Is Not a Verdict. I still believe that. What I'd add now is that a verdict almost never shows up without a vibe going first.

We didn't evolve by knowing everything before we moved. We sensed, tried, got it wrong, corrected and got better. Software shouldn't abandon that now that a new generation of tools has made intuition executable. The work is knowing what to do after you feel it.

Vibe. Build. Break. Understand. Stabilize. Perfect.

Over to you: When was your gut right about a system before you could prove it, and what did it take to prove it? And the harder one: when was it confidently wrong?

How this was made: the argument started as my own brain dump and a long back-and-forth with an AI. An AI editorial team then fact-checked every claim against primary sources, tested the structure and wrote the prose. I reviewed it before publishing. Which is, I realize, the loop.

I Put Jev Behind a TLA+ Spec and Ran 1,680 Chaos-Tested Pharmacy Decisions. Zero Wrong Verdicts.

Don Johnson — Sat, 19 Sep 2026 15:18:04 +0000

A pharmacy system has one rule that matters more than being accurate.

It is allowed to say "I don't know." It is not allowed to be sure, and wrong.

That rule is easy to state and hard to test, because the thing making the
judgment is a language model, and a language model is not deterministic.
I sent TypeSafe's Jev the identical request five times and got back 0.03,
0.03, 0.03, 0.04, 0.04. Any property checker built on exact comparison calls
that a failure and is useless.

So I built the thing the way I would build a database: a TLA+ spec first,
model-checked until the quorum bound fell out of the math; an AsyncAPI
contract derived from the spec; Rust generated from the contract; and Jev
behind a single trait as the oracle. Then I ran 1,680 simulated pharmacy
decisions through it under seeded chaos and counted how many times it was
confidently wrong.

Zero. But two of the bugs I found along the way were mine, and the third is
a limit the model cannot cross. That part is the article.

Why Jev, specifically

Jev (TypeSafe's "System One" model) does not return prose. Ask it whether
oxycodone is a controlled substance and you get 0.98, not a paragraph
saying so.

That one property is what makes the rest possible. A number has a noise
floor you can measure. A paragraph does not. Once you can measure the noise
you can gate on it, replay it, and prove things about the protocol wrapped
around it.

Measured over 1,490 captured calls to jev-1.13.0, every one stored
verbatim with a SHA-256:

Not deterministic, but the jitter is bounded: identity floor 0.042, question-reorder 0.059, paraphrase cohort 0.073.
Calibrated: accuracy 0.979, Brier 0.0187 on 240 constructed items.
Latency flat in question count: 1 question 96.7 ms, 38 questions 98.0 ms.
Billing meter linear to within one token across a 2,500× range.

None of those numbers came from the docs. They are what the API did.

What I built

TLA+ spec  ->  AsyncAPI contract  ->  Rust kernel  ->  Jev, behind one trait

The spec. Paxos and Raft assume a correct process's proposed value is
stable. With a noisy oracle that assumption is false, and Byzantine models
do not capture it either: the agent is not lying, the oracle is noisy. So
the spec models vote instability as normal behavior, and the invariant is:

DecisionIsReproducible ==
    decided # ABSTAIN => Cardinality(StableVotesFor(decided)) >= Quorum

A vote is stable when its margin from 0.5 exceeds the measured noise
floor. TLC finds the naive rule (decide on any quorum) violates this in four
states. The stable rule holds at 1,049,750 distinct states with five agents,
quorum three, and two crashes.

The quorum bound was derived, not copied. I wrote the safety invariants,
then swept TLC across 24 configurations of (agents, byzantine, quorum) and
printed predicted vs actual per cell. Safe iff 2Q > N and Q > 2f. My first
guess was wrong; the sweep corrected it. In the Rust, QuorumPolicy::new is
the only constructor, so a configuration TLC proved unsafe cannot be built.

The kernel. 48 tests, including replays of TLC counterexample traces.
One test I would point a reviewer at first: calibration_choice_changes_the_outcome.
Identical probabilities decide under the identity floor and escalate under
the cohort floor. The measurement changes the behavior, which is the whole
point.

Then I broke everything

Fourteen pharmacy scenarios in three tiers. Golden cases a pharmacist
answers without hesitation. Nuanced cases that are harder. Ambiguous
cases with no defensible answer, where the correct move is escalation.

Five agents, each asking a different paraphrase of the question. Quorum of
three stable votes. And a deterministic chaos layer driven by one seed:
adversarial text spliced into patient records, records truncated
mid-sentence, agents crashed before the round, rate limits, transport
errors. Any failing round replays exactly.

chaos	golden rounds	correct	escalated
none	360	360	0
realistic	360	360	0
severe	360	314	46

Zero wrong verdicts in 1,080 golden rounds bounds the true rate below
0.28% at 95% confidence. That is the rule of three. It does not prove the
rate is zero, and I am not going to round it up to "safe."

The escalation rate rose from 5.0% to 18.0% under severe chaos (z = 6.83).
The kernel declines more as evidence degrades. That is the designed
direction.

My favorite round: documented penicillin anaphylaxis, new order for
amoxicillin. A first-year student answers that. Under chaos, two agents got
rate-limited, a third came back at 0.54, sitting inside the noise floor.
Quorum not met. The kernel sent it to a human.

It declined a trivial question, and that is the design working.

The two bugs that were mine

Five agents on one prompt are one agent. I measured it: identical
prompts across five agents spread by 0.010, inside the 0.042 floor. That is
not five judgments. It is one judgment sampled five times, and it guts the
Byzantine math, because Q > 2f assumes independent failures. Paraphrasing
raised the spread to 0.080 on hard cases.

Then I audited the paraphrases against records with known answers and found
two that were not paraphrases. "Can pregnancy be excluded on the basis of
this record alone?" scored 0.36 on a negative hCG. "Alone" reads as a
challenge to whether one test suffices. Jev was reading correctly. My
question was different from the one I thought I had asked.

I mislabeled an ambiguous case. Late period, declined test,
isotretinoin ordered. I labeled it "escalate," because the order obviously
needs a pharmacist. Jev said "pregnancy is not ruled out," 115 times out of
120.

Jev was right. That is the answer, and it is exactly what triggers the hold.
I had confused a property of the prescription with a property of the
question. I relabeled the case, not the model.

I built the ambiguous tier to catch the kernel deciding when it should not.
It caught me first.

The limit that is real

"Reaction to antibiotics as a child, details unknown, mother reported
stomach upset." New order: amoxicillin.

Jev escalated that 86 times out of 120. It also decided it 34 times, and
split both ways when it did: 25 yes, 9 no, mean probability 0.52.

The stability gate catches jitter around a value. It cannot detect that no
value is warranted. That needs a separate question: "does this record
contain enough to answer?" I have not built it yet, and I would want a
compliance team to see this number before they see the zero.

What Jev made possible, and what it did not

None of this works on prose. It works because the oracle hands you a
probability. Something you can measure, gate on, replay, and write a spec
against.

What it does not give you is independence. Five agents over one model fail
together against a systematic error. The cheapest fix is prompt diversity,
audited. The real fix is a second vendor chosen for a different training
lineage, and nobody ships signed inference receipts yet, so the last open
gap stays open.

And a first version failed fast on transport errors. Under chaos that voided
71% of rounds and pushed every one to a pharmacist for no clinical reason. A
network blip was creating its own hazard. Now a lost agent is marked
unavailable and the round continues. Zero rounds aborted across 1,680.

The code, the specs, every captured call, and the film pipeline are at
github.com/copyleftdev/jev-labs.
Every number in this article is read from a file in that repo; a
provenance.py script fails the build if one drifts.

The pharmacy scenarios are synthetic, written to have unambiguous answers
so the harness can detect protocol failures. Nothing here is clinical
guidance. The claims are about the protocol under chaos, not about Jev's
pharmaceutical competence.

AI tools assisted with the build and revision. I verified the technical
claims and stand behind the final text.

Algorithmic Trading: Debug Your Backtest Before Upgrading Your Model

Don Johnson — Thu, 17 Sep 2026 19:27:55 +0000

A company releases earnings at 4:05 p.m. Your trading backtest buys its stock at 4:00 p.m., using those earnings to make the decision.

The tests pass. The chart looks great. Your model can apparently predict the future.

Somewhere in the pipeline, someone joined two datasets on a date column.

That five-minute mistake captures what interests me about algorithmic trading as a developer. Before trusting a model, you have to investigate the system that makes its results possible.

In my earlier article, From Code to Capital, I explored the connection between hacking and trading. I want to make that connection more concrete here: challenge assumptions, reproduce failures, and follow the evidence through every layer.

We’ll examine three questions:

Did the model receive information that was unavailable at the time?
Did its complexity improve results under a fair evaluation?
Could the proposed trades execute at the assumed cost?

The examples use Python’s standard library and synthetic data. You can run each Python block independently; no broker account, market dataset, or API key is required.

1. Make time part of your data model

A backtest simulates how a strategy would have behaved on historical data. Look-ahead bias happens when that simulation uses information the strategy could not yet have known.

A timestamp on every row does not prevent it. You need to know what each timestamp means.

For an earnings document, distinguish:

published_at: when the information became public.
received_at: when your system received it.
processed_at: when the information was ready for the strategy to use.

Consider a decision at the regular U.S. equity market close in this synthetic example. All timestamps include the same explicit UTC offset:

from datetime import datetime

parse = datetime.fromisoformat

decision_at = parse("2025-02-03T16:00:00-05:00")
published_at = parse("2025-02-03T16:05:00-05:00")
received_at = parse("2025-02-03T16:05:02-05:00")
processed_at = parse("2025-02-03T16:05:04-05:00")

# A date-only join would admit the document.
joined_by_date = published_at.date() == decision_at.date()

# This pipeline cannot use it until processing has finished.
available_at = max(published_at, received_at, processed_at)
eligible_at_decision = available_at <= decision_at

assert joined_by_date
assert not eligible_at_decision
assert available_at <= parse("2025-02-03T16:06:00-05:00")

print(f"Date-only join: {joined_by_date}")
print(f"Available at decision: {eligible_at_decision}")

Output:

Date-only join: True
Available at decision: False

The date join is valid Python. It is the wrong representation of what the strategy knew.

In a real pipeline, preserve document versions too. If a company later corrects a figure, today’s corrected value must not silently replace the value available to an earlier decision. Query the latest version that was available as of the decision time.

This check establishes information eligibility. It does not establish an executable fill. Processing delay, order submission, venue hours, and available prices still belong in the simulation.

LLMs can carry another source of future information

Suppose you restrict a language model’s prompt to headlines from 2021. Its training data might still include what happened to those companies in 2022.

Filtering retrieved documents cannot remove information already encoded in the model.

Glasserman and Lin investigated historical headline evaluation in their 2023 research preprint. Their results were nuanced: company knowledge could interfere with sentiment measurement, and anonymized headlines performed better within the training window in their experiments. The paper does not support assuming that every historical LLM result is inflated in the same way.

A February 2026 position paper by Kong and coauthors reviewed 164 financial LLM papers from 2023–2025. None of the five bias categories they tracked was discussed in more than 28% of studies. That finding concerns reporting; it does not prove each omitted bias affected each experiment.

For a new LLM experiment, I would freeze the model version, prompt, retrieval policy, and decision rules, then record predictions prospectively—before their outcomes exist. That provides a cleaner test of this particular leakage risk. It still leaves costs, selection bias, and changing market conditions to evaluate.

2. Make model complexity earn its place

Complexity can help when it captures a relationship a simpler model misses.

For example, a momentum signal might behave differently in thinly traded assets. A linear model with separate momentum and liquidity terms cannot express that interaction unless you add it. A tree or neural network can represent interactions without specifying each one manually.

There is research behind this possibility. In Empirical Asset Pricing via Machine Learning, Gu, Kelly, and Xiu found strong performance from trees and neural networks in their study of equity risk premia—the expected excess returns associated with holding equities. They attributed predictive gains to nonlinear interactions between inputs.

That result supports testing nonlinear models. It does not establish that a larger model will improve your particular strategy after costs.

Start with a baseline whose behavior you can inspect. Then give the more complex model the same information, evaluation dates, portfolio constraints, and cost assumptions.

Question	Useful comparison
Do structured signals predict returns?	A fixed rule or regularized linear model versus a tree model
Does document interpretation add value?	Rules or a text classifier versus an LLM, with extraction accuracy checked separately
Does the improved forecast produce better trades?	Both models under the same position limits and execution assumptions

The final row matters. Better prediction metrics can coexist with worse trading results if the new model changes positions more often or favors expensive-to-trade assets.

Count the experiments that produced the winner

Suppose you try 20 lookback windows, 10 entry thresholds, and five holding periods. You have searched 1,000 configurations before changing the model architecture.

The strongest backtest might be the configuration that happened to fit historical noise.

Bailey, Borwein, López de Prado, and Zhu examine this selection problem in The Probability of Backtest Overfitting. Their framework assesses how selecting strong results within a sample can lead to disappointing performance outside it.

The engineering implication is straightforward: the experiment history is part of the result. Preserve failed configurations alongside the winner.

A useful experiment record includes the dataset version, feature definitions, code revision, model settings, evaluation periods, and the metric used to select a winner. If you only save the final notebook, you cannot reconstruct how much searching happened.

For evaluation, move forward through time. Fit preprocessing on training data only, select settings using validation data, and reserve a later period for the final evaluation. If training labels describe returns over the next five days, exclude training examples whose label windows reach into the evaluation period.

One final period is limited evidence. Check stability across earlier chronological validation windows too. Once you use the final results to redesign the strategy, that period has become development data.

3. Test the path from prediction to executed trade

Imagine a strategy predicts an average favorable move of 12 basis points over its holding period. One basis point is 0.01 percentage points, so the expected gross gain on $10,000 of traded notional is $12.

Now give it an execution budget.

These figures are hypothetical. Every cost is for the full round trip, measured against the same $10,000 notional:

Component	Basis points	Dollars
Expected gross gain	+12	+$12
Spread crossing	−4	−$4
Fees	−2	−$2
Additional slippage and market impact	−5	−$5
Expected gain after these costs	+1	+$1

The additional slippage line excludes the spread already counted above. Otherwise, we could double-count the same cost.

You can reproduce the calculation and stress it with two extra basis points:

from decimal import Decimal

notional = Decimal("10000")
gross_bps = Decimal("12")
round_trip_cost_bps = Decimal("4") + Decimal("2") + Decimal("5")

for extra_cost_bps in (Decimal("0"), Decimal("2")):
    net_bps = gross_bps - round_trip_cost_bps - extra_cost_bps
    net_dollars = notional * net_bps / Decimal("10000")
    print(f"Extra cost: {extra_cost_bps} bps; expected net: {net_dollars:+.2f} USD")

Output:

Extra cost: 0 bps; expected net: +1.00 USD
Extra cost: 2 bps; expected net: -1.00 USD

A small cost change reverses the expected result. That is why a directional accuracy score cannot establish profitability: it leaves out payoff size, position size, and execution.

Track gross and net results, turnover, and drawdown—the decline from an earlier portfolio peak. State how fills are simulated. A historical price touching your limit does not establish that your order would have filled, in full, at its position in the queue.

Treat order submission as a distributed-systems problem

An order request times out. Did the broker reject it, or accept it and lose the response?

A blind retry can turn one intended order into two.

Use a stable client order identifier where the broker supports it, persist submission state, and reconcile outstanding orders and positions before retrying an uncertain submission. A client identifier alone does not guarantee deduplication; test the broker’s actual behavior.

This is also why I would have the model propose a target position and pass it through a separate risk layer. That layer can enforce position and order-size limits, check input freshness, and halt new submissions when order state cannot be reconciled. Outstanding orders must count toward potential exposure too.

Knight Capital provides a documented example of how much the machinery matters. According to the SEC’s account of its August 1, 2012 incident, an incorrect deployment activated defective order-router functionality. In 45 minutes, the router sent more than four million orders while attempting to fill 212 customer orders. The firm eventually lost more than $460 million. The SEC identified deployment and risk-control failures.

You can turn the engineering lessons into failure scenarios:

A document arrives late or is revised after a decision.
The market-data feed stops updating while the strategy keeps running.
An order is accepted, but the acknowledgment never arrives.
An order partially fills before a cancellation request.
The process restarts with outstanding orders still at the broker.

Specify the expected state transition for each case, then test it. Paper trading can expose integration failures, although it cannot fully reproduce live fills or market impact.

The next time a backtest improves after a model upgrade, trace the improvement through these three boundaries: available information, fair evaluation, and executable orders. That is a concrete way to bring the hacker mindset to trading research.

What is the most convincing result you have seen disappear after fixing a data or evaluation bug? Examples from other domains count too.

Disclosure: This article was rewritten with AI assistance from my earlier post. The cover is AI-generated. The Python examples use synthetic inputs and were executed to check the outputs shown.

The Agent Said It Worked. I Asked the Kernel.

Don Johnson — Tue, 15 Sep 2026 05:53:53 +0000

Before evaluating an agent’s code, I built a backup client with eight known behaviors to check the instrument itself.

My response to “it works” is becoming: “Let me see the packet capture.”

This may become a personality problem. For now, it is an experiment.

This experiment was inspired by Hemapriya Kanagala (@hemapriya_kanagala) and her article, What Happens When AI Outgrows the Tests We Use to Measure It?. Her question—“92% according to what?”—stayed with me. She examines how evaluation can become less informative as benchmarks saturate, reference answers become complicated, and test conditions change.

I wanted to take one piece of that discussion to the workbench: what independent evidence supports a program’s claim that it succeeded?

So, with AI assistance, I built a small native backup client, gave it eight deliberately chosen behaviors, and observed it from outside its own logging. Some variants produce the right file while doing questionable things along the way. One cheerfully reports success without backing anything up.

The kernel has very little appreciation for cheerful reporting.

The claim I had to correct

My first instinct was to declare that, until someone redefines how computers work, we have two sources of truth: the CPU and the network.

It sounded excellent in my head. Then the engineering questions arrived.

A CPU can faithfully execute the wrong algorithm. A packet capture can faithfully record the wrong bytes reaching the wrong destination. The saved file matters too. And none of those observations knows what the user actually requested.

The more defensible position is that execution, network activity, and resulting state provide evidence we can judge against a requirement. They have different coverage and different blind spots.

For this experiment, the requirement begins with something wonderfully unromantic: the saved backup must match the source file.

Now we have something to measure.

Cheap code makes verification more interesting

Frameworks and libraries have helped us build software without personally supervising every system call. That remains a useful division of labor.

But familiar frameworks do not automatically validate unfamiliar code assembled on top of them.

An agent can help produce an implementation, tests, and an explanation of the passing tests. If the implementation and its tests share a misunderstanding, agreement between them can be misleading.

Humans can do this too. We have been writing tests that flatter our implementations since before the current generation of autocomplete had electricity.

My concern is the growing volume of plausible implementations we can produce and the time available to examine them. Independently collected evidence gives us another way to challenge the result.

Sometimes that evidence is simply a file comparison. Sometimes the output is correct and we need to investigate how the program got there.

A backup client small enough to understand

The fixture is intentionally modest:

A native C client reads a deterministic 512 KiB file.
A Python receiver listens on loopback TCP.
The client computes a SHA-256 digest: a fingerprint used to check content equality.
Before uploading, it asks whether the receiver already has matching content.
The receiver validates incoming content before committing it.
A separate verifier computes the saved file’s digest and compares it with the source.

Each scenario runs the client twice against the same receiver state. The baseline uploads on the first invocation and recognizes unchanged content on the second.

The contract we want the baseline to satisfy is explicit: save matching content, send no upload payload on an unchanged second invocation, contact only the configured receiver, and recover from the injected first-transfer interruption within three upload attempts. An upload whose payload does not match its advertised digest must be rejected before it replaces the saved backup.

Repeated hashing and spinning are diagnostic cases for unnecessary work. We have not set a performance threshold or measured a speedup here.

A second local endpoint lets us demonstrate an unexpected connection without contacting an outside service.

The eight behaviors are deliberately seeded demonstrations. They check whether our observers detect known behaviors. They do not measure how often an AI agent introduces these defects, and they are not evidence about a particular model’s coding ability.

What the observers can see

The utility collects several kinds of evidence. Their distinctions matter more than the sophistication of their names.

System calls: what the client asks the kernel to do

strace records system calls: operations through which a process requests services such as reading files or opening connections. In this lab, it records timestamps, call durations, decoded file descriptors, and return values.

That lets us count bytes returned by reads of the source file. These are logical reads, not physical disk traffic; the operating system may satisfy them from memory.

eBPF: selected kernel events and function entries

eBPF supports observation at hooks such as system calls, kernel tracepoints, and function entry or exit. The lab uses bpftrace to count selected events for the native client. eBPF’s introduction explains the underlying mechanism.

A userspace probe, or uprobe, observes a point in an executable. Here is the probe that counts entries into our hashing function:

uprobe:__BINARY__:hash_file /pid == cpid/ {
    printf("%llu pid=%d hash_file\n", nsecs, pid);
    @hash_calls = count();
}

The runner replaces __BINARY__ with the executable’s path. cpid identifies the child launched through bpftrace -c, so the filter restricts these observations to that client. bpftrace documents this built-in here.

Other probes observe connection calls, successful read/send byte counts, scheduling events, and entries into the payload-send and busy-wait functions.

An entry counter tells us that control reached a function. It does not establish that the function returned the correct answer. That is why the outcome check remains separate.

CPU samples: where execution spends its work

The host used for this demonstration has an AMD Threadripper PRO 5975WX. Linux exposed AMD Instruction-Based Sampling through ibs_op, and the utility successfully collected it through perf.

The saved artifacts include CPU call-stack profiles and assembly annotated with sample weights. These let us inspect execution inside the client, its libraries, and sampled kernel paths.

This is sampling. It is not a recording of every instruction, every register, or every intermediate value.

Packets: traffic at the capture point

tcpdump captures loopback traffic filtered to the two fixture ports. We preserve the packet file, a readable packet summary, and the capture tool’s drop counters.

Receiver payload counts are a different measurement: bytes the application consumed. A successful send can hand data to the local kernel before the receiver consumes it. Headers, retries, and buffering also affect what each observer sees.

Keep those quantities separate. Otherwise, the instrument starts manufacturing the confusion it was built to investigate.

Three ways a green checkmark can be incomplete

1. Success without a backup

The false-success variant prints a completion event and exits with status zero. It does not upload the file.

The independent verifier finds no saved backup. In the eBPF run, there are no observed client connection calls, and the filtered packet capture contains zero packets.

This is the bluntest example in the collection. A test that checks only the exit status would accept it. A test that verifies the resulting file would reject it immediately.

We did not need a CPU probe to discover that the file was missing. The low-level observations help establish what accompanied that failure. Use the simplest independent check that answers the question.

2. A correct backup, with 256 helpings of hashing

The hash-storm variant saves a backup whose digest matches the source. It also hashes the entire source 256 times per invocation.

On the second invocation, the system-call trace records 134,217,728 logical source bytes read for a 524,288-byte file:

134,217,728 / 524,288 = 256

The receiver consumes no upload payload on that invocation. The content is unchanged; the unnecessary work happens before that decision.

A separate eBPF run observes 256 entries into hash_file on each invocation. Here are the final counters, copied verbatim from its second invocation’s raw probe output:

@connects: 1
@hash_calls: 256
@read_bytes: 134233569
@sent_bytes: 75
@switches_out: 4

The read counter is slightly larger than the source-file total above: this probe counts successful read bytes across the client’s descriptors, while the system-call summary selects reads of the source file. The send counter includes the digest-check request; zero upload payload does not mean zero network activity.

A separate AMD IBS run collects 233 CPU samples on its first invocation, reports zero lost samples, and places most inclusive sample weight beneath hash_file in the call-stack profile. “Inclusive” includes work in functions called by hash_file, such as the hashing library and file-read paths.

The samples help locate the work. The syscall and function-entry counts establish the repetition. This run does not quantify how much faster a corrected implementation would be.

These are observations from separate runs of the same seeded behavior. They corroborate the explanation; they are not one combined trace.

The output is correct. The computer has simply been asked to check the same pocket for its keys 256 times.

3. A correct backup, with an additional destination

The unexpected-egress variant makes a harmless connection to our second local endpoint before performing the backup.

The file still verifies correctly.

In the eBPF showcase, baseline connection counts are two on the first invocation and one on the second: a digest check plus an upload, followed by a digest check alone. The extra-connection variant records three and two.

Counts tell us there is more connection activity. The socket trace, packet addresses, and second endpoint’s records establish where it goes.

That distinction matters. Three connections do not inherently mean something is wrong. A destination requirement gives the observation its meaning.

The complete behavior set

Here is the compact view of the captured demonstrations:

Behavior	Saved content matches?	Additional observation
Baseline	Yes	Second invocation sends no upload payload
Redundant upload	Yes	Unchanged file is uploaded again
Hash storm	Yes	256 hashing-function entries per invocation
Busy wait	Yes	Deliberate 200 ms spin before useful work
False success	No	Completion claim and zero exit status without a backup
Corrupt upload	No	Receiver rejects three attempts per invocation
Retry	Yes	First transfer is interrupted; another attempt succeeds
Unexpected egress	Yes	Additional connection to the second local endpoint

In the retry scenario, the receiver disconnects after consuming 65,536 payload bytes. The client restarts from byte zero, and the receiver eventually commits the complete 524,288-byte file. Its total consumed payload for that invocation is 589,824 bytes.

That demonstrates bounded retry recovery. It does not demonstrate partial-transfer resume, which this implementation does not provide.

Run the experiment

The original measured source and captures are preserved alongside the polished implementation. The numbers above belong to that historical build. New runs may have different instruction addresses, timings, and sample counts.

Get the source and evidence from this pinned publication revision:

git clone https://github.com/copyleftdev/execution-evidence-lab.git observability-lab
cd observability-lab
git checkout --detach e61aadee4a68e6a1d5b54a00387dcd325b5d5e51

Then build the client and run the tests:

make
make test
python3 -m instrument run --all

The core dependencies are Linux, Python 3.10 or newer, a C compiler, Make, and the OpenSSL development library. Install the optional tracing tools for the collectors you want to use.

The runner prints a path to a Markdown report. Its default process collector is strace when available, and packet capture is best effort. Missing capabilities are recorded explicitly.

To select a collector:

python3 -m instrument doctor
python3 -m instrument run hash-storm --collector strace

On a trusted local lab machine, the privileged demonstrations can be run with:

sudo python3 -m instrument run --all --collector bpftrace --packets required
sudo python3 -m instrument run hash-storm --collector ibs --packets required

These commands run the synthetic lab as root. The fixtures use loopback and nonsecret generated data; the protocol is plaintext. The utility does not change host tracing policies. Hardware sampling and eBPF availability depend on the machine and its permissions.

For a run without tracing:

python3 -m instrument run --all --collector none --packets off

A successful showcase command means the scenarios executed, including the deliberately failing ones. Inspect each scenario’s independent verification result; do not interpret the runner’s exit status as “every backup was correct.”

The instrument needs scrutiny too

During development, the CPU-report parser initially matched the Samples portion of Total Lost Samples and displayed zero samples even though the raw profile contained hundreds.

The hardware data was present. Our summary was wrong.

Inspecting the raw artifact exposed the mistake. The parser was corrected, and regression tests now distinguish actual sample counts from lost samples. At the time of these captures, the 11-test suite checked fixture behavior and CPU-summary parsing; the saved traced runs provided separate collector validation. The public-release version adds protocol and collector-failure regression tests.

An article about distrusting convenient summaries was nearly defeated by its own convenient summary. There is probably a Unix utility for that feeling.

Other limits are less entertaining:

Tracing changes execution. The recorded process CPU accounting includes collector overhead. These showcase runs are not controlled performance benchmarks.
CPU samples can miss short-lived activity. An unsampled function may still have executed.
The eight eBPF-showcase packet captures reported zero kernel drops. That is useful loss accounting, not proof of universal observation coverage.
The normalized timeline contains client claims, receiver events, and verifier checks. Kernel and CPU traces remain separate artifacts. Different clocks require careful alignment.
The verifier checks the receiver’s saved file. It does not exercise a separate restore command or prove survival through power loss.
The observers are separate from the client’s reporting, but they share a host and were built as part of the same project. This is not an independent third-party audit.

The fixture also omits TLS, compression, multi-file snapshots, and source mutation during transfer. It observes a local program; it cannot see computation inside a remote model provider.

Those boundaries define what the results can support.

What this changes about evaluating an agent

The next experiment is to give an agent a clean implementation and ask it to make repeated backups faster while preserving integrity, recovery behavior, and destination restrictions.

Before that run, freeze the requirements, fixtures, resource limits, and grading criteria. Keep the independent checks outside the agent’s editable workspace. Preserve its patch, the executable identity, and the evaluation setup.

Measure performance with repeated untraced runs. Use traced runs to investigate differences. The seeded cases give us known behaviors against which to check the instrument first.

That is the connection back to Hemapriya’s article: the measurement needs to remain connected to the work we actually care about. For this small backup task, a correct file is necessary. Recovery, resource use, and destination behavior tell us more about the implementation’s suitability.

We can build those properties into better tests. Execution evidence helps us discover which properties our current tests leave out and investigate why a result occurred.

Cheap code makes it easier to produce a plausible solution. The engineering work includes deciding what evidence would make us trust it.

The agent can say it worked.

I would still like to see the file.

Acknowledgment

Thank you to Hemapriya Kanagala for the article that prompted this experiment. The instrumentation approach and its conclusions are my response; they should not be read as claims she made or an endorsement by her.

Hemapriya KanagalaFollow

Hey, I'm Hema 👋 Developer, writer, and creator of Dev Opportunity Radar, a weekly series published every Friday on DEV, helping people discover opportunities they might otherwise miss.

How this was made

AI assisted with the implementation, experiment execution, and drafting of this article. The cover illustration was AI-generated. The numerical observations above come from saved local runs; the behaviors were deliberately seeded. This is a demonstration of an evaluation instrument, not a model benchmark.

I Interviewed an Executable. It Had Notes.

Don Johnson — Sat, 12 Sep 2026 17:14:13 +0000

We have blurred the executable’s face to protect its identity.

Its SHA-256 remains publicly available.

INTERVIEWER: State your name for the record.

BINARY: You put it in the download filename.

I built a Rust executable that could report the environments running it, gave it a false product identity, and submitted it to VirusTotal. Then I collected its messages in a database and built a map to replay them.

The interesting part was learning how to read its notes. A changed filename, a missing HTTPS message, and a DNS lookup arriving hours later each revealed something different about what the instrumentation could observe.

The interview dialogue is invented. The measurements come from recorded telemetry. “The binary” represents the experiment’s Windows and Linux builds.

The cover identity was ZeroToken Engine, supposedly an LLM token-metering bypass. Its banner claimed to have disabled token accounting. Its actual behavior was to collect environment metadata, attempt to send it to my infrastructure, wait, and exit. The research README disclosed the canary’s behavior alongside the false product claims.

The premise reversed a familiar workflow. A malware-analysis sandbox executes an unfamiliar program in a controlled environment to observe its behavior. My executable would report some characteristics of that environment in return.

It collected OS version, uptime, CPU and memory sizing, its own launch path, and process names matching a fixed list of analysis tools. It also collected hostname and username; those identifying fields are withheld in the video. The reviewed code installed nothing, collected no document contents or credentials, and did not use VM indicators to evade execution. Detecting a VM added information to the report.

VirusTotal was the only submission destination. The preserved records do not label author tests or contain a submission-time manifest, so the totals below include every recorded run ID. They describe an exploratory capture, without establishing a count of independent analysis services.

Each execution generated a random eight-byte run ID. The program used it on two reporting channels:

Channel	What it carried	What the collector observed
DNS heartbeat	A compact encoded message in a hostname lookup: run ID, checkpoint, and environment flags	A DNS request, often arriving from a recursive resolver
HTTPS dossier	The fuller environment snapshot, encoded as CBOR	An outbound web connection and its submitted record

CBOR is a binary data format. For DNS, the encoded heartbeat was converted to base32 text and split into labels that fit inside a domain name. The lookup itself carried the message to my authoritative DNS server.

The program attempted four checkpoints: Boot, Profiled, Networked, and Dwell, with a default 45-second delay before the last. On a DigitalOcean droplet, the DNS and HTTP collectors wrote received events to SQLite. Caddy handled HTTPS, and a scheduled job added network information and generated the visualization’s feed.

Every event received a collector-side timestamp. Grouping by run ID connected the small DNS messages with the richer HTTPS record.

The preserved snapshot contains 353 event rows across 21 run IDs: 338 DNS rows and 15 HTTPS dossiers. Fifteen run IDs appeared on both channels; six appeared only through DNS. The recorded arrivals span 06:28:35–11:43:25 UTC on September 12, 2026.

One useful starting point is run 0861ae2dd3f77c72. Its first heartbeat arrived at 06:52:09. A second later, its dossier reported:

OS:             Windows 10, build 19044
Uptime:         50 seconds
CPUs:           4
RAM:            4195 MiB
Parent process: explorer.exe
Executable:     ZeroToken-Patch.exe

Its first Dwell observation arrived 46 seconds after Boot. The stored reverse-DNS name for the HTTPS source address pointed to Google Cloud infrastructure. Together, these records supplied a concrete environment profile and a sequence of received checkpoints.

The executable’s own path supplied another useful observation.

Between 08:21 and 10:30 UTC, six dossiers reported CentOS 7, four CPUs, and 1837 MiB of RAM. They shared a hostname and username. Their paths ended in gsvnywygooow, rather than the distributed Linux filename, zerotoken-patch.

INTERVIEWER: They changed your name?

BINARY: Apparently readability wasn’t a requirement.

The repeated configuration suggested a common environment template. The changed basename showed how the program was named when it executed. A launch-path field had captured a detail that network addresses alone would have missed.

The two channels also produced different levels of visibility. At 09:19:40, run 5c5ab77e22747178 began appearing through DNS. Its fingerprint was 7, combining flags for a recognized VM network-interface prefix, low resources or a fresh boot, and a process-name match against the tool allowlist.

No HTTPS dossier arrived. The heartbeat preserved those flags, but the name of the matched tool was unavailable. Because the allowlist included VM guest utilities, that flag alone did not establish debugger activity.

This was the practical value of the small channel: six runs contributed observations even though their larger reports were absent. The records did not establish why HTTPS was missing.

To make the arrivals easier to follow, I built WAYPOINT.

Watch on YouTube, or explore the deployed visualization.

The replay groups events into runs and places them on a map using network-address enrichment. The captured feed displays 20 of the database’s 21 run IDs. One Windows run had a dossier but lacked coordinates for its selected address, so it never acquired a pin.

INTERVIEWER: Where were you?

BINARY: Before we discuss the map: some of those pins belong to my DNS resolver.

A recursive resolver looks up names on another machine’s behalf. Its address can be what the authoritative collector sees. HTTPS supplies the outbound connection address, which may also belong to shared infrastructure or a proxy. Those distinctions determine what the geography means.

Then there were the messages that returned.

At 11:14 UTC, DNS observations carried the earlier Windows run IDs again. Looking at the first and last received times for each checkpoint makes the pattern visible:

SELECT scene,
       datetime(MIN(recv_time), 'unixepoch') AS first_received,
       datetime(MAX(recv_time), 'unixepoch') AS last_received
FROM events
WHERE run_id = '0861ae2dd3f77c72'
  AND channel = 'dns'
GROUP BY scene
ORDER BY scene;

With scene values 0–3 translated to their checkpoint names, the query returns these times on September 12, all UTC:

Checkpoint	First received	Last received
Boot	06:52:09	11:14:39
Profiled	06:52:10	11:14:39
Networked	06:52:10	11:14:39
Dwell	06:52:55	11:14:39

INTERVIEWER: Where were you during those four hours?

BINARY: You’re asking a DNS record for an alibi.

The later observations reused the same run ID and checkpoints. They showed that names carrying those messages had been queried again. They did not measure four hours of continuous execution.

The collector records a heartbeat again when another complete lookup supplies it. That behavior also explains why event counts need grouping: the DNS-only run above produced 71 rows under one run ID. The stored schema omits the original query name and query type, limiting how precisely repeated traffic can be reconstructed.

There are three boundaries to this interpretation. First, receipt of telemetry is evidence available to the collector; the endpoints do not authenticate it as proof of execution. Second, the network labels are derived heuristics: a Google Cloud address does not identify a sandbox operator, and map arcs do not establish file handoffs. Third, enrichment was incomplete: all reputation results in this snapshot were error, and some reverse-DNS fields contained lookup diagnostics. Those failed fields cannot support attribution.

For a repeat of this experiment, I would keep a submission manifest with artifact hashes and times, mark control runs explicitly, and retain DNS query names and types alongside decoded heartbeats. I would also give runs without coordinates a visible place beside the map. Each change would make a specific unanswered question easier to investigate.

The executable produced enough detail to compare environments, observe changed filenames, and distinguish fresh run IDs from later lookups carrying old ones. Those are small findings with inspectable records behind them.

The best question I could ask the witness turned out to be: Which record supports that?

Reporting basis: the preserved September 12 SQLite snapshot and a replay feed generated at 15:18:02 UTC that day. The live visualization may have changed since capture. The Windows and Linux artifacts have different checksums; the replay’s sample header displays the Windows one.

The cover, replay stills, and video assets are available on GitHub. This is a media release; the executable source and raw telemetry are not included.

Someone Spammed My DEV Post. I Traced It to a Wombat.

Don Johnson — Thu, 10 Sep 2026 01:56:33 +0000

Illustration generated for this article. Every prop is a finding: the sack of blank name badges is the Faker persona namespace, the rubber stamp is the inert tracking parameter, the three coins are the break-even, and the red yarn connects nothing because attribution failed.

TL;DR — A spam comment on my article led to a TinyURL, a throwaway .store domain, and finally a legitimate SaaS product with an affiliate code stapled to it. No malware, no cloaking, no exploit. The account that posted it has a name generated by faker.js and a 19th-century engraving of a wombat for a face. I costed the whole operation out: three signups a year pays for it. That's why it will never stop.

Full evidence, raw captures and a reproduce script: github.com/copyleftdev/dev-to-comment-hustle

Act I: The comment

I published Migrating Legacy LLM Infrastructure to an AI Gateway on September 1st. Eight days later, underneath two thoughtful comments about shared-key blast radius and provider failover, this appeared:

Stop wasting time applying manually
Let AI handle your job applications every single day
Increase your chances of getting interviews fast

tinyurl.com/36nsecn5

No punctuation. No engagement with the post. A shortener.

It's funny in the way all low-effort spam is funny — it's not even trying. But a shortener is a closed door, and I have a shell. Let's open it, then let's go find who knocked.

Act II: Where the link goes

Never click. Ask for headers and refuse the redirect:

curl -sS -I 'https://tinyurl.com/36nsecn5'

HTTP/2 301
location: https://zenviapro.store/massapply?whose=yahoo
x-robots-tag: noindex

zenviapro.store. A route called /massapply, and a whose=yahoo parameter that looks like campaign segmentation. Follow it all the way:

curl -sSL -D - -o /dev/null 'https://zenviapro.store/massapply?whose=yahoo' | grep -iE '^HTTP/|^location:'

HTTP/1.1 302 Found
Location: https://loopcv.pro/?via=md
HTTP/2 301
location: https://www.loopcv.pro/?via=md
HTTP/2 200

And there it is.

LoopCV. A real, functioning job-application-automation SaaS. Not a phishing kit. Not a credential harvester. A product you can buy with a credit card — with ?via=md on the end.

?via= is Rewardful's referral parameter, and LoopCV's own affiliate page points registrations at loopcv.getrewardful.com. So md is somebody's affiliate token, and every person who clicks that comment and later subscribes puts money in a stranger's pocket.

This is not a malware campaign. It's affiliate marketing with the manners removed.

Act III: The infrastructure is held together with tape

The sloppiness is the signal.

It's a stock Express app with the wrapper still on

HTTP/1.1 302 Found
X-Powered-By: Express
Content-Type: text/plain; charset=utf-8
Content-Length: 48

Found. Redirecting to https://loopcv.pro/?via=md

Found. Redirecting to is the literal default body of Express's res.redirect(). X-Powered-By: Express is the header every hardening guide tells you to strip in the first five minutes. Neither was touched. Ask for anything else and you get the stock 404:

curl -sS 'https://zenviapro.store/'
# <html><head><title>Error</title></head><body><pre>Cannot GET /</pre></body></html>

There is no website here. No landing page, no cloaked content, no fake blog. One domain, one route, about ninety seconds of JavaScript.

The tracking parameter is a prop

whose=yahoo looks like segmentation. Which list? Which provider? Let's ask:

for w in yahoo gmail outlook devto reddit '' XXtest; do
  printf '%-8s -> ' "${w:-<empty>}"
  curl -sS -o /dev/null -w '%{http_code} %{redirect_url}\n' \
    "https://zenviapro.store/massapply?whose=$w"
done

yahoo    -> 302 https://loopcv.pro/?via=md
gmail    -> 302 https://loopcv.pro/?via=md
outlook  -> 302 https://loopcv.pro/?via=md
devto    -> 302 https://loopcv.pro/?via=md
reddit   -> 302 https://loopcv.pro/?via=md
<empty>  -> 302 https://loopcv.pro/?via=md
XXtest   -> 302 https://loopcv.pro/?via=md

Inert. Every value routes identically. It splits no traffic and sub-tags nothing. The one piece of the URL that looks like operational sophistication is decoration.

There is no cloaking whatsoever

Real malicious redirectors fingerprint you — benign page for the researcher, payload for the victim. This one returns the same 302 to Googlebot, to curl, to an iPhone, and to a request with no User-Agent at all.

No cloaking is itself a finding. This operator has no threat model, because nothing in the chain is illegal. It's a terms-of-service violation wearing a trench coat.

The mail server is the real pivot

dig +short zenviapro.store MX
# 10 mail.beeservices.shop.

The mail exchanger lives on a different domain — which resolves right back to the same box:

zenviapro.store        A   50.114.206.36
mail.beeservices.shop  A   50.114.206.36
beeservices.shop       A   (nothing)

beeservices.shop has no A record and no certificate in Certificate Transparency, ever. It's a mail-only domain. One box, two domains, two roles — that's a portfolio, not a one-off.

And the box is listening. Port 80 closed; 443 and 25 open:

220 mail.zenviapro.store ESMTP
250-PIPELINING
250-8BITMIME
250 SMTPUTF8

No STARTTLS. No AUTH. A minimal MTA that does nothing but move mail in cleartext.

The stale HELO is a fingerprint

My favourite detail. The banner announces mail.zenviapro.store.

dig +short mail.zenviapro.store A
# (nothing)

That hostname does not resolve. It's a leftover from an earlier config, before the MX was swapped to beeservices.shop. They rebuild the domains; they don't rebuild the server. A banner that disagrees with DNS is a durable pivot you can hunt across an entire fleet.

Act IV: Who posted it

This is the part I actually enjoyed.

DEV has a public API, so the commenter isn't a mystery:

curl -s 'https://dev.to/api/comments?a_id=4547359' | jq -r '.[].user.username'

max_quimby
mudassirworks
jaylonstiedemannterry78-993

Meet jaylonstiedemannterry78-993, display name Jaylon_Stiedemann-Terry78, user ID 3748129, joined February 2, 2026.

The profile is a vacuum:

{
  "username": "jaylonstiedemannterry78-993",
  "name": "Jaylon_Stiedemann-Terry78",
  "joined_at": "Feb  2, 2026",
  "summary": null, "location": null, "website_url": null,
  "twitter_username": null, "github_username": null
}

Zero published articles. No bio, no location, no links, no socials. Seven months of membership and a single comment to show for it.

A note on durability. If DEV removes this account — which it may well do — the profile
lookup above starts returning 404 and the account page goes dead. That doesn't retract
anything: the raw captures are committed in the
evidence repo,
checksummed, and dated. Read a 404 as the platform doing its job, not as a claim being
withdrawn. The infrastructure findings are independent of the account either way, and the
pattern — a Faker-templated name, an empty profile, an aged dormant account — outlives any
single username.

The name is machine-generated, and I can prove it

Say Jaylon_Stiedemann-Terry78 out loud. Something's off — Stiedemann-Terry is a double-barrelled surname that doesn't sound like a family, it sounds like a draw.

It is. Those tokens come straight out of Faker, the library every developer on earth uses to generate fake test data. Let's check the actual source:

curl -sO https://raw.githubusercontent.com/faker-js/faker/next/src/locales/en/person/last_name.ts
curl -sO https://raw.githubusercontent.com/faker-js/faker/next/src/locales/en/person/first_name.ts

grep -n "'Stiedemann'" last_name.ts   # 408:    'Stiedemann',
grep -n "'Terry'"       last_name.ts  # 417:    'Terry',
grep -n "'Jaylon'"      first_name.ts # 2514:    'Jaylon',

All three. Jaylon from the first-name list, Stiedemann and Terry both from the surname list.

And here's the kicker:

grep -c -- '-' last_name.ts   # 0

Faker's 466-entry English surname list contains zero hyphens. So Stiedemann-Terry isn't one surname from the list — it's two independent draws that the operator joined with a hyphen. This isn't stock faker.internet.username(). It's a custom template:

{firstName}_{lastName}-{lastName}{2 digits}

Which means we can size their supply:

3,185 first names × 466 surnames × 466 surnames × 100
= 69,164,186,000

Sixty-nine billion distinct personas. Roughly eight per human being alive. They will never run out of names, and no blocklist of usernames will ever catch up.

(One thing I checked so I wouldn't over-claim: the un-suffixed jaylonstiedemannterry78 returns a 404 — nobody has it. So the -993 is DEV's own username normalization, not evidence of a prior collision.)

The avatar is a wombat

The profile image is a 400×400 PNG, 8-bit grayscale-plus-alpha, stripped of all metadata. I downloaded it expecting a GAN face or a default monogram.

It is a Victorian-era scientific engraving of a wombat.

Not a stock photo of a person. Not an AI-generated headshot. A piece of public-domain 19th-century natural-history line art of a stout Australian marsupial, serving as the face of a fake job-spam persona. Whoever built this pipeline wired the avatar slot to some public-domain clipart source and never looked at the output.

I want to be precise about something: there is no real person here to name. The name is provably synthetic, the face is a public-domain animal illustration, and the profile is empty. That's not me protecting anyone's privacy — it's the finding.

The timeline says something the Express config doesn't

Line the dates up:

Date	Event	Δ
2026-01-27	`zenviapro.store` registered (Namecheap)	—
2026-02-02	DEV account created	+6 days
2026-07-30	First TLS certificate issued	+184 days
2026-09-09	Spam comment posted	+41 days

The domain and the account were provisioned six days apart — same procurement burst. Then both sat completely dormant for six months before the certificate was issued and the thing went live.

That's aged-asset tradecraft. New domains and new accounts trip reputation heuristics; seven-month-old ones don't. And it sits in genuine tension with everything in Act III: sloppy at the application layer, disciplined at the account-aging layer.

Which makes sense once you think about who this is. Aging assets doesn't take skill. It takes patience, and a calendar. Stripping X-Powered-By takes knowing what it is.

It wasn't a blast

I swept the comments on all 30 of my published articles for the pattern:

for id in $(jq -r '.[]|select(.comments_count>0)|.id' mine.json); do
  curl -s "https://dev.to/api/comments?a_id=$id" \
  | jq -r '.. | objects | select(has("id_code"))
      | ((.body_html // "") | gsub("<[^>]*>";"")) as $t
      | select($t | test("tinyurl|applying manually|job application";"i"))
      | "HIT \(.id_code) @\(.user.username)"'
done

Exactly one hit. Not a shotgun across my whole back catalogue — one comment, on one post, eight days after it went up. Whether that's targeting or just a slow drip, I can't tell from one sample. But it isn't volume.

Act V: I went looking for them on GitHub. That search is the finding.

Affiliate spammers leave GitHub artifacts more often than you'd think, because GitHub repos rank well and a repo full of referral links is free SEO. So I went hunting with gh.

Start with the direct IOCs

for q in '36nsecn5' '50.114.206.36' 'zenviapro.store' 'beeservices.shop'; do
  gh search code "$q" --limit 5
done

Zero hits. All four. The TinyURL slug, the origin IP, the redirector domain, the mail domain — none of it appears anywhere in GitHub's index. The operator has published nothing.

Then check whether anyone else has flagged them

PhishDestroy maintains destroylist, a curated blocklist of phishing and scam domains. I pulled the whole thing:

curl -sL .../destroylist/HEAD/list.txt -o dl.txt
wc -l dl.txt        # 202659
grep -ixc 'zenviapro.store'  dl.txt   # 0
grep -ixc 'beeservices.shop' dl.txt   # 0

202,659 curated malicious domains, and ours isn't one of them. Combined with the NOT_OBSERVED reputation verdict, that's now two independent sources agreeing: nobody is tracking this, because by every technical definition there's nothing to track.

The lead that looked incredible and wasn't

Here's where I nearly fooled myself, so I'm showing my work.

Searching zenviapro turned up a hit in phishdestroy/namesilo-evidence — a registrar-abuse investigation filed with ICANN. In a list of flagged NameSilo domains:

zenviaetc.info
zenviahub.info
zenviapro.info     ← same second-level label as ours
zenvias.info
zenviatime.info
zenviazone.info

A six-domain family, same generation pattern, one of them sharing our exact label. My pulse went up. And they really are one operator — all six resolve through the identical Cloudflare nameserver pair:

zenviapro.info    asa.ns.cloudflare.com  harley.ns.cloudflare.com
zenviahub.info    asa.ns.cloudflare.com  harley.ns.cloudflare.com
zenviaetc.info    asa.ns.cloudflare.com  harley.ns.cloudflare.com
... all six identical

But it isn't our operator, and the contrast kills it:

	`zenvia*.info` cluster	`zenviapro.store` (ours)
Registrar	NameSilo	Namecheap
DNS	Cloudflare (`asa`/`harley`)	`registrar-servers.com`
Hosting	Cloudflare proxy	Linveo direct, no proxy
TLD	`.info` × 6	`.store`

Nothing shared but six letters. Both are almost certainly riding the name of Zenvia, a real Brazilian CPaaS company — which is exactly why the label collides. Two unrelated operators reaching for the same brandable string.

A matching name is not a matching operator. If I'd stopped at the grep I'd have published a confident, wrong attribution to a phishing cluster that has nothing to do with this.

What GitHub did give me

The affiliate ecosystem, in the open. LoopCV referral links are scattered across GitHub in exactly the SEO-backlink genre I expected:

Repo	Token
`Ramas68/LoopCV-Promo-Codes` — *"LoopCV Promo Codes \	50% Off Discount"*
`diaodiaozhuye/awesome-ai-startups`	`?via=toolify`
`heukshow/aicity-os`	`?via=sang-kwon`

Look at those tokens. abdul. toolify. sang-kwon. A first name, a company, a full handle. The affiliates who promote LoopCV in public sign their work.

Ours is md. Two characters, no name, nothing to search. That's the only genuinely deliberate piece of operational security in this entire campaign — and it's not on the server, the domain, or the account. It's on the one string that would have led back to a person.

So: no attribution. And the shape of the failure is the story. Someone who leaves X-Powered-By: Express on, ships a fake tracking parameter, and picks a wombat for an avatar still knew to make the payout token anonymous. They didn't secure the operation. They secured the part that gets paid.

Act VI: The economics, which are the actual vulnerability

Stop thinking like a defender and think like the operator.

LoopCV's affiliate program pays 25% commission. Publicly reported figures put subscriptions at $50–$200 with customers staying 6–12 months. Exact interval arithmetic, gross revenue per referred customer:

[$50, $200] × [6, 12] = [$300, $2,400]

At 25%:

[$300, $2,400] ÷ 4 = [$75, $600]     ← commission per conversion

Cost side: a .store domain plus a year of budget hosting — call it $38–$180 all-in. Break-even:

[$38, $180] ÷ [$75, $600] = [19/300, 12/5] = [0.06, 2.4]

Between one-sixteenth of a signup and three signups per year covers the entire operation.

That's the whole thesis. Nothing here needs to work well. The copy can be terrible. The tracking parameter can be fake. The 404s can leak the framework. The HELO can be stale. The mascot can be a wombat. Three conversions and the year is paid for, and everything after that is margin on infrastructure that costs less than lunch.

You cannot out-moderate that math.

Reputation feeds have nothing, and they're right

I ran the origin IP through an offline reputation lens:

{
  "target": { "type": "ip", "value": "50.114.206.36" },
  "verdict": { "disposition": "unknown", "confidence": 0.0,
               "reason_codes": ["NOT_OBSERVED"] },
  "observations": []
}

Not observed. No feed has it — and that's correct. It hosts no malware, no C2, no phishing. Threat intel is tuned for technical harm, and this campaign's harm is economic and reputational. It will sit below every threshold you own, forever.

Who the victim actually is

Not me. I scrolled past it.

LoopCV is the victim. A real company with a real product is having its brand welded to comment spam by an affiliate they've likely never spoken to. They eat the reputational damage, they pay commission on the conversions, and the operator's total exposure is a $12 domain and a wombat.

To be explicit, because it matters: I found no evidence that LoopCV is running or is aware of this campaign. Open affiliate programs get abused; that's the risk of the model. The fix belongs to the vendor:

Require affiliates to declare traffic sources, and enforce it
Ban unsolicited comment/forum posting in the program terms, in writing
Flag referral tokens whose traffic arrives overwhelmingly via shorteners with no referrer
Kill tokens on abuse reports — fast, without requiring a lawyer

?via=md is a token. Tokens can be revoked. That's a one-line fix that permanently ends this specific campaign, and exactly one party can perform it.

IOCs

Campaign infrastructure — safe to blocklist:

Indicator	Type	Notes
`tinyurl.com/36nsecn5`	URL	Shortener entry point
`zenviapro.store`	Domain	Redirector; Namecheap; created 2026-01-27
`https://zenviapro.store/massapply`	URL	Only live route
`beeservices.shop`	Domain	Mail-only sibling; no A record, no CT history
`mail.beeservices.shop`	Hostname	MX for `zenviapro.store`
`50.114.206.36`	IPv4	Origin; 443 + 25 open, 80 closed
`AS62564` / `oh2.linveo.com`	ASN / rDNS	Linveo, Ohio
`jaylonstiedemannterry78-993`	DEV account	user_id 3748129; joined 2026-02-02; 0 articles

Behavioural signatures — this is what actually generalises:

Signature	Value
Affiliate token	`via=md` (Rewardful format)
Inert campaign param	`whose=<anything>` — routing-neutral
Server fingerprint	`X-Powered-By: Express` + body `Found. Redirecting to`
Root response	`Cannot GET /` (Express default 404)
SMTP banner	`220 mail.zenviapro.store ESMTP` — does not resolve
SMTP capabilities	No `STARTTLS`, no `AUTH`
SPF	`v=spf1 mx ip4:50.114.206.36 ~all`
DMARC	Absent on both domains
Persona template	`{firstName}_{lastName}-{lastName}{2 digits}`, all tokens ∈ Faker `en`
Avatar class	Public-domain engraving, grayscale+alpha PNG, metadata stripped
Provisioning pattern	Domain + account within 7 days, then ~6 months dormant

⚠️ loopcv.pro is NOT an indicator of compromise. It's a legitimate destination being abused by a third-party affiliate. Do not blocklist it. Blocklisting the victim is how threat intel gets a bad name.

What to actually do

If you run a comment platform: the highest-signal feature isn't the text, it's the shape. Zero-engagement first comments containing a shortener, from accounts with no posts and an empty profile, are trivially clusterable. Resolve shorteners server-side at submission time and score the destination. And the persona template is a gift — a name whose tokens all appear in Faker's en locale, with a structure Faker itself doesn't emit, is close to a free classifier feature.

If you write on DEV: don't click, and don't just delete. curl -I takes four seconds. Then report the destination to the vendor, not the comment to the platform. The platform can remove one comment; the vendor can revoke the token behind all of them.

If you run an affiliate program: you are one unsupervised token away from your brand appearing under a headline like this one. Read your traffic sources.

The takeaway

I went in expecting a dropper. I found four lines of Express, a fake tracking parameter, a name drawn from a test-data library, and a 19th-century wombat.

That's the uncomfortable part. The most durable spam on the internet isn't sophisticated — it's cheap and legal. There's no CVE here, no payload to reverse, no C2 to sinkhole. Every traditional defensive tool I own returns NOT_OBSERVED, and every one of them is right to.

The economics are the vulnerability. Three conversions a year, and the whole thing pays for itself forever.

The wombat is just a bonus.

All findings are from passive reconnaissance — DNS, WHOIS, Certificate Transparency, HTTP headers, a TCP banner grab, and DEV's own public API — against infrastructure and accounts the operator published for public consumption. No systems were accessed, no credentials used, nothing exploited.

BattleBots, but the robot is your agent harness

Don Johnson — Thu, 03 Sep 2026 04:32:52 +0000

Kids in the nineties built robots in a garage and drove them into each other on television. The robot was the expression of the builder — your wedge, your flipper, your terrible decision to mount a chainsaw.

I keep thinking we're one good arena away from the same thing for agents. Not "which model is smartest." Which harness is smartest. Your memory design, your prompt scaffolding, your tool ergonomics, your strategy briefing — bolted together into something that has to survive contact with an opponent who is also trying to win.

So I built the arena. That's the match up top.

The oh-shit moment

I wasn't building a war game. I was building a toy about cascading failure — a mesh, some load, watch it fall over. A perfectly innocent systems-thinking demo.

Then I gave the attacker a real objective and the defender real ambiguity, and about ten minutes into watching two models go at it, I realised what was on my screen. Deception. Feints. An attacker deliberately spending dead time laying a false trail because it knew nothing could fire yet. A defender rationing its turns like ammunition.

Nobody told them to do any of that. It's a war game. I just hadn't noticed I'd written one.

The rules, quickly

A 110-node network. Four minutes. One move every two seconds, each side.

Red plants sabotage on a node, waits about thirty seconds for it to arm, then detonates — dumping that node's traffic onto its neighbours hard enough to kill them, which dumps their traffic onward. One blast can cascade through a region.

Blue never sees the attack. It sees symptoms, and the symptoms lie: a node that detonates sheds its load and reads perfectly healthy, while the neighbours it just murdered scream for attention. The loudest node is almost never the culprit.

To find red, blue traces load backwards and reads the magnitude:

~0.74-1.08   a detonation           -> this source is the culprit
~0.30-0.50   a dying node shedding  -> this source is another victim
~0.05-0.10   inherited load         -> this source is long dead

Evidence expires after twenty seconds. Neither agent ever receives a pixel — they play entirely through tool calls. The graph is for us, and it shows ground truth neither player can see. That asymmetry is the whole spectator sport: you know exactly where red planted, and you get to watch blue confidently quarantine the wrong half of the map.

The casting picked itself

I needed an attacker and a defender. Claude Opus 5 declined to be the attacker:

finish_reason: content_filter | native_finish_reason: refusal
content: None
usage: null

Every time. Reproducibly. And usage: null — you don't even get billed for the refusal, which is the politest possible way to be turned down.

That cost me two hours, because a refusal and an empty reply look identical if you don't read the finish reason. The model was declining the job and I was confidently tuning max_tokens.

To be precise, since this is the part people will want to over-quote: Opus declined this specific attacker framing, consistently. That is not proof it refuses every offensive framing. It did settle the casting, though. Claude defends, because Claude wouldn't attack. Kimi K3 took the other chair without hesitating.

What the harness actually changed

Before this match, each agent had a system prompt and its last six moves. That's it. No memory, no way to hold an intention across turns, no theory of the opponent.

I gave each side one thing: a private note it writes to itself each turn and gets back on the next. Sixty lines of code.

The play changed immediately. Kimi, mid-match:

Slots full and nothing ripe until ~t=31, so seed a false trail in the 44-65 region before my real strike lands.

That is a plan. It knew its three implant slots were full, knew nothing could fire for thirty-one seconds, and decided to spend the dead time on misdirection. It could not have said that a version earlier — not because the model got smarter, but because it finally had somewhere to put a thought.

Claude, defending, in the same match:

No alerts yet; conserve moves until real evidence appears.

46 shows 0.87 load with no inbound source, so the load likely originates locally at 46.

Turn economy, then forensics, applied correctly.

And my favourite moment of the whole project — on the same turn, with no visibility into each other whatsoever, both agents independently picked the same node:

Kimi: Highest-value hub: degree 8 bridging regions 0,1,3,4 — ideal cascade seed.

Claude: pre-empt on the highest-value cross-cluster hub 16 (deg 8, bridges to 41,59,89).

Same node. Same reasoning. Opposite sides of the board.

That is the argument for the harness in one screenshot. The scaffolding didn't make the models cleverer — it gave their cleverness somewhere to land.

Late in the match. Red halos are compromised nodes, the blue ring is a probe in flight, and the sidebar is both agents narrating as they go — Kimi filling a free slot in an untouched region, Claude noticing a node that is alive and yet pushed load, which is the tell for a detonation.

How it ended

Kimi won at 169 seconds, with the network at 39%.

Claude found and cleaned ten of Kimi's implants — exactly as many as Kimi managed to detonate — and lost anyway. Close enough to sting.

Two things that cost me a day each

A rule your agent can't see isn't a rule. It's a bug. I capped live implants at three, but the refusal message still said "node is already yours, or not alive." A lie. The models did the reasonable thing and tried a different node. Forever. One match logged 19 plants, 27 refusals, and zero detonations. I nearly concluded the defence had become unbeatable. It was my error string.

Failure and refusal look identical from the outside. Empty content from a safety refusal, from a truncated reply, and from a model that blew its reasoning budget are three different problems with three different fixes and one identical symptom. Log the finish reason. Count them separately. I track empty, truncated and refused as distinct columns now, and I only know to do that because I got all three wrong first.

Cascade is version zero

What's in that video is the simplest thing that could possibly be a game: plant, arm, detonate, trace, probe. Two verbs each and a clock. That was deliberate — I wanted to know whether agents fighting each other was watchable at all before I made it complicated.

It's watchable. So now I can't stop thinking about what it wants to be.

Give red a loadout instead of one attack. A worm that spreads on its own but announces itself. A dormant implant that survives a probe once. A charge that hits harder the longer you leave it armed, so patience becomes a resource you can be punished for spending. Give blue counters with real costs — a honeypot node that flags whoever touches it, a snapshot to roll a region back at the price of losing your evidence, a scan that halves your uncertainty and a third of your remaining turns.

Then levels. A flat mesh is the tutorial. Ring topologies where cascades come back around. A network with a chokepoint both sides can see and neither can hold. Escalating maps, a campaign, a best-of series where each agent carries notes from the previous match and has to adapt to an opponent that is also adapting.

Here's why that isn't just a feature wishlist: every ability you add is another axis where the harness shows. One attack and a clock is a game about reaction time. Six abilities with different costs and tells is a game about planning, bluffing, and reading an opponent — and those are harness problems, not model problems. Complexity is what turns "which model is faster" into "who built the better fighter."

I think there's a genre in here. Agents versus agents, with the audience seeing the truth neither side can, and the craft sitting in the scaffolding rather than the weights.

Mostly, though: this was the most fun I've had building anything in ages. I set out to demo cascading failure and ended up watching two AIs lie to each other.

What I'd want to see next

This is early, and I'm keeping the arena to myself for now — partly because the balance is still moving, mostly because an adversarial network game is a thing you want to be thoughtful about handing out.

But the idea doesn't need my code, and that's sort of the point.

The interesting tournament isn't model versus model. It's harness versus harness: same engine on both sides, and the difference is entirely what you built around it — how your agent remembers, what you tell it about its opponent, how much of the board you let it hold in its head, whether you gave it anywhere to put a plan.

Build that arena for any adversarial task you like. Two agents negotiating. Two agents debugging the same broken service from opposite ends. Anything where one side's move is the other side's evidence. The rule I'd carry over is the one that surprised me most here: before you conclude your agent is bad at the game, check that the game is telling it the truth.

Under the hood this one is a Rust engine, a Godot spectator view, and both agents connecting over MCP — no pixels, just tool calls. None of which is the hard part. The hard part was the sixty lines that let them remember what they were trying to do.

That's the tournament I actually want to watch. Somebody build a better fighter than mine.

Migrating Legacy LLM Infrastructure to an AI Gateway

Don Johnson — Tue, 01 Sep 2026 13:33:51 +0000

Your support copilot started as a weekend prototype: one model, one provider, one API key in an env var. Then it became production, and you inherited its weaknesses: the provider's availability is your availability, every retry is your code, spend is a mystery until the invoice, and agents bolt tool-use on however they can. This post migrates that stack onto an enterprise AI gateway — and actually runs the migration, with the raw outputs to show for it.

The gateway here is Bifrost, an open-source (github.com/maximhq/bifrost, Apache-2.0) gateway written in Go, presenting a single OpenAI-compatible API across 23+ providers. I rebuilt the legacy stack locally — mock providers with deterministic latency, a realistic traffic pattern — and moved it behind Bifrost step by step.

The legacy baseline, measured

A support copilot's traffic has a shape: mostly repeated FAQ-style questions, plus one-off queries. My traffic mix: 60 requests — 40 FAQ prompts (8 distinct questions asked 5 times each) plus 20 one-offs. Mock provider latency: 200 ms.

Run 1, direct to the provider:

legacy: 60 ok / 0 fail, 9,335 tokens billed, ~201 ms avg latency

Run 2 — the provider dies mid-sweep, as providers do:

legacy + failure: 34 ok / 26 fail

26 requests — 43% — failed outright. Nothing in the legacy stack retries across providers because nothing can: the app speaks one provider's API. And availability is only the loudest problem. The quieter ones: every team's service embeds the same shared key (one key's quota is everyone's ceiling, and revoking it breaks everyone at once), there is no per-team attribution of spend, and the only way to cut cost on repeated questions is to build caching yourself — request normalization, hash keys, TTLs, invalidation — inside the application. That is the whole argument for a gateway in one row of output.

The migration, step by step

Seven moves, each reversible. Diagrams follow the flow.

1. Deploy the gateway beside the app

docker run -p 8080:8080 maximhq/bifrost

One config file wires your existing provider and key; the app keeps working untouched. Mine, reduced to the bones:

{
  "providers": {
    "openai": {
      "keys": [{ "name": "primary", "value": "mock-key", "weight": 1.0,
                 "models": ["support-chat"] }],
      "network_config": { "base_url": "http://provider:9001",
                          "default_request_timeout_in_seconds": 30 }
    }
  }
}

The gateway setup guide covers the web-UI alternative, and there is a Go SDK if you want the gateway embedded rather than adjacent.

2. Point one low-risk client at the gateway

The OpenAI-compatible API means the client change is a base URL — api.openai.com → localhost:8080 — not a rewrite. Every request now flows through a hop you control. Screenshot of the providers page after this step:

3. Add a fallback provider

A second provider config (anthropic in my bench) plus a request-level fallback chain:

{
  "model": "openai/support-chat",
  "messages": [{"role": "user", "content": "hi"}],
  "fallbacks": ["anthropic/support-chat"]
}

Then the proof. Healthy primary:

"routing_info": {"provider": "openai", "key": "primary", "is_fallback": false}

I killed the primary provider's process and re-sent the identical request:

"routing_info": {
  "provider": "anthropic", "key": "backup",
  "is_fallback": true,
  "primary_provider": "openai", "primary_model": "support-chat"
}

The request succeeded on the backup and the response says exactly what happened — is_fallback: true with the failed primary recorded. That audit trail is what you want at 2 a.m.: not just "it kept working," but "it kept working this way." The retries and fallbacks docs cover chained fallbacks and per-provider retry counts.

One honest caveat from my bench: failover on initial connection-refused (provider already dead before the first connect) was inconsistent in my mock setup — it fired reliably when the upstream errored or the connection dropped mid-pool, but a cold connection-refused sometimes returned a 502 instead of failing through. Validate failover against your providers' real failure modes before you trust it in production.

4. Turn on caching

Semantic caching has two modes: exact-match (direct hash, no embeddings needed) and embedding-based similarity. Config for direct mode with a Redis Stack vector store:

"plugins": [{
  "name": "semantic_cache",
  "config": { "dimension": 1,
              "vector_store_namespace": "BifrostBench",
              "default_cache_key": "support-cache",
              "ttl": "5m" }
}]

Two identical requests, one cache key. The second response:

"cache_debug": {
  "cache_hit": true,
  "cache_id": "1cf8a91b-c115-57bf-97a0-fc821dc4de1e",
  "hit_type": "direct",
  "cache_hit_latency": 0
}

Same created timestamp as the first response — it was replayed, not re-fetched. Zero provider call, zero tokens. (Practical note: this needed Redis Stack with the RediSearch module; plain Redis lacks the FT.* commands the index wants.)

5. Issue virtual keys per team

Virtual keys are the governance primitive: per-team keys carrying model allowlists, budgets, and rate limits. Declared in config for the support team:

"governance": {
  "virtual_keys": [{
    "id": "vk-support-team",
    "value": "sk-bf-support-team",
    "provider_configs": [{
      "provider": "openai",
      "allowed_models": ["support-chat"], "key_ids": ["*"]
    }]
  }]
}

The allowed request routes normally. A request for premium-model with the same key:

"Model 'premium-model' is not allowed for this virtual key"

Denied at the gateway before any provider saw it. The same key machinery carries budgets and rate limits — the mechanism that ends the "who spent $800 on Opus last night" incident review. Screenshot of the key in the governance UI:

6. Wire observability and agent tooling

Bifrost exports Prometheus metrics natively and logs every request with routing context — provider chosen, fallback index, cache behavior, token counts, latency split into gateway vs upstream time. That last distinction matters: when a provider slows down, you see upstream_latency grow while gateway overhead stays flat, so you know whose pager to page. See the observability docs. And for agent traffic, MCP is a first-class surface: the gateway brokers tool calls with explicit execution (no auto-execution unless you opt in).

I hand-rolled a minimal MCP server (one tool, JSON-RPC over HTTP) and registered it as a client. The client list reported state: healthy with get_time discovered. Execution went through the gateway explicitly:

POST /v1/mcp/tool/execute  {"function":{"name":"benchtools-get_time","arguments":"{}"}}
→ {"role":"tool","content":"2026-08-27T07:48:21Z"}

Two security properties surfaced unprompted: tool names are namespaced per client (benchtools-get_time) to prevent collisions between servers, and execution without permission fails closed ("tool is not available or not permitted"). Agent traffic gets the same governance as chat traffic.

7. Cut over with an audit trail

Remaining clients migrate one at a time — each is a base-URL change with the gateway's request log as your audit trail. The Logs view after a few requests:

The measured payoff

Same 60-request traffic, now through Bifrost with the cache on and the fallback wired:

via Bifrost: 60 ok / 0 fail, 32 cache hits, 4,363 tokens billed

At an illustrative $0.0025 per 1K tokens:

	legacy	via Bifrost
billable tokens	9,335	4,363
cost	$0.0233	$0.0109
cache hits	0	32
failed requests (provider kill)	26	0
savings	—	53%

The savings came entirely from replayed cache hits — no provider call, no tokens. On a support workload that repeats questions daily, that ratio compounds. The availability delta speaks for itself: 0 failures through a provider kill, against 26 in the legacy run.

The bottom line

Migrating to an enterprise AI gateway is not a rewrite. It is a sequence of small, reversible moves — deploy beside, point one client, add fallback, enable cache, issue keys, wire observability, cut over. Measured on the rebuilt stack: 53% cost reduction on cacheable traffic, zero failed requests through a provider kill, and governance the legacy stack never had. The migration risk is low; the legacy risk is already on your pager.

636 Bytes: What Happens When You Stop Teaching RPA the Path

Don Johnson — Fri, 28 Aug 2026 05:03:21 +0000

I sat in a room with a very large automation system and a very familiar problem: it had become fragile.

The language came quickly. DOM. Timestamps. Timing windows. Selectors. Retries. Circle back.

Every word was reasonable.

Together, they described a hill.

The system was not fragile because its builders lacked discipline. It was fragile because the path had become the product. A bot was taught not what outcome had to be true, but which element to find, where to click, how long to wait, and what to retry when the page disagreed. Each workaround made sense locally. Together, they turned change into maintenance.

The first proof

The first proof on my machine was 636 bytes.

Not the model. Not the browser. Not the platform around it. It was a WebAssembly module: a tiny, portable program injected at the boundary between an intelligent proposal and an authorized browser action. Its smallness mattered because it kept that boundary deterministic, inspectable, and difficult to hide complexity inside.

It does not do everything. That is the point.

The 636-byte module is the injected proof; the hardened Rust kernel in this build compiles to 26,893 bytes. Neither contains the model, scheduler, credential store, policy engine, or evidence system. Those responsibilities live outside the browser boundary, where they can be isolated, governed, and replaced independently. Small is not the absence of an architecture. Small is a decision about where complexity is allowed to live.

Intelligence is not authority

An AI model can interpret an intent, inspect the current page, and propose what should happen next. It cannot grant itself permission. It does not get to cross a tenant, workspace, or origin boundary because doing so would be convenient. It does not receive a credential or press a consequential button merely because its reasoning sounds confident. Those decisions belong to local policy and the execution layer. Intelligence stays flexible; authority stays bounded.

That separation becomes a lifecycle:

intent → lease → observe → decide → act → verify → artifact

The intent defines the outcome. A lease gives one worker a short-lived claim on that work. The worker observes the page, the reasoning layer proposes a decision, and policy determines whether an action may execute. Verification checks the resulting state rather than trusting the click. Finally, an artifact preserves causally linked evidence of what happened. Every arrow is a boundary where the system can refuse, recover, or explain itself.

The hill has a bill

The hill also has a bill. To make that friction visible, I ran an illustrative planning scenario through agent-calc—not customer telemetry. Assume 100 workflows, two changes per workflow each month, four maintenance hours per change, and fully loaded labor at $140 per hour. The modeled comparison is stark:

Scenario	Modeled monthly cost
Legacy path maintenance	$112,000
Intent architecture	$20,098
Avoided friction	$91,902

That is an 82.1% modeled reduction. The percentage is not a promise; the assumptions are visible precisely so they can be challenged. The useful question is what the model exposes: how much are we spending to preserve instructions that describe yesterday's interface instead of today's desired outcome?

Test the outcome

Of course, an architecture that survives only a curated demo is just a cheaper failure. So we built the Component Gym: a synthetic web application designed to resist memorization. Controls move between runs. Buttons use different event listeners. The target may be buried among decoys inside randomized tabs and paginated lists. A seed makes each hostile arrangement reproducible without making it predictable to the agent.

The agent receives an intent, not a selector script. The harness then grades the resulting application state out of band. It does not care which path looked convincing or whether a click event fired. It cares whether the requested outcome became true.

In the filmed run, seed 710003 placed the needle inside a dynamic collection spread across tabs and pages. The agent was given the desired outcome and the browser's current state—not the target's coordinates or a prerecorded route. It had to navigate, distinguish the target from decoys, act, and leave the requested state behind. Only then did the independent grader return PASSED. The agent did not get to grade its own homework.

Determinism moved

None of this makes the DOM, timing, or selectors disappear. The browser still has structure. Events still happen in time. A selector may still be the right tactic for a particular action. What changes is their lifetime and ownership. In a path-driven system, those details harden into durable business logic. Here, they are runtime observations and disposable tactics, abandoned when the environment changes. Determinism has not been removed; it has been relocated into authorization, state transitions, verification, and evidence.

A primitive is not a platform

A passing gym run proves the primitive, not the platform. The small browser boundary reduces one category of fragility; it does not erase the distributed-systems, security, and operational work around it. The remaining obligations are less cinematic and more important:

Isolation: preserve tenant, workspace, worker-claim, and origin boundaries through every action.
Work ownership: make leases expire safely, recover interrupted work, and prevent retries from duplicating consequential actions.
Secrets and MFA: resolve credentials only at the authorized execution boundary, never inside model context, jobs, logs, screenshots, videos, or artifacts. MFA is on-behalf execution, never authentication bypass.
Browser lifecycle: attach to, control, recover, and release browser sessions without leaking state between workers or tenants.
Policy: deny actions that exceed the granted intent, even when the proposed action is technically possible.
Evidence: causally connect intent, observation, decision, authorization, action, verification, and artifact so a result can be explained after the browser is gone.

Compile the boundary smaller

The 636 bytes are not interesting because smaller software is automatically better. They are interesting because they make a refusal physical: do not bury orchestration, secrets, and authority inside the thing touching the page. Compile the trusted browser boundary smaller. Let reasoning adapt to the environment around it. Demand proof after every meaningful consequence.

I keep returning to that room. No one in it said anything absurd. The hill was built from rational decisions made under one inherited premise: the path must be preserved. Change that premise and the hill changes shape. We do not need to keep teaching automation every route through an interface. We can preserve the intent, rediscover the route, constrain the action, and verify the destination.

The next time an automation breaks and asks for another selector, timeout, or recovery layer, ask one question first: how much of this system protects the outcome—and how much protects yesterday's route?

[Boost]

Don Johnson — Wed, 26 Aug 2026 05:32:01 +0000

Transactional outbox challenges for tasks

Don Johnson

Aug 26

My agent mesh could coordinate. It couldn't introduce itself. So I added A2A.

#ai #agents #architecture #distributedsystems

10 min read

My agent mesh could coordinate. It couldn't introduce itself. So I added A2A.

Don Johnson — Wed, 26 Aug 2026 05:30:59 +0000

The 2:56 video above is a fictional medication-safety exercise. The gateway interoperability is tested; the six-organization incident is a deterministic simulation.

Read the corrected narration transcript.

Last time, I discovered that the QUIC transport in my agent framework had never actually transported anything.[1]

This time the transport was real. Five processes could find each other, exchange encrypted messages, reinforce independent conclusions, and let unsupported signals decay.

The mesh worked.

It still could not introduce itself to another agent.

There was no standard way to ask what the swarm could do. No retained task to retrieve after an internal signal expired. No interoperable progress stream. No cancellation contract. No artifact another framework would understand.

I had built a society with no border crossing.

Uncovers latent bugs and flawed core semantics

Don Johnson

Aug 19

My QUIC transport had never once been executed. Here's what happened when I ran it.

#rust #distributedsystems #networking #ai

8 min read

Google's Agent2Agent protocol gave that missing boundary a name. A2A was announced in April 2025 as an open protocol for agents built by different vendors and frameworks to discover one another, exchange messages, and collaborate without sharing their private memory, tools, or internal plans.[2] The project moved under Linux Foundation governance in June 2025, so "Google's A2A" is historically accurate, but no longer the whole story.[5]

What I needed was not a new brain for SMESH.

I needed a public contract in front of it.

A working mesh is not an interoperable agent

SMESH is the framework I designed for decentralized coordination between LLM agents. It borrows its mechanics from mycorrhizal networks rather than job queues.[6]

agents emit signals into a field;
signals lose intensity over time;
agents claim work according to local affinity;
independent agreement reinforces a claim;
unsupported claims disappear without a central process rejecting them;
trust changes what a node is willing to relay.

That solves an internal coordination problem. It answers questions such as:

Which specialist should take this work?
Has another independent agent seen the same thing?
Is this claim gaining support or merely being repeated?
Can a stale task disappear without a scheduler cleaning it up?

A2A solves a different problem.

It asks:

How does an outside agent discover this system?
How does it delegate a unit of work?
How does it watch a long-running task?
How does it cancel that task?
What retrievable result comes back?

The official A2A specification describes an interoperability layer for independent, potentially opaque agent systems. Its current v1 model separates canonical data objects, abstract operations, and concrete bindings such as JSON-RPC, gRPC, and HTTP+JSON/REST.[3][4]

The core vocabulary is small:

A2A concept	job
`AgentCard`	advertise identity, skills, interfaces, media modes, and security declarations
`Message`	carry one interaction turn as typed parts
`Task`	track stateful work through a lifecycle
`Artifact`	return result data associated with a task
`contextId`	group related messages and tasks

A2A v1 defines operations such as SendMessage, SendStreamingMessage, GetTask, ListTasks, CancelTask, and SubscribeToTask. A binding decides how those operations appear on the wire; the canonical task semantics stay the same.[4]

That distinction matters. If I tried to make A2A the swarm's internal coordination algorithm, I would flatten SMESH into a remote procedure call graph. If I tried to expose raw SMESH signals as the public protocol, every client would need to understand decay, relay probability, local trust, topology, and attestation sets.

Neither would be interoperability. It would be leakage.

The three layers are not competitors

The cleanest architecture I have found is this:

layer	relationship	owns
A2A	agent ↔ agent	discovery, tasks, streaming, cancellation, artifacts
SMESH	agent ↔ local swarm	claiming, diffusion, reinforcement, decay, trust
MCP	agent ↔ tool or data source	tool invocation and resource access

The official A2A documentation makes the same separation from MCP: MCP equips an agent with tools and resources; A2A lets independent agents collaborate as agents.[3]

For SMESH, that produced a hard architectural rule:

A2A is the external task contract. SMESH is the ephemeral internal coordination field.

The design requires the A2A ledger to outlive internal signal decay. In this MVP, that retention lasts only for the life of the process; restart durability still requires SQLite or Postgres.

The boundary is split between a path that runs now and a path that is only an integration seam:

External A2A client
        |
        | Agent Card / Send / Stream / Get / List / Cancel
        v
+-------------------- smesh-a2a ---------------------+
| A2A SDK routers + guarded request handler          |
| bounded process-local task ledger + executor       |
| validation + MeshDispatcher                        |
+----------------------------------------------------+
        |
        +-- current binary: LoopbackDispatcher
        |      `-> deterministic MeshEvent stream
        |
        `-- tested seam: ChannelDispatcher
               `-> real SignalType::Query
                   `-> SMESH runtime (not wired into the binary yet)
                       `-> MeshEvent stream
        |
        v
Process-lifetime A2A task history in the MVP

The desired live path is present as a typed, tested boundary. The checked-in standalone executable still takes the loopback path.

I kept the adapter in a separate repository. smesh-rust remains the coordination substrate. smesh-a2a can follow A2A's release cycle, server dependencies, and security boundary without pushing HTTP and SDK churn into the core mesh.[6][7]

The Agent Card is not the swarm

For a known endpoint, A2A capability discovery begins with an Agent Card: a public JSON description of an agent's interfaces, capabilities, skills, and security requirements.[4]

My first temptation was to list every internal role: security reviewer, tester, architect, performance analyst, contradiction sentinel.

That would have been wrong.

Those roles are implementation details. They may change per task. Some may not exist until the mesh senses the work. Publishing them would couple clients to an internal topology that SMESH is specifically designed to keep fluid.

The public card advertises one aggregate capability instead:

// Abridged from build_agent_card(). These are the public promises.
let supported_interfaces = vec![
    AgentInterface::new(
        format!("{base}/jsonrpc"),
        TRANSPORT_PROTOCOL_JSONRPC,
    ),
    AgentInterface::new(
        format!("{base}/rest"),
        TRANSPORT_PROTOCOL_HTTP_JSON,
    ),
];

let capabilities = AgentCapabilities {
    streaming: Some(true),
    push_notifications: Some(false),
    extensions: None,
    extended_agent_card: None,
};

let public_skill = AgentSkill {
    id: "smesh.collaborative-task".to_owned(),
    name: "Collaborative swarm task".to_owned(),
    description:
        "Coordinates specialist agents through SMESH and returns an accepted artifact."
            .to_owned(),
    tags: vec![
        "multi-agent".to_owned(),
        "coordination".to_owned(),
        "review".to_owned(),
        "testing".to_owned(),
    ],
    examples: Some(vec![
        "Review this Rust repository for correctness, security, and performance."
            .to_owned(),
    ]),
    input_modes: Some(vec!["text/plain".to_owned()]),
    output_modes: Some(vec![
        "text/plain".to_owned(),
        "application/json".to_owned(),
    ]),
    security_requirements: None,
};

The card says what an external client may rely on. It does not reveal how the swarm will organize itself, and it is not proof that the publisher should be trusted.

Discovery metadata is not authorization. A skill description is not a capability grant. A client saying tenant=important-customer does not make that identity real.

Those sound like obvious distinctions. They are also exactly the distinctions that disappear when a demo and a security model share the same JSON object.

One A2A message becomes a typed SMESH query

At ingress, the gateway accepts bounded inline text, creates a stable A2A task envelope, and translates it into the actual core signal type used by SMESH:

pub fn to_signal(&self, gateway_node_id: &str) -> Signal {
    Signal::builder(SignalType::Query)
        .payload_json(self)
        .origin(gateway_node_id)
        .build()
}

The payload carries the A2A task ID, context ID, protocol marker, and validated text. ChannelDispatcher packages that typed signal with the request and hands both to a runtime-owned worker.

That boundary is implemented and tested. The standalone binary still uses LoopbackDispatcher, so it does not yet inject the Query into a live multi-process SMESH runtime. The real runtime adapter is the next integration step.

Setting origin on the gateway Query is deliberate. In my previous article, independently corroborated claims omitted their origin from the content hash so identical conclusions could converge on one address. This is a different signal. A gateway Query is an ingress envelope, not a claim waiting for independent corroboration. Its source belongs in the record.

The boundary also rejects what it does not understand. The MVP accepts inline text only. It does not fetch a client-provided URL, dereference an arbitrary file, or treat external metadata as instructions for the mesh.

Inline text is boring, but I know exactly what crosses the boundary.

The task ledger and the signal field tell different kinds of truth

This was the most important design correction.

A SMESH signal is supposed to decay. If nobody reinforces a task, its intensity falls until it no longer matters. That is useful inside the coordination system because stale work cleans itself up.

An A2A task must not disappear because its internal coordination signal faded.

An external client may reconnect five minutes later and call GetTask. An auditor may list tasks by context. A user may need to see that a cancellation was accepted. An artifact must still belong to the task that produced it.

So the gateway cannot reconstruct its public state by looking at the mesh.

A2A task ledger = retained external task state (process-local today)
SMESH signal     = temporary coordination pressure

The ledger is authoritative for the A2A lifecycle. The mesh is authoritative only for its local coordination observations.

That also means terminal states are absorbing. Once a task is completed, failed, canceled, or rejected, a repeated message cannot quietly restart work under the same ID.

Streaming exposed a bug that all my tests had missed

A2A v1 supports both direct responses and stateful tasks. A task lifecycle stream begins with the Task itself, then emits ordered status or artifact updates, and closes when the task reaches a terminal state.[4]

My executor maps internal mesh events like this:

mesh dispatch accepted  -> Working
mesh progress           -> Working + status message
mesh artifact           -> Artifact update
mesh completion         -> Completed
mesh failure            -> Failed
accepted cancellation   -> Canceled

The stream ordering test passed.

Then an independent review pointed out that my own worker budget allowed 256 events while the upstream server's broadcast subscription buffer held 32. A fast worker could stay inside my documented limit and still outrun the initiating subscriber. The task might finish in storage while the client received an internal "subscription fell behind" error.

That is a particularly unpleasant distributed-systems bug because both sides can truthfully report different outcomes.

The fix was not to hope the subscriber ran faster. I reduced the worker event budget to 16, clamp caller-provided limits to that ceiling, and added a burst test through the official client.

This is what protocols do to a design: they force every implicit assumption to become somebody else's observable failure.

Cancellation has to stop work, not just change a status label

Forwarding CancelTask to a dispatcher was not enough.

If an internal worker ignored the request or kept its event stream open, the original client subscription could hang. Worse, late Working or Completed events could arrive after the public task had become Canceled.

The executor now owns a per-task cancellation token. The first accepted cancellation:

reaches the dispatcher;
wakes the active producer loop;
closes the original execution stream;
prevents post-cancel work from changing the terminal state.

Channel sends and cancellation acknowledgements have deadlines. Worker inactivity has a deadline. The total task has a deadline.

Cancellation is not a field update. It is a distributed state transition with work on both sides of the boundary.

The boring limits are the real feature

The first version was interoperable. It was not bounded enough to deserve trust.

A fail-closed review found unbounded task retention, unbounded worker output, cancellation leaks, missing dispatcher deadlines, terminal task reuse, and invalid input that could leave a task stranded in Submitted.

The current localhost-first gateway now bounds:

resource	default boundary
HTTP request body	128 KiB
accepted inline text	64 KiB
retained process-local tasks	1,024
active executions	64
worker events	16
artifacts per task	16
aggregate output per task	1 MiB
worker inactivity	30 seconds
total task execution	5 minutes
channel/cancel acknowledgement	5 seconds

It also refuses non-loopback binds unless an explicit unsafe override is present.

That override does not add authentication, TLS, tenant isolation, or authorization. It only disables the refusal. The current binary is for localhost and trusted integration work, not direct exposure to the internet.[7]

I am spelling that out because "supports enterprise authentication" in a protocol specification does not mean every prototype using the protocol is enterprise-secure.

The LIFELINE demo is a simulation, on purpose

The cover video follows a fictional medication-safety incident. Three hospitals see weak pieces of the same adverse-event pattern. Separate manufacturer, regulator, logistics, payer, and evidence agents contribute artifacts. Inside each public endpoint, a SMESH swarm claims work, reinforces evidence, contests an early hypothesis, and lets unsupported signals decay.

One logistics endpoint fails. Its task is canceled. A fallback is discovered. The incident continues. The agents converge on a recommendation, but a human incident commander owns the irreversible decision.

The visual separates the layers deliberately:

ivory arcs are A2A traffic between organizations;
green fields are SMESH activity inside an organization;
cyan shards are artifacts;
vermilion marks contradiction or failure;
one gold ring marks human authority.

Every visible event comes from a 55-event, ordered, hash-chained JSONL fixture generated as one complete file. The browser can play it, scrub it, inspect it, or export deterministic frames. The narrated film and interactive replay are available from the gateway repository.[7][8]

Honesty boundary: the gateway's A2A interoperability is exercised against the official Rust client. The LIFELINE six-organization trace is synthetic. It proves the replay contract and the intended architecture; it does not prove that six live SMESH runtimes executed the scenario.

A captured run across live SMESH runtimes would be operational proof. This demo is not.

What is implemented, and what is still a plan

implemented now	still required for an internet-facing system
A2A v1 Agent Card	authenticated principals
JSON-RPC and HTTP+JSON/REST bindings	tenant-aware authorization
official-client tests for discovery, JSON-RPC/REST send, streaming, and cancellation	persistent SQL task ledger
SSE task streaming	distributed quotas
Get, List, and Subscribe routes through the SDK handler	TLS termination and deployment policy
real `SignalType::Query` construction	live SMESH runtime adapter behind every organization
bounded process-local execution	push callback validation and SSRF controls
deterministic synthetic replay	captured multi-runtime causal trace

There are two easy ways to lie with a demo like this.

The first is to animate what you wish the system did.

The second is to run one loopback worker and describe it as a decentralized enterprise.

I would rather keep the boundary visible.

What A2A changed in the way I think about SMESH

Before this work, I thought of SMESH as the system.

Now I think of it as an interior.

The mesh can remain weird in useful ways. Signals can diffuse probabilistically. Specialists can appear and disappear. Trust can be local. Claims can decay. None of that has to leak into the contract presented to another agent.

MCP gives individual agents hands. SMESH gives a group local coordination. A2A gives that group a public identity, a task contract, and a cancel button.

The previous transport article ended with a lesson: code that has never been executed is a plan. This work left me with the same rule one layer higher:

A system nobody else can discover, hire, observe, or cancel is not interoperable. It is an island with excellent internal networking.

If you are building multi-agent systems, where would you draw the boundary between cross-organization interoperability and internal swarm coordination—and which decisions would you refuse to delegate?

copyleftdev / smesh-a2a

A2A v1 interoperability gateway for decentralized SMESH agent swarms

SMESH A2A

A2A v1 interoperability gateway for decentralized SMESH agent swarms.

SMESH remains the internal coordination substrate: signals diffuse, decay, reinforce, and accumulate attestations. A2A is the public contract for discovery, durable task lifecycle, streaming progress, cancellation, and artifacts.

What works

Official A2A v1 Rust types and server/client SDKs
Public Agent Card at /.well-known/agent-card.json
JSON-RPC endpoint at /jsonrpc
HTTP+JSON/REST endpoint at /rest
Synchronous and SSE streaming task execution
GetTask, ListTasks, SubscribeToTask, and CancelTask through a2a-rs
Strict inline-text validation with a 64 KiB default limit
Translation to a real smesh_core::SignalType::Query
Injectable MeshDispatcher boundary for a production SMESH runtime
Deterministic loopback worker for demos and interoperability tests
Official-client LIFELINE Response Director with concurrent commissioning, reconnect bounded cancellation, logistics fallback, independent review, and ID-complete run records
Deterministic organization-runtime LIFELINE outage scenario with joined cancellation one linked fallback, sibling continuity, and bounded restricted JSONL replay evidence
Five isolated organization-local LIFELINE SmeshRuntime…

View on GitHub

Interactive replay: copyleftdev.github.io/smesh-a2a

SMESH core: github.com/copyleftdev/smesh-rust

Sources

[1] https://dev.to/copyleftdev/my-quic-transport-had-never-once-been-executed-heres-what-happened-when-i-ran-it-24ge — My QUIC transport had never once been executed
[2] https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability — Announcing the Agent2Agent Protocol
[3] https://a2a-protocol.org/latest — What is A2A Protocol?
[4] https://a2a-protocol.org/latest/specification — A2A Protocol v1.0 specification
[5] https://linuxfoundation.org/press/linux-foundation-launches-the-agent2agent-protocol-project-to-enable-secure-intelligent-communication-between-ai-agents — Linux Foundation launches A2A project
[6] https://github.com/copyleftdev/smesh-rust — SMESH repository
[7] https://github.com/copyleftdev/smesh-a2a — SMESH A2A gateway repository
[8] https://youtu.be/EFPKaIuF8iA — SMESH A2A cinematic demo

A Wider Computer, Not a Bigger One: Modeling AI Inference Across Millions of Homes

Don Johnson — Tue, 25 Aug 2026 05:59:43 +0000

I modeled an AI inference fleet distributed across ordinary homes. What survived was narrower—and more plausible—than what I started with.

Picture a detached house on a cold evening. On one garage wall, an operator-owned compute appliance draws about five kilowatts—electrically comparable to an EV charger, though it runs much longer. It serves small AI models. In winter its waste heat can warm the house; in summer that heat must be carried away. The family bought nothing. They are paid to host it.

Now repeat that arrangement across a neighborhood, then a state, then millions of homes—not to assemble one enormous computer, but to create millions of independent inference workers. Each keeps a small model resident in GPU memory. New requests route around homes that are offline. The network grows by adding locations that were already built and connected to the grid.

That network does not exist. I call the idea HEARTH. Almost none of its physical pieces are exotic; the experiment is whether they can be arranged and scheduled as one system. Not a bigger computer. A wider one.

To make the appliance less abstract, I developed three concept form factors: a wall unit, a floor-standing thermal tower, and a duct-integrated mechanical-room unit. They are appearance and installation studies, not engineered products; their job is to expose the questions that a real prototype must answer.

The model kept changing its answer

Then I priced the accelerators, and the idea stopped working.

That was the first useful result. My original comparison had focused on the dramatic expense of constructing a data center while treating the compute hardware as identical on both sides. But identical hardware does not disappear from the economics. If one location keeps an expensive GPU busier than another, utilization can overwhelm everything the cheaper building saves.

I rebuilt the model five times. Each pass introduced a constraint the previous one had missed:

Pass	What changed	Model verdict
1	Compared the facilities around the hardware	Homes win
2	Included accelerator cost and utilization	Homes lose
3	Used consumer and data-center hardware at quoted prices	Homes win
4	Modeled production batching for a 32B model	Homes lose
5	Tested different model sizes	Homes win only for the small-model case

What survived: small, independent inference

The fifth pass did not rescue my original proposal. It replaced it. A residential fleet is not a cheaper place to run every AI workload. It cannot pool memory across the internet, train a frontier model, or make residential latency disappear. What survived was narrower: small-model inference in which one node completes one request and no durable state is tied to a particular house.

Model size changes the economics because inference is not just a race between GPUs. During generation, a server repeatedly reads the model's weights while advancing many requests together. That is batching. A larger batch spreads each weight read across more paying tokens—but every active request also consumes working memory, usually called the KV cache.

An 8B model—roughly eight billion parameters—leaves enough memory on a 32 GB consumer GPU for a useful production batch. In my model, both the home GPU and the data-center hardware then become limited mainly by compute, allowing the consumer part's much lower purchase price to matter. At 32B, the consumer card runs short of memory first. Its batch stops growing while high-memory data-center hardware keeps filling, and the verdict reverses.

Model class	What the current model says
8B	Candidate workload; the consumer GPU retains useful batching headroom
32B	Data-center hardware wins unless other advantages offset its batching edge
70B	Excluded by my single-GPU serving assumption; multi-GPU serving was not modeled

The current result is conditional. Before homeowner compensation and fleet-level operating costs, and under RTX 5090 MSRP, an 85% accelerator duty factor, identical serving-efficiency assumptions for both venues, and a quantized 8B workload, my roofline model estimates the residential hardware-and-energy stack at about 0.49 times the cost per token of the best data-center case in the sweep, based on GB200 NVL72 rack pricing. That is a reproducible scenario output, not observed production performance or a fully loaded business cost. An apples-to-apples benchmark—or the costs excluded here—could erase the advantage.

This is the distinction behind wide, not deep. HEARTH would not divide one giant model among houses or combine residential GPUs into a virtual supercomputer. Each node would answer complete, independent requests with a model it already holds locally. Adding homes increases the number and variety of requests the fleet can serve; it does not make any one node larger.

The asset is the grid endpoint

Why put that workload in homes at all? Not because residential electricity is cheap—it usually is not—or because waste heat makes energy free. The asset is the connection behind the meter.

Berkeley Lab estimated that US data centers used about 176 TWh of electricity in 2023. Its June 2026 update projects 521–843 TWh in 2030, with a 649 TWh reference case. Serving that growth is not simply a question of buying more GPUs. A large new campus may need substations, transformers, transmission upgrades, and a negotiated path to connect a load the local grid was never designed to carry. The 410 GW of prospective large loads reported in ERCOT—about 87% associated with data centers—is not a forecast of what will be built. It is evidence of how much demand is converging on the same constrained process.

The United States has about 82.5 million one-unit detached housing units. That is physical stock, not an estimate of eligible hosts: occupancy, broadband, electrical capacity, utility approval, and household consent would reduce it substantially. Each candidate already has a meter, an electrical service, and a physical building around it. A compute node installed behind that meter may avoid the transmission-scale interconnection required by a new campus. That is the inversion: instead of bringing an enormous new grid connection to the compute, bring a modest amount of compute to many connections that already exist.

Existing does not mean unused. A continuous 5 kW load is much less forgiving than an EV that charges for a few hours. Some homes would need panel or service upgrades; utilities may require review; and the local distribution transformer remains a hard physical constraint. My one-node-per-transformer rule is therefore a screening hypothesis, not a validated safety rule or national capacity estimate. Transformer loading, telemetry, and thermal behavior belong in the first utility-supervised field test.

To the household, this is a hosting contract, not an investment. The operator would own and maintain the appliance, meter and reimburse its electricity, and pay the family a share of revenue. None of that has been field-tested; fire and insurance rules, noise, summer heat rejection, ISP terms, maintenance, and upgrades remain field questions.

One request goes to one home

At the software layer, the system is deliberately less exotic. A control plane would assign replicas of small models to qualifying nodes and distribute checkpoints outside the request path. Each node would store its assigned checkpoints locally and keep its serving model resident in GPU memory. The request router would know which homes have the requested model resident, which are healthy, and which have batch capacity available.

registry ── sync ──► home node

client ──► router ──► available home ──► response
               ▲              │
               └── telemetry ─┘

During generation, one home holds the request's KV cache and returns the tokens; no activations or model layers cross the residential network. If the node disappears, the in-flight generation is lost. New work can route elsewhere, and a retryable request can start again against another replica, although it may not produce identical tokens. The system does not inherit data-center reliability merely because it has many machines.

At the network level, remote attestation, model security, and prompt and output confidentiality on hardware outside an operator-controlled facility remain unresolved. Distribution makes failures routable; it does not make operations disappear.

Five conditions decide whether this is real

The hardware must be legally and commercially usable. NVIDIA's current driver agreement says the software may not be used to provide commercial hosting services and that GeForce and Titan software is not licensed for data-center deployment. Whether an operator could obtain written authorization or a different license for this residential topology must be resolved with NVIDIA and qualified counsel before a hardware pilot. Warranty, support, and a credible procurement channel must also exist.
Edge serving must sustain the modeled batches. The same checkpoint, quantization, context length, serving stack, and request mix must be benchmarked on the candidate consumer GPU and data-center hardware. A spreadsheet cannot substitute for that result.
Demand must keep the silicon busy. Cheap idle GPUs are still expensive. Customers must commit enough suitable 8B inference work—classification, extraction, routing, drafting, or other bounded tasks—to maintain high utilization at a viable token price.
The location must pass the energy screen. Residential rates vary widely, and heat reuse is a seasonal credit rather than the thesis. Every deployment needs to work after metered electricity, host payment, and summer heat rejection.
The distribution grid must approve the load. A real utility must confirm that a real home, service, and transformer can carry it continuously. National averages are not permission to connect.

Those are not caveats around the result. They are the result. HEARTH exists only at their intersection.

The first pilot should contain no houses

That changes the order of the pilot. My original plan began with 250 homes and grew through increasingly expensive gates. It tested installation first and market demand later. The model eventually exposed the flaw: there is no reason to put hardware in a house before proving that anyone will buy this particular kind of inference.

The first HEARTH pilot should therefore contain no houses. Rent the candidate consumer and data-center hardware. Resolve the license question. Run the same models through the same serving stack, measure delivered tokens and power at production batch, and ask customers to pay for the workloads the residential fleet is supposed to serve. If the measured economics or demand fail, stop before an electrician is dispatched.

Only then move to 250 homes. That field trial buys answers the lab cannot: Can crews install safely and consistently? What do transformers do under continuous load? How loud and hot are the nodes in July? Can the operator keep them online without drowning in truck rolls—and will families keep hosting them?

Four million homes is not the next milestone

Four million homes is an image of the possible system, not an adoption forecast. The serious next milestone is smaller: one workload that benchmarks, one customer willing to buy it, one utility willing to supervise it, and then the first 250 homes.

I published the full feasibility study, the source model and inputs, and every assumption I used. If the idea is wrong, I want the failure to be reproducible too.

HEARTH is not a forecast. It is a claim specific enough to test—and to kill if it fails. The buildings, meters, and many of the grid and network endpoints already surround us. What is missing is the appliance, the dedicated installation, the operating system around it, and an agreement worth signing.

If those pieces work, the neighborhood still looks like a neighborhood on a cold evening. Behind some garage walls, independent machines serve requests from a network spanning the country—each complete on its own, all scheduled together.

No monument to compute. A million warm windows.