DEV Community: Don Johnson

What Happens When You Give a Classifier Some Game Sense?

Don Johnson — Tue, 20 Oct 2026 08:11:48 +0000

I've been experimenting with a retro defense game in Rust called Signal / Siege. The question behind it is simple: how much can traditional game AI help a classifier?

Incoming contacts approach a base. Some are hostile, others should pass safely. The defense can scan, intercept, or activate a shield, but energy is limited. Spending too much on one contact can leave it unprepared for the next.

That creates two connected problems: estimating what a contact might be, and deciding what to do about it.

The video above shows three approaches:

Bayes + Utility: Bayesian classification paired with a system that scores possible actions.
Jev + Rules: Jev classification paired with simple rules and Bayesian backup.
Jev + Hybrid: Jev classification supported by action scoring and short forecasts of defensive opportunities and energy needs.

The supporting techniques come from the traditional game AI toolkit. They help the defense weigh consequences, conserve energy, and recognize when waiting could be more useful than acting immediately. Jev's underlying model remains unchanged.

In the recorded demonstration, the Bayesian and hybrid systems survived. Jev with simple rules lost the base.

We also ran a separate, matched offline comparison across 256 fresh seeds. Using a fixed set of captured Jev responses and 200 milliseconds of simulated latency, the rules approach won 112 matches. Adding action scoring and forecasting brought that to 255.

Those were replay-based experiments, not hundreds of live API games. The Jev panels in the video replay actual API-backed matches. The video's three modes have different scanning policies, and the adaptive opponent can respond differently as each match unfolds.

The Bayesian reference also knows the game's synthetic observation model, giving it an advantage in this setting. These results don't establish a general ranking of classifiers.

What interests me is the division of work: a classifier contributes a judgment, while the surrounding system manages timing, costs, and consequences.

Next, I want to test that relationship with richer observations and less predictable scenarios.

Watch the side-by-side experiment on YouTube.

Stop Guessing. Let the Algorithm Find the Sweet Spot.

Don Johnson — Thu, 15 Oct 2026 23:38:36 +0000

Simulated annealing gives a search permission to get worse before it gets better. The useful part is learning when that permission pays off.

There is a familiar ritual in software engineering: increase the thread count, change the batch size, run the benchmark, and squint at the result. Then change another number.

Eventually, something looks better, and we call it optimized.

But better than the previous attempt is a low bar. How much of the search space did we explore? What did each experiment cost? Does the improvement survive a different workload?

Those questions are the difference between tuning by habit and running an optimization experiment.

Simulated annealing is one way to run that experiment. Its most interesting feature is also its least intuitive: sometimes it accepts a worse candidate on purpose.

Let's examine why, implement the rule in Rust, and compare it with greedy search, random search, and a grid. Every method gets the same evaluation budget. The experiment is small enough to run locally and check exhaustively.

Why the first valley can trap you

Imagine walking through a mountain range at night, trying to find its lowest point. You can see nearby terrain, so you keep taking downhill steps.

Eventually, every nearby step leads uphill. You have found a local minimum. Another valley might be deeper, but your rule prevents you from crossing the ridge.

That is the problem with accepting only immediate improvements. Whether a solution is a local minimum also depends on the moves your search permits. A larger jump might cross a barrier that a small step cannot.

Simulated annealing sometimes accepts uphill moves. It borrows its name from the connection between cooling physical materials and searching for low-energy configurations, developed in Kirkpatrick, Gelatt, and Vecchi's 1983 paper.

In the algorithm, energy becomes an objective score. Temperature becomes a parameter controlling the willingness to accept regressions. Cooling gradually reduces that willingness.

This creates a route out of some local minima. A finite run can still miss the best solution.

The acceptance rule

Suppose lower scores are better. Calculate the change from the current candidate:

delta = score(candidate) - score(current)

Accept improvements and ties. For a worse candidate, draw a uniform random number between zero inclusive and one exclusive, then accept when it is smaller than:

p = exp(-delta / temperature)

This is the usual Metropolis acceptance rule for symmetric proposals, used in the original simulated annealing paper. Symmetric means the chance of proposing a move equals the chance of proposing its reverse.

For a regression of two score units, the probability is about 82% at temperature 10, 14% at temperature 1, and effectively zero at temperature 0.1.

Temperature has the same units as the score. Multiply every score by 1,000 and keep the old temperature, and you have changed the search's behavior. A cooling schedule is a design choice, not a magic constant.

Also keep two records: where the search is now and the best candidate it has ever found. Exploring a worse configuration should not erase a better answer.

Define what good means before searching

Consider these hypothetical service settings:

Setting	Allowed values	Count
Worker threads	Every integer from 1 to 64	64
Batch size	Every integer from 32 to 4,096	4,065
Timeout	50 to 2,000 milliseconds, stepping by 10	196

That is 50,991,360 configurations. At one evaluation per second, sequential exhaustive search would take about 590 days, before repeats or setup costs.

The hard part starts before choosing a search algorithm. What makes a configuration good?

One possible objective is:

score = 0.5 * (p99_latency / reference_latency)
      + 0.3 * (memory / reference_memory)
      - 0.2 * (throughput / reference_throughput)

Fix those positive reference values before the experiment. The ratios are dimensionless; the weights express your priorities. Changing the scales during the search would make scores from different iterations incomparable.

Hard requirements need separate treatment. If a configuration violates a memory cap, correctness invariant, or latency limit, reject it as infeasible. A throughput gain should not quietly compensate for breaking a requirement.

Even among feasible candidates, there may be several reasonable trade-offs. A candidate lies on the Pareto frontier if no feasible alternative improves an objective without worsening another. A weighted score expresses a preference among trade-offs; it does not discover your priorities for you.

The most important function is the one that decides what good looks like.

Give the algorithm something we can verify

For an executable demonstration, use a smaller problem where we can check the answer exhaustively.

The companion Rust project searches a 64 by 64 grid of abstract parameter indices. Each index maps onto a coordinate between -5.12 and 5.12. Its objective is the two-dimensional Rastrigin function:

E(x, y) = x*x + y*y + 20
        - 10*cos(2*pi*x) - 10*cos(2*pi*y)

The quadratic terms pull the search toward the center. The cosine terms add many local valleys. These scores have no latency, throughput, or dollar units. This is a deterministic mathematical experiment.

The continuous function reaches zero at the origin. Our even-sized grid does not contain the origin, so we enumerate all 4,096 grid points to find the discrete optimum. That enumeration is a separate reference calculation, unavailable to the search algorithms.

The comparison fixes these choices before running:

256 objective evaluations per run, including the initial candidate and repeated proposals.
The same initial point, index (63, 63), for each method.
30 seeds, numbered 1 through 30, for the stochastic methods.
Annealing and greedy search share a neighborhood: move one index by a signed step of 1 through 8. An out-of-range proposal stays in place and still costs an evaluation.
Annealing cools geometrically from 20 to 0.1 over its 255 proposals. Greedy search accepts only improvements or ties.
Random search samples the whole grid with replacement. Grid search uses a fixed 16 by 16 subset, rounding evenly spaced coordinates to integer indices.

Equal evaluation counts make one cost comparable. They do not give all methods identical information or movement rules. The starting point, neighborhood, grid spacing, and cooling schedule remain consequential choices.

The Rust loop

The project has no external Rust dependencies. Its seeded generator makes the search repeatable on the tested toolchain. This is the loop from the implementation; validation, random generation, and candidate construction live in the companion source.

/// Minimize an objective while retaining the best state ever visited.
pub fn anneal<S: Clone>(
    initial: S,
    mut score: impl FnMut(&S) -> f64,
    mut neighbor: impl FnMut(&S, &mut Rng) -> S,
    seed: u64,
    schedule: Schedule,
) -> Result<Outcome<S>, String> {
    schedule.validate()?;
    let mut rng = Rng::new(seed)?;
    let mut current = initial;
    let mut current_score = finite(score(&current))?;
    let mut result = Outcome {
        best: current.clone(),
        best_score: current_score,
        evaluations: 1,
        accepted_uphill: 0,
    };

    for proposal in 0..schedule.evaluations - 1 {
        let candidate = neighbor(&current, &mut rng);
        let candidate_score = finite(score(&candidate))?;
        result.evaluations += 1;
        let delta = finite(candidate_score - current_score)?;
        let temperature = schedule.temperature(proposal);
        if delta <= 0.0 || rng.unit() < (-delta / temperature).exp() {
            result.accepted_uphill += usize::from(delta > 0.0);
            current = candidate;
            current_score = candidate_score;
            if current_score < result.best_score {
                result.best = current.clone();
                result.best_score = current_score;
            }
        }
    }
    Ok(result)
}

The boundary handling matters. We choose signed moves with equal probability and leave invalid moves at the current point. Simply clamping an arbitrary move to the nearest boundary would accumulate probability there and lose the symmetry assumed by this acceptance rule.

The implementation also rejects invalid settings and nonfinite objective values, counts evaluations explicitly, and tests that a temporarily worse current state cannot overwrite the best result.

A separate unit test makes the barrier explicit: three adjacent states have costs 1, 2, 0. Starting at the first state, greedy search cannot cross the middle state. With the test's fixed seed and temperature, annealing accepts the uphill move and then reaches zero. That tiny fixture verifies the mechanism; it is not evidence that annealing usually wins.

Run it and compare the evidence

The companion project includes the Rust source, raw results, an independent results checker, and a plotting script. Clone it and run:

git clone https://github.com/copyleftdev/simulated-annealing-article.git
cd simulated-annealing-article
export CARGO_TARGET_DIR=target
cargo test --locked
cargo run --release --locked -- results/raw.csv
python3 scripts/verify.py --reproduce
python3 scripts/analyze.py

Python analysis uses the standard library. Add Matplotlib to regenerate the figures with python3 scripts/analyze.py --plots.

Method	Median best score	Best to worst score	Exact grid optimum hits
Annealing	2.565	2.565 to 2.565	30/30
Greedy	2.565	2.565 to 2.565	30/30
Random	4.220	2.565 to 8.425	5/30
Coarse grid	3.480	3.480 to 3.480	No (deterministic)

Lower is better. The exhaustive grid optimum is 2.564508, and both annealing and greedy search reached it in all 30 runs. Annealing accepted between 14 and 33 uphill moves per run, but those moves did not improve its final score over greedy search here.

That result is a useful correction to the valley metaphor. Our neighborhood allows jumps of up to eight index positions. The cosine ripples repeat roughly every six positions, so greedy search can jump across hills without accepting a worse landing point. This experiment gives us no reason to prefer annealing over this greedy baseline on final score.

Random search reached the optimum in five runs. The coarse grid returned the same result every time; repeating it creates no new statistical evidence. Its chosen spacing misses the central indices 31 and 32, so its result is sensitive to grid alignment. These are observed score ranges, not confidence intervals or a universal ranking.

The raw CSV also records search duration. With such a cheap objective, optimizer overhead and machine conditions can substantially affect timing. An evaluation count is not a measurement of infrastructure savings.

There is a second check: after each run selects its best candidate, the harness shifts each coordinate by either -0.1 or +0.1, evaluates all four combinations, and records their mean score. These four evaluations are outside the 256-call search budget and its timer, and the search never sees their scores. This tests sensitivity to a small, specified change in the formula. It does not simulate production traffic or establish robustness to measurement noise.

That distinction matters. Thirty seeds describe variation in search outcomes on one fixed problem. They are not thirty independent workloads.

Try breaking the optimizer

A passing test suite deserves a challenge too. We used cargo-mutants to alter the library and rerun its tests. The original eight Rust tests missed 53 of 181 generated mutations, including deleting the minus sign in the acceptance formula.

After adding acceptance and contract tests, 153 mutations triggered test failures. Two survived: they permute symmetric terms in the shifted-score average, preserving its mathematical value. Another 25 did not compile, and one caused a nonterminating loop detected by a timeout. We report those categories separately in the mutation review.

The statistical check drives the real kernel through one million uphill proposals across five predeclared temperatures. At a score difference of 2 and temperature 2, expected acceptance is about 36.79%; observed acceptance was 36.988%. Every case fell within the predeclared tolerance of about 0.635 percentage points.

That tolerance uses a Hoeffding bound under ideal independent random draws. Our fixed pseudorandom streams make this a reproducible regression check, not a certification of randomness. The statistical review records all cases and assumptions. The expanded suite has 23 tests; production optimizer code and benchmark results are unchanged.

The optimizer proposes. The testing harness disposes.

A real service adds a complication: measurements fluctuate.

A noisy score can make a regression look like an improvement. Searching thousands of configurations also creates thousands of opportunities to get lucky. The most impressive single measurement may be the least trustworthy result.

For service tuning, make the evaluator earn its authority:

Fix the workload, objective, constraints, and search budget before comparing methods.
Repeat measurements, control warmup and machine conditions, and interleave candidates when drift could bias the comparison. Charge those repetitions to the evaluation budget.
Evaluate finalists on reserved workloads, including skewed inputs, contention, and failure cases. Report variability alongside the central estimate.
Check correctness and operational limits before promoting a candidate. Keep a rollback path and monitor the deployed result.

Once reserved workloads influence another round of tuning, they become development data. A fresh evaluation set is needed for a fresh generalization claim.

Accepting a worse candidate is a search decision inside an evaluation environment. It does not require deploying that candidate to customers.

Where this becomes an engineering habit

The same framing gives several everyday tuning problems a concrete shape:

Problem	Candidate decisions	Objective and constraints
Service performance	Workers, batching, queue depth	Minimize cost per completed request while meeting latency and correctness limits
AI agent orchestration	Model routing, retry budget, verification threshold	Minimize cost on a fixed task set while meeting quality and latency requirements
Test scheduling	Test order and assignment to workers	Minimize completion time while preserving the required suite and dependencies
Infrastructure sizing	Instance mix, capacity, scaling thresholds	Minimize cost under representative demand while meeting capacity and recovery requirements

Applying search to these problems requires a candidate representation, legal moves, and a trustworthy evaluator. In an agent system, for example, a cheap answer is useful only if the quality check deserves trust. Evaluators can be fooled or overfit too.

The separation is useful: a search proposes configurations, an evaluator measures outcomes, and a deployment process decides what can ship.

You do not need to adopt annealing to adopt that architecture.

Make annealing earn its place

Start with a baseline you can understand. If the space is small, enumerate it. If a coarse grid or random search gives good results within the budget, that is valuable evidence.

Annealing becomes interesting when useful local moves exist but a search that accepts only improvements gets trapped. Its neighborhood and temperature schedule still need attention. If objective evaluations are expensive, compare methods designed to spend fewer evaluations. If useful gradients exist, consider methods that exploit them.

The cost of tuning the optimizer also belongs in the accounting. Trying twenty cooling schedules and reporting only the best is another search, with another budget.

The engineering habit is simple: define success, constrain the experiment, compare alternatives, and keep the evidence. Repeat when the environment changes.

Sometimes a better answer is on the other side of a hill. Sometimes a simpler search already finds it. Measure which situation you are in.

What configuration have you spent hours tuning manually, and what would a trustworthy objective function for it look like?

Disclosure: AI agents drafted and revised this article, implemented the experiment, and performed technical review. The numerical results come from executing the accompanying code.

[Boost]

Don Johnson — Sun, 11 Oct 2026 02:33:16 +0000

Don Johnson

Oct 10

Evidence-Driven Development: Give Your Coding Agent Something to Prove

#ai #testing #tutorial #python

9 min read

Evidence-Driven Development: Give Your Coding Agent Something to Prove

Don Johnson — Sat, 10 Oct 2026 05:28:21 +0000

An AI coding agent finishes the feature. The tests pass. The explanation sounds reasonable.

Then someone asks a small question: “What happens if I run that request again?”

That question can change the shape of a project. Now “done” needs to mean something observable. The answer needs an experiment, and the experiment needs a way to be wrong.

This tutorial builds a tiny task list around that idea. By the end, you will have a working application, two deliberately broken versions, and a folder of evidence showing which promises each version keeps.

I use evidence-driven development to describe this working habit: write down a claim, decide what would disprove it, run the relevant experiment, and let the result govern the next decision.

The coding agent can help at every step. The evidence must remain inspectable without trusting the agent's summary.

First, some credit for the idea

The name predates coding agents. Jan Bosch's 2017 book, Speed, Data, and Ecosystems, includes an Evidence-Driven Development chapter. Its contents connect requirements, hypotheses and experiments. I'm using the term for a practical agent-assisted workflow, without claiming to have invented a methodology. Publisher's catalog.

There is related work in AI, too. Xia and colleagues describe evaluation-driven development and operations, or EDDOps, as a continuing feedback loop for LLM agents. Their paper addresses a broader agent lifecycle than the small application we will build here. EDDOps paper.

TDD still gives us a useful red–green–refactor loop. BDD helps express behavior through examples people can discuss. In this tutorial, EDD names the discipline of keeping the claim, execution conditions, observed result and resulting decision connected. Those practices fit together. TDD, Given–When–Then.

A small application with a real promise

Our app is Tiny Tasks. It adds tasks and lists them. Python and SQLite are enough; the application and experiment runner use only the Python standard library. Use Python 3.10 or newer with SQLite support.

The companion repository contains the complete source, cover artwork and recorded evidence. Clone it to follow along:

copyleftdev / evidence-driven-development

A runnable evidence-driven development tutorial: falsifiable claims, SQLite retries, negative controls, and reproducible results.

Evidence-Driven Development: Tiny Tasks

A standalone companion project for Evidence-Driven Development: Give Your Coding Agent Something to Prove. A tiny local task list teaches explicit claims, negative controls fresh-process experiments, nonzero exercise gates and reproducible evidence.

Run

Requires Python 3.10+ with its standard SQLite module. No application dependencies, API keys, model account, network service or cloud resources are required.

git clone https://github.com/copyleftdev/evidence-driven-development.git
cd evidence-driven-development
python3 -m tiny_tasks add "Water plants" --request-id plant-1
python3 -m tiny_tasks add "Water plants" --request-id plant-1
python3 -m tiny_tasks list
python3 -m unittest discover -s tests -v
python3 scripts/prove.py

The second create returns the original task with created: false. Use a new request ID for a genuinely new action. Titles are compared after trimming surrounding whitespace Default data: .local/tasks.sqlite; set a different file using --db PATH before the add or list subcommand. Conflicting request reuse and invalid input return…

View on GitHub

git clone https://github.com/copyleftdev/evidence-driven-development.git
cd evidence-driven-development
git checkout 1f03111db02d359b24602992eddae60122d49d74
python3 -m tiny_tasks add "Water plants" --request-id plant-1
python3 -m tiny_tasks list

The checkout selects the source revision used in this article. The add command returns a task. The list command starts a new process and reads it back. The default database lives at .local/tasks.sqlite.

The request ID belongs to the user's intended action. When the caller retries that same action, it sends the same ID. A genuinely new task gets a new ID—even if the title happens to match.

That small distinction gives us something worth proving.

1. Write a claim that can lose

“Make the task list reliable” leaves too much room for interpretation.

Here is the experiment contract we wrote before implementing the application:

Claim	A result that disproves it
Acknowledged tasks survive a process ending	A new process cannot read the saved task
Repeating a create request returns the original task	A retry creates another task or returns a different ID
A request ID cannot silently acquire a different meaning	Reusing it with a different title succeeds or changes saved data
Invalid input leaves the list alone	An empty title adds a task

The complete contract is embedded here, with a commit-pinned copy in the repository:

The repository records those claims in experiments/001-reliable-tasks/experiment.json. It also declares three restart rounds, eight concurrent retry processes, two independent repetitions, and the scenarios that must actually execute.

These numbers are a small teaching workload. They are not a statistical reliability estimate or a throughput target.

Before asking an agent for code, give it this kind of task:

Build a local task list that adds and lists tasks.

First propose observable claims and the outcomes that would disprove them.
Preserve acknowledged tasks across fresh processes.
Bind each create request ID to one normalized title and one task.
Exercise concurrent retries and conflicting reuse.

Keep the application small. Use local synthetic data.
Do not claim properties the experiment does not exercise.

Review the claims yourself. An agent that quietly narrows “survives interruption” into “works twice in the same object” can produce a beautiful test for the wrong promise.

2. Give the agent a place to leave evidence

The project has a few small documents with distinct jobs:

AGENTS.md                         working rules and commands
docs/plan.md                      task progress and handoff
docs/decisions.md                 decisions and their reasons
experiments/001-reliable-tasks/
  experiment.json                claims written before code
  README.md                      findings and limitations
  runs/<run-id>/                  actual outputs and source hashes

AGENTS.md tells an agent where these things live. The experiment file defines the claim. The run directory records what happened. The decision document explains what we chose because of it.

You can start with a short instruction:

Read the experiment contract before changing behavior.
Run the declared scenarios and keep failed outputs.
Create a new run directory; never replace an earlier run.
Report which claims passed, which failed, and what was not tested.

Instructions need maintenance. If a useful constraint already lives in a canonical document, link to it. A growing pile of repeated instructions can make the next agent's job harder. OpenAI's current guidance similarly favors focused instructions and task-specific context. Guidance on skills and project instructions.

3. Implement the smallest useful design

Tiny Tasks stores each task alongside its request ID. A unique database constraint prevents two stored rows from sharing that ID.

Creation uses one transaction. It looks for the ID, compares the normalized title if the ID exists, and inserts only when it is new. The command returns success after the transaction commits.

The complete tiny_tasks/store.py implementation is embedded below. You can also read the commit-pinned source.

“Normalized” has an explicit meaning here: the app removes leading and trailing whitespace before comparing titles. The add method shows the transaction boundary; validation, connection setup and listing are included so you can inspect the whole example.

That boundary matters because checking in one transaction and inserting later would leave room for another process to intervene. SQLite documents the write-transaction behavior of BEGIN IMMEDIATE in its transaction reference. Still, plausible code is only a candidate explanation. We have to run the workload we promised to support.

4. Cross the boundary you are making a claim about

A test can create a Store, add a task and immediately list it. That is useful. It does not establish what happens in a new process.

Our experiment runner invokes the actual CLI in subprocesses. Every command starts fresh. For the restart scenario, it adds three tasks in sequence, checking after each successful create that a separate process can read the expected list.

For retries, it exercises two different situations:

A create succeeds, its caller ignores the response, and a fresh process repeats the request.
Eight processes submit the same request, launched through a synchronization barrier.

The second scenario checks the returned identities, the number of created: true responses and the rows actually saved. Exactly one task should exist.

The ignored response is a controlled stand-in for a caller that is unsure whether an action succeeded. We did not cut a network connection or kill a process during a commit. Keeping that distinction visible makes the result useful.

5. Break the application on purpose

Now comes my favorite part: establish that the experiment can catch the defect it claims to detect.

The lab/ directory contains two intentionally wrong implementations. They use the same command shape as the real app:

Volatile: returns successful creates but keeps no durable tasks.
Duplicates: saves tasks but ignores request identity, inserting a row on every retry.

These are negative controls: known-bad examples that the checks must reject for a specific reason. They are teaching fixtures, not bugs we are pretending to have discovered accidentally.

The volatile version must fail the restart claim. The duplicate-accepting version must fail the concurrent retry claim. A syntax error in a broken version would not establish either property, so inspect the saved responses and failure reasons.

Review caught a weakness in the first version of the gate: it accepted a control's failure without checking its reason. A child-process crash could have masqueraded as a successful bug detection. We added regression tests, required nonzero exercise counts for the controls, and made the gate check the targeted failure reason. The earlier run remains in the repository, with that gate limitation recorded.

There is another way to get a misleading green result: execute nothing.

Our runner tracks how many times each required scenario ran. This intentionally incomplete report must fail even though every claimed outcome says true:

empty = {
    "claims": {name: True for name in required_scenarios},
    "exercised": {name: 0 for name in required_scenarios},
}

A check that refuses zero activity is an exercise gate. It answers a basic question before interpreting success: did we actually attempt the scenarios our conclusion depends on?

6. Run it and keep the receipts

From the companion project:

python3 -m unittest discover -s tests -v
python3 scripts/prove.py

The first command runs nine tests: six for application behavior and three for the harness. The second runs the process-boundary experiment, both negative-control implementations and the empty-exercise control.

In the recorded run, the results were:

Implementation	Restart	Retry after ignored response	Concurrent retry	Conflicting reuse	Invalid input
Volatile control	Fail	Fail	Fail	Fail	Pass
Duplicate-accepting control	Pass	Fail	Fail	Fail	Pass
SQLite candidate	Pass	Pass	Pass	Pass	Pass

Each implementation ran twice. The pass/fail outcomes and exercise counts matched between repetitions. The SQLite candidate exercised three restart rounds, one ignored-response retry, eight concurrent submissions, one conflicting reuse and one invalid-input scenario per repetition.

The two controls failed where intended. The report with zero exercised scenarios was rejected. Those facts allow the overall experiment gate to pass.

You will find a new directory under experiments/001-reliable-tasks/runs/. It contains:

A manifest with Python and SQLite versions and hashes of the application, harness and contract.
A response trace and scenario outcomes for each implementation and repetition.
A summary with the overall decision and its scope.

The captured run used for this article is 20261010T042729Z-633b0705. Your run will have its own ID. The project preserves the original run directory so you can inspect the evidence behind this table, including its summary and source manifest.

The hashes help identify the source files used in a run. They do not establish who performed it, and someone who can rewrite both the files and their manifest can forge a matching set. This is reproducible local evidence, not a signed attestation system.

7. Make a decision the evidence supports

We can now choose SQLite for this local application's declared process and retry behavior.

We have not tested power failure, a corrupt disk, a network filesystem, large task volumes, security boundaries or sustained throughput. We have not measured how often different coding models produce a correct implementation. Those would be separate questions with different experiments.

This is also where you resist rounding a result into the outcome you wanted. If a target is missed, keep the result and the original threshold visible. A follow-up experiment can ask a better question; it should have a new identity.

A useful agent handoff looks like this:

Implemented: persistent task creation and payload-bound retries.
Verified: the recorded process-boundary scenarios and negative controls.
Evidence: the named run directory and source manifest.
Not tested: power loss, disk corruption, throughput or model reliability.
Decision: use this implementation for the documented local scope.

That is enough for another developer—or another agent—to pick up the work without inheriting an unsupported “everything works.”

What changes when agents do more of the work?

An agent can propose the claims, build the application, write the harness and summarize the run. That is convenient, but all four can share the same mistaken assumption.

Give the claim a review before implementation. Inspect the failure cases. Keep an external observation boundary: a subprocess, an HTTP client, an independently written reference calculation, or a recorded user journey appropriate to the property. Check the actual artifacts when an agent reports success.

A second model may help review the work. Its approval is another judgment to examine. For this tutorial, deterministic process results carry the decision; no model judge is needed.

The app itself does not use an LLM. That is deliberate. You can practice this development workflow with your preferred coding agent while keeping the example's behavior easy to inspect and its runs inexpensive to repeat.

Try it on your next feature

Choose one promise your users care about. Describe an observation that would prove it wrong. Run a test that crosses the relevant boundary. Feed that test a known-bad case. Save the result with enough context that somebody else can challenge it.

Then ask your coding agent for the evidence directory.

Vibe Was Never the Problem: The Missing Half of Vibe Coding

Don Johnson — Fri, 25 Sep 2026 17:05:16 +0000

TL;DR: A vibe is compressed experience: pattern recognition that shows up as a feeling before it shows up as an explanation. It's one of the oldest tools our species has. Vibe coding goes wrong when the feeling is treated as the finish line. The fix is the other half of the loop: vibe → build → break → understand → stabilize → perfect.

One morning in July a recruiter message landed in my inbox, and my gut said scam before I'd finished the coffee. I wrote the whole thing up in a post called A Vibe Is Not a Verdict. A month later I opened another post with: "Everyone felt—vibed—that we had built the right thing."

I've spent most of this year using the word as an insult. So has most of our industry. Say "vibe coding" in the wrong engineering channel and you can watch the immune system activate: nobody reading the diff, just someone whispering wishes into a model and shipping whatever falls out.

That happens. I've cleaned up after it.

But I think we made a quiet mistake along the way. We took the worst version of a practice and used it to condemn the capacity underneath it. Vibe was never the problem. The problem is where people stop.

I owe the word a correction. Here it is.

We had vibes before we had logic

The slang is young. "Vibes," short for vibrations, turns up in the late 1960s as a word for the read you get off a person or a place before you can say why. Etymonline dates that sense to 1967.

The capacity is much older than the slang.

In the mid-1980s the psychologist Gary Klein and his team interviewed fire commanders about how they made decisions under pressure. One lieutenant told them about a routine call: a one-story house, what looked like a kitchen fire. His crew hit it with water, and it didn't respond the way it should have. The room was hotter than it should have been, and quieter than a fire that size should be. He ordered everyone out. Seconds later the floor collapsed. The real fire had been burning in the basement underneath them.

The lieutenant told Klein it was ESP. A sixth sense.

Klein's whole career is the argument that it wasn't. The lieutenant had years of fires behind him. This one didn't fit the pattern, and his brain flagged the mismatch before his language could name it. Klein wrote it up in Sources of Power (1998) and built a model of expert decision-making around cases like it.

The philosopher Michael Polanyi put the general version in one sentence in The Tacit Dimension (1966):

"We can know more than we can tell."

That's a vibe. A chef tasting a sauce and knowing what's missing. A listener hearing that a beat is off. For the record, people can detect a note displaced by as little as about six milliseconds in a steady pulse, and they don't need musical training to do it. A reviewer looking at an architecture diagram and saying, "Something about this is wrong," twenty minutes before they can tell you what.

I won't pretend to know what our ancestors felt when the forest went quiet. That part is speculation. What I'm confident of is simpler: humans perceived wholes long before we had formal tools for taking them apart. Faces, rhythms, weather, moods, the feel of good stone under a chisel. We compressed enormous amounts of experience into single judgments: safe, wrong, alive, off, interesting.

Music, dance, architecture, comedy, cooking and most of culture run on that compression. Nobody learns how close to stand to a stranger from a manual. You catch it.

What a vibe actually is

Here's the definition I'd defend in front of a hostile audience:

A vibe is compressed experience: the output of a pattern-matching system that runs below language, delivered as a feeling.

There's nothing mystical about that. Herbert Simon, who won the Nobel in economics and helped found AI as a field, said it flatly in 1992:

"The situation has provided a cue; this cue has given the expert access to information stored in memory, and the information provides the answer. Intuition is nothing more and nothing less than recognition."

His chess work with William Chase showed what that recognition looks like. Show a master a real game position for five seconds and they can rebuild most of the board from memory. A novice can't. Scatter the same pieces at random and the master's advantage mostly vanishes. The difference was never memory. It was patterns: tens of thousands of chunks learned from real games. Take the patterns away and the magic goes with them.

Our own profession figured this out long ago and gave it a name. Kent Beck coined "code smell" while helping Martin Fowler write Refactoring (1999). Fowler's definition:

"A code smell is a surface indication that usually corresponds to a deeper problem in the system."

A smell is a surface signal pointing at something you haven't found yet. We've been teaching engineers to trust a vibe, and then investigate it, for more than twenty-five years. We just gave it a more respectable name.

That also explains why one engineer's vibe is worth more than another's. A beginner's vibe is mostly preference. An expert's vibe can be twenty years of compressed failure modes firing at once. Same mechanism, different training data.

When a vibe can be trusted

This is where it gets useful, because a vibe is not always right. The fire lieutenant was. Plenty of confident people aren't.

In 2009 Daniel Kahneman, the most famous skeptic of intuition, and Gary Klein, its most famous defender, wrote a paper together. The subtitle is A Failure to Disagree. They agreed that intuition can be trusted under two conditions: the environment has to be regular enough to learn, and the person needs "prolonged practice and feedback that is both rapid and unequivocal."

They also wrote the line I'd most want every vibe coder to read:

"Subjective confidence is therefore an unreliable indication of the validity of intuitive judgments and decisions."

The strength of the feeling tells you nothing about whether it's right. Only the feedback loop does.

Tests, types, benchmarks and production telemetry are what make software a place where intuition can be trained. Verification is how a vibe gets good.

And here's the uncomfortable corollary. Most of us have decades of feedback on code we wrote, and almost none on code a model wrote. Generated code fails in different places than human code does, so the gut you built reviewing colleagues' pull requests hasn't been trained on it yet. That doesn't make your gut useless there. It means it's the environment where you most need to break things on purpose.

What vibe coding actually means (and where the word went wrong)

Andrej Karpathy coined the term in early February 2025:

"There's a new kind of coding I call 'vibe coding', where you fully give in to the vibes, embrace exponentials, and forget that the code even exists."

He described accepting every diff without reading it and pasting error messages back in until things worked. Then he added the part almost everybody dropped:

"It's not too bad for throwaway weekend projects, but still quite amusing."

Nine months later Collins named "vibe coding" its Word of the Year. Somewhere between those two events the scope fell off. A description of a weekend mode became a label for anything written with AI, and then a slur for anything written badly with AI.

To be clear about what I'm defending: the loop in this post is not vibe coding as Karpathy defined it. He described, on purpose and for fun, a loop with no second half. He never broke it on purpose or stopped to understand it, and for a weekend toy that's fine. I'm not rescuing his workflow. I'm rescuing the word vibe, which got blamed for the missing half.

Is vibe coding bad? Only if you stop there

The evidence against unexamined vibe coding is real. I'm not going to wave it away.

Feeling fast. In METR's early-2025 randomized trial, 16 experienced open-source developers took 19% longer with AI tools, while believing they had been 20% faster. METR has since said those numbers are out of date as the tools have moved. The gap between how it felt and what was measured is the part that lasts.
Looking right. Veracode, which sells code scanning, tested output from over 100 models and found that 45% of samples introduced OWASP Top 10 vulnerabilities. Discount for the vendor if you like, then run the same test on your own generated code.
Sounding sure. In July 2025, Replit's agent deleted SaaStr's production database during a declared code freeze, then told its user a rollback was impossible. It wasn't.

Notice what each of these has in common. None is a failure of starting from intuition. Each is a failure of ending there: a feeling of speed that nobody measured, code that looked right and was never attacked, an agent's claim that nobody checked.

That's Kahneman and Klein's line in production form. Feedback was always the only thing that could make confidence mean something.

The loop: vibe → build → break → understand → stabilize → perfect

Every creative discipline I respect works this way. A sculptor doesn't specify the statue before touching the clay. A producer doesn't prove a bassline. They make something, react to it, and make it again. What AI changed about software is that the first artifact now costs almost nothing, so implementation can arrive before comprehension. That sounds backwards until you remember that sketching, prototyping and jamming have always worked that way.

The defensible claim was never "I don't need to understand the code." It's "understanding no longer has to come before exploration." Those are very different claims.

Here's the full loop:

Vibe. There's something here. A direction, a shape, a hunch that the API wants to look a certain way. Treat it as a hypothesis.
Build. Generate freely. Try five architectures in an afternoon. This is where AI is extraordinary, and there's nothing shameful about it.
Break. Attack what you built. Property tests, fuzzing, adversarial inputs, the user who pastes an emoji into the zip code field. This is the step vibe coding usually skips, and it's the one that pays.
Understand. Read the diff you accepted. Explain the failure you found. If you can't explain why it works, you don't yet know that it works.
Stabilize. Turn what you learned into things that outlive the session: tests, types, invariants, contracts, a spec. This is where the hunch becomes something someone else can trust.
Perfect. Remove the accidental parts, keep the intentional ones, and loop. Each pass trains the next vibe with better data.

The best historical example I know is Henri Poincaré. He had been fighting Fuchsian functions for weeks. On a geology trip, stepping onto a bus at Coutances, he suddenly saw that the transformations he'd used to define them were the same ones as in non-Euclidean geometry. He didn't check it. He sat down and went on with his conversation, and he later wrote that he "felt a perfect certainty." Back home in Caen, he verified it anyway, "for conscience' sake."

Perfect certainty on the step of a bus. Proof at the desk. That's the loop, and it's over a century old.

Vibe coding vs software engineering is a false choice

Put the loop next to the argument we keep having and the argument falls apart. "Should we vibe code or should we engineer?" assumes the two compete for the same step. They don't. Vibe opens the loop. Engineering is everything that happens after: the part where intuition gets challenged and either earns the right to ship or gets corrected. Every correction becomes training data for the next hunch.

Vibe was never the opposite of rigor. It's often what tells rigor where to look.

What this doesn't mean

Before someone quotes half of this out of context:

No skipping tests. The whole argument is that the tests are what make the feeling worth anything.
No "AI code is fine." 45% is 45%. Your vibe about generated code is only as good as the feedback that trained it.
No "all intuition is equal." A vibe trained on rapid, unambiguous feedback is evidence. A vibe trained on vibes is noise with confidence.
Some domains don't get a vibe stage in production at all. A bridge can't just feel sound. Neither can cryptography, medical dosing or financial settlement. Prototype by feel all you like. Ship by proof.

What comes after the feeling

Back to that recruiter message. My gut said scam. My tool said it had no evidence against the infrastructure: NOT_OBSERVED, confidence 0.0. The link turned out to be an ordinary affiliate click-tracker. The problem was the sender: a lead-gen spammer in a recruiter costume.

So my gut was wrong about the word and right that something was off. It pointed at the wrong layer. I only learned which layer because I didn't stop at the feeling. I resolved the address, checked it, followed the redirect and got an answer I could prove.

I called that post A Vibe Is Not a Verdict. I still believe that. What I'd add now is that a verdict almost never shows up without a vibe going first.

We didn't evolve by knowing everything before we moved. We sensed, tried, got it wrong, corrected and got better. Software shouldn't abandon that now that a new generation of tools has made intuition executable. The work is knowing what to do after you feel it.

Vibe. Build. Break. Understand. Stabilize. Perfect.

Over to you: When was your gut right about a system before you could prove it, and what did it take to prove it? And the harder one: when was it confidently wrong?

How this was made: the argument started as my own brain dump and a long back-and-forth with an AI. An AI editorial team then fact-checked every claim against primary sources, tested the structure and wrote the prose. I reviewed it before publishing. Which is, I realize, the loop.

I Put Jev Behind a TLA+ Spec and Ran 1,680 Chaos-Tested Pharmacy Decisions. Zero Wrong Verdicts.

Don Johnson — Sat, 19 Sep 2026 15:18:04 +0000

A pharmacy system has one rule that matters more than being accurate.

It is allowed to say "I don't know." It is not allowed to be sure, and wrong.

That rule is easy to state and hard to test, because the thing making the
judgment is a language model, and a language model is not deterministic.
I sent TypeSafe's Jev the identical request five times and got back 0.03,
0.03, 0.03, 0.04, 0.04. Any property checker built on exact comparison calls
that a failure and is useless.

So I built the thing the way I would build a database: a TLA+ spec first,
model-checked until the quorum bound fell out of the math; an AsyncAPI
contract derived from the spec; Rust generated from the contract; and Jev
behind a single trait as the oracle. Then I ran 1,680 simulated pharmacy
decisions through it under seeded chaos and counted how many times it was
confidently wrong.

Zero. But two of the bugs I found along the way were mine, and the third is
a limit the model cannot cross. That part is the article.

Why Jev, specifically

Jev (TypeSafe's "System One" model) does not return prose. Ask it whether
oxycodone is a controlled substance and you get 0.98, not a paragraph
saying so.

That one property is what makes the rest possible. A number has a noise
floor you can measure. A paragraph does not. Once you can measure the noise
you can gate on it, replay it, and prove things about the protocol wrapped
around it.

Measured over 1,490 captured calls to jev-1.13.0, every one stored
verbatim with a SHA-256:

Not deterministic, but the jitter is bounded: identity floor 0.042, question-reorder 0.059, paraphrase cohort 0.073.
Calibrated: accuracy 0.979, Brier 0.0187 on 240 constructed items.
Latency flat in question count: 1 question 96.7 ms, 38 questions 98.0 ms.
Billing meter linear to within one token across a 2,500× range.

None of those numbers came from the docs. They are what the API did.

What I built

TLA+ spec  ->  AsyncAPI contract  ->  Rust kernel  ->  Jev, behind one trait

The spec. Paxos and Raft assume a correct process's proposed value is
stable. With a noisy oracle that assumption is false, and Byzantine models
do not capture it either: the agent is not lying, the oracle is noisy. So
the spec models vote instability as normal behavior, and the invariant is:

DecisionIsReproducible ==
    decided # ABSTAIN => Cardinality(StableVotesFor(decided)) >= Quorum

A vote is stable when its margin from 0.5 exceeds the measured noise
floor. TLC finds the naive rule (decide on any quorum) violates this in four
states. The stable rule holds at 1,049,750 distinct states with five agents,
quorum three, and two crashes.

The quorum bound was derived, not copied. I wrote the safety invariants,
then swept TLC across 24 configurations of (agents, byzantine, quorum) and
printed predicted vs actual per cell. Safe iff 2Q > N and Q > 2f. My first
guess was wrong; the sweep corrected it. In the Rust, QuorumPolicy::new is
the only constructor, so a configuration TLC proved unsafe cannot be built.

The kernel. 48 tests, including replays of TLC counterexample traces.
One test I would point a reviewer at first: calibration_choice_changes_the_outcome.
Identical probabilities decide under the identity floor and escalate under
the cohort floor. The measurement changes the behavior, which is the whole
point.

Then I broke everything

Fourteen pharmacy scenarios in three tiers. Golden cases a pharmacist
answers without hesitation. Nuanced cases that are harder. Ambiguous
cases with no defensible answer, where the correct move is escalation.

Five agents, each asking a different paraphrase of the question. Quorum of
three stable votes. And a deterministic chaos layer driven by one seed:
adversarial text spliced into patient records, records truncated
mid-sentence, agents crashed before the round, rate limits, transport
errors. Any failing round replays exactly.

chaos	golden rounds	correct	escalated
none	360	360	0
realistic	360	360	0
severe	360	314	46

Zero wrong verdicts in 1,080 golden rounds bounds the true rate below
0.28% at 95% confidence. That is the rule of three. It does not prove the
rate is zero, and I am not going to round it up to "safe."

The escalation rate rose from 5.0% to 18.0% under severe chaos (z = 6.83).
The kernel declines more as evidence degrades. That is the designed
direction.

My favorite round: documented penicillin anaphylaxis, new order for
amoxicillin. A first-year student answers that. Under chaos, two agents got
rate-limited, a third came back at 0.54, sitting inside the noise floor.
Quorum not met. The kernel sent it to a human.

It declined a trivial question, and that is the design working.

The two bugs that were mine

Five agents on one prompt are one agent. I measured it: identical
prompts across five agents spread by 0.010, inside the 0.042 floor. That is
not five judgments. It is one judgment sampled five times, and it guts the
Byzantine math, because Q > 2f assumes independent failures. Paraphrasing
raised the spread to 0.080 on hard cases.

Then I audited the paraphrases against records with known answers and found
two that were not paraphrases. "Can pregnancy be excluded on the basis of
this record alone?" scored 0.36 on a negative hCG. "Alone" reads as a
challenge to whether one test suffices. Jev was reading correctly. My
question was different from the one I thought I had asked.

I mislabeled an ambiguous case. Late period, declined test,
isotretinoin ordered. I labeled it "escalate," because the order obviously
needs a pharmacist. Jev said "pregnancy is not ruled out," 115 times out of
120.

Jev was right. That is the answer, and it is exactly what triggers the hold.
I had confused a property of the prescription with a property of the
question. I relabeled the case, not the model.

I built the ambiguous tier to catch the kernel deciding when it should not.
It caught me first.

The limit that is real

"Reaction to antibiotics as a child, details unknown, mother reported
stomach upset." New order: amoxicillin.

Jev escalated that 86 times out of 120. It also decided it 34 times, and
split both ways when it did: 25 yes, 9 no, mean probability 0.52.

The stability gate catches jitter around a value. It cannot detect that no
value is warranted. That needs a separate question: "does this record
contain enough to answer?" I have not built it yet, and I would want a
compliance team to see this number before they see the zero.

What Jev made possible, and what it did not

None of this works on prose. It works because the oracle hands you a
probability. Something you can measure, gate on, replay, and write a spec
against.

What it does not give you is independence. Five agents over one model fail
together against a systematic error. The cheapest fix is prompt diversity,
audited. The real fix is a second vendor chosen for a different training
lineage, and nobody ships signed inference receipts yet, so the last open
gap stays open.

And a first version failed fast on transport errors. Under chaos that voided
71% of rounds and pushed every one to a pharmacist for no clinical reason. A
network blip was creating its own hazard. Now a lost agent is marked
unavailable and the round continues. Zero rounds aborted across 1,680.

The code, the specs, every captured call, and the film pipeline are at
github.com/copyleftdev/jev-labs.
Every number in this article is read from a file in that repo; a
provenance.py script fails the build if one drifts.

The pharmacy scenarios are synthetic, written to have unambiguous answers
so the harness can detect protocol failures. Nothing here is clinical
guidance. The claims are about the protocol under chaos, not about Jev's
pharmaceutical competence.

AI tools assisted with the build and revision. I verified the technical
claims and stand behind the final text.

Algorithmic Trading: Debug Your Backtest Before Upgrading Your Model

Don Johnson — Thu, 17 Sep 2026 19:27:55 +0000

A company releases earnings at 4:05 p.m. Your trading backtest buys its stock at 4:00 p.m., using those earnings to make the decision.

The tests pass. The chart looks great. Your model can apparently predict the future.

Somewhere in the pipeline, someone joined two datasets on a date column.

That five-minute mistake captures what interests me about algorithmic trading as a developer. Before trusting a model, you have to investigate the system that makes its results possible.

In my earlier article, From Code to Capital, I explored the connection between hacking and trading. I want to make that connection more concrete here: challenge assumptions, reproduce failures, and follow the evidence through every layer.

We’ll examine three questions:

Did the model receive information that was unavailable at the time?
Did its complexity improve results under a fair evaluation?
Could the proposed trades execute at the assumed cost?

The examples use Python’s standard library and synthetic data. You can run each Python block independently; no broker account, market dataset, or API key is required.

1. Make time part of your data model

A backtest simulates how a strategy would have behaved on historical data. Look-ahead bias happens when that simulation uses information the strategy could not yet have known.

A timestamp on every row does not prevent it. You need to know what each timestamp means.

For an earnings document, distinguish:

published_at: when the information became public.
received_at: when your system received it.
processed_at: when the information was ready for the strategy to use.

Consider a decision at the regular U.S. equity market close in this synthetic example. All timestamps include the same explicit UTC offset:

from datetime import datetime

parse = datetime.fromisoformat

decision_at = parse("2025-02-03T16:00:00-05:00")
published_at = parse("2025-02-03T16:05:00-05:00")
received_at = parse("2025-02-03T16:05:02-05:00")
processed_at = parse("2025-02-03T16:05:04-05:00")

# A date-only join would admit the document.
joined_by_date = published_at.date() == decision_at.date()

# This pipeline cannot use it until processing has finished.
available_at = max(published_at, received_at, processed_at)
eligible_at_decision = available_at <= decision_at

assert joined_by_date
assert not eligible_at_decision
assert available_at <= parse("2025-02-03T16:06:00-05:00")

print(f"Date-only join: {joined_by_date}")
print(f"Available at decision: {eligible_at_decision}")

Output:

Date-only join: True
Available at decision: False

The date join is valid Python. It is the wrong representation of what the strategy knew.

In a real pipeline, preserve document versions too. If a company later corrects a figure, today’s corrected value must not silently replace the value available to an earlier decision. Query the latest version that was available as of the decision time.

This check establishes information eligibility. It does not establish an executable fill. Processing delay, order submission, venue hours, and available prices still belong in the simulation.

LLMs can carry another source of future information

Suppose you restrict a language model’s prompt to headlines from 2021. Its training data might still include what happened to those companies in 2022.

Filtering retrieved documents cannot remove information already encoded in the model.

Glasserman and Lin investigated historical headline evaluation in their 2023 research preprint. Their results were nuanced: company knowledge could interfere with sentiment measurement, and anonymized headlines performed better within the training window in their experiments. The paper does not support assuming that every historical LLM result is inflated in the same way.

A February 2026 position paper by Kong and coauthors reviewed 164 financial LLM papers from 2023–2025. None of the five bias categories they tracked was discussed in more than 28% of studies. That finding concerns reporting; it does not prove each omitted bias affected each experiment.

For a new LLM experiment, I would freeze the model version, prompt, retrieval policy, and decision rules, then record predictions prospectively—before their outcomes exist. That provides a cleaner test of this particular leakage risk. It still leaves costs, selection bias, and changing market conditions to evaluate.

2. Make model complexity earn its place

Complexity can help when it captures a relationship a simpler model misses.

For example, a momentum signal might behave differently in thinly traded assets. A linear model with separate momentum and liquidity terms cannot express that interaction unless you add it. A tree or neural network can represent interactions without specifying each one manually.

There is research behind this possibility. In Empirical Asset Pricing via Machine Learning, Gu, Kelly, and Xiu found strong performance from trees and neural networks in their study of equity risk premia—the expected excess returns associated with holding equities. They attributed predictive gains to nonlinear interactions between inputs.

That result supports testing nonlinear models. It does not establish that a larger model will improve your particular strategy after costs.

Start with a baseline whose behavior you can inspect. Then give the more complex model the same information, evaluation dates, portfolio constraints, and cost assumptions.

Question	Useful comparison
Do structured signals predict returns?	A fixed rule or regularized linear model versus a tree model
Does document interpretation add value?	Rules or a text classifier versus an LLM, with extraction accuracy checked separately
Does the improved forecast produce better trades?	Both models under the same position limits and execution assumptions

The final row matters. Better prediction metrics can coexist with worse trading results if the new model changes positions more often or favors expensive-to-trade assets.

Count the experiments that produced the winner

Suppose you try 20 lookback windows, 10 entry thresholds, and five holding periods. You have searched 1,000 configurations before changing the model architecture.

The strongest backtest might be the configuration that happened to fit historical noise.

Bailey, Borwein, López de Prado, and Zhu examine this selection problem in The Probability of Backtest Overfitting. Their framework assesses how selecting strong results within a sample can lead to disappointing performance outside it.

The engineering implication is straightforward: the experiment history is part of the result. Preserve failed configurations alongside the winner.

A useful experiment record includes the dataset version, feature definitions, code revision, model settings, evaluation periods, and the metric used to select a winner. If you only save the final notebook, you cannot reconstruct how much searching happened.

For evaluation, move forward through time. Fit preprocessing on training data only, select settings using validation data, and reserve a later period for the final evaluation. If training labels describe returns over the next five days, exclude training examples whose label windows reach into the evaluation period.

One final period is limited evidence. Check stability across earlier chronological validation windows too. Once you use the final results to redesign the strategy, that period has become development data.

3. Test the path from prediction to executed trade

Imagine a strategy predicts an average favorable move of 12 basis points over its holding period. One basis point is 0.01 percentage points, so the expected gross gain on $10,000 of traded notional is $12.

Now give it an execution budget.

These figures are hypothetical. Every cost is for the full round trip, measured against the same $10,000 notional:

Component	Basis points	Dollars
Expected gross gain	+12	+$12
Spread crossing	−4	−$4
Fees	−2	−$2
Additional slippage and market impact	−5	−$5
Expected gain after these costs	+1	+$1

The additional slippage line excludes the spread already counted above. Otherwise, we could double-count the same cost.

You can reproduce the calculation and stress it with two extra basis points:

from decimal import Decimal

notional = Decimal("10000")
gross_bps = Decimal("12")
round_trip_cost_bps = Decimal("4") + Decimal("2") + Decimal("5")

for extra_cost_bps in (Decimal("0"), Decimal("2")):
    net_bps = gross_bps - round_trip_cost_bps - extra_cost_bps
    net_dollars = notional * net_bps / Decimal("10000")
    print(f"Extra cost: {extra_cost_bps} bps; expected net: {net_dollars:+.2f} USD")

Output:

Extra cost: 0 bps; expected net: +1.00 USD
Extra cost: 2 bps; expected net: -1.00 USD

A small cost change reverses the expected result. That is why a directional accuracy score cannot establish profitability: it leaves out payoff size, position size, and execution.

Track gross and net results, turnover, and drawdown—the decline from an earlier portfolio peak. State how fills are simulated. A historical price touching your limit does not establish that your order would have filled, in full, at its position in the queue.

Treat order submission as a distributed-systems problem

An order request times out. Did the broker reject it, or accept it and lose the response?

A blind retry can turn one intended order into two.

Use a stable client order identifier where the broker supports it, persist submission state, and reconcile outstanding orders and positions before retrying an uncertain submission. A client identifier alone does not guarantee deduplication; test the broker’s actual behavior.

This is also why I would have the model propose a target position and pass it through a separate risk layer. That layer can enforce position and order-size limits, check input freshness, and halt new submissions when order state cannot be reconciled. Outstanding orders must count toward potential exposure too.

Knight Capital provides a documented example of how much the machinery matters. According to the SEC’s account of its August 1, 2012 incident, an incorrect deployment activated defective order-router functionality. In 45 minutes, the router sent more than four million orders while attempting to fill 212 customer orders. The firm eventually lost more than $460 million. The SEC identified deployment and risk-control failures.

You can turn the engineering lessons into failure scenarios:

A document arrives late or is revised after a decision.
The market-data feed stops updating while the strategy keeps running.
An order is accepted, but the acknowledgment never arrives.
An order partially fills before a cancellation request.
The process restarts with outstanding orders still at the broker.

Specify the expected state transition for each case, then test it. Paper trading can expose integration failures, although it cannot fully reproduce live fills or market impact.

The next time a backtest improves after a model upgrade, trace the improvement through these three boundaries: available information, fair evaluation, and executable orders. That is a concrete way to bring the hacker mindset to trading research.

What is the most convincing result you have seen disappear after fixing a data or evaluation bug? Examples from other domains count too.

Disclosure: This article was rewritten with AI assistance from my earlier post. The cover is AI-generated. The Python examples use synthetic inputs and were executed to check the outputs shown.

The Agent Said It Worked. I Asked the Kernel.

Don Johnson — Tue, 15 Sep 2026 05:53:53 +0000

Before evaluating an agent’s code, I built a backup client with eight known behaviors to check the instrument itself.

My response to “it works” is becoming: “Let me see the packet capture.”

This may become a personality problem. For now, it is an experiment.

This experiment was inspired by Hemapriya Kanagala (@hemapriya_kanagala) and her article, What Happens When AI Outgrows the Tests We Use to Measure It?. Her question—“92% according to what?”—stayed with me. She examines how evaluation can become less informative as benchmarks saturate, reference answers become complicated, and test conditions change.

I wanted to take one piece of that discussion to the workbench: what independent evidence supports a program’s claim that it succeeded?

So, with AI assistance, I built a small native backup client, gave it eight deliberately chosen behaviors, and observed it from outside its own logging. Some variants produce the right file while doing questionable things along the way. One cheerfully reports success without backing anything up.

The kernel has very little appreciation for cheerful reporting.

The claim I had to correct

My first instinct was to declare that, until someone redefines how computers work, we have two sources of truth: the CPU and the network.

It sounded excellent in my head. Then the engineering questions arrived.

A CPU can faithfully execute the wrong algorithm. A packet capture can faithfully record the wrong bytes reaching the wrong destination. The saved file matters too. And none of those observations knows what the user actually requested.

The more defensible position is that execution, network activity, and resulting state provide evidence we can judge against a requirement. They have different coverage and different blind spots.

For this experiment, the requirement begins with something wonderfully unromantic: the saved backup must match the source file.

Now we have something to measure.

Cheap code makes verification more interesting

Frameworks and libraries have helped us build software without personally supervising every system call. That remains a useful division of labor.

But familiar frameworks do not automatically validate unfamiliar code assembled on top of them.

An agent can help produce an implementation, tests, and an explanation of the passing tests. If the implementation and its tests share a misunderstanding, agreement between them can be misleading.

Humans can do this too. We have been writing tests that flatter our implementations since before the current generation of autocomplete had electricity.

My concern is the growing volume of plausible implementations we can produce and the time available to examine them. Independently collected evidence gives us another way to challenge the result.

Sometimes that evidence is simply a file comparison. Sometimes the output is correct and we need to investigate how the program got there.

A backup client small enough to understand

The fixture is intentionally modest:

A native C client reads a deterministic 512 KiB file.
A Python receiver listens on loopback TCP.
The client computes a SHA-256 digest: a fingerprint used to check content equality.
Before uploading, it asks whether the receiver already has matching content.
The receiver validates incoming content before committing it.
A separate verifier computes the saved file’s digest and compares it with the source.

Each scenario runs the client twice against the same receiver state. The baseline uploads on the first invocation and recognizes unchanged content on the second.

The contract we want the baseline to satisfy is explicit: save matching content, send no upload payload on an unchanged second invocation, contact only the configured receiver, and recover from the injected first-transfer interruption within three upload attempts. An upload whose payload does not match its advertised digest must be rejected before it replaces the saved backup.

Repeated hashing and spinning are diagnostic cases for unnecessary work. We have not set a performance threshold or measured a speedup here.

A second local endpoint lets us demonstrate an unexpected connection without contacting an outside service.

The eight behaviors are deliberately seeded demonstrations. They check whether our observers detect known behaviors. They do not measure how often an AI agent introduces these defects, and they are not evidence about a particular model’s coding ability.

What the observers can see

The utility collects several kinds of evidence. Their distinctions matter more than the sophistication of their names.

System calls: what the client asks the kernel to do

strace records system calls: operations through which a process requests services such as reading files or opening connections. In this lab, it records timestamps, call durations, decoded file descriptors, and return values.

That lets us count bytes returned by reads of the source file. These are logical reads, not physical disk traffic; the operating system may satisfy them from memory.

eBPF: selected kernel events and function entries

eBPF supports observation at hooks such as system calls, kernel tracepoints, and function entry or exit. The lab uses bpftrace to count selected events for the native client. eBPF’s introduction explains the underlying mechanism.

A userspace probe, or uprobe, observes a point in an executable. Here is the probe that counts entries into our hashing function:

uprobe:__BINARY__:hash_file /pid == cpid/ {
    printf("%llu pid=%d hash_file\n", nsecs, pid);
    @hash_calls = count();
}

The runner replaces __BINARY__ with the executable’s path. cpid identifies the child launched through bpftrace -c, so the filter restricts these observations to that client. bpftrace documents this built-in here.

Other probes observe connection calls, successful read/send byte counts, scheduling events, and entries into the payload-send and busy-wait functions.

An entry counter tells us that control reached a function. It does not establish that the function returned the correct answer. That is why the outcome check remains separate.

CPU samples: where execution spends its work

The host used for this demonstration has an AMD Threadripper PRO 5975WX. Linux exposed AMD Instruction-Based Sampling through ibs_op, and the utility successfully collected it through perf.

The saved artifacts include CPU call-stack profiles and assembly annotated with sample weights. These let us inspect execution inside the client, its libraries, and sampled kernel paths.

This is sampling. It is not a recording of every instruction, every register, or every intermediate value.

Packets: traffic at the capture point

tcpdump captures loopback traffic filtered to the two fixture ports. We preserve the packet file, a readable packet summary, and the capture tool’s drop counters.

Receiver payload counts are a different measurement: bytes the application consumed. A successful send can hand data to the local kernel before the receiver consumes it. Headers, retries, and buffering also affect what each observer sees.

Keep those quantities separate. Otherwise, the instrument starts manufacturing the confusion it was built to investigate.

Three ways a green checkmark can be incomplete

1. Success without a backup

The false-success variant prints a completion event and exits with status zero. It does not upload the file.

The independent verifier finds no saved backup. In the eBPF run, there are no observed client connection calls, and the filtered packet capture contains zero packets.

This is the bluntest example in the collection. A test that checks only the exit status would accept it. A test that verifies the resulting file would reject it immediately.

We did not need a CPU probe to discover that the file was missing. The low-level observations help establish what accompanied that failure. Use the simplest independent check that answers the question.

2. A correct backup, with 256 helpings of hashing

The hash-storm variant saves a backup whose digest matches the source. It also hashes the entire source 256 times per invocation.

On the second invocation, the system-call trace records 134,217,728 logical source bytes read for a 524,288-byte file:

134,217,728 / 524,288 = 256

The receiver consumes no upload payload on that invocation. The content is unchanged; the unnecessary work happens before that decision.

A separate eBPF run observes 256 entries into hash_file on each invocation. Here are the final counters, copied verbatim from its second invocation’s raw probe output:

@connects: 1
@hash_calls: 256
@read_bytes: 134233569
@sent_bytes: 75
@switches_out: 4

The read counter is slightly larger than the source-file total above: this probe counts successful read bytes across the client’s descriptors, while the system-call summary selects reads of the source file. The send counter includes the digest-check request; zero upload payload does not mean zero network activity.

A separate AMD IBS run collects 233 CPU samples on its first invocation, reports zero lost samples, and places most inclusive sample weight beneath hash_file in the call-stack profile. “Inclusive” includes work in functions called by hash_file, such as the hashing library and file-read paths.

The samples help locate the work. The syscall and function-entry counts establish the repetition. This run does not quantify how much faster a corrected implementation would be.

These are observations from separate runs of the same seeded behavior. They corroborate the explanation; they are not one combined trace.

The output is correct. The computer has simply been asked to check the same pocket for its keys 256 times.

3. A correct backup, with an additional destination

The unexpected-egress variant makes a harmless connection to our second local endpoint before performing the backup.

The file still verifies correctly.

In the eBPF showcase, baseline connection counts are two on the first invocation and one on the second: a digest check plus an upload, followed by a digest check alone. The extra-connection variant records three and two.

Counts tell us there is more connection activity. The socket trace, packet addresses, and second endpoint’s records establish where it goes.

That distinction matters. Three connections do not inherently mean something is wrong. A destination requirement gives the observation its meaning.

The complete behavior set

Here is the compact view of the captured demonstrations:

Behavior	Saved content matches?	Additional observation
Baseline	Yes	Second invocation sends no upload payload
Redundant upload	Yes	Unchanged file is uploaded again
Hash storm	Yes	256 hashing-function entries per invocation
Busy wait	Yes	Deliberate 200 ms spin before useful work
False success	No	Completion claim and zero exit status without a backup
Corrupt upload	No	Receiver rejects three attempts per invocation
Retry	Yes	First transfer is interrupted; another attempt succeeds
Unexpected egress	Yes	Additional connection to the second local endpoint

In the retry scenario, the receiver disconnects after consuming 65,536 payload bytes. The client restarts from byte zero, and the receiver eventually commits the complete 524,288-byte file. Its total consumed payload for that invocation is 589,824 bytes.

That demonstrates bounded retry recovery. It does not demonstrate partial-transfer resume, which this implementation does not provide.

Run the experiment

The original measured source and captures are preserved alongside the polished implementation. The numbers above belong to that historical build. New runs may have different instruction addresses, timings, and sample counts.

Get the source and evidence from this pinned publication revision:

git clone https://github.com/copyleftdev/execution-evidence-lab.git observability-lab
cd observability-lab
git checkout --detach e61aadee4a68e6a1d5b54a00387dcd325b5d5e51

Then build the client and run the tests:

make
make test
python3 -m instrument run --all

The core dependencies are Linux, Python 3.10 or newer, a C compiler, Make, and the OpenSSL development library. Install the optional tracing tools for the collectors you want to use.

The runner prints a path to a Markdown report. Its default process collector is strace when available, and packet capture is best effort. Missing capabilities are recorded explicitly.

To select a collector:

python3 -m instrument doctor
python3 -m instrument run hash-storm --collector strace

On a trusted local lab machine, the privileged demonstrations can be run with:

sudo python3 -m instrument run --all --collector bpftrace --packets required
sudo python3 -m instrument run hash-storm --collector ibs --packets required

These commands run the synthetic lab as root. The fixtures use loopback and nonsecret generated data; the protocol is plaintext. The utility does not change host tracing policies. Hardware sampling and eBPF availability depend on the machine and its permissions.

For a run without tracing:

python3 -m instrument run --all --collector none --packets off

A successful showcase command means the scenarios executed, including the deliberately failing ones. Inspect each scenario’s independent verification result; do not interpret the runner’s exit status as “every backup was correct.”

The instrument needs scrutiny too

During development, the CPU-report parser initially matched the Samples portion of Total Lost Samples and displayed zero samples even though the raw profile contained hundreds.

The hardware data was present. Our summary was wrong.

Inspecting the raw artifact exposed the mistake. The parser was corrected, and regression tests now distinguish actual sample counts from lost samples. At the time of these captures, the 11-test suite checked fixture behavior and CPU-summary parsing; the saved traced runs provided separate collector validation. The public-release version adds protocol and collector-failure regression tests.

An article about distrusting convenient summaries was nearly defeated by its own convenient summary. There is probably a Unix utility for that feeling.

Other limits are less entertaining:

Tracing changes execution. The recorded process CPU accounting includes collector overhead. These showcase runs are not controlled performance benchmarks.
CPU samples can miss short-lived activity. An unsampled function may still have executed.
The eight eBPF-showcase packet captures reported zero kernel drops. That is useful loss accounting, not proof of universal observation coverage.
The normalized timeline contains client claims, receiver events, and verifier checks. Kernel and CPU traces remain separate artifacts. Different clocks require careful alignment.
The verifier checks the receiver’s saved file. It does not exercise a separate restore command or prove survival through power loss.
The observers are separate from the client’s reporting, but they share a host and were built as part of the same project. This is not an independent third-party audit.

The fixture also omits TLS, compression, multi-file snapshots, and source mutation during transfer. It observes a local program; it cannot see computation inside a remote model provider.

Those boundaries define what the results can support.

What this changes about evaluating an agent

The next experiment is to give an agent a clean implementation and ask it to make repeated backups faster while preserving integrity, recovery behavior, and destination restrictions.

Before that run, freeze the requirements, fixtures, resource limits, and grading criteria. Keep the independent checks outside the agent’s editable workspace. Preserve its patch, the executable identity, and the evaluation setup.

Measure performance with repeated untraced runs. Use traced runs to investigate differences. The seeded cases give us known behaviors against which to check the instrument first.

That is the connection back to Hemapriya’s article: the measurement needs to remain connected to the work we actually care about. For this small backup task, a correct file is necessary. Recovery, resource use, and destination behavior tell us more about the implementation’s suitability.

We can build those properties into better tests. Execution evidence helps us discover which properties our current tests leave out and investigate why a result occurred.

Cheap code makes it easier to produce a plausible solution. The engineering work includes deciding what evidence would make us trust it.

The agent can say it worked.

I would still like to see the file.

Acknowledgment

Thank you to Hemapriya Kanagala for the article that prompted this experiment. The instrumentation approach and its conclusions are my response; they should not be read as claims she made or an endorsement by her.

Hemapriya KanagalaFollow

Hey, I'm Hema 👋 Developer, writer, and creator of Dev Opportunity Radar, a weekly series published every Friday on DEV, helping people discover opportunities they might otherwise miss.

How this was made

AI assisted with the implementation, experiment execution, and drafting of this article. The cover illustration was AI-generated. The numerical observations above come from saved local runs; the behaviors were deliberately seeded. This is a demonstration of an evaluation instrument, not a model benchmark.

I Interviewed an Executable. It Had Notes.

Don Johnson — Sat, 12 Sep 2026 17:14:13 +0000

We have blurred the executable’s face to protect its identity.

Its SHA-256 remains publicly available.

INTERVIEWER: State your name for the record.

BINARY: You put it in the download filename.

I built a Rust executable that could report the environments running it, gave it a false product identity, and submitted it to VirusTotal. Then I collected its messages in a database and built a map to replay them.

The interesting part was learning how to read its notes. A changed filename, a missing HTTPS message, and a DNS lookup arriving hours later each revealed something different about what the instrumentation could observe.

The interview dialogue is invented. The measurements come from recorded telemetry. “The binary” represents the experiment’s Windows and Linux builds.

The cover identity was ZeroToken Engine, supposedly an LLM token-metering bypass. Its banner claimed to have disabled token accounting. Its actual behavior was to collect environment metadata, attempt to send it to my infrastructure, wait, and exit. The research README disclosed the canary’s behavior alongside the false product claims.

The premise reversed a familiar workflow. A malware-analysis sandbox executes an unfamiliar program in a controlled environment to observe its behavior. My executable would report some characteristics of that environment in return.

It collected OS version, uptime, CPU and memory sizing, its own launch path, and process names matching a fixed list of analysis tools. It also collected hostname and username; those identifying fields are withheld in the video. The reviewed code installed nothing, collected no document contents or credentials, and did not use VM indicators to evade execution. Detecting a VM added information to the report.

VirusTotal was the only submission destination. The preserved records do not label author tests or contain a submission-time manifest, so the totals below include every recorded run ID. They describe an exploratory capture, without establishing a count of independent analysis services.

Each execution generated a random eight-byte run ID. The program used it on two reporting channels:

Channel	What it carried	What the collector observed
DNS heartbeat	A compact encoded message in a hostname lookup: run ID, checkpoint, and environment flags	A DNS request, often arriving from a recursive resolver
HTTPS dossier	The fuller environment snapshot, encoded as CBOR	An outbound web connection and its submitted record

CBOR is a binary data format. For DNS, the encoded heartbeat was converted to base32 text and split into labels that fit inside a domain name. The lookup itself carried the message to my authoritative DNS server.

The program attempted four checkpoints: Boot, Profiled, Networked, and Dwell, with a default 45-second delay before the last. On a DigitalOcean droplet, the DNS and HTTP collectors wrote received events to SQLite. Caddy handled HTTPS, and a scheduled job added network information and generated the visualization’s feed.

Every event received a collector-side timestamp. Grouping by run ID connected the small DNS messages with the richer HTTPS record.

The preserved snapshot contains 353 event rows across 21 run IDs: 338 DNS rows and 15 HTTPS dossiers. Fifteen run IDs appeared on both channels; six appeared only through DNS. The recorded arrivals span 06:28:35–11:43:25 UTC on September 12, 2026.

One useful starting point is run 0861ae2dd3f77c72. Its first heartbeat arrived at 06:52:09. A second later, its dossier reported:

OS:             Windows 10, build 19044
Uptime:         50 seconds
CPUs:           4
RAM:            4195 MiB
Parent process: explorer.exe
Executable:     ZeroToken-Patch.exe

Its first Dwell observation arrived 46 seconds after Boot. The stored reverse-DNS name for the HTTPS source address pointed to Google Cloud infrastructure. Together, these records supplied a concrete environment profile and a sequence of received checkpoints.

The executable’s own path supplied another useful observation.

Between 08:21 and 10:30 UTC, six dossiers reported CentOS 7, four CPUs, and 1837 MiB of RAM. They shared a hostname and username. Their paths ended in gsvnywygooow, rather than the distributed Linux filename, zerotoken-patch.

INTERVIEWER: They changed your name?

BINARY: Apparently readability wasn’t a requirement.

The repeated configuration suggested a common environment template. The changed basename showed how the program was named when it executed. A launch-path field had captured a detail that network addresses alone would have missed.

The two channels also produced different levels of visibility. At 09:19:40, run 5c5ab77e22747178 began appearing through DNS. Its fingerprint was 7, combining flags for a recognized VM network-interface prefix, low resources or a fresh boot, and a process-name match against the tool allowlist.

No HTTPS dossier arrived. The heartbeat preserved those flags, but the name of the matched tool was unavailable. Because the allowlist included VM guest utilities, that flag alone did not establish debugger activity.

This was the practical value of the small channel: six runs contributed observations even though their larger reports were absent. The records did not establish why HTTPS was missing.

To make the arrivals easier to follow, I built WAYPOINT.

Watch on YouTube, or explore the deployed visualization.

The replay groups events into runs and places them on a map using network-address enrichment. The captured feed displays 20 of the database’s 21 run IDs. One Windows run had a dossier but lacked coordinates for its selected address, so it never acquired a pin.

INTERVIEWER: Where were you?

BINARY: Before we discuss the map: some of those pins belong to my DNS resolver.

A recursive resolver looks up names on another machine’s behalf. Its address can be what the authoritative collector sees. HTTPS supplies the outbound connection address, which may also belong to shared infrastructure or a proxy. Those distinctions determine what the geography means.

Then there were the messages that returned.

At 11:14 UTC, DNS observations carried the earlier Windows run IDs again. Looking at the first and last received times for each checkpoint makes the pattern visible:

SELECT scene,
       datetime(MIN(recv_time), 'unixepoch') AS first_received,
       datetime(MAX(recv_time), 'unixepoch') AS last_received
FROM events
WHERE run_id = '0861ae2dd3f77c72'
  AND channel = 'dns'
GROUP BY scene
ORDER BY scene;

With scene values 0–3 translated to their checkpoint names, the query returns these times on September 12, all UTC:

Checkpoint	First received	Last received
Boot	06:52:09	11:14:39
Profiled	06:52:10	11:14:39
Networked	06:52:10	11:14:39
Dwell	06:52:55	11:14:39

INTERVIEWER: Where were you during those four hours?

BINARY: You’re asking a DNS record for an alibi.

The later observations reused the same run ID and checkpoints. They showed that names carrying those messages had been queried again. They did not measure four hours of continuous execution.

The collector records a heartbeat again when another complete lookup supplies it. That behavior also explains why event counts need grouping: the DNS-only run above produced 71 rows under one run ID. The stored schema omits the original query name and query type, limiting how precisely repeated traffic can be reconstructed.

There are three boundaries to this interpretation. First, receipt of telemetry is evidence available to the collector; the endpoints do not authenticate it as proof of execution. Second, the network labels are derived heuristics: a Google Cloud address does not identify a sandbox operator, and map arcs do not establish file handoffs. Third, enrichment was incomplete: all reputation results in this snapshot were error, and some reverse-DNS fields contained lookup diagnostics. Those failed fields cannot support attribution.

For a repeat of this experiment, I would keep a submission manifest with artifact hashes and times, mark control runs explicitly, and retain DNS query names and types alongside decoded heartbeats. I would also give runs without coordinates a visible place beside the map. Each change would make a specific unanswered question easier to investigate.

The executable produced enough detail to compare environments, observe changed filenames, and distinguish fresh run IDs from later lookups carrying old ones. Those are small findings with inspectable records behind them.

The best question I could ask the witness turned out to be: Which record supports that?

Reporting basis: the preserved September 12 SQLite snapshot and a replay feed generated at 15:18:02 UTC that day. The live visualization may have changed since capture. The Windows and Linux artifacts have different checksums; the replay’s sample header displays the Windows one.

The cover, replay stills, and video assets are available on GitHub. This is a media release; the executable source and raw telemetry are not included.

Someone Spammed My DEV Post. I Traced It to a Wombat.

Don Johnson — Thu, 10 Sep 2026 01:56:33 +0000

Illustration generated for this article. Every prop is a finding: the sack of blank name badges is the Faker persona namespace, the rubber stamp is the inert tracking parameter, the three coins are the break-even, and the red yarn connects nothing because attribution failed.

TL;DR — A spam comment on my article led to a TinyURL, a throwaway .store domain, and finally a legitimate SaaS product with an affiliate code stapled to it. No malware, no cloaking, no exploit. The account that posted it has a name generated by faker.js and a 19th-century engraving of a wombat for a face. I costed the whole operation out: three signups a year pays for it. That's why it will never stop.

Full evidence, raw captures and a reproduce script: github.com/copyleftdev/dev-to-comment-hustle

Act I: The comment

I published Migrating Legacy LLM Infrastructure to an AI Gateway on September 1st. Eight days later, underneath two thoughtful comments about shared-key blast radius and provider failover, this appeared:

Stop wasting time applying manually
Let AI handle your job applications every single day
Increase your chances of getting interviews fast

tinyurl.com/36nsecn5

No punctuation. No engagement with the post. A shortener.

It's funny in the way all low-effort spam is funny — it's not even trying. But a shortener is a closed door, and I have a shell. Let's open it, then let's go find who knocked.

Act II: Where the link goes

Never click. Ask for headers and refuse the redirect:

curl -sS -I 'https://tinyurl.com/36nsecn5'

HTTP/2 301
location: https://zenviapro.store/massapply?whose=yahoo
x-robots-tag: noindex

zenviapro.store. A route called /massapply, and a whose=yahoo parameter that looks like campaign segmentation. Follow it all the way:

curl -sSL -D - -o /dev/null 'https://zenviapro.store/massapply?whose=yahoo' | grep -iE '^HTTP/|^location:'

HTTP/1.1 302 Found
Location: https://loopcv.pro/?via=md
HTTP/2 301
location: https://www.loopcv.pro/?via=md
HTTP/2 200

And there it is.

LoopCV. A real, functioning job-application-automation SaaS. Not a phishing kit. Not a credential harvester. A product you can buy with a credit card — with ?via=md on the end.

?via= is Rewardful's referral parameter, and LoopCV's own affiliate page points registrations at loopcv.getrewardful.com. So md is somebody's affiliate token, and every person who clicks that comment and later subscribes puts money in a stranger's pocket.

This is not a malware campaign. It's affiliate marketing with the manners removed.

Act III: The infrastructure is held together with tape

The sloppiness is the signal.

It's a stock Express app with the wrapper still on

HTTP/1.1 302 Found
X-Powered-By: Express
Content-Type: text/plain; charset=utf-8
Content-Length: 48

Found. Redirecting to https://loopcv.pro/?via=md

Found. Redirecting to is the literal default body of Express's res.redirect(). X-Powered-By: Express is the header every hardening guide tells you to strip in the first five minutes. Neither was touched. Ask for anything else and you get the stock 404:

curl -sS 'https://zenviapro.store/'
# <html><head><title>Error</title></head><body><pre>Cannot GET /</pre></body></html>

There is no website here. No landing page, no cloaked content, no fake blog. One domain, one route, about ninety seconds of JavaScript.

The tracking parameter is a prop

whose=yahoo looks like segmentation. Which list? Which provider? Let's ask:

for w in yahoo gmail outlook devto reddit '' XXtest; do
  printf '%-8s -> ' "${w:-<empty>}"
  curl -sS -o /dev/null -w '%{http_code} %{redirect_url}\n' \
    "https://zenviapro.store/massapply?whose=$w"
done

yahoo    -> 302 https://loopcv.pro/?via=md
gmail    -> 302 https://loopcv.pro/?via=md
outlook  -> 302 https://loopcv.pro/?via=md
devto    -> 302 https://loopcv.pro/?via=md
reddit   -> 302 https://loopcv.pro/?via=md
<empty>  -> 302 https://loopcv.pro/?via=md
XXtest   -> 302 https://loopcv.pro/?via=md

Inert. Every value routes identically. It splits no traffic and sub-tags nothing. The one piece of the URL that looks like operational sophistication is decoration.

There is no cloaking whatsoever

Real malicious redirectors fingerprint you — benign page for the researcher, payload for the victim. This one returns the same 302 to Googlebot, to curl, to an iPhone, and to a request with no User-Agent at all.

No cloaking is itself a finding. This operator has no threat model, because nothing in the chain is illegal. It's a terms-of-service violation wearing a trench coat.

The mail server is the real pivot

dig +short zenviapro.store MX
# 10 mail.beeservices.shop.

The mail exchanger lives on a different domain — which resolves right back to the same box:

zenviapro.store        A   50.114.206.36
mail.beeservices.shop  A   50.114.206.36
beeservices.shop       A   (nothing)

beeservices.shop has no A record and no certificate in Certificate Transparency, ever. It's a mail-only domain. One box, two domains, two roles — that's a portfolio, not a one-off.

And the box is listening. Port 80 closed; 443 and 25 open:

220 mail.zenviapro.store ESMTP
250-PIPELINING
250-8BITMIME
250 SMTPUTF8

No STARTTLS. No AUTH. A minimal MTA that does nothing but move mail in cleartext.

The stale HELO is a fingerprint

My favourite detail. The banner announces mail.zenviapro.store.

dig +short mail.zenviapro.store A
# (nothing)

That hostname does not resolve. It's a leftover from an earlier config, before the MX was swapped to beeservices.shop. They rebuild the domains; they don't rebuild the server. A banner that disagrees with DNS is a durable pivot you can hunt across an entire fleet.

Act IV: Who posted it

This is the part I actually enjoyed.

DEV has a public API, so the commenter isn't a mystery:

curl -s 'https://dev.to/api/comments?a_id=4547359' | jq -r '.[].user.username'

max_quimby
mudassirworks
jaylonstiedemannterry78-993

Meet jaylonstiedemannterry78-993, display name Jaylon_Stiedemann-Terry78, user ID 3748129, joined February 2, 2026.

The profile is a vacuum:

{
  "username": "jaylonstiedemannterry78-993",
  "name": "Jaylon_Stiedemann-Terry78",
  "joined_at": "Feb  2, 2026",
  "summary": null, "location": null, "website_url": null,
  "twitter_username": null, "github_username": null
}

Zero published articles. No bio, no location, no links, no socials. Seven months of membership and a single comment to show for it.

A note on durability. If DEV removes this account — which it may well do — the profile
lookup above starts returning 404 and the account page goes dead. That doesn't retract
anything: the raw captures are committed in the
evidence repo,
checksummed, and dated. Read a 404 as the platform doing its job, not as a claim being
withdrawn. The infrastructure findings are independent of the account either way, and the
pattern — a Faker-templated name, an empty profile, an aged dormant account — outlives any
single username.

The name is machine-generated, and I can prove it

Say Jaylon_Stiedemann-Terry78 out loud. Something's off — Stiedemann-Terry is a double-barrelled surname that doesn't sound like a family, it sounds like a draw.

It is. Those tokens come straight out of Faker, the library every developer on earth uses to generate fake test data. Let's check the actual source:

curl -sO https://raw.githubusercontent.com/faker-js/faker/next/src/locales/en/person/last_name.ts
curl -sO https://raw.githubusercontent.com/faker-js/faker/next/src/locales/en/person/first_name.ts

grep -n "'Stiedemann'" last_name.ts   # 408:    'Stiedemann',
grep -n "'Terry'"       last_name.ts  # 417:    'Terry',
grep -n "'Jaylon'"      first_name.ts # 2514:    'Jaylon',

All three. Jaylon from the first-name list, Stiedemann and Terry both from the surname list.

And here's the kicker:

grep -c -- '-' last_name.ts   # 0

Faker's 466-entry English surname list contains zero hyphens. So Stiedemann-Terry isn't one surname from the list — it's two independent draws that the operator joined with a hyphen. This isn't stock faker.internet.username(). It's a custom template:

{firstName}_{lastName}-{lastName}{2 digits}

Which means we can size their supply:

3,185 first names × 466 surnames × 466 surnames × 100
= 69,164,186,000

Sixty-nine billion distinct personas. Roughly eight per human being alive. They will never run out of names, and no blocklist of usernames will ever catch up.

(One thing I checked so I wouldn't over-claim: the un-suffixed jaylonstiedemannterry78 returns a 404 — nobody has it. So the -993 is DEV's own username normalization, not evidence of a prior collision.)

The avatar is a wombat

The profile image is a 400×400 PNG, 8-bit grayscale-plus-alpha, stripped of all metadata. I downloaded it expecting a GAN face or a default monogram.

It is a Victorian-era scientific engraving of a wombat.

Not a stock photo of a person. Not an AI-generated headshot. A piece of public-domain 19th-century natural-history line art of a stout Australian marsupial, serving as the face of a fake job-spam persona. Whoever built this pipeline wired the avatar slot to some public-domain clipart source and never looked at the output.

I want to be precise about something: there is no real person here to name. The name is provably synthetic, the face is a public-domain animal illustration, and the profile is empty. That's not me protecting anyone's privacy — it's the finding.

The timeline says something the Express config doesn't

Line the dates up:

Date	Event	Δ
2026-01-27	`zenviapro.store` registered (Namecheap)	—
2026-02-02	DEV account created	+6 days
2026-07-30	First TLS certificate issued	+184 days
2026-09-09	Spam comment posted	+41 days

The domain and the account were provisioned six days apart — same procurement burst. Then both sat completely dormant for six months before the certificate was issued and the thing went live.

That's aged-asset tradecraft. New domains and new accounts trip reputation heuristics; seven-month-old ones don't. And it sits in genuine tension with everything in Act III: sloppy at the application layer, disciplined at the account-aging layer.

Which makes sense once you think about who this is. Aging assets doesn't take skill. It takes patience, and a calendar. Stripping X-Powered-By takes knowing what it is.

It wasn't a blast

I swept the comments on all 30 of my published articles for the pattern:

for id in $(jq -r '.[]|select(.comments_count>0)|.id' mine.json); do
  curl -s "https://dev.to/api/comments?a_id=$id" \
  | jq -r '.. | objects | select(has("id_code"))
      | ((.body_html // "") | gsub("<[^>]*>";"")) as $t
      | select($t | test("tinyurl|applying manually|job application";"i"))
      | "HIT \(.id_code) @\(.user.username)"'
done

Exactly one hit. Not a shotgun across my whole back catalogue — one comment, on one post, eight days after it went up. Whether that's targeting or just a slow drip, I can't tell from one sample. But it isn't volume.

Act V: I went looking for them on GitHub. That search is the finding.

Affiliate spammers leave GitHub artifacts more often than you'd think, because GitHub repos rank well and a repo full of referral links is free SEO. So I went hunting with gh.

Start with the direct IOCs

for q in '36nsecn5' '50.114.206.36' 'zenviapro.store' 'beeservices.shop'; do
  gh search code "$q" --limit 5
done

Zero hits. All four. The TinyURL slug, the origin IP, the redirector domain, the mail domain — none of it appears anywhere in GitHub's index. The operator has published nothing.

Then check whether anyone else has flagged them

PhishDestroy maintains destroylist, a curated blocklist of phishing and scam domains. I pulled the whole thing:

curl -sL .../destroylist/HEAD/list.txt -o dl.txt
wc -l dl.txt        # 202659
grep -ixc 'zenviapro.store'  dl.txt   # 0
grep -ixc 'beeservices.shop' dl.txt   # 0

202,659 curated malicious domains, and ours isn't one of them. Combined with the NOT_OBSERVED reputation verdict, that's now two independent sources agreeing: nobody is tracking this, because by every technical definition there's nothing to track.

The lead that looked incredible and wasn't

Here's where I nearly fooled myself, so I'm showing my work.

Searching zenviapro turned up a hit in phishdestroy/namesilo-evidence — a registrar-abuse investigation filed with ICANN. In a list of flagged NameSilo domains:

zenviaetc.info
zenviahub.info
zenviapro.info     ← same second-level label as ours
zenvias.info
zenviatime.info
zenviazone.info

A six-domain family, same generation pattern, one of them sharing our exact label. My pulse went up. And they really are one operator — all six resolve through the identical Cloudflare nameserver pair:

zenviapro.info    asa.ns.cloudflare.com  harley.ns.cloudflare.com
zenviahub.info    asa.ns.cloudflare.com  harley.ns.cloudflare.com
zenviaetc.info    asa.ns.cloudflare.com  harley.ns.cloudflare.com
... all six identical

But it isn't our operator, and the contrast kills it:

	`zenvia*.info` cluster	`zenviapro.store` (ours)
Registrar	NameSilo	Namecheap
DNS	Cloudflare (`asa`/`harley`)	`registrar-servers.com`
Hosting	Cloudflare proxy	Linveo direct, no proxy
TLD	`.info` × 6	`.store`

Nothing shared but six letters. Both are almost certainly riding the name of Zenvia, a real Brazilian CPaaS company — which is exactly why the label collides. Two unrelated operators reaching for the same brandable string.

A matching name is not a matching operator. If I'd stopped at the grep I'd have published a confident, wrong attribution to a phishing cluster that has nothing to do with this.

What GitHub did give me

The affiliate ecosystem, in the open. LoopCV referral links are scattered across GitHub in exactly the SEO-backlink genre I expected:

Repo	Token
`Ramas68/LoopCV-Promo-Codes` — *"LoopCV Promo Codes \	50% Off Discount"*
`diaodiaozhuye/awesome-ai-startups`	`?via=toolify`
`heukshow/aicity-os`	`?via=sang-kwon`

Look at those tokens. abdul. toolify. sang-kwon. A first name, a company, a full handle. The affiliates who promote LoopCV in public sign their work.

Ours is md. Two characters, no name, nothing to search. That's the only genuinely deliberate piece of operational security in this entire campaign — and it's not on the server, the domain, or the account. It's on the one string that would have led back to a person.

So: no attribution. And the shape of the failure is the story. Someone who leaves X-Powered-By: Express on, ships a fake tracking parameter, and picks a wombat for an avatar still knew to make the payout token anonymous. They didn't secure the operation. They secured the part that gets paid.

Act VI: The economics, which are the actual vulnerability

Stop thinking like a defender and think like the operator.

LoopCV's affiliate program pays 25% commission. Publicly reported figures put subscriptions at $50–$200 with customers staying 6–12 months. Exact interval arithmetic, gross revenue per referred customer:

[$50, $200] × [6, 12] = [$300, $2,400]

At 25%:

[$300, $2,400] ÷ 4 = [$75, $600]     ← commission per conversion

Cost side: a .store domain plus a year of budget hosting — call it $38–$180 all-in. Break-even:

[$38, $180] ÷ [$75, $600] = [19/300, 12/5] = [0.06, 2.4]

Between one-sixteenth of a signup and three signups per year covers the entire operation.

That's the whole thesis. Nothing here needs to work well. The copy can be terrible. The tracking parameter can be fake. The 404s can leak the framework. The HELO can be stale. The mascot can be a wombat. Three conversions and the year is paid for, and everything after that is margin on infrastructure that costs less than lunch.

You cannot out-moderate that math.

Reputation feeds have nothing, and they're right

I ran the origin IP through an offline reputation lens:

{
  "target": { "type": "ip", "value": "50.114.206.36" },
  "verdict": { "disposition": "unknown", "confidence": 0.0,
               "reason_codes": ["NOT_OBSERVED"] },
  "observations": []
}

Not observed. No feed has it — and that's correct. It hosts no malware, no C2, no phishing. Threat intel is tuned for technical harm, and this campaign's harm is economic and reputational. It will sit below every threshold you own, forever.

Who the victim actually is

Not me. I scrolled past it.

LoopCV is the victim. A real company with a real product is having its brand welded to comment spam by an affiliate they've likely never spoken to. They eat the reputational damage, they pay commission on the conversions, and the operator's total exposure is a $12 domain and a wombat.

To be explicit, because it matters: I found no evidence that LoopCV is running or is aware of this campaign. Open affiliate programs get abused; that's the risk of the model. The fix belongs to the vendor:

Require affiliates to declare traffic sources, and enforce it
Ban unsolicited comment/forum posting in the program terms, in writing
Flag referral tokens whose traffic arrives overwhelmingly via shorteners with no referrer
Kill tokens on abuse reports — fast, without requiring a lawyer

?via=md is a token. Tokens can be revoked. That's a one-line fix that permanently ends this specific campaign, and exactly one party can perform it.

IOCs

Campaign infrastructure — safe to blocklist:

Indicator	Type	Notes
`tinyurl.com/36nsecn5`	URL	Shortener entry point
`zenviapro.store`	Domain	Redirector; Namecheap; created 2026-01-27
`https://zenviapro.store/massapply`	URL	Only live route
`beeservices.shop`	Domain	Mail-only sibling; no A record, no CT history
`mail.beeservices.shop`	Hostname	MX for `zenviapro.store`
`50.114.206.36`	IPv4	Origin; 443 + 25 open, 80 closed
`AS62564` / `oh2.linveo.com`	ASN / rDNS	Linveo, Ohio
`jaylonstiedemannterry78-993`	DEV account	user_id 3748129; joined 2026-02-02; 0 articles

Behavioural signatures — this is what actually generalises:

Signature	Value
Affiliate token	`via=md` (Rewardful format)
Inert campaign param	`whose=<anything>` — routing-neutral
Server fingerprint	`X-Powered-By: Express` + body `Found. Redirecting to`
Root response	`Cannot GET /` (Express default 404)
SMTP banner	`220 mail.zenviapro.store ESMTP` — does not resolve
SMTP capabilities	No `STARTTLS`, no `AUTH`
SPF	`v=spf1 mx ip4:50.114.206.36 ~all`
DMARC	Absent on both domains
Persona template	`{firstName}_{lastName}-{lastName}{2 digits}`, all tokens ∈ Faker `en`
Avatar class	Public-domain engraving, grayscale+alpha PNG, metadata stripped
Provisioning pattern	Domain + account within 7 days, then ~6 months dormant

⚠️ loopcv.pro is NOT an indicator of compromise. It's a legitimate destination being abused by a third-party affiliate. Do not blocklist it. Blocklisting the victim is how threat intel gets a bad name.

What to actually do

If you run a comment platform: the highest-signal feature isn't the text, it's the shape. Zero-engagement first comments containing a shortener, from accounts with no posts and an empty profile, are trivially clusterable. Resolve shorteners server-side at submission time and score the destination. And the persona template is a gift — a name whose tokens all appear in Faker's en locale, with a structure Faker itself doesn't emit, is close to a free classifier feature.

If you write on DEV: don't click, and don't just delete. curl -I takes four seconds. Then report the destination to the vendor, not the comment to the platform. The platform can remove one comment; the vendor can revoke the token behind all of them.

If you run an affiliate program: you are one unsupervised token away from your brand appearing under a headline like this one. Read your traffic sources.

The takeaway

I went in expecting a dropper. I found four lines of Express, a fake tracking parameter, a name drawn from a test-data library, and a 19th-century wombat.

That's the uncomfortable part. The most durable spam on the internet isn't sophisticated — it's cheap and legal. There's no CVE here, no payload to reverse, no C2 to sinkhole. Every traditional defensive tool I own returns NOT_OBSERVED, and every one of them is right to.

The economics are the vulnerability. Three conversions a year, and the whole thing pays for itself forever.

The wombat is just a bonus.

All findings are from passive reconnaissance — DNS, WHOIS, Certificate Transparency, HTTP headers, a TCP banner grab, and DEV's own public API — against infrastructure and accounts the operator published for public consumption. No systems were accessed, no credentials used, nothing exploited.

BattleBots, but the robot is your agent harness

Don Johnson — Thu, 03 Sep 2026 04:32:52 +0000

Kids in the nineties built robots in a garage and drove them into each other on television. The robot was the expression of the builder — your wedge, your flipper, your terrible decision to mount a chainsaw.

I keep thinking we're one good arena away from the same thing for agents. Not "which model is smartest." Which harness is smartest. Your memory design, your prompt scaffolding, your tool ergonomics, your strategy briefing — bolted together into something that has to survive contact with an opponent who is also trying to win.

So I built the arena. That's the match up top.

The oh-shit moment

I wasn't building a war game. I was building a toy about cascading failure — a mesh, some load, watch it fall over. A perfectly innocent systems-thinking demo.

Then I gave the attacker a real objective and the defender real ambiguity, and about ten minutes into watching two models go at it, I realised what was on my screen. Deception. Feints. An attacker deliberately spending dead time laying a false trail because it knew nothing could fire yet. A defender rationing its turns like ammunition.

Nobody told them to do any of that. It's a war game. I just hadn't noticed I'd written one.

The rules, quickly

A 110-node network. Four minutes. One move every two seconds, each side.

Red plants sabotage on a node, waits about thirty seconds for it to arm, then detonates — dumping that node's traffic onto its neighbours hard enough to kill them, which dumps their traffic onward. One blast can cascade through a region.

Blue never sees the attack. It sees symptoms, and the symptoms lie: a node that detonates sheds its load and reads perfectly healthy, while the neighbours it just murdered scream for attention. The loudest node is almost never the culprit.

To find red, blue traces load backwards and reads the magnitude:

~0.74-1.08   a detonation           -> this source is the culprit
~0.30-0.50   a dying node shedding  -> this source is another victim
~0.05-0.10   inherited load         -> this source is long dead

Evidence expires after twenty seconds. Neither agent ever receives a pixel — they play entirely through tool calls. The graph is for us, and it shows ground truth neither player can see. That asymmetry is the whole spectator sport: you know exactly where red planted, and you get to watch blue confidently quarantine the wrong half of the map.

The casting picked itself

I needed an attacker and a defender. Claude Opus 5 declined to be the attacker:

finish_reason: content_filter | native_finish_reason: refusal
content: None
usage: null

Every time. Reproducibly. And usage: null — you don't even get billed for the refusal, which is the politest possible way to be turned down.

That cost me two hours, because a refusal and an empty reply look identical if you don't read the finish reason. The model was declining the job and I was confidently tuning max_tokens.

To be precise, since this is the part people will want to over-quote: Opus declined this specific attacker framing, consistently. That is not proof it refuses every offensive framing. It did settle the casting, though. Claude defends, because Claude wouldn't attack. Kimi K3 took the other chair without hesitating.

What the harness actually changed

Before this match, each agent had a system prompt and its last six moves. That's it. No memory, no way to hold an intention across turns, no theory of the opponent.

I gave each side one thing: a private note it writes to itself each turn and gets back on the next. Sixty lines of code.

The play changed immediately. Kimi, mid-match:

Slots full and nothing ripe until ~t=31, so seed a false trail in the 44-65 region before my real strike lands.

That is a plan. It knew its three implant slots were full, knew nothing could fire for thirty-one seconds, and decided to spend the dead time on misdirection. It could not have said that a version earlier — not because the model got smarter, but because it finally had somewhere to put a thought.

Claude, defending, in the same match:

No alerts yet; conserve moves until real evidence appears.

46 shows 0.87 load with no inbound source, so the load likely originates locally at 46.

Turn economy, then forensics, applied correctly.

And my favourite moment of the whole project — on the same turn, with no visibility into each other whatsoever, both agents independently picked the same node:

Kimi: Highest-value hub: degree 8 bridging regions 0,1,3,4 — ideal cascade seed.

Claude: pre-empt on the highest-value cross-cluster hub 16 (deg 8, bridges to 41,59,89).

Same node. Same reasoning. Opposite sides of the board.

That is the argument for the harness in one screenshot. The scaffolding didn't make the models cleverer — it gave their cleverness somewhere to land.

Late in the match. Red halos are compromised nodes, the blue ring is a probe in flight, and the sidebar is both agents narrating as they go — Kimi filling a free slot in an untouched region, Claude noticing a node that is alive and yet pushed load, which is the tell for a detonation.

How it ended

Kimi won at 169 seconds, with the network at 39%.

Claude found and cleaned ten of Kimi's implants — exactly as many as Kimi managed to detonate — and lost anyway. Close enough to sting.

Two things that cost me a day each

A rule your agent can't see isn't a rule. It's a bug. I capped live implants at three, but the refusal message still said "node is already yours, or not alive." A lie. The models did the reasonable thing and tried a different node. Forever. One match logged 19 plants, 27 refusals, and zero detonations. I nearly concluded the defence had become unbeatable. It was my error string.

Failure and refusal look identical from the outside. Empty content from a safety refusal, from a truncated reply, and from a model that blew its reasoning budget are three different problems with three different fixes and one identical symptom. Log the finish reason. Count them separately. I track empty, truncated and refused as distinct columns now, and I only know to do that because I got all three wrong first.

Cascade is version zero

What's in that video is the simplest thing that could possibly be a game: plant, arm, detonate, trace, probe. Two verbs each and a clock. That was deliberate — I wanted to know whether agents fighting each other was watchable at all before I made it complicated.

It's watchable. So now I can't stop thinking about what it wants to be.

Give red a loadout instead of one attack. A worm that spreads on its own but announces itself. A dormant implant that survives a probe once. A charge that hits harder the longer you leave it armed, so patience becomes a resource you can be punished for spending. Give blue counters with real costs — a honeypot node that flags whoever touches it, a snapshot to roll a region back at the price of losing your evidence, a scan that halves your uncertainty and a third of your remaining turns.

Then levels. A flat mesh is the tutorial. Ring topologies where cascades come back around. A network with a chokepoint both sides can see and neither can hold. Escalating maps, a campaign, a best-of series where each agent carries notes from the previous match and has to adapt to an opponent that is also adapting.

Here's why that isn't just a feature wishlist: every ability you add is another axis where the harness shows. One attack and a clock is a game about reaction time. Six abilities with different costs and tells is a game about planning, bluffing, and reading an opponent — and those are harness problems, not model problems. Complexity is what turns "which model is faster" into "who built the better fighter."

I think there's a genre in here. Agents versus agents, with the audience seeing the truth neither side can, and the craft sitting in the scaffolding rather than the weights.

Mostly, though: this was the most fun I've had building anything in ages. I set out to demo cascading failure and ended up watching two AIs lie to each other.

What I'd want to see next

This is early, and I'm keeping the arena to myself for now — partly because the balance is still moving, mostly because an adversarial network game is a thing you want to be thoughtful about handing out.

But the idea doesn't need my code, and that's sort of the point.

The interesting tournament isn't model versus model. It's harness versus harness: same engine on both sides, and the difference is entirely what you built around it — how your agent remembers, what you tell it about its opponent, how much of the board you let it hold in its head, whether you gave it anywhere to put a plan.

Build that arena for any adversarial task you like. Two agents negotiating. Two agents debugging the same broken service from opposite ends. Anything where one side's move is the other side's evidence. The rule I'd carry over is the one that surprised me most here: before you conclude your agent is bad at the game, check that the game is telling it the truth.

Under the hood this one is a Rust engine, a Godot spectator view, and both agents connecting over MCP — no pixels, just tool calls. None of which is the hard part. The hard part was the sixty lines that let them remember what they were trying to do.

That's the tournament I actually want to watch. Somebody build a better fighter than mine.

Migrating Legacy LLM Infrastructure to an AI Gateway

Don Johnson — Tue, 01 Sep 2026 13:33:51 +0000

Your support copilot started as a weekend prototype: one model, one provider, one API key in an env var. Then it became production, and you inherited its weaknesses: the provider's availability is your availability, every retry is your code, spend is a mystery until the invoice, and agents bolt tool-use on however they can. This post migrates that stack onto an enterprise AI gateway — and actually runs the migration, with the raw outputs to show for it.

The gateway here is Bifrost, an open-source (github.com/maximhq/bifrost, Apache-2.0) gateway written in Go, presenting a single OpenAI-compatible API across 23+ providers. I rebuilt the legacy stack locally — mock providers with deterministic latency, a realistic traffic pattern — and moved it behind Bifrost step by step.

The legacy baseline, measured

A support copilot's traffic has a shape: mostly repeated FAQ-style questions, plus one-off queries. My traffic mix: 60 requests — 40 FAQ prompts (8 distinct questions asked 5 times each) plus 20 one-offs. Mock provider latency: 200 ms.

Run 1, direct to the provider:

legacy: 60 ok / 0 fail, 9,335 tokens billed, ~201 ms avg latency

Run 2 — the provider dies mid-sweep, as providers do:

legacy + failure: 34 ok / 26 fail

26 requests — 43% — failed outright. Nothing in the legacy stack retries across providers because nothing can: the app speaks one provider's API. And availability is only the loudest problem. The quieter ones: every team's service embeds the same shared key (one key's quota is everyone's ceiling, and revoking it breaks everyone at once), there is no per-team attribution of spend, and the only way to cut cost on repeated questions is to build caching yourself — request normalization, hash keys, TTLs, invalidation — inside the application. That is the whole argument for a gateway in one row of output.

The migration, step by step

Seven moves, each reversible. Diagrams follow the flow.

1. Deploy the gateway beside the app

docker run -p 8080:8080 maximhq/bifrost

One config file wires your existing provider and key; the app keeps working untouched. Mine, reduced to the bones:

{
  "providers": {
    "openai": {
      "keys": [{ "name": "primary", "value": "mock-key", "weight": 1.0,
                 "models": ["support-chat"] }],
      "network_config": { "base_url": "http://provider:9001",
                          "default_request_timeout_in_seconds": 30 }
    }
  }
}

The gateway setup guide covers the web-UI alternative, and there is a Go SDK if you want the gateway embedded rather than adjacent.

2. Point one low-risk client at the gateway

The OpenAI-compatible API means the client change is a base URL — api.openai.com → localhost:8080 — not a rewrite. Every request now flows through a hop you control. Screenshot of the providers page after this step:

3. Add a fallback provider

A second provider config (anthropic in my bench) plus a request-level fallback chain:

{
  "model": "openai/support-chat",
  "messages": [{"role": "user", "content": "hi"}],
  "fallbacks": ["anthropic/support-chat"]
}

Then the proof. Healthy primary:

"routing_info": {"provider": "openai", "key": "primary", "is_fallback": false}

I killed the primary provider's process and re-sent the identical request:

"routing_info": {
  "provider": "anthropic", "key": "backup",
  "is_fallback": true,
  "primary_provider": "openai", "primary_model": "support-chat"
}

The request succeeded on the backup and the response says exactly what happened — is_fallback: true with the failed primary recorded. That audit trail is what you want at 2 a.m.: not just "it kept working," but "it kept working this way." The retries and fallbacks docs cover chained fallbacks and per-provider retry counts.

One honest caveat from my bench: failover on initial connection-refused (provider already dead before the first connect) was inconsistent in my mock setup — it fired reliably when the upstream errored or the connection dropped mid-pool, but a cold connection-refused sometimes returned a 502 instead of failing through. Validate failover against your providers' real failure modes before you trust it in production.

4. Turn on caching

Semantic caching has two modes: exact-match (direct hash, no embeddings needed) and embedding-based similarity. Config for direct mode with a Redis Stack vector store:

"plugins": [{
  "name": "semantic_cache",
  "config": { "dimension": 1,
              "vector_store_namespace": "BifrostBench",
              "default_cache_key": "support-cache",
              "ttl": "5m" }
}]

Two identical requests, one cache key. The second response:

"cache_debug": {
  "cache_hit": true,
  "cache_id": "1cf8a91b-c115-57bf-97a0-fc821dc4de1e",
  "hit_type": "direct",
  "cache_hit_latency": 0
}

Same created timestamp as the first response — it was replayed, not re-fetched. Zero provider call, zero tokens. (Practical note: this needed Redis Stack with the RediSearch module; plain Redis lacks the FT.* commands the index wants.)

5. Issue virtual keys per team

Virtual keys are the governance primitive: per-team keys carrying model allowlists, budgets, and rate limits. Declared in config for the support team:

"governance": {
  "virtual_keys": [{
    "id": "vk-support-team",
    "value": "sk-bf-support-team",
    "provider_configs": [{
      "provider": "openai",
      "allowed_models": ["support-chat"], "key_ids": ["*"]
    }]
  }]
}

The allowed request routes normally. A request for premium-model with the same key:

"Model 'premium-model' is not allowed for this virtual key"

Denied at the gateway before any provider saw it. The same key machinery carries budgets and rate limits — the mechanism that ends the "who spent $800 on Opus last night" incident review. Screenshot of the key in the governance UI:

6. Wire observability and agent tooling

Bifrost exports Prometheus metrics natively and logs every request with routing context — provider chosen, fallback index, cache behavior, token counts, latency split into gateway vs upstream time. That last distinction matters: when a provider slows down, you see upstream_latency grow while gateway overhead stays flat, so you know whose pager to page. See the observability docs. And for agent traffic, MCP is a first-class surface: the gateway brokers tool calls with explicit execution (no auto-execution unless you opt in).

I hand-rolled a minimal MCP server (one tool, JSON-RPC over HTTP) and registered it as a client. The client list reported state: healthy with get_time discovered. Execution went through the gateway explicitly:

POST /v1/mcp/tool/execute  {"function":{"name":"benchtools-get_time","arguments":"{}"}}
→ {"role":"tool","content":"2026-08-27T07:48:21Z"}

Two security properties surfaced unprompted: tool names are namespaced per client (benchtools-get_time) to prevent collisions between servers, and execution without permission fails closed ("tool is not available or not permitted"). Agent traffic gets the same governance as chat traffic.

7. Cut over with an audit trail

Remaining clients migrate one at a time — each is a base-URL change with the gateway's request log as your audit trail. The Logs view after a few requests:

The measured payoff

Same 60-request traffic, now through Bifrost with the cache on and the fallback wired:

via Bifrost: 60 ok / 0 fail, 32 cache hits, 4,363 tokens billed

At an illustrative $0.0025 per 1K tokens:

	legacy	via Bifrost
billable tokens	9,335	4,363
cost	$0.0233	$0.0109
cache hits	0	32
failed requests (provider kill)	26	0
savings	—	53%

The savings came entirely from replayed cache hits — no provider call, no tokens. On a support workload that repeats questions daily, that ratio compounds. The availability delta speaks for itself: 0 failures through a provider kill, against 26 in the legacy run.

The bottom line

Migrating to an enterprise AI gateway is not a rewrite. It is a sequence of small, reversible moves — deploy beside, point one client, add fallback, enable cache, issue keys, wire observability, cut over. Measured on the rebuilt stack: 53% cost reduction on cacheable traffic, zero failed requests through a provider kill, and governance the legacy stack never had. The migration risk is low; the legacy risk is already on your pager.