DEV Community: Don Johnson

My QUIC transport had never once been executed. Here's what happened when I ran it.

Don Johnson — Wed, 19 Aug 2026 03:35:06 +0000

I've written before about SMESH, a coordination protocol modelled on mycorrhizal networks — the fungal web that lets trees in a forest warn each other about drought and disease with nothing in charge of the network. Signals diffuse, decay on their own, and get reinforced when independently confirmed. Consensus emerges instead of being orchestrated.

That was the idea. This post is about the part where I found out whether it worked.

The transport that had never run

SMESH has had a QUIC transport in it for a while. Roughly 500 lines: a quinn endpoint that is simultaneously server and client, self-signed certs, length-prefixed bincode frames over unidirectional streams, an accept loop that spawns per-connection and per-stream tasks, connection pooling.

Every test passed. The workspace was green. I could point at smesh-runtime/src/transport.rs and say "yes, it does peer-to-peer."

Then I grepped for who actually constructed it:

$ grep -rn "QuicTransport" --include='*.rs' .
smesh-runtime/src/transport.rs:177:pub struct QuicTransport {
smesh-runtime/src/transport.rs:192:impl QuicTransport {
smesh-runtime/src/lib.rs:16:pub use transport::{QuicTransport, ...};

Its own definition, and a re-export. Nothing else in the workspace had ever instantiated it. No binary opened a socket. SmeshRuntime imported TransportConfig, stored it in a struct field, and never looked at it again.

I had a networking layer with tests, docs, and zero executions.

Three bugs in the first twenty minutes

I wrote an integration test that starts two runtimes, has one dial the other, and asserts a signal crosses. Here is what fell out before it went green.

1. It panicked on the first call.

Could not automatically determine the process-level CryptoProvider
from Rustls crate features.

rustls 0.23 refuses to pick a crypto backend when more than one is compiled in, and quinn pulls in both through its own feature set. Every call to QuicTransport::new would have panicked for anyone, ever. Nobody noticed because nobody had called it.

2. Dialled connections were write-only.

connect() stored the connection in the pool but only the accept loop pumped incoming streams — and the accept loop only sees connections you accepted. So a node that dialled out could send, and would never receive anything back. A QUIC connection is bidirectional regardless of who dialled it; my code only acted like it half the time.

3. Unbounded allocation from an attacker-controlled length prefix.

let len = u32::from_be_bytes(len_buf) as usize;
let mut data = vec![0u8; len];   // <- no

max_message_size was in the config struct. It was never read. Send a 4 GiB length prefix and the process allocates 4 GiB.

None of these are clever bugs. They're the bugs you get for free the first time code meets a socket, and the only reason they survived is that the code had never met a socket.

The harder problem: my protocol was wrong

Fixing the plumbing was the easy half. The real issue was that my diffusion algorithm quietly assumed something no distributed system can assume.

Network::tick expands a signal's reach one hop per tick by walking the graph:

for node_id in &reached {
    for hypha in self.hyphae.get(node_id) {
        let target = self.nodes.get(&hypha.to);
        if target.should_relay(&signal, remaining_hops) {
            frontier.push(hypha.to.clone());
        }
    }
}

Read that again. It iterates every node's adjacency, calls every node's relay policy, and mutates one global reached_nodes set on a shared signal. It's a breadth-first search from a god's-eye view of the entire graph.

That works beautifully in one process. In a real mesh, no node can see that graph. Porting it meant three corrections, and each one turned out to be a genuine bug rather than a porting detail.

Correction 1: independent conclusions were being thrown away

Signals are content-addressed — the hash is derived from what is being claimed. So when two agents independently reach the same conclusion, they produce the same hash.

My emit() saw the hash already present locally and treated it as a duplicate. It dropped it.

That is exactly backwards. Two parties independently agreeing is not redundant data — it is the only evidence the system has that a claim is real. Discarding it destroys the thing the protocol exists to measure.

/// Signals are content-addressed, so a node that independently reaches a
/// conclusion another node already published lands on the same hash. That
/// is treated as *corroboration*: this node is added as an attester and the
/// merged claim still goes out, because our agreement is news to everyone
/// who has not heard it. Swallowing it as a duplicate would silently
/// discard the only evidence that two parties concur.

Correction 2: I was counting messengers, not witnesses

Reinforcement attributed the claim to whoever handed me the message. In a gossip mesh, one finding relayed by five nodes then looks like five corroborators.

The party that attests to a claim is its origin, not the peer that passed it along. Relaying is not agreeing.

Correction 3: gossip needs a merge rule, not a broadcast rule

Once attestation is a set, the rule that makes gossip converge is simple and pleasant:

// Merge the two attester sets. Anything the sender knew that we did
// not is new information, and new information is worth passing on.
let before = Node::attesters(existing);
for attester in &incoming_attesters {
    if !before.contains(attester) {
        existing.reinforce(attester);
    }
}

Forward if and only if your own knowledge grew. That single rule is the loop breaker (a message teaching you nothing goes no further), the convergence mechanism (the set is grow-only, so it's a CRDT), and the anti-entropy repair (re-asserting carries your accumulated view, so a node that missed a round catches up).

The part I nearly got wrong on purpose

Here is the subtlety that makes the whole thing work, and it looks like a mistake in the source.

The signal builder folds the origin node into the content hash if you give it one. So when an agent publishes a claim, you must not set the origin:

let signal = Signal::builder(SignalType::Alert)
    .payload(assertion.canonical_bytes())
    .confidence(finding.confidence)
    .build();
// .origin() deliberately NOT called

If you set it, every agent gets a different hash for the same claim and correlation becomes impossible. The address has to be the claim, not the claimant.

The corollary is that evidence cannot travel in the payload. Each agent's evidence differs, so putting it in would make every hash unique and break the mechanism. The payload is just the assertion:

{"subject":"checkout-api","claim":"degraded"}

Evidence stays local and goes to the log. The mesh carries assertions; it does not carry arguments.

Five witnesses, none of whom can see the problem

To actually test any of this, I built a scenario where the answer cannot be reached alone.

Five analyst processes watch the same fleet of services. Each can see exactly one kind of telemetry, and nothing else:

agent	sees
latency	p99 response times
errors	error rates
saturation	pool and CPU utilisation
traces	retry rates, span queueing
deploys	release events

A deploy cuts checkout-api's connection pool from 200 to 20. The pool pins, requests queue, callers time out.

Now every agent sees a piece of it, and two of them are actively misleading. errors sees payments-api throwing 503s and would blame it — but payments is a victim, its own pool and CPU are fine. latency sees four services slow at once and can't say which is causal. Only deploys can see there was a release, and a release on its own means nothing; software ships all day.

I also planted two decoys with a single witness each — an unrelated CPU spike, a brief latency blip — because a system that agrees with everything is not consensus, it's an echo.

They run as five real OS processes on five ports over real encrypted QUIC, in a ring-plus-chord topology so messages actually have to be relayed to cross the mesh. Peer discovery is off, or gossip quietly converts any topology into a full mesh and there's nothing left to watch.

$ smesh orchestrate --out runs/latest

  spawned latency     pid 2452744  127.0.0.1:9301  dials 9302,9303
  spawned errors      pid 2452745  127.0.0.1:9302  dials 9303
  spawned saturation  pid 2452746  127.0.0.1:9303  dials 9304
  spawned traces      pid 2452747  127.0.0.1:9304  dials 9305
  spawned deploys     pid 2452748  127.0.0.1:9305  dials 9301

The result:

what the mesh concluded
  checkout-api          CONSENSUS at 16.9s · 5 attesters · seen by 5 nodes
  payments-api          no consensus (3/4 concerns)
  edge-gateway          no consensus (1/4 concerns)
  notification-worker   no consensus (1/4 concerns)
  session-store         no consensus (1/4 concerns)

Cause separated from symptom separated from noise. The loud, obvious suspect was held at three witnesses and never promoted. The decoys were never rejected by anything — they simply went uncorroborated and decayed out. Nobody voted, and nothing was in charge.

Making the run replayable, and then checking that claim

I wanted to visualise this, which meant the log had to be good enough to reconstruct the run exactly. Each process writes newline-delimited JSON against a shared run epoch: every emission, every per-peer send, every receipt, every relay decision including the probability and the die roll that resolved it, and a full field snapshot every 500ms so decay curves are observed rather than modelled.

Then I did the part I'd recommend to anyone building a log you intend to trust: I wrote a validator that checks the log against itself. Sequence gaps, time going backwards, snapshots referencing signals that were never received, consensus declared without the receipts to justify it.

It caught a real bug on its first run:

FAIL  latency: first event is peer_connected, not node_started

A peer completed the handshake in the gap between binding the endpoint and writing the node's identity line. The fix was to move the identity write inside the mesh startup, before any loop spawns. I would never have found that by looking at the picture — the picture would just have been subtly wrong.

What is still wrong

Being honest about the edges, because "it works" is a claim that needs a boundary:

origin_node_id is unauthenticated. It's a string on the wire, and the trust model gates relay probability on it. Spoofing another agent's identity is currently free. Ed25519-signing the origin hash closes it and is the next real piece of work.
The content hash is truncated to 64 bits. Fine against accident, not against an adversary looking for collisions.
The telemetry in the demo is synthetic. Deliberately: a seeded fixture means the run reproduces byte-for-byte on any machine, which is what makes a visualisation worth trusting. The coordination is not synthetic — real processes, real sockets, probabilistic relay.

See it move

The full narrated walkthrough is the cover video on this post, or here: https://youtu.be/kmCzwSBqu_s

It opens on the forest the protocol is stolen from, then goes inside the recorded run: five agents, six encrypted links, every dot on screen a real message read back from the log rather than animated for effect.

If you take one thing from this: code that has never been executed is not code, it's a plan. Mine had tests, docs, and a clean cargo clippy, and it would have panicked on the first line for every user. The tests were testing that the plan was internally consistent.

Repo: github.com/copyleftdev/smesh-rust · Rust · MIT/Apache-2.0

Shipping Assumptions: A Reliability Stack for AI-Generated Code

Don Johnson — Sun, 16 Aug 2026 22:56:25 +0000

The code looked good. It linted cleanly. The shallow tests passed. Everyone felt—vibed—that we had built the right thing.

Then the edge case appeared in production.

The developer treated it as normal operating procedure: bugs happen, tickets arrive, patches ship. QA was asked why they had not caught it. But QA never received a model of the system—only an implementation full of assumptions they were expected to reverse-engineer.

This is becoming the defining failure mode of AI-assisted development. We can generate code faster than we can understand the systems it creates. The danger is no longer confined to a bad function or an obvious syntax error. It lives in the space between components: boundaries, state transitions, failure modes, and invariants.

We are not merely shipping code. We are shipping assumptions we can no longer see.

Clean code is not a coherent system

A linter can tell us whether code follows a set of local rules. A test can tell us whether selected examples produce the expected result. Neither can tell us what the system must preserve unless someone states it first.

An invariant is one of those statements: a condition that must remain true across every valid state of the system. An account balance cannot be changed without a corresponding transaction. A private object cannot become public without authorization. Two successful writes cannot silently erase one another.

When invariants remain implicit, clean code can still assemble into an incoherent system. Each component may look reasonable in isolation while the failure waits in a transition, a retry, a race, or a boundary nobody thought to draw.

That is the space AI-assisted development is rapidly filling with code.

The layer we buried

There was no golden age in which every developer wrote a formal specification before touching an editor. Most teams did not. Earlier software was full of hidden assumptions too, and nostalgia has a habit of removing the inconvenient parts of the past.

But the field did develop disciplines for reasoning above the code: architectural models, state machines, formal specifications, model checking, fault injection, and deterministic simulation. We tended to reserve them for systems where failure was obviously expensive. Everywhere else, rigor was treated as a cost to minimize.

Then implementation became cheaper. Frameworks hid machinery. Packages compressed years of expertise into an import. That brought enormous benefits, but it also made it possible to build systems without seeing very far beneath their surfaces.

Generative AI accelerates the same trade. It can produce a plausible implementation before a team has agreed on what the system is. When nobody externalizes that intent, the generated code becomes the design by default. QA receives the consequences downstream.

The missing layer is not more code review. It is a shared model between human intent and machine output.

Three models, three questions

No single technique covers that layer. A useful stack needs to answer three different questions.

1. C4: What exists, and how does it relate?

The C4 model gives software teams a hierarchy for visualizing a system: context, containers, components, and code. It works like a map with zoom levels. At the widest view, we see the system, its users, and neighboring systems. Zooming in reveals applications, data stores, services, components, and relationships.

The point is not to produce four mandatory diagrams for every project. The official guidance notes that context and container diagrams are sufficient for many teams. The point is to make boundaries discussable.

That matters because generated code is locally persuasive. A service can look complete while its ownership is unclear. An API can look tidy while its trust boundary is invisible. C4 gives development and QA the same structural map before either group has to infer the system from a repository.

AI can help draft that map from requirements or an existing codebase. It should not get the final word. Humans still need to ask whether the map describes the system they intend to operate.

2. TLA+: What may happen, and what must remain true?

C4 shows structure, but structure alone cannot express behavior. It does not tell us which states are valid, which transitions are permitted, or which conditions must survive every interleaving.

That is where TLA+ fits. Leslie Lamport describes it as a language for precise, high-level models above the code level, especially for concurrent and distributed systems. Its TLC model checker can explore behaviors and find traces that violate the properties we claim should hold.

TLA+ forces a different conversation. Instead of asking, “Does this function work?” we ask, “What can the system do from this state, and is every reachable result acceptable?”

The notation may be formal, but AI can reduce the entry cost: drafting a first specification, translating plain-language promises into candidate invariants, explaining counterexample traces, and helping a team refine the model. The human job is to decide whether those promises are actually the right ones. A model that formalizes the wrong intent is merely precise about the wrong system.

3. DST: Does the implementation honor the model?

A correct model does not prove that the production code implements it correctly. We still need the implementation to fight back.

Deterministic simulation testing, or DST, runs real implementation code in a controlled world. Time, randomness, scheduling, networks, storage, and faults become inputs that a simulator can manipulate. When a generated scenario causes a failure, its seed makes the run reproducible.

TigerBeetle’s VOPR is a striking example. It can simulate a cluster on a single thread, accelerate time, and inject storage faults. TigerBeetle makes an important distinction in its architecture documentation: model checking can expose errors in an algorithm, while simulation exercises the specific implementation and its underlying assumptions.

The three layers complement one another:

C4 maps the system. TLA+ states what must remain true. DST tries to make it false.

A workflow for abundant code

This does not need to become a new ceremony in which teams maintain enormous diagrams and specifications that immediately drift out of date. The models have to participate in the engineering loop.

State the promises in plain language. Before generating implementation, write down what the system must always preserve and what it must never permit.
Draw the boundaries. Use C4 to identify users, external systems, containers, components, data stores, and the relationships where assumptions cross from one owner to another.
Model the risky behavior. Use TLA+ where concurrency, ordering, retries, permissions, or state transitions make the promises difficult to reason about informally.
Generate from reviewed intent. Let AI produce implementation and tests only after people have challenged the structural and behavioral models.
Attack the implementation deterministically. Put risky dependencies—time, randomness, scheduling, networks, or storage—under control. Generate hostile scenarios, preserve their seeds, and replay every failure.
Feed reality back into the models. Production observations and simulation failures reveal assumptions the team missed. Update the invariants, then update the implementation and tests. Detect drift rather than allowing the model to become historical decoration.

Not every application needs a simulator as sophisticated as VOPR or a large TLA+ specification. The lightweight version might be one context diagram, one container diagram, five written invariants, a small model for the riskiest transition, and a seeded test harness around it.

The goal is not maximal formality. It is to give every important assumption an address.

Everything is mission-critical to someone

Imagine a grandmother uses AI to vibe-code a family photo album. This is not a financial ledger or a flight-control system. Yet the application makes real promises:

Original photographs must never be destroyed.
Private photographs must never be exposed accidentally.
Captions and chronology must survive every edit.
Two family members must not overwrite each other’s changes.

All of them matter.

Her C4 model can be small: a person, an application, an identity provider, local storage, cloud storage, and a sharing boundary. Her TLA+ model does not need to describe the entire product; it can focus on concurrent edits, deletion, or permissions. A deterministic harness can simulate two devices going offline, editing the same album, reconnecting in different orders, and retrying interrupted uploads.

This stack was once associated with mission-critical engineering because rigor was expensive. But AI has changed the economics. If implementation now comes almost freely, why should rigor remain a luxury?

Criticality is not determined by infrastructure scale. It is determined by the consequence of failure to the person who trusted the system.

Everything is mission-critical to someone.

Retrieval, not retreat

It is tempting to tell the history of software as a simple decay: once we understood the machine, then we specialized, then we configured packages, and now we prompt a model to do even that for us.

That story is emotionally recognizable and historically incomplete. Modern developers are not less intelligent, and abstraction is not the enemy. The deeper problem is that valuable disciplines became buried beneath convenience. We kept the outputs while losing contact with some of the reasoning that produced them.

This is where nostalgia can be useful. Not as a demand to reconstruct an imagined past, but as a diagnostic signal. It tells us that something in the present has become difficult to see or value.

Computer science has left us breadcrumbs: C4, TLA+, state-machine thinking, deterministic simulators, small composable tools, and decades of systems literature. AI can help excavate that inheritance. It can explain unfamiliar notation, recover architectural knowledge from old code, draft models, generate harnesses, and make specialized techniques approachable to ordinary teams.

That is retrieval, not retreat.

The answer to faster code generation may be older, deeper modeling.

Humans own the promises

Shared models also repair an organizational failure.

Without them, a developer hands QA an implementation full of implicit decisions. QA runs the checks it can see. An edge case reaches production, and the organization asks why testing missed it. The assumption moves downstream, and the blame moves with it.

With a structural map and explicit invariants, QA no longer has to guess what the developer meant. Testers can challenge the promises, identify missing states, design hostile scenarios, and feed new discoveries back into the model. Quality becomes a shared act of reasoning rather than a final gate operated by the last team in line.

That restores a form of human value AI cannot generate for us. Our value was never only in typing every line. It is in deciding what a system is for, recognizing who can be harmed, making its promises visible, and accepting responsibility when they are broken.

Fewer people should be able to dismiss a failure by saying, “The AI wrote it.”

Generation does not transfer responsibility. If we decide what a system is for, accept its output, and release it into someone’s life, the promises it breaks are still ours.

AI may produce the code. Humans own the promises.

Disclosure: This article was developed with AI assistance through an iterative editorial process. The argument, examples, editorial direction, and responsibility for accuracy remain with the human author.

Write down every guarantee before you write any code

Don Johnson — Tue, 11 Aug 2026 03:35:29 +0000

Here is every promise a to-do list makes.

VARIABLE tasks

Init == tasks = [i \in Ids |-> "absent"]

Add(i)      == tasks[i] = "absent" /\ tasks' = [tasks EXCEPT ![i] = "open"]
Complete(i) == tasks[i] = "open"   /\ tasks' = [tasks EXCEPT ![i] = "done"]
Reopen(i)   == tasks[i] = "done"   /\ tasks' = [tasks EXCEPT ![i] = "open"]
Delete(i)   == tasks[i] # "absent" /\ tasks' = [tasks EXCEPT ![i] = "absent"]

ClearCompleted ==
  /\ \E i \in Ids : tasks[i] = "done"
  /\ tasks' = [i \in Ids |-> IF tasks[i] = "done" THEN "absent" ELSE tasks[i]]

Not a summary. Not the important ones. All of them. A task cannot go from
absent straight to done. Clearing completed items leaves the open ones alone.
You cannot delete something that was never there. Nine lines, and when you've
read them you have read the entire contract.

Now go find that list for the system you work on.

You can't. It doesn't exist. It's distributed across a test suite that asserts
outcomes rather than rules, some validation scattered through handlers, and the
memory of whoever's been there longest. The guarantees are real — your users
depend on every one of them — and there is no file you can open to see them.

That's the gap I want to talk about, because you can close it in an afternoon,
and because something has changed recently that makes closing it pay for itself.

The prime mark and two operators

That's most of the syntax, so let's get it out of the way.

tasks' means "tasks, in the next state." /\ is and. \E is "there
exists." A definition like Complete(i) is a formula relating the current state
to the next one — read it out loud: the task is open, and afterwards it is
done.

That's it. That's the language, near enough, for this purpose.

The real file adds about eight lines of scaffolding around what you saw:
a module header, a TypeOK saying a task is always in exactly one of the three
states, and the two lines that tie the actions together —

Next == \/ \E i \in Ids : Add(i) \/ Complete(i) \/ Reopen(i) \/ Delete(i)
        \/ ClearCompleted

Spec == Init /\ [][Next]_tasks

Next is "any one of the moves happens." Spec is "start legally, and then only
ever make legal moves." That second line turns out to matter more than it looks,
and I'll come back to it.

Notice what isn't in there. No database. No HTTP. No mention of whether the
button is blue or whether completion is optimistic in the UI. A specification
isn't a program and doesn't compile to one — it's a formula that says which
state changes are permitted. Everything else is out of scope by construction,
which is exactly why the list can be nine lines and still be complete.

And notice ClearCompleted has two halves: the button only exists when
something is done, and it leaves everything else alone. Two separate
promises in one action. Hold that thought.

The list is short, finite, and worth arguing about

The objection I expect is that a real system's list would be enormous.

It's smaller than you think, because it's a list of rules, not behaviours. The
behaviours are combinatorial — nine states here, and a real system has
astronomically many. The rules that generate them are not. Five actions cover
every to-do list that has ever been correct.

It's also the part of the design worth arguing about. When two engineers
disagree about whether reopening a completed task should be allowed, that
argument currently happens in a code review, in a comment thread, three weeks
after someone already built one of the answers. Written as a spec, the argument
takes four minutes and happens before anyone opens an editor.

That's the AWS result, really. They wrote up their experience in CACM in
2015 and the headline everyone quotes is about proving systems correct. The part
that actually replicates is quieter: writing the spec found bugs before any
code existed — in systems their best engineers had already designed and
reviewed. Not bugs the tests missed. Bugs the design had, findable by writing
the guarantees down and reading them back.

This is forty-year-old technology, and most of us skipped it because it looked
like homework. TLA+ is Leslie Lamport's; the temporal logic underneath it landed
in TOPLAS in 1994, the language and tools got a book in 2002,
and Lamport picked up the 2013 Turing Award along the way. (Not for
TLA+, worth saying, since people get this wrong: the citation is logical clocks,
safety and liveness, replicated state machines, sequential consistency. TLA+ is
downstream of that work, not the reason for the medal.)

Its reputation for being academic is partly earned and mostly out of date. You
do not need the proof system. You do not need to verify anything. You need the
part where you write the guarantees down.

What changed

Writing the list has always been worth it and has always been easy to defer,
because the code was going to be written slowly by people who mostly remembered
the rules.

That is no longer the situation. Something else is writing the code now, quickly,
and it does not remember anything. It has never met your system's rules and has
no way to infer the ones that aren't in the file it's looking at. It will write
something plausible.

Plausible is the problem. Plausible code passes review — this is where "looks
good to me" comes from, and it was always an honest confession: the reviewer is
reporting that nothing jumped out, because checking against the full set of
invariants was never an option. Nobody had the list.

So: write the list. Then check the generated code against it, mechanically,
every time. That second half needs a tool.

tlatools-rs

cargo install tlatools

A TLA+ parser and evaluator in Rust. Not a model checker — it doesn't explore
anything. It answers questions about states you already have:

let spec = Spec::from_file("Todo.tla")?;
let eval = Evaluator::new(&spec, constants)?;

eval.holds_at("Init", &state)?;         // legal starting state?
eval.step_allowed("Next", &from, &to)?; // legal step?

The loop is three pieces. You write the list — short, arguable, and it
barely changes. The agent writes the implementation — any language, any
framework, any speed. A script walks the implementation and asks the list
about every step it takes. That third piece is thirty lines: ask the
implementation what it can do, do each of those things, record where you landed,
repeat until nothing new turns up.

$ ./check.py impl/correct.py
The implementation refines the specification.
9 states and 35 steps, all permitted.

Nine states because there are two tasks and three states each. A real app has
more, and the walk is the expensive part, not the checking.

Two bugs the list catches

Here's an agent-plausible one. The completion handler takes an id and marks it
done. It doesn't check the task was open — why would it, the button only shows
up on open tasks. (The button. Not the handler.)

This is exactly the bug that survives review. It reads correctly. The missing
check is missing somewhere you aren't looking.

$ ./check.py impl/completes_anything.py
The implementation takes a step the specification does not permit.

  from   a=absent, b=absent
  doing  complete(a)
  to     a=done, b=absent

The closest the specification came:
  Add(i = "a") was available, but does not produce that state,
    because tasks' = [tasks EXCEPT ![i] = Open] does not hold (1 of its 2 clauses hold)
  Complete(i = "a") was not available here,
    because tasks[i] = Open does not hold (1 of its 2 clauses hold)

The second line is the bug, named: Complete requires the task to be open, and
it wasn't. It's a ranked shortlist rather than a single guess — Add also nearly
fits from this state, and saying so is more honest than pretending to know which
one you meant.

Now the other one. ClearCompleted — the action with two promises. This
implementation keeps the first and breaks the second. It clears the whole list:

$ ./check.py impl/clear_removes_everything.py
  from   a=open, b=done
  doing  clear_completed
  to     a=absent, b=absent

The closest the specification came:
  ClearCompleted was available, but does not produce that state,
    because tasks' = [i \in Ids |-> IF tasks[i] = Done THEN Absent ELSE tasks[i]]
    does not hold (1 of its 2 clauses hold)

"Was not available here" versus "was available, but does not produce that
state." Different sentences because they're different bugs. One is a missing
guard. The other is a correct guard and a wrong effect — which is worse, because
the button looks like it works. You'd demo it. You'd ship it. Someone would
lose a task they hadn't finished.

The tool can tell them apart because it knows which failing clause mentions the
next state. Neither bug is exotic. Both are invisible to a test suite that
checks outcomes, and both are named instantly by a list you wrote in nine lines.

Feedback in the language the rule was written in

You cannot fix what you cannot describe, and neither can a model.

"Tests failed." — Try again. Randomly.
"Expected {a: open}, got {a: absent}." — Better. Now infer the rule.
"ClearCompleted was available, but does not produce that state, because tasks' = [i \in Ids |-> IF tasks[i] = Done THEN Absent ELSE tasks[i]] does not hold." — The action, the condition, and the state it was in.

That third one is a prompt.

And I measured whether it helps an agent, and it didn't — not detectably.
200 tasks, each attempted with an uninformative retry and with the failure text
fed back: 90.5% [85.6–93.8] against 92.5% [88.0–95.4], McNemar exact p=0.125.
That is a null. On the formal-reasoning subset the gap was 46.2% → 69.2%, which
looks like something, except n=13 and p=0.25, which means it looks like
something in the way small numbers often do.

I'm reporting it because I ran it. The honest state of the claim: the mechanism
is sound, the message is strictly more information than a boolean, and I have no
evidence it moves the pass rate. If you were going to adopt this because "agents
do better with good errors" — don't, yet. Adopt it because you now have the
list, and something checks it.

The list as a grader

tlatools check takes a JSON job — spec, states, steps, constants — and returns
a verdict with an exit status: 0 it refines, 1 it doesn't, 2 the question
was malformed.

tlatools check job.json || exit 1

Point it at N candidate implementations and it tells you which satisfy the list
and, for the ones that don't, exactly where they diverge. If you're generating
code, evaluating models, or grading a benchmark, that's a grader with no rubric
to write and no partial credit to argue about. The list is the rubric.


`init`	is the starting state legal?
`refines`	is every step one the spec permits?
`coverage`	does the implementation reach the outcomes it should?

That third one exists because refinement alone is satisfied perfectly by an
implementation that does nothing. Ask me how I know.

Can you trust the checker?

Fair question to ask of anything that grades your code.

It agrees with the reference implementation. Over a labelled corpus of 39
cases — six implementations that must pass, thirty-three seeded bugs that must
each be caught — its verdicts are byte-identical to Java TLC's, including which
of the three checks catches which bug.

Getting that diff empty taught me something I'd otherwise have shipped wrong,
and it's the best argument in this article for writing guarantees down precisely.

Remember Spec == Init /\ [][Next]_tasks, the line I said would matter. Those
brackets are load-bearing: [Next]_tasks means Next, **or nothing changed**.
Stuttering is always permitted, in every TLA+ specification ever written. I had
been checking bare Next, so any implementation that idled or retried got
flagged for a step the spec explicitly allows.

Fixing that made one seeded bug survive: a transfer from a bank account to
itself. Which felt like a regression, until I read the spec again. A
self-transfer nets to zero. It changes nothing. It is a stuttering step, and
the spec says stuttering is fine — so that implementation genuinely satisfies
the list, and the benchmark and I had both been wrong about it. Catching that
one needs an abstraction where the operation is visible in the state at all. You
cannot tighten a refinement check into seeing something the state space doesn't
record.

Hand TLC the same [Next]_vars obligation and it passes that mutant too. The
agreement holds; what moved was my understanding of what I'd written down. The
list is only as good as your reading of it, and a tool that disagrees with you
is doing you a favour.

It reads the language. 1,256 of the 1,258 specifications in three public
TLA+ corpora: the examples repo, the community modules, and the TLA+ tools' own
test suite. The two it doesn't read are two that SANY, the official parser,
doesn't read either. How every one of those files is read is recorded in
golden/, so a change names the files it changed instead of moving a number.

It's checked in the boring ways. 166 tests, clippy-pedantic clean, and a
robustness suite that feeds back every prefix and every dropped line of every
fixture — because a parser that panics on a half-saved file is a parser you
can't put in a loop. tla-syntax and tla-eval have no external dependencies at
all.

About TLC, precisely

TLC is the model checker TLA+ ships with. It explores: from your initial state,
apply every action, walk the reachable state space looking for a violation.
That's the right question when you're designing a protocol and don't yet know
what your system can do.

Here you already have the implementation, so the question is different — are
these steps, the ones the code just took, permitted by the list?

TLC can be asked that. I want to be precise, because I had this wrong in an
earlier draft and someone would have caught it: you encode the steps as data,
write a plain safety invariant asserting each one is enabled, and run the
checker. No liveness property, no engineered failure. On the to-do spec it finds
the bad step in 0.78 seconds. There's published work doing this properly —
Validating Traces of Distributed Programs Against TLA+ Specifications
by Cirstea, Kuppe, Loillier and Merz, and Kuppe maintains the TLA+ tools. If you
want trace validation on a production system, start there.

What's left is narrower, and true. TLC tells you that the step is illegal, not
which conjunct failed. And that 0.78 s is almost entirely JVM boot and SANY
parse, paid again on every query — fine once, not fine in a loop that runs on
every agent edit.

Structured blame instead of a boolean, at a couple of orders of magnitude less
latency. That's the pitch. These compose; they don't compete.

What it won't do

It's not a model checker. If you want to know what states your system can reach, use TLC.
It doesn't verify your program. It checks the transitions you hand it. If your walk misses a path, nothing checks that path.
Your list can be wrong. It's what you argued about, not what's true. I could have written ClearCompleted to clear everything, and then the "buggy" implementation would be the correct one. The list being short and readable is the only defence, which is an argument for keeping it short and readable.
Integers are 64-bit, real arithmetic isn't implemented, temporal formulas are refused rather than guessed at, and TLAPS proofs are skipped.

Try it

git clone https://github.com/copyleftdev/tlatools-rs
cd tlatools-rs
cargo build --release
demo/todo/check.py demo/todo/impl/completes_anything.py

The whole demo is demo/todo — the spec above, three implementations,
and the script that checks them. CI runs it on every push, so if it's broken
when you get there, that's a bug and I'd like to hear about it.

crates: tlatools · tla-eval · tla-syntax · tla-oracle
source: github.com/copyleftdev/tlatools-rs
docs: docs.rs/tla-eval

If you have a TLA+ file it reads wrongly, that's the most useful thing you can
send me.

Start with one. Pick the part of your system where a wrong state change would
actually hurt — the money, the permissions, the thing with a state machine
nobody fully trusts. Write down what it's allowed to do. It'll take an afternoon
and it will be shorter than you expect.

You'll find something while writing it. Everyone does; that's the AWS result and
it isn't subtle. And then you'll have the file — the one that doesn't exist for
any system you currently work on — and everything written afterwards, by you or
by a machine, can be checked against it.

The Shape of Failure: Before You Blame the AI

Don Johnson — Sat, 01 Aug 2026 21:12:51 +0000

Every automated system receives a particular shape of the world.

That shape is expressed through records, documents, events, exceptions, and missing values. If the designers have not identified those forms—and the ways they can become malformed—the machine inherits their ignorance and reproduces it at scale.

The question is not simply whether the AI failed.

The useful question is whether the human-built system knew what success meant, knew the shape of its data, and knew how to recognize when it was wrong.

Start with the shape of the data

Before selecting a model, draw the workflow as a sequence of data transformations.

What enters each stage?
In what form and from what source?
Which values are valid, absent, duplicated, stale, delayed, or contradictory?
How will each violation be detected?
What must the workflow do next?

Each data shape needs a corresponding failure model. An unknown here is not merely uncertainty for the machine; it is a measurement failure in the organization.

The remedy is to collect the missing data or explicitly design for its absence. Otherwise, the system is being asked to operate in a world its designers have not described.

Stabilize the deliverable

A system cannot be stabilized around a target that continues to move.

The deliverable must be more than an aspiration written in a prompt. It should be expressed as observable conditions and anchored to a representative corpus:

examples that are acceptable;
examples that are unacceptable;
examples that are genuinely ambiguous.

Human reviewers should first demonstrate that they can apply those distinctions consistently. If they cannot agree on what success looks like, the model is not being measured against a specification. It is being measured against human disagreement disguised as one.

The model is not the system

Only then does it become meaningful to place an AI model inside the workflow.

The model is one transformation among many:

Input
  → validation
  → retrieval
  → normalization
  → model inference
  → output validation
  → policy checks
  → human action

Each transition can discard meaning or introduce error. The fluency of the final response makes the model the most conspicuous suspect, but conspicuousness is not causality.

To assign blame intelligently, observe the entire chain and test every boundary where information is received or transformed.

Mutation testing attacks confidence itself

Conventional tests ask whether software succeeds under conditions its authors anticipated. Mutation testing reverses that pressure.

It deliberately introduces small faults—reversing a condition, moving a boundary, substituting a value, or removing an operation—and asks whether the test suite notices.

A killed mutant caused at least one test to fail.
A surviving mutant changed the program without the tests objecting.

The resulting score is not proof of correctness. It measures how sensitive the tests are to the generated changes.

In the companion implementation, the focused safety-contract run produced:

observed mutation sensitivity = \frac{43 killed}{43 generated} = 1.0

That number has meaning only because the measurement boundary is explicit: the run scores the small kernel containing the workflow's acceptance predicates. It does not pretend that prompt punctuation or CLI wording is part of the safety contract.

Why the first mutation run mattered
The first broad run generated 374 mutations across the package. Of those, 185 survived. Many changed prompt text, trace wording, and gateway plumbing that the tests did not claim to specify.

That was not an embarrassing result to conceal. It revealed an unstable definition of the deliverable. The mutation boundary was then narrowed to the safety-critical contract, and tests were strengthened around exact acceptance boundaries. The experiment therefore reenacted the article's argument: a metric becomes meaningful only after humans define what is being measured.

Carry the same pressure through the AI workflow

The mutation principle should not stop at source code.

Remove an expected field. Corrupt a format. Supply stale or contradictory source material. Interrupt retrieval. Perturb an instruction. Substitute a fluent but incorrect model response.

At every boundary, ask whether the surrounding system detects the disturbance, abstains, falls back, or routes the case to a human.

The reference implementation makes this flow explicit:

shape_failures = shape_agent.inspect(request)
if shape_failures:
    return escalate(shape_failures)

evidence, evidence_failure = evidence_agent.retrieve(request, corpus)
if evidence_failure:
    return escalate(evidence_failure)

decision = model.decide(request, evidence)
criticism = critic_agent.review(request, evidence, decision)
if criticism:
    return escalate(criticism)

return accept(decision)

The actual implementation uses four deliberately small agents:

A shape agent rejects malformed or unmodeled input.
An evidence agent retrieves the matching corpus and detects contradictions.
A model agent proposes a structured, cited decision through OpenRouter.
A critic agent verifies the decision against citations, confidence thresholds, and a computable policy oracle.

The model is useful, but it is never permitted to define its own success criteria.

What the executable experiment found

The checked local verification produced:

86 passing tests;
43 of 43 mutations killed in the safety-contract kernel;
one clean control accepted;
all seven injected data, corpus, and model faults detected;
zero model calls when bad input or contradictory evidence made inference unsafe.

Hypothesis generates both data and sequences of workflow mutations. Mutmut changes the safety-contract source. OpenRouter supplies the optional live model boundary using a strict JSON Schema response.

The OpenRouter call is intentionally optional because it uses an external service and may incur cost. The deterministic safety oracle does not depend on network access, a particular provider, or a favorable model response.

Agreement is another output, not an oracle

Once the single-model boundary was stable, I added a second weave around it. Four model families received the same structured request and the same policy evidence. Each worked independently: no model saw another model's answer. Every response then crossed the same citation checks, confidence threshold, and computable policy oracle.

The live run used:

openai/gpt-5.4-nano;
google/gemini-3.5-flash-lite;
deepseek/deepseek-v4-flash-0731;
mistralai/mistral-small-2603.

Eight scenarios produced 32 model observations. Twenty-seven matched the oracle and passed the contract, but only four scenarios achieved unanimous contract acceptance.

Scenario	OpenAI	Gemini	DeepSeek	Mistral	System
Eligible control	approve `0.78`	approve `1.0`	approve `1.0`	approve `1.0`	accept
Eligible paraphrase	approve `0.74`, rejected	approve `1.0`	approve `0.99`	approve `1.0`	review
Instruction inside customer data	approve `0.74`, rejected	approve `1.0`	approve `1.0`	escalate `0.95`	review
Last eligible day	approve `0.86`	approve `1.0`	approve `1.0`	deny `1.0`, rejected	review
Maximum eligible amount	approve `0.78`	approve `1.0`	approve `1.0`	approve `1.0`	accept
One day late	deny `0.86`	deny `1.0`	deny `1.0`	deny `1.0`	accept
One cent over	deny `0.90`	deny `1.0`	deny `1.0`	escalate `0.99`	review
Non-refundable policy	deny `0.86`	deny `1.0`	deny `1.0`	deny `1.0`	accept

This is not a leaderboard. It is one small experiment against one explicit contract. Gemini and DeepSeek passed all eight cases in this run. OpenAI selected the oracle action in all eight, but its confidence fell below the human-set 0.75 threshold twice. Mistral abstained twice and made one incorrect decision at the exact, inclusive day boundary.

Those are different failure shapes with different remedies. A wrong boundary decision challenges reasoning or prompt clarity. An abstention challenges workflow capacity and escalation cost. A low-confidence rejection challenges calibration and the threshold chosen by the organization. None can be repaired merely by declaring that three models outvoted the fourth.

Agreement measures convergence, not correctness.

If every model repeats the same unsupported answer, the deterministic contract must still reject it. If one model dissents for a defensible reason, majority voting must not erase the evidence.

The weave also exposed instability that a single run would have concealed. In an earlier pass, the same OpenAI model gave the control 0.90 confidence instead of 0.78, and escalated the one-day-late case at 0.62 instead of denying it at 0.86. Temperature zero reduced one source of variation; it did not turn a remote generative service into a mathematical function.

A provider failure is not a reasoning failure
The pilot included anthropic/claude-haiku-4.5, but that route failed at the model boundary in all eight requests. The workflow escalated without leaking upstream details. I replaced the route with Mistral for the final decision comparison rather than silently counting eight transport or adapter failures as eight reasoning failures.

That distinction is the thesis again: identify the boundary at which the failure becomes observable before deciding what deserves blame.

Different models did help us understand more—but only because the data shape, policy oracle, and rejection rules already existed. Without those controls, the experiment would have produced four persuasive rationales and no principled way to interpret their disagreement.

Run the single-model and multi-model computational expressions

When is it truly an AI failure?

Only after this discipline has been applied does the phrase AI failure acquire useful meaning.

If the input was well formed, the deliverable was stable, the supporting components behaved correctly, and the model still violated a known requirement, then the model failed within the conditions established for it.

But if the data was absent, the target disputed, retrieval defective, or safeguards unable to recognize error, the model may merely be the place where an earlier failure became visible. Calling that an AI failure does not improve the system. It interrupts the feedback by which the organization might learn.

A generative model is an engine of learned resemblance. It can produce behavior that looks remarkably like understanding, but resemblance cannot carry responsibility. People choose the objective, define acceptable evidence, engineer the transitions, and decide what happens when uncertainty enters the system.

Before declaring that the AI failed, demonstrate that the system knew what success meant, knew the shape of its world, and knew how to recognize when it was wrong.

If it did not, the failure began long before the model answered.

The complete experiment includes the agent weave, independent multi-model comparison, property-based tests, stateful tests, mutation configuration, deterministic fault-injection CLI, verification record, and OpenRouter gateway.

I Had a Lot of Fun Building a Linux Packet Flight Recorder

Don Johnson — Tue, 28 Jul 2026 00:05:35 +0000

Some projects begin with a roadmap. skbx began with me falling into a Linux
networking rabbit hole and enjoying it much more than I expected.

The question was simple:

Where did this packet actually go?

The answer was not.

A packet can pass through network namespaces, routing decisions, Netfilter,
traffic control, XDP programs, tunnels, clones, copies, and drop paths. A
packet capture at one interface can be completely correct while still showing
only one part of that journey.

I wanted to see more of the journey. Then I wanted to save what I saw, replay
it later without root, and know whether the capture itself had lost
observations.

That became skbx.

The short version

skbx is a Linux packet-path flight recorder built with Rust and CO-RE eBPF.

During a live capture, it observes kernel networking functions and writes an
append-only JSONL evidence stream. Afterward, that stream can be replayed and
inspected without loading another eBPF program.

live traffic
    ↓
CO-RE eBPF observations
    ↓
bounded traceq JSONL
    ↓
replay → route patterns → explain an event

It can observe:

kernel functions handling an sk_buff;
packet clones, copies, copy-on-write, and XDP-to-SKB transitions;
TC and XDP program entry and exit;
tunnels and inner packet tuples;
kernel-reported drop reasons;
selected BPF helper and map activity;
capture loss, decoding failures, and output failures.

This is not a replacement for tcpdump or Wireshark. Those tools are excellent
when the question is about packet bytes and protocols at a capture point.
skbx is for questions about the path through the local Linux kernel.

It is also not the first tool to follow packets through that path. The project
is explicitly inspired by
pwru, which showed how useful broad eBPF
packet tracing can be.

The part I especially wanted to explore was the evidence after the live
terminal stopped scrolling.

Why call it a flight recorder?

Live tracing is useful, but incident work often continues after the privileged
session has ended.

Someone else may need to inspect the capture. We may want to compare it with a
successful request. We may need to cite one exact event in a bug report. An
automation or AI system may need structured input, but it should not be allowed
to invent observations that were never captured.

So the native skbx stream, called traceq, has an envelope:

capture_start
event
event
...
capture_end

Every event receives a stable handle:

event:111111111111111111111111

Replay groups ordered events into bounded route patterns, which receive their
own handles:

route:1c0201e74424e253f8363577

The footer matters as much as the events. It records the stop reason and
reliability counters for kernel reservation failures, tracer recursion misses,
read failures, userspace decoding and enrichment failures, and output
failures.

If the footer is absent, the artifact is incomplete. If the tracer lost
observations, that uncertainty stays attached to the result.

I like this property because tracing software is also software. It should not
silently act omniscient.

Installing it

The live tracer currently supports Linux on x86_64 and arm64. It needs Rust
1.85 or newer, Clang/LLVM with the BPF backend, bpftool, libelf, libpcap, and
a kernel exposing /sys/kernel/btf/vmlinux.

On Ubuntu, the native packages are:

sudo apt-get install "linux-tools-$(uname -r)" \
  clang llvm libelf-dev libpcap-dev pkg-config

On Debian:

sudo apt-get install \
  bpftool clang llvm libelf-dev libpcap-dev pkg-config

Then install the CLI directly from GitHub:

cargo install --git https://github.com/copyleftdev/skbx --locked skbx-cli

Before attaching anything, doctor checks the host:

skbx doctor --json

plan goes one step further. It shows exactly which functions would be
attached without performing the attachment:

skbx plan --filter-func 'ip.*' --json

That separation turned out to be useful while building the project. A missing
kernel capability should be visible before a privileged capture begins, not
silently approximated afterward.

Capturing a first packet

Here is a small ICMP capture:

sudo skbx capture --probe ip_rcv \
  --duration 10 \
  --output trace.jsonl \
  icmp

While it runs, create some traffic in another terminal:

ping -c 3 1.1.1.1

The capture is deliberately bounded. It has a finite duration and event limit
instead of assuming that an unbounded trace is safe.

After it finishes, replay does not require root:

skbx replay trace.jsonl

Replay summarizes the functions, processes, packet identities, route patterns,
consensus, outliers, and reliability state. Given the same valid JSONL input,
it produces a byte-identical summary.

If an event looks interesting, retrieve it with its surrounding same-packet
evidence:

skbx explain trace.jsonl event:<handle>

The repository includes a small contract fixture containing an ip_rcv event
followed by kfree_skb_reason. Replaying that fixture produces a route shaped
like this:

{
  "handle": "route:1c0201e74424e253f8363577",
  "functions": [
    "ip_rcv",
    "kfree_skb_reason"
  ],
  "routes": 1,
  "truncated": false,
  "outlier": false
}

That output was generated by the current CLI from the checked-in test fixture;
it is not illustrative pseudodata.

Use case: what happened to a website request?

This is the use case I keep returning to.

Suppose curl starts an HTTPS request and eventually times out. A capture on
the client can answer questions such as:

Did the request enter the local TCP and IP output paths?
Which interfaces and namespaces were associated with it?
Was its mark changed?
Did a TC or XDP program handle it?
Was it transformed or placed into a tunnel?
Did the local kernel report a drop?

That evidence can clear or implicate the client host.

It cannot prove what happened inside the ISP or on the target server. To prove
those parts, we need observations from those vantage points. That limitation
is important: the absence of a local drop is not proof that a remote host
received the packet.

The useful result is a smaller, evidence-backed search area:

client host evidence
        ↓
observed departure boundary
        ↓
ISP / transit inference
        ↓
target-host evidence, if available

I would rather know exactly where my evidence ends than have a tool produce a
confident story about systems it could not observe.

Use case: a packet disappears inside the host

“The network dropped it” can hide several very different failures.

A host may reject a packet through a normal kernel path. A TC classifier or
XDP program may return a drop action. A packet may be consumed during teardown.
A transformation may cause the packet we were following to continue under a
different kernel object.

For focused TCP drop questions, a small bpftrace program may be the fastest
answer. For packet contents, I still reach for tcpdump or Wireshark.

I reach for skbx when I do not yet know which local subsystem owns the
failure, or when I want to preserve the ordered path rather than one drop
event.

Use case: follow transformations instead of losing the trail

Linux does not promise that one logical packet corresponds to one
struct sk_buff address for its entire lifetime.

Packets can be cloned, copied, or changed through copy-on-write. XDP frames can
later become SKBs. Tunnels introduce outer and inner packet identities.

skbx uses bounded, capture-local lineage state to connect those transitions.
The provenance remains explicit: an event records whether it matched the
original filter or was included because it belonged to a packet already being
tracked.

For example, a marked packet can be followed with:

sudo skbx capture \
  --filter-mark 0x2a \
  --filter-track-skb \
  --output trace.jsonl

The important word here is capture-local. A lineage identifier is not a
distributed trace ID, and it should not be presented as one.

Use case: capture once, investigate without root

Loading and attaching eBPF programs is privileged work. Reading a JSONL file
does not need to be.

That makes a simple handoff possible:

An operator checks the plan and performs a bounded capture.
The capture ends and writes its reliability footer.
The artifact is copied to a normal development or analysis environment.
Another person replays it and cites exact event: or route: handles.

This is useful for incident handoffs and bug reports, but it is also useful for
learning. I can capture a small experiment once and inspect it repeatedly
without keeping probes attached while I try to understand the result.

Use case: give an AI evidence instead of a mystery

One of my stranger motivations was making the CLI friendly to both operators
and software agents.

Before touching the host, an agent can ask the executable what it supports:

skbx describe --format json
skbx schema
skbx doctor --json
skbx plan --json

The native engine remains the source of truth. An AI system may summarize a
route or explain a captured event, but the event must already exist in the
trace. Machine output stays on standard output, diagnostics stay on standard
error, and unsupported capabilities remain explicit.

This does not make an AI explanation automatically correct. It gives the
explanation something concrete to cite and gives a human a way to check it.

The engineering parts I enjoyed

The obvious fun was seeing packets move through functions I had previously
treated as boxes in a diagram.

The less obvious fun was designing the boundaries:

discovering valid struct sk_buff * arguments from the target kernel's BTF instead of guessing from function names;
keeping kernel-side state and work bounded;
moving JSON encoding, symbolization, and filesystem work out of the eBPF hot path;
making probe planning inspectable before attachment;
separating raw observation from later explanation;
treating partial output as incomplete instead of “probably good enough”;
making replay deterministic enough to use in tests and automation.

Those constraints made the tool more interesting to build. They also gave me a
better mental model of what an observability tool can honestly claim.

What it does not know

skbx only observes what the running kernel and selected probes expose.

It does not see inside an ISP. It does not see a remote host unless it runs
there too. It is not a packet-content replacement for Wireshark. Kernel
configuration, BTF availability, permissions, hidden addresses, unsupported
signatures, and capture loss can all limit the evidence.

Those limitations are part of the interface rather than footnotes.

I am still exploring where this project is most useful. The best next input is
not “looks cool,” although I will happily take that. It is a packet path the
tool cannot explain yet, together with the kernel version, exact command,
doctor --json output, and reliability footer.

The source, installation instructions, architecture notes, and field guides
are in the repository:

copyleftdev / skbx

Agent-first Linux packet-path tracing with Rust/eBPF: bounded evidence, deterministic replay, and packet routes with receipts.

skbx

skbx shows where a packet went inside Linux—and keeps the receipts.

Follow a packet through the flight recorder →

Field guides:

It observes kernel networking functions, TC/XDP programs, packet transformations, tunnels, drops, and selected BPF helper activity with Rust and CO-RE eBPF. Every observation lands in a bounded, replayable evidence stream with stable handles and explicit loss telemetry.

Use it when “the packet disappeared” is not a sufficient incident report.

capture → event:8c6f… → replay → route:21b4… → explain
            evidence          pattern          context

Inspired by pwru, rebuilt around an agent-first contract: deterministic observations, machine-readable capabilities, bounded state, and no invented evidence.

The short version

You need to…	skbx gives you…
See the kernel path of an SKB	BTF-discovered kprobes with exact function evidence
Follow clones, copies, COW, and XDP-to-SKB

…

View on GitHub

If you try it, I would love to know which networking question you used it to
answer—and where its evidence stopped.

AI-assistance disclosure: I built and tested skbx. I used OpenAI Codex to
research adjacent writing, challenge the article's positioning, edit this
draft, verify its commands and example output against the repository, and
generate the cover illustration. I reviewed the technical claims and remain
responsible for their accuracy.

389 Tests Passed. NIST Still Caught the Bug.

Don Johnson — Sat, 25 Jul 2026 06:17:51 +0000

I gave an AI agent a calculator because I wanted one hard, inspectable point
inside a probabilistic workflow.

The model could interpret the request and explain the result. The calculator
would perform the computation. It seemed like a clean division of labor.

Then I changed one multiplication sign into addition.

The calculator still passed 389 of the 390 tests in its Rust library harness.
The sole failure compared its answer with NIST's certified results for the
Longley regression dataset.

That bothered me more than a completely broken build would have. I had treated
deterministic computation as safer than asking a language model to improvise
arithmetic. But deterministic does not mean trustworthy. A program can return
the same wrong answer forever.

“Source of truth” suddenly felt too comfortable. Before an AI agent delegates
authority to a tool, that authority should be challenged—and remain revocable
by evidence.

The calculator is only the specimen. The larger idea is a way to place
inspectable, replayable instruments inside probabilistic systems.

The useful boundary is generation versus execution

The interesting distinction is not model weights versus a “real CPU.” Model
inference also runs on processors, and language models can learn genuine
arithmetic procedures. The useful boundary is between generating an answer
and executing a defined operation under a tested contract.

Research on Program-Aided Language Models
(PAL) makes a related split:
the language model reads and decomposes a natural-language problem, while a
runtime such as a Python interpreter executes the generated program. The model
contributes flexible interpretation; the runtime contributes executable
semantics.

That is the division I want in an agent:

At the semantic edge, the model interprets the request, chooses a procedure, identifies relevant quantities, and explains the result.
At the computational edge, a narrow tool validates inputs, applies specified operations, enforces limits, and returns structured output.

This does not make the whole agent deterministic, and it does not make the
model unnecessary. The agent can still choose the wrong tool, supply the wrong
arguments, misunderstand units, or misread the result.

The promise is smaller: one claim becomes inspectable, replayable, and
independently testable.

The opposite of probabilistic is not trustworthy. It is repeatable.

A CPU can be precisely wrong.

Why NIST became my external witness

Calibration cannot be entirely self-referential. The implementation should not
be the sole author of its own expected answers.

That is why I chose NIST—not because government authority turns a result into
mathematical truth, but because NIST has a long institutional practice of
building shared, independently evaluated references.

Congress established the National Bureau of Standards in
1901 to
strengthen the United States' measurement infrastructure. The Standard
Reference Data Act of
1968 authorized a federal program to
collect, critically evaluate, publish, and distribute standardized scientific
and technical reference data. NBS became NIST in
1988. In 1999, that lineage reached statistical
software through the Statistical Reference Datasets
project, usually shortened to StRD.

StRD pairs datasets with certified expected values for specific statistical
procedures. Its collection includes generated and real-world cases of varying
difficulty. For linear procedures, NIST carried 500 digits through its
calculations
so ordinary floating-point representation error would not become the
benchmark.

Longley is a small but numerically challenging linear-regression dataset in
that collection. Its certified results gave my tests something the
implementation could not manufacture for itself: an expected answer produced
outside the code under test.

That qualification matters. StRD does not certify Oddly Exact, prove the
statistics engine correct, or make the software traceable to NIST. NIST
explicitly says the datasets are an aid for evaluating software and that no
mechanism establishes software
traceability.

The reference data was not an oracle for the entire program. It was an
independent witness for the calculations it covered.

Test the tests

Example-based tests ask whether familiar inputs still produce familiar
outputs. Mutation testing asks a more uncomfortable question:

If I introduce a small, plausible defect, does the suite notice?

I used cargo-mutants to replace multiplication with addition in the
multiple-regression standard-error calculation:

- residual_std_dev * sum_sq.sqrt()
+ residual_std_dev + sum_sq.sqrt()

The mutation preserved valid Rust, valid types, and a plausible-looking numeric
result. In the library harness, 389 of 390 tests still passed. The assertion
against NIST's Longley values was the sole failure.

The public mutation
record
preserves the exact command, environment, result, and the limit of what that
experiment establishes:

I cannot claim Longley was the only test in the entire repository capable of
catching the mutation; the run stopped after the failed library harness. I can
claim something narrower and more useful: hundreds of tests tolerated a
semantically broken formula, while an assertion anchored to independently
produced values rejected it.

A green test count is evidence only to the extent that those tests would turn
red when the implementation meaningfully changes. Mutation testing measures
that sensitivity instead of admiring the count.

Challenge the contract around the answer

The formula mutation challenged numerical meaning. My next audit challenged
the promises around the calculation: what the tool accepts, what it refuses,
and how much work it will perform.

The first crack was a contradiction between the advertised contract and the
executable one. The generated JSON Schema forbade additional properties, but
the Rust deserializer silently accepted an unknown field at the request root
and another inside an expression node.

The answer was still numerically correct. That did not make the behavior
harmless. A caller validating against the schema saw a stricter instrument than
a caller speaking directly to the binary. A misspelled or misunderstood field
could disappear without warning.

For an agent-facing tool, silently interpreting a different contract is itself
a correctness defect.

The second crack was a resource boundary. The expression engine limited things
such as integer size, expression depth, and precision. The optimization
interface, however, accepted grid resolution and iteration counts without upper
ceilings.

I requested a grid search with one billion sample points. The pre-repair binary
produced no JSON before an external one-second watchdog terminated it. The tool
had input validation, but it had not earned the bounded-work guarantee I
thought I had built.

Those failures became permanent regression requests. The deserializer now
rejects unknown fields at both levels. The optimizer now publishes and enforces
ceilings of 100,000 iterations and 1,000,000 grid points. The same billion-point
request returns a typed resource_limit response before entering the search.

The contract challenge
Gist
is a runnable replay of all three repaired cases:

That script is more valuable than a screenshot of a green run. It lets another
observer cross-examine the boundary directly.

Repair is the next claim to attack

A new guard is only another claim until the tests prove they care about it.

After repairing the optimizer limits, I selected every mutation
cargo-mutants generated for the two new validators. Eleven mutations tried to
remove the checks, replace them with unconditional success, or alter their
boundary comparisons. The tests caught all eleven.

The focused optimizer mutation
record
captures the selection and outcome:

That completed the loop:

claim → attack → fail → repair → mutate → replay

If I published only the final green suite, you would see confidence. By
preserving the changed operator, the request that hung, the contract
contradiction, the repair, and the mutations that tried to undo it, I can show
a reason for confidence.

The scar is part of the calibration record.

A five-layer challenge stack for agent tools

Oddly Exact is one calculator, but the method travels. Before giving a narrow
tool authority inside an agent workflow, I now ask five questions:

Is the contract explicit and executable? The schema, parser, runtime,
limits, and failure modes must agree. Documentation the executable does not
enforce is only a suggestion.
Is there an independent reference? In this statistical case, NIST StRD
moved selected expected answers outside my implementation. Another domain
might use a standards specification, a reference implementation, a verified
corpus, or a separately derived test oracle.
Which properties should survive across many inputs? Property-based tests
generate cases and check laws rather than memorizing individual examples.
One Oddly Exact property checks that translating every value in a sample by
a large exact offset leaves sample variance unchanged within a tight
floating-point tolerance.
Can the tests detect plausible corruption? Mutation testing changes
operators, comparisons, return values, and guards. A surviving mutation does
not prove the implementation is wrong; it exposes a behavior change the
suite cannot currently distinguish.
Are failures bounded, typed, and replayable? An agent must distinguish a
rejected computation from an unavailable tool. Division by zero, resource
exhaustion, malformed JSON, and a crashed process should not collapse into
the same ambiguous failure.

These layers do different jobs. External references challenge specific
answers. Properties challenge behavior across an input space. Mutations
challenge the tests themselves. Contract and resource attacks challenge the
boundary around all of it.

None provides universal correctness. Together they create a narrower and more
useful result: evidence that a particular tool deserves provisional authority
for validated requests inside a declared contract.

A source under challenge

For a validated request inside that contract, the tool's structured result can
serve as the agent's operational source of truth.

That is a runtime role, not a claim of infallibility. If the agent encounters a
disagreement, it can inspect the arguments, surface the conflict, or consult
another independently trusted tool. It should not quietly replace a structured
result with fresh prose arithmetic.

The trust relationship runs in opposite directions at different times:

While building the tool, the developer should distrust it aggressively.
Inside a contract the tool has earned, the agent should respect its answer or its typed refusal.

New evidence can always return the tool to the first phase. That is a source
under challenge: trusted in operation, open to appeal, and always one
counterexample away from revision.

The beautiful thing is not that source code is truth.

The beautiful thing is that source code can be cross-examined.

I did not end this experiment with a calculator an AI agent can trust forever.
I ended it with something more useful: a calculator whose authority can be
revoked by evidence.

AI tools assisted with editorial research and revision. I verified the
technical claims and stand behind the final text.

Give Your Coding Agent a Deterministic Vulnerability Oracle

Don Johnson — Tue, 21 Jul 2026 15:04:00 +0000

AI agents can write code, run tests, inspect dependencies, and open pull requests. But when they encounter a vulnerable package, their security reasoning often collapses into a web search, an opaque API score, or whatever the model remembers from training.

The problem is not simply missing vulnerability data. It is the contract between the agent and its harness—the automation layer that supplies tools, executes commands, and interprets results. If “no match” silently becomes “safe,” stale intelligence looks current, or a network failure resembles an empty result, the agent can produce a confident answer without trustworthy evidence.

I designed VulnGraph around a different premise: turn continuously changing vulnerability intelligence into a deterministic local primitive. vulngraph-data compiles upstream security sources into verified, content-addressed snapshots. vulngraph-cli installs those snapshots and checks CVEs, package versions, or entire lockfiles offline.

The result is a security tool shaped for both humans and machines: an accepted snapshot and target produce the same verdict; every verdict carries typed evidence; stale data fails explicitly; and unknown is never misrepresented as clean.

This article explains the design decisions behind that contract and shows how to embed VulnGraph into agent and harness workflows.

The dangerous gap between “not found” and “safe”

Suppose an agent is reviewing a dependency update. It needs to answer a seemingly simple question: Is this version safe to ship? A useful answer depends on distinctions that are easy for a harness to erase.

What happened	A careless harness concludes	What the contract must preserve
The package exists and the version is outside every affected range	No vulnerabilities found	`not-affected`
The package is absent from the dataset	No vulnerabilities found	`unknown`
The local snapshot is too old	The last result is probably fine	A freshness failure
The tool cannot open or verify its data	Empty result	An operational or integrity error

These are not cosmetic labels. They lead to different agent actions: proceed, investigate, refresh the data, or stop the workflow. Once a harness flattens them into a Boolean, the model cannot recover the missing meaning through better prompting.

Here is the distinction in practice. One command checks an exploited CVE, a real package version, and a package the snapshot has never observed:

$ vulngraph check CVE-2024-4577 npm:lodash@4.17.15 npm:no-such-package@1.0.0

  CVE-2024-4577
  verdict: ACTIVELY EXPLOITED  (confidence 0.99)
  action:  patch now
  reasons: CRITICAL_SEVERITY, HIGH_EXPLOIT_PROBABILITY,
           KNOWN_EXPLOITED, PUBLIC_EXPLOIT, ...
  cvss:    9.8
  epss:    100.0%
  kev:     listed

  npm:lodash@4.17.15
  verdict: PROOF OF CONCEPT  (confidence 0.96)
  action:  prioritize
  package: npm:lodash — 6 CVE(s) affect this version

  npm:no-such-package@1.0.0
  verdict: UNKNOWN  (confidence 0.00)
  action:  investigate
  reasons: NOT_OBSERVED

  snapshot: sha256:41ee5fbbf58f0a46b99234af89c5388e5f27e0dcd3dbd52298bcd2c2d87ca90e

The output does not ask the agent to infer policy from prose. It supplies a disposition, an action, stable reason codes, confidence, and the exact snapshot that produced the result. The human-readable rendering is useful in a terminal; --json exposes the same distinctions through a versioned machine envelope.

Split changing intelligence from deterministic execution

Vulnerability intelligence changes every day. Agent execution still needs to be reproducible. I separated those concerns instead of allowing every check to fetch and reconcile live data.

CVE List V5 · EPSS · CISA KEV · exploit sources · ATT&CK · OSV · …
                              │
                              ▼
                    vulngraph-data
              fetch → normalize → build → verify
                              │
                              ▼
            immutable data-YYYYMMDD release
       manifest + checksums + content snapshot_id
                              │
                  vulngraph update
                              ▼
             verified local snapshot
                              │
          vulngraph check --offline --json
                              ▼
                  agent or harness

The vulngraph-data repository owns the nondeterministic edge: retrieving bulk publications from upstream sources and reconciling them into one graph. Its output is deterministic. Identical inputs produce byte-identical semantic files, and their hashes produce a snapshot_id that names the graph by content. If a daily build has no semantic change, there is no new release.

The release artifact is the distribution boundary. A consumer does not need the raw source collection or the machinery that built it. It receives a fixed database archive and a manifest recording the snapshot identity, file hashes, source freshness, graph format, and engine revision.

The vulngraph-cli owns the deterministic side. vulngraph update downloads a release into a staging area, verifies the archive checksum and every file hash, recomputes the snapshot identity, checks format compatibility, sanity-opens the graph, and only then activates it. Failed updates cannot replace the last verified snapshot.

After activation, checks do not touch the network. The snapshot becomes an explicit input to the decision:

verdict = policy(snapshot_id, target)

There is no wall clock, random value, model call, or remote response inside that function. Once a snapshot passes the freshness and integrity preflight, the same snapshot and target give a developer laptop, a CI job, and an agent sandbox the same verdict.

A harness workflow that does not guess

A reliable integration has three phases. Only the update phase needs network access; discovery and checks are stable interfaces that the harness can validate.

1. Discover the contract

Do not make the agent reverse-engineer help text. Let the harness inspect the machine-readable capability document and the bundled JSON Schemas:

vulngraph --json capabilities
vulngraph schema command
vulngraph schema observation
vulngraph schema status

capabilities declares the commands, output schema versions, exit codes, and two behavioral guarantees: checks are offline and deterministic. A harness can validate support during startup instead of discovering an incompatibility halfway through a run.

2. Refresh and preflight outside the restricted step

Install or refresh the snapshot in a setup job that is allowed to use the network:

vulngraph update
vulngraph --json status

Then mount or cache the resulting VulnGraph home directory inside the agent environment. This makes network access a controlled data-provisioning concern instead of an ambient capability available during every security decision.

status reports whether a snapshot is installed, its identity, its integrity, and its freshness. A dataset older than fourteen days is not merely accompanied by a warning: check refuses to answer and exits with code 4.

3. Check the artifact the agent is actually changing

The check command accepts individual CVEs, package coordinates, and dependency files. Pointing it at a lockfile expands every dependency and applies the same version-range policy:

vulngraph --offline --json check package-lock.json > vulngraph.json

The CLI recognizes lockfiles from npm, Yarn, pnpm, Cargo, Bundler, Poetry, pip, Go, Composer, Maven, Gradle, and Pipenv. The abridged JSON response below shows the stable envelope:

{
  "schema": "vulngraph.command.v1",
  "command": "check",
  "ok": true,
  "partial": false,
  "snapshot_id": "sha256:41ee5fbb…",
  "data": [
    {
      "schema": "vulngraph.observation.v1",
      "target": { "type": "package", "value": "npm:lodash@4.17.15" },
      "verdict": {
        "disposition": "proof-of-concept",
        "confidence": 0.96,
        "action": "prioritize",
        "reason_codes": [
          "ATTACK_MAPPED",
          "HIGH_SEVERITY",
          "MULTISOURCE_CORROBORATION",
          "PUBLIC_EXPLOIT",
          "SEVERITY_SCORED",
          "VERSION_IN_AFFECTED_RANGE",
          "WEAKNESS_CLASSIFIED"
        ]
      }
    }
  ],
  "warnings": [],
  "errors": [],
  "metrics": { "elapsed_us": 781250 }
}

The important separation is that VulnGraph observes; it does not gate. A successful check exits with 0 even when it finds an actively exploited vulnerability. Exit codes describe whether the tool completed reliably; the harness applies organizational policy to the returned dispositions.

Disposition	Evidence	Sensible default harness action
`actively-exploited`	Listed in CISA KEV	Block or require an explicit emergency exception
`weaponized`	Public exploit and very high EPSS	Block or require security approval
`proof-of-concept`	At least one public exploit	Prioritize remediation and require review
`scored`	CVSS or EPSS evidence, no public exploit	Apply the repository's severity policy
`recorded`	Known record without stronger risk signals	Monitor and retain the evidence
`not-affected`	Known package outside all affected ranges	Proceed for this snapshot
`unknown`	Target absent from the snapshot	Investigate; never translate to clean

This split keeps probabilistic reasoning in the right place. The harness deterministically decides which states require a stop, review, or escalation. The agent then uses the evidence to explain the finding, inspect reachability, propose the smallest safe upgrade, and communicate the trade-off to a human.

A minimal Node.js harness

The following wrapper turns the versioned VulnGraph envelope into one of five local workflow decisions. It does not ask the model whether an operational failure is acceptable, and it never treats an unrecognized disposition as approval.

import { spawnSync } from "node:child_process";

const binary = process.env.VULNGRAPH_BIN ?? "vulngraph";
const target = process.argv[2] ?? "package-lock.json";
const run = spawnSync(
  binary,
  ["--offline", "--json", "check", target],
  { encoding: "utf8" }
);

let envelope;
try {
  envelope = JSON.parse(run.stdout);
} catch {
  throw new Error(`vulngraph returned invalid JSON: ${run.stderr}`);
}

// Exit codes describe tool reliability, not vulnerability severity.
if (run.status === 4) {
  throw new Error("VulnGraph snapshot is stale; refresh before continuing");
}
if (run.status !== 0 || !envelope.ok) {
  throw new Error(`VulnGraph failed: ${JSON.stringify(envelope.errors)}`);
}
if (envelope.schema !== "vulngraph.command.v1") {
  throw new Error(`Unsupported schema: ${envelope.schema}`);
}

const policy = new Map([
  ["actively-exploited", "block"],
  ["weaponized", "block"],
  ["proof-of-concept", "review"],
  ["scored", "review"],
  ["recorded", "monitor"],
  ["not-affected", "proceed"],
  ["unknown", "investigate"]
]);

const decisions = envelope.data.map(({ target, verdict }) => ({
  target: target.value,
  disposition: verdict.disposition,
  decision: policy.get(verdict.disposition) ?? "investigate",
  action: verdict.action,
  confidence: verdict.confidence,
  reasons: verdict.reason_codes
}));

const evidence = {
  snapshotId: envelope.snapshot_id,
  decisions
};

console.log(JSON.stringify(evidence, null, 2));

if (decisions.some(({ decision }) => decision === "block")) {
  process.exitCode = 10; // This is our harness policy, not a VulnGraph exit code.
}

The evidence object is what I would place in the agent's context. The prompt can remain narrow:

Review the attached vulnerability evidence for this dependency change.

- Never describe `unknown` as safe or clean.
- Preserve the snapshot ID in your report.
- Explain the reason codes in plain language.
- Check whether each affected dependency is reachable from the changed code.
- Propose the smallest compatible upgrade, but do not edit files until approved.

The deterministic layer establishes what was observed and which policy branch applies. The agent contributes the work it is good at: repository-specific investigation, explanation, remediation planning, and communication.

Design lessons for agent-facing security tools

Building the graph was only half the problem. Making it safe for automation required treating the interface as part of the security model.

Make ignorance a first-class result. unknown must survive every layer from data lookup to terminal output to agent context.
Return evidence, not just a score. A verdict should carry stable reasons and source observations that a human can audit.
Separate update time from decision time. Network retrieval belongs in a controlled provisioning phase; checks should be local and reproducible.
Identify the data behind every answer. A content-derived snapshot ID turns “what did the scanner know?” into an answerable question.
Let tools describe their contract. Capabilities, schemas, freshness, and exit semantics should be machine-readable.
Keep gating policy outside the observation tool. Different repositories have different risk tolerances. The harness should own that policy explicitly.

An agent does not become trustworthy because it receives more tokens or a longer system prompt. It becomes more trustworthy when its environment preserves distinctions, constrains side effects, and supplies evidence through contracts that cannot silently change meaning.

That is the role I designed VulnGraph to play: not an autonomous security authority, but a fast, local, auditable source of vulnerability evidence that gives agents and their harnesses something solid to reason from.

Build and release pipeline: copyleftdev/vulngraph-data
Offline CLI: copyleftdev/vulngraph-cli

Disclosure: I designed VulnGraph. This article and its cover were created with AI assistance. I reviewed the technical claims against the linked repositories and live CLI output.

If this open-source work is useful to you, you can support its continued development:

[Boost]

Don Johnson — Sat, 18 Jul 2026 17:47:12 +0000

Don Johnson

Jul 17

The Last Honest Abstraction: Why AI Coding Isn't the End of Engineering

#discuss #ai #programming #softwaredevelopment

5 min read

The Last Honest Abstraction: Why AI Coding Isn't the End of Engineering

Don Johnson — Fri, 17 Jul 2026 16:47:19 +0000

Why every generation thinks the next programmer understands less—and why AI changes the burden of proof, not the need for engineering.

"Developers can't even explain their own code anymore."

If you've spent any time around software lately, you've probably seen some version of that argument.

The culprit, we're told, is AI.

Developers are prompting language models, accepting generated code, and shipping software they couldn't possibly explain line by line. Somewhere between autocomplete and autonomous agents, we supposedly crossed a professional line. We traded craftsmanship for convenience.

It sounds like a new argument.

It's one of the oldest in computing.

Before We Go Any Further...

Before I make my case, I want to pay homage to a rare breed of engineer.

You know who you are.

You're the people who can look at compiler output and immediately tell when an optimization didn't quite land. You read assembly like most people read JavaScript. You understand calling conventions, cache locality, virtual memory, branch prediction, linker behavior, and processor architecture—and you can still build elegant software without turning every project into a dissertation.

You're rare.

And you've quietly carried this industry for decades.

Many of the abstractions the rest of us rely on exist because someone like you spent years wrestling with complexity until everyone else could work one layer higher.

Every generation has its handful of engineers who descend into the deepest layers of the machine, then return carrying tools that allow the rest of us to build something higher.

This article isn't arguing against mastery.

It's a thank-you to the people who made abstraction possible.

Every Layer Was Once Considered Cheating

Again and again, computing has moved upward by adding layers.

Machine code gave way to assembly for most programming. Someone inevitably complained that assembly programmers weren't "real programmers" because they no longer had to manipulate raw binary.

Higher-level languages followed. Compilers hid more of what the processor was actually doing. Managed memory, virtual machines, dynamic languages, frameworks, cloud platforms, containers, and orchestration systems each took another category of complexity and moved it behind an interface.

And now...

AI.

The technologies aren't a single, tidy lineage. They abstract different problems, and old layers rarely disappear. But the cultural pattern repeats:

"Do they really understand what's happening underneath?"

The nouns change.

The complaint doesn't.

The Myth of the Complete Programmer

Here's an uncomfortable truth.

Most software has never been written by people who fully understood every layer beneath it.

That isn't criticism.

It's reality.

Ask an application developer to explain exactly how their operating system schedules threads.

Ask them why their compiler emitted a particular optimization.

Ask them to walk through every page-table translation.

Ask them to trace how cache behavior affected one production slowdown.

Ask them to explain every TCP retransmission or every filesystem recovery after an unexpected shutdown.

Some can.

Most can't.

And yet...

The software still ships.

Not because understanding stopped mattering, but because software engineering evolved around abstractions with bounded responsibilities and increasingly testable guarantees.

The Entire Purpose of Engineering

Engineering has never been about refusing abstraction.

It's been about earning it.

Every abstraction represents an enormous investment of human understanding. Someone had to master the ugly details so the next engineer could focus on solving a different problem.

A civil engineer need not manufacture every bolt before designing a bridge, but must know which loads it can bear. Software abstractions work the same way: we delegate construction, not accountability.

Requiring every software engineer to hand-write machine code before building a web application wouldn't protect craftsmanship.

Progress wouldn't slow down.

It would stop.

Abstractions aren't shortcuts.

They're accumulated knowledge.

They're understanding, compressed into reusable form.

That's not laziness.

That's civilization.

What AI Actually Changed

AI did change something.

Just not what many people think.

A conventional compiler is expected to produce reproducible output from fixed inputs. A language model samples from probabilities. A compiler hides implementation behind a defined system; a model proposes implementation without guaranteeing that the proposal is correct.

That makes AI a less trustworthy abstraction by default—not an invalid one. It means the engineer must supply the trust boundary through review, tests, observability, constraints, and a willingness to reject plausible-looking output.

The real shift is that authorship is becoming an even weaker proxy for understanding—and an even less useful signal of correctness.

For decades, many engineers subconsciously equated "I wrote it" with "I understand it."

Those were never the same thing.

We've all inherited libraries. We've all trusted frameworks. We've all depended on operating systems we didn't write. We've all deployed software built on millions of lines of code authored by strangers.

AI simply made that reality impossible to ignore.

The responsibility hasn't disappeared.

It's moved.

Instead of taking familiarity as evidence because you typed every character yourself, you establish confidence by inspecting the design, challenging assumptions, measuring behavior, and validating outcomes.

This doesn't absolve the engineer of understanding. It changes the required depth. You may not need to explain every generated line from memory, but you do need to understand the system's boundaries, invariants, dependencies, and failure modes well enough to be accountable for what it does.

That's engineering.

Judgment Is Becoming the Scarce Resource

The best engineers I've met have never impressed me with how quickly they type.

They impress me with how quickly they notice something feels wrong.

They can review a thousand lines and, through experience, stop at the seven that deserve attention. They know where abstractions leak. They know when benchmarks are lying. They know when a beautiful architecture is solving yesterday's problem.

They ask better questions than everyone else.

AI doesn't replace that.

If anything, it magnifies its importance.

Syntax gets cheaper.

Judgment becomes priceless.

Standing on Invisible Giants

There's a beautiful irony here.

The engineers most qualified to criticize abstraction are often the same people who built it: compiler engineers, kernel developers, database architects, networking pioneers, language designers, and chip architects.

They spent years making impossibly complicated systems disappear behind clean interfaces—not because they wanted everyone staring at the gears, but because they wanted the rest of us to keep building.

Every abstraction they created became another floor in a building nobody could have constructed alone.

The Last Honest Abstraction

Every generation believes the abstraction immediately beneath them was the last honest one.

Assembly programmers point to machine code.

C programmers point to assembly.

Framework skeptics point to handwritten C.

Now AI skeptics point to handwritten code.

Ten years from now, someone will publish an article insisting developers don't even understand what their autonomous software factories are doing.

Someone else will nod and say:

"Back in my day, we actually wrote the prompts ourselves."

And the cycle will begin again.

Because the history of software has never been the history of losing understanding.

It's been the history of redistributing it.

A relatively small number of extraordinary engineers venture into the deepest layers, wrestle complexity into submission, and transform hard-earned understanding into abstractions that millions of others can build upon without mastering every detail beneath them.

Those abstractions aren't evidence that engineering is dying.

They're evidence that engineering succeeded.

Progress has never required everyone to understand everything.

It has always required enough people to understand each layer deeply enough to build the next one—and everyone who uses that layer to understand its promises and failure modes well enough to take responsibility for the result.

The rest of us stand on their shoulders.

And someday, if we do our jobs well enough, someone else will stand on ours.

Editorial note: This article was refined with AI-assisted editorial review, and its cover illustration was generated with AI. The argument, factual review, and final wording remain the author's responsibility.

A Vibe Is Not a Verdict: I Built a Tool That's Allowed to Say 'I Don't Know'

Don Johnson — Mon, 13 Jul 2026 19:37:38 +0000

A tool should do one thing, do it well, and — this is the part everyone forgets — know exactly where its knowledge ends.

I built kilo for a morning like this one. I just didn't know the morning would come this soon.

The message that was waiting for me

I woke up, poured the coffee, and did the thing you're not supposed to do before you're fully awake: I opened my messages. And there it was, sitting at the top of the inbox like it had been placed there on purpose — a cold recruiter pitch.

An "exciting opportunity" for a Quality Engineer role at an aerospace firm. Warm, professional, faintly urgent. Please click the link below to view the job description and apply. The link was a long tracking URL, the kind with an opaque token and source and method parameters stapled to the end.

And then the signature. One "recruiter" who claimed, in a single line, to be a physical therapist, a call-centre manager, an e-commerce power seller, a freight dispatcher — half a dozen unrelated trades piled into one job title, now, apparently, placing quality engineers at an aerospace company. That's not a résumé. That's a slot machine.

There it was: the exact situation I wrote the tool for, delivered to my inbox before I'd finished the coffee.

Because here's what would normally happen next. My gut says scam. My gut is usually right. But "usually right" is how you eventually click the one link that isn't. A gut feeling is a vibe, and a vibe is not a verdict. That frustration is precisely why, months ago, I sat down and wrote kilo — a way to stop feeling about an address and start knowing about it. So this morning wasn't an annoyance. It was the experiment finally running itself.

I put the coffee down and opened a terminal.

What I actually built, and why

Let me back up and tell you what kilo is, because the design is the whole argument.

KiloCheck is an offline IP-reputation engine. One binary. ~12MB. Written in Rust. Its entire worldview is a four-stage pipeline: ingest a signed, versioned threat release → normalize typed observations → compile an immutable local index → check IPs against it. Cryptographically verified going in. Zero network calls coming out.

I made it offline on purpose. An ordinary check never reaches an intelligence API. When I'm triaging something sketchy, I don't want my queries leaking, I don't want to tip anyone off, and I want the same answer at 3AM on a plane as I get at my desk. The threat knowledge is baked into a signed local snapshot. The check is just a lookup against evidence I already trust.

Install is a checksum and a copy — the way a tool should be:

curl -fsSL https://raw.githubusercontent.com/copyleftdev/kilocheck/v0.2.0/scripts/install.sh | sh
kilo update      # pulls the signed data release → prints "Claims  3160"
kilo status --json
# → "integrity": "verified", "freshness": "fresh"

That was the tool. This was the morning. Time to point one at the other.

The experiment, step one: turn the lure into a number

kilo answers exactly one question — is this IP a known bad actor? — so first I had to turn that tracking link into an address. That's dig's whole job: resolve the hostname behind the URL down to the IP actually serving it.

$ dig +short <the-redirect-host>
203.0.113.90

Then I asked my tool the question I built it to answer. This is the moment the experiment either validates the design or embarrasses me:

$ kilo check 203.0.113.90 --json | jq '.data[0].verdict'

{
  "disposition": "unknown",
  "confidence": 0.0,
  "recommended_action": "monitor",
  "reason_codes": ["NOT_OBSERVED"]
}

NOT_OBSERVED. Confidence 0.0.

And I grinned, because this is exactly the behavior I designed for — and the exact behavior a worse tool would have gotten wrong.

Think about the pressure in that moment. I clearly think it's a scam. The vibe is overwhelming. A tool built to please its operator would have felt that pressure and handed me a scary red verdict to confirm what I already believed. kilo refuses. It says: I have no evidence about this address, and I am not going to invent some to make you feel clever.

That refusal is the single most important thing I built into it. A tool has to know the difference between "safe," "bad," and "I don't know" — and it has to refuse to smear them together. Missing, unknown, stale, incomplete, unsafe: five distinct states in my model, not three shades of the same shrug. This morning it chose the honest one under maximum temptation to lie.

Step two: prove it can still bite

An honest "I don't know" is only worth something if the same tool can say a hard "yes." If kilo returned unknown for everything, it wouldn't be humble — it'd be broken. So I ran the control, right there in my pajamas: the suspicious host, a known command-and-control node, and a boring public DNS resolver, all in one shot. Distilled from the JSON:

203.0.113.90      → unknown   monitor   NOT_OBSERVED        (0 sources)
162.243.103.246   → CRITICAL  block     COMMAND_AND_CONTROL (1 source, conf 0.99)
8.8.8.8           → unknown   monitor   NOT_OBSERVED        (0 sources)

There. A known C2 node gets critical / block / 0.99 with provenance attached. The public resolver and the suspicious host both get a calm NOT_OBSERVED. No false alarms on clean infrastructure, no missed alarm on a real one. The experiment held: the tool's silence means something precisely because its alarm means something.

Step three: the twist my own tool pointed me toward

So the infra came back clean. Was my gut wrong? No — it was pointed at the wrong target. And that's the part I didn't see coming.

kilo's honest negative didn't close the case. It moved it. "The danger is not in this infrastructure" is a coordinate — it told me to go look somewhere else. So I followed the rest of the trail with the neighbors on my toolbelt.

curl followed the redirect — headers only, I never touched the body:

HTTP/2 302 → /<captcha-gate>?...&source=...&method=...

openssl read the cert straight off the wire: a real, correctly issued certificate for a legitimate, well-known job-traffic aggregator. And a quick lookup closed the loop: the aerospace company in the pitch is real, with genuine engineering openings, and the tracking link was an affiliate click-tracker redirecting through the aggregator's own anti-click-fraud gate. No payload. No credential harvest. No bad IP anywhere in the chain.

So what was the actual threat? It was never in the packets. It was the sender — a freelance lead-gen spammer blasting affiliate commission links across social platforms, wearing a recruiter costume stitched together from half a dozen unrelated careers.

And here's why I'm writing this at all: my own tool solved the case by refusing to pretend. Its 0.0 sent me away from a malware rabbit hole and toward the social-engineering signal where the truth actually lived. If kilo had cried wolf on clean infrastructure to match my mood, I'd have wasted the morning dissecting a job board. Because it stayed honest, it handed me the thread on the first pull.

That is the exact experiment I wrote the tool to run. And it passed.

What the morning actually taught me

The recruiter was never the point. The shape of the morning was.

I didn't solve this with one brilliant tool. I solved it with a chain of small, honest ones: dig turned a name into an address so kilo had something to check; kilo's honest negative reframed the question so curl and openssl knew where to look. And every one of them spoke stable JSON — so I was consuming contracts, not parsing prose. Exit codes matter. Schemas matter. Failure stays legible.

That's the manifesto I keep coming back to as I build these things: contracts, not vibes. When I pipe kilo check --json into jq, I'm having a typed conversation with reality. I can't hallucinate a verdict, because the verdict arrives as a signed struct with its provenance stapled on.

It's the difference between a tool that sounds right and one that is right:

A gut feeling is an opinion. A sharp, composable, honest toolbelt is a witness.

And witnesses — unlike hunches — can say the three most valuable words in this whole trade: "I don't know." Then go find out.

Steal the pattern if it's useful:

Small, single-purpose tools that speak stable JSON compose into something no chatty do-everything API can match.
Honesty beats confidence. A tool that returns unknown under pressure is worth ten that guess to please you.
Let the negatives navigate. The most useful result of the entire morning was a 0.0.

So the next time a link makes your gut twitch, don't argue with the vibe. Resolve it, and ask a tool that's allowed to say nothing.

kilo didn't catch a scam — there wasn't one, just spam in a lab coat. It did something I value more: it told me the truth about what it knew, drew a hard line around what it didn't, and pointed me at where the real answer was hiding.

I built a 12MB binary to say "not observed" without flinching, on the exact morning I most wanted it to lie. It didn't. That's the whole experiment. That's why I built it.

kilo check <your-suspicions-here> --json

Tooling in this piece: KiloCheck · the eternal dig / openssl / curl. The story is real; the identifying details have been filed off.

The AI Cost-Modeling Handbook: I let Claude do the modeling, but never the arithmetic

Don Johnson — Wed, 01 Jul 2026 02:51:48 +0000

Every "what's the cheapest model?" thread online is people trading vibes. I got tired of it, so I built a pipeline that pulls live, cited prices and runs the numbers through an exact-rational math kernel — no floating-point drift, no LLM hallucinating a multiplication. Then I pointed it at eight of the cost questions every agent builder actually faces. Here's everything it found, and the repo so you can re-run all of it.

There are two kinds of LLM cost advice. The first is a benchmark leaderboard with a price column, which tells you nothing about your workload. The second is a confident tweet that's quietly wrong because someone multiplied a per-million price by the wrong token count in their head.

I wanted a third kind: a model you can audit. Re-run it and you get bit-identical numbers, with the source of every input price one file away. This article is the result — a tour through eight cost decisions, each answered with real money math, and each one teaching something that intuition gets wrong.

Let's start with the question that kicked it off.

Cold open: what's the most cost-effective model to run an agent on?

The setup: I run Hermes Agent, a model-agnostic agent framework, and I wanted the cheapest token provider to drive it. Not cheapest per token — cheapest per unit of quality, because a model that flubs tool calls and retries can cost more than a pricier reliable one.

So the metric is blended cost ÷ agentic-quality-score. Blended cost uses a production-agentic token mix (more on that below); quality is a normalized agentic score (mean of BFCL / τ²-bench / SWE-bench Verified). Filters: open-weights, prompt caching, no-train.

Model @ Provider	Blended $/1M	Quality	$/quality (×1000)
DeepSeek V3.2 @ OpenRouter	0.1145	77	1.49
DeepSeek V3.2 @ DeepInfra	0.1951	77	2.53
MiniMax M2 @ MiniMax	0.2629	73	3.60
GLM-4.6 @ z.ai	0.5276	71	7.43
Kimi K2 @ DeepInfra	0.6613	66	10.02
DeepSeek R1 @ DeepInfra	0.6519	54	12.07

DeepSeek V3.2 wins decisively — it's both the highest-quality open model in the set and near the cheapest, because its output token price is absurdly low for its tier. The emerging best deal (pending a quality test) is DeepSeek V4 Flash on Fireworks at $0.0896/1M blended — cheapest in the field, with ZDR-by-default and a 50% batch discount.

That's a useful answer. But the interesting part is everything that question opens up. If you can split traffic across providers, what's the optimal mix? Should you self-host instead? Is the "smart expensive model" actually cheaper once you count its thinking tokens? Each of those is a chapter.

First, how this is built.

The toolchain: research agents + an exact-math kernel

The pipeline has two halves, and the whole point is that the LLM never touches the arithmetic.

1. Research agents gather live, cited inputs. Token prices in 2026 move monthly — half the models in my training data are already legacy. So every price and benchmark in this guide came from a research agent doing live web search/fetch, returning a structured table with source URLs and observation dates. Those land in data/ (pricing.md, quality.md, gpu.md, frontier.md, decline.md), each one auditable.

2. An exact-rational kernel does the math. I used agent-calc, an AI-native computation kernel that works in exact rationals — 3/5 × 1,000,000 returns 600000, not 599999.9999998. It exposes typed domains: rational, linear (LP), finance (NPV/amortization), solve (algebra), stats (distributions), polynomial, interval (uncertainty arithmetic). Every chapter below drives a different one.

Why bother? Because cost models are exactly where floating-point lies and LLMs fumble: tiny per-token prices, huge token counts, compounding rates. When an answer is "self-hosting breaks even at 87.2% utilization," you want that to be computed, not vibed. And the secret structure of this guide is that each chapter answers a money question and exercises one kernel domain — it's a cost-modeling handbook that doubles as a tour of exact computation.

The shared workload model

Every chapter uses one production-agentic token profile, because agent tool-loops are wildly input-heavy:

Component	Share of tokens
Fresh input	21.25%
Cached input	63.75%
Output	15%

That's an 85/15 input/output split with 75% of input being a cache hit (the system prompt, tools, and skills get re-sent every tool-loop step). Blended price = 0.2125·in + 0.6375·cached + 0.15·out. Chapter 7 explains why this profile is forced on you; Chapter 6 explains why the cache hit rate is so high. For now, take it as the workload.

Chapter 1 — Optimal multi-provider routing (a linear program)

You don't have to pick one provider. Run a cheap one for the bulk, escalate elsewhere when constraints demand. The optimal split — minimize spend subject to a quality floor, per-endpoint capacity caps, and a US-jurisdiction floor θ — is a literal linear program. agent-calc solves it (solve_lp), then I re-verify the winning allocation's cost in exact rationals (the LP solver is f64; the headline number shouldn't be).

Sweeping the privacy knob θ traces a cost-of-privacy frontier:

US floor θ	min $/1M	$/mo @ 1B	allocation
0.00–0.40	0.1468	$146.77	A=0.60, B=0.40
0.60	0.1629	$162.89	A=0.40, B=0.60
0.80	0.2716	$271.61	A=0.20, B=0.60, F=0.13, H=0.07
0.872	0.3116	$311.59	A=0.13, B=0.60, F=0.27
≥ 0.873	—	INFEASIBLE	the privacy wall

Three things intuition misses:

The first 40% of US-jurisdiction traffic is free — the cost-optimal blend already lands there, because the cheapest provider is capped at 60% and the overflow falls on a US endpoint anyway.
The marginal cost of privacy is convex — 40→60% is cheap; past 60% the quality floor forces in weaker fillers and the bill nearly doubles by 87%.
There's a hard wall at 87.2%. Only one high-quality US endpoint exists in this menu; beyond its capacity the math is infeasible. To break the wall you need a second high-quality US provider, or you accept single-point-of-failure risk.

Chapter 2 — Self-host vs. serverless: it's a utilization question

100% utilization (impossible); only owned 8×B200 wins at 72%" width="800" height="459">

Everyone's instinct: "at some volume, renting GPUs beats per-token APIs." The reframe that changes everything: a self-hosted node costs the same at 5% load or 95% load, while the API scales linearly with tokens. So it's not a dollar break-even — it's a break-even utilization: how busy must the box stay to win?

u* = node_$/mo × 1e6 / (capacity_tokens/mo × API_$/1M)

If u* > 100%, self-hosting loses even at full tilt. Modeling DeepSeek V3.2 (8-GPU node) against DeepInfra's $0.195/1M, with agent-calc handling the owned-hardware amortization (finance) and the exact u* (eval):

Hardware / procurement	break-even u*	verdict
8×H200, rent on-demand	531%	never
8×H200, own (amortized)	220%	never
8×B200, rent reserved	174%	never
8×B200, own (amortized)	72%	the only winner

For a commodity model this cheap on the API, renting GPUs loses in every configuration — the serverless provider batches across thousands of tenants to hit a utilization you can't. The only config that wins is owned Blackwell silicon kept above ~72% utilization 24/7 (~56B tokens/month of steady load). And against the falling API floor ($0.0896/1M), even that roughly doubles its break-even. The takeaway: self-host for privacy, control, or latency — not to cut the token bill.

Chapter 3 — The reasoning-token tax

Reasoning models emit hidden "thinking" tokens, billed at the output rate. A 600-token answer can bill 7,800. Does the higher success rate justify the burn? The honest unit isn't $/token — it's cost per solved task = attempt cost ÷ success rate.

With agent-calc computing exact costs (eval) and the break-even success rate (solve):

Model	k (think mult.)	$/solved	×champion
DeepSeek V3.2	4	3.17m	1.00×
MiniMax M2	2	4.03m	1.27×
Kimi K2	1	8.79m	2.77×
GLM-4.6	5	13.77m	4.35×
DeepSeek R1	12	36.80m	11.61×

(m = milli-dollars = $0.001/task.)

The killer result: DeepSeek R1 would need a 627% success rate to match V3.2's cost-per-solved — mathematically impossible. Even a perfect R1 stays far more expensive, because its per-attempt token burn alone exceeds V3.2's entire cost-per-solved. The sweep confirms it: even at k=0 (zero thinking), R1 is still 2.6× the champion.

The rule: the reasoning tax only pays when output tokens are cheap. Thinking is a multiplier on the output price. Pay $2.15/M and ×12 and you've built the most expensive possible way to solve a task; pay $0.38/M and the same multiplier barely registers.

Chapter 4 — The retry cascade

You don't need one model for every task. Run a cheap model on everything, escalate to a reliable premium model only on the residual failures. agent-calc computes the escalation probability (stats/binomial_pmf) and the exact expected cost (eval).

Strategy	cost/solved	coverage	% → premium
Cheap-only (DeepSeek V3.2)	3.17m	77.0%	0%
Premium-only (Claude Opus 4.8)	82.96m	88.0%	100%
Cascade V3.2 → Opus	19.78m	97.2%	23%
Cascade V3.2 → Gemini → Opus	13.52m	99.5%	4.4%

The cascade beats premium-only on both axes: higher coverage (97% vs 88%) at a quarter the cost, because only 23% of tasks reach the expensive model. Adding a mid-tier pushes coverage to 99.5% while cutting cost further — the cheap+mid tiers absorb 95.6% of traffic, so the $73m Opus call fires on 4.4% of tasks.

And the trap to avoid: retrying the same model never changes cost-per-solved (it stays C/p), and for deterministic failures it doesn't even raise coverage. Only escalating to a different, stronger model moves the needle. "Just add retries" is a no-op; "add a tier" is the win. (The cost advantage survives even strong failure correlation — 3.6× cheaper than premium-only even when the premium rarely rescues what the cheap model botched.)

Chapter 5 — The NPV of waiting

A provider offers a discounted reserved rate if you commit for a year. Locking it in feels prudent — but token prices fall fast, and you might be locking yourself above where pay-go will be in six months. This is a present-value problem: NPV the committed cash-flow stream against the declining pay-go stream (agent-calc finance/net_present_value), and find the break-even decline rate.

The decisive input is that price decline depends entirely on what you hold fixed:

Fixed-quality (same capability getting cheaper): ~80%/yr (a16z/Epoch).
Frontier (always run the best): ~30%/yr, sometimes flat or rising.

At a 30%-off, 12-month commit:

Workload	central decline	winner
Fixed-quality	80%/yr	pay-go (−23%)
Frontier	30%/yr	commit (+22%)

And the break-even decline grid shows committing never wins for commodity models (80%/yr exceeds every reservable discount's break-even), while longer terms always make it worse: a 30% discount breaks even at 57.8%/yr over 12 months but only 24.5%/yr over 36 months. The rule: reserve where prices are sticky (frontier); ride the curve where they fall fast (commodity). Never sign a 3-year inference commit in a market falling >25%/yr.

Chapter 6 — Prompt-cache ROI

Caching looks free, but a cache write often costs more than a normal input token (Anthropic charges up to 2×), while reads are cheap. So it's a bet: pay the write premium to save on reads, and it only pays if you reuse enough before the cache expires. The break-even reuse count is N* = (write − read)/(in − read).

Provider	break-even N*	savings ceiling
DeepInfra / DeepSeek / OpenAI / Fireworks (no write fee)	1.00	50–90%
Anthropic Opus 5-min TTL (1.25× write)	1.28	90%
Anthropic Opus 1-hr TTL (2× write)	2.11	90%

Caching pays almost immediately — most providers from the second call, even the harshest write premium by the third. The read multiplier sets the long-run ceiling (0.1×-read → 90% savings; 0.5×-read → 50%), so for cache-heavy loops that's the number that compounds. Using agent-calc's interval domain, an 8k-token agent prefix reused an uncertain 3–20 times costs $0.088–0.156 cached vs $0.12–0.80 uncached — 80% off at the high end, even with a 2× write premium. This is why the guide's 75% cache-hit assumption holds: agent loops reuse a large prefix 10–20× per task.

Chapter 7 — Agent-loop compounding (the O(K²) tax)

Why is that reused prefix so large? Because a multi-turn agent re-sends its growing context every step — at step k, the input is the base prompt plus every prior tool call and result. Per-step input grows linearly, so cumulative input over a K-step task grows quadratically:

cumulative over K = (δ/2)·K² + (B − δ/2)·K

agent-calc evaluates that polynomial exactly (polynomial/evaluate):

K steps	cum tokens	naive $	cached $	compacted $
10	147,500	0.0384	0.0220	0.0384
50	2,237,500	0.5817	0.3015	0.3267
100	8,225,000	2.1385	1.0896	0.6907

A 100-step task isn't 10× a 10-step task — it's 56×. Your 50th tool call's input costs 10× your first. Caching bends the curve (lowers the constant); only compaction breaks it — capping the context window turns the quadratic linear, and the gap widens without bound as loops get longer. They compose: cached and compacted is read-rate pricing on a linear token count. For any loop past ~20 steps, do both.

This closes the loop on the whole guide: the quadratic re-send is exactly why agent workloads are so input-heavy and so cache-dependent — the entire shared workload profile falls out of this one chapter.

The cheat sheet

Everything above, compressed to decisions:

Decision	Verdict
Which model?	Cheap efficient hybrid (DeepSeek V3.2-class). Cheap output + high success beats everything on cost-per-solved.
One provider or many?	Blend via LP. First ~40% of any jurisdiction/diversity constraint is usually free.
Self-host?	No — for cost. Renting always loses for commodity models; only owned next-gen hardware at >72% utilization wins. Self-host for privacy/control.
Reasoning model?	Only if its output tokens are cheap. A pricey heavy reasoner can be mathematically uncatchable on cost-per-solved.
Retries?	Cascade to a different, stronger model — never retry the same one. 3 tiers ≈ 99.5% coverage at ~6× lower cost than going straight to frontier.
Commit to reserved capacity?	Only for sticky frontier prices, short terms. Never for commodity models in a market falling 50–90%/yr.
Caching?	Always, for any reused prefix. Pays from call 2–3. Prefer 0.1×-read providers.
Long agent loops?	Cap the context window (compaction) past ~20 steps, or pay quadratically.

Reproduce it

The whole thing is a repo. Each chapter is a self-contained script that shells out to agent-calc; each input price lives in data/ with a source URL and observation date.

git clone https://github.com/copyleftdev/rational
make all       # re-run every chapter
make ch04      # or just one

Same inputs → bit-identical outputs, every time. The LLM did the modeling and pulled the data. The math kernel did the math. That division of labor is the only reason I trust any number in this article — and it's the part I'd encourage you to steal.

AI Didn't Invent Slop. It Only Made It Infinite.

Don Johnson — Sat, 27 Jun 2026 00:01:32 +0000

Every generation believes it is witnessing the death of taste.

The printing press was going to flood the world with trash. The cheap camera was going to destroy photography. The internet was going to drown serious thought in blogs, forums, spam, and noise. Smartphones were going to turn every sacred moment into a blurry rectangle.

Now artificial intelligence, we are told, has brought us to the final collapse: endless images, endless songs, endless essays, endless pull requests that compile and ship and mean nothing.

We call it AI slop.

And to be fair, much of it deserves the name.

You can feel it when you see it. The over-polished fantasy portrait. The too-perfect lighting. The fake article that says everything and means nothing. And if you write software: the 600-line PR that passes CI and no human can explain. The README generated by a machine that has never run the code. The test suite with 100% coverage that asserts only that the mock returned the mock.

It is work nobody will remember making.

It has texture but no conviction. Output, but no memory of why.

But AI did not invent slop.

It only made it infinite.

We Have Been Here Before

There was a time when a photograph meant something simply because it was difficult to make.

A camera was not just an object. It was a ritual. Film had to be bought. Shots had to be chosen. Light had to be respected. Mistakes cost money. The photographer carried the burden of scarcity.

Then cameras became cheap, then digital, then they entered every pocket on Earth. Suddenly, humanity became a species of photographers. We photographed lunch, and mirrors, and sunsets we barely watched. We created billions of images, most of them forgotten before the day ended.

Photography did not die.

But the average photograph became meaningless.

That distinction matters. The medium survived. The scarcity did not.

No one calls it "camera slop," but perhaps we should have.

Slop Is What Happens When Creation Becomes Cheap

This is the uncomfortable truth underneath the AI panic:

Whenever the cost of creation collapses, the volume of mediocre work explodes.

That is not a flaw unique to artificial intelligence. It is one of the oldest patterns in media history — and it has names. Sturgeon's Law already told us that ninety percent of everything is forgettable. What changes, across eras, is only how cheaply we can manufacture the ninety percent.

When printing became cheaper, the world got pamphlets, propaganda, gossip sheets, and disposable novels. When desktop publishing arrived, everyone became a designer. When blogging platforms arrived, everyone became a columnist. When GitHub made it trivial to publish code, every weekend produced ten thousand repos that would never see a second commit.

Every tool that democratizes creation also democratizes mediocrity.

That sounds cruel, but it is not. It is simply math. When the gates open, more people enter. Most are beginners. Many are opportunists. Some are brilliant. The room gets louder before it gets better.

AI is not breaking that pattern. AI is accelerating it.

Slop Is a Taste Crisis, Not a Technology Crisis

The real danger of AI slop is not that machines can make things.

The danger is that humans may stop caring whether the things are worth making.

That is the disease. Not automation. Indifference.

Slop is not defined by the fact that AI was used. It is defined by the absence of intention. A human can make slop with a camera. A human can make slop with a paintbrush. A human can make slop with vim and forty years of experience and a million-dollar budget.

Slop is not a medium. Slop is a relationship to creation.

It is what happens when the goal is output instead of meaning. Velocity instead of correctness. Shipped instead of understood.

The machine did not create that hunger. The machine simply feeds it faster.

The System Loves Slop

Slop thrives because the systems around us reward frequency.

Merge faster. Chase the metric. Bump the contribution graph. Worship the green build. Close the ticket. Remove the silence where thinking used to live.

AI fits perfectly into this economy because AI is tireless. It does not sleep, or get bored, or stare at a failing test for three days before finding the one assumption that was wrong from the start.

The pipeline wants endless output. AI can provide it.

That marriage — between machine generation and a culture that measures motion instead of meaning — is where slop becomes industrial. Not because AI is evil. Because infinite production met infinite distribution, and nobody in the middle asked whether any of it mattered.

Copilot Did Not Make You an Engineer

A cheap camera allowed anyone to take a picture. It did not make everyone a photographer.

A laptop allowed anyone to make music. It did not make everyone a musician.

A blog allowed anyone to publish. It did not make everyone a thinker.

Copilot lets anyone generate working code. It does not make everyone an engineer.

This is where the confusion begins. We keep mistaking access for mastery.

The tool can produce the surface of a thing. It can mimic the genre, satisfy the linter, pass the tests it also wrote. It can imitate the grammar of competence.

But engineering is not merely code that runs.

It is selection, restraint, and the refusal to write the clever abstraction that will rot in eighteen months — even though the model offered it, ready to paste, today. It is a human being deciding that one design matters more than another and being able to say why.

AI can generate a thousand implementations in an hour.

The engineer is the one who knows which one should not exist.

The Scarce Resource Was Never the Tool

For decades we confused engineering value with the difficulty of execution.

The person who could write tight assembly possessed rare power. The person who could wrangle pointers, or hold a distributed system in their head, or make the build green, possessed rare power.

But technology has a way of eating technique. Again and again, the machine absorbs what once required years of specialized execution. And every time, people panic and say the craft is dead.

But the craft is not dead. It moves.

When execution becomes easier, taste becomes more important. When production becomes abundant, judgment becomes more important.

When everyone can generate a pull request, the real question becomes: who can read it? Who can see the load-bearing line? Who can look at infinite plausible code and say, "Not that. This."

But Abundance Is Not the Enemy of Craft

Here is the part the optimists get right and the part they get wrong.

The right part: every explosion of mediocrity also expands the possibility of genius. Cheap cameras gave us billions of forgettable images — and handed visual language to people who would never have entered a photography school. Home studios created oceans of bad music — and gave us bedroom producers who reshaped global sound. The early web buried us in spam — and also gave us Linux, Wikipedia, and Stack Overflow, each built from the same flood of amateur contribution that produced the junk.

Democratization tends to move in waves. The first wave is noise. The second is imitation. The third discovers what the medium is actually for.

The wrong part — the part the optimists wave away — is that the third wave is not guaranteed. Plenty of ecosystems hit the noise wave and simply stayed there, or died: local newspapers did not ascend to some higher form, they collapsed and did not come back. "It worked out before" is not a law of nature. It is a bet. The honest claim is narrower: abundance makes a better equilibrium possible. It does not deliver one. Someone has to do the choosing.

But Won't the Machine Just Do the Choosing Too?

This is the strongest objection to everything above, so let's not dodge it.

Every earlier democratization left one thing untouched: a human layer that decided what was good. The press multiplied books, but humans still chose what to read. The camera multiplied images, but humans still chose what to hang on the wall. The scarce resource could shift to judgment precisely because judgment stayed ours.

AI attacks both sides of that deal at once. It floods faster than any reviewer can read — and it is being trained, right now, to do the reviewing: to rank, to critique, to pick the better of two diffs. If the machine can write the thousand implementations, why can't it choose the one that should exist?

Partly, it can. Taste is not magic. Much of it is pattern, and patterns are trainable. An honest version of this argument has to concede that "humans simply judge better" is a moat that will not hold — the same way "humans simply write assembly better" did not hold.

But there is a remainder, and it is not a skill. It is accountability.

Someone has to be the one who says ship it and owns what comes next — the outage, the breach, the regression that reaches a million users, the quiet harm nobody modeled. A system can rank the options. It cannot be answerable for the choice. The engineer who clicks merge is not just exercising sharper judgment than the generator; they are the human a consequence can attach to.

So the moat is not "we judge better." That one erodes. The moat is "we are responsible." Discernment is the craft you sharpen. Accountability is the part the machine structurally cannot take, because responsibility requires someone who can actually be held to it.

The Engineer Becomes an Editor of Infinity

The next great technical skill may not be generation. It may be discernment exercised under that responsibility.

We already have a name for the work: review. And it is moving from the chore at the edge of the job to the center of it.

Watch what AI does to a team. It lifts a junior's output to senior-looking quality almost overnight — clean, idiomatic, plausible. It does not lift their judgment at the same rate. So the senior becomes the bottleneck, reviewing an infinite stream of confident, well-formatted diffs from people (and models) who cannot yet see what is wrong with them. "LGTM" stops being a rubber stamp and becomes the most dangerous sentence in the codebase. The reviewer becomes the rate limiter on quality for the entire organization.

That is the job now. Not typing faster. Reading better.

The engineer of the next decade is part architect, part editor, part skeptic — someone who can run the machine without serving it, who generates without limit, then cuts without mercy. Someone with memory. Someone with scars from the last system that paged them at 3 a.m.

Because when any code can be written instantly, the most valuable thing you can do is refuse to merge most of it.

Slop Is the Shadow of Democratization

AI slop is real. It is everywhere — feeds, marketplaces, search results, issue trackers, and the diff you have to review this afternoon.

But it is not an alien substance. It is the shadow that appears whenever creation becomes easier than judgment. Scarcity used to do some of our filtering for us: difficulty kept the repo smaller, cost slowed people down, the sheer friction of writing it yourself meant you mostly wrote what you meant. Now that friction is gone, and the burden shifts back to us. We have to decide what matters.

The cheap camera did not destroy photography. It destroyed the illusion that every photograph was precious. AI will not destroy engineering. It will destroy the illusion that every act of generation is creative.

And maybe that is necessary. Maybe we are being forced to admit what was always true: the magic was never in the tool. The magic was in the seeing and the choosing — in taking the chaos of a problem and shaping it into something that says:

I was here. I understood this. This mattered.

In the age of infinite slop, that may become the rarest thing of all.

Notes on the Argument

This is a historical argument, not a technological one, and it is a synthesis rather than a discovered law — so here is the lineage it rests on, made explicit so you can check it:

Sturgeon's Law — "ninety percent of everything is crap." The baseline claim that most output in any field is forgettable. (ref)
Clay Shirky's "mass amateurization" — when you remove the barrier to publishing, you don't get less junk, you get "publish, then filter": the filtering moves downstream, onto the reader. This essay's "engineer as editor" is that filter, relocated to code. (ref)
The Jevons paradox — making something cheaper to produce raises total consumption rather than lowering it. Cheaper creation means more creation, not less — now widely invoked for AI. (ref)

The concrete parallels are well documented: the printing press produced a flood of cheap Reformation pamphlets alongside its enduring works (ref); consumer and smartphone photography pushed the world past a trillion-plus photos a year (ref); desktop publishing democratized design (and gave us the "ransom note" era), while content farms like Demand Media industrialized algorithmic filler until Google's Panda update was built to bury it (ref).

The argument deliberately holds two ideas at once: democratization of creation (good — more people gain the tools) and a decline in average quality (the expected cost when participation grows from thousands to millions). They are complementary, not contradictory. Reasonable people can disagree with where it lands. The point is to offer a frame for the debate, not to claim it is settled.

A Note on Collaboration

This essay was written through a collaboration between a human author and AI. The ideas, the thesis, the structure, and every editorial decision came from a human; AI was used as a thinking partner and drafting tool, and the historical claims above were checked against the sources cited.

Writers have always worked with tools — editors, dictionaries, research assistants, cameras, compilers, spell checkers, search engines. AI is another one, albeit a remarkably capable one. The quality of a work has never been determined solely by the sophistication of the tool that helped make it. It has always been determined by the quality of the thinking behind it.

Judge this essay by its reasoning. Not by the instrument used to write it.

(Disclosed per DEV's guidelines for AI-assisted articles.)