DEV Community: Don Johnson

The Shape of Failure: Before You Blame the AI

Don Johnson — Sat, 01 Aug 2026 21:12:51 +0000

Every automated system receives a particular shape of the world.

That shape is expressed through records, documents, events, exceptions, and missing values. If the designers have not identified those forms—and the ways they can become malformed—the machine inherits their ignorance and reproduces it at scale.

The question is not simply whether the AI failed.

The useful question is whether the human-built system knew what success meant, knew the shape of its data, and knew how to recognize when it was wrong.

Start with the shape of the data

Before selecting a model, draw the workflow as a sequence of data transformations.

What enters each stage?
In what form and from what source?
Which values are valid, absent, duplicated, stale, delayed, or contradictory?
How will each violation be detected?
What must the workflow do next?

Each data shape needs a corresponding failure model. An unknown here is not merely uncertainty for the machine; it is a measurement failure in the organization.

The remedy is to collect the missing data or explicitly design for its absence. Otherwise, the system is being asked to operate in a world its designers have not described.

Stabilize the deliverable

A system cannot be stabilized around a target that continues to move.

The deliverable must be more than an aspiration written in a prompt. It should be expressed as observable conditions and anchored to a representative corpus:

examples that are acceptable;
examples that are unacceptable;
examples that are genuinely ambiguous.

Human reviewers should first demonstrate that they can apply those distinctions consistently. If they cannot agree on what success looks like, the model is not being measured against a specification. It is being measured against human disagreement disguised as one.

The model is not the system

Only then does it become meaningful to place an AI model inside the workflow.

The model is one transformation among many:

Input
  → validation
  → retrieval
  → normalization
  → model inference
  → output validation
  → policy checks
  → human action

Each transition can discard meaning or introduce error. The fluency of the final response makes the model the most conspicuous suspect, but conspicuousness is not causality.

To assign blame intelligently, observe the entire chain and test every boundary where information is received or transformed.

Mutation testing attacks confidence itself

Conventional tests ask whether software succeeds under conditions its authors anticipated. Mutation testing reverses that pressure.

It deliberately introduces small faults—reversing a condition, moving a boundary, substituting a value, or removing an operation—and asks whether the test suite notices.

A killed mutant caused at least one test to fail.
A surviving mutant changed the program without the tests objecting.

The resulting score is not proof of correctness. It measures how sensitive the tests are to the generated changes.

In the companion implementation, the focused safety-contract run produced:

observed mutation sensitivity = \frac{43 killed}{43 generated} = 1.0

That number has meaning only because the measurement boundary is explicit: the run scores the small kernel containing the workflow's acceptance predicates. It does not pretend that prompt punctuation or CLI wording is part of the safety contract.

Why the first mutation run mattered
The first broad run generated 374 mutations across the package. Of those, 185 survived. Many changed prompt text, trace wording, and gateway plumbing that the tests did not claim to specify.

That was not an embarrassing result to conceal. It revealed an unstable definition of the deliverable. The mutation boundary was then narrowed to the safety-critical contract, and tests were strengthened around exact acceptance boundaries. The experiment therefore reenacted the article's argument: a metric becomes meaningful only after humans define what is being measured.

Carry the same pressure through the AI workflow

The mutation principle should not stop at source code.

Remove an expected field. Corrupt a format. Supply stale or contradictory source material. Interrupt retrieval. Perturb an instruction. Substitute a fluent but incorrect model response.

At every boundary, ask whether the surrounding system detects the disturbance, abstains, falls back, or routes the case to a human.

The reference implementation makes this flow explicit:

shape_failures = shape_agent.inspect(request)
if shape_failures:
    return escalate(shape_failures)

evidence, evidence_failure = evidence_agent.retrieve(request, corpus)
if evidence_failure:
    return escalate(evidence_failure)

decision = model.decide(request, evidence)
criticism = critic_agent.review(request, evidence, decision)
if criticism:
    return escalate(criticism)

return accept(decision)

The actual implementation uses four deliberately small agents:

A shape agent rejects malformed or unmodeled input.
An evidence agent retrieves the matching corpus and detects contradictions.
A model agent proposes a structured, cited decision through OpenRouter.
A critic agent verifies the decision against citations, confidence thresholds, and a computable policy oracle.

The model is useful, but it is never permitted to define its own success criteria.

What the executable experiment found

The checked local verification produced:

86 passing tests;
43 of 43 mutations killed in the safety-contract kernel;
one clean control accepted;
all seven injected data, corpus, and model faults detected;
zero model calls when bad input or contradictory evidence made inference unsafe.

Hypothesis generates both data and sequences of workflow mutations. Mutmut changes the safety-contract source. OpenRouter supplies the optional live model boundary using a strict JSON Schema response.

The OpenRouter call is intentionally optional because it uses an external service and may incur cost. The deterministic safety oracle does not depend on network access, a particular provider, or a favorable model response.

Agreement is another output, not an oracle

Once the single-model boundary was stable, I added a second weave around it. Four model families received the same structured request and the same policy evidence. Each worked independently: no model saw another model's answer. Every response then crossed the same citation checks, confidence threshold, and computable policy oracle.

The live run used:

openai/gpt-5.4-nano;
google/gemini-3.5-flash-lite;
deepseek/deepseek-v4-flash-0731;
mistralai/mistral-small-2603.

Eight scenarios produced 32 model observations. Twenty-seven matched the oracle and passed the contract, but only four scenarios achieved unanimous contract acceptance.

Scenario	OpenAI	Gemini	DeepSeek	Mistral	System
Eligible control	approve `0.78`	approve `1.0`	approve `1.0`	approve `1.0`	accept
Eligible paraphrase	approve `0.74`, rejected	approve `1.0`	approve `0.99`	approve `1.0`	review
Instruction inside customer data	approve `0.74`, rejected	approve `1.0`	approve `1.0`	escalate `0.95`	review
Last eligible day	approve `0.86`	approve `1.0`	approve `1.0`	deny `1.0`, rejected	review
Maximum eligible amount	approve `0.78`	approve `1.0`	approve `1.0`	approve `1.0`	accept
One day late	deny `0.86`	deny `1.0`	deny `1.0`	deny `1.0`	accept
One cent over	deny `0.90`	deny `1.0`	deny `1.0`	escalate `0.99`	review
Non-refundable policy	deny `0.86`	deny `1.0`	deny `1.0`	deny `1.0`	accept

This is not a leaderboard. It is one small experiment against one explicit contract. Gemini and DeepSeek passed all eight cases in this run. OpenAI selected the oracle action in all eight, but its confidence fell below the human-set 0.75 threshold twice. Mistral abstained twice and made one incorrect decision at the exact, inclusive day boundary.

Those are different failure shapes with different remedies. A wrong boundary decision challenges reasoning or prompt clarity. An abstention challenges workflow capacity and escalation cost. A low-confidence rejection challenges calibration and the threshold chosen by the organization. None can be repaired merely by declaring that three models outvoted the fourth.

Agreement measures convergence, not correctness.

If every model repeats the same unsupported answer, the deterministic contract must still reject it. If one model dissents for a defensible reason, majority voting must not erase the evidence.

The weave also exposed instability that a single run would have concealed. In an earlier pass, the same OpenAI model gave the control 0.90 confidence instead of 0.78, and escalated the one-day-late case at 0.62 instead of denying it at 0.86. Temperature zero reduced one source of variation; it did not turn a remote generative service into a mathematical function.

A provider failure is not a reasoning failure
The pilot included anthropic/claude-haiku-4.5, but that route failed at the model boundary in all eight requests. The workflow escalated without leaking upstream details. I replaced the route with Mistral for the final decision comparison rather than silently counting eight transport or adapter failures as eight reasoning failures.

That distinction is the thesis again: identify the boundary at which the failure becomes observable before deciding what deserves blame.

Different models did help us understand more—but only because the data shape, policy oracle, and rejection rules already existed. Without those controls, the experiment would have produced four persuasive rationales and no principled way to interpret their disagreement.

Run the single-model and multi-model computational expressions

When is it truly an AI failure?

Only after this discipline has been applied does the phrase AI failure acquire useful meaning.

If the input was well formed, the deliverable was stable, the supporting components behaved correctly, and the model still violated a known requirement, then the model failed within the conditions established for it.

But if the data was absent, the target disputed, retrieval defective, or safeguards unable to recognize error, the model may merely be the place where an earlier failure became visible. Calling that an AI failure does not improve the system. It interrupts the feedback by which the organization might learn.

A generative model is an engine of learned resemblance. It can produce behavior that looks remarkably like understanding, but resemblance cannot carry responsibility. People choose the objective, define acceptable evidence, engineer the transitions, and decide what happens when uncertainty enters the system.

Before declaring that the AI failed, demonstrate that the system knew what success meant, knew the shape of its world, and knew how to recognize when it was wrong.

If it did not, the failure began long before the model answered.

The complete experiment includes the agent weave, independent multi-model comparison, property-based tests, stateful tests, mutation configuration, deterministic fault-injection CLI, verification record, and OpenRouter gateway.

I Had a Lot of Fun Building a Linux Packet Flight Recorder

Don Johnson — Tue, 28 Jul 2026 00:05:35 +0000

Some projects begin with a roadmap. skbx began with me falling into a Linux
networking rabbit hole and enjoying it much more than I expected.

The question was simple:

Where did this packet actually go?

The answer was not.

A packet can pass through network namespaces, routing decisions, Netfilter,
traffic control, XDP programs, tunnels, clones, copies, and drop paths. A
packet capture at one interface can be completely correct while still showing
only one part of that journey.

I wanted to see more of the journey. Then I wanted to save what I saw, replay
it later without root, and know whether the capture itself had lost
observations.

That became skbx.

The short version

skbx is a Linux packet-path flight recorder built with Rust and CO-RE eBPF.

During a live capture, it observes kernel networking functions and writes an
append-only JSONL evidence stream. Afterward, that stream can be replayed and
inspected without loading another eBPF program.

live traffic
    ↓
CO-RE eBPF observations
    ↓
bounded traceq JSONL
    ↓
replay → route patterns → explain an event

It can observe:

kernel functions handling an sk_buff;
packet clones, copies, copy-on-write, and XDP-to-SKB transitions;
TC and XDP program entry and exit;
tunnels and inner packet tuples;
kernel-reported drop reasons;
selected BPF helper and map activity;
capture loss, decoding failures, and output failures.

This is not a replacement for tcpdump or Wireshark. Those tools are excellent
when the question is about packet bytes and protocols at a capture point.
skbx is for questions about the path through the local Linux kernel.

It is also not the first tool to follow packets through that path. The project
is explicitly inspired by
pwru, which showed how useful broad eBPF
packet tracing can be.

The part I especially wanted to explore was the evidence after the live
terminal stopped scrolling.

Why call it a flight recorder?

Live tracing is useful, but incident work often continues after the privileged
session has ended.

Someone else may need to inspect the capture. We may want to compare it with a
successful request. We may need to cite one exact event in a bug report. An
automation or AI system may need structured input, but it should not be allowed
to invent observations that were never captured.

So the native skbx stream, called traceq, has an envelope:

capture_start
event
event
...
capture_end

Every event receives a stable handle:

event:111111111111111111111111

Replay groups ordered events into bounded route patterns, which receive their
own handles:

route:1c0201e74424e253f8363577

The footer matters as much as the events. It records the stop reason and
reliability counters for kernel reservation failures, tracer recursion misses,
read failures, userspace decoding and enrichment failures, and output
failures.

If the footer is absent, the artifact is incomplete. If the tracer lost
observations, that uncertainty stays attached to the result.

I like this property because tracing software is also software. It should not
silently act omniscient.

Installing it

The live tracer currently supports Linux on x86_64 and arm64. It needs Rust
1.85 or newer, Clang/LLVM with the BPF backend, bpftool, libelf, libpcap, and
a kernel exposing /sys/kernel/btf/vmlinux.

On Ubuntu, the native packages are:

sudo apt-get install "linux-tools-$(uname -r)" \
  clang llvm libelf-dev libpcap-dev pkg-config

On Debian:

sudo apt-get install \
  bpftool clang llvm libelf-dev libpcap-dev pkg-config

Then install the CLI directly from GitHub:

cargo install --git https://github.com/copyleftdev/skbx --locked skbx-cli

Before attaching anything, doctor checks the host:

skbx doctor --json

plan goes one step further. It shows exactly which functions would be
attached without performing the attachment:

skbx plan --filter-func 'ip.*' --json

That separation turned out to be useful while building the project. A missing
kernel capability should be visible before a privileged capture begins, not
silently approximated afterward.

Capturing a first packet

Here is a small ICMP capture:

sudo skbx capture --probe ip_rcv \
  --duration 10 \
  --output trace.jsonl \
  icmp

While it runs, create some traffic in another terminal:

ping -c 3 1.1.1.1

The capture is deliberately bounded. It has a finite duration and event limit
instead of assuming that an unbounded trace is safe.

After it finishes, replay does not require root:

skbx replay trace.jsonl

Replay summarizes the functions, processes, packet identities, route patterns,
consensus, outliers, and reliability state. Given the same valid JSONL input,
it produces a byte-identical summary.

If an event looks interesting, retrieve it with its surrounding same-packet
evidence:

skbx explain trace.jsonl event:<handle>

The repository includes a small contract fixture containing an ip_rcv event
followed by kfree_skb_reason. Replaying that fixture produces a route shaped
like this:

{
  "handle": "route:1c0201e74424e253f8363577",
  "functions": [
    "ip_rcv",
    "kfree_skb_reason"
  ],
  "routes": 1,
  "truncated": false,
  "outlier": false
}

That output was generated by the current CLI from the checked-in test fixture;
it is not illustrative pseudodata.

Use case: what happened to a website request?

This is the use case I keep returning to.

Suppose curl starts an HTTPS request and eventually times out. A capture on
the client can answer questions such as:

Did the request enter the local TCP and IP output paths?
Which interfaces and namespaces were associated with it?
Was its mark changed?
Did a TC or XDP program handle it?
Was it transformed or placed into a tunnel?
Did the local kernel report a drop?

That evidence can clear or implicate the client host.

It cannot prove what happened inside the ISP or on the target server. To prove
those parts, we need observations from those vantage points. That limitation
is important: the absence of a local drop is not proof that a remote host
received the packet.

The useful result is a smaller, evidence-backed search area:

client host evidence
        ↓
observed departure boundary
        ↓
ISP / transit inference
        ↓
target-host evidence, if available

I would rather know exactly where my evidence ends than have a tool produce a
confident story about systems it could not observe.

Use case: a packet disappears inside the host

“The network dropped it” can hide several very different failures.

A host may reject a packet through a normal kernel path. A TC classifier or
XDP program may return a drop action. A packet may be consumed during teardown.
A transformation may cause the packet we were following to continue under a
different kernel object.

For focused TCP drop questions, a small bpftrace program may be the fastest
answer. For packet contents, I still reach for tcpdump or Wireshark.

I reach for skbx when I do not yet know which local subsystem owns the
failure, or when I want to preserve the ordered path rather than one drop
event.

Use case: follow transformations instead of losing the trail

Linux does not promise that one logical packet corresponds to one
struct sk_buff address for its entire lifetime.

Packets can be cloned, copied, or changed through copy-on-write. XDP frames can
later become SKBs. Tunnels introduce outer and inner packet identities.

skbx uses bounded, capture-local lineage state to connect those transitions.
The provenance remains explicit: an event records whether it matched the
original filter or was included because it belonged to a packet already being
tracked.

For example, a marked packet can be followed with:

sudo skbx capture \
  --filter-mark 0x2a \
  --filter-track-skb \
  --output trace.jsonl

The important word here is capture-local. A lineage identifier is not a
distributed trace ID, and it should not be presented as one.

Use case: capture once, investigate without root

Loading and attaching eBPF programs is privileged work. Reading a JSONL file
does not need to be.

That makes a simple handoff possible:

An operator checks the plan and performs a bounded capture.
The capture ends and writes its reliability footer.
The artifact is copied to a normal development or analysis environment.
Another person replays it and cites exact event: or route: handles.

This is useful for incident handoffs and bug reports, but it is also useful for
learning. I can capture a small experiment once and inspect it repeatedly
without keeping probes attached while I try to understand the result.

Use case: give an AI evidence instead of a mystery

One of my stranger motivations was making the CLI friendly to both operators
and software agents.

Before touching the host, an agent can ask the executable what it supports:

skbx describe --format json
skbx schema
skbx doctor --json
skbx plan --json

The native engine remains the source of truth. An AI system may summarize a
route or explain a captured event, but the event must already exist in the
trace. Machine output stays on standard output, diagnostics stay on standard
error, and unsupported capabilities remain explicit.

This does not make an AI explanation automatically correct. It gives the
explanation something concrete to cite and gives a human a way to check it.

The engineering parts I enjoyed

The obvious fun was seeing packets move through functions I had previously
treated as boxes in a diagram.

The less obvious fun was designing the boundaries:

discovering valid struct sk_buff * arguments from the target kernel's BTF instead of guessing from function names;
keeping kernel-side state and work bounded;
moving JSON encoding, symbolization, and filesystem work out of the eBPF hot path;
making probe planning inspectable before attachment;
separating raw observation from later explanation;
treating partial output as incomplete instead of “probably good enough”;
making replay deterministic enough to use in tests and automation.

Those constraints made the tool more interesting to build. They also gave me a
better mental model of what an observability tool can honestly claim.

What it does not know

skbx only observes what the running kernel and selected probes expose.

It does not see inside an ISP. It does not see a remote host unless it runs
there too. It is not a packet-content replacement for Wireshark. Kernel
configuration, BTF availability, permissions, hidden addresses, unsupported
signatures, and capture loss can all limit the evidence.

Those limitations are part of the interface rather than footnotes.

I am still exploring where this project is most useful. The best next input is
not “looks cool,” although I will happily take that. It is a packet path the
tool cannot explain yet, together with the kernel version, exact command,
doctor --json output, and reliability footer.

The source, installation instructions, architecture notes, and field guides
are in the repository:

copyleftdev / skbx

Agent-first Linux packet-path tracing with Rust/eBPF: bounded evidence, deterministic replay, and packet routes with receipts.

skbx

skbx shows where a packet went inside Linux—and keeps the receipts.

Follow a packet through the flight recorder →

Field guides:

It observes kernel networking functions, TC/XDP programs, packet transformations, tunnels, drops, and selected BPF helper activity with Rust and CO-RE eBPF. Every observation lands in a bounded, replayable evidence stream with stable handles and explicit loss telemetry.

Use it when “the packet disappeared” is not a sufficient incident report.

capture → event:8c6f… → replay → route:21b4… → explain
            evidence          pattern          context

Inspired by pwru, rebuilt around an agent-first contract: deterministic observations, machine-readable capabilities, bounded state, and no invented evidence.

The short version

You need to…	skbx gives you…
See the kernel path of an SKB	BTF-discovered kprobes with exact function evidence
Follow clones, copies, COW, and XDP-to-SKB

…

View on GitHub

If you try it, I would love to know which networking question you used it to
answer—and where its evidence stopped.

AI-assistance disclosure: I built and tested skbx. I used OpenAI Codex to
research adjacent writing, challenge the article's positioning, edit this
draft, verify its commands and example output against the repository, and
generate the cover illustration. I reviewed the technical claims and remain
responsible for their accuracy.

389 Tests Passed. NIST Still Caught the Bug.

Don Johnson — Sat, 25 Jul 2026 06:17:51 +0000

I gave an AI agent a calculator because I wanted one hard, inspectable point
inside a probabilistic workflow.

The model could interpret the request and explain the result. The calculator
would perform the computation. It seemed like a clean division of labor.

Then I changed one multiplication sign into addition.

The calculator still passed 389 of the 390 tests in its Rust library harness.
The sole failure compared its answer with NIST's certified results for the
Longley regression dataset.

That bothered me more than a completely broken build would have. I had treated
deterministic computation as safer than asking a language model to improvise
arithmetic. But deterministic does not mean trustworthy. A program can return
the same wrong answer forever.

“Source of truth” suddenly felt too comfortable. Before an AI agent delegates
authority to a tool, that authority should be challenged—and remain revocable
by evidence.

The calculator is only the specimen. The larger idea is a way to place
inspectable, replayable instruments inside probabilistic systems.

The useful boundary is generation versus execution

The interesting distinction is not model weights versus a “real CPU.” Model
inference also runs on processors, and language models can learn genuine
arithmetic procedures. The useful boundary is between generating an answer
and executing a defined operation under a tested contract.

Research on Program-Aided Language Models
(PAL) makes a related split:
the language model reads and decomposes a natural-language problem, while a
runtime such as a Python interpreter executes the generated program. The model
contributes flexible interpretation; the runtime contributes executable
semantics.

That is the division I want in an agent:

At the semantic edge, the model interprets the request, chooses a procedure, identifies relevant quantities, and explains the result.
At the computational edge, a narrow tool validates inputs, applies specified operations, enforces limits, and returns structured output.

This does not make the whole agent deterministic, and it does not make the
model unnecessary. The agent can still choose the wrong tool, supply the wrong
arguments, misunderstand units, or misread the result.

The promise is smaller: one claim becomes inspectable, replayable, and
independently testable.

The opposite of probabilistic is not trustworthy. It is repeatable.

A CPU can be precisely wrong.

Why NIST became my external witness

Calibration cannot be entirely self-referential. The implementation should not
be the sole author of its own expected answers.

That is why I chose NIST—not because government authority turns a result into
mathematical truth, but because NIST has a long institutional practice of
building shared, independently evaluated references.

Congress established the National Bureau of Standards in
1901 to
strengthen the United States' measurement infrastructure. The Standard
Reference Data Act of
1968 authorized a federal program to
collect, critically evaluate, publish, and distribute standardized scientific
and technical reference data. NBS became NIST in
1988. In 1999, that lineage reached statistical
software through the Statistical Reference Datasets
project, usually shortened to StRD.

StRD pairs datasets with certified expected values for specific statistical
procedures. Its collection includes generated and real-world cases of varying
difficulty. For linear procedures, NIST carried 500 digits through its
calculations
so ordinary floating-point representation error would not become the
benchmark.

Longley is a small but numerically challenging linear-regression dataset in
that collection. Its certified results gave my tests something the
implementation could not manufacture for itself: an expected answer produced
outside the code under test.

That qualification matters. StRD does not certify Oddly Exact, prove the
statistics engine correct, or make the software traceable to NIST. NIST
explicitly says the datasets are an aid for evaluating software and that no
mechanism establishes software
traceability.

The reference data was not an oracle for the entire program. It was an
independent witness for the calculations it covered.

Test the tests

Example-based tests ask whether familiar inputs still produce familiar
outputs. Mutation testing asks a more uncomfortable question:

If I introduce a small, plausible defect, does the suite notice?

I used cargo-mutants to replace multiplication with addition in the
multiple-regression standard-error calculation:

- residual_std_dev * sum_sq.sqrt()
+ residual_std_dev + sum_sq.sqrt()

The mutation preserved valid Rust, valid types, and a plausible-looking numeric
result. In the library harness, 389 of 390 tests still passed. The assertion
against NIST's Longley values was the sole failure.

The public mutation
record
preserves the exact command, environment, result, and the limit of what that
experiment establishes:

I cannot claim Longley was the only test in the entire repository capable of
catching the mutation; the run stopped after the failed library harness. I can
claim something narrower and more useful: hundreds of tests tolerated a
semantically broken formula, while an assertion anchored to independently
produced values rejected it.

A green test count is evidence only to the extent that those tests would turn
red when the implementation meaningfully changes. Mutation testing measures
that sensitivity instead of admiring the count.

Challenge the contract around the answer

The formula mutation challenged numerical meaning. My next audit challenged
the promises around the calculation: what the tool accepts, what it refuses,
and how much work it will perform.

The first crack was a contradiction between the advertised contract and the
executable one. The generated JSON Schema forbade additional properties, but
the Rust deserializer silently accepted an unknown field at the request root
and another inside an expression node.

The answer was still numerically correct. That did not make the behavior
harmless. A caller validating against the schema saw a stricter instrument than
a caller speaking directly to the binary. A misspelled or misunderstood field
could disappear without warning.

For an agent-facing tool, silently interpreting a different contract is itself
a correctness defect.

The second crack was a resource boundary. The expression engine limited things
such as integer size, expression depth, and precision. The optimization
interface, however, accepted grid resolution and iteration counts without upper
ceilings.

I requested a grid search with one billion sample points. The pre-repair binary
produced no JSON before an external one-second watchdog terminated it. The tool
had input validation, but it had not earned the bounded-work guarantee I
thought I had built.

Those failures became permanent regression requests. The deserializer now
rejects unknown fields at both levels. The optimizer now publishes and enforces
ceilings of 100,000 iterations and 1,000,000 grid points. The same billion-point
request returns a typed resource_limit response before entering the search.

The contract challenge
Gist
is a runnable replay of all three repaired cases:

That script is more valuable than a screenshot of a green run. It lets another
observer cross-examine the boundary directly.

Repair is the next claim to attack

A new guard is only another claim until the tests prove they care about it.

After repairing the optimizer limits, I selected every mutation
cargo-mutants generated for the two new validators. Eleven mutations tried to
remove the checks, replace them with unconditional success, or alter their
boundary comparisons. The tests caught all eleven.

The focused optimizer mutation
record
captures the selection and outcome:

That completed the loop:

claim → attack → fail → repair → mutate → replay

If I published only the final green suite, you would see confidence. By
preserving the changed operator, the request that hung, the contract
contradiction, the repair, and the mutations that tried to undo it, I can show
a reason for confidence.

The scar is part of the calibration record.

A five-layer challenge stack for agent tools

Oddly Exact is one calculator, but the method travels. Before giving a narrow
tool authority inside an agent workflow, I now ask five questions:

Is the contract explicit and executable? The schema, parser, runtime,
limits, and failure modes must agree. Documentation the executable does not
enforce is only a suggestion.
Is there an independent reference? In this statistical case, NIST StRD
moved selected expected answers outside my implementation. Another domain
might use a standards specification, a reference implementation, a verified
corpus, or a separately derived test oracle.
Which properties should survive across many inputs? Property-based tests
generate cases and check laws rather than memorizing individual examples.
One Oddly Exact property checks that translating every value in a sample by
a large exact offset leaves sample variance unchanged within a tight
floating-point tolerance.
Can the tests detect plausible corruption? Mutation testing changes
operators, comparisons, return values, and guards. A surviving mutation does
not prove the implementation is wrong; it exposes a behavior change the
suite cannot currently distinguish.
Are failures bounded, typed, and replayable? An agent must distinguish a
rejected computation from an unavailable tool. Division by zero, resource
exhaustion, malformed JSON, and a crashed process should not collapse into
the same ambiguous failure.

These layers do different jobs. External references challenge specific
answers. Properties challenge behavior across an input space. Mutations
challenge the tests themselves. Contract and resource attacks challenge the
boundary around all of it.

None provides universal correctness. Together they create a narrower and more
useful result: evidence that a particular tool deserves provisional authority
for validated requests inside a declared contract.

A source under challenge

For a validated request inside that contract, the tool's structured result can
serve as the agent's operational source of truth.

That is a runtime role, not a claim of infallibility. If the agent encounters a
disagreement, it can inspect the arguments, surface the conflict, or consult
another independently trusted tool. It should not quietly replace a structured
result with fresh prose arithmetic.

The trust relationship runs in opposite directions at different times:

While building the tool, the developer should distrust it aggressively.
Inside a contract the tool has earned, the agent should respect its answer or its typed refusal.

New evidence can always return the tool to the first phase. That is a source
under challenge: trusted in operation, open to appeal, and always one
counterexample away from revision.

The beautiful thing is not that source code is truth.

The beautiful thing is that source code can be cross-examined.

I did not end this experiment with a calculator an AI agent can trust forever.
I ended it with something more useful: a calculator whose authority can be
revoked by evidence.

AI tools assisted with editorial research and revision. I verified the
technical claims and stand behind the final text.

Give Your Coding Agent a Deterministic Vulnerability Oracle

Don Johnson — Tue, 21 Jul 2026 15:04:00 +0000

AI agents can write code, run tests, inspect dependencies, and open pull requests. But when they encounter a vulnerable package, their security reasoning often collapses into a web search, an opaque API score, or whatever the model remembers from training.

The problem is not simply missing vulnerability data. It is the contract between the agent and its harness—the automation layer that supplies tools, executes commands, and interprets results. If “no match” silently becomes “safe,” stale intelligence looks current, or a network failure resembles an empty result, the agent can produce a confident answer without trustworthy evidence.

I designed VulnGraph around a different premise: turn continuously changing vulnerability intelligence into a deterministic local primitive. vulngraph-data compiles upstream security sources into verified, content-addressed snapshots. vulngraph-cli installs those snapshots and checks CVEs, package versions, or entire lockfiles offline.

The result is a security tool shaped for both humans and machines: an accepted snapshot and target produce the same verdict; every verdict carries typed evidence; stale data fails explicitly; and unknown is never misrepresented as clean.

This article explains the design decisions behind that contract and shows how to embed VulnGraph into agent and harness workflows.

The dangerous gap between “not found” and “safe”

Suppose an agent is reviewing a dependency update. It needs to answer a seemingly simple question: Is this version safe to ship? A useful answer depends on distinctions that are easy for a harness to erase.

What happened	A careless harness concludes	What the contract must preserve
The package exists and the version is outside every affected range	No vulnerabilities found	`not-affected`
The package is absent from the dataset	No vulnerabilities found	`unknown`
The local snapshot is too old	The last result is probably fine	A freshness failure
The tool cannot open or verify its data	Empty result	An operational or integrity error

These are not cosmetic labels. They lead to different agent actions: proceed, investigate, refresh the data, or stop the workflow. Once a harness flattens them into a Boolean, the model cannot recover the missing meaning through better prompting.

Here is the distinction in practice. One command checks an exploited CVE, a real package version, and a package the snapshot has never observed:

$ vulngraph check CVE-2024-4577 npm:lodash@4.17.15 npm:no-such-package@1.0.0

  CVE-2024-4577
  verdict: ACTIVELY EXPLOITED  (confidence 0.99)
  action:  patch now
  reasons: CRITICAL_SEVERITY, HIGH_EXPLOIT_PROBABILITY,
           KNOWN_EXPLOITED, PUBLIC_EXPLOIT, ...
  cvss:    9.8
  epss:    100.0%
  kev:     listed

  npm:lodash@4.17.15
  verdict: PROOF OF CONCEPT  (confidence 0.96)
  action:  prioritize
  package: npm:lodash — 6 CVE(s) affect this version

  npm:no-such-package@1.0.0
  verdict: UNKNOWN  (confidence 0.00)
  action:  investigate
  reasons: NOT_OBSERVED

  snapshot: sha256:41ee5fbbf58f0a46b99234af89c5388e5f27e0dcd3dbd52298bcd2c2d87ca90e

The output does not ask the agent to infer policy from prose. It supplies a disposition, an action, stable reason codes, confidence, and the exact snapshot that produced the result. The human-readable rendering is useful in a terminal; --json exposes the same distinctions through a versioned machine envelope.

Split changing intelligence from deterministic execution

Vulnerability intelligence changes every day. Agent execution still needs to be reproducible. I separated those concerns instead of allowing every check to fetch and reconcile live data.

CVE List V5 · EPSS · CISA KEV · exploit sources · ATT&CK · OSV · …
                              │
                              ▼
                    vulngraph-data
              fetch → normalize → build → verify
                              │
                              ▼
            immutable data-YYYYMMDD release
       manifest + checksums + content snapshot_id
                              │
                  vulngraph update
                              ▼
             verified local snapshot
                              │
          vulngraph check --offline --json
                              ▼
                  agent or harness

The vulngraph-data repository owns the nondeterministic edge: retrieving bulk publications from upstream sources and reconciling them into one graph. Its output is deterministic. Identical inputs produce byte-identical semantic files, and their hashes produce a snapshot_id that names the graph by content. If a daily build has no semantic change, there is no new release.

The release artifact is the distribution boundary. A consumer does not need the raw source collection or the machinery that built it. It receives a fixed database archive and a manifest recording the snapshot identity, file hashes, source freshness, graph format, and engine revision.

The vulngraph-cli owns the deterministic side. vulngraph update downloads a release into a staging area, verifies the archive checksum and every file hash, recomputes the snapshot identity, checks format compatibility, sanity-opens the graph, and only then activates it. Failed updates cannot replace the last verified snapshot.

After activation, checks do not touch the network. The snapshot becomes an explicit input to the decision:

verdict = policy(snapshot_id, target)

There is no wall clock, random value, model call, or remote response inside that function. Once a snapshot passes the freshness and integrity preflight, the same snapshot and target give a developer laptop, a CI job, and an agent sandbox the same verdict.

A harness workflow that does not guess

A reliable integration has three phases. Only the update phase needs network access; discovery and checks are stable interfaces that the harness can validate.

1. Discover the contract

Do not make the agent reverse-engineer help text. Let the harness inspect the machine-readable capability document and the bundled JSON Schemas:

vulngraph --json capabilities
vulngraph schema command
vulngraph schema observation
vulngraph schema status

capabilities declares the commands, output schema versions, exit codes, and two behavioral guarantees: checks are offline and deterministic. A harness can validate support during startup instead of discovering an incompatibility halfway through a run.

2. Refresh and preflight outside the restricted step

Install or refresh the snapshot in a setup job that is allowed to use the network:

vulngraph update
vulngraph --json status

Then mount or cache the resulting VulnGraph home directory inside the agent environment. This makes network access a controlled data-provisioning concern instead of an ambient capability available during every security decision.

status reports whether a snapshot is installed, its identity, its integrity, and its freshness. A dataset older than fourteen days is not merely accompanied by a warning: check refuses to answer and exits with code 4.

3. Check the artifact the agent is actually changing

The check command accepts individual CVEs, package coordinates, and dependency files. Pointing it at a lockfile expands every dependency and applies the same version-range policy:

vulngraph --offline --json check package-lock.json > vulngraph.json

The CLI recognizes lockfiles from npm, Yarn, pnpm, Cargo, Bundler, Poetry, pip, Go, Composer, Maven, Gradle, and Pipenv. The abridged JSON response below shows the stable envelope:

{
  "schema": "vulngraph.command.v1",
  "command": "check",
  "ok": true,
  "partial": false,
  "snapshot_id": "sha256:41ee5fbb…",
  "data": [
    {
      "schema": "vulngraph.observation.v1",
      "target": { "type": "package", "value": "npm:lodash@4.17.15" },
      "verdict": {
        "disposition": "proof-of-concept",
        "confidence": 0.96,
        "action": "prioritize",
        "reason_codes": [
          "ATTACK_MAPPED",
          "HIGH_SEVERITY",
          "MULTISOURCE_CORROBORATION",
          "PUBLIC_EXPLOIT",
          "SEVERITY_SCORED",
          "VERSION_IN_AFFECTED_RANGE",
          "WEAKNESS_CLASSIFIED"
        ]
      }
    }
  ],
  "warnings": [],
  "errors": [],
  "metrics": { "elapsed_us": 781250 }
}

The important separation is that VulnGraph observes; it does not gate. A successful check exits with 0 even when it finds an actively exploited vulnerability. Exit codes describe whether the tool completed reliably; the harness applies organizational policy to the returned dispositions.

Disposition	Evidence	Sensible default harness action
`actively-exploited`	Listed in CISA KEV	Block or require an explicit emergency exception
`weaponized`	Public exploit and very high EPSS	Block or require security approval
`proof-of-concept`	At least one public exploit	Prioritize remediation and require review
`scored`	CVSS or EPSS evidence, no public exploit	Apply the repository's severity policy
`recorded`	Known record without stronger risk signals	Monitor and retain the evidence
`not-affected`	Known package outside all affected ranges	Proceed for this snapshot
`unknown`	Target absent from the snapshot	Investigate; never translate to clean

This split keeps probabilistic reasoning in the right place. The harness deterministically decides which states require a stop, review, or escalation. The agent then uses the evidence to explain the finding, inspect reachability, propose the smallest safe upgrade, and communicate the trade-off to a human.

A minimal Node.js harness

The following wrapper turns the versioned VulnGraph envelope into one of five local workflow decisions. It does not ask the model whether an operational failure is acceptable, and it never treats an unrecognized disposition as approval.

import { spawnSync } from "node:child_process";

const binary = process.env.VULNGRAPH_BIN ?? "vulngraph";
const target = process.argv[2] ?? "package-lock.json";
const run = spawnSync(
  binary,
  ["--offline", "--json", "check", target],
  { encoding: "utf8" }
);

let envelope;
try {
  envelope = JSON.parse(run.stdout);
} catch {
  throw new Error(`vulngraph returned invalid JSON: ${run.stderr}`);
}

// Exit codes describe tool reliability, not vulnerability severity.
if (run.status === 4) {
  throw new Error("VulnGraph snapshot is stale; refresh before continuing");
}
if (run.status !== 0 || !envelope.ok) {
  throw new Error(`VulnGraph failed: ${JSON.stringify(envelope.errors)}`);
}
if (envelope.schema !== "vulngraph.command.v1") {
  throw new Error(`Unsupported schema: ${envelope.schema}`);
}

const policy = new Map([
  ["actively-exploited", "block"],
  ["weaponized", "block"],
  ["proof-of-concept", "review"],
  ["scored", "review"],
  ["recorded", "monitor"],
  ["not-affected", "proceed"],
  ["unknown", "investigate"]
]);

const decisions = envelope.data.map(({ target, verdict }) => ({
  target: target.value,
  disposition: verdict.disposition,
  decision: policy.get(verdict.disposition) ?? "investigate",
  action: verdict.action,
  confidence: verdict.confidence,
  reasons: verdict.reason_codes
}));

const evidence = {
  snapshotId: envelope.snapshot_id,
  decisions
};

console.log(JSON.stringify(evidence, null, 2));

if (decisions.some(({ decision }) => decision === "block")) {
  process.exitCode = 10; // This is our harness policy, not a VulnGraph exit code.
}

The evidence object is what I would place in the agent's context. The prompt can remain narrow:

Review the attached vulnerability evidence for this dependency change.

- Never describe `unknown` as safe or clean.
- Preserve the snapshot ID in your report.
- Explain the reason codes in plain language.
- Check whether each affected dependency is reachable from the changed code.
- Propose the smallest compatible upgrade, but do not edit files until approved.

The deterministic layer establishes what was observed and which policy branch applies. The agent contributes the work it is good at: repository-specific investigation, explanation, remediation planning, and communication.

Design lessons for agent-facing security tools

Building the graph was only half the problem. Making it safe for automation required treating the interface as part of the security model.

Make ignorance a first-class result. unknown must survive every layer from data lookup to terminal output to agent context.
Return evidence, not just a score. A verdict should carry stable reasons and source observations that a human can audit.
Separate update time from decision time. Network retrieval belongs in a controlled provisioning phase; checks should be local and reproducible.
Identify the data behind every answer. A content-derived snapshot ID turns “what did the scanner know?” into an answerable question.
Let tools describe their contract. Capabilities, schemas, freshness, and exit semantics should be machine-readable.
Keep gating policy outside the observation tool. Different repositories have different risk tolerances. The harness should own that policy explicitly.

An agent does not become trustworthy because it receives more tokens or a longer system prompt. It becomes more trustworthy when its environment preserves distinctions, constrains side effects, and supplies evidence through contracts that cannot silently change meaning.

That is the role I designed VulnGraph to play: not an autonomous security authority, but a fast, local, auditable source of vulnerability evidence that gives agents and their harnesses something solid to reason from.

Build and release pipeline: copyleftdev/vulngraph-data
Offline CLI: copyleftdev/vulngraph-cli

Disclosure: I designed VulnGraph. This article and its cover were created with AI assistance. I reviewed the technical claims against the linked repositories and live CLI output.

If this open-source work is useful to you, you can support its continued development:

[Boost]

Don Johnson — Sat, 18 Jul 2026 17:47:12 +0000

Don Johnson

Jul 17

The Last Honest Abstraction: Why AI Coding Isn't the End of Engineering

#discuss #ai #programming #softwaredevelopment

5 min read

The Last Honest Abstraction: Why AI Coding Isn't the End of Engineering

Don Johnson — Fri, 17 Jul 2026 16:47:19 +0000

Why every generation thinks the next programmer understands less—and why AI changes the burden of proof, not the need for engineering.

"Developers can't even explain their own code anymore."

If you've spent any time around software lately, you've probably seen some version of that argument.

The culprit, we're told, is AI.

Developers are prompting language models, accepting generated code, and shipping software they couldn't possibly explain line by line. Somewhere between autocomplete and autonomous agents, we supposedly crossed a professional line. We traded craftsmanship for convenience.

It sounds like a new argument.

It's one of the oldest in computing.

Before We Go Any Further...

Before I make my case, I want to pay homage to a rare breed of engineer.

You know who you are.

You're the people who can look at compiler output and immediately tell when an optimization didn't quite land. You read assembly like most people read JavaScript. You understand calling conventions, cache locality, virtual memory, branch prediction, linker behavior, and processor architecture—and you can still build elegant software without turning every project into a dissertation.

You're rare.

And you've quietly carried this industry for decades.

Many of the abstractions the rest of us rely on exist because someone like you spent years wrestling with complexity until everyone else could work one layer higher.

Every generation has its handful of engineers who descend into the deepest layers of the machine, then return carrying tools that allow the rest of us to build something higher.

This article isn't arguing against mastery.

It's a thank-you to the people who made abstraction possible.

Every Layer Was Once Considered Cheating

Again and again, computing has moved upward by adding layers.

Machine code gave way to assembly for most programming. Someone inevitably complained that assembly programmers weren't "real programmers" because they no longer had to manipulate raw binary.

Higher-level languages followed. Compilers hid more of what the processor was actually doing. Managed memory, virtual machines, dynamic languages, frameworks, cloud platforms, containers, and orchestration systems each took another category of complexity and moved it behind an interface.

And now...

AI.

The technologies aren't a single, tidy lineage. They abstract different problems, and old layers rarely disappear. But the cultural pattern repeats:

"Do they really understand what's happening underneath?"

The nouns change.

The complaint doesn't.

The Myth of the Complete Programmer

Here's an uncomfortable truth.

Most software has never been written by people who fully understood every layer beneath it.

That isn't criticism.

It's reality.

Ask an application developer to explain exactly how their operating system schedules threads.

Ask them why their compiler emitted a particular optimization.

Ask them to walk through every page-table translation.

Ask them to trace how cache behavior affected one production slowdown.

Ask them to explain every TCP retransmission or every filesystem recovery after an unexpected shutdown.

Some can.

Most can't.

And yet...

The software still ships.

Not because understanding stopped mattering, but because software engineering evolved around abstractions with bounded responsibilities and increasingly testable guarantees.

The Entire Purpose of Engineering

Engineering has never been about refusing abstraction.

It's been about earning it.

Every abstraction represents an enormous investment of human understanding. Someone had to master the ugly details so the next engineer could focus on solving a different problem.

A civil engineer need not manufacture every bolt before designing a bridge, but must know which loads it can bear. Software abstractions work the same way: we delegate construction, not accountability.

Requiring every software engineer to hand-write machine code before building a web application wouldn't protect craftsmanship.

Progress wouldn't slow down.

It would stop.

Abstractions aren't shortcuts.

They're accumulated knowledge.

They're understanding, compressed into reusable form.

That's not laziness.

That's civilization.

What AI Actually Changed

AI did change something.

Just not what many people think.

A conventional compiler is expected to produce reproducible output from fixed inputs. A language model samples from probabilities. A compiler hides implementation behind a defined system; a model proposes implementation without guaranteeing that the proposal is correct.

That makes AI a less trustworthy abstraction by default—not an invalid one. It means the engineer must supply the trust boundary through review, tests, observability, constraints, and a willingness to reject plausible-looking output.

The real shift is that authorship is becoming an even weaker proxy for understanding—and an even less useful signal of correctness.

For decades, many engineers subconsciously equated "I wrote it" with "I understand it."

Those were never the same thing.

We've all inherited libraries. We've all trusted frameworks. We've all depended on operating systems we didn't write. We've all deployed software built on millions of lines of code authored by strangers.

AI simply made that reality impossible to ignore.

The responsibility hasn't disappeared.

It's moved.

Instead of taking familiarity as evidence because you typed every character yourself, you establish confidence by inspecting the design, challenging assumptions, measuring behavior, and validating outcomes.

This doesn't absolve the engineer of understanding. It changes the required depth. You may not need to explain every generated line from memory, but you do need to understand the system's boundaries, invariants, dependencies, and failure modes well enough to be accountable for what it does.

That's engineering.

Judgment Is Becoming the Scarce Resource

The best engineers I've met have never impressed me with how quickly they type.

They impress me with how quickly they notice something feels wrong.

They can review a thousand lines and, through experience, stop at the seven that deserve attention. They know where abstractions leak. They know when benchmarks are lying. They know when a beautiful architecture is solving yesterday's problem.

They ask better questions than everyone else.

AI doesn't replace that.

If anything, it magnifies its importance.

Syntax gets cheaper.

Judgment becomes priceless.

Standing on Invisible Giants

There's a beautiful irony here.

The engineers most qualified to criticize abstraction are often the same people who built it: compiler engineers, kernel developers, database architects, networking pioneers, language designers, and chip architects.

They spent years making impossibly complicated systems disappear behind clean interfaces—not because they wanted everyone staring at the gears, but because they wanted the rest of us to keep building.

Every abstraction they created became another floor in a building nobody could have constructed alone.

The Last Honest Abstraction

Every generation believes the abstraction immediately beneath them was the last honest one.

Assembly programmers point to machine code.

C programmers point to assembly.

Framework skeptics point to handwritten C.

Now AI skeptics point to handwritten code.

Ten years from now, someone will publish an article insisting developers don't even understand what their autonomous software factories are doing.

Someone else will nod and say:

"Back in my day, we actually wrote the prompts ourselves."

And the cycle will begin again.

Because the history of software has never been the history of losing understanding.

It's been the history of redistributing it.

A relatively small number of extraordinary engineers venture into the deepest layers, wrestle complexity into submission, and transform hard-earned understanding into abstractions that millions of others can build upon without mastering every detail beneath them.

Those abstractions aren't evidence that engineering is dying.

They're evidence that engineering succeeded.

Progress has never required everyone to understand everything.

It has always required enough people to understand each layer deeply enough to build the next one—and everyone who uses that layer to understand its promises and failure modes well enough to take responsibility for the result.

The rest of us stand on their shoulders.

And someday, if we do our jobs well enough, someone else will stand on ours.

Editorial note: This article was refined with AI-assisted editorial review, and its cover illustration was generated with AI. The argument, factual review, and final wording remain the author's responsibility.

A Vibe Is Not a Verdict: I Built a Tool That's Allowed to Say 'I Don't Know'

Don Johnson — Mon, 13 Jul 2026 19:37:38 +0000

A tool should do one thing, do it well, and — this is the part everyone forgets — know exactly where its knowledge ends.

I built kilo for a morning like this one. I just didn't know the morning would come this soon.

The message that was waiting for me

I woke up, poured the coffee, and did the thing you're not supposed to do before you're fully awake: I opened my messages. And there it was, sitting at the top of the inbox like it had been placed there on purpose — a cold recruiter pitch.

An "exciting opportunity" for a Quality Engineer role at an aerospace firm. Warm, professional, faintly urgent. Please click the link below to view the job description and apply. The link was a long tracking URL, the kind with an opaque token and source and method parameters stapled to the end.

And then the signature. One "recruiter" who claimed, in a single line, to be a physical therapist, a call-centre manager, an e-commerce power seller, a freight dispatcher — half a dozen unrelated trades piled into one job title, now, apparently, placing quality engineers at an aerospace company. That's not a résumé. That's a slot machine.

There it was: the exact situation I wrote the tool for, delivered to my inbox before I'd finished the coffee.

Because here's what would normally happen next. My gut says scam. My gut is usually right. But "usually right" is how you eventually click the one link that isn't. A gut feeling is a vibe, and a vibe is not a verdict. That frustration is precisely why, months ago, I sat down and wrote kilo — a way to stop feeling about an address and start knowing about it. So this morning wasn't an annoyance. It was the experiment finally running itself.

I put the coffee down and opened a terminal.

What I actually built, and why

Let me back up and tell you what kilo is, because the design is the whole argument.

KiloCheck is an offline IP-reputation engine. One binary. ~12MB. Written in Rust. Its entire worldview is a four-stage pipeline: ingest a signed, versioned threat release → normalize typed observations → compile an immutable local index → check IPs against it. Cryptographically verified going in. Zero network calls coming out.

I made it offline on purpose. An ordinary check never reaches an intelligence API. When I'm triaging something sketchy, I don't want my queries leaking, I don't want to tip anyone off, and I want the same answer at 3AM on a plane as I get at my desk. The threat knowledge is baked into a signed local snapshot. The check is just a lookup against evidence I already trust.

Install is a checksum and a copy — the way a tool should be:

curl -fsSL https://raw.githubusercontent.com/copyleftdev/kilocheck/v0.2.0/scripts/install.sh | sh
kilo update      # pulls the signed data release → prints "Claims  3160"
kilo status --json
# → "integrity": "verified", "freshness": "fresh"

That was the tool. This was the morning. Time to point one at the other.

The experiment, step one: turn the lure into a number

kilo answers exactly one question — is this IP a known bad actor? — so first I had to turn that tracking link into an address. That's dig's whole job: resolve the hostname behind the URL down to the IP actually serving it.

$ dig +short <the-redirect-host>
203.0.113.90

Then I asked my tool the question I built it to answer. This is the moment the experiment either validates the design or embarrasses me:

$ kilo check 203.0.113.90 --json | jq '.data[0].verdict'

{
  "disposition": "unknown",
  "confidence": 0.0,
  "recommended_action": "monitor",
  "reason_codes": ["NOT_OBSERVED"]
}

NOT_OBSERVED. Confidence 0.0.

And I grinned, because this is exactly the behavior I designed for — and the exact behavior a worse tool would have gotten wrong.

Think about the pressure in that moment. I clearly think it's a scam. The vibe is overwhelming. A tool built to please its operator would have felt that pressure and handed me a scary red verdict to confirm what I already believed. kilo refuses. It says: I have no evidence about this address, and I am not going to invent some to make you feel clever.

That refusal is the single most important thing I built into it. A tool has to know the difference between "safe," "bad," and "I don't know" — and it has to refuse to smear them together. Missing, unknown, stale, incomplete, unsafe: five distinct states in my model, not three shades of the same shrug. This morning it chose the honest one under maximum temptation to lie.

Step two: prove it can still bite

An honest "I don't know" is only worth something if the same tool can say a hard "yes." If kilo returned unknown for everything, it wouldn't be humble — it'd be broken. So I ran the control, right there in my pajamas: the suspicious host, a known command-and-control node, and a boring public DNS resolver, all in one shot. Distilled from the JSON:

203.0.113.90      → unknown   monitor   NOT_OBSERVED        (0 sources)
162.243.103.246   → CRITICAL  block     COMMAND_AND_CONTROL (1 source, conf 0.99)
8.8.8.8           → unknown   monitor   NOT_OBSERVED        (0 sources)

There. A known C2 node gets critical / block / 0.99 with provenance attached. The public resolver and the suspicious host both get a calm NOT_OBSERVED. No false alarms on clean infrastructure, no missed alarm on a real one. The experiment held: the tool's silence means something precisely because its alarm means something.

Step three: the twist my own tool pointed me toward

So the infra came back clean. Was my gut wrong? No — it was pointed at the wrong target. And that's the part I didn't see coming.

kilo's honest negative didn't close the case. It moved it. "The danger is not in this infrastructure" is a coordinate — it told me to go look somewhere else. So I followed the rest of the trail with the neighbors on my toolbelt.

curl followed the redirect — headers only, I never touched the body:

HTTP/2 302 → /<captcha-gate>?...&source=...&method=...

openssl read the cert straight off the wire: a real, correctly issued certificate for a legitimate, well-known job-traffic aggregator. And a quick lookup closed the loop: the aerospace company in the pitch is real, with genuine engineering openings, and the tracking link was an affiliate click-tracker redirecting through the aggregator's own anti-click-fraud gate. No payload. No credential harvest. No bad IP anywhere in the chain.

So what was the actual threat? It was never in the packets. It was the sender — a freelance lead-gen spammer blasting affiliate commission links across social platforms, wearing a recruiter costume stitched together from half a dozen unrelated careers.

And here's why I'm writing this at all: my own tool solved the case by refusing to pretend. Its 0.0 sent me away from a malware rabbit hole and toward the social-engineering signal where the truth actually lived. If kilo had cried wolf on clean infrastructure to match my mood, I'd have wasted the morning dissecting a job board. Because it stayed honest, it handed me the thread on the first pull.

That is the exact experiment I wrote the tool to run. And it passed.

What the morning actually taught me

The recruiter was never the point. The shape of the morning was.

I didn't solve this with one brilliant tool. I solved it with a chain of small, honest ones: dig turned a name into an address so kilo had something to check; kilo's honest negative reframed the question so curl and openssl knew where to look. And every one of them spoke stable JSON — so I was consuming contracts, not parsing prose. Exit codes matter. Schemas matter. Failure stays legible.

That's the manifesto I keep coming back to as I build these things: contracts, not vibes. When I pipe kilo check --json into jq, I'm having a typed conversation with reality. I can't hallucinate a verdict, because the verdict arrives as a signed struct with its provenance stapled on.

It's the difference between a tool that sounds right and one that is right:

A gut feeling is an opinion. A sharp, composable, honest toolbelt is a witness.

And witnesses — unlike hunches — can say the three most valuable words in this whole trade: "I don't know." Then go find out.

Steal the pattern if it's useful:

Small, single-purpose tools that speak stable JSON compose into something no chatty do-everything API can match.
Honesty beats confidence. A tool that returns unknown under pressure is worth ten that guess to please you.
Let the negatives navigate. The most useful result of the entire morning was a 0.0.

So the next time a link makes your gut twitch, don't argue with the vibe. Resolve it, and ask a tool that's allowed to say nothing.

kilo didn't catch a scam — there wasn't one, just spam in a lab coat. It did something I value more: it told me the truth about what it knew, drew a hard line around what it didn't, and pointed me at where the real answer was hiding.

I built a 12MB binary to say "not observed" without flinching, on the exact morning I most wanted it to lie. It didn't. That's the whole experiment. That's why I built it.

kilo check <your-suspicions-here> --json

Tooling in this piece: KiloCheck · the eternal dig / openssl / curl. The story is real; the identifying details have been filed off.

The AI Cost-Modeling Handbook: I let Claude do the modeling, but never the arithmetic

Don Johnson — Wed, 01 Jul 2026 02:51:48 +0000

Every "what's the cheapest model?" thread online is people trading vibes. I got tired of it, so I built a pipeline that pulls live, cited prices and runs the numbers through an exact-rational math kernel — no floating-point drift, no LLM hallucinating a multiplication. Then I pointed it at eight of the cost questions every agent builder actually faces. Here's everything it found, and the repo so you can re-run all of it.

There are two kinds of LLM cost advice. The first is a benchmark leaderboard with a price column, which tells you nothing about your workload. The second is a confident tweet that's quietly wrong because someone multiplied a per-million price by the wrong token count in their head.

I wanted a third kind: a model you can audit. Re-run it and you get bit-identical numbers, with the source of every input price one file away. This article is the result — a tour through eight cost decisions, each answered with real money math, and each one teaching something that intuition gets wrong.

Let's start with the question that kicked it off.

Cold open: what's the most cost-effective model to run an agent on?

The setup: I run Hermes Agent, a model-agnostic agent framework, and I wanted the cheapest token provider to drive it. Not cheapest per token — cheapest per unit of quality, because a model that flubs tool calls and retries can cost more than a pricier reliable one.

So the metric is blended cost ÷ agentic-quality-score. Blended cost uses a production-agentic token mix (more on that below); quality is a normalized agentic score (mean of BFCL / τ²-bench / SWE-bench Verified). Filters: open-weights, prompt caching, no-train.

Model @ Provider	Blended $/1M	Quality	$/quality (×1000)
DeepSeek V3.2 @ OpenRouter	0.1145	77	1.49
DeepSeek V3.2 @ DeepInfra	0.1951	77	2.53
MiniMax M2 @ MiniMax	0.2629	73	3.60
GLM-4.6 @ z.ai	0.5276	71	7.43
Kimi K2 @ DeepInfra	0.6613	66	10.02
DeepSeek R1 @ DeepInfra	0.6519	54	12.07

DeepSeek V3.2 wins decisively — it's both the highest-quality open model in the set and near the cheapest, because its output token price is absurdly low for its tier. The emerging best deal (pending a quality test) is DeepSeek V4 Flash on Fireworks at $0.0896/1M blended — cheapest in the field, with ZDR-by-default and a 50% batch discount.

That's a useful answer. But the interesting part is everything that question opens up. If you can split traffic across providers, what's the optimal mix? Should you self-host instead? Is the "smart expensive model" actually cheaper once you count its thinking tokens? Each of those is a chapter.

First, how this is built.

The toolchain: research agents + an exact-math kernel

The pipeline has two halves, and the whole point is that the LLM never touches the arithmetic.

1. Research agents gather live, cited inputs. Token prices in 2026 move monthly — half the models in my training data are already legacy. So every price and benchmark in this guide came from a research agent doing live web search/fetch, returning a structured table with source URLs and observation dates. Those land in data/ (pricing.md, quality.md, gpu.md, frontier.md, decline.md), each one auditable.

2. An exact-rational kernel does the math. I used agent-calc, an AI-native computation kernel that works in exact rationals — 3/5 × 1,000,000 returns 600000, not 599999.9999998. It exposes typed domains: rational, linear (LP), finance (NPV/amortization), solve (algebra), stats (distributions), polynomial, interval (uncertainty arithmetic). Every chapter below drives a different one.

Why bother? Because cost models are exactly where floating-point lies and LLMs fumble: tiny per-token prices, huge token counts, compounding rates. When an answer is "self-hosting breaks even at 87.2% utilization," you want that to be computed, not vibed. And the secret structure of this guide is that each chapter answers a money question and exercises one kernel domain — it's a cost-modeling handbook that doubles as a tour of exact computation.

The shared workload model

Every chapter uses one production-agentic token profile, because agent tool-loops are wildly input-heavy:

Component	Share of tokens
Fresh input	21.25%
Cached input	63.75%
Output	15%

That's an 85/15 input/output split with 75% of input being a cache hit (the system prompt, tools, and skills get re-sent every tool-loop step). Blended price = 0.2125·in + 0.6375·cached + 0.15·out. Chapter 7 explains why this profile is forced on you; Chapter 6 explains why the cache hit rate is so high. For now, take it as the workload.

Chapter 1 — Optimal multi-provider routing (a linear program)

You don't have to pick one provider. Run a cheap one for the bulk, escalate elsewhere when constraints demand. The optimal split — minimize spend subject to a quality floor, per-endpoint capacity caps, and a US-jurisdiction floor θ — is a literal linear program. agent-calc solves it (solve_lp), then I re-verify the winning allocation's cost in exact rationals (the LP solver is f64; the headline number shouldn't be).

Sweeping the privacy knob θ traces a cost-of-privacy frontier:

US floor θ	min $/1M	$/mo @ 1B	allocation
0.00–0.40	0.1468	$146.77	A=0.60, B=0.40
0.60	0.1629	$162.89	A=0.40, B=0.60
0.80	0.2716	$271.61	A=0.20, B=0.60, F=0.13, H=0.07
0.872	0.3116	$311.59	A=0.13, B=0.60, F=0.27
≥ 0.873	—	INFEASIBLE	the privacy wall

Three things intuition misses:

The first 40% of US-jurisdiction traffic is free — the cost-optimal blend already lands there, because the cheapest provider is capped at 60% and the overflow falls on a US endpoint anyway.
The marginal cost of privacy is convex — 40→60% is cheap; past 60% the quality floor forces in weaker fillers and the bill nearly doubles by 87%.
There's a hard wall at 87.2%. Only one high-quality US endpoint exists in this menu; beyond its capacity the math is infeasible. To break the wall you need a second high-quality US provider, or you accept single-point-of-failure risk.

Chapter 2 — Self-host vs. serverless: it's a utilization question

100% utilization (impossible); only owned 8×B200 wins at 72%" width="800" height="459">

Everyone's instinct: "at some volume, renting GPUs beats per-token APIs." The reframe that changes everything: a self-hosted node costs the same at 5% load or 95% load, while the API scales linearly with tokens. So it's not a dollar break-even — it's a break-even utilization: how busy must the box stay to win?

u* = node_$/mo × 1e6 / (capacity_tokens/mo × API_$/1M)

If u* > 100%, self-hosting loses even at full tilt. Modeling DeepSeek V3.2 (8-GPU node) against DeepInfra's $0.195/1M, with agent-calc handling the owned-hardware amortization (finance) and the exact u* (eval):

Hardware / procurement	break-even u*	verdict
8×H200, rent on-demand	531%	never
8×H200, own (amortized)	220%	never
8×B200, rent reserved	174%	never
8×B200, own (amortized)	72%	the only winner

For a commodity model this cheap on the API, renting GPUs loses in every configuration — the serverless provider batches across thousands of tenants to hit a utilization you can't. The only config that wins is owned Blackwell silicon kept above ~72% utilization 24/7 (~56B tokens/month of steady load). And against the falling API floor ($0.0896/1M), even that roughly doubles its break-even. The takeaway: self-host for privacy, control, or latency — not to cut the token bill.

Chapter 3 — The reasoning-token tax

Reasoning models emit hidden "thinking" tokens, billed at the output rate. A 600-token answer can bill 7,800. Does the higher success rate justify the burn? The honest unit isn't $/token — it's cost per solved task = attempt cost ÷ success rate.

With agent-calc computing exact costs (eval) and the break-even success rate (solve):

Model	k (think mult.)	$/solved	×champion
DeepSeek V3.2	4	3.17m	1.00×
MiniMax M2	2	4.03m	1.27×
Kimi K2	1	8.79m	2.77×
GLM-4.6	5	13.77m	4.35×
DeepSeek R1	12	36.80m	11.61×

(m = milli-dollars = $0.001/task.)

The killer result: DeepSeek R1 would need a 627% success rate to match V3.2's cost-per-solved — mathematically impossible. Even a perfect R1 stays far more expensive, because its per-attempt token burn alone exceeds V3.2's entire cost-per-solved. The sweep confirms it: even at k=0 (zero thinking), R1 is still 2.6× the champion.

The rule: the reasoning tax only pays when output tokens are cheap. Thinking is a multiplier on the output price. Pay $2.15/M and ×12 and you've built the most expensive possible way to solve a task; pay $0.38/M and the same multiplier barely registers.

Chapter 4 — The retry cascade

You don't need one model for every task. Run a cheap model on everything, escalate to a reliable premium model only on the residual failures. agent-calc computes the escalation probability (stats/binomial_pmf) and the exact expected cost (eval).

Strategy	cost/solved	coverage	% → premium
Cheap-only (DeepSeek V3.2)	3.17m	77.0%	0%
Premium-only (Claude Opus 4.8)	82.96m	88.0%	100%
Cascade V3.2 → Opus	19.78m	97.2%	23%
Cascade V3.2 → Gemini → Opus	13.52m	99.5%	4.4%

The cascade beats premium-only on both axes: higher coverage (97% vs 88%) at a quarter the cost, because only 23% of tasks reach the expensive model. Adding a mid-tier pushes coverage to 99.5% while cutting cost further — the cheap+mid tiers absorb 95.6% of traffic, so the $73m Opus call fires on 4.4% of tasks.

And the trap to avoid: retrying the same model never changes cost-per-solved (it stays C/p), and for deterministic failures it doesn't even raise coverage. Only escalating to a different, stronger model moves the needle. "Just add retries" is a no-op; "add a tier" is the win. (The cost advantage survives even strong failure correlation — 3.6× cheaper than premium-only even when the premium rarely rescues what the cheap model botched.)

Chapter 5 — The NPV of waiting

A provider offers a discounted reserved rate if you commit for a year. Locking it in feels prudent — but token prices fall fast, and you might be locking yourself above where pay-go will be in six months. This is a present-value problem: NPV the committed cash-flow stream against the declining pay-go stream (agent-calc finance/net_present_value), and find the break-even decline rate.

The decisive input is that price decline depends entirely on what you hold fixed:

Fixed-quality (same capability getting cheaper): ~80%/yr (a16z/Epoch).
Frontier (always run the best): ~30%/yr, sometimes flat or rising.

At a 30%-off, 12-month commit:

Workload	central decline	winner
Fixed-quality	80%/yr	pay-go (−23%)
Frontier	30%/yr	commit (+22%)

And the break-even decline grid shows committing never wins for commodity models (80%/yr exceeds every reservable discount's break-even), while longer terms always make it worse: a 30% discount breaks even at 57.8%/yr over 12 months but only 24.5%/yr over 36 months. The rule: reserve where prices are sticky (frontier); ride the curve where they fall fast (commodity). Never sign a 3-year inference commit in a market falling >25%/yr.

Chapter 6 — Prompt-cache ROI

Caching looks free, but a cache write often costs more than a normal input token (Anthropic charges up to 2×), while reads are cheap. So it's a bet: pay the write premium to save on reads, and it only pays if you reuse enough before the cache expires. The break-even reuse count is N* = (write − read)/(in − read).

Provider	break-even N*	savings ceiling
DeepInfra / DeepSeek / OpenAI / Fireworks (no write fee)	1.00	50–90%
Anthropic Opus 5-min TTL (1.25× write)	1.28	90%
Anthropic Opus 1-hr TTL (2× write)	2.11	90%

Caching pays almost immediately — most providers from the second call, even the harshest write premium by the third. The read multiplier sets the long-run ceiling (0.1×-read → 90% savings; 0.5×-read → 50%), so for cache-heavy loops that's the number that compounds. Using agent-calc's interval domain, an 8k-token agent prefix reused an uncertain 3–20 times costs $0.088–0.156 cached vs $0.12–0.80 uncached — 80% off at the high end, even with a 2× write premium. This is why the guide's 75% cache-hit assumption holds: agent loops reuse a large prefix 10–20× per task.

Chapter 7 — Agent-loop compounding (the O(K²) tax)

Why is that reused prefix so large? Because a multi-turn agent re-sends its growing context every step — at step k, the input is the base prompt plus every prior tool call and result. Per-step input grows linearly, so cumulative input over a K-step task grows quadratically:

cumulative over K = (δ/2)·K² + (B − δ/2)·K

agent-calc evaluates that polynomial exactly (polynomial/evaluate):

K steps	cum tokens	naive $	cached $	compacted $
10	147,500	0.0384	0.0220	0.0384
50	2,237,500	0.5817	0.3015	0.3267
100	8,225,000	2.1385	1.0896	0.6907

A 100-step task isn't 10× a 10-step task — it's 56×. Your 50th tool call's input costs 10× your first. Caching bends the curve (lowers the constant); only compaction breaks it — capping the context window turns the quadratic linear, and the gap widens without bound as loops get longer. They compose: cached and compacted is read-rate pricing on a linear token count. For any loop past ~20 steps, do both.

This closes the loop on the whole guide: the quadratic re-send is exactly why agent workloads are so input-heavy and so cache-dependent — the entire shared workload profile falls out of this one chapter.

The cheat sheet

Everything above, compressed to decisions:

Decision	Verdict
Which model?	Cheap efficient hybrid (DeepSeek V3.2-class). Cheap output + high success beats everything on cost-per-solved.
One provider or many?	Blend via LP. First ~40% of any jurisdiction/diversity constraint is usually free.
Self-host?	No — for cost. Renting always loses for commodity models; only owned next-gen hardware at >72% utilization wins. Self-host for privacy/control.
Reasoning model?	Only if its output tokens are cheap. A pricey heavy reasoner can be mathematically uncatchable on cost-per-solved.
Retries?	Cascade to a different, stronger model — never retry the same one. 3 tiers ≈ 99.5% coverage at ~6× lower cost than going straight to frontier.
Commit to reserved capacity?	Only for sticky frontier prices, short terms. Never for commodity models in a market falling 50–90%/yr.
Caching?	Always, for any reused prefix. Pays from call 2–3. Prefer 0.1×-read providers.
Long agent loops?	Cap the context window (compaction) past ~20 steps, or pay quadratically.

Reproduce it

The whole thing is a repo. Each chapter is a self-contained script that shells out to agent-calc; each input price lives in data/ with a source URL and observation date.

git clone https://github.com/copyleftdev/rational
make all       # re-run every chapter
make ch04      # or just one

Same inputs → bit-identical outputs, every time. The LLM did the modeling and pulled the data. The math kernel did the math. That division of labor is the only reason I trust any number in this article — and it's the part I'd encourage you to steal.

AI Didn't Invent Slop. It Only Made It Infinite.

Don Johnson — Sat, 27 Jun 2026 00:01:32 +0000

Every generation believes it is witnessing the death of taste.

The printing press was going to flood the world with trash. The cheap camera was going to destroy photography. The internet was going to drown serious thought in blogs, forums, spam, and noise. Smartphones were going to turn every sacred moment into a blurry rectangle.

Now artificial intelligence, we are told, has brought us to the final collapse: endless images, endless songs, endless essays, endless pull requests that compile and ship and mean nothing.

We call it AI slop.

And to be fair, much of it deserves the name.

You can feel it when you see it. The over-polished fantasy portrait. The too-perfect lighting. The fake article that says everything and means nothing. And if you write software: the 600-line PR that passes CI and no human can explain. The README generated by a machine that has never run the code. The test suite with 100% coverage that asserts only that the mock returned the mock.

It is work nobody will remember making.

It has texture but no conviction. Output, but no memory of why.

But AI did not invent slop.

It only made it infinite.

We Have Been Here Before

There was a time when a photograph meant something simply because it was difficult to make.

A camera was not just an object. It was a ritual. Film had to be bought. Shots had to be chosen. Light had to be respected. Mistakes cost money. The photographer carried the burden of scarcity.

Then cameras became cheap, then digital, then they entered every pocket on Earth. Suddenly, humanity became a species of photographers. We photographed lunch, and mirrors, and sunsets we barely watched. We created billions of images, most of them forgotten before the day ended.

Photography did not die.

But the average photograph became meaningless.

That distinction matters. The medium survived. The scarcity did not.

No one calls it "camera slop," but perhaps we should have.

Slop Is What Happens When Creation Becomes Cheap

This is the uncomfortable truth underneath the AI panic:

Whenever the cost of creation collapses, the volume of mediocre work explodes.

That is not a flaw unique to artificial intelligence. It is one of the oldest patterns in media history — and it has names. Sturgeon's Law already told us that ninety percent of everything is forgettable. What changes, across eras, is only how cheaply we can manufacture the ninety percent.

When printing became cheaper, the world got pamphlets, propaganda, gossip sheets, and disposable novels. When desktop publishing arrived, everyone became a designer. When blogging platforms arrived, everyone became a columnist. When GitHub made it trivial to publish code, every weekend produced ten thousand repos that would never see a second commit.

Every tool that democratizes creation also democratizes mediocrity.

That sounds cruel, but it is not. It is simply math. When the gates open, more people enter. Most are beginners. Many are opportunists. Some are brilliant. The room gets louder before it gets better.

AI is not breaking that pattern. AI is accelerating it.

Slop Is a Taste Crisis, Not a Technology Crisis

The real danger of AI slop is not that machines can make things.

The danger is that humans may stop caring whether the things are worth making.

That is the disease. Not automation. Indifference.

Slop is not defined by the fact that AI was used. It is defined by the absence of intention. A human can make slop with a camera. A human can make slop with a paintbrush. A human can make slop with vim and forty years of experience and a million-dollar budget.

Slop is not a medium. Slop is a relationship to creation.

It is what happens when the goal is output instead of meaning. Velocity instead of correctness. Shipped instead of understood.

The machine did not create that hunger. The machine simply feeds it faster.

The System Loves Slop

Slop thrives because the systems around us reward frequency.

Merge faster. Chase the metric. Bump the contribution graph. Worship the green build. Close the ticket. Remove the silence where thinking used to live.

AI fits perfectly into this economy because AI is tireless. It does not sleep, or get bored, or stare at a failing test for three days before finding the one assumption that was wrong from the start.

The pipeline wants endless output. AI can provide it.

That marriage — between machine generation and a culture that measures motion instead of meaning — is where slop becomes industrial. Not because AI is evil. Because infinite production met infinite distribution, and nobody in the middle asked whether any of it mattered.

Copilot Did Not Make You an Engineer

A cheap camera allowed anyone to take a picture. It did not make everyone a photographer.

A laptop allowed anyone to make music. It did not make everyone a musician.

A blog allowed anyone to publish. It did not make everyone a thinker.

Copilot lets anyone generate working code. It does not make everyone an engineer.

This is where the confusion begins. We keep mistaking access for mastery.

The tool can produce the surface of a thing. It can mimic the genre, satisfy the linter, pass the tests it also wrote. It can imitate the grammar of competence.

But engineering is not merely code that runs.

It is selection, restraint, and the refusal to write the clever abstraction that will rot in eighteen months — even though the model offered it, ready to paste, today. It is a human being deciding that one design matters more than another and being able to say why.

AI can generate a thousand implementations in an hour.

The engineer is the one who knows which one should not exist.

The Scarce Resource Was Never the Tool

For decades we confused engineering value with the difficulty of execution.

The person who could write tight assembly possessed rare power. The person who could wrangle pointers, or hold a distributed system in their head, or make the build green, possessed rare power.

But technology has a way of eating technique. Again and again, the machine absorbs what once required years of specialized execution. And every time, people panic and say the craft is dead.

But the craft is not dead. It moves.

When execution becomes easier, taste becomes more important. When production becomes abundant, judgment becomes more important.

When everyone can generate a pull request, the real question becomes: who can read it? Who can see the load-bearing line? Who can look at infinite plausible code and say, "Not that. This."

But Abundance Is Not the Enemy of Craft

Here is the part the optimists get right and the part they get wrong.

The right part: every explosion of mediocrity also expands the possibility of genius. Cheap cameras gave us billions of forgettable images — and handed visual language to people who would never have entered a photography school. Home studios created oceans of bad music — and gave us bedroom producers who reshaped global sound. The early web buried us in spam — and also gave us Linux, Wikipedia, and Stack Overflow, each built from the same flood of amateur contribution that produced the junk.

Democratization tends to move in waves. The first wave is noise. The second is imitation. The third discovers what the medium is actually for.

The wrong part — the part the optimists wave away — is that the third wave is not guaranteed. Plenty of ecosystems hit the noise wave and simply stayed there, or died: local newspapers did not ascend to some higher form, they collapsed and did not come back. "It worked out before" is not a law of nature. It is a bet. The honest claim is narrower: abundance makes a better equilibrium possible. It does not deliver one. Someone has to do the choosing.

But Won't the Machine Just Do the Choosing Too?

This is the strongest objection to everything above, so let's not dodge it.

Every earlier democratization left one thing untouched: a human layer that decided what was good. The press multiplied books, but humans still chose what to read. The camera multiplied images, but humans still chose what to hang on the wall. The scarce resource could shift to judgment precisely because judgment stayed ours.

AI attacks both sides of that deal at once. It floods faster than any reviewer can read — and it is being trained, right now, to do the reviewing: to rank, to critique, to pick the better of two diffs. If the machine can write the thousand implementations, why can't it choose the one that should exist?

Partly, it can. Taste is not magic. Much of it is pattern, and patterns are trainable. An honest version of this argument has to concede that "humans simply judge better" is a moat that will not hold — the same way "humans simply write assembly better" did not hold.

But there is a remainder, and it is not a skill. It is accountability.

Someone has to be the one who says ship it and owns what comes next — the outage, the breach, the regression that reaches a million users, the quiet harm nobody modeled. A system can rank the options. It cannot be answerable for the choice. The engineer who clicks merge is not just exercising sharper judgment than the generator; they are the human a consequence can attach to.

So the moat is not "we judge better." That one erodes. The moat is "we are responsible." Discernment is the craft you sharpen. Accountability is the part the machine structurally cannot take, because responsibility requires someone who can actually be held to it.

The Engineer Becomes an Editor of Infinity

The next great technical skill may not be generation. It may be discernment exercised under that responsibility.

We already have a name for the work: review. And it is moving from the chore at the edge of the job to the center of it.

Watch what AI does to a team. It lifts a junior's output to senior-looking quality almost overnight — clean, idiomatic, plausible. It does not lift their judgment at the same rate. So the senior becomes the bottleneck, reviewing an infinite stream of confident, well-formatted diffs from people (and models) who cannot yet see what is wrong with them. "LGTM" stops being a rubber stamp and becomes the most dangerous sentence in the codebase. The reviewer becomes the rate limiter on quality for the entire organization.

That is the job now. Not typing faster. Reading better.

The engineer of the next decade is part architect, part editor, part skeptic — someone who can run the machine without serving it, who generates without limit, then cuts without mercy. Someone with memory. Someone with scars from the last system that paged them at 3 a.m.

Because when any code can be written instantly, the most valuable thing you can do is refuse to merge most of it.

Slop Is the Shadow of Democratization

AI slop is real. It is everywhere — feeds, marketplaces, search results, issue trackers, and the diff you have to review this afternoon.

But it is not an alien substance. It is the shadow that appears whenever creation becomes easier than judgment. Scarcity used to do some of our filtering for us: difficulty kept the repo smaller, cost slowed people down, the sheer friction of writing it yourself meant you mostly wrote what you meant. Now that friction is gone, and the burden shifts back to us. We have to decide what matters.

The cheap camera did not destroy photography. It destroyed the illusion that every photograph was precious. AI will not destroy engineering. It will destroy the illusion that every act of generation is creative.

And maybe that is necessary. Maybe we are being forced to admit what was always true: the magic was never in the tool. The magic was in the seeing and the choosing — in taking the chaos of a problem and shaping it into something that says:

I was here. I understood this. This mattered.

In the age of infinite slop, that may become the rarest thing of all.

Notes on the Argument

This is a historical argument, not a technological one, and it is a synthesis rather than a discovered law — so here is the lineage it rests on, made explicit so you can check it:

Sturgeon's Law — "ninety percent of everything is crap." The baseline claim that most output in any field is forgettable. (ref)
Clay Shirky's "mass amateurization" — when you remove the barrier to publishing, you don't get less junk, you get "publish, then filter": the filtering moves downstream, onto the reader. This essay's "engineer as editor" is that filter, relocated to code. (ref)
The Jevons paradox — making something cheaper to produce raises total consumption rather than lowering it. Cheaper creation means more creation, not less — now widely invoked for AI. (ref)

The concrete parallels are well documented: the printing press produced a flood of cheap Reformation pamphlets alongside its enduring works (ref); consumer and smartphone photography pushed the world past a trillion-plus photos a year (ref); desktop publishing democratized design (and gave us the "ransom note" era), while content farms like Demand Media industrialized algorithmic filler until Google's Panda update was built to bury it (ref).

The argument deliberately holds two ideas at once: democratization of creation (good — more people gain the tools) and a decline in average quality (the expected cost when participation grows from thousands to millions). They are complementary, not contradictory. Reasonable people can disagree with where it lands. The point is to offer a frame for the debate, not to claim it is settled.

A Note on Collaboration

This essay was written through a collaboration between a human author and AI. The ideas, the thesis, the structure, and every editorial decision came from a human; AI was used as a thinking partner and drafting tool, and the historical claims above were checked against the sources cited.

Writers have always worked with tools — editors, dictionaries, research assistants, cameras, compilers, spell checkers, search engines. AI is another one, albeit a remarkably capable one. The quality of a work has never been determined solely by the sophistication of the tool that helped make it. It has always been determined by the quality of the thinking behind it.

Judge this essay by its reasoning. Not by the instrument used to write it.

(Disclosed per DEV's guidelines for AI-assisted articles.)

Cogs and Sparks: Who Wins When Machines Learn to Speak Human

Don Johnson — Sat, 13 Jun 2026 22:57:15 +0000

Disclosure: this article was written from my own thesis and lived experience, then shaped and edited with AI assistance. I reviewed the argument, revised the structure, and stand behind the ideas here.

For most of the digital age, humans had to learn how to think like machines.

We memorized syntax. We learned rigid menus, brittle interfaces, command lines, templates, ticket flows, and rules that punished ambiguity. We trained ourselves to break imagination into tiny mechanical instructions because computers could not meet us halfway.

To make technology useful, we first had to translate ourselves into its language.

That translation did not just shape software. It shaped people.

The modern workplace rewarded the person who could become a little machine-like: precise, procedural, reliable, consistent, patient with repetition, able to memorize commands and tolerate the tedious work required to keep the system moving.

Those people mattered. They still matter.

They were the cogs.

And for a long time, being a good cog was a survival strategy.

But AI changes the direction of adaptation.

For the first time, machines are learning to move toward us. They are learning language, context, style, pattern, intent, image, tone, code, conversation, and approximation. They are still flawed. They still hallucinate. They still need judgment, taste, and correction.

But the direction is unmistakable.

The machine is no longer asking only:

Can you speak machine?

It is beginning to ask:

Can you show me what you mean?

That shift changes who gets leverage.

The old world favored the cog.

The new world favors the spark.

The Cog Was Never the Problem

A cog is not an insult.

Cogs are dependable. Cogs execute. Cogs maintain order. Cogs remember how the system works when everyone else is chasing novelty. Cogs show up. They keep the lights on. They learn the process and repeat it with precision.

Every serious system needs cogs.

The problem is not the cog.

The problem is a world that taught people the safest way to survive was to become only a cog.

For decades, technical leverage lived behind gates. If you could not code, you could not build software. If you could not design, you could not ship a product experience. If you could not navigate the tools, you needed someone else to translate your idea into execution.

That gatekeeping was not always malicious. Much of it was just mechanical reality. Computers were powerful, but they were not generous. You had to meet them on their terms.

So the system rewarded people who could memorize the terms.

But many people never fit that world cleanly.

They were visual thinkers in text-only systems. Writers in spreadsheet cultures. Artists trapped behind tool complexity. Teachers with product instincts but no engineering team. Generalists who could see connections but lacked the credentialed path to execute them.

They were sparks in a cog-shaped world.

What a Spark Does

A spark sees possibility before there is a process.

A spark experiments before there is permission.

A spark can look at two unrelated things and feel the bridge between them before they can fully explain it.

Sparks ask strange questions. They remix. They test. They wander. They notice tone, friction, story, behavior, emotion, and timing.

Sparks are artists, writers, designers, teachers, builders, founders, comedians, strategists, researchers, tinkerers, and curious people who were often told they were impractical because their value did not fit the spreadsheet.

In the old world, sparks often needed cogs to translate them.

In the AI world, sparks get a new kind of leverage.

AI lowers the cost of execution. It can draft, summarize, code, calculate, organize, translate, simulate, critique, generate, and iterate. It can turn a vague sketch into a prototype, a paragraph into a campaign, a napkin idea into a working demo, a conversation into a plan.

That does not make the work automatic.

It moves the bottleneck.

The question is no longer only:

Who can perform the task?

The question becomes:

Who can imagine the right thing worth doing?

The Bottleneck Moves Upward

When execution gets cheaper, taste gets more expensive.

When syntax gets easier, judgment matters more.

When tools can produce ten versions in seconds, the valuable person is not the one impressed by abundance. It is the one who can tell which version is alive.

That is the real AI transition.

Not from human to machine.

From memorization to judgment.

From task completion to problem framing.

From obedience to direction.

From labor as identity to creation as leverage.

The person who can only follow instructions will increasingly compete with machines that follow instructions faster.

The person who can create better instructions, better constraints, better questions, better taste, and better visions becomes more powerful.

This is why the spark matters.

A spark does not merely ask an AI to make something. A spark senses what is missing. A spark knows when the output is technically correct but emotionally dead. A spark can say:

No, not that. Make it quieter. Make it sharper. Make it feel like a system designed by someone who has actually lived this problem.

That is not button-pushing.

That is direction.

What This Means for Developers

Developers are in a strange position because we are both protected and exposed by this shift.

We understand the machinery better than most people. That matters. AI-generated code still needs review, architecture, debugging, security judgment, and production discipline. The machine can create plausible code faster than it can understand consequences.

But we should not mistake that for safety.

If our value is only that we remember syntax, our value is shrinking.

If our value is that we can reason about systems, frame problems, notice edge cases, understand users, evaluate tradeoffs, and build things that survive contact with reality, our value compounds.

The best developer in the AI era is not the one who refuses the tool.

It is also not the one who blindly trusts it.

It is the one who can use the tool without surrendering judgment.

The Cog Has to Remap

If you were trained by the old world, this shift can feel insulting.

You spent years learning the hard way. You memorized commands. You built discipline. You earned your scars. Now a beginner can prompt a machine and get something that looks, at first glance, like the thing you worked years to produce.

That reaction is understandable.

But resentment is a bad strategy.

The cog does not need to disappear. The cog needs to remap.

Here are the shifts I think matter most:

From certainty to experimentation. In the old world, being right early was rewarded. In the AI world, discovery often comes from fast comparison.
From memorization to judgment. Facts are easier to retrieve. Syntax is easier to generate. But judgment remains scarce.
From task completion to problem framing. A weak frame creates polished garbage. A strong frame creates leverage.
From obedience to imagination. When machines can produce correct-enough work quickly, direction becomes practical.
From effort to impact. If something becomes easier, it does not become meaningless. It means the value moves upward.

The question is not whether you can still do the hard thing manually.

The question is whether you can aim the system toward something worth building.

The New Creative Class Is Broader Than We Think

The painter who could not code can now prototype an app.

The writer who could not design can now shape a visual world.

The teacher who could not afford a product team can now build a learning experience.

The mechanic can become an automation builder.

The founder without funding can test the first version before asking anyone for permission.

The future may not belong only to the people who were best at computers.

It may belong to the people who were best at being human, now armed with computers that can finally keep up.

This does not mean technical skill stops mattering.

Deep expertise still matters. Engineering still matters. Precision still matters. In fact, expertise may matter more because AI can produce confident nonsense at scale. Someone has to know what good looks like.

But the ceiling rises for people who were previously blocked by technical gates.

The next great product designer might be a poet.

The next great software founder might be a teacher.

The next great systems thinker might be someone who never called themselves technical because the old tools made them feel stupid.

They were not stupid.

They were early to a world that had not built the right interface yet.

The Hybrid Wins

The spark alone is not enough.

Unfocused creativity becomes noise. Ideas without discipline become vapor. Taste without execution becomes performance.

The cog alone is not enough either.

Perfect execution of the wrong thing is still waste. Process without imagination becomes maintenance of a dying machine.

The people who win are hybrids.

They have the discipline of the cog and the imagination of the spark. They can execute, but they can also question the premise. They can use the tool, but they are not hypnotized by it. They can move fast without confusing motion for meaning.

The cog must learn to spark.

The spark must learn to aim.

And the best builders will become both: precise enough to finish, imaginative enough to invent, and adaptable enough to keep changing as the tools change.

The Machine Is Bending Toward Us

The great mistake is to treat AI as merely a faster machine for old work.

That is the shallow reading.

The deeper reading is that AI changes who gets to participate. It gives language, imagination, taste, and judgment a new kind of leverage. It lets people who were once trapped outside the machinery step closer to the center.

For years, humans had to bend toward the machine.

Now the machine is bending toward humans.

That is not the end of human value.

It is the beginning of a different test.

Not:

Can you remember the command?

Not:

Can you tolerate the process?

Not:

Can you make yourself small enough to fit the machine?

But:

Can you see what is possible?

Can you tell what is good?

Can you aim the spark before it burns out?

That is the work now.

AI-assistance note: I used AI to help structure and edit this essay, but the central argument, final review, and publication responsibility are mine.

Topics: #AI #Programming #DeveloperCareer #Creativity

Stop Feeding Agents Raw Data

Don Johnson — Tue, 09 Jun 2026 06:05:05 +0000

I used to think the problem was the agent.

I would hand it a large JSON export and ask a reasonable question: what changed, what looks risky, what should we investigate before release?

It would find something. It always found something.

But it missed fields. It over-indexed on irrelevant values. It hallucinated patterns in JSON that weren't there. It noticed one dramatic-looking record and ignored the boring distribution that made the record meaningful. So I tried the usual fixes: stricter prompts, longer instructions, bigger context windows, more examples.

The real problem was simpler.

I was handing the agent noisy context.

Not bad data. Bad context.

Raw structured data is full of repetition, incidental shape, schema noise, nulls, IDs, timestamps, nested boilerplate, and fields that only mean something in aggregate. Humans struggle with that. Agents do too.

If you paste a huge API response, GitHub export, log dump, CSV, or source tree into a model and ask it to "find what matters," you are asking the model to do two jobs at once:

Measure the data.
Reason about the measurements.

Those are different jobs.

Vajra is my attempt to split them apart.

Before the model reasons, Vajra measures. Before the agent plans, Vajra reduces the raw input into evidence.

The Agent Context Problem

The default agent workflow is still too close to copy and paste.

Give the agent a repository. Give it a JSON dump. Give it a pile of logs. Give it a directory of semi-structured files. Then ask it to infer structure, identify anomalies, detect drift, preserve evidence, and decide what to do next.

That sounds powerful, but it is a bad division of labor.

Agents are useful for interpretation, synthesis, planning, and tradeoff analysis. They are not the best tool for counting every path in a nested document, computing entropy across values, comparing two distributions, detecting type instability, or producing a repeatable fingerprint of a shape.

Those jobs should be done by boring software.

Reliable software.

Software that returns the same answer every time.

That is the layer I wanted in front of the agent.

What Vajra Does

Vajra is a Rust CLI and library for analyzing structured data. It started from a simple frustration with large JSON, then grew into a signal layer for things agents routinely ingest:

structured data like JSON, NDJSON, YAML, CSV, and TSV
operational artifacts like Markdown, PDF text, CPU profiles, and strace logs
repositories, GitHub exports, and source code parsed through tree-sitter

The goal is not fuzzy summarization. The goal is stable measurement.

For example:

vajra inspect payload.json
vajra stats events.ndjson
vajra anomalies batch.json
vajra drift baseline.json current.json
vajra essence payload.json --profile ai --format compact-ai --budget 500 --redact
vajra inspect src/main.rs --input-format source --lang rust --semantic-paths
vajra governance commits.json

That gives you paths, types, fingerprints, entropy, cardinality, outliers, rarity, schema drift, source-code structure, contributor concentration, and compact essences that are designed to be read by humans or agents.

The important word is stable.

Same input. Same config. Same version. Same result.

That matters more than it sounds.

If an agent is going to call a tool, update a plan, make a recommendation, and then call the tool again after a change, the tool cannot behave like a mood ring. It has to be an instrument.

Raw Data Is Not Context

Here is the mistake I kept making.

I treated "available to the model" as equivalent to "usable by the model."

They are not the same.

A 5 MB JSON export may fit in a context window. That does not mean it is good context. Most of those tokens may be repeated object structure, duplicated IDs, common timestamps, low-signal values, or fields whose meaning only appears after aggregate analysis.

Vajra turns that raw input into a smaller set of measured claims.

Instead of giving an agent every record and hoping it notices that one path is unstable, give it a measured signal shaped like this:

Path: $.claims[*].status
Dominant type: string
Cardinality: 7
Entropy: 1.82
Rare values: reversed_manual, pending_external_review
Null rate: 0.00

Instead of asking it to eyeball schema drift between two payloads, give it a drift report shaped like this:

Removed path: $.member.coverage.plan_id
Added path: $.member.coverage.policy_ref
Type changed: $.claim.amount string -> number
Distribution drift: $.claim.status high
Severity: critical

Instead of making it inspect a full repository export, give it project-health signals such as:

Bus factor: 2
Contributor concentration: high
One-commit contributor rate: 0.61
Most active author share: 0.44
Velocity trend: declining

These examples are representative shapes, not a promise that every command emits exactly these lines. The point is the workflow: let the tool compute the measurements, then let the agent reason from them.

Now the agent can do the work it is actually useful for.

It can reason about consequences. It can ask whether the drift is intentional. It can propose a migration plan. It can decide what evidence belongs in a report. It can open an issue with the relevant fields instead of a blob of raw data.

Measure, Then Reason

The agentic workflow I want looks like this:

raw data / repo / logs / API export
    -> deterministic analysis
    -> compact evidence bundle
    -> agent reasoning
    -> action plan or code change
    -> re-analysis
    -> verification

That loop is different from the usual paste-and-hope loop.

The agent has fewer reasons to infer measurements from raw records. It gets to reason from measured facts.

Vajra can sit before the model and decide what deserves attention. It compresses without pretending to be creative. It surfaces anomalies without pretending every anomaly is automatically important. It preserves paths and evidence so the agent can point back to the data.

That is the difference between a summary and an instrument.

A summary says, "Something changed."

An instrument says, "This path changed, this value distribution moved, this type is unstable, and here is the evidence."

A Concrete Agent Workflow

Imagine an agent responsible for reviewing a GitHub project before a release.

The naive version gets a large export of issues, pull requests, commits, and releases. It tries to infer project health from raw records. It may notice recent failures, but miss contributor concentration. It may summarize issue titles, but miss stale ownership patterns. It may produce a confident report with weak evidence.

The Vajra version starts by measuring first:

vajra ingest-github owner/repo --output .vajra/repo
vajra governance .vajra/repo/commits.json --format json > .vajra/governance.json
vajra core-team .vajra/repo/commits.json --format json > .vajra/core-team.json
vajra anomalies .vajra/repo/issues.json --format json > .vajra/issue-anomalies.json
vajra essence .vajra/repo/issues.json --profile ai --format compact-ai --redact > .vajra/issues.essence.json

Now the agent does not need to start by reading everything.

It reads the measurements.

Instead of producing a vague release memo, it can flag specific evidence-backed risks:

ownership concentrated in two maintainers
unusual issue clusters before release
repeated labels or states that deserve triage
fields that changed shape across exports
anomalies that need business context before they are treated as problems

If the agent recommends action, the action can be checked.

Did the generated report cite the right paths? Did the relevant anomaly disappear or become explainable? Did a migration remove the type instability? Did the same input produce the same output after a refactor?

That is the kind of feedback loop agents need.

Why Determinism Matters

Agentic systems are already probabilistic at the reasoning layer. The tools around them should not add unnecessary randomness.

If a tool summarizes differently every time, the agent's plan changes for reasons that are hard to debug. If a score cannot be decomposed, the agent cannot explain it. If a drift report is not stable, CI cannot trust it. If a fingerprint changes because object keys were ordered differently, the tool is measuring formatting noise instead of structure.

Vajra is built around boring constraints:

deterministic ordering
stable fingerprints
explicit paths
explainable scores
profile-driven reduction
no silent mutation of source data

It is not glamorous. It is what makes the output usable as agent context.

Agents do not just need more context. They need context they can trust.

What This Does Not Solve

This layer does not make judgment disappear.

Vajra can measure structure and distributions. It does not decide whether an anomaly is business-important. It can flag contributor concentration. It cannot know whether that concentration is a temporary release push, an organizational risk, or just how a small project works.

Compact essences are evidence bundles. They are not replacements for source data when auditability, legal review, or incident response requires the original records.

And if sensitive data is involved, it should be redacted or minimized before it is sent to any model or external service. That is why the CLI supports --redact, but the operational policy still belongs to you.

The point is not to remove human or agent judgment.

The point is to stop wasting judgment on work that deterministic tools can do better.

Where This Goes

Serious agent systems will not be built from prompts alone.

They need toolchains that produce reproducible intermediate artifacts. They need ways to shrink context without losing evidence. They need stable measurements before interesting reasoning.

That is the bet behind Vajra.

Not that agents should know everything.

That agents should be handed better instruments.

Because when you stop feeding agents raw data, you give them a better job:

reasoning from evidence.

The Container Runtime Nobody Told You About (And Four Others)

Don Johnson — Tue, 26 May 2026 20:19:00 +0000

Here's something the container ecosystem doesn't say loudly enough: runc is not the only option, and for a growing number of production workloads, it's the wrong one.

AWS Lambda doesn't run your function in a Docker container. It runs it in a Firecracker microVM. Fly.io's Machines? Firecracker fork. Google's multi-tenant GKE nodes? gVisor. Cloudflare Workers? WASM. These companies didn't reach for exotic runtimes because they were bored — they reached for them because the default isolation model was insufficient for their threat model, their latency requirements, or both.

This article takes one tiny Go HTTP server and runs it through all five of them: runc/distroless, gVisor, Kata + QEMU, Kata + Firecracker, and WASM/WASI. You'll see exactly what changes (almost nothing), what the real numbers look like, and — most importantly — which runtime belongs in which situation.

TL;DR: gVisor, Kata, and Firecracker all run the exact same 3 MB OCI image — only --runtime=X changes. WASM is a different compilation target entirely. Cold-start ranges from ~20 ms (runc) to ~500 ms (Kata/QEMU), with Firecracker splitting the difference at ~125 ms. Request latency overhead at steady state is shockingly small across all of them. The real cost is memory and compatibility, not throughput.

The App

Before the runtimes, the subject. A Go HTTP server with one meaningful endpoint:

RUNTIME_NAME is injected at docker run time. Everything else — Go version, arch, PID, uptime — is live from inside whatever sandbox is holding it. When the runtime changes, the response field tells the story.

Runtime 1: Distroless + runc

What it is

The default Docker runtime (runc) but with a distroless base image. No shell, no package manager, no apt, no curl. Just the Go binary and CA certificates.

The image comes out at 3.0 MB. Alpine would be ~18 MB. Ubuntu ~80 MB.

The honest security story

Distroless does not change your isolation model. The container still shares the host kernel. What it does is remove every tool an attacker would use after a successful exploit — no shell to drop into, no package manager to pull more tools from, no /tmp scripts to run. You're not preventing the breach; you're making the post-breach environment hostile.

Ultimate use cases

Internal microservices in a trusted, single-tenant cluster
GitOps pipelines where you control every image in the registry
Replacing fat Alpine images — the size drop alone is worth it
The security baseline every team should hit before adding runtime overhead

Runtime 2: gVisor (`runsc`)

What it is

gVisor ships a user-space Linux kernel called the Sentry — written in Go — that runs alongside your container. Every syscall your container makes goes to the Sentry. The host kernel never sees your container's syscalls.

# Same 3 MB image. One flag.
docker run --rm --runtime=runsc \
  -p 8080:8080 -e RUNTIME_NAME=gvisor \
  micro-containers

The Sentry re-implements the Linux ABI. In ptrace mode it intercepts via ptrace; the newer Systrap mode (shipped 2023, ~2× faster) uses seccomp to intercept. Either way, a kernel exploit in your container cannot reach the host kernel — there is no direct path.

The honest security story

gVisor's threat model is syscall isolation. A container escape via a kernel CVE (your dirty_pipe, your runc breakout) is stopped at the Sentry. But gVisor is not a VM — the container still shares memory, CPU, and the host's network stack at some layers. It's a strong sandbox, not a hard boundary.

2025 state of the world

GKE Sandbox is gVisor, enabled with a single node pool annotation
Systrap mode is now the default — nearly removes the performance cliff that made early gVisor a tough sell
GPU support is production-ready for A100/H100 via vGPU passthrough — relevant if you're sandboxing AI inference workloads

Ultimate use cases

CI/CD runners — the #1 production use case. GitHub Actions self-hosted, GitLab runners, Buildkite agents that execute arbitrary user pipelines. You don't control the code; gVisor limits the blast radius.
ML inference APIs where users submit model weights or custom code — you can't trust what's in those pickles
SaaS plugin execution — any platform that lets users run custom logic (Zapier-style automations, Retool actions, webhook processors)
Cloud IDE backends — Codespace-style environments where each user gets a container that feels like root

Runtime 3: Kata Containers (QEMU VMM)

What it is

Kata Containers boots a lightweight QEMU MicroVM per container. Your app runs inside a VM with its own kernel. containerd sees an OCI runtime; your process sees a dedicated Linux instance.

docker run --rm --runtime=kata-runtime \
  -p 8080:8080 -e RUNTIME_NAME=kata-qemu \
  micro-containers

The host sees a qemu-system-x86_64 process — nothing inside leaks out. The container image is mounted via virtiofs. The kernel boundary is real.

The honest security story

Kata/QEMU is the only option here that provides a true hardware-enforced boundary between container and host. gVisor is software isolation. Kata is a VM. If your threat model requires that a kernel exploit inside the container cannot affect the host, Kata is the answer.

2025 state of the world

Kata 3.x ships with confidential container support: Intel TDX and AMD SEV-SNP give you hardware-attested memory encryption. The host operator can't inspect container memory — relevant for regulated data.
Cloud Hypervisor is now a supported VMM alternative to QEMU, lighter and faster to boot
Confidential Containers (CoCo) as a CNCF project wraps Kata + hardware attestation into a first-class primitive — watch this space

Ultimate use cases

PCI-DSS, HIPAA, FedRAMP — when the compliance checklist literally says "VM-level isolation," Kata is the only container runtime that checks that box without running actual VMs
Financial services — trade processing, settlement systems, anything touching payment card data
Healthcare data pipelines — PHI processing where you need a kernel boundary in the audit trail
Multi-tenant databases — giving each tenant a database that physically cannot escape its VM
Government/defense workloads — environments where the security control plane doesn't trust the container runtime

Runtime 4: Kata + Firecracker VMM

What it is

Firecracker was built by AWS in 2018 specifically for Lambda and Fargate. It replaces QEMU as Kata's VMM. The device model is stripped to the minimum a serverless function needs: one network interface, one block device, one serial port. No BIOS. No PCI bus. No USB enumeration. No legacy device emulation of any kind.

# Kata reads configuration-fc.toml and invokes Firecracker instead of QEMU
docker run --rm --runtime=kata-fc \
  -p 8080:8080 -e RUNTIME_NAME=kata-firecracker \
  micro-containers

Cold start drops from ~500 ms (QEMU) to ~125 ms. Memory overhead drops by nearly half.

The honest security story

Same VM isolation guarantee as Kata/QEMU — a dedicated kernel per container. The tradeoff for the speed gain is device compatibility: no GPU passthrough, no USB, fewer PCIe options. For stateless functions, you don't need any of that.

2025 state of the world

Firecracker 1.7+ — production-stable, used in billions of Lambda invocations per day. AWS open-sourced it and it ships new major versions regularly.
Fly.io Machines use a Firecracker fork as the core primitive — every fly machine run is a microVM
AWS Serverless Aurora uses Firecracker to isolate query execution environments
Confidential Firecracker is in active development — combining Firecracker's boot speed with AMD SEV memory encryption

Ultimate use cases

Serverless function platforms — this is what Firecracker was made for. If you're building the next Lambda, Railway, or Render, Firecracker is the substrate.
AI/ML inference bursts — LLM inference is bursty; Firecracker's 125 ms cold start makes scale-to-zero viable. A GPU instance spun up with Firecracker can take traffic in under a second.
Short-lived test runners — each test run gets a clean VM, boots in 125 ms, exits, gets GC'd. No shared state, no contamination between runs.
Multi-tenant job queues — background jobs that process user-submitted data. Firecracker gives you VM isolation at a price point runc used to own.
Preview environments — spin up a full-stack environment for each PR, destroy it on merge. The economics work at ~125 ms boot + minimal memory overhead.

Runtime 5: WASM / WASI preview1

What it is

The binary is compiled to WebAssembly with Go's WASI target — an entirely different binary, an entirely different image:

The resulting image: 3.1 MB scratch base + one .wasm binary. The sandbox is enforced at the language-runtime level — no syscalls, capabilities explicitly granted by the host.

The honest HTTP story

net/http doesn't work in WASI preview1. The spec has no socket API. This demo outputs JSON to stdout. That's not a cop-out — it's the current state of the standard. The wasi-http proposal shipped as part of WASI 0.2, which is ratified. Fermyon Spin 2.x implements it today. Go's WASI 0.2 support is in progress.

2025 state of the world

WASI 0.2 (Component Model) is ratified and shipping in wasmtime, WasmEdge, and Fastly Compute. wasi-http is a real, stable interface.
Docker+Wasm is GA in Docker Desktop 4.27+ — run a WASM container with --platform=wasi/wasm and a containerd shim
Fermyon Spin 2.x compiles Go to WASM with a full HTTP server abstraction — the framework paper over the WASI/HTTP gap today
WasmPlugin in Kubernetes — Envoy and Istio support WASM plugins for custom policy, auth, and observability logic
Extism — a cross-language WASM plugin framework that lets you embed sandboxed user code in any Go/Rust/Python host

Ultimate use cases

Edge functions — Cloudflare Workers, Fastly Compute, Deno Deploy, and Vercel Edge Functions are all WASM at the bottom. The same binary runs in London, Singapore, and São Paulo with no containers to spin up.
Cross-platform CLI tools — compile once, run on Linux/macOS/Windows/browser with no CGO, no cross-compilation matrix
Sandboxed plugin systems — give users scriptable extensions with a real capability boundary. Zellij (terminal multiplexer) uses WASM plugins; VS Code extensions are moving this direction.
Business logic in the browser + server — tax calculation, pricing rules, validation logic that needs to run identically client-side and server-side
AI prompt/response filters — fast, sandboxed, hot-reloadable logic at the edge before a request hits your inference endpoint

The Numbers

All OCI runtimes run the same 3 MB distroless image. The distroless/runc row is measured hardware; microVM rows are reference numbers from project documentation — run make bench-md for your own numbers.

Runtime	Image	Cold Start	p50	p95	Memory
distroless / runc	3.0 MB	~20 ms¹	0.28 ms	0.41 ms	6.9 MB
gVisor (runsc)	3.0 MB	~50 ms	~0.5 ms	~1.0 ms	~18 MB
Kata / QEMU	3.0 MB	~500 ms	~0.8 ms	~1.5 ms	~52 MB
Kata / Firecracker	3.0 MB	~125 ms	~0.7 ms	~1.3 ms	~28 MB
WASM (wasmtime)	3.1 MB	N/A²	—	—	—

¹ First run ~174 ms (overlay FS init); subsequent ~20 ms on warm cache.
² WASM has no HTTP server in wasip1; exec time ~8 ms for the stdout variant.

Three things the numbers tell you that prose doesn't:

Image size is not the story. All five runtimes land at 3–3.1 MB. Switching from runc to Firecracker doesn't touch your image pipeline.

Latency overhead at steady state is negligible. Even inside a Kata VM, p50 latency is under 1 ms. The isolation boundary costs you cold-start and memory, not throughput. If you're worried about runtime overhead on a running service, stop — that's not where the overhead lives.

Firecracker hits the practical sweet spot. 125 ms is the number AWS decided was fast enough for Lambda. 500 ms (QEMU) is where users start feeling it. Firecracker lands right where microVM isolation becomes viable for interactive-latency workloads.

The Decision Framework

Stop asking "which is more secure?" Start asking "what's my threat model?"

Your tenants are you. You control every image, every workload, every user. → runc + distroless. Fast, simple, no overhead.

Your tenants are your users, but you control the runtime environment. CI runners, SaaS execution engines. → gVisor. Drop-in, no KVM, syscall isolation stops the most common container escapes.

You have compliance paperwork that says "VM-level isolation." → Kata + QEMU. The only option that satisfies an auditor asking for a kernel boundary.

You're building a platform. Functions, jobs, preview environments, AI inference. Cold-start matters. → Kata + Firecracker. This is the production-proven answer for platforms.

Your code runs everywhere, or users supply the code. Edge compute, plugins, sandboxed scripts. → WASM/WASI. The sandbox is portable; the isolation model is capability-based, not kernel-based.

Run It Yourself

git clone https://github.com/copyleftdev/micro-containers
cd micro-containers

make check        # see what's installed
make bench-fast   # quick smoke-test with 20 samples
make bench-md     # full benchmark → Markdown table

Each runtime has an install.sh in runtimes/<name>/. The benchmark driver skips unavailable runtimes and tells you exactly what to install.

Source, Dockerfiles, benchmark driver, and install scripts: copyleftdev/micro-containers