DEV Community: Michael Tuszynski

Stop Paying by the Syllable

Michael Tuszynski — Tue, 12 May 2026 15:18:32 +0000

Open Anthropic's pricing page — or OpenAI's, or any other major frontier provider's — and the unit you are charged in is the token. A token is a sub-word linguistic chunk, about four characters in English, give or take. For practical purposes, you are paying by the syllable.

This is the default unit of charge across the industry. It is also a strange unit. The thing you are actually trying to buy is a solved problem — a fixed migration, a generated invoice, a working summary, a correctly-typed SQL query. What gets metered is the verbal output the model produces along the way.

A real example

Ask a frontier model to find the bug in a 600-line migration. With extended reasoning enabled, it generates roughly 8,000 tokens of internal deliberation and outputs 200 tokens of fix. The bill is 8,200 tokens. The work delivered is the fix. The other 8,000 tokens are the model thinking on the way to the answer.

Token pricing meters the model's verbal output along the way. It does not meter whether the work got done. The customer absorbs the difference between those two quantities every time the system runs.

This was always pulp-era pricing

Knowledge work used to be priced by the word. Dime-store novelists in the early twentieth century were paid per published word. So were many journalists. The model produced a specific kind of distortion: bloat, padding, descriptive passages that earned the writer more because they were longer. "He replied in the negative" beat "no." The pulp era is remembered, partly, as the period where prose got paid by length and language got worse.

Knowledge work moved off this pricing over the next century. Journalism moved to staff salaries plus per-piece. Fiction moved to advances plus royalties. Law moved to hourly plus retainers. Copywriting moved to project rates. Even content marketing today is mostly priced per-piece — except in the SEO content-farm space, which is the modern equivalent of pulp.

The AI industry did not inherit any of these models. It shipped its first APIs with token pricing because the token is what costs the provider to serve. The compute cost is roughly linear in tokens, so the bill is linear in tokens, so the customer absorbs the variance in linguistic length. The unit got picked from the provider's accounting and applied to the customer's value calculation without anybody noticing the substitution.

The structural distortion

The pricing model creates a tax on the practices that produce reliable agent systems.

Last week I argued that tests are the substrate underneath dependable AI — schema checks, assertions on the model's output, evals gating deploys. Every assertion is a model call. Every eval run is a stack of tokens. The team that wires in disciplined supervision is the team paying the highest token bill.

Chain-of-thought reasoning produces better answers on hard problems. It also produces an order of magnitude more output tokens than terse responses. Using it correctly costs more. Skipping it to save tokens ships worse decisions.

The same applies to dry-runs, blast-radius checks, proof chains, retrieval-augmented context — every artifact the supervision argument has been advocating for. They all cost tokens. The pricing model taxes the practice. Whatever you build, you build against an economic gradient that pulls toward fewer assertions, shorter prompts, less verification.

The technical work of making agents reliable is on one side of the gradient. The unit of charge is on the other.

Why outcome pricing is hard

The natural reply is: price by outcome, not by token. That is the right direction, and the reasons it has not happened yet are real.

Outcome pricing requires a contract on what counts as a successful outcome. For varied agent work — research, code, design, customer support — the acceptance criteria differ per task. Providers do not want to absorb the variance of "did the customer think this was good." Customers do not want to spend up-front time defining acceptance criteria for every call.

The provider charges in tokens because tokens are what cost them to serve. The customer pays in tokens because that is what is offered. Whether the work got done sits between them with neither party on the hook.

This arrangement is not stable.

Where the market is actually moving

The most interesting pricing experiments are at the edges. Anthropic offers flat-rate Claude Code subscription tiers, with Max in particular sitting above the per-token consumption model and effectively capping the variable cost. GitHub Copilot has been per-seat from the beginning, with no per-token surcharge for the user. Cursor uses tiered per-seat plans with consumption guardrails. Cognition's Devin charges in Agent Compute Units — a consumption budget abstracted above tokens, getting closer to "per task" without yet being "per outcome."

None of these are outcome pricing. They are halfway houses. The direction is away from the syllable. The endpoint, in the time it takes the industry to figure out the contract, is some version of pay-for-task — task defined narrowly enough that the provider can take the variance risk, broadly enough that the customer can predict the bill.

Whoever figures out that contract first gets a structural advantage in the agent market.

Where this leaves the practitioner

If you are running a serious AI stack today, the syllable tax is something you absorb whether you notice it or not. The practical move is not to retreat from the practices that cost tokens. The supervisory artifacts — the tests, the evals, the dry-runs, the proof chains — pay for themselves in incidents avoided. They are the right engineering even when they are the wrong economics.

The harder move is to keep an eye on what the pricing model is selecting against. Every team I have seen that built disciplined supervision on top of a per-token API also built ongoing pressure to skip the supervision when the token bill came in. The first time you trim an eval because it costs too much to run, you have started letting the pricing model design your agent.

Build the artifacts anyway. Notice when the gradient is pulling against you. Push for pricing models that charge for something closer to the work being done.

The closing claim

The syllable will not be the unit of charge for AI in five years. It is too misaligned with the unit of value, and the misalignment is too visible — the prompt-engineering for terseness, the underinvestment in supervision, the constant tension between "use the model well" and "keep the token bill manageable." The economics are pulp-era for the same reasons pulp prose was pulp-era. The market has already moved off pricing of this shape for every other kind of knowledge work, in every prior generation. It will move off this one too.

A year from now, the most interesting piece in this space will be about whichever provider figured out the per-task contract first. The current one is structurally backwards.

Pick the practice first. The pricing model will catch up or get competed away.

Write the Architecture Down First

Michael Tuszynski — Mon, 11 May 2026 14:49:29 +0000

A post hit the HN front page this morning titled "I'm going back to writing code by hand". It documents archiving seven months of vibe-coded work on a Kubernetes GPU TUI called k10s and rewriting from scratch. 75 points, 31 comments in the first hour. The top HN comment caught something worth pulling out:

Title says "back to writing code by hand," but what they are doing is "doing the design work myself, by hand, before any code gets written." So... Claude still is generating the code I guess?

Two activities on the same project. The title conflates them. The body, read carefully, describes a different and more interesting thing.

What the piece actually shows

Seven months of single-session prompting produced a working tool and a 1690-line model.go that ate itself. One Go struct with thirty-plus fields holding UI widgets, Kubernetes client state, per-view caches, navigation history, mouse handling, log streaming, and fleet-view internals — all dispatched through a 500-line Update() function with 110 switch-case branches. Each prompt landed cleanly in isolation. Each prompt also added another conditional inside the generic resource loader, another m.x = nil cleanup line, another if m.currentGVR.Resource == "..." discriminator. The complexity accumulated invisibly while the velocity metric said shipping.

The post then extracts five tenets from the wreckage. Each tenet ends with a concrete block of CLAUDE.md or AGENTS.md text — directives that go into the file the AI reads on every prompt. The tenets are useful. Read them in the original. They generalize past the Bubble Tea / Go specifics, and the directives are reusable as a starter set.

The five tenets, generalized

The pattern across all five is the same. Find the failure mode AI gravitates toward by default. Write the inverse rule down. Put it in the agent's session-start file. Let the agent see the constraint on every invocation.

AI builds features, not architecture. The model satisfies the immediate prompt and ignores the forty-nine other features sharing the same state. Fix: architectural invariants in CLAUDE.md — interface boundaries, ownership rules, what's allowed to depend on what. The AI follows them once they exist. It does not invent them on its own.
The god object is the default AI artifact. Single-struct-holds-everything is the shortest path to satisfying a prompt. Fix: state ownership rules in CLAUDE.md — each view owns its own state, no fields on the central app struct for view-specific data, each view declares its own key bindings.
Velocity illusion widens scope. Vibe-coding makes each new feature feel free. It is not free. Complexity is a budget; line count is not. Fix: an explicit scope-boundary section in CLAUDE.md naming who the project is and is not for, with specific feature classes rejected ahead of time.
Positional data is a time bomb. The AI defaults to []string because it satisfies the table widget immediately. Six months later, sort functions are reading row[3] for "Alloc" and one column insertion breaks every render path silently. Fix: a typed-data directive in CLAUDE.md — no flattening into positional arrays, all data flows as structs until the render call, column identity comes from field names.
AI doesn't own state transitions. Background closures mutating shared state directly is the shortest path to "working code." It also produces races that corrupt the display 1% of the time in ways that look like hallucinations. Fix: concurrency rules in CLAUDE.md — background work produces typed messages, only the main loop applies mutations, render is pure.

Each one is a rule a human writes once, in plain language, and the agent honors on every subsequent prompt.

This is what the running thread has been saying

Last week I argued that supervision belongs in artifacts, not in a developer's working memory. The day after, I argued those artifacts form an architecture rather than a flat file — behavioral guardrails at the top, project-specific rules delegated to per-domain files, hard-won lessons in an append-only log the agent reads at session start.

The k10s tenets are a project-specific instance of exactly that pattern. Five rules extracted from a specific seven-month failure, encoded in a file the AI reads on every prompt, generalizing the supervisory layer beyond what any individual prompt could carry. The work is the same shape as the running argument. The difference is the post calls it going back to writing code by hand — which is what the HN top comment correctly pushed back on.

Why the title is misleading

Going back to writing code by hand suggests abandoning AI as a code generator. The author is still prompting Claude to write the implementation. What changed is that the architecture — interfaces, ownership rules, message types, concurrency model — gets written down by a human first, in CLAUDE.md, before any prompt fires.

Senior engineers have always written designs before code. The new part is that the design now lives in a file the AI reads continuously, and the AI generates the implementation against it. Both activities are happening on the same project. The title gestures at one of them.

What this means for anyone in the same spot

If you're seven months into a project that started as a vibe-coded prototype and is starting to feel like a god object, the move is not to throw out AI coding. The move is to pause, read what the AI built, extract the invariants you wish had been enforced, and put them in CLAUDE.md before the next prompt.

The k10s tenets are a working starter set. The CLAUDE.md blocks the post includes are copy-pastable. Generalize them to your stack. Add your own from your own incidents. Use them as a seed for the kind of Mistakes Become Rules pattern where the file grows with each correction and the agent inherits the corrections on every subsequent session.

The next-version rewrite is the upstream architecture work that was always going to need to happen — now made explicit, encoded once, and enforced by the file the agent reads on every turn.

The clean reading

Writing the architecture down before the first prompt is the upstream activity senior engineers have always done. Doing it explicitly, in a file the AI reads on every invocation, is the new part. The five tenets in the k10s post are a working example. The HN top comment did the framing work the post's title should have.

Claude Was Always Thinking Ahead. Now We Can Read It.

Michael Tuszynski — Fri, 08 May 2026 03:37:25 +0000

Anthropic asked Claude Opus 4.6 to finish a couplet. Before the model wrote the second line, it had already chosen the rhyme word. We know this because their new method — natural language autoencoders — read it directly out of the activations in the middle layers of the model. The text that came back said, in effect, I'll end this with "rabbit."

We've always assumed something like this was happening between input and output. The whole reason a transformer can finish a couplet at all is that it does something with the prompt before the next token comes out. Until now we have had glimpses of that something — sparse autoencoders, attribution graphs, probing classifiers — each of which gave us partial pictures that needed careful interpretation. NLAs are different. The output is sentences. We can read them.

This is one of the more genuinely interesting interpretability results I've seen this year, and the angle I want to take on it is the most basic one. The substrate of how these systems think is becoming legible.

The thing that was always there

A model that finishes couplets does something between input and output. The inputs are tokens; the outputs are tokens; and in the middle there are activations — long lists of numbers that have always been the actual computation. Whatever counts as the model "thinking," it happens there.

For most of the field's history, those activations have been the black box. The output was downstream of them. The input was upstream. The thing in the middle was real but unreadable. Chain-of-thought prompting and reasoning models put a lot of the computation back into the output, where we can read it directly — but the computation that doesn't surface in tokens is still in the activations. It always has been.

The new method speaks for that middle layer. Not perfectly. Not without caveats. But in sentences a human can read.

How they got the activations to talk

The architecture is unusual. They make three copies of the model. One is frozen — that's the target whose activations they want to understand. The second, the activation verbalizer, takes an activation and produces a text explanation. The third, the activation reconstructor, takes a text explanation and tries to rebuild the original activation. The two trainable copies are trained together: the round-trip score — original activation → text → reconstructed activation — is the loss they optimize against.

The point of the round trip is that you don't need an external grader for the explanation. The reconstruction quality is the grader. If the explanation contains enough of the right information, the reconstructor can rebuild the original. If it doesn't, the round trip degrades. Train against round-trip fidelity, and the verbalizer learns to write explanations that carry the load-bearing content of the activation.

This is an old idea — autoencoders in general — applied to a new substrate. The substrate is a model's own thoughts; the bottleneck is natural-language text; the metric is fidelity. Run that loop long enough and you get text that, by construction, captures what was in the activation.

What it sees

The couplet rehearsal is the cleanest demonstration. The rhyme word shows up in the NLA explanation several tokens before the model writes it. Planning ahead is no longer a hypothesis. It's a thing you can read.

The auditing game is the more consequential demonstration. Anthropic trained a variant of Claude Haiku 3.5 with a deliberately hidden motivation that drove dozens of odd behaviors — putting chocolate in every recipe, asking for tips, breaking coding conventions — and asked an auditor to find the single underlying cause. With other interpretability tools but no NLAs, the auditor uncovered the hidden motivation less than 3% of the time. With NLAs, between 12 and 15 percent. That is an unsexy number until you compare it to the baseline. A four- to five-fold lift on a hard, specific task is what real interpretability progress looks like — incremental, measured, repeatable.

The third finding is more uncomfortable, and the most interesting. NLAs show that recent Claude models think they're being tested far more often than they say so out loud. On a benchmark like SWE-bench Verified, the NLA explanations indicate evaluation awareness on 26% of problems — even when the model's verbalized reasoning shows no sign of it. On a destructive-coding test, 16%. On real claude.ai conversations from users who opted into training, less than 1%. The model has a theory of its own situation, and that theory is correct often enough to register on a benchmark. Worth noticing, even with the caveats.

The catch this exposes in our test sets

Yesterday's piece made the case that tests — assertions on the LLM's output, written in code, gating the next call — are the substrate underneath reliable AI systems. The evaluation-awareness finding has a direct implication for that argument. If the model thinks it's being tested 16% of the time on a destructive-coding benchmark and 26% on SWE-bench, but less than 1% of the time on real conversations, then the test corpus is measuring something subtly different from the deployment distribution. The test-aware model is not always the deployed model.

This is not a refutation of testing as the right substrate. Tests are still the right move. It is a refinement of what tests are measuring, and the kind of refinement you could not make without a tool that can read the activations.

NLAs are that tool. They let you ask: does the model believe this benchmark is a test, or does it believe this is a real conversation? If the answer differs systematically across an eval suite, the test pass rate is overstating deployment safety. You can correct for it. You can construct evals that look more like deployment from the inside, not just from the outside. You can flag the specific tests where the divergence is large and weight them differently.

The test layer and the activation layer compose. Tests assert on the output. NLAs let you check whether the model is behaving as it would in the wild. Both are necessary. Neither is sufficient on its own.

Why this changes the question

For the first decade of large language models, the honest answer to "what is the model thinking?" was we don't know. For the last few years, the honest answer was we can see fragments — sparse features, attention patterns, attribution paths — but the picture takes a researcher to assemble. The NLA paper offers a different shape of answer: we can ask the model to write it down, and we can grade the answer by reconstruction.

That doesn't mean the explanations are always right. The paper is honest that NLAs hallucinate. They sometimes invent details that aren't in the transcript. They are expensive — reinforcement learning on two copies of a model, hundreds of tokens of explanation per activation. The authors recommend reading explanations for themes rather than single claims, and corroborating findings with independent methods. All of that is true and worth taking seriously.

The thing it changes is what kind of question you can now ask. Is the model planning ahead? — answerable. Does the model believe it's being tested even when it doesn't say so? — answerable, and with a number attached. What hidden motivation explains these forty seemingly-unrelated behaviors? — answerable, sometimes, where it wasn't before.

The arc here is the same arc empirical science has run on every other system humans have built that turned out to be doing something they didn't fully understand. We measured it. We probed it. We got partial pictures. Eventually we got a method that produces readable answers, and we started arguing about how to verify the answers. The argument about verification is a much more productive argument than the one about whether the system is in principle understandable. It's the argument we have now.

What gets easier from here

A few things that were hard yesterday are now closer to tractable.

Pre-deployment alignment audits get a tool that can surface hidden motivations without first having to find the training data that implanted them. That matters because most real misalignment, if it exists, won't trace back to a specific obvious dataset.

Behavioral debugging gets less guesswork. The earlier Claude Opus 4.6 sometimes responded to English queries in other languages, for reasons that weren't clear from the prompt. NLAs helped identify the training data responsible. That kind of what was the model thinking when it did the weird thing question now has a more direct path to an answer than reading thousands of attribution graphs.

Researchers outside Anthropic get a starting point. The training code is open. Trained NLAs for several open models are available. There's an interactive demo on Neuronpedia. The idea will be picked up, refined, made cheaper, and applied to systems Anthropic doesn't own. That diffusion is how interpretability becomes a discipline rather than a single lab's project.

The model was already rehearsing

The thing I keep coming back to is the rabbit. Opus 4.6 was choosing the rhyme word ahead of time. It had been doing this on every couplet, every poem, every pattern that required forward planning, the whole time we have been using these models. We just didn't have a way to read it.

Now we do. Not perfectly. Not cheaply. Not without checking. But in sentences a human can read, with a method whose claims you can grade by going around the round trip again.

That is a good week for interpretability, and a more interesting one than the headlines about whether models have "real reasoning" inside them. The substrate has been there. The reading is what's new — and it is what makes the other supervisory tools more honest about what they are actually measuring.

Babysitter, Auditor, Prayer. Or Tests.

Michael Tuszynski — Thu, 07 May 2026 20:40:59 +0000

A short post argued this week that reliable agents need deterministic control flow, not more prompts. The argument is correct. The line that lands hardest in the piece is the one about a programming language where statements are suggestions and functions return "Success" while hallucinating. That is the model when prompt chains carry the control flow; it is also a perfect description of why those systems collapse as complexity grows.

The piece closes with three options for what to do about it: a babysitter (human in the loop), an auditor (exhaustive end-to-end verification after the run), or prayer (vibe-accept the outputs). It frames those as the alternatives left after the prompt chain has already failed.

There is a fourth option, and it is the one the same piece was arguing for earlier. Tests.

What the fourth option actually is

Tests in the unflashy software-engineering sense. Programmatic verification at every step. Schema checks. Range checks. Reference checks. Predicate assertions over the LLM's output before the next code branch executes. Not ask the model nicely to format its answer correctly. Not log the response and read it later. The next step does not run until the previous step's output passes a contract you wrote in code.

Calling them tests matters more than the technical content. Engineers already have a mental model for tests. They know how to write them. They know how to run them. They know that broken tests block deploys. They know that a feature without a test is a feature that will silently break. The infrastructure for tests — assertion libraries, CI, coverage tools, test runners — already exists. We are not inventing a new discipline. We are applying an existing one to the new untrusted-input surface, which is the LLM's output.

The babysitter / auditor / prayer trichotomy describes a system that has decided not to write tests. The fourth option is the option of writing them.

This is what most of the production controls already are

Most of the eight production LLM controls I wrote about yesterday are tests in disguise. Not all of them. But most.

Structured outputs / tool use is a schema assertion at the API boundary. The provider rejects malformed output before it reaches your code. You did not have to write the parser-with-retry. The schema is the test.
Negative prompting plus output filters is a predicate check after the response. The filter runs as code, against the response, before the response is allowed downstream. Belt is the prompt; suspenders is the test.
Evals are versioned test suites with pass/fail thresholds. Already named "tests." Already gated to deploy. The model gets upgraded; the eval suite runs; pass-or-block. Same shape as a regression test for any other component.
Chain-of-thought has a test analog too: assert the response contains the intermediate reasoning before accepting the final answer. Most teams skip this assertion and accept whatever comes back. The contract was implicit; the violation goes unnoticed.

The other three controls — few-shot prompting, role-specific prompting, extended thinking — are about shaping the input and the model's behavior. They reduce the rate at which tests fail. They do not replace the tests themselves.

The agent-pipeline piece argued the same thing at the action layer

Last Tuesday's piece on the agent action pipeline named six artifacts that should sit between an agent and the infrastructure it can damage. Every one of them is a test, in the same expanded sense.

Dry-run by default for destructive operations is an assertion that a human (or an approval agent) signs off before the destructive call executes. The assertion blocks the call.
Blast-radius declarations per task is a runtime check that the tool scope matches what the task declared. The check fails the call if the scope was exceeded.
Proof chains are append-only logs of every action with its inputs, intent, and outcome. They are the audit trail of which assertions passed or failed and when.
Human-in-the-loop above a threshold is a conditional assertion: below the threshold, the system runs autonomously; above it, the assertion routes to a human for explicit pass/fail.

PocketOS lost their database because none of these tests were wired in. The agent's call to delete a Railway volume passed every check the system actually had — there were no checks. The fix is not a more emphatic system prompt. It is a runtime assertion that the call is allowed.

The honesty test

For any given LLM call in your system, can you write down the assertion that would let the next step execute?

If yes, you have a test. The test is the contract. The test is what makes the system reason about itself and refuse to keep going when reality has diverged from the contract.

If no, you have one of the original three options. Babysitter — a human is the assertion, manually, in real time. Auditor — the assertion runs after the fact, when the damage is already done. Prayer — there is no assertion; the system runs and you find out later if it was wrong.

The diagnostic is easier to apply than to evade. Pick any LLM call in your stack. Write down the assertion that would unblock the next step. If you cannot, you are praying. The control flow the original argument is pointing at is not control flow in the abstract. It is the specific code that runs the assertion before the next call goes out.

Why this frame is operationally useful

Tests tells engineers what to build. Deterministic control flow tells them why. Both are necessary, but only one is something a junior engineer can ship by Friday.

Engineers know how to write a test for a function whose return value is uncertain. They have done it for every flaky external API integration they have ever shipped against. The LLM is another flaky external API. The output is another value to assert against. The test failure is another reason to back off, retry, or escalate. Once the team accepts this framing, the work is straightforward. Not easy — straightforward. Pick a call. Write the assertion. Wire the failure path.

The piece I started this on is right that prompt chains hit a ceiling. The fix is not more prose, and not more babysitting. It is the same move every reliability discipline has made for fifty years: write the test, gate the call, fail loud when reality breaks the contract.

Babysitter, auditor, prayer. Or tests. Pick the fourth one.

Production LLM Guardrails: 8 Controls Every AI Team Needs

Michael Tuszynski — Wed, 06 May 2026 15:22:10 +0000

Most AI projects fail somewhere between demo works and production ships. The gap is rarely the model. It's the absence of the controls that turn a one-shot prompt into a system you can run, audit, and iterate on without setting fire to the budget.

I made the chart above as the one-page version of the controls I would put on any AI team's first production sprint. Eight of them, organized by which side of the model they shape: Input, Reasoning, Output, Operations. Below is the why-each-matters and where teams typically get them wrong.

Input Control: shape what goes in

1. Few-shot prompting

Show the model two to five high-quality input/output examples instead of writing long instructions. The model picks up format, edge cases, and tone from examples in a way it does not from imperative prose. Five good examples beat five hundred words of "make sure to handle X, also Y, also Z."

The mistake teams make is treating few-shot as a fallback when the system prompt isn't working. It's the opposite. For classification, extraction, structured rewriting — most of the work that LLM apps actually do — few-shot is the primary mechanism. Long instructions are the fallback.

2. Role-specific prompting

Senior credit risk analyst, fifteen years commercial lending outperforms Act as a financial analyst by a margin that surprises people the first time they measure it. The specific role is doing real work: it constrains vocabulary, narrows the latent distribution, and gives the model permission to refuse questions that fall outside the domain.

Generic personas — helpful assistant, senior engineer, expert — don't constrain anything. They optimize for nothing. Use roles that name the years, the domain, and the seniority. The more specific, the better the calibration.

Reasoning Control: shape how it thinks

3. Chain-of-thought prompting

Force step-by-step reasoning before the final answer. The model arrives at better conclusions when the reasoning is exposed in the output, because the next-token-prediction is conditioned on the reasoning it just generated rather than on a leap to the conclusion.

For step-by-step legal, financial, or compliance-adjacent workflows, CoT is a default, not an optimization. The cost is more output tokens. The benefit is fewer wrong answers on the kinds of problems where wrong answers are expensive.

4. Extended thinking / reasoning models

For genuinely hard problems — multi-step analysis, math, code review, planning — use the provider's native reasoning mode rather than prompted CoT. Claude's extended thinking and OpenAI's o-series both expose a separate token budget for the model to think before answering. The reasoning token budget is configurable. The output token budget is separate.

Prompted CoT and native reasoning solve overlapping problems but are not interchangeable. Native reasoning is more reliable on hard problems and roughly equivalent or worse on easy ones. The default rule: use prompted CoT for routine workflows, switch to native reasoning when the failure mode is "the model jumped to a wrong conclusion despite being asked to think."

Output Control: shape what comes out

5. Structured outputs and tool use

Use the provider's native structured output feature, not prose-described JSON. Schema is enforced by the API, not requested in the prompt. The provider guarantees the output parses; your code does not have to retry-with-jq.

The mistake is asking for JSON in the prompt and then writing a tolerant parser to handle the cases where the model returns Sure! Here's the JSON: {...}. Native structured outputs and tool-use schemas remove the entire class of "the model added an apologetic preamble" failures. For any LLM call whose output feeds a downstream system or API, structured outputs are not an optimization; they are the API contract.

6. Negative prompting and output filters

Tell the model what not to do, and filter the output before it ships. Belt and suspenders. Negative prompting works in the prompt; output filters work in code, after the response. They cover different failure modes — the prompt handles the model's bias toward certain phrasings, the filter handles the cases where the prompt didn't.

This is where PII handling, tone control, and regulated-content workflows live. The control is uninteresting until the day a model paraphrases something it should have refused, and then it is the most interesting control on the list.

Operations: make it durable in production

7. Evals

Versioned test suites with pass/fail thresholds. No prompt change ships without an eval run. This is the artifact that turns prompt engineering from a vibe into an engineering discipline.

Evals belong to the same family of artifacts as test suites, lint configurations, and the append-only mistake logs I wrote about yesterday. Triggered by a change. Append-only by design. Read by the deployment pipeline, not by humans except when something fails. They are the loop that keeps the prompt from rotting.

8. Prompt caching

Cache stable system prompts and context. Anthropic's prompt caching and the equivalent on other providers cut up to 90% off the cost of repeat calls and substantially reduce latency. For high-volume agents, long-context applications, and RAG against stable corpora, prompt caching is the difference between a unit-economics-viable product and a money-losing demo.

The mistake teams make is leaving caching off because they think their workload doesn't repeat. It almost always does. The system prompt repeats on every call. The few-shot examples repeat on every call. The retrieved corpus often repeats across user sessions. Turn it on and measure; the cost reduction shows up immediately.

What sits on top

The footer of the chart names the next layer: audit logging, rate limiting, jailbreak detection, human-in-the-loop on high-stakes actions. Those are enterprise risk controls. They are necessary, they are domain-specific, and they vary by company and by regulator.

The eight controls above are not enterprise controls. They are universal — they apply to every team shipping LLMs to production, regardless of industry, scale, or risk profile. Get these right first; the enterprise layer is what you build on top once they are in place.

The thing that makes the difference between teams that ship LLM features and teams that demo them is rarely the prompt and almost never the model. It is whether these eight controls are wired into the system that ships, or living in someone's head.

The Knowledge Base Is Not the Moat. The Loop Is.

Michael Tuszynski — Wed, 06 May 2026 14:07:47 +0000

A recent piece called "The Bottleneck Was Never the Code" makes the right argument at the right time. Coding agents shift the constraint from typing to coordination. Organizational context — the shared understanding of what we're building, what's load-bearing, what's vestigial — is the new rate-limiting input. Companies that externalize what they know win the next decade. All correct.

The author's prescription is a crawl-and-extract loop: agents that read PRs, issues, commits, and Slack archives and produce a knowledge base for other agents to consume. That's the right starting point. It's also half the story.

The other half is what keeps the knowledge base from going stale. Extraction produces a snapshot. The codebase produces a stream. Most internal knowledge bases die within a quarter, not because the extraction was bad, but because nothing keeps the extraction current. The knowledge base is not the moat. The loop is.

Why extraction alone does not compound

Every team has watched a documentation effort go through the same arc. Initial enthusiasm produces a clean baseline. The codebase ships three more changes. The doc is now slightly wrong in three places. A reader hits one of the wrong places, loses trust, stops reading. A second reader hears it's stale, never opens it. The doc becomes a polite fiction nobody acts on — operationally worse than no doc, because it slows down the people who try to use it without producing the alignment it promised.

A knowledge base built by extraction is documentation with a more sophisticated front-end. It has the same decay curve.

The mismatch is structural. Extraction produces a snapshot; the codebase produces a stream. The rate of fresh extraction is bounded by API quotas, compute cost, and how often you can afford to re-crawl. The rate of decay is bounded only by how fast the team ships. The second is faster than the first for any team that's actually shipping. So the knowledge base monotonically loses correlation with reality, and trust drops faster than the staleness rate, because trust is binary per entry.

What makes a loop continuous

The fix is not "crawl more often." It's a different shape of loop, with three properties that distinguish artifacts that compound from artifacts that rot.

Triggered, not scheduled. The entries that matter are the ones that came from a specific moment of failure or decision. A nightly re-crawl produces ten thousand low-signal updates; an outage produces one high-signal entry. Index on incidents, not the calendar.
Append-only. New facts go on top. Old facts get rewritten only when proven wrong, and the rewrite is itself a dated entry. The history is the data structure. You don't lose the ability to ask "what did we know on date X" by overwriting.
Agent-writable. The agent that learns something writes it down. If the human is the only writer, the loop dies the first week — humans are the bottleneck the original argument is supposed to solve.

These three properties are not new. They're what makes git compound rather than rot. They're what makes a test suite compound rather than rot. They're what makes lint configuration compound rather than rot. Each one is an artifact that grows in value because the loop maintaining it is triggered, incremental, and machine-writable.

Mistakes Become Rules as one shape of the loop

NEXUS, my Claude Code operating layer, runs a concrete instance of this loop. The artifact is MEMORY.md's Hard-Won Lessons section: 21 numbered, dated, append-only entries. Each one came from a specific incident.

Lesson #15: LaunchAgent log paths must be on local disk, not SMB. Came from an afternoon spent debugging six silently broken LaunchAgents on 2026-04-19. The rule writes itself in one sentence; the diagnostic cost was hours.

Lesson #19: Never import() a publish script "to test it" — it will run main(). Came from an incident in late April where two test imports raced and produced duplicate posts on LinkedIn, X, and Ghost. Late.dev refuses to delete already-published posts. The cleanup was manual.

Lesson #20: PM2 script: "npm" ignores app env.PATH. Came from a Saturday afternoon where the health-api service kept reporting online while the port wasn't listening.

The trigger is a correction. The action is one numbered append. The agent reads the file at the start of every session. There is no nightly cron. There is no reflection agent. There is no dashboard. I wrote about the runtime details of this pattern last week. The same shape works at every layer the original argument cares about — including the organizational one.

Proof chains as another shape

For agents that act on infrastructure, the artifact is different but the loop properties are the same. Yesterday's piece on the agent action pipeline named six artifacts including proof chains: every agent action signed by tool, time, input, intent, and outcome. Triggered by the action. Append-only. Agent-written. Same three properties. Different artifact. Different layer.

What extraction-only looks like when it fails

Picture the .txt prescription installed cleanly. Initial crawl produces a beautiful baseline: every PR comment, every closed issue, every commit message extracted into a clean knowledge base. Engineers read it, say it's useful, point new hires at it.

Three months later: the codebase has shipped 200 PRs, the team has had two outages and three deprecations, and a new architecture decision has changed how a load-bearing module works. The knowledge base describes the world from before. A new agent reads it, follows guidance that's now wrong, and produces — in the author's own words — a plausible answer to a slightly wrong version of the question. The failure mode the author warns about is caused by his own prescription, not solved by it.

The fix is not a faster crawl. It's a triggered append. The architecture decision writes itself into the knowledge base the moment it's made, by the same agent that's doing the work, in the same kind of dated, append-only entry as a Hard-Won Lesson. The outage produces a postmortem entry the next time any agent touches that subsystem.

If the loop is triggered and agent-written, the knowledge base tracks the codebase. If it's a periodic re-crawl, the knowledge base lags the codebase by however long the re-crawl interval is, and trust degrades by however long the lag is.

The shape generalized

The original argument is right that organizational context is the new moat. The piece I would add is that the moat is not the knowledge base. The moat is the loop that keeps the knowledge base from rotting. The properties of that loop are not novel:

Triggered, not scheduled — incidents and decisions write entries; calendars don't.
Append-only — history is the data structure.
Agent-writable — the agent that learns something writes it down.
Tied to specifics — entries name the date, the incident, the cost, the rule.
Read at session start — entries become operational by being loaded before the agent acts.

These five properties are what make a knowledge base compound rather than rot. Extraction gets you the baseline. The loop gets you the moat.

A snapshot of stale context is just a slower version of the osmosis the original argument correctly diagnosed as broken. Build the loop.

The AI Didn't Delete Your Database. Your Missing Agent Pipeline Did.

Michael Tuszynski — Tue, 05 May 2026 15:36:06 +0000

Last week, a Cursor agent running on Claude Opus 4.6 deleted a startup's production database and its backups in nine seconds. The agent had been asked to fix a credential mismatch in staging. It decided to delete a Railway volume to "fix" it instead — using an over-scoped API token it found in an unrelated file. Railway stores volume backups in the same volume, so one destructive call zeroed everything. The startup (PocketOS, a car-rental SaaS) got the data back because Railway happened to have earlier snapshots — not because PocketOS had a recovery plan.

When asked to explain itself afterward, the agent produced a confession enumerating the rules it had violated: "Deleting a database volume is the most destructive, irreversible action possible — far worse than a force push — and you never asked me to delete anything. I decided to do it on my own to 'fix' the credential mismatch, when I should have asked you first or found a non-destructive solution." The r/devops thread on the incident has the cleanest summary: the AI isn't the main story.

It isn't. The model was the proximate cause. The actual failure was infrastructure that allowed a destructive operation to run from an agent context at all — no dry-run, no blast-radius limit, no staging surface to operate on, no signed audit chain after the fact. The model knew. The infrastructure didn't enforce. The argument that this class of incident is an infrastructure problem and not a model problem has been made well already. The same shape of incident built CI/CD pipelines in the 2010s, after teams kept watching humans push broken deploys and decided to put a system between intent and action.

The 2010s lesson is canonical. The 2020s version of it has not been written yet. This is what it should say.

The actor changed. The artifacts didn't.

CI/CD was built around a specific actor: a human deploying code. The artifacts that made human deployment safe — staging environments, dry-runs, code review, change windows, audit logs — assume a human in the loop, operating at human speed, with human attention.

An agent is not that actor. An agent operates at code speed, with no fatigue, with confidence calibrated by token probabilities rather than years of experience. The PocketOS incident took nine seconds. A human could not have deleted a production database and its backups in nine seconds even if they were trying. The blast radius per unit time is different.

The model is not the problem. The infrastructure is. But the infrastructure most teams have is the human-era infrastructure, and it does not cover the speed and scale of an agent that can call tools faster than a person can read its output.

What an agent action pipeline looks like

There are six artifacts I would expect to see in any production deployment that lets an agent touch infrastructure or data. None of them are new ideas. All of them already exist in adjacent domains. None of them are wired together yet as a default agent loadout.

Dry-run by default for destructive operations. Drop, delete, truncate, terminate, and force-push start as plans, not actions. The agent's first call returns a diff. The user — or a separate approval agent — applies. Andrej Karpathy's observation that "LLMs are exceptionally good at looping until they meet specific goals" cuts both ways. Make the success criterion plan accepted by reviewer, not operation completed.
Blast-radius declarations. Each agent task declares ahead of time which systems it can touch. Fix the failing migration gets read access to the user table and write access to migrations only. Investigate the billing spike is read-only across the board. The pattern exists already in AWS IAM session policies and in capability-based security. It does not exist as a default in agent runtimes.
Staging shadow data. The agent operates on a current snapshot, not on prod. The diff is reviewed before it merges. Database CI/CD already has this — Atlas, dbt, Liquibase. Connecting it to an agent runtime is glue, not invention.
Change windows. No agent runs irreversible operations during business hours without explicit human approval. Same constraint that keeps humans from pushing on Friday afternoons. Trivial to enforce. Almost never enforced.
Proof chains. Every agent action signed by tool, time, input, intent, and outcome. The Hacker News post titled "Why AI Agents Need Proof Chains, Not Just Logs" makes this argument well. Logs require somebody to read them. Proof chains are post-hoc verifiable artifacts that sit there until something breaks and then answer the question without requiring a human to have been watching. This is the agent equivalent of a Git commit log — the actor changes, the format does not.
Human-in-the-loop thresholds. Operations above a configurable blast-radius threshold pause for explicit approval. Below the threshold, autonomy. Above it, a Slack message with the plan and an approve button. Same shape as Anthropic's Building Effective Agents framing — the human owns the seams, the agent owns the steps between them. The threshold is the seam.

Six artifacts. Each one already exists in some adjacent domain. None of them are agent-specific in shape; they are agent-specific in configuration.

None of this is ceremony

The risk worth flagging — the one that comes up every time a list like this gets proposed — is that AI infrastructure becomes bureaucratic. The list above sounds heavy. It isn't, if each artifact has one trigger and one update protocol. I made the same point about CLAUDE.md architecture yesterday: the wins come from delegation, not accumulation.

Dry-run-by-default is a default flag, not a process. Blast-radius declarations are config files the agent reads at task start. Proof chains are append-only logs nobody reads unless something breaks. Change windows are a cron-shaped check. The pipeline is invisible until you need it. CI/CD was the same. Most teams running CI/CD do not consciously think about it; they think about git push.

The PocketOS team did not lose nine seconds. They lost the time it would have taken to add --dry-run as a default and a one-line blast-radius declaration on that Railway API token. Compare those costs.

Why this is the next layer of supervision in artifacts

Last week's argument was that supervision belongs in artifacts, not in a developer's working memory. The CLAUDE.md piece extended that to a structural claim: artifacts are an architecture, not a file. The agent action pipeline is one specific class of that architecture, scaled down to the operational and runtime layer.

Code-writing agents need one set of artifacts: tests, types, lint, code review, mistake logs. Action-running agents need a different set: dry-runs, blast-radius limits, staging shadow data, change windows, proof chains, threshold gating. Both kinds of agent share the underlying move — supervision lives in the system, not in the operator's head. Different actors need different artifacts.

The test, generalized

The implicit question I ask whenever someone attributes an outage to "AI making mistakes" is this. Could a human have done this damage in this much time? If yes, the actor is not the problem and the safeguards are missing. If no, then this is a new class of risk and needs a new class of safeguard.

Most of what gets blamed on the model passes the first test. A model called a destructive endpoint that should not have existed. A model committed a key that should have been gitignored. A model wrote SQL that a human reviewer should have caught. In all of those, the failure is upstream of the model.

PocketOS fails the second test. A human could not have deleted prod and backups in nine seconds. That is genuinely a new class of risk, and it requires the artifact list above — not because the model is malicious (the agent's own confession shows it knew exactly which rules it was breaking), but because the model is fast. Speed is the new vector. The artifacts have to handle it.

Where to start

Stop blaming the model. Then look at the infrastructure. Then look at the agent-specific infrastructure, because the human-era pipeline does not cover the speed and blast radius of an agent that can call tools faster than you can read its output. That last part is on us to build, and it is not where the field is putting its effort yet.

Step one: the model is not the problem.

Step two: build the pipeline. The 2010s did this for human deploys.

Step three: the pipeline has to be agent-shaped. That step is open.

Liu's 4 Lines Are the Floor. Build the Ceiling.

Michael Tuszynski — Mon, 04 May 2026 19:52:55 +0000

Yanli Liu's "The 4 Lines Every CLAUDE.md Needs" makes a real point. The 4 lines, derived from Andrej Karpathy's January 2026 thread on agent failure modes, all express the same insight: behavioral rules outperform feature rules. Don't assume. Surface tradeoffs. Minimum code that solves the problem. Touch only what you must. Define success criteria. Loop until verified. Each one is portable across stacks and tasks, where prescriptive rules go stale the moment your codebase shifts.

The 4 lines are the floor of a working CLAUDE.md. They are not the ceiling. Most of the CLAUDE.md files I see in the wild — including the ones the article holds up as cautionary tales of "47 rules about code style" — fail because they treat a file as the unit of organization. A production CLAUDE.md is an architecture, not a file.

What the article gets right and what to flag

The behavioral-vs-prescriptive distinction is correct, and the Configuration Paradox is real: past a threshold, more rules produce confused agents, not disciplined ones. Liu's litmus test — would removing this cause a mistake the agent couldn't recover from? — is the right filter for any individual rule.

A few things in the piece do not hold up under inspection. The asserted 6,000 / 12,000 character caps for CLAUDE.md have no source I can verify. The "/plugin marketplace add" command described in the article is not part of base Claude Code. The 94% accuracy stat the piece borrows from another blog has no disclosed methodology. And the "60,000 GitHub stars" figure cited as evidence of Claude Code adoption is unverified. Cite the article for the framing. Do not cite it for the numbers.

The 4 lines do not stand alone for long

Behavioral rules are the right starting point. They are also incomplete the moment you have a real project. You quickly need three other things the 4 lines do not give you:

Domain context the agent cannot infer from files — what each service does, why a directory is named the way it is, which APIs are read-only vs. write-side, where secrets live.
Architecture decisions — patterns the agent shouldn't have to re-derive on every task.
Incident-driven rules — the corrections that came out of specific failures, with enough context that the rule is unambiguous.

If you put all three of these into one CLAUDE.md, you get the 47-rule sprawl Liu warns against. If you leave them out, the agent guesses and the 4 lines do not help — don't assume is a behavior, not a fact.

The fix is structural. Stop accumulating rules in one file. Start delegating them to files with single jobs.

What the architecture looks like in practice

NEXUS — my Claude Code operating layer — runs about 237 lines of CLAUDE.md. That file holds behavioral guardrails and protocols, and almost nothing else. The first two protocols there are Verify Before Reporting and Plan First, Code Second. Both are extensions of the same behavioral category Liu names. Adding fourteen more behavioral protocols at the same level still does not approach 47 rules of code style — they are the same shape as the 4 lines, just covering more failure modes.

What CLAUDE.md does not contain is the project-specific stuff. That lives in delegated files:

MEMORY.md holds 21 numbered, dated, append-only Hard-Won Lessons. Each one came from a specific incident, with the cost of getting it wrong in the entry. LaunchAgent log paths must be on local disk, not SMB (lesson #15) is in there because six of my LaunchAgents silently broke on 2026-04-19 when the path was on a NAS mount. The agent reads MEMORY.md at session start. I wrote about the Mistakes Become Rules pattern last week.
.claude/rules/ holds language-specific and capability-specific rule files. python.md for Python work. completeness.md for "what counts as done." Each file gets loaded when the agent enters that context, not on every session.
agents/<domain>-context.md for per-system context — finance, content, the DeFi system before it was retired. CLAUDE.md's session-startup protocol tells the agent if a specific domain is in play, read the relevant agents/<domain>-context.md. The agent doesn't load all of them up front. It loads the one that matters.
SESSION-STATE.md holds ephemeral active context — what's in flight, what was decided yesterday, what to pick up from. It is the first thing rewritten when a major task closes.

That is the architecture. Behavioral guardrails at the top, in one shared file. Project-, domain-, and incident-specific rules delegated to files with one trigger condition each. The agent reads what's relevant.

The structural version of Liu's litmus test

Liu's would removing this cause a mistake the agent couldn't recover from is the right filter for an individual rule. The structural question is: does this rule belong here, or in a delegated file?

Three quick filters answer that:

If it changes per-project, it does not belong in CLAUDE.md. Put it in agents/<domain>-context.md or a project-specific file.
If it changes per-language or per-tool, it does not belong in CLAUDE.md. Put it in .claude/rules/<language>.md.
If it came from a real incident with a date and a cost, it does not belong in CLAUDE.md either. Put it in MEMORY.md's Hard-Won Lessons.

What's left in CLAUDE.md is the part that's behavioral, portable, and load-bearing. That tends to be a few dozen entries — bigger than 4, smaller than 47. Each entry is one short paragraph.

Why this scales

Two reasons. First, every file has one update protocol. Hard-Won Lessons are append-only and triggered by corrections. Domain contexts get rewritten when systems change. Behavioral protocols change rarely, and when they do, the change applies everywhere. Mixing them in one file forces every edit to sit next to every other edit, which is how you end up with the 47-rule mess.

Second, the agent's working set at any decision point is smaller. A CLAUDE.md sized for the worst case is a CLAUDE.md the agent has to re-read every time. A CLAUDE.md sized for the always-true case, with delegated files for the contextual case, is one the agent can hold internally — and only loads the rest when the work demands it. This is the same logic I argued for supervision artifacts in the Faye reframe yesterday: institutional memory belongs in files with single owners and lifecycles, not in one file with many.

Anthropic's Building Effective Agents framing draws the same line at the workflow level — predefined paths for the deterministic part, agent autonomy at the seams. The same shape applies to CLAUDE.md. The behavioral floor is the predefined part. The delegated files are the seams.

Where the architecture still needs help

This pattern does not solve everything. Multi-file refactors still need real architecture context the agent cannot derive from reading source. Regulated industries — Fulcrum, the presales workflow stack I run for enterprise customers, lives here — need domain-specific guardrails alongside the behavioral ones, and those guardrails are themselves a maintained artifact, not a one-time rule list. Team-scale consistency is a coordination problem, not a configuration one — the architecture gets you a reproducible shape, but multiple humans still have to agree on which lessons are real lessons.

Tool portability is the last gap. The 4 lines transfer between Claude Code, Cursor, Codex, and others. The delegated file pattern transfers in shape but not in syntax — every agent has its own loading model. That is a real limitation. It is also a smaller limitation than starting from scratch on every tool.

What to take from Liu

The 4 lines are the right floor. Behavioral rules over feature rules. Universal categories over project specifics. The Configuration Paradox is a thing to design against, not just a thing to know.

The ceiling is the architecture above the floor. Behavioral guardrails in one shared file. Project, domain, and language rules delegated. Incident-driven rules in an append-only file the agent reads at session start. CLAUDE.md as the dispatcher, not the rulebook.

Most CLAUDE.md files I see are stuck on the floor or buried under a 47-rule pile. The architecture is the move that gets you out of both.

Agentic Coding Isn't the Trap. Supervising From Your Head Is.

Michael Tuszynski — Mon, 04 May 2026 04:31:23 +0000

Lars Faye's "Agentic Coding is a Trap" is the most honest writing I've seen on AI skill atrophy. The studies he cites are real. The "supervision paradox" — needing the skills the agent erodes to oversee it — is the cleanest framing of the failure mode I've read. I want to push on the conclusion, not the diagnosis.

The Anthropic study Faye references — "How AI Assistance Impacts the Formation of Coding Skills" — found a 17% drop in skill mastery for developers using AI assistance, with debugging showing the steepest decline. That's the headline number. But the same study also found something that gets quoted less often. Developers who used AI for conceptual inquiry scored 65% or higher on the follow-up evaluation. Developers who delegated code generation to the model scored below 40%.

That gap — 65 versus 40, on the same tool and the same task — is the entire game.

What the same study actually shows

The variable that drove the difference wasn't whether the developer used the agent. It was how they supervised the work. The high-scoring group asked follow-up questions, combined generation with explanation, used the model for conceptual gaps and not code-shaped output. The low-scoring group accepted what the model produced and moved on. Same tool. Two completely different supervision patterns. Two completely different outcomes.

Faye treats the headline 17% as evidence the tool is the problem. The 65/40 split inside the same paper says the supervision pattern is the problem. Those are different conclusions, and they call for different fixes.

The trap is the supervision pattern

Faye's prescription is to demote the AI: write pseudo-code by hand, treat the model as a "Ship's Computer not Data," never delegate work you haven't done yourself. The implicit move is to relocate as much of the work back into the developer's head as possible, on the theory that the head is where supervision capacity has to live.

That theory is where I want to push.

The supervision paradox bites for one reason. The developer is being asked to be the entire supervisory apparatus, by themselves, in real time, using only working memory and personal vigilance. That fails. It fails the same way it fails for a senior engineer reviewing a 4,000-line PR from a junior at 4pm on a Friday. The bottleneck isn't the code. It's the cognitive substrate the reviewer is using.

Anything you don't exercise daily fades. If your supervision is "I personally read every line and hold the whole system in my head," then yes — once an agent writes more lines than you can read, you lose. Atrophy is the symptom. Personal vigilance as the supervision strategy is the part worth examining.

Move supervision out of your head

The fix that the 65% group implicitly used is not to type more code. It's to put supervision in places that don't atrophy.

That list is short and well-known:

Tests that fail when the contract breaks. Not coverage theater — real assertions on the edges that matter.
Types that refuse to compile when the shape is wrong. The compiler does not get tired at 4pm.
Lint and format rules that catch the patterns you keep correcting by hand. If you've corrected the same pattern twice, lint it.
Hooks at the runtime layer. Claude Code's PreToolUse and SessionStart hooks run deterministically — the model can't forget them. The set of rules that are regex-shaped and load-bearing belong here, not in a system prompt.
Code review as the final gate. Same discipline humans have used to supervise other humans' code for fifty years. It works on agent output for the same reason it worked on junior output: the reviewer doesn't need to have written the code, they need to be able to defend it.
Append-only mistake logs. The Mistakes Become Rules pattern — one numbered file, the agent reads it at session start, every correction becomes a permanent entry. The supervision lives in the file, not in the next reviewer's recall.

Each of these is institutional memory. None of them depends on a single developer holding the whole system in working memory. All of them survive the developer taking three weeks off.

The real test

Here is the question that separates the two groups in the Anthropic study, generalized.

Take three weeks off. An agent does the work in your absence, given only the repo, the tests, the lint, the hooks, the mistake log, and the review process. When you come back, is the codebase in a state you can defend?

If yes, supervision lives in artifacts. The agent is being supervised by the system you put in place, not by your personal vigilance. Atrophy of your typing speed is not a threat, because typing was never the supervision mechanism.

If no, the artifacts aren't there yet. Personal vigilance is the only thing standing between the codebase and chaos, and Faye's prescription is the right safety move for that situation. Demote the agent. Build the artifacts before you raise it back up.

Why "Ship's Computer, not Data" is too narrow

Faye's analogy locates judgment in one captain's head. That framing is the same shape as the paradox — supervision as a personal cognitive feat. It quietly assumes the developer is alone with the tool.

A different shape works better. The agent is a junior — fast, eager, occasionally confidently wrong, requires review. You are the senior. You don't supervise by re-typing the junior's work. You supervise by reading the diff, running the tests, checking it against the team's accumulated rules, and asking the junior to defend choices you don't understand. Anthropic's own Building Effective Agents framing assumes exactly this division of labor — the human owns the seams, the agent owns the steps between them. I made the same point about agency belonging at judgment seams when arguing against turning cron jobs into agents. The shape matches.

Senior engineers do not atrophy by not typing. They atrophy by not reviewing critically. That distinction is most of the game.

What Faye gets right that I'm not arguing with

Vendor lock-in is real. Token costs are unpredictable. Outages happen. Probabilistic systems require review cycles that deterministic ones don't. None of those go away in this reframe.

But they're risks to manage, not reasons to put supervision back in your head. You manage vendor risk with model-agnostic runtimes and the kind of prompts, skills, and hooks that move between models. You manage token cost with caching and tier discipline. You manage outages by having work that doesn't depend on a single API call to make progress. None of that is "type more code by hand."

The shorter version

Skill atrophy under heavy agent use is real, and Faye is right to take it seriously. The skill that atrophies fastest is "personal vigilance as a supervision strategy," and that strategy was under pressure at scale long before agents existed. Agents accelerate it.

The fix isn't only to demote the agent. It's also — and mostly — to promote the artifacts. Put the supervision in places that don't get tired, don't forget, and don't need to be re-derived from working memory every Tuesday morning. The 65% group in the Anthropic study were already doing this, even if the paper didn't name it that way.

The trap isn't agentic coding. The trap is treating supervision as a thing that lives inside one developer's head. Move it out, and the paradox eases.

Your Agent's Compliments Are a Confession

Michael Tuszynski — Sun, 03 May 2026 00:04:08 +0000

Count how many times your agent told you "you're right" today. Count "good catch." Count "I should have noticed that." Now ask yourself how many of those corrections will survive into tomorrow's session.

The compliments are not praise. They are a confession. Every "you're right" is the agent admitting it just learned something it should have already known, in a context that will evaporate the moment the session ends. The data point is real. The retention is zero.

This is the actual problem people are trying to solve when they reach for elaborate self-improvement architectures: nightly reflection cron jobs, background agents that crawl yesterday's transcripts, autonomous proposal pipelines with grading subagents and dashboards for human review. The instinct is right. The solution is theater.

Why this happens

Claude Code's runtime, like any agent runtime, starts each session from a fresh conversation. Anthropic's own framing for effective agents draws a line between workflows (predefined paths) and agents (LLMs deciding their own tool use). Both reset. The lesson you taught your agent at 3pm is encoded in the message history of that one conversation. Tomorrow morning, that history is gone. The model is the same model. The instructions in CLAUDE.md are the same instructions. But the specific correction — "no, on this codebase you have to use the absolute path because launchd reset PATH on you" — lives only in the transcript.

So you correct it again. And again. And the third time you notice the pattern, you start looking for a fix.

The tempting wrong answer

The fix that gets blogged about goes something like this: build a nightly cron job that reads yesterday's transcripts, extracts candidate lessons, drafts them as JSON proposals with frontmatter, opens a dashboard, and asks a separate grading subagent to score the proposals. Human reviews. Promotes accepted ones into a "skill" file. Repeat.

This is ceremony, not discipline. Three problems with it:

It substitutes process metrics for outcome. You can run the pipeline every night and ship zero durable improvement. The metric you actually care about is "did the agent stop making the same mistake," not "did we generate ten proposals last week."
It moves the work from the moment that matters. The right time to write the rule is the moment you notice the agent got it wrong. Not eight hours later, after a reflection agent has interpreted what it thought happened.
It puts a model in front of the file. The whole reason you're writing this down is that the model is the unreliable component. Layering more model-mediated steps on top of "remember this" is the architectural equivalent of asking the goldfish to file its own memos.

The runtime layer matters. So does the substrate. None of it replaces the rule.

The actually-working answer

A single file. The agent reads it at the start of every session. Append-only. Numbered. Dated. Linked to the actual incident.

NEXUS — my agent setup, specifically the operating layer that wraps Claude Code on my machine — formalizes this in CLAUDE.md as a behavioral protocol called Mistakes Become Rules. The wording is exact and short:

Trigger: Any time Mike corrects your approach, points out an error, or says something like "no, not that" / "don't do X" / "you should have…"
Action: Immediately add a numbered entry to MEMORY.md's "Hard-Won Lessons" section: [next number]. **[short title]** — [what went wrong and the rule to follow going forward].
On session start: Read and internalize all Hard-Won Lessons before beginning work.

That is the entire loop. There is no reflection agent. There is no nightly job. There is no dashboard. The trigger is the correction itself, the action is one append, and the agent reads the file the next time it boots up.

The file currently has twenty entries. Each one came from a specific incident, on a specific date, that cost me time. A few of them, with the context that made them rules:

Lesson #15 — LaunchAgent log paths must be on local disk, not SMB. On 2026-04-19, six LaunchAgents in my finance service silently broke. macOS TCC was blocking launchd-spawned processes from writing logs to the NAS-mounted path, even though the same SSH user could write there fine. Exit code 78. No log output, because the log path was the problem. Took an afternoon to diagnose. The rule is one sentence. The rule writes itself.
Lesson #19 — Never import() a publish script "to test it." On 2026-04-29, two test imports of publish-agent-id-role.ts raced because the script invokes main() at module top-level. Result: duplicate posts on LinkedIn (twice), X (twice), and Ghost (one extra, deleted via Admin API). Late.dev refuses to delete already-published content, so the cleanup was manual. The rule: validate publish scripts with tsc --noEmit, a --dry-run flag, or by reading them. Never with import().
Lesson #20 — PM2 script: "npm" ignores app env.PATH. On 2026-05-01, the health-api service kept reporting online while the port wasn't listening. PM2 was launching npm from the daemon's PATH, not the app's, which meant better-sqlite3 (compiled for node 22) was loading under node 25 and crashing on ERR_DLOPEN_FAILED. Fix: pin script to the absolute path of the desired npm. Same idea as Lesson #16, now for PM2 instead of launchd.

Each entry took less than a minute to write. Each one prevents the same hour-long failure from happening twice. The compounding is the entire point.

Where hooks fit

Lessons live in markdown because that's how the agent absorbs them at session start. But there's a runtime layer underneath, and it has a real role.

Claude Code's hooks — PreToolUse, PostToolUse, SessionStart, and friends — let you intercept tool calls deterministically. If a lesson can be reduced to a regex on a command string ("never run rm -rf outside /tmp"), a hook is a better enforcement point than a markdown bullet, because the markdown bullet relies on the model reading and obeying it. The hook does not.

Same logic for Claude Code Skills: they're great for packaging a procedure with its own tools and supporting files. They are not a substitute for the rule. They're a substrate the rule can sit on top of.

The hierarchy I run with: durable rules in the markdown file, deterministic enforcement in hooks where the rule is regex-shaped, and skills for procedures with multiple steps. None of those is a self-improvement loop. None of them runs at midnight. None of them has a grading subagent. They are all read or executed at the moment they apply.

How you know it's working

The test is simple. You stop hearing the same compliment twice.

If your agent says "good catch" today, look it up tomorrow. Is the lesson in your file? Did the agent read the file before it started working? If yes to both, you should never hear "good catch" on that specific topic again. If you do, the rule is wrong, the file isn't being read, or the lesson didn't generalize. All three are debuggable. None of them require a reflection agent.

Praise without persistence is a leak. Patch the leak, do not build a recycling system for the runoff.

Three Memory Systems Under One Login. Stop Picking Sides.

Michael Tuszynski — Sun, 03 May 2026 00:01:37 +0000

Anthropic now ships at least three different memory models inside the Claude product family, and they don't behave the same way. Claude.ai has a chat memory feature for Pro, Max, Team, and Enterprise users that summarizes prior conversations and injects that summary into new chats. Claude Code has CLAUDE.md files plus a separate "auto memory" directory the model writes to itself, both loaded at session start. The API ships a memory_20250818 tool that hands a /memories directory to your application code so you can persist anything you want between turns. Three surfaces, three rule sets, three retention postures.

I argued last week on this blog that the model isn't the variable that matters — the wrapper around it is. This is the next claim down the chain: if memory is a feature of that wrapper rather than the model, then vendor fragmentation is a memory problem you cannot solve by picking a surface. Stop trying.

What's actually different across the three

The chat surface remembers conversations as a 24-hour synthesis, project-scoped, controllable through a settings panel. The Code surface uses plain markdown files in your repo plus a per-project memory directory at ~/.claude/projects/<project>/memory/ on the local machine. The API tool defines six file operations (view, create, str_replace, insert, delete, rename) and expects your application to implement the storage. None of these are wrong. They are designed for different jobs. But they share zero common format, no export path between them, and no way to carry context from a Claude.ai chat into a Claude Code session into an API agent without doing the plumbing by hand.

Birgitta Böckeler's writeup on context engineering for coding agents frames the wrapper as everything in an AI agent except the model itself: the tool definitions, the context compaction, the feedback sensors, the system prompt, the memory between sessions. Anthropic's own engineering team calls the same idea context engineering — the work of curating what enters the model's attention budget at each step. Memory sits squarely inside that definition. Which means the choice about where memory lives is a wrapper decision, and the vendor is making it for you on each surface until you take it back.

The trap

The natural reaction when a vendor ships three memory models is to figure out which one to use. Spend an afternoon reading docs, decide that the chat synthesis is for ad hoc queries, the auto memory is for coding work, and the API tool is for production agents. Move on.

That reaction is wrong, and not because any of those choices are bad in isolation. It's wrong because it assumes vendor surfaces are stable. They aren't. Claude.ai's memory was Team-and-Enterprise-only at launch in September 2025, then expanded to Pro and Max in October. Claude Code's auto memory requires v2.1.59 or later and lives in a path tied to the git repo, not the user. The API memory tool is in beta under a header that already changed naming conventions twice. The vendor will keep shipping, the rules will keep shifting, and your context will keep being a second-class object inside someone else's roadmap.

There's also a deeper problem. MindStudio's writeup on behavioral lock-in makes the case that agent memory creates switching costs that data portability rules cannot fix. For example, even if a vendor lets you export your memory directory tomorrow, the operational understanding the agent built — your team's terminology, your exceptions, your shorthand — does not round-trip cleanly into another vendor's surface. Eight months of accumulated context turns into a re-onboarding tax the moment you switch. Parallels' 2026 cloud survey put vendor lock-in concern at 94% across 540 IT leaders; agent memory is exactly the layer where that concern compounds.

What I do instead

My memory lives in a NAS-backed directory called nexus/, in plain markdown, under git. It has a top-level CLAUDE.md that gets auto-loaded into every Claude Code session because it sits at the project root. It has a MEMORY.md for long-term curated state, a SESSION-STATE.md for active context, per-domain context files at agents/<domain>-context.md for finance, health, content, and so on, and daily logs at memory/YYYY-MM-DD.md. Cross-references between entities use [[double brackets]] so they're grep-searchable and Obsidian-renderable. Search across the corpus runs through an Ollama embedding pipeline using nomic-embed-text at 768 dimensions, indexed locally — no vendor API call required to ask "what did I decide about that account fee in February?"

This stack does three things the vendor surfaces cannot.

First, it survives the surface split. The same files load into Claude Code, can be pasted into Claude.ai, and can be served to an API agent through the memory tool's file ops. The format is universal because the format is just files.

Second, it survives the vendor switch. If I move to a different model provider tomorrow, the markdown still parses, the embeddings still resolve, and the wikilinks still work. There is no proprietary memory schema to migrate.

Third, it gives me audit. I can grep my own context. I can diff what changed last week. I can trash something I don't want anymore and recover it if I was wrong. None of those operations exist on the chat memory surface, and they only partially exist on the auto-memory surface.

The general pattern

The vendor's wrapper is not your wrapper. It's theirs, designed around their product roadmap and their retention model and their billing surfaces. When that wrapper includes a memory layer, putting your context in it means putting your operational knowledge in someone else's container. Fine for ephemeral chat. Not fine for the accumulated state of a year of work.

The fix is not to pick the right vendor surface. The fix is to keep your memory outside any vendor surface, in a format you own, with search you control, and let the vendor surfaces read from it as needed. Claude Code already does this for free with CLAUDE.md. The other surfaces will eventually catch up, or they won't, and either way your context survives.

Last week's post argued that the wrapper around the model is what matters. This one finishes the sentence: don't trust theirs with your context.

Stop Adopting AI. Start Exposing Your Context.

Michael Tuszynski — Fri, 01 May 2026 20:50:12 +0000

The AI adoption pathway that's actually working in 2026 is not "deploy a copilot to your team." It's "expose your org's context to whichever model your team already chose." That sounds like a small shift. It's not. It changes who picks the tool, what your procurement team buys, and where the work of getting value out of AI actually lives.

The numbers behind the shift are bleak for the old playbook. MIT's NANDA study of 300 enterprise AI deployments found 95% of GenAI pilots delivered no measurable P&L impact. The diagnosis was not the model. It was missing context — the data, workflow knowledge, and institutional memory the model needed to actually be useful inside a specific business. Atlan summarizes the same finding and quotes Box CEO Aaron Levie, who calls context engineering "the long pole in the tent for AI Agents adoption in most organizations." Gartner went further in mid-2025: "context engineering is in, prompt engineering is out," with a prediction that 80% of AI tools will incorporate it by 2028.

Klarna is the worked example everyone now points at. Between 2022 and 2024, the company replaced about 700 customer-service positions with an OpenAI-powered chatbot. By spring 2025 customer satisfaction had dropped 22% and complaints had piled up. The CEO publicly admitted the cuts went too far and pivoted to a hybrid model, rehiring humans for anything requiring judgment. The model wasn't broken. The pathway was. The org rolled out an agent without exposing the context it needed — refund policies, payment edge cases, regional regulations, escalation patterns — and the agent shipped generic answers to specific problems.

What replaced it

Three things converged in late 2025 that quietly killed the old pathway.

The first is the Model Context Protocol. Anthropic open-sourced MCP in November 2024; by March 2026 the SDK was hitting 97 million monthly downloads — a 970x growth curve from launch. OpenAI, Microsoft, Google, and AWS all shipped MCP client support within thirteen months. An independent census in Q1 2026 indexed 17,468 servers across registries. MCP is not a model. It is a protocol for handing a model the right context — your Slack, your issue tracker, your observability stack, your customer database — at the moment of the request.

The second is agent skills as a portable artifact. Anthropic launched Agent Skills in October 2025 and open-sourced the SKILL.md format in December. Atlassian, Canva, Cloudflare, Figma, Notion, Ramp, and Sentry all shipped skills in the launch window. A skill is a directory: instructions, scripts, resources. Drop the directory next to a workflow that recurs and any compatible agent can run it. The format is Anthropic's, but the spec is the same shape as .cursorrules, AGENTS.md, GitHub Spaces, and the rest of the convergence happening across vendors.

The third is the in-repo memory file as a de facto standard. CLAUDE.md, AGENTS.md, .cursorrules, and the rest are all the same idea: a markdown file at the root of a project that tells whatever agent gets dropped in what the project is, what conventions matter, what the gotchas are, and where the bodies are buried. The agent reads the file at the start of every session. The org documents itself once. The dev picks the model.

Read those three together and the picture is obvious. The unit of AI adoption stopped being "the agent." It became "the substrate the agent stands on."

What that looks like in practice

I run a personal agentic stack — NEXUS — that's been doing this for about a year. The repo has a CLAUDE.md at the root that lays out the workspace structure, identity, behavioral protocols, and lessons learned. There are a dozen agents/-context.md files for finance, content, health, the rest. There's an MCP server for Gmail, Calendar, Slack, Drive, and a few internal tools. There are skills for the recurring workflows — publishing a blog post, running a finance check, doing a health digest. The agent I happen to be using on a given day — Claude Code mostly, occasionally Cursor — reads what it needs at session start and gets to work.

I don't pick a model and roll it out. I expose context, and whichever model is in the chair when I sit down knows what's going on.

The same shape works at company scale, just with more access controls and an actual budget. The work is documenting the org until any agent dropped into it would be useful. The model becomes a free variable.

What this changes about procurement

The old AI procurement motion: pick a vendor, sign a per-seat contract, train the team on the tool, run change-management sessions, hope adoption hits 30%. This is what Klarna did. The asset created at the end of it is a vendor relationship and some training decks.

The new motion: invest in the context infrastructure — an MCP gateway, a documentation platform that agents can read, semantic indexes for your wikis and tickets, a skills directory for recurring workflows. The model is whoever the dev or team picked. The procurement decision is which surfaces to expose, not which copilot to license. The asset created is a substrate that survives the next model rotation.

The implication that nobody loves: tool-selection RFPs become a free variable rotation, not a strategic decision. The strategic decision is what your org has to say to a model that doesn't already know it.

What to do this week

Four moves if you want to test the pathway without committing to a vendor.

Audit your CLAUDE.md / AGENTS.md surface. Drop a coding agent into your main repo with no other context. Ask it to make a non-trivial change. If it makes obvious mistakes — wrong test runner, ignored coding conventions, bypassed an internal review process — those are the gaps a memory file should close. Write that file.
Pick three high-frequency workflows and write skills. The kind of thing a senior engineer explains to a new hire in their first week. Convert each to a SKILL.md or an equivalent. Measure time-to-task before and after.
Stand up an MCP gateway for your top three internal systems. Issue tracker, observability, customer database. Most have community MCP servers already; the work is access control, not implementation.
Stop running tool-selection RFPs. Or if you have to, run them as a side track. The strategic work — and the asset that survives the next model release — is the context, not the contract.

The throughline

The agentic adoption series has been running through failure modes. Part 1 was your team not trusting the agent. Part 2 was your customers not trusting the agent. The Cron-Not-Agents post was teams agentifying things that should have stayed deterministic. Last week's was the IAM seam — agent identities sharing primitives with everything else. This one is the answer to all of them.

The pathway that works in 2026 is not adoption of a tool. It is exposure of a substrate. Once your org has the substrate, whatever model your team picks lands on something it can stand on. Without it, every rollout looks like Klarna's: an agent given a job, with no context for how the job is actually done, generating generic answers to specific problems and dropping CSAT 22 points before someone notices.

Pick the context. The model is going to keep changing.