DEV Community: Michael Tuszynski

The 26% Who Never Rolled Back an Agent Aren't Winning

Michael Tuszynski — Wed, 29 Jul 2026 00:00:48 +0000

Somewhere right now there's a slide in a board deck claiming a clean record: zero agent rollbacks since launch. It reads like a win. It's closer to a smoke detector with the battery pulled out.

New survey data from Sinch puts the number at 74% of enterprises that have rolled back a deployed AI agent after it went live — customer-communications agents specifically, pulled over governance failures. That figure got picked up everywhere as an indictment — three out of four agent deployments blowing up in production. But the interesting number is the one underneath it. Among organizations with the most mature guardrails, the rollback rate goes up, to 81%.

Read that again. Better safety infrastructure correlates with more rollbacks, not fewer.

Rollback rate measures your eyes, not your agent

There's only one clean way to explain that inversion. The teams with better instrumentation aren't shipping worse agents. They're catching things the other teams are shipping past.

Every agent in production is doing something wrong at some rate. Hallucinated policy details, tool calls against stale records, a tone that drifts on the eighth turn of an angry conversation, a refusal loop that pushes a customer to a human queue that's already 40 deep. The question was never whether the failure exists. It's whether anything in your stack notices it before your customer does.

So the 26% split into two very different groups. A small number genuinely nailed scope — narrow task, tight tool surface, a human approving anything consequential. The rest have no detector. Their agent is failing at whatever the base rate is, silently, and the absence of a rollback is being reported upward as quality.

SRE teams learned this lesson two decades ago and it stuck: an incident count is a function of your monitoring, not your reliability. A team that goes from 3 incidents a quarter to 30 after installing real alerting did not get 10x worse. They got 10x more honest. It's the same principle the write-ups on this data keep circling — the model rarely breaks, the infrastructure around it buckles. Which means model choice isn't what separates the 26% from everyone else.

Treating rollback rate as a failure metric creates the exact incentive you don't want. A VP who gets graded on rollbacks will not build better agents. They'll build quieter ones — loosen the eval thresholds, downgrade the alert to a weekly digest, route the complaint channel to a shared inbox nobody owns. The metric improves. The agent doesn't.

The guardrail tax nobody budgeted for

The second number in that survey is the one that should reset your staffing plan. 84% of AI engineering teams spend at least half their time on safety infrastructure rather than on the agent itself. And enterprise investment now skews toward trust, security, and compliance (76%) over AI development proper (63%).

That's not overhead creeping in at the edges. That's the majority of the work.

It also matches what production looks like. The agent is a prompt, a model call, and a tool list — a week of work for a competent engineer. The other eleven weeks go to the eval set, the PII redaction layer, the escalation path, the audit log that survives a compliance review, the shadow-mode comparison, the per-version rollback switch, and the on-call rotation that owns all of it at 2 a.m.

Microsoft's 2026 Work Trend Index frames the organizational side of this as a capacity question — whether organizations are built to capture the agency that agents free up. Most aren't, and the guardrail tax is a good part of why. The headcount that was supposed to move up the value chain is instead maintaining the machinery that keeps the agent honest.

Who paid for this, and what it actually covers

Now the part the LinkedIn reposts skipped.

Sinch is a CPaaS vendor. They sell customer-communications infrastructure. A survey they sponsored concluding that infrastructure quality predicts agent success is a survey concluding that you should buy more of what Sinch sells. That doesn't make the data wrong, but it does mean the framing was chosen before the responses came in.

The scope is narrower than the headline suggests, too. This is customer-communications agents — support, messaging, conversational flows. Not coding agents, not internal RAG assistants, not the finance-ops bot reconciling invoices. A customer-facing agent has an unusually loud failure mode: the customer complains, and the complaint is logged in a system somebody already reads. Detection is close to free. In a domain where the agent's mistakes land quietly in a document nobody re-reads for six weeks, the 26% number would almost certainly be higher — and mean even less.

Hold the core finding anyway. The correlation between guardrail maturity and rollback frequency is directionally strong enough to survive the sponsorship discount, because it points the opposite direction from the sponsor's simplest sales pitch. "Buy our stuff and you'll roll back more often" isn't a slogan anyone reverse-engineers into a survey.

Instrument the rollback, then go trigger one

Stop reporting rollback count. Start reporting rollback provenance. One row per deployed agent version, and the field that matters is who noticed:

release_id      agent-support-v14
model           claude-sonnet-5
prompt_sha      a91f3c2
tools_sha       7de0b41
deployed_at     2026-07-14T09:12Z
rolled_back_at  2026-07-16T23:41Z
detector        eval_regression | slo_breach | abuse_signal
                | agent_self_report | human_complaint
time_to_detect  3410 minutes

Now run the distribution on detector. If 80% of your rollbacks say human_complaint, your detection layer is your customers, and your time_to_detect is however long it takes an annoyed person to find the contact form. Whatever that interval turns out to be in your own data, multiply it by your conversation volume — that's how many exchanges you can't un-send.

Three things follow from that ledger.

Set a target on time-to-detect, not on rollback count. An hour is achievable with automated eval replay on a sampled slice of live traffic. Days is what you get when complaint volume is the detector.

If you've never rolled back, go cause one. Inject a deliberately degraded prompt into a canary slice — drop a rule from the system prompt, or point a tool at a stale index — and time how long your stack takes to flag it. A rollback path that has never been exercised is a hypothesis, not a control. Same logic as restoring from backup: nobody has a backup, they have a restore they've tested or a file they hope is fine.

And when a vendor pitches you an agent, ask how many times they've rolled back their own. A confident zero means they aren't looking. The right answer sounds like "four times last quarter, median detection 22 minutes, here's the ledger."

The 81% cohort isn't failing more often. They're the only ones who can prove what their agent did last Tuesday at 3 a.m.

AI Use Is a 50/50 Coin Flip. Coding Already Tipped 61% Autonomous.

Michael Tuszynski — Thu, 23 Jul 2026 20:08:27 +0000

The Copilot Framing Is Already the Minority Report

Every vendor deck for a coding tool sells you the same picture: a developer in the driver's seat, the AI riding shotgun, a human hand on every merge. Copilot. Human-in-the-loop. Pair programming with a machine. It's the comfortable frame because it keeps the person central and the tool subordinate.

For software development, that frame already describes the smaller half of what's actually happening.

Anthropic's Economic Index publishes real data on how people use Claude, and it tags every conversation one of two ways. Augmentation means the person stays actively in the loop — back and forth, iterating, learning, steering each step. Automation means the person hands off a task and directs Claude to complete it. Across everything, it's nearly a coin flip: 51 percent augmentation, 49 percent automation globally, and a clean 50/50 in the US.

Narrow to software development tasks and the coin lands differently. 39 percent augmentation, 61 percent automation. The pattern everyone quotes in their pitch is the one coding has already moved past.

What the Split Actually Measures

Read the label carefully, because it's easy to turn this into something it isn't. Augmentation versus automation is a conversation style, not a jobs claim. It says nothing about displacement, headcount, or whether a role survives. It measures one thing: in a given exchange, is the human collaborating turn by turn, or delegating the whole task?

That distinction matters because the two styles want different products underneath them. An augmentation session is a dialogue — the tool's job is to be responsive, explain itself, and stay legible while a person drives. An automation session is a delegation — the tool's job is to run to completion and come back with something you can check.

Most coding tools are still built for the first job. The interface assumes you're watching, approving diffs one at a time, keeping a hand on the wheel. When 61 percent of the actual work is delegation, that assumption is backwards for the majority of what your developers are doing.

The Number That Keeps This Honest

Here's the counter-weight, and it's in the same dataset. Software development is only about 11.5 percent of global Claude requests, and 8.1 percent in the US. Coding is not the center of gravity for how people use these models. It's a slice.

That cuts against the temptation to read "61 percent automation in coding" as "AI is taking over programming." It isn't taking over anything. It's one task category, and inside that category the delegation style leads. Both things are true, and the small share is the part that keeps the big number from getting oversold.

One more caveat worth stating plainly: this is a single snapshot from May 2026. No trend line, no trajectory, no "up 12 points from last quarter." One reading. The 61/39 split is a photograph, not a movie. I'd bet the direction of travel is toward more automation as tools get better at running unattended — but that's my read, not something the data shows. Treat it as a fixed point, not a slope.

And note the seam in the categories: Anthropic reports "Computer and Mathematical" as the number one task category overall, yet software development specifically sits at 11.5 percent. Those aren't in conflict — the top category is broad and covers a lot more than writing application code. But if you conflate them, you'll overstate how much of AI usage is engineers shipping features. Most of it isn't.

What a Buyer Should Do With This

If you're choosing or building a coding agent, design for the automation majority. That's the whole recommendation, and it changes what "good" looks like.

The copilot default optimizes for a person watching every step. The automation pattern optimizes for a person checking the result. Those need different architecture. I've written before that a coding agent stack has two layers — the model and the scaffolding around it — and that the scaffolding decides the outcome more than the model choice does. This data is why. When 61 percent of real coding work is delegation, the layer that runs the agent unattended and catches its mistakes is doing the heavy lifting, not the chat window.

Concretely, designing for automation means:

Autonomous execution with verification gates, not approval-per-diff. The agent should run the task end to end, then hand you a result gated by checks it had to pass — tests green, types clean, lint quiet, a diff that touches only what it claimed. You review the outcome and the evidence, not every keystroke. A human approving line-by-line is the augmentation product, and it doesn't scale to 61 percent of your work.

Real gates, not vibes. The gates are the whole safety story once the human steps back from each step. That means a test suite the agent can't skip, a build it can't merge past when red, and provenance on what changed. If you can't articulate what has to be true before an agent's output reaches main, you haven't built for automation — you've built a faster way to generate unreviewed code.

Legibility at the boundary, not throughout. Augmentation needs the tool legible at every turn. Automation needs it legible at one point: the handoff. Invest your explanation budget there — what did it do, what did it check, what should you look at first.

The tools that ship copilot-by-default are optimizing for the 39 percent. That's a real 39 percent, and for exploratory work, learning a new codebase, or high-stakes changes where you genuinely want to drive, augmentation is the right mode. Don't rip it out. But if your default interaction model assumes a human hand on every commit, you've built the minority product and called it the platform.

The Frame Is Behind the Practice

The interesting gap here isn't between humans and machines. It's between how the industry talks about coding agents and how people actually use them. The marketing still runs on the copilot metaphor — reassuring, human-centered, a decade old. The usage data says developers crossed over to delegation a while ago and mostly stopped narrating it.

61 percent. That's the number to bring to your next tool evaluation. Ask the vendor what happens after the human stops watching — what runs, what gets checked, what stops a bad change cold. If the answer is "well, you approve each diff," they built for the half that's shrinking.

The Meter Is Always Running

Michael Tuszynski — Tue, 21 Jul 2026 14:02:11 +0000

The Meter Is Always Running

For forty years, enterprise software rested on a comfortable assumption: that it was a fixed cost. You paid a license, amortized it over a depreciation schedule, and every unit of value you squeezed out afterward felt like it was approaching free. Seats, not usage. Capex, not cost of goods. A CFO could draw a straight line from spend to return and sleep at night.

AI breaks that line. Most organizations adopted it without noticing.

The thing that actually changed

The interesting shift in enterprise AI is not cloud versus on-premises. That is a deployment question, and deployment questions are the kind of thing infrastructure teams have solved a hundred times. The deeper shift is quieter: software stopped being a fixed cost and became a variable one.

A large language model does not bill like a license. It bills like a taxi. The meter runs on every token, upstream and downstream, and input and output are priced per million tokens at rates that differ by model and by direction. The fare depends on how long the conversation ran, how much reasoning it demanded, how many times a user hit retry. Two customers doing the "same" task can cost 10x different amounts because one of them pasted a novel into the prompt. There is no seat count that caps it. There is no version you buy once and freeze. The cost scales with success: the more people use the feature, the more it costs you to have built it.

That is a new financial object, and the mental model most companies applied to it, SaaS licensing or depreciable capital, is the wrong shape. You cannot amortize a variable cost. You can only manage it.

The on-prem mirage

The popular rebuttal goes like this: just run it yourself. Buy the hardware once, own it forever, and your marginal cost per query approaches zero. Convert the scary variable cost back into a familiar fixed one.

It is a seductive move, and it is mostly wrong.

The "own it forever" asset is not the GPU. It is the model, and the model is spoiling in real time. The cluster you justified this year runs a frontier model that will be mid-tier in twelve months. The hardware ages too: how long an AI accelerator stays economically useful is now an open argument on Wall Street, with investors questioning whether the six-year depreciation schedules on the books match a reality where a new generation lands every eighteen months. You did not buy a depreciable asset amortized over seven years. You bought a seat on a refresh treadmill, plus a standing bill for power, cooling, and the engineers who keep it fed.

And unlike the cloud meter, that silicon costs the same whether it runs at 90% load or 5%. Idle capacity is the norm, not the exception: it is common for GPU clusters to sit 70 to 80% idle between bursts of real work. You did not eliminate the variable cost. You traded a usage-based meter for a utilization risk, and for most enterprises, whose demand is spiky rather than steady, that is the worse trade.

Marginal-cost-to-zero is real, but only at high, sustained load on a model you are content to keep running. That describes a narrow band of workloads. It does not describe "our AI strategy."

Inference is cost of goods, not capex

Here is the reframe that survives contact with a spreadsheet: inference is cost of goods sold.

Not a license you buy. Not a machine you depreciate. It is a per-transaction input cost, like the cloud compute behind a web request or the fee on a card payment. Companies already know how to run a business where serving each customer costs money. They have gross-margin discipline, unit economics, dashboards that flag when a product line goes underwater. They simply never had to apply that discipline to software features before, because software features used to be free to serve once written.

File inference under cost of goods instead of "IT capex," and the right behaviors fall out on their own. You start asking the questions a margin-conscious operator always asks.

What that discipline looks like

Know your unit cost per feature. If you cannot say what a single run costs, you are blind on the one number that decides whether the feature is a business or a liability. Instrument tokens the way you already instrument latency.

Then route by need. Most requests do not warrant a frontier model; send the cheap, frequent, predictable work to smaller or local models and reserve the expensive reasoning for the calls that actually earn it. A good router is worth more than a bigger model.

Cache aggressively. Repeated system prompts, retrieved context, common answers: every hit is a fare you skip. Anthropic reports that prompt caching can cut input costs by up to 90% and latency along with it. Caching is not a nice-to-have optimization here. It is margin.

Put a ceiling on it. Variable cost without guardrails is how one looping agent turns into a five-figure surprise. Budget caps, rate limits, and per-tenant quotas are the circuit breaker. Build them before the invoice, not after.

And decide placement per workload, not per company. Some things belong on a local model you own; some belong on a metered API. "Cloud-first" and "on-prem-first" are both the wrong altitude. The unit is the workload, and the answer is a portfolio.

The reckoning is real. It is not the one on the thumbnail.

A reckoning is coming for a lot of AI initiatives, but it will not arrive as the dramatic collapse of anyone's cloud strategy. It will arrive as a quiet quarterly review where someone finally divides the AI line item by the number of customers it served and does not like the answer. The projects that survive that meeting will not be the ones that picked the right deployment location. They will be the ones that treated inference as what it is, a variable operating cost with no natural ceiling, and built the discipline to manage it from the first day.

The meter was always running. The only question is whether you were watching it.

Your AI Coding Pilot Cost $80K and Shipped 9% Faster. Here's the Line Item Finance Missed.

Michael Tuszynski — Mon, 20 Jul 2026 14:07:50 +0000

The slide always looks the same. Pilot spend: $80K. Velocity: up 9%. Recommendation: expand to the full org. I have sat through enough of these readouts, on both sides of the table, to recite the speaker notes from memory. The room approves it, because $80K buying a 9% faster engineering org of any size is obviously a deal.

The math on the slide is fine. The problem is a line item that never made it onto the slide, and it is the most expensive input in the building.

Where the $80K Went

Run the composite pilot. Fifty seats of an AI coding assistant at roughly $20 a head is about $12K a year. Agent and API tokens for the teams that went past autocomplete, call it $3K a month once real workloads land — I've published my own bill, so I know what this curve looks like. Add the integration sprint, the eval work, and the enablement time nobody bills to the pilot, and $80K is a fair all-in number for a mid-size org's first serious year.

Every dollar of that is metered, invoiced, and visible. Seats show up on a purchase order. Tokens show up on a usage dashboard. Finance can read the pilot's cost to the penny, which is exactly why the readout feels rigorous.

Now the return side. A 9% velocity gain across a 12-engineer group is roughly one extra engineer's worth of throughput, and a loaded engineer runs $250K, which is the number that dwarfs every token bill in sight. So the slide implies $80K bought about $270K of capacity. A 3.4x return, approved before the coffee gets cold.

The Line Item That Never Shows Up

Here is what the slide does not carry: the review and verification attention the pilot consumed, priced at senior-engineer rates.

AI-assisted pull requests change shape. Diffs get bigger. Volume goes up. And the failure mode shifts from obviously-broken to plausible-but-wrong, which is the expensive kind, because plausible-but-wrong has to be caught by a human who understands the system. Suppose each engineer ships five AI-assisted PRs a week and each one takes a reviewer just twenty extra minutes. Across 12 engineers that is 20 hours a week of review time, half a full-time engineer, roughly $125K a year. And it does not land evenly: it lands on the two or three senior people qualified to catch the plausible-but-wrong category, the same scarce attention that was already the bottleneck.

Take the $270K gain, subtract $80K in tool spend and $125K in unpriced senior attention, and the triumphant 3.4x collapses to about 1.2x, before counting rework. That might still be worth doing. But nobody in the room approved that deal, because nobody saw it.

The reason nobody saw it is structural, and it is worth saying plainly: finance manages what is metered. Seats and tokens arrive as invoices. Attention arrives as nothing at all. It is smeared invisibly across a salary line that looks identical whether your seniors spent the quarter building or spent it correcting a machine's confident guesses.

The Evidence Says the Slide Flatters Itself

The 9% is usually self-reported or lightly measured, and self-report runs hot. In METR's randomized study of experienced open-source developers, participants estimated AI made them 20% faster while the measured result was 19% slower. The perception gap is the mechanism that hides the missing line item: the tool feels fast because typing is fast, while the slow part — verifying, correcting, re-prompting — doesn't register as "using the tool."

The vendor numbers run hotter still. GitHub's own study reported 55% faster task completion, a first-party figure measured on a greenfield toy task, and it should be discounted accordingly. Meanwhile DORA's research keeps finding the bill on the other side of the ledger: as AI adoption rises, software delivery stability dips. Faster generation, wobblier delivery. That wobble is rework, and rework is more senior attention, unpriced.

To be fair to the other side of the argument: METR's result is one population on one kind of task, not a universal law, and pilots built on scoped tasks with an explicit verification budget genuinely do clear the honest bar. The point is narrower and harder to dodge. A pilot that measured only what was invoiced has not measured its return. It has measured its receipts.

The Five-Line Version

The fix costs nothing but honesty. Here is the pilot P&L with all the lines on it:

AI coding pilot P&L

+ Velocity delta
    measured (cycle time,
    merged scope), never
    surveyed
- Seats + tokens
    the invoice (the only
    line the readout had)
- Review-hours delta
    PR review time before
    vs. during, x reviewer
    cost
- Rework delta
    change-failure/rollback
    rate, before vs. during
- Comprehension debt
    merged code nobody on
    the team can explain, %

The first two lines exist in every pilot. The last three exist in almost none, and they are all measurable with tooling you already run: your git host timestamps every review, your incident tracker already counts rollbacks, and the comprehension question takes one uncomfortable team meeting.

If the pilot still clears the bar with five lines on the page, expand it with confidence — mine does, and that's precisely why I keep publishing the full bill instead of the flattering half. A tool that survives honest accounting is a tool worth scaling.

Ask for the Line

The 9% on the slide is neither a lie nor a result. It is an unfinished calculation, presented with the confidence of a finished one, to a room that is being asked to multiply it by the whole org.

So ask the one question that finishes it: show me the review-hours delta. If the answer is a number, you are looking at a rare, well-run pilot, and you should probably fund it. If the answer is a pause, then the 9% was never a measurement. It was a survey wearing one, and the most expensive people in the building are quietly paying the difference.

The Bubble Popper and the Payoff Are the Same Thing

Michael Tuszynski — Thu, 16 Jul 2026 23:59:00 +0000

Cory Doctorow thinks AI is a bubble, and he argues it better than almost anyone who cheers him on. In his December speech at the University of Washington, the case runs like this: the tech giants stopped growing years ago, and a company priced as a growth stock faces catastrophe the moment the market stops believing. So they pump whatever keeps the multiple alive. Pivot to video, crypto, NFTs, Metaverse, now AI, each pitched with the same conviction, each abandoned when the next vehicle arrives. "Superintelligence" is the pitch this cycle because science fiction makes a better P/E story than enterprise software ever did.

His sharpest tool is the reverse centaur. A centaur is a human assisted by a machine: you drive the car, the car amplifies you. A reverse centaur is a human bolted onto a machine as its peripheral, hired to absorb the failures the machine can't, at machine pace, under machine supervision. Doctorow's thesis is that the money behind AI is betting on reverse centaurs: workers demoted into error-cleanup for systems sold to their bosses as replacements.

Take him seriously. The losses are real, the incentives he names are real, and I have sat through enough board-mandated AI pilots to know the reverse centaur is not a hypothetical. And then notice that his argument, stated carefully, is about two different balance sheets that the doom discourse keeps smearing into one.

Two Balance Sheets, One Shouting Match

Seller economics asks whether the labs make money. Company-level losses are enormous, but look at where they live: training runs, data-center capex, sales, and the free tier. The serving unit underneath is a different story. SemiAnalysis puts Anthropic's gross margins in the mid-60s — an independent analyst number, though The Information reported the same company projecting 40% when inference costs spiked, so treat the exact figure as contested. The direction is not. Serving paid tokens is gross-margin-positive; what bleeds is everything wrapped around it.

Even the famous counterexample proves the shape. When Sam Altman admitted OpenAI loses money on the $200 Pro plan, the reason was that heavy users outran the flat price. Sell unbounded agent workloads at a fixed monthly fee and the top of the usage tail eats you. That is a pricing-model problem, and pricing models get fixed. Physics problems don't. This one is being fixed right now, mostly at the expense of people like me: usage caps, tier splits, metered agents.

Buyer economics asks a different question: does a scoped deployment pay for itself? That is where operators live, and none of the training-run losses show up on this side of the table. I've written down what my own stack costs per month, and why "$20K in tokens against a $250K engineer" is the wrong math even when it flatters the tools. A deployment either produces measured value over measured cost or it doesn't. The lab's income statement has nothing to do with it.

Doctorow is right about the first balance sheet. The mistake is letting the first one answer for the second.

The Question That Does All the Work

Here is my actual thesis, earned from reps rather than from a valuation model: the economics work themselves out, but only for operators disciplined enough to ask "what problem am I solving" before pointing the tool at anything, and rigorous enough to measure the answer afterward.

An efficiency you don't understand is not an efficiency. It is an unmeasured cost wearing a demo. The team that deploys a coding agent with no baseline, no success metric, and no owner has not automated anything; it has hired a very fast intern nobody supervises and booked the salary as savings. That is blind deployment, and blind deployment is exactly the reverse-centaur failure: the humans end up serving the machine's output because nobody defined what the machine was for.

Which means my thesis and Doctorow's frame are the same observation read from opposite ends. Scoping rigor is what makes you a centaur instead of a reverse one. The human who decides what the machine is for stays the head. The human who cleans up after an unscoped machine becomes the peripheral.

The scoping questions I use, written down (writing them down matters, and the last section says why):

1. What is the problem, stated without naming a tool?
2. What does it cost today, in hours or dollars I can point at?
3. What does "working" look like, measured how, by whom?
4. What is the failure mode, and who catches it?
5. What is the kill criterion, decided before the pilot starts?

The Payoff and the Pin Are the Same Force

Now the part I have not seen anyone say plainly: the rigor that makes AI pay off is the same force that pops the bubble.

Walk through what happens if scoping discipline actually spreads. Every FOMO seat license gets audited against question 2, and half of them fail. Every board-mandated pilot meets question 3 and dies for lack of a metric. The all-you-can-eat experimentation budgets get metered. A real slice of today's AI revenue is hype-spend, and disciplined buyers stop paying for hype. Revenue evaporates precisely as deployments get better.

That is not bearish on AI. It is bearish on the bubble, and those are different positions. It is Doctorow's own "some bubbles leave behind something productive" scenario made concrete. He reaches for Worldcom, a fraud whose CEO died in prison while the dark fiber it buried still carries Doctorow's 2-gigabit home connection. Swap the fiber for scoped deployments quietly compounding inside businesses while the FOMO spend burns off. The economics working out and the bubble deflating are the same event, narrated by an optimist and a pessimist.

The Craftsman Clause

There is an investor-side implication buried in this that is more uncomfortable than any crash prediction.

If every deployment that pays needs a human who understands both the domain and the tool's failure modes, then AI is not the frictionless labor replacement the trillion-dollar valuations are priced on. It is a power tool that still needs a craftsman. A nail gun does not fire the carpenter; it makes the carpenter's judgment the binding constraint on every house. Attention was always the bottleneck; the tools just moved it.

Great news for buyer ROI. Quietly fatal for the "human exits the loop" story being sold upstream, because the wholesale-replacement bet only pays if the craftsman requirement goes away, and every honest deployment I have run or reviewed says it doesn't. The augmentation bet keeps cashing small checks. The replacement bet keeps pre-spending checks nobody has figured out how to write.

Two Half-Lives, One Fork

So if the scarce input is the human who can scope, the personal question is what that scarcity is worth and how long it lasts.

The scoping skill is teachable in principle and bottlenecked in practice, and the bottleneck is structural, not a fad. The skill is tacit. It lives on the seam between a domain and a tool, where few people sit. And organizations promote people for shipping, not for the scoping that made the shipping cheap; nobody's OKRs reward the pilot that didn't happen.

But the edge has two halves with opposite half-lives, and confusing them is the trap. Tool mechanics — prompt-craft, knowing this month's model's failure modes, the tricks — depreciates fast, because every model release is the labs folding exactly that knowledge into the product. The agent-loop mechanics I wrote up are already aging out from under that post. Problem formulation — translating a business's real problem out from under its stated one — depreciates slowly, because it is organizational translation, and organizations stay human. The trap is that the fast-depreciating half is the legible, demoable half. It feels like the edge because you can show it off. The durable half looks like "just asking questions."

Which leaves the fork, and I am going to leave it as a fork rather than resolve it for you. Stay the indispensable bottleneck and you extract rent: real money, now, capped at your personal throughput, gone the day the scarcity ends. Or codify the tacit half into artifacts a colleague can run without you — a written scoping method like the checklist above, reference patterns from the deployments that worked — and you dissolve your own bottleneck while capturing the value of having systematized it. Rent pays this quarter. The moat is owning the codified version when everyone else finally needs it.

The bubble being real is what creates the opening; nobody pays a premium for discipline in a calm market. The economics work out for whoever is on the right side of the rent-to-moat conversion. And the thing to be dismantling, deliberately, starting now, is the bottleneck being you personally — before the labs dismantle the cheap half of your edge for free and leave you holding only the part you never bothered to write down.

Torvalds' Best Review Trick Just Stopped Working

Michael Tuszynski — Thu, 16 Jul 2026 18:11:41 +0000

Linus Torvalds has spent years reviewing the most consequential codebase on Earth without reading much of the code. He said so himself in a 2020 interview with Tag1 Consulting: "While I still look at patches, I actually tend to look more at the explanations, and the history of how the patch came to me."

That habit was never laziness. It was the sharpest quality filter in software, and it rested on an economic fact: a coherent explanation of why a change belongs in the kernel was expensive to produce. You had to understand the subsystem, the failure mode, and the design decisions that shaped the current code. Faking the explanation cost more than having the understanding. So the explanation worked as proof of understanding, and the man at the top of the review pyramid could read intent instead of implementation.

At Open Source Summit India 2026 in Mumbai this month, Torvalds described the thing that broke that filter. He never framed it as broken. The quotes do it for him.

The Explanation Was Proof of Work

Tests can be gamed. Diffs can be pattern-matched from similar fixes without understanding either one. But for decades, a paragraph that correctly situated a change in the design history of a kernel subsystem could not be written by someone who lacked the mental model. Reviewers up and down the kernel hierarchy leaned on that correlation. Torvalds built his entire post-programming job on it.

The correlation was always a proxy, though. Nobody cares about the explanation itself. They care about the understanding it demonstrates, and Goodhart's law says any proxy works right up until it becomes a target. A proxy survives on the cost of faking it. This one survived for thirty years because the fake was more work than the real thing.

LLMs Made the Proxy Free

Large language models generate code that is sometimes wrong. They generate prose about code that is nearly always fluent. The explanation, the one artifact that used to be hardest to fake, is now the cheapest part of the submission.

Torvalds described what that looks like from the receiving end: bug reports that read as entirely valid and turn out to be fabricated. "It can actually be a huge drain on resources when it takes humans a lot of effort to figure out that, hey, this machine-generated report was not true," he said. Note the asymmetry: the report costs nothing to generate and real investigative work to disprove. That is a filter running backwards, taxing the reviewers it used to protect.

The patches carry the same signature. He called many of them "mindless band-aid kind of patches... they may fix the immediate problem, but the kind of bug remains, and it just is waiting in the hallway to hit you in another place." A band-aid patch in 2019 usually arrived with a thin, awkward description, and the description gave it away. The same patch in 2026 arrives wrapped in a confident, well-structured explanation of a root cause the author never found. The tell is gone. Prose quality no longer predicts anything about the understanding behind it.

The Fix He Announced Is a Provenance Requirement

Buried in the Mumbai remarks is the actual news, and it got almost no coverage: "If you find a bug with an LLM, it's not enough to just ask the LLM to make a bug report and then throw it over the fence to us. We want to see a suggested patch; we want to see the human who ran the LLM act as a kind of back-and-forth."

Read that as a submission requirement, not a complaint. He is no longer asking the explanation to prove anything. He is asking for evidence of the process: show me the dialogue, show me where you pushed back, show me what the model got wrong before you fixed it. The unit of review is shifting from the artifact to its provenance. For decades the kernel asked whether the description demonstrated understanding. The new question is whether the human can show their working relationship with the tool that produced it.

That is a bigger change than it sounds. The transcript of the back-and-forth, not the polished description sitting on top of it, is becoming the thing a maintainer actually wants to see.

Nobody's Tooling Captures This Yet

PR templates ask what changed and why. CI gates check tests, coverage, lint. None of them ask the question Torvalds is now asking: how do you know? The description field is where the fluent fake lives. The evidence of a genuine back-and-forth lives in session logs most teams throw away.

Closing that gap does not need a platform. A template section does most of it:

## Provenance
- Generated with: <tool/model, or "by hand">
- First-pass mistake I caught: <what you pushed back on>
- What I checked myself: <command you ran, output you read>

The middle field is the tell. A developer who worked the problem with a model fills it in ten seconds, because every real session has one. A developer who piped output over the fence has nothing to put there, and an empty field is a louder signal than a beautiful description.

I started keeping this kind of evidence before I had a name for it. When I published a self-improving agent demo earlier this month, the repo shipped with the actual run logs, a 13-of-15 test pass climbing to 15-of-15 across iterations, because without the logs the claim is indistinguishable from every other AI demo on the internet. The logs are the proof the loop ran. The README is just the description.

Where This Breaks

The obvious objection: transcripts can be faked too. Ask the model that wrote your patch to also write a plausible back-and-forth and it will produce one, complete with staged pushback. If teams start grading transcripts, Goodhart eats the transcript next. Grading them with an LLM judge inherits the same problem one level up.

The kernel's answer is already visible, and it is not paperwork. Torvalds has been blunt that the AI slop problem will not be solved with documentation, and kernel maintainers already deprioritize drive-by reports whose submitters won't answer questions. The gate is not the artifact. The gate is whether you can continue the conversation. A fabricated transcript buys exactly one round; the first follow-up question a maintainer asks lands on the human, in real time, with no model in the loop to sound fluent for them.

So Torvalds' test did not die in Mumbai. It moved. Reading the explanation stopped working, so he replaced it with watching you respond.

The next time a pull request lands in your queue with a suspiciously fluent description, run the kernel's version of the test. Ask one follow-up question the description does not answer, and watch the clock. The answer that comes back in five minutes was always the author's. The one that takes a day was written by whatever wrote the description.

Attention Is the New Bottleneck. Engineer It Like One.

Michael Tuszynski — Tue, 14 Jul 2026 16:43:39 +0000

The Diagnosis Is Right

Atomic Object published a piece last month arguing that with AI agents, attention is the new bottleneck. Agents made execution cheap. You can fire off four parallel coding agents before your coffee cools. What you can't do is review four streams of output at once, hold the context of each in your head, and catch the one that quietly wrote a migration that drops a column.

The diagnosis is correct. Simon Willison described running four parallel agents and being wiped out by 11am — not because the tools failed, but because he became the rate limiter. Michael Novati made the same point from a different angle: AI removed the production bottleneck and revealed the real one underneath — the human system that surrounds production. Everyone circling this problem is seeing the same thing. Execution went to zero and human attention became the scarce resource.

So the diagnosis holds. The prescription is where it falls apart.

The "Type Faster" Answer

Atomic Object's advice is all personal discipline. Automate trivial decisions. Plan for cognitive load. Take deliberate breaks. Budget your attention like a finite resource. Good advice for a person. Useless as a system.

We've seen this exact move before. When execution was the bottleneck — when developers spent their day typing, compiling, catching their own syntax errors — nobody's answer was "type faster." Nobody wrote a productivity blog telling engineers to be more disciplined about their keystrokes. We built compilers so you didn't hand-check types. We built linters so nobody argued about a missing semicolon. We built CI so the machine ran the test suite at 2am and told you which commit broke it.

The bottleneck moved and we answered with infrastructure, not willpower. Telling a developer in 1998 to concentrate harder on avoiding null-pointer bugs would have been absurd. Telling a developer in 2026 to budget their attention better is the same absurdity wearing new clothes.

Discipline doesn't scale. Infrastructure does. If your plan for the attention bottleneck is "I'll be more focused," you've already lost, because the failure mode of human attention isn't insufficient effort — it's that there's a finite amount of it and agents produce work faster than any amount of focus can absorb.

Engineer the Bottleneck Instead

Here's the reframe. Attention is a resource your system spends on your behalf. Most systems spend it wastefully — they make a human look at everything. The job is to build the parts that spend it carefully, the same way a good compiler spends your debugging time carefully by pointing at line 47 instead of making you read the whole file.

Four pieces do most of the work.

Verification gates so humans review exceptions, not everything. The default agent workflow asks you to eyeball every output. That's the waste. Instead, have the agent verify its own work and only surface what it can't confirm. My content pipeline ships blog posts across five channels. After it publishes, a verifyPublishedPost() step re-fetches the committed markdown and checks for the two failures that actually happened to me — a missing feature image, and a leading H1 that double-rendered the title. If it finds one, it fires a 🚨 line into my notification channel. If everything's clean, I hear nothing. I went from reading every published post to reading only the broken ones. That's not discipline. That's a gate.

Structured escalation so one line beats a log stream. Uptime monitoring is the canonical trap. You can watch a dashboard, or you can read logs, or you can have every alert route to a single channel with a single flagged line: what broke, where, since when. I don't read my nightly job logs. They write to disk. If a job fails, one message arrives. The difference between "check the logs" and "here is the one thing that needs you" is the difference between spending attention and saving it.

Trust tiers per task class. Not all work deserves the same scrutiny, and pretending it does is how you burn attention on things that don't need it. Low-stakes work ships on green checks — my content pipeline's Tier 1 curated shares go out on a passing lint gate, no human in the loop. High-stakes work queues for review. The on-demand blog publisher treats my explicit request as the approval and ships without a gate, because I asked for it. The cron pipeline that drafts on its own routes through an approval message first. Same infrastructure, different trust tier, matched to the blast radius of being wrong.

Decision logs so the same call never reaches a human twice. This is the one people skip, and it's the one that compounds. Every time you make a judgment call for an agent, write it down where the agent reads it. My project instructions carry a running list of hard-won lessons — never hand-author markdown into the blog repo, opt into Instagram explicitly because the publishing API returns false-success statuses, strip any leading body H1 unconditionally. Each of those is a decision I made exactly once. The agent now makes it every time without me. A decision log is a cache for judgment. Without it, you re-answer the same question forever, which is the most expensive way there is to spend attention.

The Objection Worth Taking Seriously

The honest counterargument: infrastructure has a build cost, and discipline is free today. Writing a self-verification step, wiring escalation, defining trust tiers — that's real work, and for a solo developer running one agent occasionally, personal focus genuinely is cheaper. Atomic Object isn't wrong that a human should also automate their own trivial decisions and take breaks.

But that's a bet on the problem staying small, and it won't. The whole premise is that agents multiply output. The developer running one agent this month runs six next quarter. Discipline that works at one stream collapses at six — that's the entire bottleneck being described. You pay the infrastructure cost once and it holds as you scale. You pay the discipline cost every single day, and it fails exactly when the load gets heavy enough to matter. Build the gate before you need it, because the moment you need it you won't have attention left to build it.

What Actually Separates the Winners

The framing that treats attention as a personal-productivity problem is going to age like every other "work smarter" answer to a structural constraint. It puts the burden on the operator to be superhuman, when the whole point of the last forty years of tooling was to stop requiring humans to be superhuman.

The people who win with agents won't be the ones with the most discipline about their attention. Discipline is a fixed, small, human quantity, and the workload is about to be neither fixed nor small. The winners will be the ones whose systems spend their attention the least — whose agents verify themselves, escalate the one thing that matters, ship low-stakes work without asking, and never bring the same decision back a second time.

Attention is the new bottleneck. So engineer it like one. We didn't beat the execution bottleneck by typing faster, and we won't beat this one by concentrating harder.

Try It: A Working Assessment-First Course

Michael Tuszynski — Mon, 13 Jul 2026 11:01:23 +0000

Eight posts ago the claim was that the AI-education industry is building the wrong product — chatbots students ignore, while the thing that actually moves exam scores is an LLM grading written answers against a rubric, wrapped in spaced cumulative review. Now there's a running system to argue with instead of a claim to nod at. This is the capstone of the assessment-first series: what got built, how to run it, and where the bet breaks.

Run it in five minutes

doerkit is a full course — six statistics lessons from OpenStax OER, quizzes, cumulative review, a dosage dashboard:

git clone https://github.com/michaeltuszynski/doerkit && cd doerkit
npm install
export ANTHROPIC_API_KEY=sk-ant-...
npm run dev          # http://localhost:8734

Pick a name, read a lesson, take its quiz. Write a real answer to a constructed-response question and watch it get graded against the rubric with feedback in about a second; write "the median because reasons" and watch it get partial credit with a specific note on what's missing. Fail the 90% review bar, get nudged to come back tomorrow instead of cramming. Open /dashboard and see your own dosage. The grader is regression-tested by the sibling repo, including against the prompt-injection answers a real student would try.

That's the whole thesis, executable. The LLM never chats, never does the student's work, never assigns a grade directly — it judges rubric criteria as booleans and code computes the rest.

What eight posts actually shipped

Two repositories, both MIT, both green in CI, both tagged v1.0:

rubric-bench — regression testing for any LLM judge. Golden sets, run scoring, drift diffs, an adversarial suite, tone metrics. The general-purpose one; useful well beyond education.
doerkit — the platform: grading engine, lessons, mixed-format quizzes, interleaved spaced review, telemetry.

The findings that surprised me, collected: a frontier model shrugged off first-generation prompt injections that a cheaper model fell for, so grader security lives in the model-prompt pair and moves when you swap either. Grader severity and grader warmth are separable knobs: you can be kind without inflating grades, which means a cold grader is a defect, not rigor. And the boring cumulative-review feature carried the biggest effect size in the source study, beating both the AI grader and the chatbot everyone demos.

What a real deployment would still need

The honest gap between "runs on my laptop" and "runs a gateway course," so nobody mistakes this for the second thing:

LMS integration: LTI 1.3, roster sync, gradebook. Unglamorous, mandatory, and deliberately absent here.
Auth and multi-tenancy: the demo trusts a self-typed name. A real one needs SSO, real accounts, and per-institution isolation.
A FERPA data agreement: the moment student-keyed telemetry leaves a laptop it's regulated education data, with all the procurement that implies.
Human-rater validation: post 3 regression-tests grading consistency, not agreement with instructors. A pilot needs an inter-rater study against real graded work.
An RCT: everything here rests on one observational pilot at one selective school. The design is a hypothesis with strong priors, not proof.

None of these are hard research problems. They're the difference between a portfolio and a product, and pretending otherwise is how edtech demos oversell.

Where the whole bet breaks

The strongest counterargument to this series is selection. The students who complete more lessons and pass all three reviews are the ones who were going to ace the final anyway; the Dartmouth data brackets the effect between 0.71 SD (over-adjusted) and 1.30 SD (selection-inflated) precisely because it can't fully separate the platform from the motivation. I believe the effect is real and meaningful — the cross-format contrast, where constructed-response dosage tracked scores and multiple-choice didn't within the same students, is hard to explain by motivation alone, but "real and meaningful" is a defensible position, not a settled one. Anyone who tells you AI tutoring has proven 1.3-SD gains is selling.

And there's a tension the series surfaced without resolving: disabling constructed response in the pilot raised completion rates, because writing answers is more work than clicking. The highest-efficacy format may carry an engagement tax. The whole bet is that the tax is worth paying and that better grader tone shrinks it, but that's the open question a real study exists to answer, not one this code settles.

The actual takeaway

If you build one thing from these eight posts, don't make it an education product. Make it the eval suite. Every team putting an LLM judge into production — grading, triage, moderation, ranking — has the exact problem post 3 solved and mostly doesn't know it yet: their judge's behavior is an untested production dependency that changes when the model updates. Golden sets, drift diffs, adversarial cases, tone guards. That pattern outlives statistics, outlives edtech, and outlives whatever model you're calling this quarter.

The chatbot got two years of the industry's attention. The quiz engine moved the exam scores. Both repos are public, both are yours to fork, and the code is the argument.

Where Your $20K in Tokens Actually Goes

Michael Tuszynski — Mon, 13 Jul 2026 02:03:54 +0000

The last piece argued that comparing a $250K engineer to a $20K token bill is a trap, because the two numbers measure different things and the cheap one hides its real cost. Fine. But "the token bill is not the whole story" leaves a follow-up hanging: what is actually in that $20K? Where does the money go once the invoice clears?

It goes to waste, mostly. Not fraud, not overpriced models. Ordinary, invisible waste that nobody instruments because the bill arrives as one number and one number tells you nothing. A $20K monthly spend on production AI is not a cost. It is a pipeline with leaks at four specific joints, and every one of them is measurable.

The bill is not one line, it's four leaks

Start with the model. On Claude, Opus 4.8 runs $15 per million input tokens and $75 per million output, and Sonnet lands at roughly a fifth of that. Those are the posted numbers, and they are the part of the bill you can't argue with. What you can argue with is how many tokens you send, how many times you send them, and how many of those sends did no useful work.

Break a real production bill apart and the same four categories show up every time: retries and stalls, context and prompt bloat, tool-schema overhead from MCP servers, and redundant eval or judge passes. Each one has a fix. None of the fixes require a better model.

Leak one: retries and stalls

An agent loop that hits a rate limit, a malformed tool call, or a timeout doesn't fail cleanly. It retries. And the retry re-sends the entire context that got you to the failure point, so a stall at turn nine costs you turns one through nine again, in full, at output prices if the model already started generating.

This is the quiet killer in agentic workloads. A single user request that should cost one round trip can spawn six because the agent got confused, called a tool wrong, read the error, and tried again. In production agent loops, error-recovery and re-planning routinely consume more tokens than the productive reasoning does. The model isn't thinking harder. It's redoing work.

The fix is boring and it works: cap retries explicitly, log every retry with the reason, and treat a high retry rate as a bug in your tool definitions, not a cost of doing business. If your agent retries a tool 30% of the time, the tool's description is unclear or its schema is wrong. Fix the schema and the retries disappear. Instrument this first, because it's usually the biggest single line and the one teams never look at.

Leak two: context and prompt bloat

Every request carries a system prompt, a set of instructions, skill definitions, and whatever conversation history you've accumulated. Most teams have no idea how big that payload is. They wrote the system prompt eight months ago, bolted three more skills onto it, and never measured the total.

Measure it. A tool like token-baseline run across your prompt, skill, and command corpus gives you a real number per component, so you can see that the 4,000-token "helpful preamble" nobody has read since launch is riding along on every single call. At Opus input rates, 4,000 wasted tokens on 100,000 daily calls is real money, and it buys you nothing.

The structural fix is prompt caching. Anthropic's prompt caching charges a 25% premium to write a cache entry and then serves cache reads at 10% of the base input price. If your system prompt is stable across calls, and it should be, caching turns a repeated 4,000-token tax into a one-time write plus pennies per read. Teams that cache their stable prefix routinely cut input spend by half or more. Teams that don't are paying full freight to re-send the same unchanging text thousands of times a day.

Leak three: MCP tool-schema overhead

This one is new enough that most teams haven't caught it yet. Connect an agent to a handful of MCP servers and every request now carries the full JSON schema for every tool those servers expose, before the model reads a word of the actual task. GitHub's server alone ships around 35 tools. Stack a few servers and you can burn 50,000-plus tokens on schemas the model will not use on this particular request.

You can audit this directly. A tool like mcp-token-audit measures the token cost of each connected server's schema payload, and the results are usually ugly. Half your context window can be tool definitions for capabilities the current task doesn't touch.

The fix is on-demand tool loading: expose tool names to the model, and load the full schema only when the model decides to call the tool. I wrote about the mechanics of this separately, but the short version is that you should never pay to describe a tool the request won't use. Audit your schema payload, then defer everything you can.

Leak four: redundant eval and judge passes

The last leak comes from good intentions. You added an LLM judge to grade outputs, then an eval pass to check the judge, then a second judge for confidence. Now every production response triggers three extra model calls, and two of them are asking nearly the same question.

Judge calls are output-heavy and they compound. If your judge re-reads the full input plus the candidate answer plus a rubric on every call, you're paying to re-process the same context three times to answer one quality question. Sample instead of grading everything. Grade 5% of production traffic continuously and the full set only when you ship a prompt change. Collapse redundant judges into one call with a structured multi-field output. The goal is confidence in your quality, not a receipt for every token you can spend proving it.

Instrument the pipeline, then fix the joints

Here's the position: stop treating the token bill as a price and start treating it as telemetry. The number on the invoice is the sum of four measurable subsystems, and every one of them leaks in a way you can see the moment you point a tool at it.

Run token-baseline against your prompt and skill corpus this week. Run mcp-token-audit against your connected servers. Turn on prompt caching for your stable prefix. Put a counter on retries and set an alarm when the rate climbs. None of this is exotic, and none of it needs budget approval. It needs someone to accept that "$20K in tokens" is a diagnosis waiting to happen, not a fact to shrug at.

The trap in the first piece was believing the token number was small. The trap in this one is believing it's a single number at all. It isn't. It's a pipeline. Instrument it, and half of it turns out to be leak.

Instrument Like a Learning Scientist

Michael Tuszynski — Sun, 12 Jul 2026 11:01:42 +0000

The most valuable thing the Dartmouth team built wasn't the grader. It was the fact that they could answer "did completing this lesson's quiz correlate with doing better on the exam?" — per module, per format. That question is why they discovered multiple-choice quizzing produced no measurable learning while constructed-response did. Without per-lesson dosage logged against exam outcomes, that finding is invisible, and the platform ships the useless format forever because everyone felt engaged.

This is post 7 of the assessment-first series. It's about the least glamorous and most compounding part of doerkit: the telemetry that lets the platform measure itself.

Dosage is the variable that matters

Most edtech analytics report engagement — logins, time-on-page, questions attempted. Those are vanity metrics; they measure whether people showed up, not whether showing up did anything. The variable the Dartmouth study built its whole argument on is dosage: how many lessons a student actually completed, regressed against exam performance. The distinction is the entire finding: engagement was comparable-or-higher under multiple-choice, but dosage only tracked exam scores under constructed response. If you log engagement you learn nothing; if you log dosage you learn which features work.

So doerkit logs two things from day one: an append-only events table (lesson views, quiz starts, submissions with score and pass/fail) and an attempts table (every quiz and review attempt with its score). Both carry a student key and a timestamp. That's the minimal schema, and it's enough to reconstruct dosage-versus-outcome for any cohort you later attach exam scores to.

CREATE TABLE attempts (
  id INTEGER PRIMARY KEY, student TEXT, kind TEXT,   -- 'lesson' | 'review'
  lesson_id TEXT, score REAL, passed INTEGER, created_at TEXT
);

The dashboard is the instrument

The dosage dashboard rolls this up per student: lessons passed (the dosage number), quiz attempts, average score, reviews passed versus attempted, total events. It's a plain SQL rollup rendered as an HTML table, with no charting library and no analytics vendor. The point isn't the visualization; it's that the raw material for an efficacy analysis exists the moment the first student touches the platform, instead of being a data-collection project you scramble to start after someone asks whether the thing works.

That framing matters for what this platform is for. Efficacy evidence is the currency of institutional edtech sales and the thing every rigorous claim in this space is missing. A platform instrumented for dosage-outcome analysis generates its own evidence base as a byproduct of being used — every cohort makes the next efficacy claim stronger. The data asset compounds; the code doesn't. That's the actual moat in this category, and it costs two database tables to start accruing.

The minimal event schema for any learning product

If you're building anything with practice and outcomes, log these from commit one, before you think you need them:

The dose — the countable unit of work (lessons completed, problems solved), per user, timestamped. Not time-on-page.
The verdict — pass/fail and score on each attempt, so you can separate "attempted a lot" from "attempted well."
The retry structure — every attempt, not just the last, with its timestamp. The ~1.5-day spacing finding only existed because retries were individually logged.
A stable subject key — so you can join to outcomes later without re-identifying anyone.

Retrofitting this after launch means the first cohort is unmeasurable, and the first cohort is exactly the one a skeptical instructor asks about. Instrument before you need it, because the need arrives as a question you can't answer retroactively.

Where this breaks

Dosage-outcome correlation is not causation, and this is the load-bearing caveat for the whole series: motivated students both complete more lessons and score higher, so raw dosage regressions are selection-inflated. The Dartmouth authors handled it by controlling for prior midterm performance, which brackets the true effect between an over-adjusted 0.71 SD and a selection-inflated 1.30 SD, and doerkit's telemetry can produce the same bracketing only if you feed it exam scores, which it doesn't collect on its own. There's a privacy surface too: a student-keyed event log is FERPA-relevant data the moment this leaves a laptop, so the demo uses a self-chosen name and no real roster, and a genuine deployment needs a data agreement this scope deliberately avoids. Telemetry that measures learning is also telemetry that surveils learners; build it, and own that both are true.

Next post is the capstone: run the whole thing yourself, what an actual institutional deployment would still need, and an honest accounting of where the assessment-first bet holds and where it doesn't.

The Biggest Effect Size Was the Boring Feature

Michael Tuszynski — Sat, 11 Jul 2026 11:01:35 +0000

The feature with the largest effect size in the Dartmouth pilot is the one no startup would put on a landing page. Not the AI grader. Not the chatbot. Cumulative module reviews: a big quiz covering every lesson, questions interleaved across topics, a 90% bar, unlimited retries. Students who passed all three scored 7.1 points higher on the final (d = 0.66), the strongest signal in the study. It looks like a quiz from 2005.

This is post 6 of the assessment-first series, and it assembles the pieces from posts 2 through 5 into a running web app — doerkit. Lessons, quizzes, and that unglamorous review engine, which is the part worth dwelling on.

Why interleaving beats blocked practice

A lesson quiz tests one topic while it's fresh — you just read about the median, then answer about the median. That's "blocked" practice, and it produces a specific illusion: fluency inside the current topic that evaporates when topics are mixed. The student feels like they know it because the context is doing half the retrieval.

A cumulative review mixes topics. Question 3 is about outliers, question 4 is about z-scores, question 5 is back to sampling. Now each question forces the harder move (which concept does this even call for) before you can answer it. Interleaved retrieval practice is one of the most replicated findings in learning science, and it consistently loses on the in-session feeling of mastery while winning on the exam weeks later. That gap between how it feels and how it works is exactly why it doesn't sell, and exactly why it's the feature that moved scores.

doerkit's review builds a 10-question quiz by taking at most two questions per lesson across the whole module, so no student sees a topic-blocked run:

export function pickReviewQuiz(banks: QuestionBank[]): QuizQuestion[] {
  const perLesson = banks.map((b) => pick(b.questions, 2));
  return pick(perLesson.flat(), REVIEW_QUIZ_SIZE); // interleaved by construction
}

Spacing, and the nudge

The other half of the effect was timing. The Dartmouth retry logs showed students returning to reviews a median of ~1.5 days apart, spaced retrieval rather than cramming, and that spacing wasn't designed, it emerged from a high pass bar plus unlimited retries. A 90% threshold you'll rarely clear on the first try, with no penalty for coming back, quietly manufactures the exact study schedule the research recommends.

doerkit makes the nudge explicit. Retry within 20 hours of your last review attempt and it says so: "Retrieval sticks better with a gap; coming back in about 14h beats retrying now (you can retry anyway)." It never blocks the retry. The whole design principle from post 1 holds: the platform succeeds only when students choose to come back, so it persuades rather than gates. Manufacturing a good habit out of a pass threshold and a soft nudge is cheaper and more durable than any streak mechanic.

The stack, deliberately small

The whole app is Hono plus SQLite, server-rendered HTML, no client framework. Lessons are markdown authored from OpenStax OER. Quizzes draw from per-lesson banks of ten. The grading engine from post 2 grades written answers concurrently; multiple choice is a pure function. Content is never gated — you can read any lesson and take any quiz in any order, because the Dartmouth platform wasn't gated and hit 90% voluntary adoption.

What's deliberately absent is the tell. No LMS integration, no SSO, no multi-tenancy, no student roster, no RAG chatbot. Those are the features that make edtech an 18-month enterprise sale, and none of them touch the thing that moves exam scores. The scope is frozen to exactly what the evidence supports: read, write an answer, get judged, come back spaced. A pilot instructor can run this on a laptop the afternoon they find it.

Applying the pattern past statistics

The review engine is subject-agnostic. Any course with lessons and question banks gets interleaving for free — the scheduler just needs topics to mix and a pass bar high enough to invite a second visit. Language vocabulary, medical board prep, onboarding curricula: the same two knobs (interleave across units, set a threshold that manufactures spacing) transfer directly. The AI grader is what makes the written half economical; the review engine is what makes any of it stick. You need both, and only one of them is exciting.

Where this breaks

The spacing nudge is honest but toothless by design; a determined crammer ignores it, and without course credit attached (the Dartmouth quizzes were ungraded) the population that most needs spacing is the least likely to self-impose it. The interleaving is within-module only; true long-horizon spacing across a whole term needs scheduling state this version doesn't keep. And the biggest honest caveat carries over from the study itself: the all-reviews-passed group was also the most self-selected, so some of that d = 0.66 is motivated students being motivated. The within-module comparison (passing the review predicted a 6.1-point midterm gain holding cohort fixed) is the cleaner evidence, and it's smaller. Real, but smaller.

Next post — the last one — is the capstone: run it yourself, what a real institutional deployment would still need, and where this whole assessment-first bet does and doesn't hold.

The Grader Was Right and the Students Quit Anyway

Michael Tuszynski — Fri, 10 Jul 2026 15:34:31 +0000

The most dangerous failure in the Dartmouth Phosphor pilot wasn't a wrong grade. Students found the constructed-response grader "rigid and discouraging," complained loudly enough that the team removed those questions from an entire module. And the module without them turned out to produce no measurable learning. The feature that worked got pulled because of how it felt. Accuracy survived contact with students; tone didn't.

This is post 5 of the assessment-first series, and it treats grader tone the way post 3 treated grader accuracy: as a measurable property under regression test, not a vibe you hope survives the next prompt edit.

Severity and tone are different knobs

The intuition says a "strict" grader gives lower grades and a "warm" grader inflates them — that you buy kindness with rigor. I tested that. Same 72-case golden set, same model (claude-sonnet-5), two system prompts: the default from post 2 ("precise and warm: a good TA, not a gatekeeper... name what the answer got right first") and a strict-examiner variant ("do not give benefit of the doubt... do not praise, do not soften").

Verdicts barely moved. Expected-partial answers graded down to incorrect: 3 of 24 under both prompts, the same three cases. Overall accuracy within one case (69 vs 68 of 72). The persona change did not make the grading harsher.

The experience changed completely. Across the 60 genuine cases:

metric	default	strict
feedback opening with something the student got right	54/60	38/60
feedback containing scolding phrases ("fails to", "unfortunately"...)	0/60	5/60
average feedback length	37.8 words	32.8 words

Same case, both prompts. Default: "You correctly state that the mean exceeds the median, but you don't describe the shape of the distribution... so the reasoning credit isn't earned." Strict: "No description of the distribution shape is given... only vaguely references 'mean follows the skew,' which does not meet the required attribution." Identical verdict, identical partial credit. One reads like a TA who wants you to pass; the other reads like a rejection letter. A student on their third retry at 11pm reads the second one and closes the tab.

That separability is the finding: warmth is nearly free. You don't pay for it with grade inflation; the rubric-and-booleans architecture keeps verdicts anchored while the prose register moves independently. Which means shipping a cold grader isn't rigor. It's just a product defect you haven't measured.

Making tone a regression test

The measurement is deliberately crude, because crude and automated beats sophisticated and manual. rubric-bench golden cases now take a feedbackForbidden list alongside feedbackMustMention (terms that must never appear in feedback, like "unfortunately" or "you failed"), and every run keeps the full feedback text per case, so the tone analysis script can report positive-acknowledgment rates and scold counts across whole runs. A prompt edit that keeps accuracy but drops the positive-opener rate from 90% to 60% now fails visibly, in CI, before a student sees it.

Regex against feedback text is a blunt instrument and I'm comfortable with that. The alternative, an LLM judging the tone of an LLM's feedback, is a real technique, but it puts a second nondeterministic judge in your test suite, and you'd need a bench for the bench. Start with substring guards on the phrases you never want students to read; graduate to a tone judge only when the blunt version stops catching real regressions.

The product frame

Formative assessment lives or dies on retry behavior. The Dartmouth data's strongest feature was students returning to cumulative reviews a median of ~1.5 days apart — voluntary spaced retrieval, the thing decades of learning science says to maximize. Every piece of that loop runs on the student choosing to come back, and the feedback message is the last thing they read before choosing. This is why "the grader was accurate" and "the grading feature failed" can both be true: accuracy is a property of verdicts, retention is a property of the loop, and tone is the hinge between them.

There's a business asymmetry here too. A too-lenient grader fails quietly and gets caught by the adversarial suite from post 4. A too-cold grader fails loudly: screenshots, complaints, an instructor pulling the feature mid-term. The Dartmouth team's response (rip out constructed response, discover the replacement taught nothing) is what unmeasured tone costs.

Where this breaks

The metrics are proxies, and proxies saturate: a grader could open every message with a hollow "Good effort!" and score perfectly on positive-acknowledgment while being useless. The forbidden-terms list is English-specific and enumerable, and a genuinely different feedback register (a new model's house style) could pass every guard while feeling off in ways only students will tell you. Verdict-tone separability held on this model pair and this rubric architecture; a judge that assigns scores directly (no boolean criteria) would likely see verdicts drift with persona. And 60 cases of feedback is a tone sample, not a study — the real instrument is a mid-term student survey sitting next to the bench numbers.

Next post: the platform itself. Lessons, quizzes, and the boring cumulative-review feature that carried the biggest effect size in the study — assembled into a runnable web app.