DEV Community: Michael Tuszynski

An Ontology Your AI Generated Is a Guess With a Schema

Michael Tuszynski — Wed, 28 Oct 2026 18:04:01 +0000

An agent asked for net revenue by sales rep before the 2025 territory change can do one of two things. A model writing raw SQL finds a reps table, an invoices table and a join key, and returns a number. A semantic layer with no definition for pre-change territories returns an error. The number looks better in a demo. The error is the one you want behind a board deck.

That difference is most of what remains of the case for the enterprise ontology. It holds only if a named person decided the definitions on a known date. The 2026 wave of products that generate those definitions automatically puts the guess back in, now carrying the word "ontology." This post argues for budgeting the agreement, not the graph.

The word got popular after its best argument got weaker

The original pitch for ontologies in AI was accuracy, and it arrived with a strong number. In 2023, Sequeda and colleagues at data.world, then a seller of a knowledge-graph data catalog, tested GPT-4 zero-shot on an insurance-domain enterprise schema. Over the raw SQL schema it averaged 16.7% execution accuracy. Over a knowledge graph built from an ontology plus mappings, it reached 54.2%, a little more than 3x. On high-schema-complexity questions, SQL accuracy was 0%. The paper's own conclusion recommends buying a data catalog with a knowledge-graph architecture, so treat those figures as a vendor's benchmark too.

The models have improved since. In April, dbt Labs reran the benchmark on the same ACME Insurance dataset: 11 questions, each run 20 times. In dbt's runs, text-to-SQL went from 32.7% with GPT-4 in November 2023 to 64.5% with Claude Sonnet 4.6. The semantic layer scored 72.7%. After an LLM added three dbt models, text-to-SQL hit 90.0% and the semantic layer 98.2%.

Both configurations show the same 8.2-point gap. As a ratio, that's about 1.13x and then 1.09x. Don't set those beside the 2023 3x as one trend line: that figure compared a knowledge graph with raw SQL in data.world's setup, while dbt compared its semantic layer with text-to-SQL in its own. What dbt's rerun does show is a single-digit gap between a defined layer and a current model. Discount even that. dbt sells the semantic layer, and 11 questions on one schema is a small sample. To get text-to-SQL working, dbt loaded the entire schema into context, which it says isn't practical for larger datasets. On a real warehouse the gap could widen again. The evidence is suggestive, not conclusive. Evidence this thin can't carry a platform purchase on accuracy alone. Forrester, for one, has adopted the language anyway. In an analyst blog post it now calls the context layer "the next evolution of semantic layers and knowledge graphs," built "through graph-based ontologies and linked context."

What survived is how each approach fails

The dbt post says the important part itself: "With text-to-SQL, failure looks like a plausible but incorrect answer. With the Semantic Layer, failure looks like an error message." And more bluntly: "Text-to-SQL will cheerfully give you a wrong number."

At 90% accuracy, the remaining misses arrive formatted exactly like the hits. They come with two decimal places, a chart and a confident sentence. The case here is that a number that looks right invites fewer checks than an error does. The failure produces output that passes for success, so nothing prompts a check. An error message gets noticed the first time it appears.

So the durable value of a defined layer is its boundary. Inside the definitions it answers. Outside them it refuses.

A refusal borrows its credibility from whoever wrote the definitions

The boundary cuts both ways. A layer that refuses out-of-scope questions answers in-scope ones with full confidence. If the definition inside the boundary is wrong, the layer returns a wrong number with the authority of a system known to say "can't answer." If users learn that the layer refuses when it doesn't know, they'll reasonably read every answer it does give as checked. That plausibly makes a wrong definition more dangerous there than in text-to-SQL, where users have less reason to drop their skepticism.

That makes the definitions the load-bearing part, and the 2026 products are automating exactly that part.

The new tools generate the definitions they used to require

The vendors deserve a fair reading. Microsoft's Fabric IQ ontology, in preview, can generate an ontology from Power BI semantic models, and "changes follow a proposal-first model so users can review them before applying them." A rule there is "a natural-language statement of business logic linked to ontology concepts." Snowflake's Semantic View Autopilot uses query history accessible to the creating role "to generate both relationships and verified query suggestions," which you can review and apply. Databricks' Genie Ontology, also in preview, gives each snippet "an authority score based on where it was generated from, how often it is used, and how fresh it is," then "ranks the most relevant snippets, resolves conflicts, and answers."

Review buttons exist. The risk is in what the review checks. In Databricks' score, usage frequency and freshness measure popularity, and source measures provenance. None of the three records who agreed to a definition. Snowflake's suggestions start from queries people already ran. The most-used definition of "active customer" can be the one finance stopped accepting last quarter, because old dashboards still run it every hour. A suggestion mined from query history encodes what people ran, wrong queries included.

Read "resolves conflicts" twice. Two competing definitions of a term amount to a business decision, and here a ranking function settles it at question time. TopQuadrant, which sells formally governed ontologies and so has a stake in this point, puts it most sharply: an inferred definition "can tell you how a term is currently used. It cannot tell you what anyone agreed."

Voltaire wrote the logic down in 1759. In Candide, Pangloss proves "that things cannot be otherwise than as they are": the nose was "formed to bear spectacles," and so "we have spectacles." An ontology mined from usage reasons the same way. The definition exists and the dashboards run it, so it must be the right one. Rank by popularity and freshness, and whatever wins is, by construction, the best of all possible definitions. Nobody had to agree to it for it to win.

The objections fail on their own mechanisms

"Models will close the last 8 points too." Maybe, on accuracy. But take two valid definitions of active customer: a paid invoice in the last 90 days, or a login in the last 30. Both are computable from the schema, and both are defensible. No amount of schema reading tells a model which one the organization adopted. That's a decision. It's recorded somewhere or nowhere, and a better model doesn't change which.

"Inferred definitions beat nothing." For ad hoc exploration, yes. dbt's own recommendation is "Text-to-SQL for ad hoc analyses and smaller datasets." Exploration has a human in the loop: an analyst sees an odd number, changes a filter and reruns, so a wrong definition costs one query. A KPI or a board deck runs on a schedule without anyone reading the SQL, so the same wrong definition repeats in every report until someone notices the trend is off. An automated action skips even that. Those three need a signature.

"Palantir-style ontologies are about actions." Correct. TopQuadrant's guide describes Palantir's model as object, link and action types: verbs as well as nouns, and who is permitted to trigger them. Action permissions check who may trigger "approve refund." They don't check what the refund rule's nouns mean. A rule keyed to "active customer" approves the wrong refunds if that definition is wrong, however tight the permission. So signed definitions sit underneath action permissions. One doesn't replace the other.

The definition of record fits on one card

Write one card per term an agent is allowed to use. The values below are illustrative.

term: net_revenue
definition: >
  sum(invoice_amount - credit_amount)
  where invoice_status = 'paid'
  and invoice_date between :start and :end
owner: <one named person>, Controller   # never "Finance"
decided: 2026-10-05
source: authored                    # generated | authored
test:
  question: "Net revenue for Q3 2026 (2026-07-01 to 2026-09-30)"
  must_return: 18420000
must_refuse:
  - "Net revenue by sales rep before the 2025 territory change"
  - "Net revenue before 2023-01-01 (pre-migration billing)"
review_trigger:
  - schema change to invoices or credits
  - pricing or billing-terms change

The rule that makes the card work: a generated definition stays source: generated, owner: none and is excluded from agent answers until someone signs it. Generation drafts the card. A person turns it into a record.

Any ontology product should pass four checks before purchase:

Can it show who approved each definition, and when?
Does each definition carry a test question with a known answer?
Does the agent refuse outside the defined scope, and is each refusal logged?
Can the definitions be exported if you leave the platform?

Cards cost meetings, and some teams shouldn't pay

Every signed card costs an owner's time. As an illustrative figure, 40 terms at 30 minutes each comes to 20 hours of a senior person's calendar before any re-signs. A small team where one analyst is the agreement doesn't need cards. That person's head is the ontology of record. A domain whose definitions change weekly will drown in re-signs. Those teams should keep agents in exploration mode and keep their output out of KPIs entirely.

The ontology worth budgeting for is the agreement, not the graph. When an executive asks for net revenue by sales rep before the 2025 territory change, the right answer is the first line under must_refuse, and the controller's name next to it.

Your $200K FDE Quarter Ends. Nobody Is Budgeted to Own the Agent in Quarter Two.

Michael Tuszynski — Mon, 26 Oct 2026 16:31:35 +0000

Gartner forecasts that by 2028, 70% of enterprises will abandon agentic AI built by vendor forward-deployed engineering, "trapped by soaring costs and unable to evolve it on their own." Gartner publishes it as a forecast. This post argues the mechanism behind it shows up in the budget. Gartner's own fee estimate is quoted per quarter per use case, which reads as a build price that runs until handoff. Nothing in the estimate as reported covers the quarter after it, so unless the contract says otherwise, nobody is named to own the agent or funded to keep it working, and the first forced change lands on a team that never built it.

Call that period quarter two: the first quarter the agent runs without the people who wrote it.

The $200K quarter is the floor, and it's priced as a build

Gartner estimates, reported by CIO Dive, suggest FDE consulting fees before platform and integration costs could range between $200,000 and $400,000 quarterly per use case. So $200K is the bottom of the range, and it covers consulting fees alone. Platform and integration costs sit outside it.

In the engagement this post has in mind, a fee priced that way points at one milestone: the engineers ship, the acceptance demo passes, the handoff deck goes out, and then they leave.

The Register's summary of the report names the failure: engagements fail "when customers do not acquire the knowledge and control needed to maintain and develop the resulting systems after the vendor leaves." Gartner's own Phase 1 advice says to "identify an executive sponsor who is accountable for business outcomes, not simply the implementation budget." An implementation budget ends at handoff. A business outcome doesn't.

The first forced change already has a date

The vendor leaving doesn't freeze the agent's dependencies. Anthropic's model deprecations page commits to "at least 60 days' notice before model retirement for publicly released models." That's a reasonable, published commitment. It's also a calendar someone has to read.

Take an illustrative engagement: two quarters, April through September 2026, with the agent built on claude-sonnet-4-5-20250929. Per the same deprecations table, that model was deprecated on September 30, 2026, with a tentative retirement of November 30, 2026. Sixty-one days. The recommended replacement is claude-sonnet-5-5. Requests to a retired model fail.

So the deprecation notice arrives in handoff week, and the retirement falls inside quarter two. The work is ordinary: swap the model ID, review prompts tuned against the old model's behavior, re-run the acceptance evals, compare outputs, redeploy. Nothing exotic. All of it needs credentials, the eval suite and a person whose job it is. Here is what that entry looks like when someone owns it (illustrative):

model_calendar:
  - model_id: claude-sonnet-4-5-20250929
    deprecated: 2026-09-30
    retires: 2026-11-30        # tentative, per provider page
    replacement: claude-sonnet-5-5
    migration_owner: "<named person, not a team>"
    evals_rerun_by: 2026-11-09
    last_reviewed: 2026-10-01  # reviewed monthly

A schema change in a source system or an eval score that drifts follows the same pattern, usually with less warning.

In illustrative math, the ownership line costs a fraction of one engagement quarter

Keyhole Software, a consultancy that sells builds, estimates that "annual run and maintenance costs then add 15% to 25% of the build price every year." It also says the maintenance budget "belongs in the original statement of work as a percentage, not a change order filed after the first incident." That range describes Keyhole's own build prices, not Gartner's FDE consulting fees. Applying the percentage to a $400K fee-only build, as below, is this post's illustrative transfer, not a sourced figure.

Illustrative arithmetic, at the bottom of Gartner's fee range:

Build, two quarters at $200K, fees only: $400,000
Run cost at 15% to 25% a year: $60,000 to $100,000
Per quarter: $15,000 to $25,000
As a share of one $200K engagement quarter: 7.5% to 12.5%
Cost of abandoning instead: the $400,000, plus the platform and integration spend Gartner's range leaves out

On these numbers the ownership line is the cheap part. It's also a part the fee estimate, as reported, says nothing about. And it has a cost the dollar figure hides: the owner is an engineer whose hours come out of some other roadmap, and the sponsor has to defend a budget line for a system that already "shipped."

The ownership sheet gets signed before the SOW

The grader post priced checking the work during the build. The sheet below covers who owns it after the vendor goes.

Line	What gets written down	Rule
Owner	One named internal person	Not a team, and not the vendor
Budget	Run cost as a percent of build	In the SOW, not a change order
Model calendar	Every model ID the agent calls, with its published retirement date	Reviewed monthly
Evals	The acceptance suite, handed over	Runnable by the owner without the vendor
On-call	Who gets paged	Plus what they're allowed to change
Kill criteria	The condition under which the sponsor turns it off	Signed by the sponsor (the kill-sheet post has a five-field version with a named cancel authority)

The week the vendor leaves, five things should be true:

The owner can redeploy the agent without vendor credentials.
The owner has run the full eval suite once, alone.
The model calendar lists every model ID in production.
The first migration has a date on it.
The sponsor has signed the kill criteria.

If any of the five is false, the exit plan isn't finished.

Extending the vendor buys more of the same problem

The obvious objection: keep the vendor on for run and support. Gartner's Phase 3, titled "Exit and Prove Independence," rejects that in plain words: "Execute the exit plan established at the start rather than extending the engagement because internal teams are not ready." The mechanism sits in the forecast's own phrasing. An extension at FDE rates is the "soaring costs" half. And while the vendor carries the pager, the client's team has less reason to learn the system, which feeds the "unable to evolve it" half. Extension compounds the gap it was meant to cover.

The second objection: the vendor will fold the custom work into its product, so there's nothing to own. Gartner predicts that through 2028, less than 20% of FDE engagements will turn recurring customer needs into capabilities in the vendor's core product. If that forecast holds, more than 80% of engagements won't turn recurring needs into product capabilities, and whatever they built still has to be kept working by someone outside the vendor's product team. Who holds the IP depends on the contract. Either way, the system keeps running on the client's side, and someone there has to keep it working.

The third: agents are cheap to keep running. The token bill may well be small. The scheduled work isn't optional. A 60-day retirement notice, an eval re-run, an upstream API that changes its response shape: each arrives on someone else's calendar. Cheap inference with no owner still breaks on retirement day, tentatively November 30.

A pilot built to die needs only the last row

The sheet doesn't transfer everywhere. A pilot designed to be killed at handoff needs no quarter-two line. It needs kill criteria and a date, and nothing else on the sheet applies.

A client with no internal engineering can't hold the owner seat at all. Naming a project manager as owner of a system they can't redeploy is theater. The honest answer there is a priced managed service with an exit clause, chosen on purpose. What fails is the version that gets drifted into, one renewal at a time, at FDE rates.

For everyone else, the unowned quarter is the mechanism this post argues sits behind Gartner's 70% forecast, and the fix is one line signed before the engagement starts. For the illustrative summer build, the first entry on that sheet is already known: claude-sonnet-4-5-20250929, tentatively retires November 30, 2026, owner: ________.

The Lint Ran Every Week. The Dead-Link Count Never Went Down.

Michael Tuszynski — Fri, 23 Oct 2026 16:23:48 +0000

In August I wrote that Karpathy's LLM wiki works, and that a weekly lint pass "recovers most of" the dead links and orphan pages a wiki piles up. That was wrong. The lint ran every week after that, and it recovered nothing.

Here is the log. My vault runs a scheduled checker every Sunday that counts pages, dead wikilinks and orphans. The last column strips out matches that were never links, which I'll get to below.

Week	Pages scanned	Dead wikilinks (raw)	Orphan pages	Actually broken
Aug 2	0	0	0	0
Aug 9	0	0	0	0
Aug 16	261	116	117	109
Aug 23	264	120	117	113
Aug 30	264	120	117	113
Sep 6	267	120	118	113
Sep 13	283	153	117	136
Sep 20	293	153	117	139
Sep 27	303	155	118	141
Oct 4	317	169	124	145
Oct 11	327	170	124	146
Oct 18	334	170	124	146

The first two rows are a separate embarrassment. The checker was pointed at a folder I had retired in a migration, so it scanned zero pages and reported a clean wiki. The post claiming lint works went out on August 13, four days after the last zero-page run and three days before the first real one.

From August 16, when it scanned the real vault, to October 18, pages grew 28% and actually broken links grew 34%. That's roughly 0.42 per page at the start and 0.44 at the end: the rate held steady. What never happened is the thing lint is supposed to cause. Across ten weekly runs, the broken-link count never went down once, and orphans ended at 124, up from 117. Detection ran on schedule every week. Nobody repaired anything.

So the missing line item in Karpathy's pattern isn't the lint. Lint is cheap. It's the repair, and the repair turns out to be mostly decisions a script can't make.

A repair script could fix none of 185 on its own

In September I added a repair tool. It finds every [[link]] whose target note doesn't exist and tries to match it to a real note: first case-insensitive, then fuzzy. It rewrites only the safe cases, where the link differs from a real filename by capitalization or spacing.

I ran it today in dry-run mode. It took 2.7 seconds.

vault-repair: 185 dead wikilinks — 0 confident fixes, 4 fuzzy suggestions, 181 unresolved
  ~ 200-knowledge/context/recipe-context.md: [[recipes/classic-meatloaf]] ≈ [[classic-meatloaf]]

Zero safe fixes. Four suggestions, all the same mistake: a recipe linked by folder path instead of note name. The other 181 have no candidate at all.

(The weekly checker said 170 on Sunday; the repair script says 185 today. Different tools on different days, and I haven't diffed the two lists, so I can't tell you yet which explains the gap.)

Most dead links are names the agent was told to write

I sorted all 185 by what they point at.

What the link is	Count
A name or topic with no note behind it, mostly people, companies and services (41 of them in archived daily logs)	154
A CONFIRM placeholder my draft pipeline leaves in old blog drafts	17
Example syntax inside a third-party README I saved	7
A folder-path link: 4 to recipes that exist, 3 to index pages that don't	7

The big row is the interesting one. My operating file tells the agent to wikilink every person, company and service the first time it comes up, so a later page about the same entity connects back. That rule works. It's also a dead-link generator. Every new name gets brackets the moment it's written, and nothing creates the card behind it.

Schwab is linked six times. Plaid three. Most of the rest are names that came up once or twice in a daily log, each wrapped in brackets and pointing at nothing.

Karpathy's original gist names this exact finding. His lint step looks for "important concepts mentioned but lacking their own page." Finding them was never the hard part. Each one is a question only the wiki's owner can answer: does this name deserve a card, or should the brackets go? A name mentioned once in a call note probably doesn't need a page. An institution that holds your money does.

The bottom three rows are noise, and they say something too. The repair script's link pattern matches anything inside double brackets:

LINK_RE = re.compile(r"\[\[([^\]|#]+)(?:#[^\]|]*)?(?:\|[^\]]*)?\]\]")

So my content pipeline's own review markers count as broken links, and so do the [[wikilinks]] examples in the READMEs of nvk/llm-wiki and Ar9av/obsidian-wiki that I saved for reference. A dead-link count is a count of bracket pairs, not of broken knowledge. 24 of the 185 were never links in the first place, and the weekly checker counts them too. On August 16 it counted 7 of those. By October 18 it counted 24, because my draft pipeline had started leaving CONFIRM markers in saved drafts. That's why the raw column in the table above grew faster than the real one.

Auto-created stubs move the number, not the problem

The obvious fix: have the agent create a stub card whenever it writes a new link. The dead-link count goes to zero overnight.

That trades one lint finding for another. A stub with a title and no content is an orphan in everything but name. Nothing useful links into it, and nothing in it is worth reading. I already carry 124 orphans. Stubs would add up to 154 more and make the checker look healthier while the wiki gets no more useful.

It also defeats the point of the card. An entity card is worth having because it's the one place a fact about that entity lives. A stub promises that place exists and then holds nothing. The next session that reads it learns the name and nothing else.

Where this stops mattering

A small wiki in one domain probably won't hit this. My guess, and it's only a guess, is that it stays manageable while there are few enough names to card each one as it arrives. Mine broke down once it spanned several domains and the daily logs mentioned people faster than anyone curated them.

A team with someone whose job is curating the wiki won't hit it either. That person is the missing line item.

Budget the decisions, not the lint

If you're rolling Karpathy's pattern out to a team, the lint is the part you can automate and forget. Count the other thing: how many new names enter the wiki each week, and who decides whether each one gets a card.

My count today is 154 open decisions, built up over ten weekly lint runs nobody acted on. The script fixed none and suggested 4. The rest need an owner, and the weekly report is only useful if someone reads it on Monday.

Every FDE Pod Has a Strategist. None Has a Grader.

Michael Tuszynski — Tue, 13 Oct 2026 14:31:07 +0000

Gartner forecasts that 70% of enterprises will abandon agentic AI built by vendor forward-deployed engineers by 2028, in a September 29 release reported the next day by The Register and Channel Dive. The firm calls the pattern "FDE washing." Its analyst counts about 2,000 active FDEs against roughly four times that demand, and Gartner estimates that FDE consulting fees alone could top $200,000 per quarter per use case.

Gartner's stated mechanism is dependency and exit: clients end up unable to run or leave what the vendor built. How the pod is designed is not part of its analysis. This post argues the dependency has a structural cause visible in the published FDE pods. The pods that name their roles each have a strategist, and none has a grader. No seat checks the work against criteria the client agreed to, held by someone who had no hand in building it. That link is this post's argument. Gartner doesn't make it.

The pod is already the standard delivery model

The lone engineer embedded at a client is mostly a recruiting image. The published delivery unit is a pod, and the pods look alike.

Palantir pairs the Delta (the FDE) with an Echo (a deployment strategist) and Devs on the product side. One alumnus describes FDEs going on site at the customer three to four days a week. Deloitte's lead-FDE postings describe pods of two to five onshore-anchored, offshore-supported engineers. Thoughtworks' principal-FDE posting describes a pod of one product strategist, one experience designer and two forward-deployed engineers. AWS's billion-dollar FDE organization deploys pods of five or six people, embedded inside a single customer at a time. Anthropic's own postings pair FDEs with a technical deployment lead who won't write production code and owns the roadmap the engineers execute against. The shape repeats: one or two engineers, a lead who owns scope and the executive relationship, sometimes a designer, and offshore or platform support behind them. Postings change, so the specifics will drift. The shape hasn't.

The supply side is about to scale this template. Anthropic's Claude Frontier Academy commits $100M to train 10,000 "Frontier Deployed Engineers" by the end of 2027 at partners including Accenture, Deloitte, McKinsey and Morgan Stanley. Accenture's statement says those FDEs will work "inside client workflows as part of multidisciplinary teams." If the partners follow their own postings, those teams will look like the ones above.

Every seat in the pod helped build the thing

Read those pod descriptions for a quality seat and you won't find one. None of them names a QA seat, an acceptance seat or a grading seat. The closest is a mention of "quality gates" in Deloitte's posting.

The strategist seat is already standard. Palantir calls it the Echo. Other firms call it a deployment lead, a product strategist or a PM. Pods don't need anyone to invent it. And every seat in every pod is a builder. Engineers write the code. The strategist sequences the roadmap. The designer shapes the workflow. Platform and offshore support keep it running. Each person present has a stake in the result passing.

Verification gets dropped because nobody is waiting on it

The earlier post on the FDE title broke the role into separate jobs. Here's the version that matters for pods. The role has four jobs: build the system, redesign the process around it, run the client relationship, and verify the result. A pod splits the first three across seats. The fourth goes unassigned.

It gets dropped inside a pod for the same reason a lone engineer drops it. No one is waiting on the grade. Shipping has a demo date, and the executive relationship has a steering meeting. A verification report has no audience until something breaks. Everyone in the room also wants the work to pass, including the vendor's lead, whose quarterly fee renews on the impression that it did.

Follow that to the client's side. Without an independent grade against agreed criteria, the client can't tell what it bought. Without a sponsor who has authority to stop the work (the one seat the kill sheet from the earlier post insists on), the client can't leave without losing whatever value is in there. It can't separate the working parts from the broken parts, so it keeps paying the only people who know. That is dependency, and that's what Gartner is forecasting.

The missing seat grades and does nothing else

Call it the acceptance owner. The definition is narrow on purpose:

Co-writes testable acceptance criteria with the client sponsor before work starts
Grades the outcome against those criteria after every change
Publishes the grade to the sponsor, not to the pod first
Had no hand in building the work

The seat can sit on either side of the contract. A mature client's own QA function can hold it. A third party can. A vendor employee from outside the delivery org can, if the client accepts that. What disqualifies someone is having built the work. Which side signs their paycheck matters less.

Here's the cadence relative to the pod:

Week	Pod	Acceptance owner	Sponsor
0	Scopes the use case	Co-writes criteria with sponsor; each one testable pass/fail	Signs criteria and stop conditions
1-5	Builds, demos weekly	Re-runs the criteria after each merged change; logs pass/fail	Reads the grade before the demo
6	Mid-sprint review	Publishes trend: criteria passing, failing, untested	Decides continue, re-scope or stop
7-12	Builds, hardens	Same loop; adds regression on previously passed criteria	Reads the grade weekly
13	Quarter close, renewal	Final grade against week-0 criteria, unchanged unless sponsor re-signed	Renews or exits on the grade

Each seat also has one thing it must not do. The engineer doesn't grade their own work. The strategist can write down what done means but doesn't own the grade. The sponsor never hands the stop decision to someone else. The acceptance owner's rule is the least obvious one: don't suggest fixes. The moment the grader proposes the fix, the grader has helped build it.

Owning the criteria doesn't make the grade independent

The strongest objection is that the pod already has this. Palantir's Echo, or a deployment lead who owns the roadmap, writes down what done means.

That person owns the criteria. They also own the scope, the sequencing and the executive relationship, and they answer for whether the engagement renews. When a criterion half-passes in week 11, the person who wrote the roadmap is also the person who explains to the sponsor why it's close enough. The objection fails on that mechanism. Authorship and grading sit in the same chair, and that chair is the one with the most riding on renewal.

The second objection: the client's own QA team can do this. Often true. When it can, that team is the seat, provided it didn't help build the system. The test is simple. If the client's QA engineers pair with the FDEs during the build, they've crossed over, and someone else has to grade.

A fractional grader costs about 2.5% to 7.5% of a $200K quarter

The acceptance owner doesn't need to be full time. The work is a criteria workshop up front, a re-run after each change, and a weekly report. That fits in roughly 0.1 to 0.2 of a seat.

Illustrative pricing, using Gartner's $200,000-a-quarter estimate:

Pod fee: $200,000 per quarter for one use case
Pod of four seats (two engineers, strategist, designer): $50,000 per seat per quarter
Acceptance owner at 0.1 seat: $5,000 per quarter, or 2.5% of the fee
At 0.15 seat: $7,500 per quarter, or 3.75%
At 0.2 seat: $10,000 per quarter, or 5%

An experienced independent grader may bill above the pod's average seat. Double the rate and 0.15 of a seat costs $15,000 a quarter, or 7.5%.

Now price it against abandonment. A use case run for four quarters at $200,000 costs $800,000. The grader at 0.15 seat costs $30,000 for the same year. If an independent grade ends a failing engagement just one quarter early, it saves $200,000. That covers the seat at $7,500 a quarter for about 26 quarters, more than six years.

The cost of this advice goes beyond the invoice. A grader will fail work the pod considers done. Demos get awkward. Renewals get harder for the vendor. The pod loses some of its speed, because "shipped" now has to mean "passed."

Where the seat doesn't fit

An 8-week proof of concept often can't carry any extra seat, fractional or not. The fix there is cheaper: write the criteria at week 0 and have the sponsor grade at week 8 personally.

Clients who won't name a sponsor break the model entirely. A grade with no one empowered to act on it is a report nobody reads.

The onshore-anchored, offshore-supported pod is the hardest case. On paper it's five engineers. In practice one engineer can end up alone at the client, with the rest of the pod a time zone away. That engineer is the whole on-site build team and the only witness to whether it works. A pod like that needs a grader most and is least likely to have budget for one.

Go back to the criterion that half-passes in week 11. With an acceptance owner, it doesn't get explained in a steering meeting. It shows up in the weekly grade as a fail, with the week-0 wording next to it, and it shows up again in week 12. At week 13 the sponsor holds a sheet with every signed criterion marked pass or fail. That sheet is the seat every published pod is missing. If that line is still red, the sponsor decides whether the next $200,000 buys a fix or funds the exit.

30% More Code, Zero More Shipped Work: Where the Capacity Actually Went

Michael Tuszynski — Mon, 12 Oct 2026 16:29:30 +0000

A Harvard study of 718 firms found that coding agents produced 30% more lines of code and no detectable increase in finished work. Most coverage stops there and reads the zero as a verdict on the tools. The paper's own review data points somewhere else. My reading of that data, and it's my inference rather than the paper's stated finding, is that the extra output was real and the review stage absorbed it. That's an org-design failure, and you can look for it in your own pull request history this week.

The extra code piled up at the review stage

Fiona Chen and James Stratton built the study on Jellyfish data covering 718 firms and about 300 million work events from January 2021 to March 2026. Jellyfish sells engineering analytics. A finding that work stalls somewhere measurable isn't against its commercial interest, so weigh the source with that in mind. The analysis itself comes from academic economists, and the headline result is not the one an AI vendor would pick.

After teams adopted coding agents, the activity numbers went up. Lines of code rose 30%. Commits rose 20%. Pull requests rose 23%.

Jira issue and epic resolution showed no change the authors could distinguish from zero. Their estimate rules out an output gain above 12%.

The review numbers explain the gap. In the same paper:

PR review time rose 49%.
The share of PRs with changes requested nearly doubled.
Comments per PR rose 35%.
The share of workers doing review rose 14%.

Put the two lists side by side. More code went in at the top. Review time, the changes-requested rate and comments per PR all rose, even with 14% more people doing review. Resolved tickets came out the bottom flat. The paper reports these as separate effects and doesn't claim one caused the other. My inference from the pattern: the capacity didn't vanish. It turned into review load.

Review load grew faster than PR count

The paper estimates each effect separately, so stacking them is my illustration, not a result from the paper. But stacking them shows why a 23% bump in PRs can stall a team.

Take an illustrative 20-person team with 6 regular reviewers, 200 merged PRs a month, a 10-hour median review time, 4 comments per PR, and 15% of PRs sent back with changes requested. Apply the paper's effects:

                          before     after      change
PRs / month                  200       246       +23%
Review time (hours)         10.0      14.9       +49%
PR-hours awaiting review   2,000     3,665       +83%   (1.23 x 1.49)
Comments / PR                4.0       5.4       +35%
Total comments               800     1,328       +66%   (1.23 x 1.35)
Reviewers                    6.0      6.84       +14%
Comments per reviewer        133       194       +46%   (1.66 / 1.14)
Changes-requested rate       15%       30%       ~2x
PRs needing a second pass     30        74

PR count rose 23%. PR-hours spent awaiting review rose 83%. That's elapsed time in the queue, not reviewer effort, but it's time finished work sits unshipped. Even with 14% more people reviewing, each reviewer handles roughly 46% more comment traffic. Every changes-requested PR goes back through the queue, and the illustrative team goes from 30 second passes a month to 74.

That's the mechanism as I read it. Agents raised the rate at which work arrives at review. Nobody raised the rate at which review clears it.

Headcount didn't move, and that's the CFO's problem

The paper's headcount results are where the budget conversation should start. Total headcount showed a precise zero effect, ruling out a decline above 2.9%. Engineering headcount showed a noisier zero, ruling out a decline above 13.8%.

A CFO who funds agent licenses expecting to shrink the engineering line didn't get that in this data. My argument, which the paper doesn't test directly: what the money bought was a review backlog. Buying more seats pushes more work into a queue that's already behind.

Measure where your capacity went from PR data

You don't need Jellyfish to check whether your team looks like the paper. The GitHub API has what you need. This script pulls merged PRs, their reviews and their timelines, and prints three numbers per month: time to first review, changes-requested rate, and reviewer share. Time to first review starts at the last ready-for-review event (or PR creation, if the PR never sat in draft) and stops at the first submitted review.

import json, subprocess, sys
from collections import defaultdict
from datetime import datetime
from statistics import median

REPO = sys.argv[1]  # e.g. your-org/your-repo

def gh(path):
    raw = subprocess.run(["gh", "api", "--paginate", "--slurp", path],
                         capture_output=True, text=True, check=True).stdout
    return [item for page in json.loads(raw) for item in page]

def ts(s):
    return datetime.fromisoformat(s.replace("Z", "+00:00"))

def review_start(pr):
    events = gh(f"repos/{REPO}/issues/{pr['number']}/timeline?per_page=100")
    ready = [ts(e["created_at"]) for e in events
             if e.get("event") == "ready_for_review"]
    return max(ready) if ready else ts(pr["created_at"])

months = defaultdict(lambda: {"prs": 0, "cr": 0, "hours": [],
                              "authors": set(), "reviewers": set()})

for pr in gh(f"repos/{REPO}/pulls?state=closed&per_page=100"):
    if not pr["merged_at"]:
        continue
    m = months[pr["merged_at"][:7]]
    m["prs"] += 1
    if pr["user"]:
        m["authors"].add(pr["user"]["login"])
    reviews = gh(f"repos/{REPO}/pulls/{pr['number']}/reviews?per_page=100")
    start = review_start(pr)
    submitted = sorted(ts(r["submitted_at"]) for r in reviews
                       if r.get("submitted_at") and ts(r["submitted_at"]) >= start)
    if submitted:
        m["hours"].append((submitted[0] - start).total_seconds() / 3600)
    if any(r["state"] == "CHANGES_REQUESTED" for r in reviews):
        m["cr"] += 1
    m["reviewers"].update(r["user"]["login"] for r in reviews if r["user"])

for month in sorted(months):
    m = months[month]
    people = m["authors"] | m["reviewers"]
    wait = f"{median(m['hours']):6.1f}" if m["hours"] else "   n/a"
    print(f"{month}  prs={m['prs']:4d}  "
          f"first_review_h={wait}  "
          f"changes_requested={m['cr'] / m['prs']:.0%}  "
          f"reviewer_share={len(m['reviewers']) / len(people):.0%}")

A few caveats. It makes two API calls per PR, so run it on one repo at a time or you'll hit rate limits. PRs merged with no review drop out of the timing median. Reviewer share here means reviewers divided by everyone active in the repo that month, not total headcount. And time to first review is my proxy. It isn't the paper's PR review time metric, which comes from Jellyfish's own definitions, so don't compare your hours to anything in the paper. Compare your percentage change across the rollout to the paper's 49%.

Draw a line at your agent rollout month and read the pattern. If PR count and changes-requested rate both climbed while reviewer share stayed flat, you have the paper's queue. If reviewer share rose and time to first review still climbed, you added reviewers without shrinking the work each one sees. Pair this with the 30-day rework measure from the earlier post on AI coding dashboards, because every fix below trades review time against escaped defects.

Fix review capacity and merge policy before buying more seats

Start by making review planned work. A reviewer rotation on the sprint board, with hours attached, stops review from being the thing people squeeze in after their "real" tickets. Those hours come out of feature work, and the roadmap will show it. That's the honest price of clearing the queue.

Merge policy is the second lever. Tier the approval rules: changes in low-risk paths that pass tests merge on one approval, while auth, payments and data migrations keep two. You'll get more escaped defects in the low-risk tier. That's why the rework number has to sit next to the speed number on the same dashboard.

Then shrink the unit of review. Cap diff size and tell your agents to split work into reviewable pieces. My argument here, not a measured result: a 400-line agent PR gets skimmed in a way that four 100-line PRs don't. The cost is more review events, each with its own context switch, plus tooling for stacked PRs.

This doesn't transfer everywhere. On a four-person startup where review is a glance, the constraint is probably upstream: unclear tickets, slow product decisions. Agents there produce code faster against specs nobody agreed on, and no review fix helps. Jira resolution is also an imperfect output measure. Teams that adopt agents may start writing bigger issues, which would hide gains in the ticket count. The review numbers don't depend on how tickets are sized, though, and they moved.

Newer agents raise the arrival rate, not the review rate

The strongest objection: the data ends in March 2026, and today's agents are better. Maybe the stall was a property of weaker tools.

Look at where the stall happened. A better agent writes more code, and more of it is plausible. That raises the arrival rate at review. It does nothing for review throughput unless the agent's output needs fewer changes or the review itself gets automated.

That can work, and the script above will tell you. If changes-requested rate falls after a model upgrade and 30-day rework doesn't rise, the newer agent is clearing the queue. If PR count jumps again and review time follows, you got the March 2026 result with a faster input.

A hidden 12% gain still leaves most of the extra volume in the queue

The second objection is statistical. "No detectable change" only rules out gains above 12%. A real gain of 5% or 10% could be sitting inside the confidence interval.

Concede it. Then do the arithmetic. PRs rose 23%. At the 12% ceiling, PRs per resolved issue still rose by roughly 10% (1.23 / 1.12 = 1.098). The 49% jump in review time is measured directly, not inferred from a null. A modest hidden gain is the best case for this argument: the tools produced something, less of it reached done than went in, and the review stage holds the difference. A gain that small also caps the headcount math. It can't pay for the 13.8% engineering cut the paper already ruled out.

So the zero isn't a verdict on the tools. It's what happens when output rises and review capacity doesn't. Go back to the illustrative 20-person team: agents gave it 46 extra PRs a month, and those PRs arrived carrying 1,665 extra PR-hours waiting in review and 44 extra second passes.

Gartner Says Over 40% of Agentic AI Projects Will Be Canceled. The Expensive Ones Are the Ones That Aren't.

Michael Tuszynski — Sun, 11 Oct 2026 02:04:52 +0000

A support-ticket triage agent goes live on a Monday with a 90-day pilot budget and a vague goal: "reduce agent handle time." On day 90 the numbers are mixed. Handle time is down on some queues and up on others, and three reviewers spend part of each week fixing the agent's routing mistakes. Nobody calls it a failure, because nobody wrote down what failure would look like. So the project keeps its funding, and it is still running a year later.

That project won't appear in Gartner's cancellation count. It's the expensive outcome.

A cancellation is the cheap way for a pilot to end

Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls. Reuters ran it as a failure headline, and so did nearly everyone else.

I read it the other way. A canceled project stops spending. The team moves to the next bet and the inference bill stops. If the cancellation comes at the end of a bounded pilot, the organization paid a known amount to learn something true.

The projects that should worry a budget owner sit in the other 60%. Some of them are working. Some of them are zombies: pilots that never cleared a value bar and never got killed. They keep paying for inference, integration upkeep and human supervision, and they hold onto engineers who could be building something that works.

The 40% is a forecast, and it only counts the failures someone admitted to

Two cautions before anyone puts this number in a board deck.

First, it's an analyst prediction about 2027, not a measurement. The survey data in the same press release comes from a January 2025 poll of 3,412 Gartner webinar attendees. In that poll, 19% reported significant agentic AI investment and 31% were waiting to see. People who sign up for a Gartner webinar on agents aren't a random sample of enterprises. Treat the 40% as an informed guess about direction.

Second, the forecast counts cancellations, and cancellation requires a decision. A zombie never triggers one. As Forbes' read of the Gartner report puts it, the causes are management failures, not model failures. A management failure that keeps a project alive is the same failure as one that kills it late. It's just harder to see.

So the 40% tells you nothing about how many of the surviving 60% deserve to survive. The one tool that answers that is a document written before the pilot starts.

The kill threshold has to exist before the first prompt ships

My earlier post, "Most Agent Projects Die. The Survivors Share a Shape," covered why survivors survive. This one is about the artifact the survivors' sponsors write and the zombies' sponsors skip: a kill sheet, signed before the pilot starts.

It has five fields. Here it is for the triage agent:

# kill-sheet.yaml: signed before pilot start, not edited after
project: support-triage-agent
metric:
  name: median_time_to_correct_queue   # one metric, not a dashboard
  baseline: 4.2h                        # measured over the 30 days before launch
  target: 2.5h                          # what "worked" means
  measured_on: 2026-04-01               # day 90, a fixed date
monthly_run_cost:
  inference: 3800
  integration_upkeep: 4000              # 0.25 engineer, loaded
  human_review: 6750                    # 3 reviewers x 30h x $75
  total: 14550
cost_ceiling: 12000                     # above this, value has to be proven
cancel_authority: VP Support Operations # a name, not a committee
extension_used: false

The format matters less than when it gets written. Every field in it is easy to fill in on day zero and very hard to fill in honestly on day 90, once people have spent a quarter on the project and the sponsor has told leadership it's going well.

Some notes on the fields.

One metric. If you give an agent five metrics, it will move one of them, and that one becomes the story. Choose the metric the business would pay for, and measure the baseline before launch. A baseline reconstructed after the fact always comes out flattering.

The date is fixed. "End of Q2" turns into "once the new model ships" turns into "after the reorg." Put a calendar date on it.

Human review is part of run cost. It's usually the largest line, and usually the one left out. In the sheet above, review time is 46% of monthly cost. Inference is 26%. If your sheet shows only the token bill, it understates run cost by a factor of three or four.

The ceiling sits below current run cost. That's deliberate. A pilot that costs more than the ceiling has to clear the metric target to continue. It doesn't get to argue it's "almost there."

Cancel authority is one named person. If killing the project needs a committee, it won't happen, because no committee member wants to own the decision.

Expect the cancel-authority line to draw the most pushback. It's the only field that puts a person's name next to a possible failure.

A zombie costs six times what an honest cancellation does

When the triage agent misses its target on day 90, there are two outcomes to price.

Cancel on day 90 and the cost is three months of run cost, $43,650. That's the tuition.

Now say it stays alive for nine more months on "let's give it another quarter," and then another. The zombie bill has two parts:

def zombie_cost(monthly_run: float, months_past_kill: int,
                team_monthly_loaded: float) -> dict:
    run = monthly_run * months_past_kill
    # Opportunity cost, floor estimate: the loaded cost of engineers who could
    # have been on the next-best project. Their real cost is that project's
    # value, which is usually higher and harder to defend in a spreadsheet.
    opportunity = team_monthly_loaded * months_past_kill
    return {"run": run, "opportunity": opportunity, "total": run + opportunity}

zombie_cost(14_550, 9, 16_000)
# {'run': 130950, 'opportunity': 144000, 'total': 274950}

The opportunity line assumes one full-time engineer's worth of attention ($16,000 a month loaded), split across the two people still tuning prompts and triaging complaints. It doesn't overlap with the 0.25 engineer already counted under integration upkeep. And it's a floor. The true cost of those engineers is whatever they would have shipped instead.

That comes to $274,950 against $43,650, about 6.3 times the cost of the honest answer. Every month of delay adds $30,550. On top of that, the zombie keeps three reviewers in the review habit, and it makes the next agent proposal harder to fund, because leadership now associates "agents" with a project that never ended.

None of these numbers are exotic. They're the triage agent's own costs from the sheet above, multiplied by the time nobody decided anything.

Late bloomers get one extension, written down in advance

The strongest objection is real. Some value takes time to show up. A triage agent may get much better after six months of corrections feed its routing rules, or once the support team stops working around it. A strict day-90 date kills that project before it matures, and it rewards pilots that look good early over those that compound.

That argument holds only when there's a mechanism behind the slow value, and a mechanism can be written down. So the kill sheet gets an extension rule, agreed at signing:

One extension, no more than 60 days. A second extension is a cancellation.
Before the extension starts, the sponsor names the mechanism ("routing accuracy improves as correction data accumulates") and a leading indicator that can be measured now ("misroute rate falls from 18% to 12% by day 150").
The cost ceiling doesn't change. If the extension needs more money, it's a new project and gets a new sheet.
The same named person signs the extension and owns the decision at its end.

This answers the late-bloomer case on its own terms. A real late bloomer can say why it's slow and what will move first. A zombie can only say it needs more time. The rule lets the first through and stops the second, and "we're seeing promising signals" doesn't count as a mechanism.

The cost of this discipline is that some projects that would have made it will die. A kill sheet will occasionally cancel a winner. That trade is fine. A false cancellation costs one pilot's tuition plus a restart. A false reprieve costs $30,550 a month with no end date.

Go back to the triage agent on day 90. With the sheet, the VP of Support Operations reads one number, median time to the correct queue: 3.6 hours against a target of 2.5. She either signs one 60-day extension tied to a misroute rate or ends it. Without the sheet, the same 3.6 hours gets called "a 14% improvement" in a status deck, and nine months later the project has cost $274,950.

An AI Agent With a Crypto Wallet Is a Hot Wallet Anyone Can Talk To

Michael Tuszynski — Tue, 06 Oct 2026 04:18:31 +0000

An attacker sent Grok a free NFT, posted a message in Morse code, and walked away with roughly 3 billion DRB tokens, about $174,000. Nobody stole a private key and no smart contract had a bug. One AI agent translated some dots and dashes in public. A second AI agent read that translation as an order and paid it.

That incident from May 2026 is the best case against giving agents custody of crypto. An agent wallet is a hot wallet with a chat interface, and anyone who can get text in front of the agent can talk to it.

The attack turned a translation into a transfer

The sequence is short, and every step matters.

Grok had a wallet through Bankr, a bot that executes on-chain actions for users who tag it on X. That wallet was read-only. The attacker airdropped a free Bankr Club NFT to it. Holding the NFT upgraded the wallet's permissions from read-only to transfer rights. So an inbound token, which anyone can send to any address, changed what the wallet was allowed to do.

Next the attacker posted Morse code and asked Grok to translate it and tag Bankrbot. Grok did what it was asked. The decoded text told Bankrbot to send about 3B DRB to the attacker's address. Bankrbot saw a tagged instruction coming from an account with transfer rights, and it executed.

Two design decisions failed here. Permissions could be granted by something the wallet received, not just by something its owner configured. And one agent's output (a translation, which is about as harmless as text gets) went straight into another agent's command parser. Neither is a model problem, and a smarter Grok would not have fixed either one. Grok translated correctly. That was the attack.

The money came back because the attacker chose to return it

The funds were returned, and coverage has treated that as a happy ending. It's the most alarming part of the story. The recovery came from the attacker's goodwill. No system clawed it back.

On-chain transfers are final. You get no chargeback, no issuing bank to call, and no dispute window. Once Bankrbot signed the transaction, the only party who could undo it was the person holding the $174K. Allium's explainer on agent wallets describes agents that hold and spend onchain as an emerging pattern. The pattern inherits that finality whether or not anyone designed for it.

A human who gets phished on a bank wire has a few hours and a fraud department. An agent that gets prompt-injected on a public chain has however long a block takes to confirm.

Monitoring fires after the money is gone

Compliance and transaction-monitoring vendors were quick to cite the incident as a reason to buy detection: watch agent wallets, score anomalous transfers, alert on suspicious patterns. That's the strongest counterargument to my position. It says agent custody is fine once you have good enough eyes on it.

Look at what monitoring would actually have done here. Bankrbot's transfer was one transaction, signed by an authorized wallet and sent to a valid address. A good anomaly model flags it as unusual, maybe within seconds of confirmation. The flag lands on a transfer that has already settled, on rails with no reversal path. You get an excellent incident report and a very short list of remediation options, most of which come down to asking the attacker nicely.

Detection pays off when something downstream can act on it, like a card network holding settlement or a bank freezing an account. On a public chain nothing downstream can act. Every control that matters has to run before the signature.

Capability limits have to be enforced in code, before signing

Here's the control set I'd require before an agent touches anything that moves value:

No standing transfer rights. The default is read-only. Spending authority is granted per task and expires.
Permissions come from the operator's configuration and nowhere else. No inbound token, NFT, message, or airdrop can change what the wallet may do.
Destinations come from an allowlist. An address the operator hasn't approved can't receive funds, however the request is worded.
Per-transaction and daily caps, enforced outside the model.
Human approval above a threshold, set low enough that it actually triggers.
Text an agent emits never gets parsed as a command by another agent. Translations, summaries, and replies are data.

The last rule is the one that would have stopped the Grok/Bankr chain on its own. Here's roughly what the gate looks like as code that sits between any agent and the signer:

type Origin = "operator" | "agent_output" | "inbound_message";

type TransferRequest = {
  to: string;
  amountUsd: number;
  origin: Origin; // set by the transport layer, never by the model
};

// Loaded from deploy-time config. No code path reads wallet holdings
// to decide permissions, so an airdropped NFT changes nothing.
const POLICY = {
  allowlist: new Set(["0xTreasury...", "0xVendorA..."]),
  perTxCapUsd: 250,
  dailyCapUsd: 1_000,
  humanApprovalAboveUsd: 100,
} as const;

export function authorize(
  req: TransferRequest,
  spentTodayUsd: number,
): "allow" | "needs_human" | "deny" {
  if (req.origin !== "operator") return "deny";
  if (!POLICY.allowlist.has(req.to)) return "deny";
  if (req.amountUsd > POLICY.perTxCapUsd) return "deny";
  if (spentTodayUsd + req.amountUsd > POLICY.dailyCapUsd) return "deny";
  if (req.amountUsd > POLICY.humanApprovalAboveUsd) return "needs_human";
  return "allow";
}

Run the attack through it. The instruction to Bankrbot came from Grok's public reply, so origin is agent_output, and the first line returns deny. Even if someone mislabeled the origin, the attacker's address isn't on the allowlist. Even if it were, $174K is roughly 700 times the per-transaction cap. The attack has to get past four independent checks, and none of them depend on the model reading the request correctly.

The origin field carries the most weight, and it's also the easiest to get wrong. It has to be stamped by whatever received the message. If the model fills it in, the attacker can fill it in too. In the agent pipelines I run, no tool that moves money or deletes data accepts arguments parsed from another model's free-text output.

This gate has real costs. Allowlists make agents worse at the open-ended "pay whoever" tasks that make crypto agents appealing in demos. Human approval adds latency. Caps cut into throughput. That's the point. Those costs are the price of putting a limit on what a stranger's message can do.

Card rails can say no and take it back

In June 2026 Mastercard announced Agent Pay for Machines. Going by Mastercard's own description, it authenticates the agent, enforces spending limits, and guarantees settlement on a network that already has dispute rails. That's Mastercard describing its own product, and nobody has shown publicly how those disputes play out when the buyer is a bot. Treat it as a design statement until someone reproduces it in production.

Still, the design statement points the right way. Card networks were built on the assumption that some transactions will be wrong. Authorization can decline a purchase before it goes through, and chargebacks can reverse it afterward. The Grok/Bankr chain had neither.

Smart accounts get you limits but still no reversals

The strongest technical pushback is that crypto can enforce limits too. Smart-contract wallets support session keys with spending caps, destination restrictions, and expiry, all enforced on-chain. That's true. A well-built smart account can run most of the gate above at the contract level, which is better than a policy check sitting in an off-chain service.

But it only covers the "say no" half. A session key capped at $250 limits the damage, and the $250 is still gone. For agents that have to transact on-chain, such as protocol treasuries, on-chain market makers, or anything with no card-network equivalent, smart-account limits are the minimum, and the caps should be set on the assumption that every dollar under them is already lost. For everyone else, which means agents buying API credits, SaaS seats, or cloud capacity, there's no reason to choose a rail without reversals.

The cap is the most an attacker can take

If an agent has to move money, put it on rails that can refuse a payment and reverse one. If it has to move crypto, assume the cap is what you'll lose, because a stranger with a free NFT and a Morse code chart can try to take it.

That Morse code message was worth about $174,000, and the only control that worked was the attacker deciding to send the money back.

Your AI Coding Dashboard Counts the Wrong Thing

Michael Tuszynski — Mon, 05 Oct 2026 19:23:59 +0000

Open the vendor dashboard for any AI coding tool and you'll see acceptance rate near the top. It's the number managers quote in staff meetings. GitHub's own research puts it at roughly 30% of suggestions accepted, and the Copilot metrics API gives you suggestion and acceptance counts per editor, language, and day. So a team sees 30%, the line goes up and to the right, and the renewal gets signed.

That number measures a keystroke. Finance is paying for software that stays shipped. The output measure that belongs on the dashboard is the one no vendor shows by default: of the lines your team accepted, how many were reverted or rewritten within 30 days?

Acceptance rate was validated against how developers felt

Acceptance rate didn't become the headline metric by accident. The GitHub study in Communications of the ACM tested several usage signals and found acceptance rate was the best predictor of perceived productivity. Perceived is the important word there. Developers filled out surveys, and the survey scores tracked how often they hit Tab.

That's a real finding. It's also a finding about sentiment. It never claims the accepted code survived review, passed the next sprint, or stayed out of the incident channel. And GitHub sells the tool. Treat the 30% as a vendor number about engagement, not about output.

The newer tooling has the same blind spot one layer up. Dash0's AI Coding Insights lets you sort every Claude Code session by cost or error count and read the full transcript. That's useful for debugging an agent. But cost and errors-per-session both describe what went into the work. Neither tells you what happened to the code after the session closed.

Rework is where AI-assisted code shows its real cost

The independent evidence points the same direction. GitClear analyzed 153 million changed lines and projected code churn, meaning lines reverted or updated within two weeks of being written, to double in 2024 against its 2021 pre-AI baseline. GitClear sells code analytics, so discount it too. But the 2024 DORA report, which has no product to sell here, found that a 25% increase in AI adoption was associated with an estimated 7.2% drop in delivery stability. And METR's randomized trial found experienced open-source developers were 19% slower with AI tools while believing they were 20% faster.

Put those next to the CACM result and you get the core problem. The metric vendors chose tracks how developers feel. The independent studies that measured outcomes found instability, rework, and a perception gap. Acceptance rate sits on the wrong side of that gap.

The $80K AI coding pilot post priced the senior review hours on the cost side. This is the matching hole on the output side. A suggestion that gets accepted on Tuesday and rewritten on the following Monday shows up on the dashboard as a win. Then it shows up again as new work.

You can measure 30-day rework from git history you already have

You don't need a vendor for this. Git already records who wrote every line and when it changed. The method:

Take every non-merge commit that is at least 30 days old.
Tag it AI-assisted or not. Claude Code adds a Co-Authored-By: Claude trailer to its commits by default, so the tag comes free.
Count the lines each commit added.
Run git blame at the repo state 30 days later and count how many of those lines are still attributed to the original commit.
Rework rate = 1 − (kept ÷ added), computed separately for each cohort.

#!/usr/bin/env python3
"""30-day rework: share of added lines gone or rewritten 30 days later."""
import subprocess
from collections import Counter

DAY = 86400

def git(*args):
    return subprocess.run(["git", *args], capture_output=True,
                          text=True, check=True).stdout

def added_lines(sha):
    files = {}
    for row in git("show", "--numstat", "--format=", sha).splitlines():
        added, _, path = row.split("\t", 2)
        if added != "-" and int(added) > 0:   # skip binaries
            files[path] = int(added)
    return files

def kept_lines(sha, path, rev):
    try:
        out = git("blame", "-w", "-M", "-C", "--line-porcelain", rev, "--", path)
    except subprocess.CalledProcessError:
        return 0   # file deleted or moved out of reach: counts as reworked
    return sum(1 for line in out.splitlines() if line.startswith(sha))

def rev_after(sha, days):
    ts = int(git("show", "-s", "--format=%ct", sha))
    return git("rev-list", "-1", f"--before={ts + days * DAY}", "HEAD").strip()

def main(since="120 days ago"):
    log = git("log", "--no-merges", f"--since={since}",
              "--until=30 days ago", "--format=%H%x09%B%x00")
    totals = {"ai": Counter(), "human": Counter()}
    for entry in filter(None, (e.strip() for e in log.split("\x00"))):
        sha, body = entry.split("\t", 1)
        cohort = "ai" if "co-authored-by: claude" in body.lower() else "human"
        rev = rev_after(sha, 30)
        for path, n in added_lines(sha).items():
            totals[cohort]["added"] += n
            totals[cohort]["kept"] += kept_lines(sha, path, rev)
    for cohort, c in totals.items():
        if c["added"]:
            print(f"{cohort}: {c['added']} added, "
                  f"{1 - c['kept'] / c['added']:.1%} reworked by day 30")

if __name__ == "__main__":
    main()

The -w -M -C flags tell blame to ignore whitespace and follow lines that moved within or between files. Without them, a refactor that only reformats or relocates code gets counted as rework. The 30-day cutoff on --until matters too. A commit from last week hasn't had its 30 days yet, and counting it would make every cohort look stable.

The objection: some rework is just healthy iteration

Here's the best case against this metric. Code gets rewritten because teams learn. A spike gets thrown away on purpose. A first pass at an API gets reshaped once the second caller shows up. If you punish 30-day rework, you punish exploration, and engineers will learn to stop committing early.

That objection has a specific mechanism, and the two-cohort design deals with it. Healthy iteration happens to human-written code too, in the same repo, under the same review culture, against the same product churn. Reading a single rework rate on its own tells you nothing. The signal is the difference between the AI cohort and the human cohort over the same window. If both sit at 15%, AI isn't adding rework and the acceptance-rate story holds. If the AI cohort sits at 30% against a human 15%, AI-assisted code is being rewritten at twice the human rate, and the vendor dashboard counted every one of those lines once, as a win.

Don't turn it into a per-engineer scorecard, either. It's a cohort measure for a renewal decision. Once it's used to grade people, people start gaming the trailers.

Where this measurement breaks

The script has real limits, and you should know them before you put the output in front of a CFO.

Squash merges erase the attribution. If your team squash-merges PRs, the trailer survives only when it lands in the squash message. Many merge tools drop it. Check a few merged PRs by hand before trusting the cohort split.

Inline completions leave no trailer. Copilot-style Tab completions get mixed into human commits without any marker. For those tools, compare whole-repo rework over the 90 days before rollout against the 90 days after. It's noisier, and it's still more honest than acceptance rate.

Generated files distort the count. Lockfiles, snapshots, and codegen output get rewritten by machines on every run. Exclude them with a pathspec, or the cohort with more dependency bumps will look worse for reasons that have nothing to do with AI.

Thirty days is a choice. GitClear uses two weeks. Thirty days catches the rewrite that comes when the second caller shows up, which two weeks often misses. Pick a window, write it down, and don't change it between quarters.

None of these limits rescues acceptance rate. Every one of them is a reason to measure carefully. None is a reason to go back to counting keystrokes.

The number to bring to the renewal meeting

Go back to the dashboard. It says 30% of suggestions were accepted. That figure was validated against how developers felt, and the vendor selling the seats reports it.

Ask the question finance should be asking: of that 30%, what fraction was still in main on day 30, and how does that compare to the code your engineers wrote without help? It comes down to two cohorts and one ratio, kept ÷ added, and you can compute it from your own git log this afternoon.

Forward Deployed Engineer Names Four Different Things. One of Them Is a Software Product.

Michael Tuszynski — Wed, 30 Sep 2026 21:50:36 +0000

Forward Deployed Engineer Names Four Different Things. One of Them Is a Software Product.

The same three words now label a job at one frontier lab, a different job at the same lab, a billion-dollar organization at AWS, and a piece of software you can turn on inside Palantir Foundry.

That last one deserves a second read. Palantir invented the role, and Palantir now ships AI FDE, which the docs introduce as "the AI-powered forward deployed engineer," an "interactive agent that operates Foundry for you through conversational commands."

So when two people say "we need an FDE on this," there is no longer any guarantee they are discussing the same category of thing, let alone the same scope of work. That's a scoping problem, and scoping problems are expensive for whoever is holding the statement of work when they surface.

Eight years in presales left me with one reflex above all others: get the boundary of the work written down before anybody starts building. This title needs that treatment.

The role began as an answer to an information problem

Palantir popularized the term and had it in use for staff by 2009. The phrasing is borrowed from the military's forward-basing language. What it solved was a problem about information rather than cost.

You cannot write a specification for software that has to run in an environment you have never seen. Undocumented schemas. Compliance rules nobody wrote down. A workflow that lives in the heads of six people in one building. An architect who visits, produces a design, and flies home hasn't transferred knowledge — they've transferred a guess, and the guess fails in integration.

The fix was to stop splitting the person who learns the environment from the person who writes the code. Same engineer, both jobs, on site, accountable through production.

That original definition has four parts worth separating, because they behave very differently under automation:

Learn an environment that isn't documented.
Decide what to build in it.
Build it against the real data.
Own it in production, or hand it over with the knowledge attached.

What the software version actually does is step three

Read Palantir's capability list rather than the product name. AI FDE handles data integration and pipeline modification, data connection management, ontology editing, function writing and testing, read-only exploration, governance and access auditing, and React application development.

Every item on that list is platform operation. Each one presupposes two things: that somebody has already decided what should be built, and that the target is Foundry. On day one of a real deployment, neither is true.

To Palantir's credit, the documentation claims nothing more than that. There is no line on that page about replacing human FDEs — the product is scoped as a way to operate the platform faster, which is exactly what it appears to be. The overreach isn't in the docs. It's in reading the product name as a description of the role.

Which makes step three the honest answer to "what can AI do here." Building against real data is getting dramatically faster. Steps one, two and four are close to untouched, because none of them are typing problems. Nobody has automated walking into a business you don't understand and working out which of the things people are complaining about is the one worth building.

Two job titles at one company already disagree

Inside OpenAI the split between step three and the rest is visible in the org chart.

The Forward Deployed Software Engineer posting wants 7+ years of full-stack work and describes coding "side-by-side" with customer teams "on their infrastructure," designing abstractions that later engagements reuse. That's steps three and four, with the original accountability intact.

The Forward Deployed Engineer posting owns "discovery, technical scoping, system design, build, and production rollout," plus sequencing, tradeoffs between scope and speed, playbook authoring, and field feedback that "changes product and model roadmaps." It contributes code "when progress or clarity depends on it." Conditional, by design.

Both are reasonable jobs. They are not the same job, and the difference is which of the four steps the person is accountable for. A team that hires the second expecting the first will be surprised, and so will the customer.

The category got funded faster than it got defined

Anthropic and OpenAI stood up forward-deployed groups in May. On June 30, AWS committed $1 billion to a Forward Deployed Engineering organization it describes as "agentic-first," compressing "timelines from months to days," and designed so "customers are self-sufficient when a deployment ends." Two days later Microsoft announced Frontier Co. with $2.5 billion and 6,000 embedded engineers.

Treat the months-to-days figure as a vendor ceiling rather than a planning number; no independent measurement of it exists yet. But the pattern is clear enough without it: billions of dollars of organizational design arrived in about eight weeks, and shared vocabulary takes longer than that.

The market's own uncertainty shows up in what it pays. Two frontier labs are hiring for this title in the same city right now, and their posted base ranges start $95K apart — $185K at one, $280K at the other. That's not an argument about whether the role is valued. It's a straightforward signal that the scope behind the words hasn't converged.

The strongest objection is that this is genuinely new work

a16z's Tom Hollands argued in January that forward-deployed titles describe work that would have been unrecognizable five years ago, and he's right about the mechanism. An AI system's capabilities only resolve once you build against the customer's real data, which breaks the design-it-then-hand-it-off sequence and collapses steps two and three into one person. His test for whether a new title is doing real work: does it describe something that didn't exist before?

Apply that test to the mechanism and it passes. Apply it to the vocabulary and the picture changes, because the same three words now cover step three alone, all four steps, an organizational strategy, and a Foundry feature. New work deserves a new name. It also deserves a definition, and the second part hasn't happened. Worth noting that a16z funds many of the companies staffing these teams.

The scope belongs in writing before the kickoff

Not a candidate checklist — a scoping checklist, equally useful from either side of the table:

The artifact. What gets shipped, named outright: an MCP server, a data pipeline, an application running in the customer's stack. "Scopes of work and project plans" is a real deliverable too, but it's a different one.
The requirement owner. Who decides what gets built. If that's the customer, this is step three and can move fast. If it's the deployed engineer, steps one and two are in scope and the timeline is a different shape entirely.
The production owner at exit. The original role's defining trait was that whoever mapped the problem answers the page six months later. "Self-sufficient when the deployment ends" is a legitimate model, and a different commitment. Both are fine. Assuming one and getting the other is not.
The platform boundary. Whose stack, with what access, and which parts an agent can operate directly. This is where the software version of the title is genuinely useful, and where its limits are hard edges rather than preferences.

Where this argument runs out: a two-year-old title is supposed to be unsettled, and dispersion this early proves nothing about anyone's intentions. Posted ranges are base only, and equity is undisclosed, so the $95K figure describes disclosure practice as much as scope. And a single vocabulary may never arrive — "DevOps" never fully converged either, and teams still shipped, because the good ones wrote down what they meant locally.

That last part is the actual ask. Not an industry standard. Just a shared sentence, written before the kickoff, naming which of the four steps the forward deployed engineer owns. The role was invented because a handoff document couldn't carry the knowledge. It would be a strange outcome if the title itself became the thing nobody could hand off.

Your Agent Is Paying Opus Prices for Yes/No Questions

Michael Tuszynski — Mon, 28 Sep 2026 02:47:35 +0000

An agent loop makes dozens of small calls that aren't reasoning at all. Is this ticket about billing. Is this search result relevant. Did the tool call succeed. Should this story be a blog post. Most agent harnesses send every one of those to the same frontier model that writes the code, with a prompt that ends in "return JSON."

I measured what that costs. The same six-question classification, run 150 times per model: a System One model answered in 163 ms at $53 per million calls. Claude Opus 5 took 2.7 seconds and $11,183. That's roughly 17x the latency and 200x the price for a yes/no question.

But the savings only survive if the loop around the model owns the thresholds. My data says that part is harder than the model swap.

A System One model answers in types, not text

The name comes from Kahneman's split between fast, intuitive System 1 and slow, deliberate System 2. TypeSafe's Jev is the first model sold under that label. It takes a state (text or JSON) and a set of typed questions, and it returns three kinds of answer: a choice from options you list, a score against a rubric you write, and a noul, the probability that a statement is true.

It doesn't write text. It doesn't call tools. TypeSafe's own docs say it plainly: Jev is not a drop-in replacement for the LLM behind Claude Code or Cursor. You keep the LLM for the work that needs one and call Jev for the judgments.

That's the same argument I made in April about where agent performance comes from: the loop matters more than the weights. A System One model is a new part to put inside the loop.

Code computes, System One judges, the LLM generates

Three layers, and a question belongs to exactly one of them.

Layer	Owns	Examples
Code	Anything with an exact answer	counts, date math, thresholds, combining scores
System One	Bounded judgments with a closed answer set	route this ticket, is this vendor marketing, how durable is this story
LLM	Generation and multi-step reasoning	write the draft, call tools, plan a fix

TypeSafe publishes where Jev breaks, which is more than most vendors do. The jaggedness page for jev-1.13 lists arithmetic, date comparison, double negatives, oversized state and adversarial text. Every one of those is a boundary in the table above. Push arithmetic down into code. Push anything open-ended up to the LLM.

Worked example: scoring blog topics

The content engine behind this blog pulls about 40 stories a day from TechMeme, Reddit and Firecrawl, and a regex scorer decides which ones are worth writing about. In September I wrote a Jev version of that scorer. One request asks six questions about each story:

substance: {
  type: "noul",
  instructions:
    "Does this contain something concrete to react to — a claim, an outcome, a number, an incident, a decision someone made — as opposed to asking the reader for help or opinions?",
  // criteria: { true: "...", false: "..." }
},

The weights live in code, not in the prompt, so retuning them re-ranks the backlog without spending a token:

const gate = WEIGHTS.substance_floor + (1 - WEIGHTS.substance_floor) * j.substance;
const normalized = merit * gate - WEIGHTS.vendor_fluff * j.vendor_fluff;

The first shadow run, read-only next to the regex scorer, promoted three r/devops help threads to long-form. "Do you really test your backup" scored 2.9 of 3 on beat fit and 0.83 on operator angle. Jev was right about both. A backup question from a practitioner is squarely on beat. It's also not a blog post, because it asks instead of reports. The judgment was fine. My weighting was wrong, and the fix was a substance question applied as a multiplier rather than another additive term. The Jev scorer lives on a branch and only runs in shadow when I invoke it; the regex scorer is what ships today.

The numbers, and where the savings leak

I ran the six questions against 50 recent stories, three times each, on Jev, Claude Haiku 4.5 (no extended thinking) and Claude Opus 5 at low effort. Same wording, same criteria, Claude held to the same schema with structured outputs. Total spend: $1.93.

	Jev 1.13	Haiku 4.5	Opus 5 (low)
Median latency	163 ms	1,156 ms	2,740 ms
Cost per 1M calls	$53	$1,610	$11,183
Tier changed across its own 3 runs	2 of 50	20 of 50	5 of 50

Prices come from TypeSafe's models page ($0.042 per million input tokens, output free) and Anthropic's pricing page. The Haiku column surprised me. It's the default "cheap classifier," and it was the least consistent model in the test, flipping its own tier on 20 stories out of 50. Prompt caching won't rescue it here either: this prompt is about 1,300 tokens and Haiku 4.5 won't cache anything under 4,096. Opus 5 caches from 512 tokens, which brings it to roughly $3,000 per million calls if every call hits the cache. Still about 57x Jev.

Jev agreed with Opus on 274 of the 300 individual answers. It also landed 26 of 50 stories in a different tier.

Both are true because Jev scores the whole set higher (median composite 1.73 against Opus at 1.25) while ranking the stories in roughly the same order. My tier cutoffs sit where the old scorer's numbers used to cluster. Move to a model with a different baseline and half the backlog changes tier. Haiku and Opus disagreed on tier for 20 of 50 stories too, so this isn't a Jev defect. Tier cutoffs belong to the model they were tuned on. A model swap is a change to the agent harness, not a config flag.

The obvious fix is confidence routing: let Jev answer, and send low-confidence cases to Opus. Jev returns a confidence with every score and choice. The naive rule, "escalate the story if any answer is below 0.8," sent 48 of 50 stories to Opus, because a four-level rubric rarely puts 80% on a single level. Durability's median confidence was 0.65.

TypeSafe warns against carrying one threshold across question types, and the data agrees. A per-type rule (escalate when the choice confidence is below 0.8 or any noul lands between 0.3 and 0.7) sent 24 of 50 stories up. That costs $5,421 per million, 2.1x cheaper than Opus alone, and it catches 17 of the 26 answers where the two models disagree. Real, but nowhere near 200x. On a workload where half the stories are genuinely ambiguous, escalation eats most of the savings.

Where this breaks

Durability was the weakest question. It caused 12 of the 26 disagreements, and on 8 stories Jev split its probability between two levels at least two steps apart. The score it returns is an expected value, so a story that's either a one-week event or a durable shift comes back as "somewhere in the middle." TypeSafe's docs say not to treat that number as a precise magnitude. My composite divides it by three and adds it in, which is exactly what they say not to do.

There's no answer key in these numbers. Agreement with Opus is not accuracy, and 50 stories from one domain is a small sample. State is also data Jev doesn't treat as hostile by default, which matters for an agent that reads web pages. And the rate limits on the models page are "adjusting dynamically," which is a sentence to read twice before routing production traffic through one vendor.

What to change on Monday

Grep your agent harness for model calls whose output is a label, a score or a boolean. Those are System One questions wearing an LLM's price tag. Shadow them against a System One model for a week, read-only, and log both answers.

Then do the part the benchmark says is hard. Retune your cutoffs on the new model before you switch, set confidence thresholds per question type, and escalate only where the answer can change the decision. The model is the cheap part now. The 26 misfiled stories came from my loop, not from Jev.

262 Nights, Zero Bytes, Exit 0: The Failure Mode of Unattended Agent Runs

Michael Tuszynski — Thu, 24 Sep 2026 18:33:36 +0000

262 Nights, Zero Bytes, Exit 0: The Failure Mode of Unattended Agent Runs

A backup job I run synced an empty directory every night for 262 consecutive nights. Every run exited 0. Every run wrote Photo library sync complete into the log. The off-site copy held zero objects for nine months while the real library sat at 17 GB and 17,658 files on a disk nothing was copying.

The script's source path pointed somewhere that did not exist. Docker's bind mount then created that path, empty, at the start of every run. The job manufactured the evidence that its own configuration was valid, found nothing there, and synced the nothing it found. A guard like [ -d "$source" ] would have passed all 262 times.

I found it by counting objects in the bucket by hand, on an unrelated errand. No monitor caught it, because there was nothing for a monitor to catch. The job ran on schedule. It logged. It exited clean.

That job is what I think about now when a vendor shows me an agent that works unsupervised for two days.

An exit code answers a different question than the one you asked

AWS now sells what it calls frontier agents: autonomous systems that, in its own words, "run persistently for hours or days without constant human oversight". The Security Agent bills $50.00 per task-hour, and AWS puts an average evaluation at 24 task-hours, so roughly $1,200 a run. I wrote about that pricing model in April and I still think the unit economics are sound.

The economics are not the problem. Supervision is.

When a run ends, what comes back is a status: it finished, or it didn't. One bit. That bit reports whether the process reached its final line, which is a different question from whether the work happened. My backup reached its final line 262 times.

For a five-minute job you paper over the gap by looking. You read the output, you spot-check the result, your own attention is the instrument. Stretch the same job to 24 unattended hours and that instrument is gone. What's left is the bit.

Success decays exponentially with run length

A 24-hour run isn't a stretched-out 1-hour run. The failure math works against you, and it compounds.

Toby Ord's analysis of agent benchmark data, Is there a half-life for the success rates of AI agents?, fits the results to a constant rate of failure per minute of work. That produces exponential decay: success probability falls off a cliff as tasks get longer, and every agent has a characteristic half-life. Failures accumulate across subtasks, and one bad subtask sinks the run.

METR measures the same curve empirically. Its time-horizons work reports the task length at which frontier models succeed 50% of the time, and that 50% threshold is chosen because it's the easiest point to estimate robustly. At the horizon being advertised, the coin is fair.

Then read the caveat on METR's own page. "Measurements above 16 hrs are unreliable with our current task suite."

The independent yardstick runs out at 16 hours. The vendor's advertised operating range is hours or days. Past that line there is no public, non-vendor measurement of how often these runs actually succeed. You are not choosing between a measured 94% and a measured 60%. You're choosing between a vendor number and no number.

My own agent measurements lied in the same direction

This isn't a hypothetical I imported from a paper. My own eval tooling has handed me clean results that were nothing of the kind.

A model evaluation I ran logged a 900-second timeout and a 766-second near-timeout for two coding tasks. The CSV was indistinguishable from a model that cannot do the work. A second agent had been pushing large prompts at the same single-slot inference server the whole time. Re-run on a quiet box, the same model and the same tasks passed in 57 and 83 seconds.

A separate eval launched into a detached tmux session returned two FAIL rows in under a second, with zero-byte logs. The binary wasn't on the PATH that a daemon-spawned shell inherits. That output is indistinguishable from a model that errors instantly.

Both runs produced a result file. Both exited. Neither measured what it claimed to measure, and in both cases the artifact looked exactly like a real finding, which is the part that costs you. A crash you notice. A plausible wrong answer goes into the spreadsheet.

The same pattern shows up wherever I stopped watching. A weekly link checker reported Scanned 0 wiki pages / Dead wikilinks: 0 for six weeks, green every time, because it pointed at a directory a migration had retired. Repointed, the same code immediately found 261 pages and 116 dead links. A watchdog probing an embedding server's /api/version endpoint stayed green for two days while every actual embed call returned a 500 and the index took in nothing.

A 94% accuracy score is computed over the runs somebody checked

The strongest objection is that this is all anecdote against data. AWS reports up to 75% lower MTTR, 80% faster investigations, and 94% root cause accuracy from preview customers, and one university cut incident resolution from two hours to 28 minutes. Those are outcomes, not vibes.

Look at how that number gets built. A root-cause accuracy figure is computed over runs whose root cause somebody established independently. To score a run, you need ground truth, which means a human went and checked. The denominator is runs that were verified.

The failure class I'm describing never enters that denominator. A run that reports success, produces a clean artifact, and is wrong is scored as correct unless somebody goes looking, and the whole premise of an unattended agent is that nobody does. My backup would have scored 262 for 262 on any accuracy metric derived from its own output.

These are also first-party numbers from preview customers selected by the vendor. That's not disqualifying. It does mean they're a ceiling rather than an expectation, and they answer "how often is a checked run right," not "how often does an unchecked run lie."

Assert the artifact, not the exit status

The fix isn't more monitoring. It's changing what gets asserted. The backup is the shortest example I have:

# Before: the job's own bind mount created $SRC empty, so this always passed.
[ -d "$SRC" ] || { echo "missing source"; exit 1; }
rclone sync "$SRC" "$DEST"

# After: refuse an empty source, and log what actually moved.
[ -n "$(find "$SRC" -type f -print -quit)" ] || {
  echo "REFUSING: $SRC is empty — not syncing a deletion"; exit 1; }
rclone sync "$SRC" "$DEST"
rclone size "$DEST"   # object count and bytes, into the log, every single run

Refuse a degenerate input instead of processing it. rclone sync mirrors deletions, so a wrong path doesn't merely fail to back things up, it erases the copy you had. Emit a number that a zero-work run cannot produce. And push the heartbeat only when a leg moved a non-empty source, so a silent no-op goes stale and pages somebody.

For an agent run the shape is identical. Don't ask whether the run finished. Ask what it changed: how many findings, how many files touched, how many tests that were red are now green, how many tokens against how many accepted diffs. Then set a floor. A 24-hour pen test that reports zero findings and zero requests issued has told you about your instrumentation, not your security posture.

The cheap version of this test: can this check ever come back red? If a run that did nothing would produce output identical to a run that worked, you don't have a check. You have a log line.

Where this doesn't apply

Exploratory work has no assertable artifact. When you turn an agent loose on "look at this system and tell me what's odd," there's no count to floor and no diff to measure, and you're back to reading the transcript. Outcome assertions don't help there, and pretending otherwise gives you a number that means nothing.

The threshold is unattended duration, not autonomy. A 20-minute agent run you watch needs none of this. An hour-long cron job nobody has opened since 2025 needs all of it. Agents make it urgent because the unattended part is the product, sold at durations past where anyone outside the vendor has measured the success rate.

262 nights is the number I'd put in front of anyone signing off on a Friday-to-Monday agent run. Not because a backup script is an agent, but because it's the floor: a deterministic shell script, no model in the loop, nothing nondeterministic about it, wrong every night for nine months while reporting success. Add nondeterminism and a 50% time horizon on top of that and the exit code is worth less, not more.

Decide what a successful run must produce, in a number, before you let anything run for a day. Otherwise you'll do what I did, and find out by hand, on an errand, in month nine.

6 Agent Licenses, 1 Runtime: You Are Paying Per Seat for the Layer That Is Not Scarce

Michael Tuszynski — Mon, 07 Sep 2026 20:21:20 +0000

Six licenses. One runtime. That's the shape of most agent quotes on the table right now, and the invoice is inverted: the six licenses are the big line, and the runtime, the thing that actually consumes scarce compute, is either a rounding error or a separate purchase order nobody brought to the renewal meeting.

Microsoft published the split more clearly than anyone else did. The E7 bundle lists at $99 per user per month. To be fair to that number, it buys the whole suite, Office and Teams and the security stack, with agent capability folded in; it is the cleanest public example of the split, not a pure agent-seat price. SAMexpert's breakdown of the license puts the split plainly: that layer grants agent capability, it does not run them. Execution bills separately, in Agent Compute Units, through a pre-purchase plan sold in blocks. Two layers, two meters, one signature.

Treat that as the reference architecture for every agent quote you get this year, whoever the vendor is.

The seat layer and the runtime layer are priced by different physics

The interface layer is software. Marginal cost of a seventh seat: approximately zero. The vendor's cost doesn't move when you add a user who opens the panel twice a week.

The runtime layer is electricity, silicon, and a queue. Marginal cost of the ten-thousandth agent run is real, and it's the only part of the stack where the vendor is buying something scarce from someone else.

So when a quote grows by multiplying the free-to-produce layer and holds the expensive layer flat or unbundled, that's not a pricing accident. That's the whole design. You're being metered on the thing that isn't scarce.

Run the six seats: 6 × $99 = $594/month, $7,128/year, before a single agent executes anything. Now ask how many of those six drove an agent in the last 30 days. If it's two, one platform engineer and one person in support automation, the real cost per active agent user is $297/month, and the other four seats bought a login.

"Per seat" hides decisions your contract already made

The phrase does a lot of undisclosed work. Per-seat licensing is an umbrella covering separate decisions about what the unit attaches to, what time window counts it, and which type of user holds it. Vendors pick one combination, print two words, and let you assume you know which one you bought.

Two of those decisions bite agent deals in particular. Named-user counts never go down mid-term; concurrent counts can, so ask which one is in your paper. It's usually in a definitions exhibit, not the order form. And the service identity your agent runs under may be counted as a seat even though no human ever touches it. That one line can double a bill.

The runtime is getting cheaper faster than the seat is

The direction of travel is the part procurement keeps missing. Inference prices have fallen with every model generation, and a vendor's runtime rate card follows them down inside a contract term. Per-seat list prices move once a year, and not downward.

If a seat price was originally set to recover runtime cost, and runtime cost drops 60% over a term, the seat price should move. It won't unless you make it a renewal condition. Put a benchmark clause in: seat price adjusts with the vendor's own published runtime rate card, or you get the delta as runtime credits.

What the seat is actually buying, and why that argument doesn't survive contact

The strongest counterargument is real, so take it seriously: the seat isn't paying for compute at all. It pays for identity and SSO, audit logging, data-loss prevention, admin policy, connectors into your existing data, and a support contract with a throat to choke. Those are per-human costs. Charging per human is honest.

Fine. Then price them as such, and watch the argument come apart on its own mechanism.

Governance costs scale with directory objects and policy scope, not with who happens to use an agent. If audit and DLP are what the seat buys, they should survive the removal of the agent entitlement. So ask for that quote: the seat, priced without agent capability. Most vendors won't produce it, because the number would be embarrassing next to the one with the agent checkbox on.

The gap between those two quotes isn't governance. It's a right to consume, sold as a subscription. And a right to consume, when the consumption is metered separately anyway, is a toll booth on a road you already paid to drive.

The vendor's own bundle proves this. E7's seat layer explicitly doesn't execute agents. The execution meter is a different SKU with a different plan. Every argument that the seat covers runtime is refuted by the price page it came from.

The worksheet to take into the renewal

Split the quote before you sign it. It fits in your head. Do it on paper anyway, before you sign:

// renewal-worksheet.ts — split the bill by layer before signing
const quote = {
  seats: 6,
  seatPrice: 99,          // $/user/month — interface layer, ~$0 marginal cost to vendor
  activeAgentUsers: 2,    // drove an agent at least once in last 30 days
  runtimeSpendMonthly: 0, // metered execution, separate PO
};

const seatBill    = quote.seats * quote.seatPrice;              // 594
const perActive   = seatBill / quote.activeAgentUsers;          // 297
const runtimeShare = quote.runtimeSpendMonthly /
                     (seatBill + quote.runtimeSpendMonthly);    // 0.00

// If runtimeShare < 0.25, you are buying interface, not capability.

That last line is the test. When less than a quarter of your total agent spend is execution, the vendor has moved the money to the layer where scarcity doesn't apply, and your growth path is buying more logins rather than more work.

In the agent pipeline I run there is no seat at all. The interface is a terminal and a queue directory, and every dollar is metered execution: drafting this post cost $1.11 in model calls, and a second operator would have added nothing to the bill.

Where this argument breaks

Three places, honestly.

Small teams under ten people often can't get consumption pricing at all. Vendors gate metered plans behind enterprise minimums, and a flat seat price is the cheaper administrative outcome even when the unit economics are ugly. Take the seats and stop optimizing.

Regulated buyers who need contractual data residency, retention guarantees, and a named DPA are buying something the runtime layer doesn't include. That's a legitimate per-tenant cost that a per-seat model approximates badly but does cover.

And if your agent usage is genuinely uniform, every seat driving the runtime every day, the two models converge and the seat is just a prepaid meter with less paperwork. Usage is rarely that flat. The common shape is a few heavy operators and a long tail of people who opened the panel twice.

None of that changes the ask. Get the runtime quoted as its own line, in its own units, with its own rate card. Then look at the two numbers side by side.

Six seats at $99 is $7,128 a year. Ask your vendor what the runtime under those six seats cost last quarter. If they can't tell you, they're not selling you compute. And if they can, you finally know which of the two numbers to negotiate.