DEV Community: Michael Tuszynski

Every FDE Pod Has a Strategist. None Has a Grader.

Michael Tuszynski — Tue, 13 Oct 2026 14:31:07 +0000

Gartner forecasts that 70% of enterprises will abandon agentic AI built by vendor forward-deployed engineers by 2028, in a September 29 release reported the next day by The Register and Channel Dive. The firm calls the pattern "FDE washing." Its analyst counts about 2,000 active FDEs against roughly four times that demand, and Gartner estimates that FDE consulting fees alone could top $200,000 per quarter per use case.

Gartner's stated mechanism is dependency and exit: clients end up unable to run or leave what the vendor built. How the pod is designed is not part of its analysis. This post argues the dependency has a structural cause visible in the published FDE pods. The pods that name their roles each have a strategist, and none has a grader. No seat checks the work against criteria the client agreed to, held by someone who had no hand in building it. That link is this post's argument. Gartner doesn't make it.

The pod is already the standard delivery model

The lone engineer embedded at a client is mostly a recruiting image. The published delivery unit is a pod, and the pods look alike.

Palantir pairs the Delta (the FDE) with an Echo (a deployment strategist) and Devs on the product side. One alumnus describes FDEs going on site at the customer three to four days a week. Deloitte's lead-FDE postings describe pods of two to five onshore-anchored, offshore-supported engineers. Thoughtworks' principal-FDE posting describes a pod of one product strategist, one experience designer and two forward-deployed engineers. AWS's billion-dollar FDE organization deploys pods of five or six people, embedded inside a single customer at a time. Anthropic's own postings pair FDEs with a technical deployment lead who won't write production code and owns the roadmap the engineers execute against. The shape repeats: one or two engineers, a lead who owns scope and the executive relationship, sometimes a designer, and offshore or platform support behind them. Postings change, so the specifics will drift. The shape hasn't.

The supply side is about to scale this template. Anthropic's Claude Frontier Academy commits $100M to train 10,000 "Frontier Deployed Engineers" by the end of 2027 at partners including Accenture, Deloitte, McKinsey and Morgan Stanley. Accenture's statement says those FDEs will work "inside client workflows as part of multidisciplinary teams." If the partners follow their own postings, those teams will look like the ones above.

Every seat in the pod helped build the thing

Read those pod descriptions for a quality seat and you won't find one. None of them names a QA seat, an acceptance seat or a grading seat. The closest is a mention of "quality gates" in Deloitte's posting.

The strategist seat is already standard. Palantir calls it the Echo. Other firms call it a deployment lead, a product strategist or a PM. Pods don't need anyone to invent it. And every seat in every pod is a builder. Engineers write the code. The strategist sequences the roadmap. The designer shapes the workflow. Platform and offshore support keep it running. Each person present has a stake in the result passing.

Verification gets dropped because nobody is waiting on it

The earlier post on the FDE title broke the role into separate jobs. Here's the version that matters for pods. The role has four jobs: build the system, redesign the process around it, run the client relationship, and verify the result. A pod splits the first three across seats. The fourth goes unassigned.

It gets dropped inside a pod for the same reason a lone engineer drops it. No one is waiting on the grade. Shipping has a demo date, and the executive relationship has a steering meeting. A verification report has no audience until something breaks. Everyone in the room also wants the work to pass, including the vendor's lead, whose quarterly fee renews on the impression that it did.

Follow that to the client's side. Without an independent grade against agreed criteria, the client can't tell what it bought. Without a sponsor who has authority to stop the work (the one seat the kill sheet from the earlier post insists on), the client can't leave without losing whatever value is in there. It can't separate the working parts from the broken parts, so it keeps paying the only people who know. That is dependency, and that's what Gartner is forecasting.

The missing seat grades and does nothing else

Call it the acceptance owner. The definition is narrow on purpose:

Co-writes testable acceptance criteria with the client sponsor before work starts
Grades the outcome against those criteria after every change
Publishes the grade to the sponsor, not to the pod first
Had no hand in building the work

The seat can sit on either side of the contract. A mature client's own QA function can hold it. A third party can. A vendor employee from outside the delivery org can, if the client accepts that. What disqualifies someone is having built the work. Which side signs their paycheck matters less.

Here's the cadence relative to the pod:

Week	Pod	Acceptance owner	Sponsor
0	Scopes the use case	Co-writes criteria with sponsor; each one testable pass/fail	Signs criteria and stop conditions
1-5	Builds, demos weekly	Re-runs the criteria after each merged change; logs pass/fail	Reads the grade before the demo
6	Mid-sprint review	Publishes trend: criteria passing, failing, untested	Decides continue, re-scope or stop
7-12	Builds, hardens	Same loop; adds regression on previously passed criteria	Reads the grade weekly
13	Quarter close, renewal	Final grade against week-0 criteria, unchanged unless sponsor re-signed	Renews or exits on the grade

Each seat also has one thing it must not do. The engineer doesn't grade their own work. The strategist can write down what done means but doesn't own the grade. The sponsor never hands the stop decision to someone else. The acceptance owner's rule is the least obvious one: don't suggest fixes. The moment the grader proposes the fix, the grader has helped build it.

Owning the criteria doesn't make the grade independent

The strongest objection is that the pod already has this. Palantir's Echo, or a deployment lead who owns the roadmap, writes down what done means.

That person owns the criteria. They also own the scope, the sequencing and the executive relationship, and they answer for whether the engagement renews. When a criterion half-passes in week 11, the person who wrote the roadmap is also the person who explains to the sponsor why it's close enough. The objection fails on that mechanism. Authorship and grading sit in the same chair, and that chair is the one with the most riding on renewal.

The second objection: the client's own QA team can do this. Often true. When it can, that team is the seat, provided it didn't help build the system. The test is simple. If the client's QA engineers pair with the FDEs during the build, they've crossed over, and someone else has to grade.

A fractional grader costs about 2.5% to 7.5% of a $200K quarter

The acceptance owner doesn't need to be full time. The work is a criteria workshop up front, a re-run after each change, and a weekly report. That fits in roughly 0.1 to 0.2 of a seat.

Illustrative pricing, using Gartner's $200,000-a-quarter estimate:

Pod fee: $200,000 per quarter for one use case
Pod of four seats (two engineers, strategist, designer): $50,000 per seat per quarter
Acceptance owner at 0.1 seat: $5,000 per quarter, or 2.5% of the fee
At 0.15 seat: $7,500 per quarter, or 3.75%
At 0.2 seat: $10,000 per quarter, or 5%

An experienced independent grader may bill above the pod's average seat. Double the rate and 0.15 of a seat costs $15,000 a quarter, or 7.5%.

Now price it against abandonment. A use case run for four quarters at $200,000 costs $800,000. The grader at 0.15 seat costs $30,000 for the same year. If an independent grade ends a failing engagement just one quarter early, it saves $200,000. That covers the seat at $7,500 a quarter for about 26 quarters, more than six years.

The cost of this advice goes beyond the invoice. A grader will fail work the pod considers done. Demos get awkward. Renewals get harder for the vendor. The pod loses some of its speed, because "shipped" now has to mean "passed."

Where the seat doesn't fit

An 8-week proof of concept often can't carry any extra seat, fractional or not. The fix there is cheaper: write the criteria at week 0 and have the sponsor grade at week 8 personally.

Clients who won't name a sponsor break the model entirely. A grade with no one empowered to act on it is a report nobody reads.

The onshore-anchored, offshore-supported pod is the hardest case. On paper it's five engineers. In practice one engineer can end up alone at the client, with the rest of the pod a time zone away. That engineer is the whole on-site build team and the only witness to whether it works. A pod like that needs a grader most and is least likely to have budget for one.

Go back to the criterion that half-passes in week 11. With an acceptance owner, it doesn't get explained in a steering meeting. It shows up in the weekly grade as a fail, with the week-0 wording next to it, and it shows up again in week 12. At week 13 the sponsor holds a sheet with every signed criterion marked pass or fail. That sheet is the seat every published pod is missing. If that line is still red, the sponsor decides whether the next $200,000 buys a fix or funds the exit.

30% More Code, Zero More Shipped Work: Where the Capacity Actually Went

Michael Tuszynski — Mon, 12 Oct 2026 16:29:30 +0000

A Harvard study of 718 firms found that coding agents produced 30% more lines of code and no detectable increase in finished work. Most coverage stops there and reads the zero as a verdict on the tools. The paper's own review data points somewhere else. My reading of that data, and it's my inference rather than the paper's stated finding, is that the extra output was real and the review stage absorbed it. That's an org-design failure, and you can look for it in your own pull request history this week.

The extra code piled up at the review stage

Fiona Chen and James Stratton built the study on Jellyfish data covering 718 firms and about 300 million work events from January 2021 to March 2026. Jellyfish sells engineering analytics. A finding that work stalls somewhere measurable isn't against its commercial interest, so weigh the source with that in mind. The analysis itself comes from academic economists, and the headline result is not the one an AI vendor would pick.

After teams adopted coding agents, the activity numbers went up. Lines of code rose 30%. Commits rose 20%. Pull requests rose 23%.

Jira issue and epic resolution showed no change the authors could distinguish from zero. Their estimate rules out an output gain above 12%.

The review numbers explain the gap. In the same paper:

PR review time rose 49%.
The share of PRs with changes requested nearly doubled.
Comments per PR rose 35%.
The share of workers doing review rose 14%.

Put the two lists side by side. More code went in at the top. Review time, the changes-requested rate and comments per PR all rose, even with 14% more people doing review. Resolved tickets came out the bottom flat. The paper reports these as separate effects and doesn't claim one caused the other. My inference from the pattern: the capacity didn't vanish. It turned into review load.

Review load grew faster than PR count

The paper estimates each effect separately, so stacking them is my illustration, not a result from the paper. But stacking them shows why a 23% bump in PRs can stall a team.

Take an illustrative 20-person team with 6 regular reviewers, 200 merged PRs a month, a 10-hour median review time, 4 comments per PR, and 15% of PRs sent back with changes requested. Apply the paper's effects:

                          before     after      change
PRs / month                  200       246       +23%
Review time (hours)         10.0      14.9       +49%
PR-hours awaiting review   2,000     3,665       +83%   (1.23 x 1.49)
Comments / PR                4.0       5.4       +35%
Total comments               800     1,328       +66%   (1.23 x 1.35)
Reviewers                    6.0      6.84       +14%
Comments per reviewer        133       194       +46%   (1.66 / 1.14)
Changes-requested rate       15%       30%       ~2x
PRs needing a second pass     30        74

PR count rose 23%. PR-hours spent awaiting review rose 83%. That's elapsed time in the queue, not reviewer effort, but it's time finished work sits unshipped. Even with 14% more people reviewing, each reviewer handles roughly 46% more comment traffic. Every changes-requested PR goes back through the queue, and the illustrative team goes from 30 second passes a month to 74.

That's the mechanism as I read it. Agents raised the rate at which work arrives at review. Nobody raised the rate at which review clears it.

Headcount didn't move, and that's the CFO's problem

The paper's headcount results are where the budget conversation should start. Total headcount showed a precise zero effect, ruling out a decline above 2.9%. Engineering headcount showed a noisier zero, ruling out a decline above 13.8%.

A CFO who funds agent licenses expecting to shrink the engineering line didn't get that in this data. My argument, which the paper doesn't test directly: what the money bought was a review backlog. Buying more seats pushes more work into a queue that's already behind.

Measure where your capacity went from PR data

You don't need Jellyfish to check whether your team looks like the paper. The GitHub API has what you need. This script pulls merged PRs, their reviews and their timelines, and prints three numbers per month: time to first review, changes-requested rate, and reviewer share. Time to first review starts at the last ready-for-review event (or PR creation, if the PR never sat in draft) and stops at the first submitted review.

import json, subprocess, sys
from collections import defaultdict
from datetime import datetime
from statistics import median

REPO = sys.argv[1]  # e.g. your-org/your-repo

def gh(path):
    raw = subprocess.run(["gh", "api", "--paginate", "--slurp", path],
                         capture_output=True, text=True, check=True).stdout
    return [item for page in json.loads(raw) for item in page]

def ts(s):
    return datetime.fromisoformat(s.replace("Z", "+00:00"))

def review_start(pr):
    events = gh(f"repos/{REPO}/issues/{pr['number']}/timeline?per_page=100")
    ready = [ts(e["created_at"]) for e in events
             if e.get("event") == "ready_for_review"]
    return max(ready) if ready else ts(pr["created_at"])

months = defaultdict(lambda: {"prs": 0, "cr": 0, "hours": [],
                              "authors": set(), "reviewers": set()})

for pr in gh(f"repos/{REPO}/pulls?state=closed&per_page=100"):
    if not pr["merged_at"]:
        continue
    m = months[pr["merged_at"][:7]]
    m["prs"] += 1
    if pr["user"]:
        m["authors"].add(pr["user"]["login"])
    reviews = gh(f"repos/{REPO}/pulls/{pr['number']}/reviews?per_page=100")
    start = review_start(pr)
    submitted = sorted(ts(r["submitted_at"]) for r in reviews
                       if r.get("submitted_at") and ts(r["submitted_at"]) >= start)
    if submitted:
        m["hours"].append((submitted[0] - start).total_seconds() / 3600)
    if any(r["state"] == "CHANGES_REQUESTED" for r in reviews):
        m["cr"] += 1
    m["reviewers"].update(r["user"]["login"] for r in reviews if r["user"])

for month in sorted(months):
    m = months[month]
    people = m["authors"] | m["reviewers"]
    wait = f"{median(m['hours']):6.1f}" if m["hours"] else "   n/a"
    print(f"{month}  prs={m['prs']:4d}  "
          f"first_review_h={wait}  "
          f"changes_requested={m['cr'] / m['prs']:.0%}  "
          f"reviewer_share={len(m['reviewers']) / len(people):.0%}")

A few caveats. It makes two API calls per PR, so run it on one repo at a time or you'll hit rate limits. PRs merged with no review drop out of the timing median. Reviewer share here means reviewers divided by everyone active in the repo that month, not total headcount. And time to first review is my proxy. It isn't the paper's PR review time metric, which comes from Jellyfish's own definitions, so don't compare your hours to anything in the paper. Compare your percentage change across the rollout to the paper's 49%.

Draw a line at your agent rollout month and read the pattern. If PR count and changes-requested rate both climbed while reviewer share stayed flat, you have the paper's queue. If reviewer share rose and time to first review still climbed, you added reviewers without shrinking the work each one sees. Pair this with the 30-day rework measure from the earlier post on AI coding dashboards, because every fix below trades review time against escaped defects.

Fix review capacity and merge policy before buying more seats

Start by making review planned work. A reviewer rotation on the sprint board, with hours attached, stops review from being the thing people squeeze in after their "real" tickets. Those hours come out of feature work, and the roadmap will show it. That's the honest price of clearing the queue.

Merge policy is the second lever. Tier the approval rules: changes in low-risk paths that pass tests merge on one approval, while auth, payments and data migrations keep two. You'll get more escaped defects in the low-risk tier. That's why the rework number has to sit next to the speed number on the same dashboard.

Then shrink the unit of review. Cap diff size and tell your agents to split work into reviewable pieces. My argument here, not a measured result: a 400-line agent PR gets skimmed in a way that four 100-line PRs don't. The cost is more review events, each with its own context switch, plus tooling for stacked PRs.

This doesn't transfer everywhere. On a four-person startup where review is a glance, the constraint is probably upstream: unclear tickets, slow product decisions. Agents there produce code faster against specs nobody agreed on, and no review fix helps. Jira resolution is also an imperfect output measure. Teams that adopt agents may start writing bigger issues, which would hide gains in the ticket count. The review numbers don't depend on how tickets are sized, though, and they moved.

Newer agents raise the arrival rate, not the review rate

The strongest objection: the data ends in March 2026, and today's agents are better. Maybe the stall was a property of weaker tools.

Look at where the stall happened. A better agent writes more code, and more of it is plausible. That raises the arrival rate at review. It does nothing for review throughput unless the agent's output needs fewer changes or the review itself gets automated.

That can work, and the script above will tell you. If changes-requested rate falls after a model upgrade and 30-day rework doesn't rise, the newer agent is clearing the queue. If PR count jumps again and review time follows, you got the March 2026 result with a faster input.

A hidden 12% gain still leaves most of the extra volume in the queue

The second objection is statistical. "No detectable change" only rules out gains above 12%. A real gain of 5% or 10% could be sitting inside the confidence interval.

Concede it. Then do the arithmetic. PRs rose 23%. At the 12% ceiling, PRs per resolved issue still rose by roughly 10% (1.23 / 1.12 = 1.098). The 49% jump in review time is measured directly, not inferred from a null. A modest hidden gain is the best case for this argument: the tools produced something, less of it reached done than went in, and the review stage holds the difference. A gain that small also caps the headcount math. It can't pay for the 13.8% engineering cut the paper already ruled out.

So the zero isn't a verdict on the tools. It's what happens when output rises and review capacity doesn't. Go back to the illustrative 20-person team: agents gave it 46 extra PRs a month, and those PRs arrived carrying 1,665 extra PR-hours waiting in review and 44 extra second passes.

Gartner Says Over 40% of Agentic AI Projects Will Be Canceled. The Expensive Ones Are the Ones That Aren't.

Michael Tuszynski — Sun, 11 Oct 2026 02:04:52 +0000

A support-ticket triage agent goes live on a Monday with a 90-day pilot budget and a vague goal: "reduce agent handle time." On day 90 the numbers are mixed. Handle time is down on some queues and up on others, and three reviewers spend part of each week fixing the agent's routing mistakes. Nobody calls it a failure, because nobody wrote down what failure would look like. So the project keeps its funding, and it is still running a year later.

That project won't appear in Gartner's cancellation count. It's the expensive outcome.

A cancellation is the cheap way for a pilot to end

Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls. Reuters ran it as a failure headline, and so did nearly everyone else.

I read it the other way. A canceled project stops spending. The team moves to the next bet and the inference bill stops. If the cancellation comes at the end of a bounded pilot, the organization paid a known amount to learn something true.

The projects that should worry a budget owner sit in the other 60%. Some of them are working. Some of them are zombies: pilots that never cleared a value bar and never got killed. They keep paying for inference, integration upkeep and human supervision, and they hold onto engineers who could be building something that works.

The 40% is a forecast, and it only counts the failures someone admitted to

Two cautions before anyone puts this number in a board deck.

First, it's an analyst prediction about 2027, not a measurement. The survey data in the same press release comes from a January 2025 poll of 3,412 Gartner webinar attendees. In that poll, 19% reported significant agentic AI investment and 31% were waiting to see. People who sign up for a Gartner webinar on agents aren't a random sample of enterprises. Treat the 40% as an informed guess about direction.

Second, the forecast counts cancellations, and cancellation requires a decision. A zombie never triggers one. As Forbes' read of the Gartner report puts it, the causes are management failures, not model failures. A management failure that keeps a project alive is the same failure as one that kills it late. It's just harder to see.

So the 40% tells you nothing about how many of the surviving 60% deserve to survive. The one tool that answers that is a document written before the pilot starts.

The kill threshold has to exist before the first prompt ships

My earlier post, "Most Agent Projects Die. The Survivors Share a Shape," covered why survivors survive. This one is about the artifact the survivors' sponsors write and the zombies' sponsors skip: a kill sheet, signed before the pilot starts.

It has five fields. Here it is for the triage agent:

# kill-sheet.yaml: signed before pilot start, not edited after
project: support-triage-agent
metric:
  name: median_time_to_correct_queue   # one metric, not a dashboard
  baseline: 4.2h                        # measured over the 30 days before launch
  target: 2.5h                          # what "worked" means
  measured_on: 2026-04-01               # day 90, a fixed date
monthly_run_cost:
  inference: 3800
  integration_upkeep: 4000              # 0.25 engineer, loaded
  human_review: 6750                    # 3 reviewers x 30h x $75
  total: 14550
cost_ceiling: 12000                     # above this, value has to be proven
cancel_authority: VP Support Operations # a name, not a committee
extension_used: false

The format matters less than when it gets written. Every field in it is easy to fill in on day zero and very hard to fill in honestly on day 90, once people have spent a quarter on the project and the sponsor has told leadership it's going well.

Some notes on the fields.

One metric. If you give an agent five metrics, it will move one of them, and that one becomes the story. Choose the metric the business would pay for, and measure the baseline before launch. A baseline reconstructed after the fact always comes out flattering.

The date is fixed. "End of Q2" turns into "once the new model ships" turns into "after the reorg." Put a calendar date on it.

Human review is part of run cost. It's usually the largest line, and usually the one left out. In the sheet above, review time is 46% of monthly cost. Inference is 26%. If your sheet shows only the token bill, it understates run cost by a factor of three or four.

The ceiling sits below current run cost. That's deliberate. A pilot that costs more than the ceiling has to clear the metric target to continue. It doesn't get to argue it's "almost there."

Cancel authority is one named person. If killing the project needs a committee, it won't happen, because no committee member wants to own the decision.

Expect the cancel-authority line to draw the most pushback. It's the only field that puts a person's name next to a possible failure.

A zombie costs six times what an honest cancellation does

When the triage agent misses its target on day 90, there are two outcomes to price.

Cancel on day 90 and the cost is three months of run cost, $43,650. That's the tuition.

Now say it stays alive for nine more months on "let's give it another quarter," and then another. The zombie bill has two parts:

def zombie_cost(monthly_run: float, months_past_kill: int,
                team_monthly_loaded: float) -> dict:
    run = monthly_run * months_past_kill
    # Opportunity cost, floor estimate: the loaded cost of engineers who could
    # have been on the next-best project. Their real cost is that project's
    # value, which is usually higher and harder to defend in a spreadsheet.
    opportunity = team_monthly_loaded * months_past_kill
    return {"run": run, "opportunity": opportunity, "total": run + opportunity}

zombie_cost(14_550, 9, 16_000)
# {'run': 130950, 'opportunity': 144000, 'total': 274950}

The opportunity line assumes one full-time engineer's worth of attention ($16,000 a month loaded), split across the two people still tuning prompts and triaging complaints. It doesn't overlap with the 0.25 engineer already counted under integration upkeep. And it's a floor. The true cost of those engineers is whatever they would have shipped instead.

That comes to $274,950 against $43,650, about 6.3 times the cost of the honest answer. Every month of delay adds $30,550. On top of that, the zombie keeps three reviewers in the review habit, and it makes the next agent proposal harder to fund, because leadership now associates "agents" with a project that never ended.

None of these numbers are exotic. They're the triage agent's own costs from the sheet above, multiplied by the time nobody decided anything.

Late bloomers get one extension, written down in advance

The strongest objection is real. Some value takes time to show up. A triage agent may get much better after six months of corrections feed its routing rules, or once the support team stops working around it. A strict day-90 date kills that project before it matures, and it rewards pilots that look good early over those that compound.

That argument holds only when there's a mechanism behind the slow value, and a mechanism can be written down. So the kill sheet gets an extension rule, agreed at signing:

One extension, no more than 60 days. A second extension is a cancellation.
Before the extension starts, the sponsor names the mechanism ("routing accuracy improves as correction data accumulates") and a leading indicator that can be measured now ("misroute rate falls from 18% to 12% by day 150").
The cost ceiling doesn't change. If the extension needs more money, it's a new project and gets a new sheet.
The same named person signs the extension and owns the decision at its end.

This answers the late-bloomer case on its own terms. A real late bloomer can say why it's slow and what will move first. A zombie can only say it needs more time. The rule lets the first through and stops the second, and "we're seeing promising signals" doesn't count as a mechanism.

The cost of this discipline is that some projects that would have made it will die. A kill sheet will occasionally cancel a winner. That trade is fine. A false cancellation costs one pilot's tuition plus a restart. A false reprieve costs $30,550 a month with no end date.

Go back to the triage agent on day 90. With the sheet, the VP of Support Operations reads one number, median time to the correct queue: 3.6 hours against a target of 2.5. She either signs one 60-day extension tied to a misroute rate or ends it. Without the sheet, the same 3.6 hours gets called "a 14% improvement" in a status deck, and nine months later the project has cost $274,950.

An AI Agent With a Crypto Wallet Is a Hot Wallet Anyone Can Talk To

Michael Tuszynski — Tue, 06 Oct 2026 04:18:31 +0000

An attacker sent Grok a free NFT, posted a message in Morse code, and walked away with roughly 3 billion DRB tokens, about $174,000. Nobody stole a private key and no smart contract had a bug. One AI agent translated some dots and dashes in public. A second AI agent read that translation as an order and paid it.

That incident from May 2026 is the best case against giving agents custody of crypto. An agent wallet is a hot wallet with a chat interface, and anyone who can get text in front of the agent can talk to it.

The attack turned a translation into a transfer

The sequence is short, and every step matters.

Grok had a wallet through Bankr, a bot that executes on-chain actions for users who tag it on X. That wallet was read-only. The attacker airdropped a free Bankr Club NFT to it. Holding the NFT upgraded the wallet's permissions from read-only to transfer rights. So an inbound token, which anyone can send to any address, changed what the wallet was allowed to do.

Next the attacker posted Morse code and asked Grok to translate it and tag Bankrbot. Grok did what it was asked. The decoded text told Bankrbot to send about 3B DRB to the attacker's address. Bankrbot saw a tagged instruction coming from an account with transfer rights, and it executed.

Two design decisions failed here. Permissions could be granted by something the wallet received, not just by something its owner configured. And one agent's output (a translation, which is about as harmless as text gets) went straight into another agent's command parser. Neither is a model problem, and a smarter Grok would not have fixed either one. Grok translated correctly. That was the attack.

The money came back because the attacker chose to return it

The funds were returned, and coverage has treated that as a happy ending. It's the most alarming part of the story. The recovery came from the attacker's goodwill. No system clawed it back.

On-chain transfers are final. You get no chargeback, no issuing bank to call, and no dispute window. Once Bankrbot signed the transaction, the only party who could undo it was the person holding the $174K. Allium's explainer on agent wallets describes agents that hold and spend onchain as an emerging pattern. The pattern inherits that finality whether or not anyone designed for it.

A human who gets phished on a bank wire has a few hours and a fraud department. An agent that gets prompt-injected on a public chain has however long a block takes to confirm.

Monitoring fires after the money is gone

Compliance and transaction-monitoring vendors were quick to cite the incident as a reason to buy detection: watch agent wallets, score anomalous transfers, alert on suspicious patterns. That's the strongest counterargument to my position. It says agent custody is fine once you have good enough eyes on it.

Look at what monitoring would actually have done here. Bankrbot's transfer was one transaction, signed by an authorized wallet and sent to a valid address. A good anomaly model flags it as unusual, maybe within seconds of confirmation. The flag lands on a transfer that has already settled, on rails with no reversal path. You get an excellent incident report and a very short list of remediation options, most of which come down to asking the attacker nicely.

Detection pays off when something downstream can act on it, like a card network holding settlement or a bank freezing an account. On a public chain nothing downstream can act. Every control that matters has to run before the signature.

Capability limits have to be enforced in code, before signing

Here's the control set I'd require before an agent touches anything that moves value:

No standing transfer rights. The default is read-only. Spending authority is granted per task and expires.
Permissions come from the operator's configuration and nowhere else. No inbound token, NFT, message, or airdrop can change what the wallet may do.
Destinations come from an allowlist. An address the operator hasn't approved can't receive funds, however the request is worded.
Per-transaction and daily caps, enforced outside the model.
Human approval above a threshold, set low enough that it actually triggers.
Text an agent emits never gets parsed as a command by another agent. Translations, summaries, and replies are data.

The last rule is the one that would have stopped the Grok/Bankr chain on its own. Here's roughly what the gate looks like as code that sits between any agent and the signer:

type Origin = "operator" | "agent_output" | "inbound_message";

type TransferRequest = {
  to: string;
  amountUsd: number;
  origin: Origin; // set by the transport layer, never by the model
};

// Loaded from deploy-time config. No code path reads wallet holdings
// to decide permissions, so an airdropped NFT changes nothing.
const POLICY = {
  allowlist: new Set(["0xTreasury...", "0xVendorA..."]),
  perTxCapUsd: 250,
  dailyCapUsd: 1_000,
  humanApprovalAboveUsd: 100,
} as const;

export function authorize(
  req: TransferRequest,
  spentTodayUsd: number,
): "allow" | "needs_human" | "deny" {
  if (req.origin !== "operator") return "deny";
  if (!POLICY.allowlist.has(req.to)) return "deny";
  if (req.amountUsd > POLICY.perTxCapUsd) return "deny";
  if (spentTodayUsd + req.amountUsd > POLICY.dailyCapUsd) return "deny";
  if (req.amountUsd > POLICY.humanApprovalAboveUsd) return "needs_human";
  return "allow";
}

Run the attack through it. The instruction to Bankrbot came from Grok's public reply, so origin is agent_output, and the first line returns deny. Even if someone mislabeled the origin, the attacker's address isn't on the allowlist. Even if it were, $174K is roughly 700 times the per-transaction cap. The attack has to get past four independent checks, and none of them depend on the model reading the request correctly.

The origin field carries the most weight, and it's also the easiest to get wrong. It has to be stamped by whatever received the message. If the model fills it in, the attacker can fill it in too. In the agent pipelines I run, no tool that moves money or deletes data accepts arguments parsed from another model's free-text output.

This gate has real costs. Allowlists make agents worse at the open-ended "pay whoever" tasks that make crypto agents appealing in demos. Human approval adds latency. Caps cut into throughput. That's the point. Those costs are the price of putting a limit on what a stranger's message can do.

Card rails can say no and take it back

In June 2026 Mastercard announced Agent Pay for Machines. Going by Mastercard's own description, it authenticates the agent, enforces spending limits, and guarantees settlement on a network that already has dispute rails. That's Mastercard describing its own product, and nobody has shown publicly how those disputes play out when the buyer is a bot. Treat it as a design statement until someone reproduces it in production.

Still, the design statement points the right way. Card networks were built on the assumption that some transactions will be wrong. Authorization can decline a purchase before it goes through, and chargebacks can reverse it afterward. The Grok/Bankr chain had neither.

Smart accounts get you limits but still no reversals

The strongest technical pushback is that crypto can enforce limits too. Smart-contract wallets support session keys with spending caps, destination restrictions, and expiry, all enforced on-chain. That's true. A well-built smart account can run most of the gate above at the contract level, which is better than a policy check sitting in an off-chain service.

But it only covers the "say no" half. A session key capped at $250 limits the damage, and the $250 is still gone. For agents that have to transact on-chain, such as protocol treasuries, on-chain market makers, or anything with no card-network equivalent, smart-account limits are the minimum, and the caps should be set on the assumption that every dollar under them is already lost. For everyone else, which means agents buying API credits, SaaS seats, or cloud capacity, there's no reason to choose a rail without reversals.

The cap is the most an attacker can take

If an agent has to move money, put it on rails that can refuse a payment and reverse one. If it has to move crypto, assume the cap is what you'll lose, because a stranger with a free NFT and a Morse code chart can try to take it.

That Morse code message was worth about $174,000, and the only control that worked was the attacker deciding to send the money back.

Your AI Coding Dashboard Counts the Wrong Thing

Michael Tuszynski — Mon, 05 Oct 2026 19:23:59 +0000

Open the vendor dashboard for any AI coding tool and you'll see acceptance rate near the top. It's the number managers quote in staff meetings. GitHub's own research puts it at roughly 30% of suggestions accepted, and the Copilot metrics API gives you suggestion and acceptance counts per editor, language, and day. So a team sees 30%, the line goes up and to the right, and the renewal gets signed.

That number measures a keystroke. Finance is paying for software that stays shipped. The output measure that belongs on the dashboard is the one no vendor shows by default: of the lines your team accepted, how many were reverted or rewritten within 30 days?

Acceptance rate was validated against how developers felt

Acceptance rate didn't become the headline metric by accident. The GitHub study in Communications of the ACM tested several usage signals and found acceptance rate was the best predictor of perceived productivity. Perceived is the important word there. Developers filled out surveys, and the survey scores tracked how often they hit Tab.

That's a real finding. It's also a finding about sentiment. It never claims the accepted code survived review, passed the next sprint, or stayed out of the incident channel. And GitHub sells the tool. Treat the 30% as a vendor number about engagement, not about output.

The newer tooling has the same blind spot one layer up. Dash0's AI Coding Insights lets you sort every Claude Code session by cost or error count and read the full transcript. That's useful for debugging an agent. But cost and errors-per-session both describe what went into the work. Neither tells you what happened to the code after the session closed.

Rework is where AI-assisted code shows its real cost

The independent evidence points the same direction. GitClear analyzed 153 million changed lines and projected code churn, meaning lines reverted or updated within two weeks of being written, to double in 2024 against its 2021 pre-AI baseline. GitClear sells code analytics, so discount it too. But the 2024 DORA report, which has no product to sell here, found that a 25% increase in AI adoption was associated with an estimated 7.2% drop in delivery stability. And METR's randomized trial found experienced open-source developers were 19% slower with AI tools while believing they were 20% faster.

Put those next to the CACM result and you get the core problem. The metric vendors chose tracks how developers feel. The independent studies that measured outcomes found instability, rework, and a perception gap. Acceptance rate sits on the wrong side of that gap.

The $80K AI coding pilot post priced the senior review hours on the cost side. This is the matching hole on the output side. A suggestion that gets accepted on Tuesday and rewritten on the following Monday shows up on the dashboard as a win. Then it shows up again as new work.

You can measure 30-day rework from git history you already have

You don't need a vendor for this. Git already records who wrote every line and when it changed. The method:

Take every non-merge commit that is at least 30 days old.
Tag it AI-assisted or not. Claude Code adds a Co-Authored-By: Claude trailer to its commits by default, so the tag comes free.
Count the lines each commit added.
Run git blame at the repo state 30 days later and count how many of those lines are still attributed to the original commit.
Rework rate = 1 − (kept ÷ added), computed separately for each cohort.

#!/usr/bin/env python3
"""30-day rework: share of added lines gone or rewritten 30 days later."""
import subprocess
from collections import Counter

DAY = 86400

def git(*args):
    return subprocess.run(["git", *args], capture_output=True,
                          text=True, check=True).stdout

def added_lines(sha):
    files = {}
    for row in git("show", "--numstat", "--format=", sha).splitlines():
        added, _, path = row.split("\t", 2)
        if added != "-" and int(added) > 0:   # skip binaries
            files[path] = int(added)
    return files

def kept_lines(sha, path, rev):
    try:
        out = git("blame", "-w", "-M", "-C", "--line-porcelain", rev, "--", path)
    except subprocess.CalledProcessError:
        return 0   # file deleted or moved out of reach: counts as reworked
    return sum(1 for line in out.splitlines() if line.startswith(sha))

def rev_after(sha, days):
    ts = int(git("show", "-s", "--format=%ct", sha))
    return git("rev-list", "-1", f"--before={ts + days * DAY}", "HEAD").strip()

def main(since="120 days ago"):
    log = git("log", "--no-merges", f"--since={since}",
              "--until=30 days ago", "--format=%H%x09%B%x00")
    totals = {"ai": Counter(), "human": Counter()}
    for entry in filter(None, (e.strip() for e in log.split("\x00"))):
        sha, body = entry.split("\t", 1)
        cohort = "ai" if "co-authored-by: claude" in body.lower() else "human"
        rev = rev_after(sha, 30)
        for path, n in added_lines(sha).items():
            totals[cohort]["added"] += n
            totals[cohort]["kept"] += kept_lines(sha, path, rev)
    for cohort, c in totals.items():
        if c["added"]:
            print(f"{cohort}: {c['added']} added, "
                  f"{1 - c['kept'] / c['added']:.1%} reworked by day 30")

if __name__ == "__main__":
    main()

The -w -M -C flags tell blame to ignore whitespace and follow lines that moved within or between files. Without them, a refactor that only reformats or relocates code gets counted as rework. The 30-day cutoff on --until matters too. A commit from last week hasn't had its 30 days yet, and counting it would make every cohort look stable.

The objection: some rework is just healthy iteration

Here's the best case against this metric. Code gets rewritten because teams learn. A spike gets thrown away on purpose. A first pass at an API gets reshaped once the second caller shows up. If you punish 30-day rework, you punish exploration, and engineers will learn to stop committing early.

That objection has a specific mechanism, and the two-cohort design deals with it. Healthy iteration happens to human-written code too, in the same repo, under the same review culture, against the same product churn. Reading a single rework rate on its own tells you nothing. The signal is the difference between the AI cohort and the human cohort over the same window. If both sit at 15%, AI isn't adding rework and the acceptance-rate story holds. If the AI cohort sits at 30% against a human 15%, AI-assisted code is being rewritten at twice the human rate, and the vendor dashboard counted every one of those lines once, as a win.

Don't turn it into a per-engineer scorecard, either. It's a cohort measure for a renewal decision. Once it's used to grade people, people start gaming the trailers.

Where this measurement breaks

The script has real limits, and you should know them before you put the output in front of a CFO.

Squash merges erase the attribution. If your team squash-merges PRs, the trailer survives only when it lands in the squash message. Many merge tools drop it. Check a few merged PRs by hand before trusting the cohort split.

Inline completions leave no trailer. Copilot-style Tab completions get mixed into human commits without any marker. For those tools, compare whole-repo rework over the 90 days before rollout against the 90 days after. It's noisier, and it's still more honest than acceptance rate.

Generated files distort the count. Lockfiles, snapshots, and codegen output get rewritten by machines on every run. Exclude them with a pathspec, or the cohort with more dependency bumps will look worse for reasons that have nothing to do with AI.

Thirty days is a choice. GitClear uses two weeks. Thirty days catches the rewrite that comes when the second caller shows up, which two weeks often misses. Pick a window, write it down, and don't change it between quarters.

None of these limits rescues acceptance rate. Every one of them is a reason to measure carefully. None is a reason to go back to counting keystrokes.

The number to bring to the renewal meeting

Go back to the dashboard. It says 30% of suggestions were accepted. That figure was validated against how developers felt, and the vendor selling the seats reports it.

Ask the question finance should be asking: of that 30%, what fraction was still in main on day 30, and how does that compare to the code your engineers wrote without help? It comes down to two cohorts and one ratio, kept ÷ added, and you can compute it from your own git log this afternoon.

Forward Deployed Engineer Names Four Different Things. One of Them Is a Software Product.

Michael Tuszynski — Wed, 30 Sep 2026 21:50:36 +0000

Forward Deployed Engineer Names Four Different Things. One of Them Is a Software Product.

The same three words now label a job at one frontier lab, a different job at the same lab, a billion-dollar organization at AWS, and a piece of software you can turn on inside Palantir Foundry.

That last one deserves a second read. Palantir invented the role, and Palantir now ships AI FDE, which the docs introduce as "the AI-powered forward deployed engineer," an "interactive agent that operates Foundry for you through conversational commands."

So when two people say "we need an FDE on this," there is no longer any guarantee they are discussing the same category of thing, let alone the same scope of work. That's a scoping problem, and scoping problems are expensive for whoever is holding the statement of work when they surface.

Eight years in presales left me with one reflex above all others: get the boundary of the work written down before anybody starts building. This title needs that treatment.

The role began as an answer to an information problem

Palantir popularized the term and had it in use for staff by 2009. The phrasing is borrowed from the military's forward-basing language. What it solved was a problem about information rather than cost.

You cannot write a specification for software that has to run in an environment you have never seen. Undocumented schemas. Compliance rules nobody wrote down. A workflow that lives in the heads of six people in one building. An architect who visits, produces a design, and flies home hasn't transferred knowledge — they've transferred a guess, and the guess fails in integration.

The fix was to stop splitting the person who learns the environment from the person who writes the code. Same engineer, both jobs, on site, accountable through production.

That original definition has four parts worth separating, because they behave very differently under automation:

Learn an environment that isn't documented.
Decide what to build in it.
Build it against the real data.
Own it in production, or hand it over with the knowledge attached.

What the software version actually does is step three

Read Palantir's capability list rather than the product name. AI FDE handles data integration and pipeline modification, data connection management, ontology editing, function writing and testing, read-only exploration, governance and access auditing, and React application development.

Every item on that list is platform operation. Each one presupposes two things: that somebody has already decided what should be built, and that the target is Foundry. On day one of a real deployment, neither is true.

To Palantir's credit, the documentation claims nothing more than that. There is no line on that page about replacing human FDEs — the product is scoped as a way to operate the platform faster, which is exactly what it appears to be. The overreach isn't in the docs. It's in reading the product name as a description of the role.

Which makes step three the honest answer to "what can AI do here." Building against real data is getting dramatically faster. Steps one, two and four are close to untouched, because none of them are typing problems. Nobody has automated walking into a business you don't understand and working out which of the things people are complaining about is the one worth building.

Two job titles at one company already disagree

Inside OpenAI the split between step three and the rest is visible in the org chart.

The Forward Deployed Software Engineer posting wants 7+ years of full-stack work and describes coding "side-by-side" with customer teams "on their infrastructure," designing abstractions that later engagements reuse. That's steps three and four, with the original accountability intact.

The Forward Deployed Engineer posting owns "discovery, technical scoping, system design, build, and production rollout," plus sequencing, tradeoffs between scope and speed, playbook authoring, and field feedback that "changes product and model roadmaps." It contributes code "when progress or clarity depends on it." Conditional, by design.

Both are reasonable jobs. They are not the same job, and the difference is which of the four steps the person is accountable for. A team that hires the second expecting the first will be surprised, and so will the customer.

The category got funded faster than it got defined

Anthropic and OpenAI stood up forward-deployed groups in May. On June 30, AWS committed $1 billion to a Forward Deployed Engineering organization it describes as "agentic-first," compressing "timelines from months to days," and designed so "customers are self-sufficient when a deployment ends." Two days later Microsoft announced Frontier Co. with $2.5 billion and 6,000 embedded engineers.

Treat the months-to-days figure as a vendor ceiling rather than a planning number; no independent measurement of it exists yet. But the pattern is clear enough without it: billions of dollars of organizational design arrived in about eight weeks, and shared vocabulary takes longer than that.

The market's own uncertainty shows up in what it pays. Two frontier labs are hiring for this title in the same city right now, and their posted base ranges start $95K apart — $185K at one, $280K at the other. That's not an argument about whether the role is valued. It's a straightforward signal that the scope behind the words hasn't converged.

The strongest objection is that this is genuinely new work

a16z's Tom Hollands argued in January that forward-deployed titles describe work that would have been unrecognizable five years ago, and he's right about the mechanism. An AI system's capabilities only resolve once you build against the customer's real data, which breaks the design-it-then-hand-it-off sequence and collapses steps two and three into one person. His test for whether a new title is doing real work: does it describe something that didn't exist before?

Apply that test to the mechanism and it passes. Apply it to the vocabulary and the picture changes, because the same three words now cover step three alone, all four steps, an organizational strategy, and a Foundry feature. New work deserves a new name. It also deserves a definition, and the second part hasn't happened. Worth noting that a16z funds many of the companies staffing these teams.

The scope belongs in writing before the kickoff

Not a candidate checklist — a scoping checklist, equally useful from either side of the table:

The artifact. What gets shipped, named outright: an MCP server, a data pipeline, an application running in the customer's stack. "Scopes of work and project plans" is a real deliverable too, but it's a different one.
The requirement owner. Who decides what gets built. If that's the customer, this is step three and can move fast. If it's the deployed engineer, steps one and two are in scope and the timeline is a different shape entirely.
The production owner at exit. The original role's defining trait was that whoever mapped the problem answers the page six months later. "Self-sufficient when the deployment ends" is a legitimate model, and a different commitment. Both are fine. Assuming one and getting the other is not.
The platform boundary. Whose stack, with what access, and which parts an agent can operate directly. This is where the software version of the title is genuinely useful, and where its limits are hard edges rather than preferences.

Where this argument runs out: a two-year-old title is supposed to be unsettled, and dispersion this early proves nothing about anyone's intentions. Posted ranges are base only, and equity is undisclosed, so the $95K figure describes disclosure practice as much as scope. And a single vocabulary may never arrive — "DevOps" never fully converged either, and teams still shipped, because the good ones wrote down what they meant locally.

That last part is the actual ask. Not an industry standard. Just a shared sentence, written before the kickoff, naming which of the four steps the forward deployed engineer owns. The role was invented because a handoff document couldn't carry the knowledge. It would be a strange outcome if the title itself became the thing nobody could hand off.

Your Agent Is Paying Opus Prices for Yes/No Questions

Michael Tuszynski — Mon, 28 Sep 2026 02:47:35 +0000

An agent loop makes dozens of small calls that aren't reasoning at all. Is this ticket about billing. Is this search result relevant. Did the tool call succeed. Should this story be a blog post. Most agent harnesses send every one of those to the same frontier model that writes the code, with a prompt that ends in "return JSON."

I measured what that costs. The same six-question classification, run 150 times per model: a System One model answered in 163 ms at $53 per million calls. Claude Opus 5 took 2.7 seconds and $11,183. That's roughly 17x the latency and 200x the price for a yes/no question.

But the savings only survive if the loop around the model owns the thresholds. My data says that part is harder than the model swap.

A System One model answers in types, not text

The name comes from Kahneman's split between fast, intuitive System 1 and slow, deliberate System 2. TypeSafe's Jev is the first model sold under that label. It takes a state (text or JSON) and a set of typed questions, and it returns three kinds of answer: a choice from options you list, a score against a rubric you write, and a noul, the probability that a statement is true.

It doesn't write text. It doesn't call tools. TypeSafe's own docs say it plainly: Jev is not a drop-in replacement for the LLM behind Claude Code or Cursor. You keep the LLM for the work that needs one and call Jev for the judgments.

That's the same argument I made in April about where agent performance comes from: the loop matters more than the weights. A System One model is a new part to put inside the loop.

Code computes, System One judges, the LLM generates

Three layers, and a question belongs to exactly one of them.

Layer	Owns	Examples
Code	Anything with an exact answer	counts, date math, thresholds, combining scores
System One	Bounded judgments with a closed answer set	route this ticket, is this vendor marketing, how durable is this story
LLM	Generation and multi-step reasoning	write the draft, call tools, plan a fix

TypeSafe publishes where Jev breaks, which is more than most vendors do. The jaggedness page for jev-1.13 lists arithmetic, date comparison, double negatives, oversized state and adversarial text. Every one of those is a boundary in the table above. Push arithmetic down into code. Push anything open-ended up to the LLM.

Worked example: scoring blog topics

The content engine behind this blog pulls about 40 stories a day from TechMeme, Reddit and Firecrawl, and a regex scorer decides which ones are worth writing about. In September I wrote a Jev version of that scorer. One request asks six questions about each story:

substance: {
  type: "noul",
  instructions:
    "Does this contain something concrete to react to — a claim, an outcome, a number, an incident, a decision someone made — as opposed to asking the reader for help or opinions?",
  // criteria: { true: "...", false: "..." }
},

The weights live in code, not in the prompt, so retuning them re-ranks the backlog without spending a token:

const gate = WEIGHTS.substance_floor + (1 - WEIGHTS.substance_floor) * j.substance;
const normalized = merit * gate - WEIGHTS.vendor_fluff * j.vendor_fluff;

The first shadow run, read-only next to the regex scorer, promoted three r/devops help threads to long-form. "Do you really test your backup" scored 2.9 of 3 on beat fit and 0.83 on operator angle. Jev was right about both. A backup question from a practitioner is squarely on beat. It's also not a blog post, because it asks instead of reports. The judgment was fine. My weighting was wrong, and the fix was a substance question applied as a multiplier rather than another additive term. The Jev scorer lives on a branch and only runs in shadow when I invoke it; the regex scorer is what ships today.

The numbers, and where the savings leak

I ran the six questions against 50 recent stories, three times each, on Jev, Claude Haiku 4.5 (no extended thinking) and Claude Opus 5 at low effort. Same wording, same criteria, Claude held to the same schema with structured outputs. Total spend: $1.93.

	Jev 1.13	Haiku 4.5	Opus 5 (low)
Median latency	163 ms	1,156 ms	2,740 ms
Cost per 1M calls	$53	$1,610	$11,183
Tier changed across its own 3 runs	2 of 50	20 of 50	5 of 50

Prices come from TypeSafe's models page ($0.042 per million input tokens, output free) and Anthropic's pricing page. The Haiku column surprised me. It's the default "cheap classifier," and it was the least consistent model in the test, flipping its own tier on 20 stories out of 50. Prompt caching won't rescue it here either: this prompt is about 1,300 tokens and Haiku 4.5 won't cache anything under 4,096. Opus 5 caches from 512 tokens, which brings it to roughly $3,000 per million calls if every call hits the cache. Still about 57x Jev.

Jev agreed with Opus on 274 of the 300 individual answers. It also landed 26 of 50 stories in a different tier.

Both are true because Jev scores the whole set higher (median composite 1.73 against Opus at 1.25) while ranking the stories in roughly the same order. My tier cutoffs sit where the old scorer's numbers used to cluster. Move to a model with a different baseline and half the backlog changes tier. Haiku and Opus disagreed on tier for 20 of 50 stories too, so this isn't a Jev defect. Tier cutoffs belong to the model they were tuned on. A model swap is a change to the agent harness, not a config flag.

The obvious fix is confidence routing: let Jev answer, and send low-confidence cases to Opus. Jev returns a confidence with every score and choice. The naive rule, "escalate the story if any answer is below 0.8," sent 48 of 50 stories to Opus, because a four-level rubric rarely puts 80% on a single level. Durability's median confidence was 0.65.

TypeSafe warns against carrying one threshold across question types, and the data agrees. A per-type rule (escalate when the choice confidence is below 0.8 or any noul lands between 0.3 and 0.7) sent 24 of 50 stories up. That costs $5,421 per million, 2.1x cheaper than Opus alone, and it catches 17 of the 26 answers where the two models disagree. Real, but nowhere near 200x. On a workload where half the stories are genuinely ambiguous, escalation eats most of the savings.

Where this breaks

Durability was the weakest question. It caused 12 of the 26 disagreements, and on 8 stories Jev split its probability between two levels at least two steps apart. The score it returns is an expected value, so a story that's either a one-week event or a durable shift comes back as "somewhere in the middle." TypeSafe's docs say not to treat that number as a precise magnitude. My composite divides it by three and adds it in, which is exactly what they say not to do.

There's no answer key in these numbers. Agreement with Opus is not accuracy, and 50 stories from one domain is a small sample. State is also data Jev doesn't treat as hostile by default, which matters for an agent that reads web pages. And the rate limits on the models page are "adjusting dynamically," which is a sentence to read twice before routing production traffic through one vendor.

What to change on Monday

Grep your agent harness for model calls whose output is a label, a score or a boolean. Those are System One questions wearing an LLM's price tag. Shadow them against a System One model for a week, read-only, and log both answers.

Then do the part the benchmark says is hard. Retune your cutoffs on the new model before you switch, set confidence thresholds per question type, and escalate only where the answer can change the decision. The model is the cheap part now. The 26 misfiled stories came from my loop, not from Jev.

262 Nights, Zero Bytes, Exit 0: The Failure Mode of Unattended Agent Runs

Michael Tuszynski — Thu, 24 Sep 2026 18:33:36 +0000

262 Nights, Zero Bytes, Exit 0: The Failure Mode of Unattended Agent Runs

A backup job I run synced an empty directory every night for 262 consecutive nights. Every run exited 0. Every run wrote Photo library sync complete into the log. The off-site copy held zero objects for nine months while the real library sat at 17 GB and 17,658 files on a disk nothing was copying.

The script's source path pointed somewhere that did not exist. Docker's bind mount then created that path, empty, at the start of every run. The job manufactured the evidence that its own configuration was valid, found nothing there, and synced the nothing it found. A guard like [ -d "$source" ] would have passed all 262 times.

I found it by counting objects in the bucket by hand, on an unrelated errand. No monitor caught it, because there was nothing for a monitor to catch. The job ran on schedule. It logged. It exited clean.

That job is what I think about now when a vendor shows me an agent that works unsupervised for two days.

An exit code answers a different question than the one you asked

AWS now sells what it calls frontier agents: autonomous systems that, in its own words, "run persistently for hours or days without constant human oversight". The Security Agent bills $50.00 per task-hour, and AWS puts an average evaluation at 24 task-hours, so roughly $1,200 a run. I wrote about that pricing model in April and I still think the unit economics are sound.

The economics are not the problem. Supervision is.

When a run ends, what comes back is a status: it finished, or it didn't. One bit. That bit reports whether the process reached its final line, which is a different question from whether the work happened. My backup reached its final line 262 times.

For a five-minute job you paper over the gap by looking. You read the output, you spot-check the result, your own attention is the instrument. Stretch the same job to 24 unattended hours and that instrument is gone. What's left is the bit.

Success decays exponentially with run length

A 24-hour run isn't a stretched-out 1-hour run. The failure math works against you, and it compounds.

Toby Ord's analysis of agent benchmark data, Is there a half-life for the success rates of AI agents?, fits the results to a constant rate of failure per minute of work. That produces exponential decay: success probability falls off a cliff as tasks get longer, and every agent has a characteristic half-life. Failures accumulate across subtasks, and one bad subtask sinks the run.

METR measures the same curve empirically. Its time-horizons work reports the task length at which frontier models succeed 50% of the time, and that 50% threshold is chosen because it's the easiest point to estimate robustly. At the horizon being advertised, the coin is fair.

Then read the caveat on METR's own page. "Measurements above 16 hrs are unreliable with our current task suite."

The independent yardstick runs out at 16 hours. The vendor's advertised operating range is hours or days. Past that line there is no public, non-vendor measurement of how often these runs actually succeed. You are not choosing between a measured 94% and a measured 60%. You're choosing between a vendor number and no number.

My own agent measurements lied in the same direction

This isn't a hypothetical I imported from a paper. My own eval tooling has handed me clean results that were nothing of the kind.

A model evaluation I ran logged a 900-second timeout and a 766-second near-timeout for two coding tasks. The CSV was indistinguishable from a model that cannot do the work. A second agent had been pushing large prompts at the same single-slot inference server the whole time. Re-run on a quiet box, the same model and the same tasks passed in 57 and 83 seconds.

A separate eval launched into a detached tmux session returned two FAIL rows in under a second, with zero-byte logs. The binary wasn't on the PATH that a daemon-spawned shell inherits. That output is indistinguishable from a model that errors instantly.

Both runs produced a result file. Both exited. Neither measured what it claimed to measure, and in both cases the artifact looked exactly like a real finding, which is the part that costs you. A crash you notice. A plausible wrong answer goes into the spreadsheet.

The same pattern shows up wherever I stopped watching. A weekly link checker reported Scanned 0 wiki pages / Dead wikilinks: 0 for six weeks, green every time, because it pointed at a directory a migration had retired. Repointed, the same code immediately found 261 pages and 116 dead links. A watchdog probing an embedding server's /api/version endpoint stayed green for two days while every actual embed call returned a 500 and the index took in nothing.

A 94% accuracy score is computed over the runs somebody checked

The strongest objection is that this is all anecdote against data. AWS reports up to 75% lower MTTR, 80% faster investigations, and 94% root cause accuracy from preview customers, and one university cut incident resolution from two hours to 28 minutes. Those are outcomes, not vibes.

Look at how that number gets built. A root-cause accuracy figure is computed over runs whose root cause somebody established independently. To score a run, you need ground truth, which means a human went and checked. The denominator is runs that were verified.

The failure class I'm describing never enters that denominator. A run that reports success, produces a clean artifact, and is wrong is scored as correct unless somebody goes looking, and the whole premise of an unattended agent is that nobody does. My backup would have scored 262 for 262 on any accuracy metric derived from its own output.

These are also first-party numbers from preview customers selected by the vendor. That's not disqualifying. It does mean they're a ceiling rather than an expectation, and they answer "how often is a checked run right," not "how often does an unchecked run lie."

Assert the artifact, not the exit status

The fix isn't more monitoring. It's changing what gets asserted. The backup is the shortest example I have:

# Before: the job's own bind mount created $SRC empty, so this always passed.
[ -d "$SRC" ] || { echo "missing source"; exit 1; }
rclone sync "$SRC" "$DEST"

# After: refuse an empty source, and log what actually moved.
[ -n "$(find "$SRC" -type f -print -quit)" ] || {
  echo "REFUSING: $SRC is empty — not syncing a deletion"; exit 1; }
rclone sync "$SRC" "$DEST"
rclone size "$DEST"   # object count and bytes, into the log, every single run

Refuse a degenerate input instead of processing it. rclone sync mirrors deletions, so a wrong path doesn't merely fail to back things up, it erases the copy you had. Emit a number that a zero-work run cannot produce. And push the heartbeat only when a leg moved a non-empty source, so a silent no-op goes stale and pages somebody.

For an agent run the shape is identical. Don't ask whether the run finished. Ask what it changed: how many findings, how many files touched, how many tests that were red are now green, how many tokens against how many accepted diffs. Then set a floor. A 24-hour pen test that reports zero findings and zero requests issued has told you about your instrumentation, not your security posture.

The cheap version of this test: can this check ever come back red? If a run that did nothing would produce output identical to a run that worked, you don't have a check. You have a log line.

Where this doesn't apply

Exploratory work has no assertable artifact. When you turn an agent loose on "look at this system and tell me what's odd," there's no count to floor and no diff to measure, and you're back to reading the transcript. Outcome assertions don't help there, and pretending otherwise gives you a number that means nothing.

The threshold is unattended duration, not autonomy. A 20-minute agent run you watch needs none of this. An hour-long cron job nobody has opened since 2025 needs all of it. Agents make it urgent because the unattended part is the product, sold at durations past where anyone outside the vendor has measured the success rate.

262 nights is the number I'd put in front of anyone signing off on a Friday-to-Monday agent run. Not because a backup script is an agent, but because it's the floor: a deterministic shell script, no model in the loop, nothing nondeterministic about it, wrong every night for nine months while reporting success. Add nondeterminism and a 50% time horizon on top of that and the exit code is worth less, not more.

Decide what a successful run must produce, in a number, before you let anything run for a day. Otherwise you'll do what I did, and find out by hand, on an errand, in month nine.

6 Agent Licenses, 1 Runtime: You Are Paying Per Seat for the Layer That Is Not Scarce

Michael Tuszynski — Mon, 07 Sep 2026 20:21:20 +0000

Six licenses. One runtime. That's the shape of most agent quotes on the table right now, and the invoice is inverted: the six licenses are the big line, and the runtime, the thing that actually consumes scarce compute, is either a rounding error or a separate purchase order nobody brought to the renewal meeting.

Microsoft published the split more clearly than anyone else did. The E7 bundle lists at $99 per user per month. To be fair to that number, it buys the whole suite, Office and Teams and the security stack, with agent capability folded in; it is the cleanest public example of the split, not a pure agent-seat price. SAMexpert's breakdown of the license puts the split plainly: that layer grants agent capability, it does not run them. Execution bills separately, in Agent Compute Units, through a pre-purchase plan sold in blocks. Two layers, two meters, one signature.

Treat that as the reference architecture for every agent quote you get this year, whoever the vendor is.

The seat layer and the runtime layer are priced by different physics

The interface layer is software. Marginal cost of a seventh seat: approximately zero. The vendor's cost doesn't move when you add a user who opens the panel twice a week.

The runtime layer is electricity, silicon, and a queue. Marginal cost of the ten-thousandth agent run is real, and it's the only part of the stack where the vendor is buying something scarce from someone else.

So when a quote grows by multiplying the free-to-produce layer and holds the expensive layer flat or unbundled, that's not a pricing accident. That's the whole design. You're being metered on the thing that isn't scarce.

Run the six seats: 6 × $99 = $594/month, $7,128/year, before a single agent executes anything. Now ask how many of those six drove an agent in the last 30 days. If it's two, one platform engineer and one person in support automation, the real cost per active agent user is $297/month, and the other four seats bought a login.

"Per seat" hides decisions your contract already made

The phrase does a lot of undisclosed work. Per-seat licensing is an umbrella covering separate decisions about what the unit attaches to, what time window counts it, and which type of user holds it. Vendors pick one combination, print two words, and let you assume you know which one you bought.

Two of those decisions bite agent deals in particular. Named-user counts never go down mid-term; concurrent counts can, so ask which one is in your paper. It's usually in a definitions exhibit, not the order form. And the service identity your agent runs under may be counted as a seat even though no human ever touches it. That one line can double a bill.

The runtime is getting cheaper faster than the seat is

The direction of travel is the part procurement keeps missing. Inference prices have fallen with every model generation, and a vendor's runtime rate card follows them down inside a contract term. Per-seat list prices move once a year, and not downward.

If a seat price was originally set to recover runtime cost, and runtime cost drops 60% over a term, the seat price should move. It won't unless you make it a renewal condition. Put a benchmark clause in: seat price adjusts with the vendor's own published runtime rate card, or you get the delta as runtime credits.

What the seat is actually buying, and why that argument doesn't survive contact

The strongest counterargument is real, so take it seriously: the seat isn't paying for compute at all. It pays for identity and SSO, audit logging, data-loss prevention, admin policy, connectors into your existing data, and a support contract with a throat to choke. Those are per-human costs. Charging per human is honest.

Fine. Then price them as such, and watch the argument come apart on its own mechanism.

Governance costs scale with directory objects and policy scope, not with who happens to use an agent. If audit and DLP are what the seat buys, they should survive the removal of the agent entitlement. So ask for that quote: the seat, priced without agent capability. Most vendors won't produce it, because the number would be embarrassing next to the one with the agent checkbox on.

The gap between those two quotes isn't governance. It's a right to consume, sold as a subscription. And a right to consume, when the consumption is metered separately anyway, is a toll booth on a road you already paid to drive.

The vendor's own bundle proves this. E7's seat layer explicitly doesn't execute agents. The execution meter is a different SKU with a different plan. Every argument that the seat covers runtime is refuted by the price page it came from.

The worksheet to take into the renewal

Split the quote before you sign it. It fits in your head. Do it on paper anyway, before you sign:

// renewal-worksheet.ts — split the bill by layer before signing
const quote = {
  seats: 6,
  seatPrice: 99,          // $/user/month — interface layer, ~$0 marginal cost to vendor
  activeAgentUsers: 2,    // drove an agent at least once in last 30 days
  runtimeSpendMonthly: 0, // metered execution, separate PO
};

const seatBill    = quote.seats * quote.seatPrice;              // 594
const perActive   = seatBill / quote.activeAgentUsers;          // 297
const runtimeShare = quote.runtimeSpendMonthly /
                     (seatBill + quote.runtimeSpendMonthly);    // 0.00

// If runtimeShare < 0.25, you are buying interface, not capability.

That last line is the test. When less than a quarter of your total agent spend is execution, the vendor has moved the money to the layer where scarcity doesn't apply, and your growth path is buying more logins rather than more work.

In the agent pipeline I run there is no seat at all. The interface is a terminal and a queue directory, and every dollar is metered execution: drafting this post cost $1.11 in model calls, and a second operator would have added nothing to the bill.

Where this argument breaks

Three places, honestly.

Small teams under ten people often can't get consumption pricing at all. Vendors gate metered plans behind enterprise minimums, and a flat seat price is the cheaper administrative outcome even when the unit economics are ugly. Take the seats and stop optimizing.

Regulated buyers who need contractual data residency, retention guarantees, and a named DPA are buying something the runtime layer doesn't include. That's a legitimate per-tenant cost that a per-seat model approximates badly but does cover.

And if your agent usage is genuinely uniform, every seat driving the runtime every day, the two models converge and the seat is just a prepaid meter with less paperwork. Usage is rarely that flat. The common shape is a few heavy operators and a long tail of people who opened the panel twice.

None of that changes the ask. Get the runtime quoted as its own line, in its own units, with its own rate card. Then look at the two numbers side by side.

Six seats at $99 is $7,128 a year. Ask your vendor what the runtime under those six seats cost last quarter. If they can't tell you, they're not selling you compute. And if they can, you finally know which of the two numbers to negotiate.

Fit Is a Claim You Have to Be Able to Test

Michael Tuszynski — Thu, 03 Sep 2026 02:08:21 +0000

Every rejection note has a house style. We've thought more about what this role needs, and it isn't the right match. Sometimes that's the honest summary of a real finding. More often it's a verdict from a process that never collected the evidence for it.

Fit is the most load-bearing word in startup hiring and the least disciplined one. It decides more outcomes than the take-home and the panel debrief combined, and unlike those it goes unwritten. Nobody signs their name to it. It shows up at the end, in the summary email, after a loop that tested something else entirely.

Nobody is a fit in the abstract

The claim always has four parts: this person, in this seat, under these constraints, against this bar. Drop one and you're describing a personality, which is neither testable nor any of your business.

Two versions of the same criterion. First: "works well with others." Second: "will carry the pager alone through a weekend for a system they didn't build, and will wake a colleague when the runbook is wrong rather than guess."

The first is a vibe. The second can be wrong, which means it can be checked. Build a round that produces evidence against it and you'll find out. There's no round you can build for "works well with others," which is why teams that lean on it end up grading candidates against whichever colleague they happen to resemble.

The rewrite costs an afternoon. Most startups skip it because the founders think they already agree on what the seat is. They don't, and the loop is where that surfaces.

If no round can test it, it isn't a reason

Every criterion in your fit definition gets a named round attached to it — the round that could produce evidence against it. If nothing on the schedule can do that, the criterion is an impression, and impressions don't get to reject people.

Interviewers aren't neutral instruments. Jason Dana, Robyn Dawes, and Nathanial Peterson ran the cleanest demonstration I know of. In the first of three studies, participants predicting other students' semester GPAs formed impressions just as confidently when the interviewee was answering with a random response system as when the answers were real, and they predicted worse than if they'd skipped the interview and used prior GPA alone. Watching an interview didn't hurt. Conducting one did. Their abstract ends on a single sentence: "Our simple recommendation for those making screening decisions is not to use them."

That's students predicting GPAs, not managers predicting job performance, and the gap matters. What transfers is the mechanism: a confident read built on noise, crowding out the information you already had.

Structure is the fix. Google reports that its structured loops predicted on-the-job performance across functions and levels, saved about 40 minutes per interview in interviewer time, and left rejected candidates 35% happier than those who went through an unstructured one. That's a company grading its own homework, so take it as a ceiling. The independent version is stronger than Google's marketing needs it to be. When Sackett and colleagues re-ran the personnel-selection meta-analyses and corrected for decades of overstated range-restriction adjustments, most selection methods lost validity. Structured interviews came out top-ranked anyway.

Sit with the rejected-candidates number for a second. People can accept losing to a bar. Losing to a mood is what they don't forgive.

You didn't fail by interviewing. You failed by grading a criterion nothing in the loop assessed. Reject someone on collaboration style after four rounds that only ever probed technical depth and you've written a conclusion with no data under it. "Fit" is the word that covers the gap.

The seat has to exist before anyone can miss it

The most common fit failure is a role-definition failure wearing a candidate's face.

Watch a fast-moving company open a seat nobody has held yet. Call it the first data engineer. One interviewer describes someone who builds the warehouse and owns the models end to end. Another describes a partner to the analysts, mostly unblocking their queries. A third thinks the job is finally making the dashboards trustworthy. All three are senior. All three are sincere. All three are describing different jobs. The candidate answers each of them accurately and then gets measured against a composite that exists in nobody's head.

This isn't only a hiring problem. Gallup's engagement indicator puts the share of U.S. employees who say they know what's expected of them at work at 49% as of May 2026. Role clarity is scarce for people who already have the job. Expecting it to show up spontaneously for a job that doesn't exist yet is optimistic.

Here's a cheap diagnostic, and a slightly brutal one. Before the req goes out, have every interviewer write one paragraph, separately: what does this person do in week six, who do they answer to day to day, what do they own that nobody else touches. Read the paragraphs side by side. If they don't converge, there's no seat yet. There's a wish, and nobody can fit a wish.

Your loop is the honest demo

Culture is behavior under constraint. A hiring loop is a constraint, so the loop is the most honest thing a candidate sees.

They read it as an operating model, because that's what it is. Rounds that appear and disappear mid-process. Headcount numbers that don't agree between two conversations a week apart. A decision-maker named early who never materializes. A timeline that quietly doubles. None of that reads as a scheduling problem — it reads as how you behave when you're busy. The people who pick up on it fastest are the ones with other options, which is a filter you didn't design and wouldn't choose.

If your loop can't hold its shape for six weeks under normal load, better to learn that before a real deadline tests the same thing.

Write the fit spec

The artifact is small: one table, agreed before the req goes live, a row per criterion. Here's one filled in for that first data engineer. The rows change per role. The shape doesn't.

Claim about the seat	What would disprove it	Round that tests it	Disqualifying signal
Ships against a charter nobody has written yet	Needs the spec settled before starting	Scenario round where the requirements contradict each other	Asks for the org chart before the problem
Makes the analysts self-sufficient instead of becoming their queue	Optimizes for doing the work themselves	Live teach-back on something they built	Can't explain the work to a non-specialist
Tolerates a role whose shape changes every quarter	Wants a fixed remit before committing	Direct question, asked the same way of every candidate	Treats a scope change as a broken promise
Picks a schema under incomplete information and defends it later	Waits for more data before deciding	Design round with a deliberately thin brief	Won't commit to a call under uncertainty

Then one governing rule. Rejection reasons must come off the table. If someone in the debrief raises a reason that isn't on it, either the spec was incomplete — add the row, re-run the round that tests it — or the reason wasn't real, and it doesn't get to decide.

I build this kind of thing for a living, though usually for machines. rubric-bench exists because an LLM scoring free-text answers is a production dependency: you pin a golden set of answers to known verdicts, re-score after every prompt edit or model bump, and fail the build when the verdicts move. Nobody would ship a grader that way and skip the tests. A hiring panel is the same object with worse instrumentation — a grader nobody regression-tests, running a rubric nobody wrote down.

The spec isn't free. Four rows with a round attached to each adds a week to a loop that a twenty-person company may not have. Published criteria are rehearsable: say "we test for scope pushback" out loud and you'll meet candidates who've practiced pushing back on scope. And a spec tight enough to be testable will screen out the atypical hire who would have redefined the seat. Pay it anyway. The alternative isn't "no spec." It's a spec living in four people's heads, applied at the end, where nobody has to defend it.

The counter-argument

Fit isn't fake, and treating it as pure bias overcorrects.

Lauren Rivera's study of elite professional service firms found that evaluators sought candidates culturally similar to themselves in leisure pursuits, experiences, and self-presentation, and that concerns about shared culture "often outweighed concerns about absolute productivity". Read closely, that's a finding about unstated fit rather than fit as such. Her setting was law, banking, and consulting, not early-stage startups, and it's a case study built from 120 interviews rather than a controlled trial. The mechanism travels anyway. A criterion nobody wrote down collapses into similarity-to-me, because that's the only yardstick left in the room.

Genuine style mismatches exist too. Someone who does their best work in a quiet queue will be miserable in a seat that's five customer calls a day, and hiring them is unkind to both parties. So style can be a real reason. What it can't be is unstated, because unstated fit is unfalsifiable, and unfalsifiable reasons attract every bias you have. Name the constraint in the job post and you've been fair. Discover it in round four and call it fit, and you haven't.

Back to the rejection note

Write the spec before the req, attach a round to every row, and forbid off-spec reasons in the debrief. That part's mechanical.

The sentence at the end is the harder part. Tell people what you actually decided against. "We wanted someone who's built a warehouse from nothing before, and we couldn't establish that from the loop" is a sentence a person can act on, and one you can only write if you did the work to know it. "Not the right match on either side" is what you send when you didn't.

Context Is Four Different Words: The Window, The Corpus, The File, and The Session

Michael Tuszynski — Sun, 16 Aug 2026 16:34:16 +0000

Someone on your team says the agent "needs better context." Four people nod. They are agreeing about a word and disagreeing about the work, and nobody finds out until two of them have built the wrong thing.

I run four systems that all get called context, and they have almost nothing in common. One is a hard capacity limit. One is a 35-million-token corpus. One is a 4.7KB file I hand-edited last week. One evaporates when the session ends. Same word, four owners, four failure modes, four different afternoons of work.

Here is how I keep them apart.

Job one: context as capacity

This is context as a number. The window. It has a ceiling, you can exceed it, and when you do, something gets dropped or summarized whether you like it or not.

Capacity is the only one of the four you can measure without judgment, which is why it dominates conversation. It is also the least interesting, because being under the limit tells you nothing about whether the right things are in there.

My own number is worth stating plainly. Before I type a single word, my agent has already loaded a global instruction file, a shared cross-tool rules file, a workspace guide, a memory index, and a lessons file. Together that is about 75,000 characters, call it 19,000 tokens, spent every session on standing orders. That is my floor, not my usage.

Job two: context as corpus

This is context as a pile of things you could retrieve. Vector store, RAG index, semantic search. It is not in the window; it is the universe you draw from to fill the window.

Mine is Postgres with pgvector, embeddings served by a local model:

select sum(token_count), count(*), round(avg(token_count))
  from chunks where status = 'active';
-- 35,035,979 | 159,669 | 219

Thirty-five million tokens of retrievable material against a window that holds a fraction of a percent of it. The binding constraint was never storage. It is selection.

The breakdown is the part that changed how I think about this. Of those 159,669 chunks, session transcripts account for 99,259. Archived history from two prior systems adds another 51,165. The hand-curated knowledge vault — the notes I actually wrote on purpose — is 2,515 chunks, about 1.6% of the corpus. The rest is exhaust.

That ratio is not a bug I am about to fix. It is what a retrieval corpus looks like after eighteen months: mostly a record of what happened, with a thin seam of what you decided.

Job three: context as instruction

This is context as a durable file you hand-author and version. CLAUDE.md, AGENTS.md, cursor rules, system prompts checked into the repo. It loads every session, it is read by a machine but written by a person, and it goes stale silently.

The artifact:

$ ls -la ~/.codex/AGENTS.md
~/.codex/AGENTS.md -> /Users/mpt/.claude/AGENTS.core.md

One 4.7KB file of rules — never commit secrets, never deploy without asking, batch privileged steps into one paste-able block — symlinked so that two different vendors' coding agents read the identical text. I edit one file; Claude Code and Codex both change behavior. That is context as configuration, and it has more in common with a dotfile than with anything in jobs one or two.

Its failure mode is drift, not overflow. A stale rule is worse than a missing one, because the agent follows it confidently.

Job four: context as session state

This is what is live right now: the conversation so far, what you already tried, the decision you made twenty minutes ago and have not written down. It is the only one of the four that dies on restart.

Everyone underestimates this one until a session compacts mid-task and the agent cheerfully re-litigates something it settled an hour ago. The fix is unglamorous. Write state to disk on purpose, or accept that it is gone.

The two axes nobody separates

Capacity and curation are not the same axis, and treating them as one is where the money goes.

	Small window	Large window
Curated input	Fast, cheap, works	Works, costs more than it needs to
Uncurated input	Overflows immediately	Fits, and quietly degrades

The bottom-right cell is the expensive one, because everything looks fine. It fits. Nothing errors. The output is just worse in ways that do not announce themselves.

Anthropic's engineering team makes this argument in their context-engineering post, framing the model as having an "attention budget" and defining good practice as finding "the smallest possible set of high-signal tokens." Take the framing seriously but note the position: the company describing your window as a scarce resource also sells the window. That is a claim worth checking against someone with no such incentive.

There is one. Liu et al., published in TACL, tested how models actually use long inputs on multi-document QA and key-value retrieval. Performance was highest when the relevant information sat at the beginning or the end of the input and dropped when the model had to reach into the middle — and that held even for models explicitly built for long contexts. That is peer-reviewed, it is not selling a context window, and it says position inside the window changes the answer. Capacity is not the variable. Placement is.

The widely-cited "context rot" work points the same direction, and the lab behind it sells a vector database, so "long contexts degrade, retrieve instead" is a conclusion they profit from. It survives the objection mainly because Anthropic cites it against its own product interest.

Where this breaks

The four-way split is vocabulary, and vocabulary has a tax. On a two-person team where the same person owns all four systems, this is overhead — go build something. The split earns its keep when the four jobs have different owners, which is exactly when the argument starts.

It also breaks at the edges. Prompt caching sits across capacity and instruction. A compaction step is session state being rewritten into capacity. I am drawing lines through something continuous because the lines make meetings shorter, not because the territory has them.

The question I ask now

When someone says we need better context, I ask which one before anything else: is this a room problem, a retrieval problem, a rules-file problem, or a state problem?

Room is a budgeting question. Retrieval is a ranking and evals question. A rules file is a twenty-minute edit. State is a persistence question. Four different afternoons, four different owners, one shared noun.

I got the shape of this from Shawn Wallace, who ran the same play on the word "agent" and found three jobs hiding under it. The form travels because the industry keeps doing this: a word gets useful, then absorbs every adjacent thing anyone shipped, then stops carrying information.

Writing this post, I ran the count against my own corpus and got 159,669 active chunks. My workspace guide — the instruction file, job three, the one loaded into every session I start — said 126,000. It had been verified eight days earlier. The file whose job is to describe the corpus was wrong about the corpus, and it had been telling every agent I ran the wrong number all week.

Job three going stale about job two. I had to write the taxonomy to catch it.

Karpathy's LLM Wiki Works. Here's What Broke When I Ran It at Real Scale.

Michael Tuszynski — Thu, 13 Aug 2026 18:38:58 +0000

Andrej Karpathy's LLM Wiki gist from April is right about the important part: for personal knowledge, a markdown wiki that Claude maintains beats a RAG pipeline. I've been running a version of this pattern in production against my own life for six months. Karpathy is right — and he stopped at exactly the point where the interesting problems start.

Here's the pattern, briefly, then what actually breaks.

The claim, in one paragraph

You keep raw sources in one directory. You keep a wiki of markdown pages in another. A CLAUDE.md file tells Claude how the wiki works: naming, folder layout, what to do on ingest, how to handle contradictions. New source comes in, Claude reads it, writes or updates concept pages, links them with [[wikilinks]], updates an index. When you ask a question, Claude reads the wiki, not the raw sources — the synthesis has already been done at ingest time. The whole thing sits under ~100k tokens, so Claude can hold the index in context and reason over it directly. No embeddings, no vector store, no infrastructure. Two clean community implementations to start from: nvk/llm-wiki and Ar9av/obsidian-wiki.

That's the whole thing. It works. It works better than you'd expect. And every problem I've hit with it shows up in the same three places.

My setup, so the failure modes are grounded

Mine is a NEXUS workspace: an Obsidian vault split into 000-inbox/, 200-knowledge/, 300-entities/, 400-daily/, plus per-domain context files under 200-knowledge/context/ — one each for finance, health, content, jobsearch. Every domain has its own schema and its own conventions. A top-level CLAUDE.md sets the operating rules; each domain file specializes them. Entities (people, companies, services, properties) are single markdown files named by their wikilink text, so [[Janney]] grep-searches straight to vault/300-entities/orgs/Janney.md. Daily logs at vault/400-daily/YYYY-MM-DD.md are the append-only session record. A MEMORY.md at the root holds hard-won lessons — 34 entries and counting.

Volume: several hundred wiki pages, four active domains, daily writes going back to February. Well past Karpathy's toy scale, well under any RAG-scale problem. This is the middle band the original gist never addresses.

Where it broke: three failure modes, all the same shape

1. Drift between the "recent changes" section and the "how it works" section of the same file.

The gist assumes ingest and update. It doesn't say what happens when something you documented in April is contradicted by what you did in July. In practice, Claude appends a new dated entry to a changelog at the top of the file, feels done, and leaves the reference section below still describing the dead system as current. A reader — human or agent — trusts the reference section for "how does this work today," and it's wrong. Writing a dated log entry feels like recording the change, so the present-tense prose that people actually read for "how does this work now" never gets revisited. The rule I now enforce: a migration isn't done when the changelog says so; it's done when the reference sections match.

2. Silent contradictions between pages.

Two entity cards for the same account — one calls the institution "Janney," the other calls it "Janney Montgomery Scott LLC" — and a downstream aggregation double-counts because the dedup key is the name. RAG failures are silent because retrieval quietly misses a chunk. Wiki failures are silent for a different reason: two pages both exist, both are internally consistent, and neither knows about the other. Claude doesn't spontaneously notice.

3. Orphan pages and dead wikilinks accumulate faster than lint catches them.

Karpathy's original suggests running lint occasionally. At three hundred pages, "occasionally" is not a frequency — it's a hope. Weekly lint recovers most of it. Nothing recovers a stale page that nobody links to and nobody re-reads.

All three are the same shape: the wiki has no immune system against its own past. It compounds knowledge, but it also compounds staleness, and it needs help distinguishing the two.

Three things I bolted on

A drift linter that fails the build. A Python check that walks the vault, cross-references any "retired" or "decommissioned" term against every file, and flags live mentions of dead systems in the canonical docs. It runs pre-commit and on a scheduled cron. When Ghost got replaced by an Astro static site, the linter caught nine files still describing the Ghost publish path as current — including the top-level project README. Without it, a reader would have reached for dead credentials.

Per-domain sub-schemas, not one giant CLAUDE.md. The gist treats the schema as one file. That doesn't survive the second domain. Finance and health have almost no overlap in vocabulary, conventions, or what a "page" is. My top-level CLAUDE.md sets universal rules (voice, safety, wikilink format, verification protocol). Each domain has its own <domain>-context.md that specializes them. A session working on finance loads the domain file; a session working on content loads a different one. Same architecture as scoped CSS.

A vector index alongside the wiki, not replacing it. Once I passed roughly 100k tokens of wiki content plus 100k tokens of session logs plus tens of thousands of legacy conversations, Claude couldn't hold the index in context anymore. The gist's answer to this is "at that point, use RAG." My answer is: keep the wiki as the authored, curated layer, and add a pgvector index over the whole corpus — vault plus session history plus legacy archives — for recall. Around 126k chunks on Postgres 17 with mxbai-embed-large embeddings running on a Mac mini. Not a replacement for the wiki; a way to find the right wiki page when your brain forgot which slug you wrote it under two months ago. The RAG layer is a retrieval tool for me and my agents. The wiki is still the canonical knowledge.

Where the pattern still wins

For synthesis, entity resolution, and the "how do these three things I noticed relate" question, nothing else comes close. RAG genuinely doesn't do it — the structural argument about retrieval having no accumulation is correct. When I ask "what's the through-line across the last month of daily logs on finance," the answer draws on synthesis pages that were written when each log came in. The work happened once, at ingest.

Entity cards are the other win. A single file per person or company, named by its wikilink text, becomes the source of truth. Everything else references it. When a fact about an entity changes, you change one file. This is the boring pattern that pays for itself every day.

Where it stops — the honest edge cases

The wiki pattern breaks when any of these are true:

Multi-actor writes. If two agents write to the same wiki page concurrently, there is no lock. Mine is single-writer by convention. If you fan out coding agents that all edit SESSION-STATE.md, you need a lockfile — the gist has nothing to say about this.
Recall across the boundary of a Claude session. A wiki isn't memory. Claude reads it fresh each session. Cross-session memory ("what did we decide last Tuesday") needs a separate store, which is what daily logs plus the vector index are for.
Anything past ~100k tokens of hot wiki content. The load-bearing assumption is that the index fits in context. When it doesn't, either you shard by domain (my approach), add a retrieval layer, or accept degraded answers.
Anything anyone else needs to read without Claude in the loop. A markdown wiki is legible to humans. A markdown wiki written by Claude, for Claude, over months has enough house-style shorthand that a new reader without the schema will miss context. The wiki is legible; the accumulated conventions are not.

What to actually do

Start where Karpathy says. One folder, one schema, five sources, in one domain you're actively thinking about. Do it for two weeks before adjusting anything.

At the point it starts feeling like it's working — around thirty or forty pages — put three things in before you regret not having them: a drift check that flags dead terms in live docs, one lint pass a week that actually runs, and a rule that any migration updates the reference section, not just a changelog. That's the operating discipline the pattern needs and doesn't ship with.

Add a vector index only when the wiki genuinely stops fitting. Not before. Karpathy is right about that too — most people reach for RAG when they should have written a schema.