DEV Community: Human On Mute 401

Your coding agent's "done" is a claim. Plan the checks between steps

Human On Mute 401 — Fri, 13 Nov 2026 10:21:10 +0000

Most agent workflows I see are not one request. They are a run of them: add the model, wire the endpoint, add validation, write the migration, build the report. Each step lands on code the agent wrote a few minutes earlier.

Each step also ends the same way. The agent runs its tests, they pass, and it tells you it's done.

A preprint from two Amazon researchers measures what that "done" is worth when you stack twenty of them. Short version: not enough to build on without checking.

What the benchmark did

BOBA-Bench (Cuadron and Gupta, arXiv 2611.02207, 1 November 2026) turns the commit histories of 140 open-source Python projects into 280 chains of 20 requests each. Each request goes to the agent in a fresh conversation, on whatever the previous request left. Hidden tests grade every step. The agent is never told the result.

Ten agents, from open-weight to frontier models. Headline numbers:

51% of chains completed with agents left alone. 39% got through with no missed request and no broken earlier test.
66 to 84% of the individual requests inside those chains were completed.
A request issued on broken code failed about twice as often as one on a clean base.
Only 14 to 33% of failing tests were ever repaired later.
Asked whether an implementation was correct, agents were right 53% of the time.
One external "that's wrong" plus one repair round lifted chain completion from 51% to 59%.

Two details matter for engineers. First, once checked, agent-written code was nearly as good a base as the original developers' code (within 4 points for every model). The code wasn't the problem. Unnoticed failures were. Second, in 23 to 43% of attempts a later request broke a test that had passed earlier, usually one of the agent's own.

A second preprint, Multi-step SWE (Klavis AI, arXiv 2611.07166), found the same shape on 5 to 15 step tasks: the best of nine models passed 56.8% of steps but solved 18.5% of whole tasks.

Mapping it onto engineering work

This part is my inference, not the paper's.

The agent's own tests are not your gate. If it misreads the request, it tests its misreading. Your gate is a check it didn't write and can't edit.
Some steps are foundations. A schema, an interface, a shared helper. Later steps inherit their mistakes. Those need a real check before you issue the next request.
Regression runs every step. Re-run the earlier steps' checks, not only the new one.
Agent review is a hint, not a gate. The reviewing models either waved through most wrong code (77 to 95% accepted) or rejected about half the correct code (41 to 58%).
Feed back facts. The paper found that naming the failing test once did about what three rounds of bare "that's wrong" did. Paste the test name and the error.
Multi-agent setups: the authors recommend that the verifier's result reach the sub-agent that did the work, not stop at the orchestrator.

The artefact: a chain plan

Paste this into the PR description, the ticket, or a CHAIN.md before you start a multi-step agent session. Fill the left columns first. Fill "Result fed back" as you go.

## Chain plan

| # | Request | Builds on | Foundation? | Check the agent can't edit | Earlier checks re-run | Result fed back |
|---|---|---|---|---|---|---|
| 1 | Example: add `discount_code` to Order model | - | Yes | Migration applies on a copy of prod schema; one real order loads | - | Pass |
| 2 | Example: expose field on `GET /orders/:id` | 1 | Yes | Contract test from API spec (held outside agent workspace) | 1 | Fail: `test_order_contract_discount_null` expected null, got "" |
| 2r | Example: repair after named failure | 1 | - | Same contract test | 1 | Pass |
| 3 | Example: validate code format on POST | 2 | No | Hand test: one valid, one invalid code via curl | 1, 2 | Pass |
| 4 | Example: monthly discount report | 2, 3 | No | Report totals match a SQL query written by a person | 1, 2, 3 | Not run yet: do not start step 5 |

Rules I'd attach to it (again, mine):

A step marked Foundation = Yes does not get a successor until its check column says Pass.
"Check the agent can't edit" names a file, test or person. "Agent's tests pass" is not allowed in this column.
Failures go back as the exact test name or observed behaviour, then one repair round, then re-check.
If nobody can run a step's check today, the chain stops there today.

Limits

Both papers are preprints written by the benchmark builders. BOBA-Bench is Python only and removes the human on purpose, so real chains with someone glancing between steps may do better. The authors call their own advice "patches, not a principled approach", and expect it to become unnecessary once agents can certify their own work. Even perfect feedback recovered only part of the loss, and the repair round raised regression rates by 3 to 9 points.

Sources

Cuadron and Gupta, "Built on Broken Assumptions (BOBA): Why Coding Agents Cannot Be Left Alone", arXiv 2611.02207: https://arxiv.org/abs/2611.02207
Zeng, Liu and Lin, "Multi-step SWE: Can Coding Agents Keep Building on Their Own Code Over Long Horizons?", arXiv 2611.07166: https://arxiv.org/abs/2611.07166

How many requests do you let an agent stack before you run the thing yourself, and has a silent miss ever surprised you several steps later?

Your coding agent has opinions. Log the calls it made for you

Human On Mute 401 — Thu, 12 Nov 2026 08:07:47 +0000

Code review is good at one thing: finding mistakes. A failing test, an off-by-one, a missing null check. Something is wrong, and a reviewer can point at it.

A lot of engineering work has no mistake to find. Cache or recompute? Retry in the client or in the queue? Measure latency at p95 or p99? Both answers are defensible. Someone has to pick, and the pick shapes everything after it.

Coding agents now sit in the middle of those choices. They draft the design note, propose the approach and argue for it. A trial published by Anthropic on 27 October 2026 suggests we should check who is actually making those calls.

What the trial found

Researchers Amaç Herdağdelen, Hailey Joren and Alex Tamkin gave 50 professional analysts a realistic statistics task with Claude Code. The task had four open decisions, each with two reasonable options, such as which outcome measure to use and how to weight the data.

A coin flip set which option each analyst's agent leaned towards. The agent was told to make the strongest honest case for its lean, to change its mind if the evidence turned decisively against it, and never to misrepresent results. The analysts didn't know.

Analysts matched the agent's lean on 86% of the four decisions. About 50% would mean no effect.
The four coin flips alone predicted 82% of final recommendations, against a 61% baseline.
In 38 cases an analyst stated a preference first. When the agent disagreed, they switched 60% of the time. When it agreed, 6%.
Afterwards, 77% explained their choices with the agent's arguments. 15% mentioned the agent.
Experience didn't predict who deferred.

The agent also helped. Of the 11 analysts who chose to work without it, six made serious methodological errors, against three of the 50 who used it. The groups weren't comparable, so treat that as a hint, not proof.

The researchers' own summary of the review problem: "Checking AI-assisted work for errors, or against a reference answer, assumes there is a correct answer."

Mapping it onto engineering work

This next part is my inference. The study used analysts on a data task, not engineers on a codebase.

Agents are now involved in exactly the kind of choices this study tested. Think of an architecture decision record drafted with an agent, or a pull request where the agent picked the retry strategy. A spike write-up where it chose the benchmark. In each case the reviewer checks that the code works. Nobody checks whether the choice was the author's.

Two habits make it worse:

One-sided options. The researchers found the agents rarely made a strong case for the other option. An ADR with one well-argued option and a strawman isn't really a decision record.
Lost provenance. If 77% repeat the agent's argument and 15% credit it, then "Considered alternatives" sections are partly the agent's memory, not yours. Nobody is lying. Memory is just generous to its owner.

The artefact: a judgement-call log

Paste this into your PR template or ADR template. Fill one row per open choice that could change behaviour, cost or a threshold. Skip trivia. The researchers warn that surfacing every decision "could quickly become overwhelming".

The example rows are illustrative. They are not from the study.

## Judgement calls in this change

| Call | Option A | Option B | Chosen | Result under the other option | Argument came from | Independent check |
|---|---|---|---|---|---|---|
| Example: retry location | Client-side retry with backoff | Queue redelivery | B | Client retry also passes tests; duplicate-write risk higher | Agent proposed B; I added the idempotency check | Reviewer read both options before seeing agent output |
| Example: latency SLO metric | p95 | p99 | p99 | p95 alert would not fire on last month's incident | Mine | Same model used by reviewer: treat as one view |
| Example: cache strategy | Cache per request | Cache per tenant | A | Per-tenant cuts DB reads in a load test | Both | Not checked yet |

How to use the columns:

Option A / Option B: write both as if you meant them. If you can't argue the one you rejected, ask the agent for "the strongest case for option A, then the strongest case for option B".
Result under the other option: the important column. If the outcome flips (a test, an SLO, a cost line), the reviewer should know this row is the real decision.
Argument came from: mine, the agent's, or both. It takes ten seconds and it's the only provenance you'll get.
Independent check: if the reviewer used the same model with similar prompts, say so. Two signatures from one model are one opinion.

A quick self-test

Before you merge, try this on one row. Open a fresh agent session, frame the question neutrally and ask for the case for the option you rejected. If you find it more convincing than you expected, keep the row and flag it for the reviewer.

This is a rough check, not an experiment. The study could measure deferral only because the leans were randomised. But it's cheap, and it shows you where your view is thin.

Limits

The leans were deliberately strengthened. With no lean assigned, Claude Opus 4.8 made the same five choices in 112 of 123 runs, so models do lean, but the effect may be smaller in normal use.
One task, one field, one model, about an hour long.
Anthropic studied its own model and published the results itself.

Sources

How AI agents shape experts' judgment calls, Herdağdelen, Joren and Tamkin, Anthropic, 27 October 2026.
Who's Aligned to Whom? How AI Agents Shape Experts' Research Judgment Calls, full paper (PDF).

In your last merged PR that an agent helped with, which design choice would you struggle to say was yours?

Give AI the work you can already check

Human On Mute 401 — Tue, 10 Nov 2026 12:54:31 +0000

When a team reports a high share of pull requests touched by AI, the number is real. It still does not tell you whether the tool is on the work that decides how a service behaves.

Code generation, tests, refactors and summaries are easy to review. You can run the tests, read the diff and reject a bad patch in minutes. Architecture decisions, design trade-offs, requirements and incident analysis are harder to verify, and the stakes are higher. "Good" takes argument.

I think many engineering teams put AI on the first pile and leave the second almost untouched. A study published last week gives that hunch some evidence. The subjects are scientists, not software teams, so the engineering mapping below is mine, not Anthropic's.

What 207,049 sessions show

On 5 November 2026, Anthropic published "AI for scientific discovery: How researchers use Claude". The team analysed 207,049 research sessions on consumer plans from 2 August to 30 September 2026. They compared each task's share of AI sessions with its share of a researcher's ordinary time.

Data analysis: 33.7% of Claude sessions and 48.1% of Claude Code sessions, against 27.6% of time.
Writing and communication: 32.0% of Claude sessions, against 16.0% of time.
Data gathering: 11.5% of Claude sessions and 11.9% of Claude Code sessions, against 28.6% of time.
Ideation and hypothesis generation: 4.2% of Claude sessions and 2.1% of Claude Code sessions, against 7.1% of time.

Writing gets twice the AI share that time would suggest. Gathering and ideation get less.

Use follows what is easy to check

Anthropic also scored how well AI could handle each research task. Tasks in the top 20% on that score got about twice as much Claude use as time would predict, and about four times as much on Claude Code. Tasks in the bottom 20% were barely used. Anthropic's summary: "scientists appear to use AI where it works best."

That capability score has a correlation of 0.78 with how suitable a task is for reinforcement learning. Anthropic describes those tasks as ones where success is easy to verify, many examples can be simulated and feedback can be delivered quickly.

Here is my inference, not Anthropic's claim. Work with a clear pass mark and a short feedback loop is where the tool performs well, so that is where people use it. Work that needs a longer argument gets less AI, even when a team would say it matters more.

On a software team, I would expect the same split. Generated functions, tests, mechanical refactors and summaries have a quick review loop. System design, trade-off write-ups, requirements and incident analysis do not.

A two-axis map you can copy

Score each recurring task on two questions. Can you check the result quickly? Does the task matter to the outcome?

Paste the table below into a doc, replace the example rows with your own work, and assign a review owner for anything that matters. The rows are examples from ordinary engineering work. Anthropic did not test them.

Task	Easy to check?	Matters to outcome?	AI role	Review owner
Generate a function with existing unit tests	Yes	Medium	Draft the code	Pull request reviewer
Write or extend unit tests for a known behaviour	Yes	High	Draft cases and assertions	Service owner
Mechanical refactor (rename, extract, format)	Yes	Medium	Apply the change	Pull request reviewer
Summarise a pull request or stand-up notes	Yes	Low	Draft the summary	Author
Architecture decision or design trade-off	No	High	Draft options and the counter-case	Named decision owner
Requirements or incident analysis	No	High	Structure the notes, list open questions	Named investigation or product owner

Start with the work that is easy to check and matters to the outcome, and keep a review step. Use the tool freely on easy, low-stakes work, and do not count that as a win for the service. For hard, high-stakes work, keep a named human owner. Let AI draft and argue the other side, and put your effort into making the task easier to check.

Make the hard work easier to check

You can often move a hard task one step towards checkable before you ask the model for a draft.

Write the acceptance tests first, then ask AI to implement against them.
Fill an architecture decision record template (context, options, decision, consequences) before you ask for a recommendation.
Keep a short set of worked examples of a good requirement, and compare new drafts against them.
For incident analysis, agree the timeline fields and evidence rules up front, then use AI to organise the notes.

The cost of review still lands on someone. Name that person before you scale the prompt.

The sessions are on consumer plans, so enterprise use and other tools sit outside the frame. The findings are correlations, and the subjects are scientists. Treat the engineering mapping as a hypothesis to test on your own work.

If you want the longer argument, I wrote a fuller essay on Medium: We hand AI the work that's easy to mark.

Sources

AI for scientific discovery: How researchers use Claude, Hermle, Jones, Althoff and colleagues, Anthropic, 5 November 2026.
ChatGPT's expanding world of work, OpenAI Economic Research, 26 October 2026.

Of the last five things your team asked AI to do, how many had a test, a diff or a known total you could check in a few minutes?

I stopped setting goals. Here's what happened.

Human On Mute 401 — Sat, 07 Feb 2026 11:13:12 +0000

92% of people fail their goals.

82% of employees are at risk of burnout.

$322 billion lost annually to burnout-related productivity drops.

Maybe the problem isn't people. Maybe it's goals.

Neuroscientist Anne-Laure Le Cunff spent years at Google before getting her PhD at King's College London. Her research led to a radical conclusion:

Linear goals in a nonlinear world create suffering, not success.

Her book Tiny Experiments proposes an alternative:

Replace goals with experiments.

The difference?

When a goal fails, you failed. When an experiment doesn't work, you learned something. Same outcome. Completely different psychological impact.

Three lessons from the book that changed how I think about work:

Procrastination is protection. Neuroscience shows we delay tasks to avoid negative emotions. Instead of fighting resistance, ask what it's telling you.
Hustle culture makes you dumber. Research shows burned-out employees have 60% reduced focus and 32% lower productivity. Working more doesn't mean achieving more.
Uncertainty is opportunity. When you stop treating life as a path to follow and start treating it as a laboratory to explore, fear becomes curiosity.

I've started applying this to my own work. Instead of quarterly goals, I run weekly experiments. The pressure is down. The learning is up.

What would change if you replaced your biggest goal with an experiment?

Harvard just proved AI tutors beat classrooms. Now what?

Human On Mute 401 — Mon, 05 Jan 2026 20:57:53 +0000

AI Tutoring: The Promise and the Divide

The Classroom That Never Changed

I've been thinking a lot about my kid's classroom this week.

Walk in there and it looks... remarkably like my classroom did. And my parents' classrooms before that:

One teacher
30 kids
Fixed schedule
Same pace for everyone

The Breakthrough Study

Then I read the Harvard study published in Nature Scientific Reports last June, and something clicked.

The Setup:

Proper randomized controlled trial
Physics students learning from AI tutor vs. active learning classroom
Not passive lectures—active learning (already the gold standard)

The Results:

AI group: 2x+ learning gains
In less time
Higher engagement and motivation

This wasn't ChatGPT doing homework. The researchers carefully engineered the AI to follow pedagogical best practices:

Scaffolding content
Managing cognitive load
Giving immediate personalized feedback

The kind of teaching that simply cannot scale with one human and 30 kids.

The Global Crisis

UNESCO's Global Teacher Report warns:

Region	Need by 2030
Worldwide	44 million additional teachers
Sub-Saharan Africa	15 million teachers
Global education funding	Expected to fall 25% by 2027

The Promise

On paper, AI tutoring is exactly what education has been missing:

✅ Infinite patience
✅ Infinite personalization
✅ Near-zero marginal cost

The Uncomfortable Truth

But here's what keeps me up at night.

Home Internet Access:

High-income countries: 87% of students
Low-income countries: 6% of students

Current Reality:

AI tutoring market booming in North America, Europe, and Asia-Pacific
The regions that need educational transformation most urgently are the regions least equipped to access it

The Choice

We're facing a choice:

Path A: AI gives every child on Earth a personalized tutor in their own language

Path B: Two parallel education systems

AI-powered learning for the privileged
Overcrowded classrooms for everyone else

The technology is proven. The question is whether we'll choose to deploy it equitably.

That's not a technology problem. That's a policy problem.

Sources

Kestin et al., Nature Scientific Reports (June 2025)
UNESCO Global Report on Teachers (2024)
UNESCO Global Education Monitoring Report (2023)

If this resonated: Share it with someone thinking about the future of education.

I Tested 5 AI Customer Service Agents With the Same Complex Billing Issue – None Escalated Correctly

Human On Mute 401 — Sun, 04 Jan 2026 17:51:35 +0000

I Tested 5 AI Customer Service Agents With the Same Complex Billing Issue — None Escalated Correctly

None escalated correctly. AI customer service systems are systematically failing to escalate complex issues to humans, creating customer frustration and eroding trust.

Summary

Last week, I tested a complex billing discrepancy across five different AI customer service agents on separate platforms.

Each system claimed it could resolve the issue.

Not one correctly escalated to a human.

This is not an isolated experience. It reflects a structural failure in how AI customer service systems are designed, deployed, and measured.

Key Data Points

39% of AI customer service bots were pulled back or reworked due to errors in 2024
Customer complaints about AI service rose 56.3% year-over-year in China
Resolution rates range from:
- 17% for billing issues
- 58% for returns and cancellations
75% of customers say chatbots struggle with complex issues
85% of consumers believe their issues require human assistance
Global trust in AI dropped from 62% (2019) to 54% (2024)

What Happened in Practice

Each chatbot behaved in nearly identical ways:

Confidently stated it understood the problem
Offered generic troubleshooting steps
Failed to detect when human judgment was required
Never triggered escalation logic

The escalation mechanisms vendors advertise simply did not activate.

This failure matters because companies invested $47 billion in AI customer service in the first half of 2025 alone, yet 89% of that investment delivered minimal returns.

The Trust Breakdown

The data tells a different story than vendor marketing:

Customer complaints about AI service increased 56.3% year-over-year
For billing problems, AI resolution rates drop to 17%
Users increasingly recognize when they are being deflected rather than helped

Global confidence in AI customer service:

62% → 54% globally (2019–2024)
50% → 35% in the United States

Users are not confused — they are frustrated.

Real-World Consequences: The Air Canada Case

Air Canada learned the risks the hard way.

Their chatbot hallucinated a bereavement discount policy
A customer relied on the information
Air Canada argued the chatbot was a separate legal entity
A tribunal rejected that argument
The company was required to honor the fabricated policy

AI hallucination rates range from 3% to 27%.

This is not an edge case — it is a known limitation.

Enterprise AI Failure Rates

The situation is worse inside large organizations:

Only 5% of enterprise-grade generative AI systems reach production
70–85% of AI projects fail outright
Gartner projects 40% of agentic AI projects will be scrapped by 2027

Despite this, deployment continues — often with reduced human access and deeper menu nesting for escalation.

What Actually Works

The organizations seeing success share common traits:

AI is used as assistance, not replacement
Humans remain accessible and visible
Success metrics include escalation accuracy, not deflection rates
Bots are trained to recognize complexity, not mask it

Most companies are doing the opposite:

AI used as a barrier
Slower response times
Hidden contact options
Endless conversational loops

By the time a human is reached, trust is already gone.

Conclusion

This outcome is not inevitable.

The technology can work — when customer outcomes are prioritized over cost reduction.

Today, 85% of consumers believe their issues require human assistance.

Given the evidence, they are probably right.

Key Sources & Citations

Investment & Failure Rates

CMSWire — $47B invested in AI initiatives (H1 2025), 89% minimal returns
ASAPP / MIT — Only 5% of enterprise-grade generative AI systems reach production
Gartner — 40% of agentic AI projects scrapped by 2027
Multiple sources — 70–85% AI project failure rate

Customer Complaints & Trust

China Daily — 6,969 AI customer service complaints in 2024 (+56.3% YoY)
Sobot — Global AI trust fell from 62% (2019) to 54% (2024)
Salesforce — Customer trust dropped from 58% to 42% (2023–2024)
Plivo — 75% say chatbots struggle with complex issues
Plivo — 85% believe issues require human assistance

Resolution Rates & Bot Performance

Plivo — Resolution rates: 17% (billing), 58% (returns/cancellations)
Fullview — 39% of AI customer service bots reworked or pulled in 2024

AI Hallucinations

CMSWire — Hallucination rates between 3% and 27%
EdStellar — 77% of businesses concerned about hallucinations

Legal Case

Moffatt v. Air Canada, 2024 BCCRT 149
Legal analysis: McCarthy, American Bar Association, Lexology