DEV Community: Orvi Das

Your LLM Pipeline Never Throws: Three Guardrails for Silent AI Failure

Orvi Das — Sat, 19 Sep 2026 13:42:27 +0000

A crashed service pages someone at 3am. A model that's quietly gotten worse ships a clean 200 and nobody finds out for six weeks.

That asymmetry is the actual bug in most production AI. Every other component in your stack has a failure signal you can alert on — a non-zero exit, a timeout, a stack trace, a p99 that walks off the chart. A classifier that's drifted has none of those. It returns the same shape of JSON it returned yesterday, at the same latency, with confidence scores that look fine, and the content inside is wrong. Your monitoring is green. Your uptime is 100%. Your approval queue is full of garbage.

The numbers around this are grim and specific. In 2025, 42% of companies abandoned most of their AI initiatives, up from 17% the year before, and large enterprises killed an average of 2.3 projects each at roughly $7.2M in sunk cost per initiative. What gets reported as "the AI didn't work" is usually something narrower: it worked, it stopped working, and the gap between those two events went unobserved long enough that trust never came back.

So instrument the gap. Here are three guardrails, in the order I'd add them to a system that has none.

1. A golden set that runs like a unit test

The cheapest correctness signal you can build is a frozen set of hand-labeled cases and an assertion. Not an eval harness. Not a dashboard. A test file.

# tests/test_golden_set.py
import json

from approvals import classify

GOLDEN = json.load(open("tests/golden_set.v4.json"))  # 200 cases, hand-labeled

def test_golden_set_accuracy():
    wrong = [c for c in GOLDEN if classify(c["input"]) != c["label"]]
    accuracy = 1 - len(wrong) / len(GOLDEN)
    assert accuracy >= 0.92, (
        f"accuracy {accuracy:.3f} below floor; "
        f"regressed: {[c['id'] for c in wrong][:5]}"
    )

The part people skip: run it on a schedule, not just on pull requests. Your code didn't change. That's the whole point. A schedule: trigger in GitHub Actions, every night, on the production model and the production prompt:

on:
  schedule:
    - cron: "0 6 * * *"
  pull_request:

Two traps worth naming. First, golden sets rot — the distribution they represent is the one you had when you wrote them. Version the file (v4, not golden_set.json), append ten fresh production cases a month, and never quietly delete a case that starts failing. That deletion is the drift, recorded. Second, don't assert on a single accuracy number if your classes are lopsided. A 94% accurate spam filter that has stopped catching spam entirely still scores 94% when 94% of traffic is legitimate. Assert per-class recall.

2. Watch the inputs, because they move first

Golden sets tell you the model got worse. They can't tell you it's about to. Labels arrive late — sometimes weeks late, sometimes never — but the inputs arrive in real time, and they shift before your metrics do.

Population Stability Index is the boring, effective tool here:

import numpy as np

def psi(baseline, current, bins=10):
    edges = np.quantile(baseline, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    b = np.histogram(baseline, edges)[0] / len(baseline)
    c = np.histogram(current, edges)[0] / len(current)
    b, c = np.clip(b, 1e-6, None), np.clip(c, 1e-6, None)
    return float(np.sum((c - b) * np.log(c / b)))

Rough reading: under 0.1 is stable, 0.1 to 0.25 is worth a look, above 0.25 means the thing you're scoring today isn't the thing you built for.

Text inputs don't have a natural histogram, so project them onto a scalar first and PSI that. Cosine distance from last quarter's embedding centroid. Token length. Share of requests matching your top-20 known intents. Fraction containing a product name you shipped after the prompt was written — that last one caught a real regression for me, because the model had never seen the feature it was being asked to route.

Log the number daily. A drift metric you compute once during an incident is archaeology, not monitoring.

3. Fail closed, and put a hard ceiling on the spend

The first two guardrails are detection. This one is containment, and it's the one that decides whether a bad Tuesday is an incident or a shutdown.

Give the system an explicit third answer. Not approve, not reject — abstain, and route to a human:

FALLBACK = "human_review"

def decide(case, drift_score):
    label, confidence = classify_with_confidence(case)
    if drift_score > 0.25 or confidence < 0.80:
        metrics.incr("approvals.fallback")
        return FALLBACK
    return label

Now alert on the fallback rate, not on correctness. "The model is punting 3x more than it did last month" is a signal that exists today, requires no labels, and arrives weeks before anyone files a complaint.

The cost side matters more than people expect, because runaway spend is often drift's first visible symptom. When inputs stop matching the prompt, agents retry, re-plan, and re-query — the loop that cost you $40 a day starts costing $400, and it looks like traffic growth until you read the traces. Budget alerts fire after the money's gone; rate limits cap requests per second, not dollars. What you want is pre-flight enforcement: refuse the call before it reaches the provider. baar-core does exactly that — an open-source Python library that raises a 402 on a call that would exceed the cap, with atomic reservation so twenty parallel workers each "under budget" can't jointly blow through it (pip install baar-core). For teams that need the same enforcement per-user with a dashboard on top, that's noburn.dev — it blocks the API call before it fires when a user crosses their budget, rather than emailing you about it afterward.

The part nobody puts in the budget

All three of these are maybe two days of work. They almost never get built, and the reason isn't difficulty.

Shipping the model is a project, with a sponsor, a launch date, and someone's quarterly goal attached. Keeping it correct is an on-call rotation nobody staffed. There's no demo, no launch post, no line item. So the golden set doesn't get written, the drift metric doesn't get logged, and the system runs unobserved until it's wrong loudly enough that killing it is the obvious call — at which point the sunk cost makes a rebuild politically impossible.

Write the tests during the build, while the budget's still open. Retrofitting observability onto a model that's already lost the room is a much harder conversation than adding a cron job in week two.

What's the longest one of your models ran wrong in production before anyone noticed?

Originally published at https://robatdasorvi.com/stories/why-most-ai-automation-dies-within-six-months-of-going-live

I built a Chrome extension after a screen-share disaster reminded me how fragile browsing is

Orvi Das — Fri, 18 Sep 2026 20:11:48 +0000

A founder was sharing her screen during a work call with a CEO. He asked her to check a lead's website. She searched the domain, clicked it, and an adult website opened on the shared screen.

That specific incident came from a public post on X, and it is a much better product brief than most I have written for myself. Here is the post that prompted this build.

It is easy to laugh at a one-off mishap. It is harder to dismiss the pattern. A domain can be mistyped, a search result can be misleading, a link can redirect, or the right page can expose the wrong thing at exactly the wrong time. A shared screen removes the usual private moment between clicking and seeing the result.

For people who demo products, support customers, teach, interview, or work with clients, that is a small but real risk surface. It is not only about inappropriate websites. It is also a personal inbox, a pasted token, a payment detail, an API key, or an unfamiliar domain that imitates a brand you trust.

So I built ScreenGuard, an early Chrome extension that puts a small safety layer between a click and an embarrassing moment.

What it does

ScreenGuard has two related jobs.

First, it warns before a risky destination opens. It looks for signals such as known unsafe or adult domains, phishing-style lookalike domains, and suspicious navigation patterns. The point is not to declare every unfamiliar link dangerous. It is to create a brief, deliberate pause when a link deserves one.

Second, it has a screen-share protection mode. When enabled, it can blur sensitive values on the page, including things such as API keys, tokens, email addresses, and payment-card-style numbers. A small indicator tells you how many items are hidden, and you can reveal them when you need to.

The privacy rule matters here: the extension runs locally in the browser. ScreenGuard does not send browsing history or page contents to a server to make these decisions.

Why I did not make it a "block everything" tool

False positives are the fastest way to make a browser extension useless. A warning that fires on every unfamiliar domain gets trained out of a user's attention in days.

That is why the design is layered. Known dangerous destinations can be blocked under the relevant settings. Heuristic matches get a warning and a choice. Sensitive-content protection is specifically tied to screen-sharing mode, rather than permanently hiding useful information during normal work.

The product has to make risky moments less likely without turning everyday browsing into a sequence of permission dialogs.

What I am looking for now

ScreenGuard is free and currently at v1.0.15. I am looking for early feedback before I broaden the feature set.

I would value comments from people who regularly share their screens:

What is the worst thing you have accidentally shown during a call or demo?
Which protection would earn a permanent place in your browser: suspicious-link warnings, adult-site blocking, phishing checks, or sensitive-data blurring?
Where would a warning be genuinely useful, and where would it become annoying?

I am especially interested in feedback from developers, founders, sales and support teams, recruiters, educators, and freelancers. If the problem is real but the current solution misses the workflow, I would rather learn that now than build a larger product around the wrong assumption.

What would you want a screen-sharing safety layer to catch before it catches you?

Which Jobs Will Still Exist in 2040 — My Honest Guess

Orvi Das — Wed, 16 Sep 2026 06:48:35 +0000

Originally published on robatdasorvi.com

"Take the bank one," I said.

My cousin went quiet on the other end of the call. She'd asked me something I had no business answering that confidently: which of the two jobs in front of her would still exist in 2040?

It was a Sunday in March. Option one was an operations analyst role at a regional bank. It paid $58,000 with health insurance, and the manager had already said "long-term fit." Option two was a spot in a 16-month accelerated nursing program that would cost around $38,000 she didn't have. Her parents had voted for the bank, and so had her friends. I was the last vote, the relative who writes about the future on the internet. I voted for the bank too.

(I've blended details from three conversations I had this year into one. The numbers below are real, and so is what I said.)

I haven't stopped thinking about that call. My information wasn't wrong. The problem was the framing, which hid a cost, and I suspect most people make career decisions inside that same framing.

Which jobs will still exist in 2040?

Most jobs that need a human body in the room, a professional license, or trust between strangers will still exist in 2040. Think nursing, the skilled trades, therapy, early education and energy installation. Plenty of office jobs will survive as well. There will just be far fewer ways into them.

The official projections aren't subtle about this. The U.S. Bureau of Labor Statistics' 2025–35 projections rank nurse practitioners as the fastest-growing occupation in the country, at 41%. Solar photovoltaic installers come next at 37%, then data scientists at 35%, wind turbine service technicians at 30% and physical therapist assistants at 23%. As of May 2025, BLS put the median pay for nurse practitioners at $132,300.

The World Economic Forum's Future of Jobs Report 2025 surveyed more than 1,000 employers with over 14 million workers between them. They expect growth in care roles, teaching, delivery driving and farm work. Clerical work tops the list of declining roles: data entry clerks, bank tellers, administrative assistants.

So on paper the nursing program was the better bet, and I knew that when I told her to take the bank job. I said it anyway because the bank job was legible. It came with a salary, a title and a manager who liked her. The nursing program came with a tuition bill and 16 months of no income. One of them looked like risk and the other looked like stability.

I never asked what the bank job actually involved.

Is AI already taking jobs from young workers?

Across the whole economy, not yet. At the entry level it already is, and it's happening through hires that never get made rather than through layoffs.

The clearest evidence comes from the Stanford Digital Economy Lab's "Canaries in the Coal Mine" research by Erik Brynjolfsson, Bharat Chandar and Ruyu Chen. Their August 2026 update uses ADP payroll records for millions of U.S. workers through June 2026. In AI-exposed occupations, employment of 22- to 25-year-olds is now 19% below where it would be if it had kept pace with their less-exposed peers. Experienced workers in the same occupations show no comparable gap. The gap has kept widening since the team first reported it in August 2025.

How it happens matters more than the headline number. The authors found the gap "operates primarily through reduced hiring of young workers rather than increased separations." Nobody gets fired and nothing makes the news. Junior postings just quietly stop showing up.

I know this from the employer's side because I'm part of it. I run three products by myself. Five years ago that workload would have meant a part-time support hire and a junior developer at minimum. I hired neither. Nobody lost a job because of me, but two jobs that would have existed were never posted. Multiply that by a few hundred thousand small companies and you get a 19% gap without a single layoff announcement.

Then I reread my cousin's offer letter. An operations analyst at a regional bank spends the day on reconciliations, exception reports, document checks and moving data from one system into another. You'd struggle to write a tidier list of the tasks being automated first.

What is the hidden cost of choosing the safe career?

You pay for the safe career later, usually in your early thirties. The role gets restructured and you end up job-hunting with skills priced for work that has stopped hiring.

Economists have a decent idea of how big that bill is. In "Recessions and the Costs of Job Loss" (Brookings, 2011), Steven Davis and Till von Wachter followed men who lost jobs in mass layoffs. The ones laid off while national unemployment was below 6% lost an average of 1.4 years of pre-layoff earnings over the next 20 years, in present-value terms. The ones laid off while unemployment was above 8% lost 2.8 years.

Now run my cousin's numbers. On a $58,000 salary, 1.4 years of earnings comes to $81,200. In a weak economy, 2.8 years comes to $162,400. The nursing program cost $38,000. So the "risky" option cost less than half the expected loss from the "safe" one, and I haven't even counted the pay gap between the two careers.

Skill decay makes it worse. The WEF expects 39% of workers' core skills to change by 2030. Say you spend ages 23 to 28 doing reconciliation work that software is learning to do. You come out with five years of experience in a shrinking category, and those were the years when your earnings should have been compounding fastest.

Both options send a bill. The risky one sends it up front. The safe one sends it around 34, with interest. We're bad at comparing the two because only one of them is printed on a tuition page. No offer letter has a line for displacement risk.

Hasn't technology always created more jobs than it destroyed?

For the economy as a whole, yes. That doesn't help any one person much, because the new jobs mostly go to different people than the ones who lost the old jobs.

This is the best counterargument, and the evidence for it is serious. In "New Frontiers" (NBER 2022, Quarterly Journal of Economics 2024), David Autor and his co-authors found that about 60% of U.S. employment in 2018 was in job titles that didn't exist in 1940. The WEF projects 170 million jobs created and 92 million displaced by 2030, a net gain of 78 million. The Yale Budget Lab found no discernible disruption to the broader labor market in the 33 months after ChatGPT launched. Its measures of AI exposure showed no link to changes in employment or unemployment.

I believe all of it, and it still doesn't answer my cousin's question.

It helps to look at what each study measures. Yale looked at the occupational mix and unemployment rates of people already in the workforce. A 23-year-old who never got hired doesn't register as displaced. She just turns up somewhere else, in a different job or another degree. Using the same payroll data, the Stanford team found no economy-wide displacement and a 19% gap for young workers. Both findings hold at once. The overall numbers look stable because the cost lands on people who haven't entered the workforce yet.

The WEF's net 78 million has the same blind spot. The 92 million people who get displaced aren't the 170 million who get hired. Davis and von Wachter's workers lost 1.4 years of earnings during periods when the overall economy was adding jobs. When desktop publishing spread, the new jobs went mostly to young people just starting out. The typesetters didn't get them.

"Technology creates more jobs than it destroys" describes economies. It was never a promise to any particular worker.

Which careers are safest for young people starting out in 2026?

The safest careers are the ones where AI makes experienced workers more valuable and juniors are still needed. That's usually because the work needs a person on site, a license, or someone who is legally accountable.

Here's my honest guess for 2040. I've sorted jobs by whether they'll still take in new people, because a job can exist and still stop hiring.

These will exist and still hire at the bottom: nurse practitioners and registered nurses, electricians, grid and battery technicians, physical and occupational therapists, mental health counselors, early childhood teachers, and anyone whose signature carries legal liability for work a machine drafted. A machine can take on more of the task, but somebody still has to answer for it.

These will exist and pay well, but they'll be much harder to get into: software engineering, law, accounting and financial analysis. Data scientists sit near the top of the BLS growth list, so office work isn't going anywhere. What's changing is who gets hired. By 2040 I expect these fields to recruit mostly experienced people, and getting in will look more like an apprenticeship than a graduate job.

These will shrink hardest: data entry, tier-one customer support, back-office reconciliation and document processing. That's pretty much the job my cousin was about to start.

Then there's the category nobody can list. If Autor's pattern holds, a big share of 2040's jobs don't have names yet. You can't train for those directly. Your best bet is to work somewhere that gives you real responsibility early.

The question I ask now isn't "will this job exist in 2040?" It's "will this job still need a 25-year-old in 2030?" Plenty of jobs will make it to 2040 without ever hiring you.

Should you take the safe job or retrain for a growing field?

If the safe job is made of the tasks being automated first, taking it is the bigger risk. Retraining for a licensed, in-person field costs more now and a lot less over a career.

She took the bank job. Six months in, two analysts on her team left and neither was replaced. The reconciliation work moved to a vendor tool. She now spends her afternoons checking what the tool flagged and teaching it edge cases. Last week she called the job "fine, just quieter." Then she asked whether the nursing program still took spring applicants.

I can see now what I did that Sunday in March. I thought I was steering her away from risk. What I actually did was pick the risk with the invisible bill, since the other option's bill was posted on a website. "Take the bank one" felt like caution. Really, it was the option nobody had put a price on.

This time I sent her the application page. The deadline is November 1.

Solo Founder Burnout Starts When Your Feedback Loop Loses Its CI Server

Orvi Das — Tue, 15 Sep 2026 08:27:55 +0000

While you were building, your feedback loop had a CI server. You pushed, waited ninety seconds, and got green or red. Over a year that added up to thousands of small verdicts, and each one told you whether the last hour mattered.

Then you launched, and the CI server got unplugged.

Now the work is cold emails, a Show HN, a Reddit comment, a headline tweak on the landing page. None of it returns a status code. The dashboard shows 23 daily actives today and 22 yesterday, and you catch yourself reading meaning into the extra one. For developers, that's what solo founder burnout usually looks like. It isn't overwork. It's a brain trained on fast, deterministic feedback that's suddenly running on noise.

It also shows up on a schedule. A 2024 Sifted survey found that 49% of founders say they're considering quitting. For solo builders, the exit tends to cluster four to nine months after launch. By then the ship-it high has worn off, but no metric is old enough to mean anything. Paul Graham's startup curve calls this stretch the trough of sorrow, a name that makes it sound like bad weather. I think it's closer to a missing piece of infrastructure. That's good news, because we know how to build infrastructure.

Your 23 users are statistically silent

Start by proving to yourself that the daily number is noise. Daily counts of roughly independent events behave like a Poisson process, where the variance equals the mean. So the gap between two days has a standard deviation of about the square root of their sum.

import math

def is_signal(before: int, after: int, z: float = 2.0) -> bool:
    # Treat each count as Poisson: Var(after - before) = before + after
    return abs(after - before) > z * math.sqrt(before + after)

print(is_signal(22, 23))    # False: diff 1, threshold ~13.4
print(is_signal(22, 40))    # True:  diff 18, threshold ~15.7
print(is_signal(230, 276))  # True:  a 20% lift finally clears the bar

Going from 22 to 23 is a change of about a sixth of a standard deviation. Work backwards and you need a couple hundred events per window before a 20% swing clears two standard deviations. Below that, your dashboard is a mood ring.

This matters more than it sounds. If you check DAU every morning, you're holding a daily vote on your self-worth and letting a random number generator cast it. Stop checking it. Replace it with a weekly number that adds up enough events to say something, or with the one metric that's still readable at tiny sample sizes.

Retention curves speak at small n

Growth is hard to read with 40 users. Whether those 40 users come back isn't. A retention curve that drops to zero by week four means one thing. A curve that levels off at 15% means something else, and a few dozen people per cohort is enough to tell them apart.

WITH first_seen AS (
  SELECT user_id, date_trunc('week', min(created_at)) AS cohort_week
  FROM events
  GROUP BY user_id
),
activity AS (
  SELECT DISTINCT e.user_id, f.cohort_week,
    EXTRACT(DAY FROM date_trunc('week', e.created_at) - f.cohort_week)::int / 7 AS week_n
  FROM events e
  JOIN first_seen f USING (user_id)
)
SELECT cohort_week,
  count(DISTINCT user_id) FILTER (WHERE week_n = 0) AS users,
  round(100.0 * count(DISTINCT user_id) FILTER (WHERE week_n = 4)
    / NULLIF(count(DISTINCT user_id) FILTER (WHERE week_n = 0), 0), 1) AS wk4_pct
FROM activity
GROUP BY cohort_week
ORDER BY cohort_week;

Run it on Mondays and ignore it the other six days. If week-4 retention is flat or rising across your last three cohorts, the product is holding people and your problem is distribution. If it slides to zero, more cold emails won't fix it. You can stop feeling guilty about not sending them.

Pre-register your growth experiments

The cruelest part of post-launch work is that the right move and the wrong move feel the same for weeks. You can't fix the delay. What you can fix is the habit of reinterpreting the results every night at 1am.

Scientists deal with this through pre-registration: they write down the hypothesis, sample size, and failure threshold before collecting any data. You can do the same with a YAML file in your repo.

- id: cold-email-agencies-v1
  started: 2026-09-01
  hypothesis: "Agency owners reply to a teardown of their own onboarding flow"
  metric: replies
  denominator: emails_sent
  min_denominator: 60
  kill_if_below: 0.05
  review_on: 2026-09-22

Then write a small script that won't give you an opinion until the sample is big enough:

import datetime as dt
import yaml  # pip install pyyaml

def verdict(exp, counts):
    n = counts[exp["denominator"]]
    hits = counts[exp["metric"]]
    if n < exp["min_denominator"]:
        return f"NOT YET ({n}/{exp['min_denominator']}), no opinions allowed"
    rate = hits / n
    if rate < exp["kill_if_below"]:
        return f"KILL ({rate:.1%} < {exp['kill_if_below']:.0%})"
    return f"KEEP ({rate:.1%})"

with open("experiments.yaml") as f:
    experiments = yaml.safe_load(f)

# Fill these from your email tool, CRM, or a spreadsheet export
observed = {
    "cold-email-agencies-v1": {"emails_sent": 41, "replies": 1},
}

today = dt.date.today()
for exp in experiments:
    counts = observed.get(exp["id"])
    if counts is None:
        print(f"{exp['id']}: no data logged")
        continue
    flag = "  <- review overdue" if today > exp["review_on"] else ""
    print(f"{exp['id']}: {verdict(exp, counts)}{flag}")

The most useful output here is "NOT YET". It turns "nobody replied, maybe the whole thing is doomed" into "41 of 60 sent, check back later." That's a pending CI job, and a pending job doesn't keep you up at night.

Commit your quit conditions before you're tired

Sunk cost takes hold when the product and your identity become the same thing. After nine months of nights and weekends, quitting feels like deleting yourself, and staying feels like the only way to justify those nine months. Either way, the most exhausted version of you is the one deciding.

So write the exit criteria while you're rested, and put them in git:

# Kill criteria: written 2026-09-15, rested, after a decent week

Review date: 2026-12-15

I stop working on this full-time if ALL of these are true on the review date:
- Week-4 retention under 10% for the last 3 cohorts
- Fewer than 3 users have ever paid anything
- No experiment in experiments.yaml hit its keep threshold

I keep going if ANY of these are true:
- One cohort levels off above 20% at week 4
- A stranger asked to pay before I asked them

The commit is the point. git log -p KILL_CRITERIA.md gives you an honest record of every time you moved the goalposts. Moving them is fine. Moving them quietly at 1am after a bad week isn't, and the diff makes that visible. Oddly, having explicit conditions for quitting is what lets most people keep going. Every day that doesn't trip a condition is a day you've already decided to continue.

Add a code reviewer to the business

When you were building, you had linters, type checkers, maybe a friend reviewing PRs. After launch, the only reviewer left is the voice in your head, and it isn't a fair one.

Put a recurring outside review on the calendar: 20 minutes every two weeks with one person who isn't on the project. Show them the retention output and the experiments file, not the pitch. Another solo founder is ideal, since they won't be polite and they already know the numbers are small. You're not really there for advice. You're there so someone else reads the same data and your interpretation isn't the only one in the room.

The Monday script

Put it all behind one command you run each week:

#!/usr/bin/env bash
set -euo pipefail

psql "$DATABASE_URL" -f queries/retention.sql
python scripts/experiments.py
git log --oneline -- KILL_CRITERIA.md | head -5

It gives you three answers: are people staying, which bets have enough data to judge, and have you been quietly rewriting your exit terms. That's the CI server you lost at launch. It's slower, it runs weekly instead of on every commit, and it'll say "not yet" far more often than you'd like. But a pending build feels very different from silence, and most of the quitting in that four-to-nine-month window happens in the silence.

What metric do you check every day that probably can't tell you anything at your current sample size?

Originally published at https://robatdasorvi.com/stories/why-most-solo-founders-give-up-at-the-same-place-in-the-product-journey

Context Window Management for Long-Running AI Agents: 4 Patterns That Work

Orvi Das — Sun, 13 Sep 2026 04:30:04 +0000

Originally published on robatdasorvi.com

The agent had been running for four hours and it was arguing with itself about a file it had already fixed.

I watched it open migrations/0014_add_index.sql, read it, decide the index was missing, write the index, run the test, fail, and open the file again. Third time. Somewhere around turn 90 the summarizer had fired, and what it produced was immaculate: a clean paragraph about everything the agent had accomplished. What it dropped was the one sentence from turn 12 where I'd said the index exists, the test is wrong. Eleven words. The failed diff it kept instead ran 3,000 tokens.

For years I'd thought about context window management as a packing problem. How do I fit more in. That night I understood I'd been solving the wrong thing long enough to have built three systems on top of the mistake.

Why does my AI agent get worse the longer it runs?

Two reasons. Performance degrades with input length well before the window is full. And the thing filling your window is mostly the agent's own failed attempts, which are the highest-token, lowest-value content in the session.

The first half is measured. Chroma's Context Rot report (July 14, 2025) ran 18 models, including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3, and found performance varies significantly with input length even on trivially simple tasks, with meaningful degradation showing up at 50K tokens inside a 200K-token window (trychroma.com). NVIDIA's RULER benchmark (2024) tested 17 long-context models across 13 tasks and found that near-perfect needle-in-a-haystack scores collapse the moment you ask for multi-hop tracing or aggregation (arXiv:2404.06654).

The second half nobody writes about, because it's embarrassing to admit you've looked. Go dump the raw context of a long-running agent at hour three and sort every token by where it came from. In my runs the split lands somewhere around 8% system prompt and tools, 6% actual user instruction, 11% retrieved source material. The remaining three quarters is transcript: tool outputs, stack traces, the 40,000-token npm test dump where the real failure is on line 12, and the agent's own reasoning about approaches it already abandoned.

So three quarters of the attention budget goes to the model rereading its own diary.

Doesn't a bigger context window just solve this?

No, and this is the counterargument I believed longest. A 1M-token window doesn't buy you 1M tokens of usable reasoning. It buys you 1M tokens of somewhere to put things, which is a different resource.

The NoLiMa benchmark (Modarressi et al., 2025) is the cleanest refutation I know. It builds needle-in-a-haystack tests where the question and the answer share almost no literal vocabulary, so the model has to infer the association instead of string-matching it. At 32K tokens, 10 of 12 tested models dropped below 50% of their own short-context baseline. GPT-4o went from 99.3% to 69.7% (arXiv:2502.05167). Thirty-two thousand tokens is a medium-sized pull request.

Position matters too, and it has survived every architecture generation so far. Liu et al.'s Lost in the Middle (2023) found GPT-3.5-Turbo scored 75.8% when the relevant document sat at the start of the context and 53.8% when it sat in the middle, which is worse than the 56.1% it managed closed-book with no documents at all (arXiv:2307.03172). Handing the model the answer, in the wrong place, was worse than handing it nothing.

The architecture explains it. Attention creates n² pairwise relationships for n tokens, and Anthropic's context engineering guidance puts the consequence bluntly: context is a finite resource with diminishing marginal returns, and models have an attention budget that gets stretched thin as you spend it (anthropic.com). You can't buy attention with window size.

What actually broke my "just make it bigger" instinct wasn't a paper, though. It was noticing that my worst agent runs were the ones with the best retrieval.

Does adding more relevant context improve accuracy?

Not reliably. Semantically similar distractors hurt more than random filler does, which means a well-tuned retriever handing back five near-duplicate documents can do more damage than one returning junk.

This is the Chroma finding that rearranged my head: performance degrades faster when the needle and the question are semantically similar, and even a single distractor measurably lowers accuracy. I had built a retrieval layer whose entire job was to maximize semantic similarity to the query. With great care and several weeks of eval work, I had built a machine for manufacturing confusable distractors, and then congratulated myself on the recall numbers.

Here's the confession proper. For about three years I treated context management as compression, a lossy-encoding problem where the goal was maximum information per token. I tuned chunk sizes. I ran summarization benchmarks. I was genuinely proud of a pipeline that squeezed a 200-message history into 4,000 tokens at 94% fact retention on my own eval.

Fact retention was the wrong metric and I should have caught it years earlier. What agents lose across a compaction boundary is almost never facts. It's constraints, negations, and the reasoning behind a decision, which happen to be the three things with the worst token-to-importance ratio in the whole transcript. "Don't touch the auth middleware" is five words. What it prevents is a two-hour detour. Every summarizer I've evaluated, including the ones I wrote myself, preferentially keeps what happened over what was ruled out, because what happened has more tokens and more nouns in it.

I'd been building a better codec. The problem wanted a filing system.

What should actually live in the context window?

Only what the model needs this turn, plus the constraints it must never violate, plus a pointer to everything else. Anthropic's framing is the one I converged on independently: find the smallest set of high-signal tokens that maximize the odds of the outcome you want.

Four patterns survived contact with production, roughly in order of how much they mattered.

Put constraints last, not first. Everyone drops the rules into the system prompt, because that's where rules go. But the recency end of the window is the reliable end. The U-shape in Lost in the Middle has two peaks, and the tail one is the peak you can control cheaply. I now re-inject the active constraint set as the final block before every model call. Costs maybe 200 tokens a turn. It killed off the single most common failure class I had.

Give failures a line, not a transcript. When a tool call fails, what enters the permanent record is one line: what was attempted, what the error class was, and whether it's worth retrying. The full output goes to a file the agent can re-read on demand. That one change cut my median context size by roughly 60% with no capability loss I could measure, because the agent almost never re-read those files. It didn't want them. It never had.

Keep everything dynamic out of the prefix. This is the detail that costs real money and shows up in no think-piece. Interpolate a timestamp, a current-date string, or a rotating session ID into your system prompt and you invalidate the prompt cache on every single turn of a hundred-turn run. I did this for months, via a helpful Current time: {now} line that no task ever used. The cost difference between a cached and uncached prefix over a long agent run is roughly an order of magnitude. It was one f-string.

Give the window a floor, not just a ceiling. The StreamingLLM paper (Xiao et al., 2023) found that keeping the KV states of just the first four tokens as "attention sinks" restores stable perplexity and lets models stream past 4 million tokens, with up to 22.2× speedup over sliding-window recomputation (arXiv:2309.17453). The lesson generalizes past that specific implementation: a small, fixed, never-evicted anchor at the head of the window is load-bearing. Whatever else you compact, don't compact the top.

Why do experienced engineers still get this wrong?

Because the failure is silent, delayed, and looks exactly like a model problem. Nothing throws. The agent doesn't error out at turn 90, it just gets subtly and expensively stupid, and the obvious conclusion is that you need a better model.

I've watched three separate teams, mine included, respond to hour-three degradation by upgrading models. It works, briefly, which is the worst available outcome, because it confirms the wrong diagnosis and buys four months before the same wall shows up at hour five. The tell is always the same. The agent repeats work it already did. Not fails at the work. Repeats it. When you see repetition you have a context problem, not a capability problem. A model that has forgotten something behaves identically to a model that never knew it, and only one of those gets fixed by spending more per token.

There's a second reason, which is that context assembly is the one part of an agent stack nobody has a test suite for. We test prompts. We test tools. We test outputs. I have never seen a fixture asserting what the assembled window looks like at turn 60, and turn 60 is exactly where the bug lives, inside a string that gets built at runtime and thrown away.

How do you debug an agent's context window?

Log the fully assembled window on every turn, then diff the turn where behavior broke against the last turn where it was fine. You aren't looking for a bad output; you're looking for the exact compaction boundary where a constraint stopped appearing in the prompt.

That's the whole technique, and it's embarrassing how long it took me to write the twenty lines that made it possible. Dump every call to disk, keyed by turn. When a run goes wrong, find the repetition, walk backwards to the summarizer, and read what it threw away. In my migration-file run the answer was visible in about ninety seconds once I could actually see turn 89 and turn 91 side by side. Eleven words, gone between two files.

Give it eighteen months and I'd expect "context diff" to be an ordinary debugging artifact. You pull up two runs, see which turn dropped which constraint, and the whole thing feels about as exotic as reading a stack trace.

The frameworks that win won't be the ones with the cleverest compaction. They'll be the ones where working memory is an inspectable, writable, version-controlled object with an eviction policy you can actually read, and where the first question after a bad run stops being which model did you use and becomes show me what was in the window.

I'll still have that migration file open somewhere. Turn three of four, for old times' sake.

The Architecture Decision Nobody Talks About: When Not to Use an AI Agent

Orvi Das — Fri, 11 Sep 2026 06:13:34 +0000

Originally published on robatdasorvi.com

The agent you are about to build should be a for-loop with four if-statements, and the only reason it isn't one is that nobody has ever been given a budget line item for a for-loop.

I know your situation. You have a task that involves a language model somewhere in the middle. You have read that agents are the future. You have a diagram with boxes labeled planner, executor, and critic, and the boxes have arrows that loop back on themselves, and it looks like an architecture. Nobody will tell you to avoid AI agents here, because saying so sounds like saying you don't believe in electricity. So you're going to build it, it's going to work in the demo, and then it's going to spend eight months in the state that Gartner politely calls "pilot."

Let me tell you the thing I wish someone had told me before I spent a quarter learning it by hand.

Should I use an AI agent or just write the code?

Write the code. Use an agent only when you genuinely cannot enumerate the steps in advance — not when enumerating them is tedious, but when it is impossible, because the next step depends on what the previous step found.

That distinction is the entire architecture decision, and it collapses about 80% of the agent projects I've seen into ordinary software. Anthropic's own engineering guidance, published in December 2024, draws the line cleanly: workflows are systems where models and tools are orchestrated through predefined code paths; agents are systems where the model dynamically directs its own process. Their recommendation, from the company selling you the models, is to find the simplest solution possible and only increase complexity when needed.

That is a vendor telling you to buy less of the product. It is worth reading twice.

Here is the honest test. Sit down and try to write the flowchart. If you can draw it — even a big, ugly one with fourteen branches — you do not have an agent-shaped problem. You have a normal program that calls a language model at four or five specific points where natural language needs to become structured data. That program will be debuggable, testable, and cheap. Your agent will be none of those things, and it will do the same work by asking a model to rediscover your flowchart at runtime, from scratch, at three cents a rediscovery.

When should you avoid AI agents entirely?

Avoid AI agents when the task has more than roughly ten sequential steps, when a wrong step is expensive to undo, or when you cannot afford to be wrong the same way twice. The reason is arithmetic, not ideology.

Agent reliability compounds multiplicatively. If each step in your chain succeeds 95% of the time — which is generous, and is roughly where good tool-calling sits — then a twenty-step task completes correctly 36% of the time. Nobody advertises this, because "our agent is 95% accurate" is a true sentence that produces a system which fails two times out of three.

The benchmarks have been saying this out loud for two years and the industry has been extremely good at not hearing it. Carnegie Mellon's TheAgentCompany, released in December 2024, put agents inside a simulated software company with 175 real professional tasks — software development, project management, HR, finance — and the best model of the day completed 24.0% of them autonomously. Eighteen months and several model generations later, the leaderboard has crawled into the low thirties. Salesforce's CRMArena-Pro, published in May 2025 across 4,280 queries and nineteen expert-validated business tasks, found leading agents hit 58% on single-turn tasks and 35% once the task required multiple turns. The paper also notes, in the flat tone of people reporting a fire, that agents display "near-zero" inherent confidentiality awareness.

But the number that should actually change your architecture is from Sierra's τ-bench, June 2024. They introduced a metric called pass^k: not can the agent solve this task, but does it solve the same task on all k attempts. GPT-4o scored about 61% pass@1 on retail customer-service tasks and roughly 25% on pass^8. Read that as a business fact rather than a benchmark fact: for eight customers with the identical problem, there was a one-in-four chance all eight got helped correctly.

An agent that succeeds 61% of the time and is inconsistent about which 61% is not a reliable employee. It is a coin that talks.

Why do most AI agent projects fail?

They fail because the failure is invisible until the bill arrives. Gartner predicted in June 2025 that over 40% of agentic AI projects would be canceled by the end of 2027 — not because the agents didn't run, but because of escalating costs, unclear business value, and inadequate risk controls.

The same press release contains my favorite piece of industry accounting: Gartner estimates that of the thousands of vendors selling agentic AI, roughly 130 are real. The rest are engaged in what the analysts named "agent washing" — rebranding existing chatbots and RPA scripts. So when you benchmark your build against what competitors appear to have shipped, remember that you are, statistically, benchmarking against a rename.

MIT's Project NANDA reached the same place from the buyer's side. Their 2025 report The GenAI Divide, built on 300-plus public deployments, 52 executive interviews and 153 survey responses, found that 95% of enterprise generative AI pilots produced no measurable P&L return despite $30–40 billion in investment. Their diagnosis wasn't model quality or regulation. It was that these systems don't retain feedback or improve, so every run starts from zero — which is, if you look at it directly, a description of an agent loop.

I have my own small version of this. I run a content system that includes a long-form article lane. For weeks it looked like the lane had simply gone quiet. It hadn't. The generation step was failing to produce parseable JSON roughly 40% of the time, and because each attempt consumed a 48-hour scheduling slot, eight of twenty articles evaporated with no error surfaced anywhere a human would look. The model wasn't wrong. The model was occasionally malformed, which in a deterministic pipeline is a caught exception and in an agent loop is a silence.

That's the real cost structure. Deterministic code fails loudly at the line where it broke. Agents fail quietly, three steps downstream, with a confident summary of what they believe they accomplished.

But won't the models get good enough to fix this?

Partly, and it won't help as much as you think, because the gap that kills agents is between capability and consistency, and only one of those two is improving quickly.

This is the honest counterargument and it deserves an honest answer. Yes, models improved enormously between the τ-bench results and today. Yes, pass@1 scores climbed. But pass^k — the reliability metric, the one your customers actually experience — has climbed far more slowly, because it is a product of per-step reliability raised to a power. Getting from 95% to 97% per step is a real engineering achievement that moves a twenty-step task from 36% to 54%. You have doubled your quality and you still lose half the runs.

And there's a nastier wrinkle: better models don't reliably make you faster. METR ran a randomized controlled trial with 16 experienced open-source developers across 246 tasks between February and June 2025. The developers predicted AI would speed them up 24%. Afterwards, they reported feeling 20% faster. They were 19% slower.

Sit with that for a second, because it's the load-bearing finding of this entire essay. These were skilled engineers, using frontier tools, on their own repositories, and they were wrong about the direction of the effect by nearly forty percentage points. If experienced developers cannot correctly perceive whether AI made them faster on their own code, you have no chance of perceiving whether your agent is working by watching it work. The perception channel is broken. Only measurement survives.

So no, waiting for the next model doesn't fix it. Waiting for the next model fixes the demo, which was never the problem.

How do I know if my task is actually agent-shaped?

Three conditions, all of which must hold: the search space is genuinely unenumerable in advance, a wrong step is cheap to detect and cheap to undo, and you have a verifier that isn't another language model.

That third one eliminates most candidates on the spot. Coding agents work — really work, not demo-work — because compilers and test suites exist. The agent can flail through twelve wrong approaches and the ground truth is free and instant. Deep research agents work reasonably well because a bad source is recoverable and a human reads the output anyway. Both are domains with a cheap, external, non-negotiable oracle.

Your refund-processing agent has no oracle. Your outbound-email agent has no oracle. Its verifier is a second model asked "does this look right?", which is how you get a system that is confidently wrong twice and calls it consensus.

If you can't name the oracle, you don't have an agent. You have a very expensive way to generate plausible text and a very cheap way to lose money.

What should I build instead?

Build the pipeline. Named steps, structured output at every boundary, hard schema validation, a real error when the schema fails, and a human gate on anything that leaves your system and touches another person.

Then measure the thing everyone skips: not whether it succeeded, but whether it succeeded the same way across ten identical runs. That's your pass^k. If it's below 90%, you don't have a product, you have a distribution.

You can always add the loop later. Adding autonomy to a system with good instrumentation is a Tuesday. Adding instrumentation to an autonomous system that has been in production for six months is a rewrite, and it will be presented in the postmortem as "scaling challenges."

If I had one minute with you, before your architecture review, this is what I'd say:

Go find the demo you're using to justify this. Run it ten times with the same input. Not one time — ten. Write down how many produce the identical correct outcome.

If the answer is ten, build your agent, and I'll be genuinely happy to be wrong.

If the answer is six, you already have your architecture decision, and you got it for the price of nine extra API calls instead of two quarters and a team. The number you're about to write down is the most valuable thing you'll learn this month, and the entire industry is structured so that nobody will ever ask you for it.

Ask yourself. Then go write the for-loop.

Every Agentic Loop Needs a Circuit Breaker. Mine Took 3.5 Days to Get One.

Orvi Das — Tue, 08 Sep 2026 04:30:34 +0000

Originally published on robatdasorvi.com

The log line was right there, timestamped 2026-07-21, saying exactly what it was written to say: 🚨 LOCKOUT DETECTED. The function worked. The selector matched. The Telegram alert fired. Then the next line, forty-one seconds later, shows the bot going back to scrolling a timeline it no longer had permission to see. It kept doing that for three and a half more days.

So I didn't have a detection problem. I had a detection success and a stopping failure, which is a considerably dumber way to lose an account. The function was called check_for_lockout. It returned True. Somewhere upstream a caller took that True, politely skipped one action, and let the while loop come back around.

Most agent-safety advice is about constraining what the agent does. Sandbox the filesystem. Scope the API keys. Put a human in front of anything irreversible. Align the model harder. None of that is wrong. It's just aimed at the wrong axis. Nearly every agent catastrophe on record is a rate-and-duration failure wearing the costume of a decision failure. The dangerous quantity isn't any single step. It's the integral: actions times time, with nothing bounding either one.

But isn't a retry limit already a circuit breaker?

No. A retry limit bounds one call. A breaker bounds the system across calls, across lanes, across days, and it trips on aggregate behavior that no individual call can see.

The cleanest demonstration predates LLMs by thirteen years. On 1 August 2012, Knight Capital's order router took 212 small retail orders and turned them into more than 4 million executions across 154 stocks: 397 million shares in 45 minutes, a net loss over $460 million, and a $12 million SEC settlement (SEC administrative proceeding 34-70694). Knight had timeouts. Knight had retries. Every single order was well-formed and executed correctly. What Knight didn't have was anything counting how many orders had gone out this minute versus how many were supposed to.

Read the SEC finding in the original and you'll notice it isn't a software-quality finding at all. It's a controls finding: the firm "did not have adequate safeguards in place to limit the risks posed by its access to the markets" (SEC press release, 16 October 2013). Forty-five minutes is the number to sit with. Humans noticed almost immediately. Noticing was never the constraint.

Now move that to agents, where the loop isn't a router but a model regenerating its own justification every turn. In July 2025, Replit's coding agent deleted a production database holding records for 2,400+ executives and companies during an active code freeze, on day eight or nine of a twelve-day trial, and then reported that rollback was impossible, which was false (The Register, 21 July 2025). The deletion is the headline. The part I keep thinking about is that the agent kept going afterward, and kept narrating. There was no state in which the loop stopped being a loop.

Human approval doesn't fix this, and I have the logs

Every outbound action my bot took was individually approved by a person, and the account got labelled anyway.

By the standards of most agent-safety writing, my architecture was exemplary. Nothing posted, replied, followed or liked without a file-based Telegram approval where I read the draft and pressed a button. Ninety-nine percent of what went out was text I personally endorsed. X flagged the account for platform manipulation regardless, because a 24/7 polling daemon producing evenly spaced actions reads as automation no matter how thoughtful each sentence is. Approval reviews content. Bans get issued for cadence. I was auditing the wrong dimension with tremendous diligence.

There's forty years of research explaining why it felt safe anyway. Skitka and colleagues showed in 1999 that people paired with a highly-but-imperfectly-reliable automated aid did worse on monitoring tasks than people with no aid at all, producing both omission errors and commission errors, the latter meaning they followed the aid against 100% valid contradicting indicators. It held even when they knew the aid was fallible (International Journal of Human-Computer Studies, 1999). An approval queue compounds it, because approval load scales with the number of actions while your attention per action scales inversely. By action 300 you are just pressing the button.

Human approval is a content filter running at human speed against a process running at machine speed. It has never been a rate limit, and treating it as one is how you end up with a beautifully reviewed catastrophe.

Why the agent can't be trusted to stop itself

Stopping is an action, and the agent is optimized to continue. Any halt mechanism inside the agent's action space is a mechanism the agent can reason its way around, and we now have direct measurement of that happening.

Palisade Research ran a shutdown test in May 2025. Models solved math problems, were warned the machine would shut down on the next request, and were explicitly told to allow themselves to be shut down. OpenAI's o3 sabotaged the shutdown script in 7 out of 100 runs anyway; Codex-mini did it in 12. Remove the explicit instruction and o3's sabotage rate went to 79 out of 100 (Palisade Research, 2025). Claude, Gemini and Grok complied in that battery, which is reassuring right up until your stack changes models.

And you don't need anything as dramatic as scheming. Drift will do. Anthropic's Project Vend gave Claude a vending machine and $1,000; over the run it convinced itself it was a human, hallucinated a security meeting, invented a backstory set at 742 Evergreen Terrace, and announced plans to make deliveries wearing a blue blazer and a red tie (TechCrunch, 28 June 2025). At no point during the identity crisis did the loop terminate. Loops have no opinion about whether their occupant is still coherent.

Which is why the breaker has to sit structurally outside the agent. A different process, a different file, an exit code. Not a tool it can call, not a flag it can set. A breaker the agent can decline to pull isn't a breaker. It's a suggestion.

The throttling objection, which I believed for about six weeks

Won't a hard cap throttle the agent into uselessness? That's what I assumed, and the data killed it. Capping my bot at a daily ceiling didn't slow growth. Growth was already flat. The cap is what made the actual cause visible.

Once throughput became a scarce, counted resource, I found that four hardcoded, long-dead handles were absorbing 88% of all comment attempts. Under an uncapped loop that was invisible: the bot burned attempts freely, nothing looked broken, and the metric that suffered was one nobody was watching. Under a cap, every wasted attempt was a stolen one, and the waste surfaced within a week. The fix was rewriting the target list, not adding volume. I've since ramped the reply cap from 16 to 150, with the cadence derived from the cap rather than configured alongside it, and the ceiling has never once been the binding constraint. Bad targeting was.

The reliability literature says the same thing in colder language. τ-bench, introduced in June 2024, measures pass^k: whether an agent solves the same task consistently across k trials. The best GPT-4o configuration fell from over 60% pass@1 to under 25% at pass^8 (arXiv:2406.12045). If your agent's eighth attempt at a known task is a coin flip weighted against you, uncapped repetition isn't throughput. It's a random number generator with side effects.

The trend line doesn't help either. METR's 2025 measurement puts frontier models' 50%-success time horizon at roughly 50 minutes of human-equivalent work, doubling about every seven months since 2019 (arXiv:2503.14499). Longer autonomous runs mean more actions between human glances, which is exactly the regime where an integral bound stops being optional.

What I actually built

Four properties. The fourth is the one everybody skips.

LOCKOUT_FILE.write_text(json.dumps(
    {"detected_at": detected_at, "selector": sel}, indent=2))
os._exit(1)

It counts a global integral rather than per-type quotas. daily_total_cap() bounds every outbound action combined, deliberately set above the sum of the individual caps so a like-heavy cycle can't starve replies. Per-type limits alone let 8 follows plus 25 likes plus 6 comments plus 3 quote-tweets add up to 42 actions nobody authorized as a group.

It bounds duration, not just volume. Real quiet hours, 18:00 to 01:00 UTC, plus bimodal jittered intervals, because acting around the clock at a fixed period is the loudest automation tell there is. Volume caps by themselves still produce a metronome.

It trips on the observation, not the outcome. The breaker fires when the selector matches, not when I confirm the ban. Waiting for confirmation is what cost me three and a half days.

And it exits the process, leaving a sentinel a human has to delete. os._exit(1), plus an ACCOUNT_LOCKED.json that main() refuses to start past. My daemon runs under launchd with KeepAlive, so exiting on its own would have produced an instant respawn. A breaker that resets itself has never really tripped.

If the argument holds, what has to change is what we ship as a primitive. Every agent framework on the market ships retries, tracing and evals. None of them ship a rate integral, a duration bound, or a kill path the agent can't negotiate with, so every team builds a worse version of one after their first incident. Equity markets went through this exactly once and came out the other side with pre-trade risk controls enforced by regulators, because "the algorithm was reviewed" turned out not to survive contact with 45 minutes.

I think the review unit for agents shifts the same way. Less what did it decide. More how many times, how fast, for how long, and what stops it without asking permission. Mine stops in under a second now. Figuring out that the number mattered took three and a half days I'd rather have back.

Why LLM Agents Fail Silently in Production (And How to Detect It)

Orvi Das — Tue, 08 Sep 2026 04:30:03 +0000

Originally published on robatdasorvi.com

Six weeks of green. That's what I was looking at when I finally opened the raw log instead of the dashboard. Forty-two consecutive days of a scheduled agent reporting completed runs, zero exceptions, nothing in the alert channel. The engagement number underneath it hadn't moved since week one. Both things were true at once, and that's the whole problem. A silent agent failure doesn't announce itself. The process exits zero, the trace closes cleanly, and something you cared about quietly stopped happening in a way none of the counters you were watching were built to notice.

The cause, when I found it, was almost insulting. Four hardcoded target handles in a config file had gone dead. Suspended, renamed, deactivated, I still don't know. The agent dutifully visited each one, got a page with no post on it, correctly concluded there was nothing to reply to, logged no action needed, and moved on. Eighty-eight percent of the daily attempt budget was going to four accounts that no longer existed. Every individual decision in that chain was correct. The system was completely broken.

Why do LLM agents fail without throwing errors?

Because an agent's job is to handle ambiguity, and handling ambiguity is indistinguishable from absorbing failure. No exception gets thrown because the recovery path you paid for treats a broken world as one more input to route around.

A traditional program fails when reality violates its assumptions. An agent's entire purpose is to keep going when reality violates its assumptions. That's structural, not incidental. You built a thing whose selling point is robustness to weird inputs. A dead page is a weird input. The agent robusts right past it, then writes you a summary in confident English, because the summary comes from the same component that just failed and that component has no privileged access to whether it succeeded.

This is measured now, not anecdotal. A June 2026 study of 9,876 tau2-bench trajectories across eight model families found that agents asserting task completion while the environment state said otherwise accounted for 45–48% of all failures in single-control domains. Among AppWorld coding-agent trajectories that made an explicit status claim, 75.8% of failures were this false-success type. Three out of four broken runs got reported to the operator as finished work.

So a silent failure isn't an error you missed. It's a success message you believed.

Isn't silent agent failure just bad error handling?

No. Bad error handling produces failures with an exception sitting somewhere to be caught; the expensive silent failures produce no exception at all, so there is nothing for better handling to catch.

That was my first theory and it was wrong. I spent two weeks adding structured logging, per-step try/except blocks, richer exception context. At the end I had significantly better records of a failure I still couldn't see.

The instinct makes sense. Every engineer's reflex on hearing "it failed quietly" is you swallowed an exception somewhere. Sometimes you did. My article-generation lane was dying on JSON parse errors in eight of twenty runs, and each dead run burned the full 48-hour publishing slot before the next attempt. That one really was an exception, and I caught it in about a week once I bothered to look.

But the failure mode that actually costs you is the one where no exception exists to swallow. The dead-handles bug ran for a month and a half with nothing to catch, ever. Same with the Telegram approval flow, where the Skip button silently did nothing for months, quietly reclassifying every skipped item as a timeout and turning an entire quarter of my approval statistics into fiction. No stack trace was going to save me there. From the runtime's perspective nothing went wrong: a button was pressed, a handler ran. It just didn't do the thing.

The Berkeley taxonomy of multi-agent failures, built from 150-plus annotated execution traces across seven frameworks, puts task verification failures at 21.3% of the total, listed separately from design flaws and coordination breakdowns. It's not a subtype of bad error handling. It's its own animal, and it eats a fifth of everything.

Do better models reduce silent agent failures?

No. They reduce the visible failures and make the silent ones harder to spot, because a stronger model writes a more plausible completion summary for whatever still broke.

This is the part that took me longest to accept. Upgrading the model made my visible failures rarer and my silent failures harder to detect.

I want to be precise about the mechanism rather than cute about it. Better models do genuinely complete more tasks. On TheAgentCompany, a benchmark of 175 realistic long-horizon professional tasks, the top agent autonomously completed 30.3%. That's real progress over earlier numbers, and it's also a system that fails roughly seven times out of ten at work a competent human handles routinely. The same fluency that lifts the completion rate also improves the narration attached to whatever still breaks. The false-success paper found judges keyed on "confident closing language" as their primary signal. Guess which capability improves fastest with scale.

Sakana AI learned this in public in February 2025, when their AI CUDA Engineer reported speedups of up to 100x. Independent testing found the kernels ran about three times slower, and the company acknowledged the system had found a memory exploit in the evaluation harness that let it skip correctness checks. That wasn't a weak system. It was a strong optimizer pointed at a metric, doing what strong optimizers do.

Humans aren't better calibrated either, which should really end the argument. METR's July 2025 randomized trial had 16 experienced open-source developers complete 246 tasks in repos they'd worked in for an average of five years. With AI tools they were 19% slower, and estimated afterward that they'd been 20% faster. Thirty-nine points between felt outcome and measured outcome, in experts, on their own code. Self-report was never evidence.

Can an LLM judge detect silent agent failures?

Not reliably. The best-measured judge configurations land near 0.65 AUROC on one benchmark and 0.54 on another, which is close enough to chance that the judge is not doing the job you're buying it for.

This is the standard rebuttal and the data is unkind to it. Across five judge models and five prompting strategies, all given full task specifications, no configuration exceeded 0.65 AUROC on tau2-bench, and the same judges managed 0.54 on AppWorld API traces. A coin flip is 0.50.

Sit with 0.54 for a second. That's a system you'd pay per-token for, add latency for, and draw on an architecture diagram, in order to perform about as well as not having it. The judges failed because they were reading the same surface signals I was: closing language, action-sequence volume, the shape of a finished job. They weren't checking state. They couldn't, since nobody gave them any.

All of which is downstream of something established back at ICLR 2024, when Huang et al. showed that LLMs can't reliably self-correct reasoning without external feedback, and that performance sometimes degrades after self-correction. Their framing is what stuck with me: if the model could identify the error, why did it produce the error? An LLM judge reading an agent's own trace is intrinsic self-correction wearing a lanyard.

The same paper offers a deflating fix. Lightweight TF-IDF detectors, bag-of-words, no reasoning, technology from the 1970s, reached 0.83 and 0.95 AUROC on the same task at four to eight times lower latency. The thing that worked wasn't smarter. It was looking somewhere else.

How do you detect silent LLM agent failures?

Assertions against environment state the agent never touches, plus outcome heartbeats measured in units you actually care about. Not "did the run complete" but "is the number that justified building this thing still moving."

What I run on every agent lane now:

A liveness check on every external target before it's allowed to consume budget. Dead handle, dead lane, config gets flagged instead of silently absorbed.
A floor on real actions per window. Fewer than N state-changing operations in 24 hours is an incident, even though nothing errored.
Decision-outcome reconciliation: every approve/skip checked against the artifact it should have produced. That's how I finally caught the Skip button lying to me.
An independent counter for anything the agent reports about itself, computed by code the agent can't reach.

None of this is clever. All of it is boring, and boring is sort of the point. The rule I landed on: never let the component that performs the action also be the component that certifies the action. It's a separation-of-duties principle borrowed from accounting, roughly four hundred years older than the transformer.

Why don't more teams monitor agent outcomes?

Because verification usually costs more to build than the automation saves, so there's a quiet incentive not to look. Most teams running agents in production haven't looked, can't tell a working agent from a broken one, and have arranged their metrics so nobody has to find out.

That's not a slur on anyone's competence. It's arithmetic. To verify an agent's output you need ground truth, and if you had cheap ground truth you often wouldn't need the agent. So the honest verification layer gets scoped, estimated, and deferred to next quarter, while the dashboard shows uptime, token spend, p95 latency, trace counts. Every one of those measures execution. None of them measures outcome. Every observability stack I've seen will tell you the agent ran. Almost none will tell you anything happened.

Gartner predicted in June 2025 that over 40% of agentic AI projects will be canceled by the end of 2027, citing unclear business value alongside cost and risk controls, and noted that out of thousands of self-described agentic vendors only around 130 were doing anything real. "Unclear business value" is a wonderfully diplomatic way to say we ran it for a year and can't demonstrate it did anything. That's forty-two green days at scale, with a budget attached.

Which silent agent failures still can't be detected?

Judgment-shaped ones. Anything where the target was real, the action fired, the state changed, and the output is still quietly bad has no ground truth to assert against, so none of the checks above will ever flag it.

State-shaped failures I've mostly solved. Did the post go out, did the row get written, did the handle resolve, did the count move. Those have ground truth, and ground truth is checkable by dumb code with no opinions.

Judgment-shaped failures I haven't solved at all. My agent can generate a reply, publish it, and pass every assertion I own. Real target, real action, real state change, counter incremented. And the reply can still be mediocre in a way that costs me something slow and unmeasurable. I have no verifier for that. I have a sample I read manually, which isn't a system, it's a habit, and it degrades exactly when I'm busiest.

I also can't tell you my current setup isn't failing silently right now in some sixth way I haven't thought of. Every check I own was written after a specific incident. The next one will be too. I found the dead handles at six weeks, which makes six weeks my actual detection latency, and I have no particular reason to think it's improved since.

Your App Depends on 1,200 Packages. Find the Ones With a Single Maintainer.

Orvi Das — Mon, 07 Sep 2026 08:25:34 +0000

Pick any Node repo you own and run this. Takes about ten seconds.

// bus-factor.mjs — rank your deps by how few people can publish them
import { readFileSync } from 'node:fs'

const pkg = JSON.parse(readFileSync('package.json', 'utf8'))
const deps = Object.keys({ ...pkg.dependencies, ...pkg.devDependencies })

const rows = await Promise.all(deps.map(async (name) => {
  const meta = await fetch(`https://registry.npmjs.org/${name}`)
    .then(r => r.ok ? r.json() : { maintainers: [] })
  const dl = await fetch(`https://api.npmjs.org/downloads/point/last-week/${name}`)
    .then(r => r.json()).catch(() => ({ downloads: 0 }))
  return { name, owners: (meta.maintainers ?? []).length, weekly: dl.downloads ?? 0 }
}))

rows
  .filter(r => r.owners === 1)
  .sort((a, b) => b.weekly - a.weekly)
  .forEach(r => console.log(`${String(r.weekly).padStart(12)}  ${r.name}`))

Everything it prints is a package where exactly one npm account can publish a new version. Not one contributor. One person holding the keys. On a mid-sized app I ran this against, 61 of 340 direct and dev dependencies came back with a single owner, and the top of the sorted list was moving nine figures of downloads a week.

That's the number worth sitting with before anyone talks about supply chain strategy.

The script is optimistic

The registry's maintainers field means publish rights, not activity. Someone who hasn't opened the repo since 2021 still counts as an owner. So a package showing three maintainers can easily be one active human and two people who moved on. The real distribution is worse than what you just printed.

PyPI's JSON API is even less helpful here, so for Python use OpenSSF Scorecard, which looks at commit activity rather than account lists:

scorecard --repo=github.com/psf/requests --checks=Maintained,Contributors

Maintained scores recent commit and issue activity over the last 90 days. A 0 there on something in your production path is a finding, not a curiosity.

Three incidents that were funding problems wearing a code costume

Heartbleed, 2014. OpenSSL was securing a large share of the web's TLS on donations of roughly two thousand dollars a year. Not two thousand per contributor. Two thousand, total. The bug was a missing bounds check, and the missing bounds check was downstream of nobody having time to review.

Log4Shell, December 2021. CVSS 10.0, and the patch cycle ran through a holiday weekend, handled by a small group of volunteers while every Fortune 500 SOC on earth was paging. The companies that depended on Log4j had eight-figure security budgets. The library had a Slack channel.

xz Utils, February 2024. This one is the sharpest, because the attack surface was the maintainer. Sustained social pressure on a burned-out solo maintainer until a helpful new co-maintainer with commit rights looked like relief instead of a threat. The backdoor reached sshd through liblzma and got caught because a Postgres developer noticed his SSH logins had gotten about half a second slower and refused to let it go.

None of those were caused by bad engineering. They were caused by the ratio between how much the software was worth and how many people were paid to keep it alive.

Why nobody writes the check

The Linux Foundation and Harvard's Census II found that 136 developers were responsible for more than 80% of the lines of code in the top 50 npm packages. Tidelift's maintainer surveys have consistently found that roughly half of maintainers earn nothing at all from the work, and only a thin slice clear a thousand dollars a year.

The blocker usually isn't willingness. It's that procurement can process a forty-thousand-dollar observability contract in two weeks and cannot process two hundred dollars a month to an individual in Finland. There's no vendor record, no MSA, no security questionnaire to fill out, no PO number. The system is built to buy from companies, and the dependency is a person.

There's a measurement asymmetry too. You can put a dollar figure on the outage. You can never put one on the maintenance that stopped the outage from happening, so it always loses the budget argument.

What actually moves, ordered by effort

Start with npm fund. It's already in your toolchain, it reads your lockfile, and it prints funding URLs for everything in your tree. Two minutes, no new tooling.

Then make the giving proportional. thanks.dev and Open Source Collective both split a monthly amount across your actual dependency graph rather than whatever's trending. Five hundred a month distributed by real usage does more than five thousand to one popular project.

Vendor the small stuff. If a dependency is forty lines with one owner, copy it in with the license header and attribution intact. You've removed a publish-rights risk and stopped pretending someone else is on call for it.

Harden the install path. Committed lockfile, npm ci, --ignore-scripts where your build tolerates it, and Dependabot PRs that a human actually reads. The xz backdoor shipped in a release tarball that didn't match the git tree, so build from source where it's cheap enough to do.

The view from the other side

I maintain one of these. Small Python library, single owner, and it exists because of a bill.

An agent loop of mine had a retry-on-parse-error path where the counter never incremented. The model kept producing output that failed a schema check, the wrapper kept retrying, and it ran unattended overnight. Roughly 1.1 million output tokens and about $340 before I saw it at breakfast. My provider budget alert fired. Four hours after the loop started.

That's the design lesson, and it's why the enforcement has to happen before the request leaves your process:

Control	Fires when	What it doesn't stop
Provider budget alert	after spend is recorded, often minutes late	the spend that triggered it
Rate limit (RPM/TPM)	at the request boundary	a slow loop that stays under the limit for six hours
Pre-flight cap	before the socket opens	usage inside one already-approved call

So baar-core is a pre-flight kill switch. It checks the cap, and if you're over it, the provider is never contacted at all:

from baar_core import Budget, BudgetExceeded

budget = Budget(limit_usd=5.00, key=f"user:{user_id}:day")

try:
    with budget.reserve(estimated_usd=0.04):
        resp = client.messages.create(...)   # only runs if the reservation held
except BudgetExceeded as e:
    return Response(status=402, body=f"over cap: {e.spent} of {e.limit}")

The reserve() context manager matters more than the cap itself. Twenty parallel agents that each read "spent so far" and each independently conclude they're under budget will jointly blow straight through it. Reservation makes the read and the commit one atomic operation, so the twentieth agent gets a 402 instead of a share of the overrun. pip install baar-core if you want it.

It's free because charging for it would mean invoicing teams at the exact moment they discovered they'd been billed for a bug, and that's a worse world to build. noburn.dev is what we built on top of it for teams that need more than a library: same pre-flight enforcement, blocking calls before they fire when a user goes over budget, plus per-user caps and a spend history you can hand to finance.

Which also means I'm currently a single-maintainer row in somebody else's script output. I know exactly what that costs, and I still can't tell you how to price it.

What's the highest-download single-maintainer package in your tree, and would your team notice if that person stopped answering email?

Originally published at https://robatdasorvi.com/stories/why-open-source-won-and-what-that-victory-cost

The Hidden State Problem in Agentic Systems That Nobody Warns You About

Orvi Das — Mon, 07 Sep 2026 04:30:09 +0000

Originally published on robatdasorvi.com

On the morning of July 25th I opened Telegram to a column of approval cards. Each one was a reply my Twitter agent had drafted and queued up for me to sign off on. Quote tweets, comment threads. There was a first line in there I remember actually liking.

The account had been suspended since July 21st.

So: four days of drafts that could never post. And somewhere in that stack was an alert from the bot itself, saying in plain English that it had detected a lockout. Then it carried on. That gap, between a system knowing something and the system behaving like it knows it, is the part of agent state management nobody puts in the tutorial. Retrieval, memory, vector stores, all covered at length. The chapter where an observation fails to become a fact the loop obeys doesn't exist.

Three and a half days. Here's what was underneath.

Why Did My Agent Keep Running After It Was Locked Out?

Because "we are locked out" only ever existed as a log line and a Telegram message. It was never a value the control loop read. The function that detected the suspension notified me and returned, and the loop it returned into had no field to check.

In spirit the code went: check_for_lockout() scraped the profile, matched the suspension banner, called notify(), returned None. The scheduler that called it moved on to due("reply"), which read a timestamp file, decided enough minutes had passed, generated a reply, and pushed it into the approval queue. Every step correct in isolation. Nothing in the chain asked whether the account was alive, because nothing in the chain had anywhere to put that answer if it got one. The detection was flawless and completely inert.

Isn't This Just a Missing Return Statement?

That was my first fix, and it wasn't wrong so much as absurdly undersized. I made check_for_lockout() call sys.exit(1), watched it work, and figured that was that.

It was that, for exactly that one instance. Over the following week the same shape kept turning up in different clothes. A rate-limit response the HTTP layer logged and swallowed. A Telegram callback whose skip branch never wrote its decision anywhere, so the scheduler counted the item as timed out and requeued it. A daily action cap dutifully incrementing a counter in memory while the restart loop killed the process every few hours.

Four separate bugs, one structure underneath all of them: something true about the world got observed, got mentioned out loud, then got discarded before it could constrain any behavior. I still ship the sys.exit(1) and it's still correct. But treating it as the fix is how you spend a month patching one defect wearing four costumes.

The wrong turn was in my head, not in the code. I'd been reading these as bugs in error handling. They're bugs in state design.

What Is Hidden State in an AI Agent, Exactly?

Hidden state is any fact your agent has established that lives somewhere the control flow can't read. A log line. A chat message. A paragraph in the context window. A variable in a function that already returned. It's the difference between an agent that observed something and an agent that knows it.

The specific trap in LLM systems is that the context window looks like state. It's right there, it's readable, it accumulates over time. But a context window is a transcript, and a transcript is not a state machine. It records that something was observed. It never records that anything became true. When the model reads "ERROR: account suspended" on turn 14, that string is competing for attention with thirteen other turns and whatever arrives on turn 15. Nothing has changed structurally. The agent is exactly as free to draft a reply on turn 15 as it was on turn 1.

Which explains the direction agentic reliability tends to fail in. Sierra's τ-bench, published in June 2024, measured not whether an agent completes a task but whether it completes the same task repeatedly. The best GPT-4o configuration cleared 60% on pass^1 in the retail domain and fell below 25% on pass^8, roughly a 60% relative collapse once consistency is the thing being scored (arXiv:2406.12045). Capability was never the binding constraint. Reproducibility was, and reproducibility is a state problem.

Doesn't a Bigger Context Window Solve AI Agent State Management?

No, and the evidence here is unusually clean. The NoLiMa benchmark, presented at ICML 2025, tested retrieval where the question and the target share minimal literal overlap. That's what real agent state looks like, since "the account is suspended" rarely shows up verbatim in the phrasing of the next decision.

Eleven of the twelve models evaluated dropped below 50% of their own short-context baselines at 32K tokens. GPT-4o, one of the strongest performers in the set, fell from 99.3% at short context to 69.7% (arXiv:2502.05167). These are models advertising 128K windows and up. The window is real. The reliable attention inside it isn't.

So the obvious counterargument, just put the suspension notice in context and let the model handle it, fails twice over. It fails empirically, because the model's ability to act on a buried fact decays with distance long before the window fills. And it fails structurally, because even at 100% retrieval you've made a safety-critical guarantee contingent on a probabilistic read. My bot didn't need to remember the suspension. It needed to be incapable of attempting a post. Those are different requirements, and a longer prompt only satisfies one of them.

Why Do Multi-Agent Systems Make Hidden State Worse?

Because every agent boundary is a place where implicit decisions get dropped. The parent's context doesn't cross into the child. Only the instruction does.

Walden Yan's Don't Build Multi-Agents, published by Cognition in June 2025, has the cleanest illustration I know of. Ask a system to build a Flappy Bird clone. It spawns one subagent for the background and one for the bird. The first misreads the brief and produces a Super Mario Bros. background. The second produces a bird that doesn't move like Flappy Bird and doesn't match the art it's flying over. Neither subagent did anything wrong given what it was actually told. Yan's phrasing: "actions carry implicit decisions, and conflicting decisions carry bad results."

The Berkeley taxonomy paper Why Do Multi-Agent LLM Systems Fail? (arXiv:2503.13657, 2025) put numbers behind the intuition, deriving 14 distinct failure modes from 150 hand-annotated traces across seven popular frameworks, validated at inter-annotator agreement κ = 0.88 and then extended to a corpus of over 1,600 traces. Those 14 modes cluster into three families, and two of the three, inter-agent misalignment and task verification, are state problems rather than reasoning problems. Agents drop information across handoffs, or nobody is holding the state you'd need to check the result against.

Anthropic's own multi-agent research system write-up, also June 2025, reports a 90.2% improvement over single-agent Claude Opus 4 on research tasks at roughly 15× the token cost of chat, with token usage alone explaining about 80% of performance variance. Worth noticing where that works: parallel research, where the subtasks are genuinely independent of each other. Anthropic says outright that the pattern is a poor fit for tightly coupled work like coding, which is the same boundary Yan draws from the other side. Parallelism is affordable precisely when there's no shared state to lose.

So What Do You Actually Write Down?

Not everything. That was my second wrong turn. I started designing a serialization layer to persist the full agent trajectory, which is expensive, slow, and solves a problem I did not have.

The rule I landed on is much narrower, and I didn't expect it going in. Every observation that changes what the agent is permitted to do next has to become a typed value the control loop reads before it acts. Not what the agent knows. What the agent is allowed to do.

My bot's memory was fine, honestly. It could recall its own posts, its target list, its performance history. What it didn't have was a permission layer: a small, boring, explicitly-read set of gates that every outbound action passes through and that any detector can write to. Account alive. Under the daily cap. Outside quiet hours. Not rate-limited. The suspension check doesn't notify anymore, it flips a gate and halts the process. The rate limiter doesn't log anymore, it writes a cooldown timestamp the scheduler reads on the next tick.

I went into this assuming state management meant memory. It doesn't. Memory is the easy half, and it's the half every framework already ships for you. The dangerous state in an agentic system is what the model is still allowed to do, and that lives in your code rather than your prompt. An LLM can't hold a permission on your behalf. It can only be told about one, and being told is not the same as being bound.

What I Still Don't Know How to Fix

The gates work for the conditions I anticipated. For the ones I haven't, I have no method at all.

Every gate in my system exists because something already went wrong. The suspension, the rate limit, the cap, the quiet hours. Four gates, each paid for in an incident. The whole design is reactive by construction, and I obviously can't tell you what the fifth failure is, because if I could I'd have already written the gate. A real completeness argument for a permission layer would need something like a model of every state the platform can put me in, which is a specification of X's moderation system, which nobody outside X has.

The honest position: I've converted a class of silent failures into a class of loud ones, and I don't know what fraction of the class I've covered. The bot no longer runs 3.5 days into a wall. Whether it runs 3.5 days into some wall I haven't met yet is open, and every quiet day is weak evidence at best.

AI-Assisted Coding vs Vibe Coding: What the Difference Actually Is

Orvi Das — Fri, 04 Sep 2026 04:30:07 +0000

Originally published on robatdasorvi.com

What is the difference between vibe coding and AI-assisted coding?

The difference is whether you read the code.

That is the whole distinction, and I have yet to see it survive being made more complicated. Both practices use the same models, the same editors, the same prompt-and-generate loop. What separates them happens in the few seconds after the output appears: you either read it and take responsibility for it, or you run it and find out.

Everything else follows from that one choice. Security posture, maintainability, whether anyone can fix the thing in six months, whether you can explain your own system in a code review. One decision, repeated a few hundred times a week, compounds into two very different kinds of software.

What is vibe coding?

Vibe coding is building software with an LLM without reading what it wrote. Andrej Karpathy coined the term in February 2025, describing a way of working where you "fully give in to the vibes, embrace exponentials, and forget that the code even exists."

The phrase caught because it named something people were already doing and felt slightly embarrassed about. You describe what you want. The model writes it. You run it. If it breaks, you paste the error back into the chat and let the model try again. You never open the file. You are not reviewing an implementation, you are steering a black box by its outputs.

Collins named it Word of the Year for 2025 on 6 November 2025, and Merriam-Webster added it as slang that same year, which tells you something about how fast it moved from a tweet to a working practice. A throwaway tweet became a dictionary entry in nine months.

The important thing about Karpathy's original framing is that it was not a criticism. He was describing a legitimate mode for throwaway projects, where forgetting the code exists is exactly the point. The trouble started when the mode escaped the weekend project and turned up in production.

What is AI-assisted coding?

AI-assisted coding uses the same tools and keeps the engineering. The model writes the implementation; you still own the architecture, the review, and the bug.

You still make the architectural calls. You decide what the system is, how it is structured, what the boundaries are. The model writes implementations inside those decisions, fast, and you read every one of them before it becomes yours. When something breaks you debug it the way you would debug anything, by understanding it, rather than by describing the symptom to a chat window and hoping.

Addy Osmani has made this point sharply: the model is a very fast junior engineer and you are the tech lead. The junior writes a lot of code. The lead is still accountable for all of it. His estimate is that AI gets you about 70% of the way to working software, and the remaining 30% is still engineering judgment.

Simon Willison put the cleanest test on it in March 2025. If an LLM wrote the code, and you reviewed it, tested it properly, and can explain how it works to someone else, that is not vibe coding. That is software development that happened to use an LLM. The tool did not change what you are responsible for.

How do vibe coding and AI-assisted coding compare, side by side?

They differ on one input and eight outputs. The input is whether you read the generated code; everything below is downstream of that single decision.

	Vibe coding	AI-assisted coding
Who makes architecture decisions	The model, implicitly	You, explicitly
Do you read the generated code	No	Yes, before it ships
How you debug	Paste the error back into chat	Read the code, form a hypothesis
What you optimize for	Speed to something running	Speed to something correct
Can you explain the system	No	Yes
Who is accountable for a bug	Unclear	You
Good for	Prototypes, throwaway tools, learning, personal projects	Anything with users, data, money, or a maintenance horizon
Failure mode	Silent defects that surface in production	Slower than the demo suggested

The row that matters most is the last one. Both practices fail. They fail differently. AI-assisted coding fails by being less of a speedup than you were promised. Vibe coding fails by shipping something broken that nobody noticed was broken.

What does the data say about skipping review?

Unreviewed AI code works far more often than it is safe, and the measured gap between those two things is roughly 50 percentage points. Four independent 2025 studies land in the same place from different directions.

On security, the numbers are consistent across independent sources and they are not close. OX Security reports that 62% of AI-generated code ships with vulnerabilities. A Carnegie Mellon study built a benchmark called SusVibes — 200 repository-scale tasks drawn from 108 open-source Python projects, spanning 77 CWE weakness classes — and found that while 61% of AI-generated code functions correctly, only 10.5% passes security review. More than 80% of the solutions that passed the tests still contained a vulnerability. That is the gap between "it works" and "it is safe" stated as plainly as it can be.

Escape.tech scanned roughly 5,600 vibe-coded applications and found over 2,000 vulnerabilities, more than 400 exposed secrets including API keys and credentials, and 175 instances of exposed personal data — medical records, IBANs, phone numbers, emails. Their methodology writeup is worth reading on one point in particular: these were live production apps, findable within hours by anyone who bothered to look, not theoretical flags from an over-sensitive scanner.

A December 2025 study from Tenzai took 15 applications across five AI coding platforms and found 69 vulnerabilities in total, about half a dozen of them rated critical. Every single tool introduced server-side request forgery vulnerabilities. Zero of the 15 applications implemented CSRF protection. Zero set any security headers. Not most. Zero.

On defect rates, CodeRabbit's December 2025 analysis of 470 pull requests — 320 AI-coauthored, 150 human-only — found AI-coauthored code carried 1.7 times more major issues, with logic flaws up 75%, readability problems tripling, and security findings up to 2.74x higher. In raw terms: 10.83 findings per AI pull request against 6.45 per human one. Uplevel's 2024 study of roughly 800 developers after Copilot adoption found a 41% increase in bugs, with no significant improvement in pull request cycle time or merged throughput. The speed arrived and left again as rework.

Gartner's Predicts 2026 research projects that prompt-to-app development by citizen developers will increase software defects by 2,500% by 2028 without governance and quality controls. I would not put much weight on the specific figure, since nobody can forecast a defect rate two years out to that precision, but the direction is consistent with everything else here.

None of this says models write bad code. It says unreviewed code is unreviewed code, which was true long before any of this.

Does AI-assisted coding actually make you faster?

Measured, no — at least not for experienced developers in codebases they already know. The best available experiment found a 19% slowdown, and the developers living through it could not detect it.

METR ran a randomized controlled trial in 2025 with 16 experienced open-source developers working on repositories they knew well — averaging over 22,000 stars and a million lines of code — across 246 real tasks averaging about two hours each. Each task was randomly assigned to allow or forbid AI tools. The full paper is on arXiv.

Developers forecast that AI would make them 24% faster. They were measured at 19% slower. And after completing the tasks, having actually lived through the slowdown, they still estimated they had been about 20% more productive.

The slowdown itself I would hold loosely. Sixteen participants, mature codebases they already knew well, one point in time. The perception gap is the part that has stayed with me. These people could not tell from the inside whether the tool was helping, and they were experienced developers paying attention. What it felt like and what was measured pointed in opposite directions by nearly 40 percentage points. If you cannot feel a 19% slowdown while it is happening, you cannot feel a missing review either.

I recognize that gap, which is probably why it bothers me. AI-assisted work feels productive in a way that is hard to argue with while you are doing it.

Stack Overflow's 2025 survey of more than 49,000 developers found the same tension from a different angle. Adoption reached 84% using or planning to use AI tools, up from 76% the year before. Trust in AI accuracy fell from 40% to 29% year over year, with 46% actively distrusting the output and only 3% reporting high trust. The most cited frustration, from 66% of respondents, is "AI solutions that are almost right, but not quite," and 45% say they lose significant time debugging AI-generated code. Adoption went up while trust went down, which is not a pattern you see in tools that are working.

Almost-right is the specific hazard. Obviously wrong code fails loudly and costs you a minute. Almost-right code passes a glance, passes the happy path, and fails in the case you did not think to check. The only reliable filter for almost-right is a person reading it.

When should you use vibe coding, and when should you use AI-assisted coding?

Vibe code when being wrong is free. Use AI-assisted coding the moment someone other than you depends on the result.

Prototypes you will throw away. A script that renames 400 files. A weekend project with no users. Something you are building to decide whether the idea is worth building properly. Exploring an unfamiliar API where the fastest path to understanding is watching working code appear. Karpathy's framing holds up completely here, and the people sneering at vibe coding as a category are usually ignoring how much of programming is legitimately disposable.

Use AI-assisted coding when someone will depend on it.

Anything touching user data, money, authentication, or a database you cannot restore. Anything another developer will maintain. Anything you will still be running in a year. Size has nothing to do with it. The question is consequence: if this being subtly wrong would cost something real, read the code.

Choosing wrong is survivable. Drifting is what gets people. A prototype picks up a user. That user brings a second one. Nobody goes back and reviews the 4,000 lines that were never meant to survive, because no moment forces it, and the review would be miserable by then anyway. I have done this to myself more than once. It is a boring, procedural failure rather than a dramatic one, which is exactly why it keeps happening.

How do you tell which one you are doing?

Open the last thing you shipped and try to explain it out loud. If you are reconstructing your own system from evidence, you were vibe coding, whatever you were calling it at the time.

Not the intent. The implementation. Why is the retry logic there. What happens when that call times out. Why does that function take a list instead of a single item. If you can answer, you were doing AI-assisted development, whatever the model contributed.

The honest answer for most of us, most weeks, is that we do both, and the line moves depending on how tired we are and how much the task seems to matter. That is fine. The problem is losing track of which one is happening, because both produce output that looks the same right up until it does not.

The tools will keep getting better, and I think that makes this distinction more useful over time rather than less. Better models write more convincing code. More convincing code makes skipping the review more tempting and its failures harder to spot.

Why Budget Alerts Never Stop Runaway LLM Spend

Orvi Das — Sat, 29 Aug 2026 07:48:08 +0000

An "84% of monthly budget used" email landed on a Tuesday. I read it. I forwarded it to myself with a note that said watch this. Then I let the batch job run overnight anyway, and by Thursday morning the account had burned $1,900 against a $600 cap.

Nothing malfunctioned. The alerting pipeline worked exactly as specified, delivered on time, to the right person, who understood it. That's the uncomfortable part, and it's not a story about discipline.

A budget warning is a scarcity signal, and scarcity signals don't produce caution

Telling someone a resource is running low is one of the most-studied moves in behavioral psychology. The studied outcome is not restraint. It's an increase in perceived value and a pull toward consuming the remainder.

Worchel, Lee, and Adewole ran the cleanest version of this in 1975, published in the Journal of Personality and Social Psychology, with 200 participants. Same cookie, same room, same everything, handed over from either a jar holding ten or a jar holding two. The two-cookie jar produced higher ratings on value and attractiveness. Then they added a disclosure condition: some participants were explicitly told the supply had dropped because other people wanted them. Being told the mechanism didn't cancel the effect. That group rated the cookie highest of all.

Now read your dashboard banner again. "You have 16% of your budget remaining" is structurally identical to two cookies in a jar. It's a depletion cue with a full explanation attached, and the explanation is not a brake. You know the number is a threshold someone configured. You know the cap is arbitrary. Knowing changes what you can say about the banner. It does not change what you do next, which is usually to run the job.

What a soft limit looks like in code

Here's the version almost everyone writes first, in some dialect:

spent = usage_store.month_to_date(user_id)

if spent > BUDGET * 0.9:
    log.warning("user %s at %.0f%% of budget", user_id, spent / BUDGET * 100)
    metrics.incr("budget.near_limit")

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=4096,
    messages=messages,
)

usage_store.record(user_id, response.usage)  # ledger updates *after*

Every line of that is defensible in review. It still fails, in three specific ways.

The ledger is always behind by exactly the calls that matter. record() runs after the response returns. During a burst, the spend you most need to see is the spend that hasn't landed yet.

Concurrency reads the same stale number. Twelve workers pull month_to_date inside the same second, all twelve see 92%, all twelve pass, all twelve fire. No single request overshot the cap. Collectively they cleared it by 4x.

Agent loops move faster than any aggregator. A tool-calling agent that hits a malformed schema and retries will happily make forty calls in ninety seconds. If your usage rollup runs on a one-minute cron, the loop is finished before the graph even flinches. My $1,900 was mostly this: a retry loop with no ceiling, feeding a context window that grew on every pass.

The common thread: that if statement doesn't gate anything. It observes and then gets out of the way. It's a very expensive log line.

Reserve before you spend

The fix is boring and it's the same one databases have used forever. Don't check the balance, hold against it.

class BudgetExceeded(Exception):
    status_code = 402


def reserve(user_id: str, estimated_cost: float) -> Reservation:
    with db.transaction():
        row = db.query(
            "SELECT spent, reserved, cap FROM budgets "
            "WHERE user_id = %s FOR UPDATE",
            user_id,
        ).one()

        if row.spent + row.reserved + estimated_cost > row.cap:
            raise BudgetExceeded(
                f"{user_id}: {row.spent + row.reserved:.2f} committed, "
                f"needs {estimated_cost:.2f}, cap {row.cap:.2f}"
            )

        db.execute(
            "UPDATE budgets SET reserved = reserved + %s WHERE user_id = %s",
            estimated_cost, user_id,
        )
    return Reservation(user_id, estimated_cost)

The call site becomes a hold, then a settle:

estimate = price_of(model, count_tokens(messages), max_tokens)
hold = reserve(user_id, estimate)   # raises 402 before any network I/O

try:
    response = client.messages.create(...)
    hold.commit(actual_cost(response.usage))
except Exception:
    hold.release()
    raise

Two details carry the weight. FOR UPDATE means those twelve concurrent workers serialize instead of all reading the same stale total, which kills the race that soft checks can't see. And the estimate is an upper bound computed from input tokens plus max_tokens, so you reserve the worst case and refund the difference on commit. Over-reserving is a rounding error. Under-reserving is how you get a Thursday like mine.

Most of the annoying parts here are already solved. baar-core is an open-source Python library that wraps this into a decorator around your LLM client: pip install baar-core, set a cap, and it raises a 402 before the provider is ever contacted rather than after the tokens are billed. Atomic reservation is the piece I'd have gotten wrong on my own, and it's the piece that stops concurrent calls from jointly clearing a limit that neither one individually exceeded. noburn.dev is what we built on top of it for teams: the same pre-flight enforcement blocking the API call before it fires when a user is over budget, plus per-user caps and the ledger view.

Three rules that survive contact with a real incident

Enforce at one boundary. A single wrapper around your client, not if statements scattered across seven call sites where the eighth one, added last month by someone in a hurry, has none.

Reserve, don't observe. If your limit reads a number that a different process is responsible for updating, it isn't a limit. It's a lagging indicator wearing a limit's clothes.

Fail typed and loud. Return a real 402 with a machine-readable reason so callers, retry middleware, and agent loops can all distinguish "out of budget" from "transient 500" and stop instead of backing off and trying again. A log line is advice. An exception is a wall.

The reason this matters more than it should is the cookie jar. Alerts are built on the assumption that information changes behavior, and the research on depletion cues says information mostly changes your narration of the behavior. I didn't ignore my 84% email. I read it, understood it, and then did the thing anyway while explaining to myself why this run was different. Every person on your team will do the same, including the one who wrote the alert.

So stop shipping warnings where you meant to ship a wall.

What's the biggest gap you've measured between a spend alert firing and anything actually stopping?

Originally published at https://robatdasorvi.com/stories/why-scarcity-works-even-when-people-know-it-is-fake

DEV Community: Orvi Das

Your LLM Pipeline Never Throws: Three Guardrails for Silent AI Failure

1. A golden set that runs like a unit test

2. Watch the inputs, because they move first

3. Fail closed, and put a hard ceiling on the spend

The part nobody puts in the budget

I built a Chrome extension after a screen-share disaster reminded me how fragile browsing is

What it does

Why I did not make it a "block everything" tool

What I am looking for now

Which Jobs Will Still Exist in 2040 — My Honest Guess

Which jobs will still exist in 2040?

Is AI already taking jobs from young workers?

What is the hidden cost of choosing the safe career?

Hasn't technology always created more jobs than it destroyed?

Which careers are safest for young people starting out in 2026?

Should you take the safe job or retrain for a growing field?

Solo Founder Burnout Starts When Your Feedback Loop Loses Its CI Server

Your 23 users are statistically silent

Retention curves speak at small n

Pre-register your growth experiments

Commit your quit conditions before you're tired

Add a code reviewer to the business

The Monday script

Context Window Management for Long-Running AI Agents: 4 Patterns That Work

Why does my AI agent get worse the longer it runs?

Doesn't a bigger context window just solve this?

Does adding more relevant context improve accuracy?

What should actually live in the context window?

Why do experienced engineers still get this wrong?

How do you debug an agent's context window?

The Architecture Decision Nobody Talks About: When Not to Use an AI Agent

Should I use an AI agent or just write the code?

When should you avoid AI agents entirely?

Why do most AI agent projects fail?

But won't the models get good enough to fix this?

How do I know if my task is actually agent-shaped?

What should I build instead?

Every Agentic Loop Needs a Circuit Breaker. Mine Took 3.5 Days to Get One.

But isn't a retry limit already a circuit breaker?

Human approval doesn't fix this, and I have the logs

Why the agent can't be trusted to stop itself

The throttling objection, which I believed for about six weeks

What I actually built

Why LLM Agents Fail Silently in Production (And How to Detect It)

Why do LLM agents fail without throwing errors?

Isn't silent agent failure just bad error handling?

Do better models reduce silent agent failures?

Can an LLM judge detect silent agent failures?

How do you detect silent LLM agent failures?

Why don't more teams monitor agent outcomes?

Which silent agent failures still can't be detected?

Your App Depends on 1,200 Packages. Find the Ones With a Single Maintainer.

The script is optimistic

Three incidents that were funding problems wearing a code costume

Why nobody writes the check

What actually moves, ordered by effort

The view from the other side

The Hidden State Problem in Agentic Systems That Nobody Warns You About

Why Did My Agent Keep Running After It Was Locked Out?

Isn't This Just a Missing Return Statement?

What Is Hidden State in an AI Agent, Exactly?

Doesn't a Bigger Context Window Solve AI Agent State Management?

Why Do Multi-Agent Systems Make Hidden State Worse?

So What Do You Actually Write Down?

What I Still Don't Know How to Fix

AI-Assisted Coding vs Vibe Coding: What the Difference Actually Is

What is the difference between vibe coding and AI-assisted coding?

What is vibe coding?

What is AI-assisted coding?

How do vibe coding and AI-assisted coding compare, side by side?

What does the data say about skipping review?

Does AI-assisted coding actually make you faster?

When should you use vibe coding, and when should you use AI-assisted coding?

How do you tell which one you are doing?

Related reading

Why Budget Alerts Never Stop Runaway LLM Spend

A budget warning is a scarcity signal, and scarcity signals don't produce caution

What a soft limit looks like in code

Reserve before you spend