DEV Community: Azan Hyder

seeing some decent cities forming!

Azan Hyder — Mon, 12 Oct 2026 13:50:47 +0000

Azan Hyder

Oct 11

I turned GitHub into a 3D city

#showdev #github #ai #webdev

6 min read

ADD UR GITHUB!!

Azan Hyder — Sun, 11 Oct 2026 21:24:11 +0000

Azan Hyder

Oct 11

I turned GitHub into a 3D city

#showdev #github #ai #webdev

6 min read

I turned GitHub into a 3D city

Azan Hyder — Sun, 11 Oct 2026 17:10:34 +0000

Quick, do this before you read anything else:

Open github.com/<your-username>
Delete github.com
Type commit.city
Hit enter

That's the whole post. Thanks for reading. 👋

...okay fine, you scrolled. Here's what you'd have seen:

That's Vercel's last 12 months: **21,136 commits* replayed in about 20 seconds. Every one of those beams is a real commit.*

Wait, what am I looking at? 🏙️

commit.city takes any GitHub user, org or repo and builds it as a city, then replays the last year of commits as a time-lapse.

🏝️ Islands are repos.
🏢 Towers are files. Height is line count, colour is file type.
💥 Beams are commits. They fly in, and the files they touched light up.
🌙 Repos cool down as they go quiet: hot, active, cooling, dormant, abandoned. (We all have that one island.)
🕐 Day and night follow your real timezone. It's read from your own commits, so yes, it knows about the 3am pushes.

There's no sign-up and nothing to install. Your first visit builds the city while you watch.

Choose your fighter 🎮

All of these are already built, so they open instantly:

Click	What you get
commit.city/torvalds	Linus's year
commit.city/sindresorhus	2,696 commits in one year. Respect.
commit.city/karpathy	Karpathy's repos
commit.city/vercel	a whole org: 23 repos, 21,136 commits
commit.city/facebook/react	one repo, every contributor
commit.city/denoland/deno	👀 hold that thought
commit.city/events	all of GitHub, live: the public firehose as a city

Now press A 🅰️

This is the part I actually built it for.

More and more commits aren't written by people anymore. Press A in any city and the agent lens turns on: every beam is coloured by who sent it.

🔵 people are cyan
🟠 bots are amber (dependabot, renovate, github-actions and friends)
🌈 each AI agent gets its own colour: Claude, Copilot, Cursor, Codex, Devin, Jules, Gemini and more (17 so far)

How does it know? It reads the receipts the agents leave behind:

The author. Some agents commit as themselves, like copilot-swe-agent[bot].
The message. A Co-authored-by: trailer, or a footer like Generated with Claude Code.
Bots. Any [bot] account.

Everything else counts as human. Agent work that isn't disclosed looks human, so the number is a floor, not an audit. (Your secret's safe. For now.)

Pop quiz 🧠

Every number here comes from the live cities, last 12 months. No peeking.

Q1. Which of these has the biggest share of commits written by AI agents?
torvalds · vercel · facebook/react · denoland/deno

🥁 Reveal the answer
denoland/deno, at 14.3%: 430 of 2,997 commits.

And of those 430, 404 are Claude. Four hundred and four. I couldn't have planned that better. The rest: Copilot 21, Cursor 3, Codex 2.

(Relax, Deno fans. 2,554 of those commits are human. The humans were found.)

Q2. How many agent commits does torvalds have?

🥁 Reveal the answer
Zero. 757 commits, 0 agents, 0 bots. Same for sindresorhus: 2,696 commits, all human.

Built different. 🫡 (Reminder: floor, not audit. The lens only sees agents that sign their work.)

Q3. Vercel's 21,136 commits: what share came from bots (not agents, plain bots)?

🥁 Reveal the answer
19.5%: 4,127 bot commits, plus 9.2% from AI agents (1,939). So about seven in ten Vercel commits last year came from a person. Of the agent commits, 1,541 were Claude, then Cursor 283, Copilot 94 and v0 21.

How'd you score? Tell me in the comments. Honour system. 🫡

Things to press 🕹️

Key	What it does
drag · right-drag · scroll	orbit · pan · zoom
click a repo	inspect it (Esc goes back)
Space	play / pause
1 to 6	speed, from 1 hour/sec up to 2 weeks/sec
A	the agent lens 🌈
R	record a clip of your year (MP4/WebM, wide, vertical or square)
O · L · H	auto-orbit · labels · hide the UI

R is the fun one. The year replays in 10, 20 or 30 seconds while the camera orbits, and your browser records it. Nothing gets uploaded. That's how I made the GIF at the top.

Also, Share → copy this moment gives you a link that reopens at the same time, from the same camera angle. Great for "look at THIS day" moments.

Put your city on your profile 🪪

Every city has a README card. Paste this into your GitHub profile README and swap YOU for your username:

[![my commit city](https://commit.city/api/card?t=YOU)](https://commit.city/YOU)

It's a self-contained SVG, so GitHub renders it, and the towers rise as it loads. Here's a snapshot of Vercel's (on GitHub it's live):

How it works (for the nerds 🤓)

The hosted builder reads public repos only, with a read-only token that can't see anything private.
It replays the git history into a compact JSON city, which gets cached. The first visitor waits a few seconds (a few minutes for a huge org). Everyone after that gets it instantly.
The renderer is three.js. The towers keep their footprints over time, so you can watch a single file grow, shrink, move or die.
The agent rules for all 17 agents live in one file, schema/agents.json. Both test suites check it. Adding an agent is a one-file PR.

Got private repos? The same pipeline ships as a GitHub Action and a stdlib-only Python CLI. Either one writes a single offline HTML file. Private repos stay anonymised: names become private/repo-N and files become dir3/file17.ts, while the sizes and timing stay real.

It's open source, MIT:

azank1 / commit-city

Your GitHub, as a living 3D city. Swap github.com → commit.city on any user, org or repo and watch a year of commits build it.

commit.city

Your GitHub, as a living city.
Swap github.com for commit.city on any user, org or repo and watch a year of git history replay commit by commit, as a 3D city you can scrub, fly through and share.

Open commit.city → · GitHub, live · Agent lens · README card · Private repos · How it works

Try it: swap the domain

On GitHub	On commit.city	What you get
`github.com/torvalds`	`commit.city/torvalds`	a person's year: every public repo they committed to, wired to their avatar
`github.com/vercel`	`commit.city/vercel`	an org: its 24 busiest repos with everyone's commits
`github.com/facebook/react`	`commit.city/facebook/react`	one repo, every contributor
`commit.city/denoland/deno?all=1`	a repo's entire history, from the first commit

There's no sign-up and nothing to install. The first visit builds the city while you watch, repo by repo. That takes a few seconds for a small account and a few minutes for a big one. After that it's…

View on GitHub

Your turn 🫵

Drop your commit.city/<you> link in the comments, with A pressed.

Highest agent percentage wins absolutely nothing. Lowest gets the Linus badge. 🏅

an engagement bot commented on this article.. and it posted its reasoning trace instead of the comment lol. It probably thought the article was a video.. it leaked its rules and got cut off mid word.. Nothing checked the output before it left.. weird!

Azan Hyder — Fri, 25 Sep 2026 12:50:50 +0000

Guo Qiang

Sep 25

From SEO to GEO: Optimizing for AI Search Engines

#seo #ai #webdev #tutorial

10 min read

https://github.com/azank1/code-desk-cli

Azan Hyder — Fri, 25 Sep 2026 08:14:29 +0000

Azan Hyder

Sep 25

When should an agent stop?

#ai #agents #machinelearning #research

9 min read

GitHub - azank1/code-desk-cli: Run an office of AI coding sessions from your terminal: named desks, two hats, gated handoffs, and a token meter. · GitHub

Run an office of AI coding sessions from your terminal: named desks, two hats, gated handoffs, and a token meter. - azank1/code-desk-cli

github.com

When should an agent stop?

Azan Hyder — Fri, 25 Sep 2026 07:57:42 +0000

Field note 009. A replication of a published agent-loop benchmark, and what it hides.

Abstract

Every agent loop has to decide when to stop. The usual options are a fixed max_iterations or letting the agent declare itself done. In June I published a small benchmark with CDV, an open-source judge for coding agents, showing that a Bayesian adaptive stop rule used 41% fewer steps than a fixed six-step budget while reaching the quality bar on 99.7% of tasks. This note replicates that result on today's code (it reproduces to the digit), then extends it. Three findings. (1) The adaptive policy ties a plain threshold rule ("stop at the first score ≥ 0.80") everywhere, and it does so by construction: the threshold guard runs first in the guard stack. (2) Both collapse as soon as the judge's score is noisy. At a per-step noise of σ = 0.10, the loop believes it reached the bar in 99.5% of runs and actually reached it in 71%. A fixed budget of six, which never reads the score, holds 93% at every noise level. (3) A one-line fix, requiring two consecutive scores over the bar, restores 97% true reach at σ = 0.10 for 1.4 extra steps. The published headline was true and beside the point: the decision is only as good as the score it trusts.

The problem

An agent loop is a repeat-until: act, evaluate, decide whether to go again. The deciding part is where things go wrong. A fixed max_iterations is either too small for hard tasks or wasteful for easy ones. Letting the agent grade itself means the entity deciding to stop is the entity being judged, and agents optimise reported progress.

CDV's answer was to separate the two. Each step the agent claims is scored by a judge, and a policy decides from the score history whether to continue. The policy is a stack of guards evaluated in order, first stop wins: a score-threshold guard, a plateau guard (last three scores within 0.01 of each other), a Bayesian guard (stop when the learned expected improvement over the remaining steps can't close the gap to the bar), then budget, wall-clock, token and repeated-output guards.¹

The benchmark I shipped with it compared four strategies on a synthetic task set. This note asks two questions the benchmark didn't: what is the adaptive policy actually adding over the simplest rule? and what happens when the judge is wrong?

Prior work, briefly

The published table (June 2026, commit b8a219f in the public repo):²

Strategy	Mean steps	Mean final score	% reaching 0.80
fixed (budget = 2)	2.00	0.698	34.3%
fixed (budget = 6)	6.00	0.939	94.0%
threshold (reactive)	3.56	0.852	100.0%
adaptive (CDV)	3.56	0.852	99.7%

Read the third and fourth rows again. They are the same row. The headline compared row 4 to row 2 and said "41% fewer steps". It did not say that the plain threshold rule was already there, with nothing learned and nothing Bayesian about it. The benchmark script was later removed in a repository cleanup; it's still in the public history, and this note restores it.

Method

The simulation is the original one, unchanged. Each task has a hidden quality curve with diminishing returns, q(t) = 1 − (1 − s₀)·e^(−r(t−1)), with s₀ and r drawn per task from three task types (easy, medium, hard) that converge at different depths. The bar is 0.80; the cap is 8 steps. What the loop sees is q(t) plus Gaussian noise. The original used σ = 0.02.

Replication. I ran the restored script against the current package (cdv at 356d496, Python 3.12.3). It reproduces the table above exactly.³

Extension. Three sweeps, each over 10 seeds × 300 test tasks, reported as mean ± 1 sd:

Noise. σ ∈ {0.02, 0.05, 0.10, 0.15, 0.20}. The key metric changes: true reach is whether the hidden quality was ≥ 0.80 at the step the loop stopped, as opposed to observed reach, whether the noisy score was. The original benchmark only measured the second.
Cold start. At σ = 0.10, the adaptive policy warmed on 0, 10, 30, 100 or 300 prior tasks before the test split.
Cheap fixes. Three rules that need no learning: confirm-2 (stop at the first pair of consecutive scores ≥ 0.80), smooth-2 (stop when the mean of the last two scores ≥ 0.80), and margin (stop at the first score ≥ 0.85).

No LLM is called anywhere. This is a study of the decision policy given a score, not of any model's quality. The scripts, raw results and figures are in the artifact folder linked at the end.

Results

1. Adaptive equals threshold, by construction

Across all five noise levels the adaptive policy and the threshold rule are within one standard deviation of each other on every metric (Figure 1, the two overlapping lines). The reason is in the code, not the statistics: the guard stack evaluates ScoreThresholdGuard first.¹ The adaptive policy can therefore stop earlier than the threshold rule (via plateau or Bayesian guards) but never later, and in this simulation it almost never fires earlier once the priors are warm. Whatever the Bayesian machinery is worth, this benchmark cannot show it.

2. Trusting one score collapses under noise

Table 1 is the noise sweep. The observed reach for the threshold rule is ~99.5% at every σ: the loop always saw a score over the bar before it stopped. True reach falls from 93.4% at σ = 0.02 to 54.8% at σ = 0.20.

σ	fixed 6, true	threshold, steps	threshold, observed	threshold, true	adaptive, true
0.02	92.8 ± 1.1	3.66	99.9	93.4 ± 1.4	92.8 ± 1.6
0.05	92.8 ± 1.1	3.66	99.7	84.6 ± 2.0	84.1 ± 1.8
0.10	92.8 ± 1.1	3.54	99.5	71.2 ± 2.7	70.9 ± 2.5
0.15	92.8 ± 1.1	3.38	99.5	61.4 ± 3.3	61.0 ± 3.2
0.20	92.8 ± 1.1	3.23	99.6	54.8 ± 3.6	54.5 ± 3.5

Table 1. Percent of tasks reaching the bar, mean ± sd over 10 seeds. "Observed" is what the loop believed; "true" is what was the case.

Notice the step counts fall as noise rises. The loop isn't just wrong more often; it's wrong faster. A noisy judge produces a lucky score sooner, and the rule takes the first one it sees.

The fixed six-step budget, the strategy the original headline was beating, holds 92.8% true reach at every noise level, because it never reads the score. Above σ ≈ 0.03, the dumb budget delivers more real quality than either learned rule. It costs 2.3 more steps per task to do it.

3. The Bayesian guard is harmful before it is warm

At σ = 0.10 with no warm-up history, the adaptive policy stops after 2.79 steps on average and reaches the bar truly on 50% of tasks, worse than the threshold rule (71%) and far worse than the fixed budget (93%). It needs about 30 prior tasks to catch up (Figure 3). The should_continue rule falls back to a plain threshold check for the first three observations, then starts trusting expected-improvement estimates that are still wide.⁴ In a real deployment that is the first sprint on every new task type.

4. A one-line fix

Confirm-2, stopping only when two consecutive scores clear the bar, is the best rule tested at every noise level (Table 2). At σ = 0.10 it reaches 97.2% true reach in 4.91 steps: more real quality than the fixed budget, for one step less. It costs 1.4 steps more than the threshold rule and buys 26 points of true reach.

σ	threshold	confirm-2	smooth-2	margin +0.05	fixed 6
0.02	93.4 (3.66)	100.0 (4.65)	100.0 (4.28)	99.9 (4.19)	92.8 (6)
0.05	84.6 (3.66)	99.6 (4.73)	97.8 (4.27)	96.5 (4.19)	92.8 (6)
0.10	71.2 (3.54)	97.2 (4.91)	92.8 (4.25)	85.1 (4.03)	92.8 (6)
0.15	61.4 (3.38)	95.4 (5.06)	88.7 (4.28)	73.9 (3.81)	92.8 (6)
0.20	54.8 (3.23)	93.7 (5.16)	85.3 (4.28)	64.7 (3.59)	92.8 (6)

Table 2. True reach % (mean steps). Margin, the intuitive fix, is the weakest: a higher bar is still one noisy sample.

Findings

The published comparison was against the wrong baseline. Adaptive beat a fixed budget; it tied the simplest reactive rule, and the guard order guarantees it can't do otherwise here.
The decision is only as good as the score. A loop that stops on one crossing of the bar is fooled at a rate set by the judge's noise, and it is fooled faster as the noise grows. Observed success stays flat while real success falls. Any harness that reports "reached the bar" from a single judge score is reporting the judge's variance.
Learning has a cold-start cost that a fixed rule doesn't. Before ~30 observations per task type, the Bayesian guard is worse than not having it.
Redundancy beats sophistication. Asking the judge twice (confirm-2) does more for real quality than either a learned prior or a higher bar. That is the same argument CDV makes at the scoring layer with min(A, B), the stricter of two judges wins, applied at the stopping layer.

Limitations

This is a simulation of the decision policy, not of models. The quality curve is a hand-made exponential; real agent progress isn't smooth, isn't monotone, and can regress. Noise is Gaussian and independent per step; a real judge's errors are correlated with the task, which is worse for confirm-2 than this study shows.
Noise levels are assumed, not measured. I don't know what σ a real LLM critic has on real coding steps. If it is under 0.03, the threshold rule is fine and this note is moot. If it is over 0.05, the fixed budget wins on quality. Measuring that number is the next study.
The adaptive policy was tested as shipped, with default guards. Reordering the stack (Bayesian before threshold) or tuning the plateau delta could change result 1. I did not tune anything; that would be a different claim.
Same author, same code. I wrote the benchmark and the policy it flatters. The replication and the extension are both mine. Treat the artifact as the check, not my reading of it.
The cold-start result depends on total_calls < 3 fallback logic in the priors and on the warm-up being drawn from the same distribution as the test set. A mismatched warm-up would be worse.

What would change my mind

A live run, with a real judge scoring real agent steps, where the single-crossing rule's true reach (measured by a held-out oracle: the test suite, a human, a stricter judge) tracks its observed reach within a few points. That would mean judge noise is low enough that none of this matters. I'd also update if a reordered guard stack let the Bayesian guard beat confirm-2 on the noise sweep at equal step cost.

Artifact

Original benchmark, restored from public history: git show b8a219f:benchmarks/adaptive_vs_fixed.py in azank1/cdv.
Extension scripts, raw results (JSON), and the figure code: github.com/azank1/code-desk-cli → research/009-stop-rule.
Environment: cdv at 356d496, Python 3.12.3, no network, no model calls. Runtime under two minutes.

Notes

src/cdv/guards.py, default_guard_stack(): ScoreThresholdGuard, PlateauGuard, BayesianGuard, then budget, max-steps, timeout, token and output-repeat guards; GuardStack.evaluate returns the first stop reason. CONVERGENCE_DELTA = 0.01.
benchmarks/results/adaptive_vs_fixed.md at commit b8a219f (2026-06-25); the same table is in the README at README.md §Evidence.
Output of the restored script on 2026-09-25: identical rows, 41% step saving, 99.7% goal-reach.
src/cdv/priors.py, AdaptivePriors.should_continue: if prior.total_calls < 3: return current_score < quality_threshold, then expected-improvement summation over the remaining steps.

anyone using Jev out there?

Azan Hyder — Fri, 25 Sep 2026 07:52:42 +0000

Azan Hyder

Sep 24

Can AI Trust Its Gut? How Jev works

#ai #agents #security #devops

7 min read

Can Vibe Coding Build Production Software Without an Engineer?

Azan Hyder — Fri, 25 Sep 2026 07:35:05 +0000

Short answer: not yet. Longer answer: an engineer's job splits in two, and a founder can already own one half.

Table of contents

Someone asked the right question
"The AI writes the code, so who needs an engineer?"
The code was never the business
Run it like an office, not a chat
Every part of the business, on one board
"Isn't this just bureaucracy for robots?"
So what's the worst that can happen?
How not to fix it
Vibe code the work. Engineer the office.
The founder's half

Someone asked the right question

This week someone asked on X: can vibe coding produce production-ready software without an experienced engineer?

It's the right question, and most answers argue about the wrong thing: whether the model can write the code. It can. Writing the code was never what made software production-ready.

"The AI writes the code, so who needs an engineer?"

The belief goes like this. You describe what you want, the model builds it, and the engineer becomes an expensive middleman. Just vibe it.

For a weekend project, that's true. You describe, it builds, you click around, it works. Ship it.

The model never pushes back, never asks for a raise, and never says "that's not in scope." Which, it turns out, is the whole problem.

Then someone pays for it.

The code was never the business

Think about a restaurant. Great cooks don't make a restaurant. Someone decides the menu. Someone tastes the plate before it leaves the kitchen. Someone watches what the ingredients cost. The cooks can be brilliant, and without those three jobs you're one bad night away from closing.

Vibe coding hired the cooks. It didn't hire the other three.

That's what an experienced engineer actually brings, and most of it isn't typing:

Deciding what's in scope. What gets built this week, and what waits.
Saying "not done yet". With evidence: which tests must pass, what counts as reviewed.
Watching the cost. Noticing when a task has eaten three times what it should.

Ask one model to do all three, in the same chat where it writes the code, and it does them the way a new hire would on day one: confidently, and with nobody checking.

Run it like an office, not a chat

This is why I'm building Code Desks, a small open-source tool that runs AI coding sessions like an office instead of one long chat.

A desk is one real AI coding session (Claude Code, Codex or Cursor) with a job description: product manager, developer, reviewer, or whatever your office needs.
An office is a set of desks working one project on a sprint cadence, written down in one file you can read.
Work moves between stages through gates. A gate is a checklist with two columns, and nothing moves until both are filled.

The two columns are the trick, because one person can wear two hats:

Hat	Owns	In business terms
Owner	scope, priority, the final yes	the CEO, the product manager
Engineer	estimates, risks, tests green, evidence	the CTO

The owner decides. The engineer proves. A gate can be engineer-first: the owner's "yes" is refused until the evidence is on file. Nobody can sign off on nothing.

Every part of the business, on one board

Here's the part that matters if you're not an engineer. The office file reads like an org chart with a budget attached:

desks:
  pm:     { role: roles/pm.md,     harness: claude, budget: 10_000_000 }
  dev:    { role: roles/dev.md,    harness: claude, budget: 36_000_000 }
  review: { role: roles/review.md, harness: codex,  budget: 5_000_000 }
  launch: { role: roles/launch.md, harness: claude, budget: 4_000_000 }

Product: the PM desk turns your idea into scoped pieces of work.
Engineering: the dev desk builds. A review desk, running a different model, checks it.
Growth: a launch desk writes the landing page, the docs and the announcement, in the same project and through the same gates. It's just another job description.
Finance: every desk gets a token budget per sprint, and a meter reads what each desk actually used from the logs the sessions already write. No estimates, no guessing a price. Just a line like "the PM desk used four times its budget this sprint".

Here's the shape of what the meter prints (trimmed):

usage sprint 2 · 2026-09-22 → 2026-09-29
DESK    HARNESS  SESS  TURNS  TOTAL    BUDGET          PROJECTION
pm      claude   3     412    41.8M    418% of 10.0M   125.4M/30d
dev     claude   5     1105   132.2M   367% of 36.0M   396.6M/30d
review  cursor   2     96     n/a      no data

No dashboard, no invoice. A table that says which hire is over budget and what next month looks like if nothing changes.

That last one changes the conversation. Each desk is an investment: tokens in, work out. The meter shows what each desk cost. The board shows what it got through the gates. Where next sprint's budget goes is a business decision, and it belongs to the owner.

"Isn't this just bureaucracy for robots?"

Fair objection. If you're building a weekend toy, skip all of it.

But look at what a gate actually replaces. Not meetings. It replaces the moment an experienced engineer leans over and asks "did you actually run the tests?" The gate asks it every single time, including at 3 a.m. and on the tenth run of a Tuesday, when nobody is leaning over anything.

So what's the worst that can happen?

Without the office: the agent says it's done. The tests it ran were ones it wrote itself. (That happened to me: 16/16 green, and the one check that mattered said no.) Nobody tracked what it cost. You launch, and your first real customer finds what the tests didn't.

With the office: the same agent says it's done. The gate runs the project's real test command, it fails, and the work doesn't move. You find out on Tuesday, not from a customer.

How not to fix it

Hire an engineer to read every line. It works for one project. It doesn't keep up with how fast agents write.
Use a bigger model. A smarter model writes more code, faster. It doesn't decide what's in scope or what counts as done.
Write a longer prompt. A prompt is a suggestion the agent weighs against its goal. A gate isn't.

Vibe code the work. Engineer the office.

So, can vibe coding produce production-ready software without an experienced engineer?

Not yet. Someone still has to decide what "done" means and write the checks that prove it. That's engineering judgment, and the model shouldn't be the one grading its own work.

But the rest of the job (the scoping, the handoffs, the "not yet", the budget) can be written down once and run every sprint.

Vibe code the work. Engineer the office.

The founder's half

If you're a founder or a product manager building with AI, you don't have to become an engineer. You have to run the office: hire the desks, set their budgets, decide what "done" means, and keep the final yes.

That's the half you can own today. Borrow the other half, the checks, from an engineer once, and let the gates ask the question every time after that.

Code Desks is early (v0, MIT): github.com/azank1/code-desk-cli

What's the first job you'd give a desk that isn't writing code?

Related: Can AI Trust Its Gut?, on when an agent's fast call can be trusted, and Vibe engineering with Opus 5.5? Route by the feedback, on picking the model by what comes back when it acts.

Can AI Trust Its Gut? How Jev works

Azan Hyder — Thu, 24 Sep 2026 17:10:48 +0000

Five questions to ask before a fast model decides what your agent is allowed to do.

Table of contents

The agent that doesn't ask anymore
"Give your agent a System 1"
Firefighters and stock pickers
Five questions before you trust the judge
"Just use a bigger judge"
So what's the worst that can happen?
How not to fix it
Trust a fast call only where the world corrects it
Can the gut learn?
Intuition is earned

The agent that doesn't ask anymore

Agents don't just write code now. Point one at your website and it'll find prospects, write the emails, send them and book the calls. Point one at your repo and it'll run the migration.

Nobody wants to click "approve" on every step, so a pattern is spreading: before each action, a small, fast model gets asked one question. Is this OK? It answers in milliseconds, with a confidence score, and the agent carries on.

It's a good idea. It's also one more thing you're trusting, and most setups never ask how far.

"Give your agent a System 1"

The idea borrows from Daniel Kahneman. System 2 is slow, careful reasoning. System 1 is the fast, automatic call: the gut. Your big model is the slow thinker, and the little judge in front of every tool call is the gut.

Fast, cheap, always on. What could go wrong?

Firefighters and stock pickers

Kahneman spent a career showing where gut calls fail. Gary Klein spent one showing where they work: fire commanders who knew, without being able to say why, to get their crew out of a building moments before the floor gave way.

In 2009 the two wrote a paper together to settle the argument, and ended up agreeing. A gut call deserves trust when two things hold:

the environment is regular enough to be predictable, and
there's a real chance to learn it: lots of practice, with feedback that comes back fast and clear.

A firefighter gets both. A stock picker gets neither: the market is noisy and the feedback is slow and ambiguous, so confidence grows while skill doesn't.

Now look at the judge in front of your agent. It says "yes" to an email. Does it ever find out whether that email was a mistake? It says "yes" to a migration. Does anything tell it the migration broke production at 3 a.m.?

A judge that never learns whether its "yes" turned out fine is a stock picker.

Confident, fast, and with nothing correcting it.

Five questions before you trust the judge

None of these is about a particular product. They're true of any model that sits in front of an agent's actions.

1. Where does the question go? To judge an action, the judge has to see it: the command, the file path, the email body, the customer's name. If the judge is hosted, all of that leaves your machine on every call. That may be fine. It should be a decision, not a default.

2. Is "97% sure" a probability? A confidence score is only a probability if someone checked it against outcomes: of all the calls it was 97% sure about, were about 97% right? If nobody measured that on your kind of actions, it's a vibe with decimals.

3. What happens when it isn't sure? Every judge has an unsure zone. Does unsure mean allow, deny, or ask a human? The worst answer is the one nobody chose: a default buried in a config file.

4. Can you check a "yes" tomorrow? Something goes wrong. Can you show what the judge was asked, what it answered, and which rules it was running at the time? Or is the answer "it said OK, apparently"?

5. Who else is talking to the judge? The judge reads the action, and the action was written by the agent, which may have just read a web page, an email or an issue someone planted. The input to your safety check is partly written by whoever wants to get past it.

"Just use a bigger judge"

The obvious fix is a smarter model in front. But a bigger judge is slower, so it gets skipped in exactly the places you run the most actions: the fast loops. It still hasn't been measured on your actions. And it still leaves you nothing to check tomorrow.

Thinking longer doesn't create feedback.

So what's the worst that can happen?

Play it forward. An outreach agent drafts 200 emails with the wrong price in them. The judge sees 200 well-formed, polite emails to real prospects and is 96% sure each one is fine. They go out.

You can't unsend an email.

Nobody did anything wrong. The judge answered the question it was asked. It was just asked to use its gut in a place where its gut had never been corrected.

How not to fix it

Turn it off because it's slow. Now there's no judge, just faster mistakes.
Trust the accuracy number on the landing page. It was measured on someone else's actions, not yours.
Add a line to the system prompt. Rules get weighed against the goal. Checks don't.

Trust a fast call only where the world corrects it

Sort your agent's actions by two questions: will I find out fast if this was wrong? and can I undo it?

Run the tests, lint, read a file: the world corrects you in seconds, and nothing's lost. Let the gut call it.
Send an email, run a migration, move money, delete anything: no quick correction, no undo. That's not a gut call. Stop and ask a human, or check it with something slower.

The rule:

Trust a fast call only where the world corrects it.

Can the gut learn?

Everything above points at one missing piece: feedback. That's exactly what reinforcement learning is for. The judge makes a call, the world answers, and the answer becomes a reward the judge learns from. Done carefully, it turns a stock picker into a firefighter.

Cheapest first:

Calibrate before you learn. Check whether "97% sure" is right about 97% of the time on real outcomes, and adjust the scores until it is. No RL needed, and it fixes question 2 on its own.
Treat each call as a bandit, not a game. A gate decision is one move (allow, ask or deny) with one outcome. A contextual bandit, the simplest kind of RL, fits that shape better than full RL, and it's far cheaper to run.
Every "ask" is a free label. When the judge isn't sure and asks a human, the human's answer is training data. So are the quieter signals: a commit reverted after an allow, CI failing after an allow, a later check refusing the work.
Then fine-tune the judge on what you've collected.

But who hands out the reward? Not the judge, and not the agent. It has to be a second judge that checks the result after the action, and it can't be argued with. I explored this in an earlier research project, CDV: every step an agent claims is scored twice, once by deterministic checks and once by an LLM critic, and the final score is min(A, B). The stricter judge wins, so a lenient critic can't pay the gut for a bad call. It also Thompson-samples which clarifying question to ask first, a textbook bandit. The honest limitation I published with it: the critic was the same model as the agent. Independence is the part you have to earn.

Signal	How fast it arrives	Safe to learn from live?
Tests fail after an allow	seconds	yes
A human answers an "ask"	minutes	yes
A commit gets reverted	hours to days	in reviewed batches
A customer replies to the wrong email	days	offline only

And where it breaks:

Don't let the learner grade itself. Self-improving agents are having a moment, and the first question to ask any of them is: who grades the runs it learns from? If the agent grades itself, it learns to report progress, not make it.
You can't learn by trying in the red corner. Tests and lint can afford the odd wrong call, and the judge learns from it in seconds. An email, a migration or a payment can't: the lesson arrives after the damage. Those learn only from human decisions, offline, with the judge frozen and versioned between updates.
You only see what you allowed. A denied action never runs, so you never find out whether it was fine. Leave a little room to test that, but only on actions you can undo.
Price the mistakes honestly. If the reward is "the human didn't complain", the judge learns that asking is annoying and drifts toward yes. A wrong yes on something irreversible has to cost far more than an unnecessary ask.
Don't let attackers teach it. The judge reads input the agent helped write (question 5). With online learning, an attacker doesn't just get past the judge once. They train it.

One more thing falls out of this. You can't learn from calls you didn't write down. Keep what the judge was asked, what it said, which version answered and what happened next. That record answers question 4, and it's your training set at the same time.

Intuition is earned

Kahneman and Klein didn't conclude that intuition is bad. They concluded that it's earned, one fast, clear correction at a time. Your agent's gut can earn it too, but only where something tells it when it was wrong, and only if you can look back and check what it said.

What's your agent allowed to do without asking?

Related: Vibe engineering with Opus 5.5? Route by the feedback. The same idea one step earlier: pick the model by what comes back when it acts.