Can AI Trust Its Gut?

Five questions to ask before a fast model decides what your agent is allowed to do.

Table of contents

• The agent that doesn't ask anymore
• "Give your agent a System 1"
• Firefighters and stock pickers
• Five questions before you trust the judge
• "Just use a bigger judge"
• So what's the worst that can happen?
• How not to fix it
• Trust a fast call only where the world corrects it
• Can the gut learn?
• Intuition is earned

The agent that doesn't ask anymore

Agents don't just write code now. Point one at your website and it'll find prospects, write the emails, send them and book the calls. Point one at your repo and it'll run the migration.

Nobody wants to click "approve" on every step, so a pattern is spreading: before each action, a small, fast model gets asked one question. Is this OK? It answers in milliseconds, with a confidence score, and the agent carries on.

It's a good idea. It's also one more thing you're trusting, and most setups never ask how far.

"Give your agent a System 1"

The idea borrows from Daniel Kahneman. System 2 is slow, careful reasoning. System 1 is the fast, automatic call: the gut. Your big model is the slow thinker, and the little judge in front of every tool call is the gut.

Fast, cheap, always on. What could go wrong?

Firefighters and stock pickers

Kahneman spent a career showing where gut calls fail. Gary Klein spent one showing where they work: fire commanders who knew, without being able to say why, to get their crew out of a building moments before the floor gave way.

In 2009 the two wrote a paper together to settle the argument, and ended up agreeing. A gut call deserves trust when two things hold:

1. the environment is regular enough to be predictable, and
2. there's a real chance to learn it: lots of practice, with feedback that comes back fast and clear.

A firefighter gets both. A stock picker gets neither: the market is noisy and the feedback is slow and ambiguous, so confidence grows while skill doesn't.

Now look at the judge in front of your agent. It says "yes" to an email. Does it ever find out whether that email was a mistake? It says "yes" to a migration. Does anything tell it the migration broke production at 3 a.m.?

A judge that never learns whether its "yes" turned out fine is a stock picker.

Confident, fast, and with nothing correcting it.

Five questions before you trust the judge

None of these is about a particular product. They're true of any model that sits in front of an agent's actions.

1. Where does the question go? To judge an action, the judge has to see it: the command, the file path, the email body, the customer's name. If the judge is hosted, all of that leaves your machine on every call. That may be fine. It should be a decision, not a default.

2. Is "97% sure" a probability? A confidence score is only a probability if someone checked it against outcomes: of all the calls it was 97% sure about, were about 97% right? If nobody measured that on your kind of actions, it's a vibe with decimals.

3. What happens when it isn't sure? Every judge has an unsure zone. Does unsure mean allow, deny, or ask a human? The worst answer is the one nobody chose: a default buried in a config file.

4. Can you check a "yes" tomorrow? Something goes wrong. Can you show what the judge was asked, what it answered, and which rules it was running at the time? Or is the answer "it said OK, apparently"?

5. Who else is talking to the judge? The judge reads the action, and the action was written by the agent, which may have just read a web page, an email or an issue someone planted. The input to your safety check is partly written by whoever wants to get past it.

"Just use a bigger judge"

The obvious fix is a smarter model in front. But a bigger judge is slower, so it gets skipped in exactly the places you run the most actions: the fast loops. It still hasn't been measured on your actions. And it still leaves you nothing to check tomorrow.

Thinking longer doesn't create feedback.

So what's the worst that can happen?

Play it forward. An outreach agent drafts 200 emails with the wrong price in them. The judge sees 200 well-formed, polite emails to real prospects and is 96% sure each one is fine. They go out.

You can't unsend an email.

Nobody did anything wrong. The judge answered the question it was asked. It was just asked to use its gut in a place where its gut had never been corrected.

How not to fix it

• Turn it off because it's slow. Now there's no judge, just faster mistakes.
• Trust the accuracy number on the landing page. It was measured on someone else's actions, not yours.
• Add a line to the system prompt. Rules get weighed against the goal. Checks don't.

Trust a fast call only where the world corrects it

Sort your agent's actions by two questions: will I find out fast if this was wrong? and can I undo it?

• Run the tests, lint, read a file: the world corrects you in seconds, and nothing's lost. Let the gut call it.
• Send an email, run a migration, move money, delete anything: no quick correction, no undo. That's not a gut call. Stop and ask a human, or check it with something slower.

The rule:

Trust a fast call only where the world corrects it.

Can the gut learn?

Everything above points at one missing piece: feedback. That's exactly what reinforcement learning is for. The judge makes a call, the world answers, and the answer becomes a reward the judge learns from. Done carefully, it turns a stock picker into a firefighter.

Cheapest first:

1. Calibrate before you learn. Check whether "97% sure" is right about 97% of the time on real outcomes, and adjust the scores until it is. No RL needed, and it fixes question 2 on its own.
2. Treat each call as a bandit, not a game. A gate decision is one move (allow, ask or deny) with one outcome. A contextual bandit, the simplest kind of RL, fits that shape better than full RL, and it's far cheaper to run.
3. Every "ask" is a free label. When the judge isn't sure and asks a human, the human's answer is training data. So are the quieter signals: a commit reverted after an allow, CI failing after an allow, a later check refusing the work.
4. Then fine-tune the judge on what you've collected.

But who hands out the reward? Not the judge, and not the agent. It has to be a second judge that checks the result after the action, and it can't be argued with. I explored this in an earlier research project, CDV: every step an agent claims is scored twice, once by deterministic checks and once by an LLM critic, and the final score is min(A, B). The stricter judge wins, so a lenient critic can't pay the gut for a bad call. It also Thompson-samples which clarifying question to ask first, a textbook bandit. The honest limitation I published with it: the critic was the same model as the agent. Independence is the part you have to earn.

Signal	How fast it arrives	Safe to learn from live?
Tests fail after an allow	seconds	yes
A human answers an "ask"	minutes	yes
A commit gets reverted	hours to days	in reviewed batches
A customer replies to the wrong email	days	offline only

And where it breaks:

• Don't let the learner grade itself. Self-improving agents are having a moment, and the first question to ask any of them is: who grades the runs it learns from? If the agent grades itself, it learns to report progress, not make it.
• You can't learn by trying in the red corner. Tests and lint can afford the odd wrong call, and the judge learns from it in seconds. An email, a migration or a payment can't: the lesson arrives after the damage. Those learn only from human decisions, offline, with the judge frozen and versioned between updates.
• You only see what you allowed. A denied action never runs, so you never find out whether it was fine. Leave a little room to test that, but only on actions you can undo.
• Price the mistakes honestly. If the reward is "the human didn't complain", the judge learns that asking is annoying and drifts toward yes. A wrong yes on something irreversible has to cost far more than an unnecessary ask.
• Don't let attackers teach it. The judge reads input the agent helped write (question 5). With online learning, an attacker doesn't just get past the judge once. They train it.

One more thing falls out of this. You can't learn from calls you didn't write down. Keep what the judge was asked, what it said, which version answered and what happened next. That record answers question 4, and it's your training set at the same time.

Intuition is earned

Kahneman and Klein didn't conclude that intuition is bad. They concluded that it's earned, one fast, clear correction at a time. Your agent's gut can earn it too, but only where something tells it when it was wrong, and only if you can look back and check what it said.

What's your agent allowed to do without asking?

---

Related: Vibe engineering with Opus 5.5? Route by the feedback. The same idea one step earlier: pick the model by what comes back when it acts.