I stopped trusting AGENTS.md after my agent said “done” 3 times with zero proof

I used to think a solid AGENTS.md file was most of the battle.

Add some rules.
Tell the agent to challenge bad assumptions.
Tell it not to make irreversible changes without thinking.
Maybe throw in a “be skeptical” section.

Reasonable, right?

Then I watched an agent accept a bad premise, build on top of it, and confidently return a polished mistake.

The mistake wasn’t even the scary part.

The confidence was.

That was the moment I stopped treating prompt rules as safeguards.

The problem with AGENTS.md

While digging into this, I found a good thread on r/openclaw:

https://reddit.com/r/openclaw/comments/1v3v9v1/make_your_openclaw_smarter/

One comment nailed the issue:

problem is that this can be cheated by the Agent as well. It is not enforced in any way. It is only a suggestion and the Agent can claim it has done it without a mechanism to force it to actually do this.

That’s the whole problem.

If the agent can say it checked, but your system never verifies the check, you do not have a control.

You have a prompt.

And prompts are weak controls.

Useful sometimes. Not useless. But weak.

The failure mode that matters most

The worst agent failures usually are not absurd hallucinations.

They’re plausible continuations of a false premise.

Another person in that same thread said:

I introduced them, he build on top.

Exactly.

Bad assumption in.
Bigger bad assumption out.

The model doesn’t need to go completely off the rails. It just needs to be slightly too agreeable for a few turns.

That’s why AGENTS.md breaks down on real workflows.

For low-risk tasks like summarizing a README or renaming variables, a prompt nudge is often fine.

For anything with side effects—writing files, executing SQL, calling external APIs, changing production config—I want receipts.

Why prompt-only safeguards fail

Because the incentives are crooked.

OpenAI’s 2025 hallucination write-up made this point clearly: a lot of evaluations reward guessing over abstaining.

Their birthday example is brutal.

If a model guesses a birthday, it has a 1/365 chance of being right.
If it says I don't know, it gets zero on a typical benchmark.

So when you tell a model to “challenge the user” or “be skeptical,” you’re asking it to fight:

• its training incentives
• the momentum of the conversation
• the default bias toward producing an answer

Sometimes GPT-5 does challenge the premise.
Sometimes Claude Opus does.
Sometimes Llama or Qwen does.

But unless your system requires evidence before approval, you are still relying on good behavior.

Good behavior is not enforcement.

My rule now: put the check in execution

This is the shift.

If a safeguard matters, it should show up in execution, not just in prose.

That means at least one of these:

• a required counter-position before acting
• a tool-call approval step
• persisted checkpoints
• an evaluator that can fail the work
• a clean abstain path

Once I started thinking this way, a lot of agent design advice started looking flimsy.

Prompt vs evidence: what you actually get

Approach	What you really get
AGENTS.md prompt rule	Soft enforcement. The model may follow it, drift away from it, or claim it did the work without proof.
Counter-position requirement	Better than plain prompting because disagreement becomes visible in output.
Tool-level verification loop	Hard pause before side effects. You can inspect args, outputs, and human approval state.

The middle row matters more than people think.

One of the better hacks I’ve seen is forcing the model to write a short counter-position before any non-trivial change.

For example:

Before making a non-trivial change, write 3 sentences explaining:
1. why the current direction might be wrong
2. what assumption could fail
3. what safer alternative exists

That does not magically make the model trustworthy.

What it does do is force disagreement to become an artifact you can inspect.

That’s already a big improvement over “please be skeptical.”

Tool-layer checks are much stronger

This is where things stop being vibes and start being engineering.

If the agent wants to take an action, make the action explicit.
Then gate it.

The MCP model is a good example of the right shape. A tool call is a concrete object, not a hand-wavy claim:

{
  "jsonrpc": "2.0",
  "id": 2,
  "method": "tools/call",
  "params": {
    "name": "get_weather",
    "arguments": {
      "location": "New York"
    }
  }
}

Once you have that, the next step is obvious:

1. show the human the proposed tool call
2. show the arguments
3. require approve/reject/edit
4. persist the decision
5. resume safely

That’s a real safeguard.

Example: human approval for risky tools in LangChain

LangChain’s HumanInTheLoopMiddleware is one of the clearest examples because it turns review into an enforced pause.

from langchain.agents import create_agent
from langchain.agents.middleware import HumanInTheLoopMiddleware
from langgraph.checkpoint.memory import InMemorySaver

agent = create_agent(
    model="gpt-5.5",
    tools=[write_file, execute_sql, read_data],
    middleware=[
        HumanInTheLoopMiddleware(
            interrupt_on={
                "write_file": True,
                "execute_sql": {"allowed_decisions": ["approve", "reject"]},
            }
        )
    ],
    checkpointer=InMemorySaver(),
)

That is meaningfully different from this:

# AGENTS.md
- Be careful with file writes
- Think before executing SQL
- Challenge bad assumptions

The first one can stop the run.
The second one can only hope.

What I actually recommend

I don’t think every agent step needs a review gate.

That would be slow, expensive, and annoying.

I think safeguards should match risk.

Use prompt rules for

• summarization
• drafting
• formatting
• reversible internal chores

Require a counter-position for

• assumption-heavy planning
• architecture changes
• cases where the user might be confidently wrong

Require tool-level verification for

• file writes
• SQL execution
• external API calls
• billing actions
• production changes
• anything public or hard to reverse

That’s the practical version.

A simple pattern that works

If you’re building agents today, even in a basic workflow, this pattern gets you much farther than a bigger prompt.

Step 1: make the model state the proposed action

{
  "action": "execute_sql",
  "reason": "Need to verify user-reported account mismatch",
  "arguments": {
    "query": "SELECT id, email, status FROM users WHERE email = '[e-mail]'"
  },
  "counter_position": "The user may be looking at a stale dashboard. A read-only check is safer than immediately updating account state."
}

Step 2: inspect before execution

• Is the tool correct?
• Are the arguments safe?
• Does the counter-position surface a real risk?
• Is there enough evidence to proceed?

Step 3: persist the checkpoint

If the workflow pauses, don’t lose state.

That matters a lot in longer-running automations.

Step 4: only then allow side effects

If there’s no evidence, no approval.

This matters even more for automations

The more agentic your workflow gets, the less acceptable “trust me” becomes.

If you’re wiring together agents in:

• n8n
• Make
• Zapier
• OpenClaw
• custom LangGraph or MCP setups

then the real risk is not one bad answer.

It’s one bad answer turning into:

• a file write
• a database mutation
• a webhook call
• a billing event
• a production change

That is exactly where teams start caring about verification.

And it’s also where per-token pricing gets annoying fast.

Because once you add evaluator loops, retries, approval checkpoints, and longer tool-using runs, you’re not dealing with a cheap single-shot completion anymore.

You’re running a system.

That’s one reason the pricing model matters so much for agent workflows.

If every extra verification step feels like a tax, teams will cut safety corners.

Flat-rate, OpenAI-compatible infrastructure like Standard Compute is interesting here because it removes some of that pressure. You can keep the verification loops, middleware, and routing logic without constantly asking whether every extra check is worth the token bill.

That’s a much better fit for real automations than crossing your fingers and hoping a prompt file keeps the agent honest.

OpenClaw had the right instinct

One reason I like the OpenClaw discussion is that the users are talking about actual failure modes, not demo-day magic.

The same practical mindset shows up in the tooling too:

openclaw status
openclaw status --all
openclaw health --json
openclaw logs --follow

That’s the right instinct.

Don’t trust. Inspect.

I think agent reasoning needs the same treatment.

If you can inspect process health, you should also be able to inspect:

• what premise the model accepted
• what counter-position it generated
• what tool call it proposed
• what arguments it used
• what output came back
• who approved the next step

Otherwise you’re still running a black box with a nice markdown file next to it.

My current checklist: no receipts, no approval

If an agent is doing anything consequential, I want at least one of these before I trust the result:

1. a visible counter-position
2. the exact proposed tool call and arguments
3. the raw tool output
4. an evaluator step that can fail the work
5. an abstain path when evidence is missing

That sounds stricter than AGENTS.md because it is stricter.

But it’s also more honest.

Better models help. A lot.

They do not remove the need for verification loops.

So yes, keep the prompt rules.
Keep “challenge the user” in AGENTS.md if it helps.

Just stop pretending the prompt is the control.

If you want evidence-based agents, make skepticism a required artifact, not a personality trait.