DEV Community

Sho Naka
Sho Naka

Posted on • Originally published at note.com

Stop Telling Your AI to "Be Careful Next Time." It Has No Memory of Yesterday.

#ai #agentic #productivity #contextengineering

This is an adapted English version of an article I first wrote in Japanese. I work with AI to shape and review my drafts, but the argument and the field observations are my own. The numbers are cited from public surveys (linked at the end).

I built an aggressive prompt-injection block to stop my AI agent from repeating the same mistakes. It worked, so I kept adding rules. By the time I noticed, the file had ballooned to 56,000 characters — and the agent had quietly stopped functioning. Too much context, attention spread too thin to act on any of it. I gutted it back to under 1,200 characters, and here's the part that still stings: it behaved better with fewer rules.

That was the day I learned my whole mental model was backwards.

This isn't a post about making your AI more accurate. It's about designing so that accuracy stops being the thing you depend on.

The mistake I made for months

My agent kept skipping the same step in a workflow. So I did what every engineer does on instinct: I added a rule. "Don't skip this step." Then it did something else dumb, so I added another rule. Then another.

I was treating the rules file like a conversation with a colleague — as if the agent would remember yesterday's correction and carry it forward. It doesn't. Every run starts cold. "Be careful next time" assumes a next time that shares state with this time. For a stateless model, there is no continuity to appeal to. You are talking to a counterparty with no memory of the conversation you think you're having.

So the rules pile up, because each correction feels like progress. And for a while the numbers even improve. But adding rules has a ceiling, and I blew straight through it: at 56,000 characters the agent wasn't reasoning over my guardrails anymore — it was drowning in them.

Knowing a rule and stopping at it are different things

Here's the distinction that took me far too long to see.

Putting a rule in the context window means the model knows the rule. It does not mean the model will stop at that rule in the middle of a workflow, when it's three tool-calls deep and optimizing for finishing the task. Knowing and stopping are two different operations. The first is retrieval. The second is control flow.

Text in a prompt is a soft suggestion competing for attention with everything else in the window. If you actually need the agent to halt, you don't write a more emphatic sentence. You make the constraint physical — a hook, a gate, a check that runs outside the model's discretion. The model can't talk its way past a script that refuses to proceed.

This is the part of context engineering nobody puts on the slides: half the job is deciding what does not belong in the context window, and moving those guarantees into code instead.

Why I stopped chasing accuracy

It's worth saying plainly how thin the "just make it more accurate" bet really is.

JUAS's 2025 survey found only 4% of companies said generative AI "greatly exceeded expectations." MIT's 2025 GenAI Divide report put the share of enterprise AI pilots that actually reached production at around 5%. And Persol's 2026 data showed only 25.4% of workers saw their working hours drop because of AI — with the heaviest AI users often logging more overtime, not less.

Read those together and a pattern falls out. The teams that win aren't the ones whose model is a few points more accurate. The accuracy is roughly the same for everyone. The difference is structural: whether the work around the AI is designed so that imperfect output is still safe to use.

That reframes the whole problem. Stop asking "how do I make it get this right every time?" Start asking "what do I do when it's wrong — and can I make that cheap?"

What designing-around-it actually looks like

A few patterns I now reach for instead of writing another rule.

Make disagreement the signal. A team I watched (estie) runs the same task through multiple models — GPT, Claude, Gemini — and routes only the items where the models disagree to a human. Where they agree, ship it. You're not trusting any single model to be right; you're using cheap redundancy to localize where human attention is worth spending. Accuracy stopped being the dependency.

Write escalation criteria, not encouragement. Instead of "be careful," I give the agent explicit hand-back conditions in the system prompt:

  • Can't solve it in two minutes → change approach
  • Stuck for fifteen minutes → hand back to a human
  • Change touches scope or design → always hand back
  • Everything else → proceed autonomously

These aren't morale. They're a decision boundary the agent can actually evaluate, with a defined exit instead of a vague aspiration to do well.

Give the agent a fixed position in the workflow. The deployments still alive three months in all share one trait: the AI owns a defined seat, not a vague mandate to "help." It drafts the meeting minutes. It handles first-line customer support. It assembles the deck. A narrow seat means a wrong answer has a small, known blast radius — and a small blast radius is what makes the thing usable at all.

The part I didn't expect

When I cut the rules file and built the real constraints into hooks, the agent didn't just work again — I started trusting it more. Counterintuitive, but it tracks. Once I could see exactly where it would stop, I could finally let it run.

That's the actual function of a guardrail. It isn't there to slow you down; it's there so you can go faster without watching the edge. The careful people — the ones who were most skeptical of handing work to an AI — are the ones who start using it heavily once the boundaries are visible and physical.

You will not lecture your AI into remembering. It has no yesterday. So stop spending your effort on the sentence, and spend it on the structure: build the place where it's allowed to stop, and then stop telling it to be careful.

References

Originally published in Japanese on note. I write under "nomuraya / 中翔 (Naka Sho)" — the same person across media. This English version is adapted, not literally translated.

Top comments (0)