In Ten Minutes, Test Whether Claude Reasons Or Just Remembers

Originally published at vinpatel.com

In under fifteen minutes you can find out whether your AI agent actually reasons through a problem or just has more room to hold information than you do.

That distinction is the whole argument in a essay making the rounds this week: AI models have access to a vastly larger working memory than the human brain, and that gap gets mistaken for intelligence constantly. A bigger context window lets a model hold an entire codebase, a whole legal brief, or a full customer history in view at once. That is not the same as reasoning through it. It is closer to a much bigger sticky note.

Here is how you check which one you actually shipped.

1. Pick a task your agent already handles well in production today.
2. Strip every bit of injected context: no RAG, no retrieved documents, no prior chat history. Bare instructions only.
3. Give it a variant of the task it has never seen, one with no lookup shortcut available.
4. Run both versions and save the full reasoning traces, not just the final answers.
5. Compare accuracy and the shape of the reasoning between the context-fed run and the cold run.
6. If accuracy collapses without the extra context, you built a lookup engine wearing a chain-of-thought costume. If it holds steady, you're closer to something that actually reasons.

Paste this as your test harness prompt:

TASK: [describe the task]
CONSTRAINT: Solve using only the instructions below. Do not assume access to any external document, prior conversation, or retrieved context.
INSTRUCTIONS: [bare task instructions, no injected data]
Output your full reasoning trace, then your final answer.

The gotcha: a model that looks brilliant in production because it's stuffed with retrieved context can fail this test badly, and you won't notice until the day your retrieval pipeline breaks or returns the wrong document. Bigger working memory buys recall, not judgment. Confusing the two is how teams end up shipping an agent that quietly can't handle a genuinely novel input, because it was never reasoning in the first place, it was just remembering harder.

If you're building agentic pipelines where this distinction actually costs money, the breakdown on orchestrating full agentic workflows is worth the read. For a longer look at where AI benchmarks quietly conflate memory with reasoning, the Claude Opus 4.8 piece digs into the same failure mode from a different angle.

One AI signal a day. 90 seconds. No fluff. Get the next one in your inbox — subscribe at vinpatel.com/subscribe/.