Claude Is Bad at Searching Its Own Memory

I had what sounded like a simple request: ask Claude Code questions about work I had already done with Claude Code.

The answers were all somewhere on disk. Claude Code stores local transcripts as JSONL files, and I had weeks of them: debugging sessions, project decisions, one-off investigations, cost summaries, configuration changes, and the usual trail of half-remembered engineering context.

In theory, this should be easy.

Give the agent access to the files. Let it search. It can grep. It can parse JSON. It can inspect timestamps. It can reason over long context. If anything should be good at answering questions from a pile of text, it should be a coding agent with shell tools.

In practice, this is exactly the kind of task where a capable agent becomes slow, expensive, and weirdly fragile.

The problem is not that Claude cannot search. The problem is that raw search is the wrong abstraction.

I already had another abstraction available. I had built claudescope: a local tool that indexes coding-agent sessions and exposes them through search, session views, analytics, and an MCP server.

The practical question was whether that MCP surface was actually worth exposing to Claude. Was it meaningfully better than simply giving the agent raw transcript files and letting it use normal shell tools?

So I ran a small benchmark on a very specific use case: answering questions about my own AI coding-agent history.

The result was blunt:

• Raw file search worked, but it was the worst option.
• An indexed retrieval interface was about 2.8x faster at the median.
• It was about 2.3x cheaper at the median.
• It was also more accurate, especially when the answer required stitching together multiple sessions.

The benchmark did not show that Claude needs more search tools.

It showed that Claude needs less raw search work.

The Use Case

I use coding agents heavily. Over time, the transcripts become a real knowledge base.

They contain answers to questions like:

• Why did we make this project decision?
• What was the root cause of that weird failure last month?
• Which session contained the fix?
• How much did a project cost over a given week?
• What did we conclude after comparing several tools?
• Did we already investigate this bug?

This is not hypothetical “memory” in the vague chatbot sense. It is concrete operational memory: local transcripts, timestamps, working directories, tool calls, costs, and final answers.

The naive solution is obvious: point Claude at the transcript files and ask it to search.

Claude Code transcripts live as JSONL. A coding agent can use rg, jq, sed, Python, SQLite, or whatever else it decides. For a human developer, this sounds like a reasonable interface. The data is there. The tools are familiar.

But from the agent’s perspective, every question now includes a tax:

That is a lot of work before the agent has even started answering the question.

I wanted to know how bad that tax was, and whether claudescope’s indexed MCP interface removed enough of it to justify being a separate tool surface.

The Experiment

I built a question bank from my real Claude Code history: 41 questions with verified reference answers and evidence.

The corpus covered three weeks of local Claude Code sessions, from June 15 to July 6, 2026. The questions were intentionally practical. Some were simple lookups. Others required reading a session. Others required aggregating cost or token usage across days. Others required remembering decisions spread across multiple conversations.

Then I ran the same questions through different tool setups. The main comparison was raw transcript access versus the claudescope MCP server; the hybrid and skill arms were there to answer follow-up questions about agent choice and interface mechanics.

• Raw files The agent could use normal file and shell tools over local transcripts.
• MCP The agent could use claudescope, the indexed retrieval system I had already built, exposed through MCP tools.
• Hybrid The agent had both MCP and raw file tools, so it could choose.
• Skill + CLI A later fourth arm used the same claudescope index, but through a skill that taught the agent a CLI workflow instead of MCP tool calls.

Each question ran three times per arm. Answers were graded blindly by a separate judge model against the reference answers.

In total: 492 measured runs and 492 blind judgings.

The Headline Result

The raw approach was not a disaster. Claude often found the answer.

But it was consistently worse.

Compared with MCP, raw files were:

• about 2.8x slower at the median
• about 2.3x more expensive at the median
• less accurate overall
• less predictable, including the only runs that hit the turn cap

That last point matters. Averages are not the whole story. When an agent is searching raw transcripts, the worst cases become ugly: broad searches, huge result sets, repeated parsing, false leads, and more turns spent deciding whether it has enough evidence.

An index gives the agent a bounded path. Search the index, open the relevant session window, ask for analytics if the question is numeric, answer with evidence.

Raw search makes the agent rebuild that path from scratch.

Where Raw Search Actually Fails

The interesting part is not simple lookup.

If the question is “find the session where I said X,” raw search can be fine. Sometimes it is even competitive. Grep is good. Literal text search works when the answer contains a rare phrase.

The gap appears when the answer is not one string in one file.

• For aggregate questions, raw files scored 63%. MCP scored 90%.
• For multi-session questions, raw files scored 65%. MCP scored 75%.

That matches the failure mode I saw while inspecting the runs. The agent was not stupid. It was doing reasonable things. But it had to infer too much structure from too little interface.

An aggregate question is not just search. It needs semantics:

• Which sessions count?
• Which project name maps to which working directory?
• Are resumed sessions duplicated?
• Are costs attached to messages, requests, or sessions?
• What date range is inclusive?
• Are we grouping by day, project, or agent?

Those are database questions pretending to be file-search questions.

The raw-file agent had to solve the database problem over and over again using generic tools. The indexed agent could call the database-shaped interface directly.

That is the whole lesson.

Hybrid Was a Useful Sanity Check

The hybrid arm mattered because it tested whether the index was actually useful to the agent when raw access was still available.

If raw files were just as good, or if the agent preferred them, hybrid would show that.

It did not.

Given both options, agents used MCP exclusively in 72% of hybrid runs.

That is a strong behavioral signal. When the indexed tool was available, Claude mostly stopped doing expensive archaeology in the transcript files.

Hybrid matched MCP accuracy at 88%, but cost more: $0.16 median instead of $0.12. That makes sense. Giving the agent more options sometimes means it spends extra turns exploring those options.

The best interface was not “everything.”

The best interface was the right abstraction.

MCP vs Skill Was the Smaller Question

After the main study, I tested one more thing.

Maybe MCP was not special. Maybe the value came from the underlying index, and any decent interface to that index would work.

So I added a fourth arm: a skill that taught the agent to use the claudescope CLI. The CLI talked to the same index and exposed the same kind of retrieval workflow, but the agent accessed it through shell commands instead of typed MCP tool calls.

The result:

• MCP: 88% accuracy, $0.12 median cost, 30s median latency
• Skill + CLI: 87% accuracy, $0.20 median cost, 34s median latency

That is the cleanest conclusion in the whole experiment:The index is the value. MCP is the efficiency layer.

The skill approach worked. It almost matched MCP accuracy. If you already have a good CLI and do not want to build an MCP server, a skill can be a perfectly reasonable way to teach an agent how to use it.

But it costs more. The agent has to load the skill, make shell calls, read human-formatted CLI output, and sometimes post-process that output. MCP gives it a more direct, typed interface.

There was one interesting advantage to the CLI path: oversized CLI outputs could persist to files, and the agent could then grep those files. That is a real affordance MCP results did not have in this setup. But for normal high-volume retrieval, MCP was cleaner and cheaper.

Why This Matters

Coding-agent history is becoming project memory.

Not because anyone designed it that way, but because the transcripts are where the work happened. The bug investigation is there. The decision rationale is there. The failed attempts are there. The exact command output is there. The cost and token metadata are there.

If you use agents seriously, eventually you ask: “What did we do last time?”

And the tempting answer is: “Just let the agent search the logs.”

This benchmark is my argument against that. Raw logs are a storage format. They are not a memory interface.

When you expose raw transcripts to an agent, you are asking it to spend reasoning tokens on plumbing:

• locating
• parsing
• deduplicating
• joining
• windowing
• aggregating
• validating

That work is not free. It shows up as latency. It shows up as cost. It shows up as missed answers. It shows up as fragile runs that depend on whether the agent guessed the right search term.

An index moves that work out of the model loop.

That is why the indexed approaches won. More specifically, that is why exposing claudescope through MCP made sense: it turned a transcript-mining problem into a few bounded retrieval calls.

The Practical Takeaway

If you want coding agents to use history, do not start by giving them a bigger pile of files. Start by giving them a retrieval layer.

For my use case, the right shape was:

• full-text search over transcripts
• session listing by project/date/agent
• “open around this hit” session windows
• analytics for cost/token/date aggregation
• stable IDs and metadata
• enough evidence in the result for the agent to cite its path

Once that existed, the agent became faster and cheaper because it no longer had to reinvent it.

For claudescope, the answer was yes: exposing the index as MCP was worth it. MCP was the best interface I tested for repeated use. A skill over a CLI was a good lightweight alternative. Raw files were useful as a fallback, but not as the main path.

The punchline is simple:

Claude can search its own memory. But if you make it grep raw chat logs every time, you are wasting the model on work your tooling should have already done.