If you have run Claude Code against a real document corpus, you have probably watched it go from snappy to sluggish as the file count climbs. Ten files feel instant. A few hundred PDFs, and the same query takes minutes, your token bill spikes, and occasionally the answer is confidently wrong.
This post is about why that happens and how to make Claude Code faster on large document sets. The short version is that the bottleneck is usually the search strategy, not the model. Fix the retrieval pattern and the speed, cost, and reliability problems mostly go away together.
The problem: direct file search does not scale
By default, Claude Code does document search by reading files directly. Ask a question and the agent opens files, scans them, and reasons over what it finds. That works beautifully for a small project because the agent can hold the whole thing in context.
The trouble is that there is no index. Nothing tells the agent where an answer lives, so to stay thorough it has to look across more and more files as the corpus grows. Three problems show up at once. Latency climbs because the model reads far more text than the question needs. Cost climbs in lockstep, since every scanned document is input tokens you pay for, relevant or not. And reliability drops, because when the answer is not actually present, a model told to scan everything tends to fabricate something plausible rather than return a clean "not found."
The core issue is that the work scales with the size of your library instead of the difficulty of your question. That is the wrong scaling curve for anything document-heavy.
The fix: retrieve first, reason second
The standard solution is retrieval augmented generation, or RAG. Instead of asking the model to find and reason in a single pass, you split the job. A dedicated retrieval layer searches a prebuilt index and returns the handful of passages most likely to contain the answer. Those chunks, with their sources, are then handed to Claude Code, which reasons over that small, focused set and produces a grounded answer. In plain terms, the flow goes from the user query, to the RAG retrieval layer, to the relevant document chunks, to Claude Code reasoning, to a grounded answer.
Each component does what it is good at. Vector search is fast and cheap at finding relevant text in a large corpus, and Claude is strong at reasoning over a focused set of facts. Direct file search forces the model to do both, including the part it is slowest at, which is locating the needle in the haystack.
The key behavioral change is that retrieval cost is roughly constant. Whether you have fifty documents or fifty thousand, the retriever returns a small set of chunks, so Claude reasons over about the same amount of text every time. Latency and cost flatten out instead of growing with the corpus.
Connecting a private RAG layer through MCP
The clean way to wire this into Claude Code is the Model Context Protocol, or MCP. MCP lets Claude Code call an external tool and get structured context back, so a retrieval system can be exposed as an MCP server and behave like any other tool in the agent loop.
A private RAG layer over MCP usually handles three jobs. It indexes your documents once, chunking and embedding them ahead of time rather than rescanning on every query. It retrieves selectively, returning only the most relevant chunks for each question along with their sources. And it keeps your data contained, with the index living in an environment you control. That last point is the operative one for enterprise teams. The retrieval layer is yours, the data does not leak into ad hoc scans, and you can apply your own access controls.
What the benchmark shows
It helps to attach numbers to the claim. CustomGPT.ai ran a controlled test of Claude Code on a 500-PDF workflow, measuring response time, cost, and completion rate as the document count scaled. With a private RAG layer in front of Claude Code, the result was 4.2x faster and 3.2x cheaper, and average response time fell from 2 minutes 31 seconds to 36 seconds. The reliability gap widened with scale as well. Without retrieval, a large share of searches failed to return within a reasonable window, while with it, completion stayed consistent. The methodology and raw data are in their Claude benchmark.
The point is not the specific tool. It is that the retrieval pattern, not the model, is what moves the numbers.
Direct file search vs. private RAG
The trade-off comes down to where the work happens. Direct file search needs no setup and always reflects the live state of your files, but its latency grows with corpus size, its cost per query rises as more files are scanned, and its grounding weakens as the haystack grows. A private RAG layer over MCP needs an upfront ingest and indexing step and has to be reindexed when documents change, but in exchange its latency stays roughly flat as the library grows, its cost per query stays low and stable, and its answers stay anchored to retrieved sources inside an access-controlled index.
Put simply, one pattern scales with library size and the other scales with question difficulty.
When direct file search is enough
Do not add RAG where it is not warranted, since the index is one more thing to maintain. Direct file search is the right call when the document set is small, on the order of a handful to a few dozen files. It is also the better choice when files change constantly and you want the agent reasoning over the live working set, or when you are doing quick, exploratory work where any ingest step would just slow you down.
When private RAG is the better pattern
Reach for a private RAG layer when the shape of the problem changes. That usually means the corpus is large or growing, the same knowledge base is queried repeatedly, cost per question matters at volume, or accuracy and data privacy are non-negotiable so a fabricated answer is unacceptable. A practical rule of thumb is that once you are past a few dozen files and querying them often, retrieval stops being optional.
Implementation checklist
If you decide to add a retrieval layer, a minimal path looks like this. Start by inventorying the corpus, counting documents, formats, and how often they change, since that tells you whether RAG is justified at all. Choose a chunking strategy that follows semantic boundaries rather than arbitrary fixed sizes, and keep source metadata on every chunk. Build the vector index once, ahead of query time. Expose that retrieval as an MCP server so Claude Code can call it as a tool and receive the top matching chunks with sources. Constrain the prompt so Claude answers only from retrieved chunks and returns "not found" when the corpus genuinely lacks the answer. Measure response time, cost per query, and completion rate before and after, so you can prove the win rather than assume it. Finally, plan how and when the index refreshes as documents change.
For a step-by-step version with the benchmark in context, this walkthrough on how to make Claude Code faster when searching documents is a useful reference.
Developer takeaway
Making Claude Code faster on large document sets is an architecture choice, not a model choice. Start with direct file search for small, fast-moving work. Watch latency and cost as the corpus grows, and the moment that curve turns against you, put a private RAG layer in front of Claude Code through MCP. Index once, retrieve selectively, and let the model reason over only the passages that matter.
Top comments (0)