What I Learned Building a RAG Pipeline for 1000 Page Books
A few weeks ago I decided to build a RAG pipeline for books. Not blog posts, not ten page PDFs, actual books, the kind that run 500 to 1000 pages. I knew big players already existed in this space, Google's NotebookLM being the obvious one. That wasn't really the point for me. I wasn't trying to invent something new or compete with anyone. I just wanted to sit down and actually understand how large scale vector embeddings behave when you throw a real book at them instead of a toy dataset.
Here's how that went, and where it eventually led me.
The Vector Search Approach
I started simple. I chunked the book, generated vector embeddings using NVIDIA's nemotron 3 embed 1b model, and stored everything in Qdrant Cloud. For most reasoning tasks this worked beautifully. Ask it a question about the author's argument, ask it to find where a certain idea shows up, ask it to compare two sections, and cosine similarity did exactly what it was supposed to do. Retrieval was fast and the answers were accurate. I felt like I had this figured out.
Then I asked it to summarize the whole book, and things fell apart.
Not occasionally either. A lot. Entire sections were missing from the summary, the narrative flow felt broken, and details that clearly mattered were just gone. To make sure this wasn't me being sloppy with my own evaluation, I tested it against something I know intimately, Clean Code by Robert C. Martin. I've read that book closely enough to spot exactly where a summary is lying to me. And it was lying constantly.
So I sat with an uncomfortable question. Was the vector store actually helping with summarization at all?
The honest answer was no. And once I started looking at how production RAG tools handle summarization, I saw the same pattern everywhere. Retrieve some random twenty to forty chunks that score highest on similarity, hand them to the LLM, ask it to summarize. But that isn't summarization. That's answering a vague query using whatever chunks happen to sit closest to it in vector space. A real summary needs everything, not the thirty most relevant looking fragments.
Maybe I Don't Need a Vector Database
So I changed the approach entirely. If summarization wasn't a retrieval problem, I figured I should stop treating it like one. The new plan was to extract the raw text straight from the PDF, pass it directly to the LLM, get a summary back, and store that summary in a normal, non vector database. From then on, whenever the agent needed to summarize the book, it wouldn't touch cosine similarity or retrieval at all. It would just pull the pre generated summary and reason over that. As a nice side effect, this also made personalization easy. Instead of returning the same canned summary to every single user, the LLM could restyle the stored summary based on someone's preference at serve time.
It sounded clean on paper. It broke in exactly the same place.
It Was Never the Token Limit
My first assumption was that I was hitting an input token limit. That seemed reasonable, a 500 page book is a lot of text after all. But modern models have enormous context windows now, so that theory didn't survive long. I fed the model well within its limits and it still lost details in exactly the same way.
The real culprit had an actual name, one I hadn't taken seriously enough until I ran into it myself. Lost in the Middle. Long context models can still show a noticeable drop in performance when the relevant information sits in the middle of a long context, even when the entire input technically fits within the context window. It isn't that every model behaves identically, or that the beginning and end are always handled perfectly. It's that the middle is where things are most likely to go wrong, and a full length book is mostly middle.
Once I saw that pattern named and described, everything clicked. My summaries were accurate about the opening chapters, accurate about the ending, and vague or flat out wrong about the two hundred pages sitting in between. That's exactly the shape you'd expect if the model is attending strongly to the edges of its context and weakly to the center.
My first instinct, probably like most people's first instinct, was to fight it with a better prompt. I tried telling the model to pay closer attention to the middle sections, to not neglect the details from chapters five through fifteen. None of it worked. You can't prompt engineer your way out of a structural attention bias. If your fix for Lost in the Middle is a stronger system prompt, you're already stuck in the same loop I was.
MapReduce to the Rescue
The actual fix wasn't a smarter prompt at all. It was accepting a fairly humbling idea, that you should never ask an LLM to summarize an entire 300 page book in a single pass, no matter how large its context window claims to be. Instead, you break the problem down recursively. Summarize small pieces first, then summarize the summaries. This is the idea behind MapReduceDocumentChain, and its close cousin TreeSummarize. Since I was already comfortable with LangChain, that's the direction I went.
The two aren't quite the same thing, even though people tend to use the names interchangeably. MapReduce, at its simplest, means summarizing each chunk independently, then combining all of those summaries in one final pass. TreeSummarize goes a step further. When the combined summaries are still too large to fit in a single context window, it recursively groups and summarizes them again, and again, until what's left fits into one final pass. In practice, for a 300 to 1000 page book, you almost always end up needing the tree version, since a single flat reduce step over dozens of chunk summaries can itself get long enough to reintroduce the same middle of context problem you were trying to escape.
Here's roughly what the pipeline looks like end to end.
For the actual summarization work at each step, I used NVIDIA's nemotron 3.5 lightning 30b a3b model. Model choice does matter here, since how badly a model suffers from Lost in the Middle isn't the same across the board, some handle long context more gracefully than others. But the map reduce restructuring fixes the underlying problem regardless of which model sits beneath it, because it sidesteps the issue structurally rather than depending on the model's raw long context ability.
So here's how the flow actually works, told simply.
First, the book gets split into small, manageable chunks, small enough that each one comfortably fits inside the model's "good attention" zone. There's no middle of context degradation here, because each chunk basically is the entire context for that particular call.
Second comes the map step. Each chunk gets sent to the LLM independently, in parallel, with a simple instruction along the lines of "summarize this section." Because every call only has to reason over one small chunk, there's no middle to lose anything in. The model sees the entire chunk clearly, from edge to edge.
Third comes the reduce step. All of those individual chunk summaries get collected and combined. If the combined summaries are still too long to fit inside a single context window, the process recurses, the summaries get grouped and summarized again, and that repeats until everything finally fits into one pass. This is essentially what TreeSummarize formalizes, you're literally building a tree, where the leaves are chunk level summaries and each level above compresses further, until you land on one root summary at the top.
Finally comes the synthesis step. That last reduce pass takes the now much smaller set of intermediate summaries and produces the final, coherent summary of the whole book.
What I like about this approach is how the "lost in the middle" problem quietly disappears. No single LLM call ever has to hold the entire book's middle in its context at once. Every call only ever deals with a small, fully attended slice of text. The tradeoff is more LLM calls, more cost, and more latency compared to a single shot summary, but you get correctness in return, and for a 500 to 1000 page book, correctness was the entire point of the exercise.
Did It Actually Get Better
I didn't run a formal benchmark here, this was a personal project, not a paper. But I did what I'd already been doing since the start, checking the output against Clean Code chapter by chapter, since I know that book well enough to catch what's missing.
The single pass summary, whether it came straight from the LLM or from the top k retrieved chunks, consistently skipped the same kind of material. Chapters covering formatting, boundaries, and error handling, the ones sitting comfortably in the middle of the book, were thin or entirely absent, while the opening chapters on naming and functions and the closing chapters on classes and systems came through clearly every time. The map reduce version was the first one that actually surfaced content from every chapter, including the ones buried in the middle. It wasn't flawless, some nuance still gets flattened when you compress twice, but it stopped skipping entire chapters, which the earlier approaches did every single time.
That, more than anything, is what convinced me this wasn't a prompting problem to begin with.
What I Learned
Looking back, a few things stand out to me now that didn't at the start. Retrieval and summarization are genuinely different problems. Vector similarity search is built to answer "find me the relevant bits," not "condense the entire thing." Reaching for a vector database just because it's already sitting in your stack isn't always the right call. A bigger context window doesn't save you from Lost in the Middle either, since it's an attention behavior, not a capacity limit. And you cannot prompt your way around a structural limitation in how a model reads long text. If the fix you keep reaching for is a stronger instruction, you're probably solving the problem at the wrong layer entirely.
The biggest lesson here wasn't that RAG is bad, or that vector databases are unnecessary, or that big context windows aren't useful. It was that context size and useful reasoning capacity are not the same thing. A model being able to accept 500 pages doesn't mean it can reason over all 500 pages equally well. Sometimes the best way to give a model more information is to hand it less information at a time, and let it build the full picture in stages instead of all at once.
Top comments (0)