Reading a 40-page spec with an LLM without getting burned

The summary that almost cost me a week

I had a 40-page integration spec to implement against. A dense PDF, the kind of document where the one sentence that matters is buried in a subclause on page 31. I did what everyone does. I dropped it into a model and asked for a summary. The summary was good. Clean bullets, sensible structure, and it read like I now understood the document.

I did not understand the document. I understood the summary, which is a different and more dangerous thing. The summary was built to sound complete, and completeness is exactly the property it could not guarantee.

I have a workflow now that has stopped burning me. It is not clever. It is mostly about refusing to trust the convenient output.

Paste the actual text

First rule, and it is dumb but people skip it. Paste the real text. Not a description of the document, not "it's the standard webhook spec", not a link the model may or may not fetch faithfully. The actual characters. If the model is working from your paraphrase, it is answering about your paraphrase, and it will happily fill the gaps you left with the most common version of that kind of spec. The gaps are where the money is. A spec is worth reading precisely because it deviates from the common version somewhere.

So the text goes in the prompt. Forty pages fits in context now, and if it does not, I split by section and keep each chunk verbatim rather than compressing it.

Ask for extractions, not a summary

A summary optimizes for sounding whole. An extraction optimizes for answering a specific question, and a specific question has a right answer you can check.

Instead of "summarize the auth section" I ask things like:

• List every field the webhook payload is required to contain, with its type.
• What are all the conditions under which the server returns a 4xx?
• Quote every sentence that uses the word "must" or "must not".
• What is the exact retry behavior, including intervals and the cap?

These have shapes. If the answer to the retry question is one line and the spec has a whole retry section, I know something got dropped and I go look. A summary would have smoothed that gap into a confident sentence and I would never have felt the missing piece.

Make it quote the exact line

This is the rule that changed the most for me. For every claim it makes, I require the model to quote the exact source sentence it is drawing from, verbatim, with the surrounding context.

Two things happen. First, if the fact is real, I now have the exact string to search for in the PDF, so verifying takes ten seconds instead of a reread of page 31. Second, if the fact is invented, the model has to invent a quote to go with it, and invented quotes are far easier to catch than invented summaries. A fabricated paraphrase blends in. A fabricated quote that is not in the document fails the instant I hit Ctrl-F.

I phrase it plainly. For each point, give me the claim, then the exact quoted sentence from the text, then the page or section. No quote, no claim.

Never act on a fact until you open that page

Here is the discipline, and it is the whole thing. I do not write a line of code against an extracted fact until I have opened that one page myself and read the sentence in its real context.

Here is what that catches. On the spec I mentioned, the model extracted, correctly, that the signature header uses HMAC SHA-256. True, quoted, right there on the page. What the summary never surfaced, and what the extraction only surfaced because I asked the "must" question, was a clause two paragraphs down. The signature is computed over the raw request body before any JSON parsing, and if your framework reparses and reserializes the body, the bytes change and every signature fails. That sentence was one line. It was the entire difference between a working integration and a day of "why is every webhook rejected." The convenient summary had "verifies an HMAC signature" and moved on.

The model did not hide that clause out of malice. It had no way to know that particular subclause was load bearing, because load bearing is a property of my implementation, not of the text. Only I know I am about to reserialize the body. So only I can catch it, and only if I actually open the page.

So the loop is simple. Paste the real text, ask narrow questions, demand the exact quote, then go read the source with my own eyes before I trust it. The model is a fast index into a document I still have to read. It points me at page 31. It does not get to read page 31 for me. Used that way it saves hours. Used as a summary machine it will hand you clean, confident, and quietly incomplete, which is the worst shape a spec can be in.

AGINE Academy is an independent product by AGINE AI (not affiliated with Anthropic). We teach building with Claude by doing the work, not watching lectures.