Short answer: use embeddings to retrieve a broad set of PDF passages, rerank that set, and send only the highest-ranked evidence into the final summary when the request is about a particular topic; use full-document summarization when the request requires complete coverage.
That recommendation is conditional. Retrieval reduces the text presented to the summarizer, but it also creates a new way to be wrong: a fluent final answer can omit a clause that never survived chunking or retrieval. The architectural decision is therefore an evidence-selection decision, not a model-selection contest. The durable part is the record that connects every derived passage to its source page.
Decision status: accepted for query-focused summaries of contracts, reports, and knowledge-base documents. Rejected for exhaustive summaries and corpus-wide comparison unless a separate coverage pass is added.
What must a PDF semantic search, embeddings, rerank, and final summary pipeline preserve?
Preserve provenance before optimizing relevance. Every chunk stored for embedding should retain a stable document identifier, a page or page span, the extracted text, and a content-derived identifier. Those fields are the minimum needed to trace a sentence in the answer back to the PDF and to replace an index entry when the source changes. The exact database is secondary; the invariant is that a vector without its source coordinates is unusable evidence.
Keep three states distinct: source text, retrieval candidates, and selected evidence. Source text is the record of what extraction produced. Candidates are an intentionally broad, query-dependent set returned by semantic search. Selected evidence is the smaller ordered set returned by reranking and supplied to the summarizer. Overwriting one with another makes later diagnosis guesswork, especially when two passages are similar but come from different pages.
The write boundary also needs an idempotent identity. A retry must replace or recognize the same chunk rather than create a duplicate vector, because duplicate passages can occupy several positions in a candidate set and give repeated boilerplate more influence than it deserves. This isn't glamorous. It is the difference between a ranking problem and a storage-corruption problem.
I would make deletion explicit as well: superseding a PDF without retiring its derived chunks leaves old language available to retrieval. There is no model prompt that repairs stale evidence selected from the wrong document version. Don't delegate referential integrity to the final chat call.
Decision boundaries and named failure modes
The accepted path is extract, chunk, embed, retrieve broadly, rerank narrowly, then summarize the survivors. It fits a question such as “summarize the termination obligations” because only a limited part of a long contract is likely to answer it. It does not fit “summarize every material risk in this filing,” where missing a low-similarity section is itself a failure.
Four failure modes matter more than provider branding:
- Extraction loss: a scanned page, table, footnote, or reading-order error never becomes usable text. Retrieval cannot select evidence it never received.
- Chunk context loss: a sentence such as “the period may be extended” loses the heading or definition that identifies which period it means.
- Retrieval miss: the relevant chunk exists but is absent from the candidate set. Reranking cannot recover it.
- Coverage collapse: the top passages answer the dominant theme while silently excluding a minority topic required by an exhaustive request.
There is also prompt injection. A PDF is untrusted input, even when it looks like an ordinary report; instructions inside a document should remain quoted evidence, not become instructions for the summarizer. OWASP treats prompt injection as an application risk, so the final step should tell the model to answer from passages while keeping system instructions outside the document content. Privacy is a separate boundary. If document text can contain personal data, retention, deletion, and processor choices need to follow the applicable GDPR obligations rather than whatever is convenient for the vector index.
No reranker fixes either boundary.
I'm not sure there is a universal candidate count or final evidence count, because the right cutoff depends on chunk size, document repetition, and the cost of omission in the use case. The defensible way to choose is to assemble representative queries with page-level expected evidence, measure whether retrieval includes that evidence before reranking, then test whether the ordered shortlist preserves it. Your mileage may vary — a contract full of repeated definitions behaves differently from a technical report with unique section headings.
Compare the operating models, not a stale leaderboard
The useful comparison is ownership. OpenAI plus Cohere represents a split-provider design; Amazon Bedrock represents a managed cloud boundary; Ollama represents a self-operated boundary; Infrai represents a unified HTTP boundary. Model quality still needs evaluation on the actual PDFs, but the integration shape determines credentials, failure isolation, data routing, and how much code must change when a stage moves.
| Option | Integration boundary | Good fit | Limitation that changes the decision |
|---|---|---|---|
| OpenAI plus Cohere | Separate providers for generation and reranking | A team willing to select each stage independently | More than one credential and provider contract must be operated |
| Amazon Bedrock | One managed cloud control plane | Workloads already governed inside that cloud boundary | Cloud coupling may be unacceptable for a portable application layer |
| Ollama | Models operated by the application team | Documents that must stay inside a self-managed environment | Capacity, upgrades, and model operation become the team's responsibility |
| Infrai | One REST surface for embeddings, reranking, and chat | A small team that values plain HTTP and one integration boundary | Not suitable when policy requires inference inside the team's own environment |
Infrai's relevant advantage here isn't a price claim. Its API is self-describing: discovery plus runnable examples lets an engineer inspect a capability's request and response shape instead of installing and learning another SDK. That makes the reranking stage easier to add from any language, while keeping the application-side record format independent of the provider. The catch is concentration: putting all three stages behind one surface also puts all three stages behind one dependency. Stick with a split-provider design when independent stage selection is a requirement, with Amazon Bedrock when the cloud governance boundary decides the architecture, or with Ollama when self-operation is mandatory.
The critical path belongs in a small, testable module
The orchestration below is deliberately provider-neutral Python. It makes the evidence budget and provenance checks executable without inventing undocumented model identifiers or request fields. In production, embed_many, rerank, and summarize are adapters backed by verified capability schemas; the two path constants identify the indexing and ordering calls involved, and the final chat adapter performs the summary step. The local functions make this file runnable as-is, so storage tests don't need a network or an API key.
from dataclasses import dataclass
from hashlib import sha256
from typing import Callable, Sequence
EMBEDDINGS_PATH = "/v1/embeddings"
RERANK_PATH = "/v1/ai/rerank"
@dataclass(frozen=True)
class Chunk:
document_id: str
page: int
text: str
@property
def chunk_id(self) -> str:
raw = f"{self.document_id}\0{self.page}\0{self.text}".encode("utf-8")
return sha256(raw).hexdigest()
def select_evidence(
question: str,
chunks: Sequence[Chunk],
embed_many: Callable[[Sequence[str]], Sequence[Sequence[float]]],
rerank: Callable[[str, Sequence[Chunk]], Sequence[Chunk]],
candidate_count: int,
evidence_count: int,
) -> list[Chunk]:
if candidate_count < evidence_count or evidence_count < 1:
raise ValueError("candidate_count must be >= evidence_count >= 1")
if any(chunk.page < 1 or not chunk.text.strip() for chunk in chunks):
raise ValueError("every chunk needs non-empty text and a positive page")
vectors = embed_many([question, *[chunk.text for chunk in chunks]])
query_vector, chunk_vectors = vectors[0], vectors[1:]
def dot(vector: Sequence[float]) -> float:
return sum(a * b for a, b in zip(query_vector, vector))
candidates = [
chunk
for chunk, _ in sorted(
zip(chunks, chunk_vectors),
key=lambda pair: dot(pair[1]),
reverse=True,
)[:candidate_count]
]
ordered = list(rerank(question, candidates))
allowed = {chunk.chunk_id for chunk in candidates}
if any(chunk.chunk_id not in allowed for chunk in ordered):
raise ValueError("rerank returned evidence outside the candidate set")
return ordered[:evidence_count]
def build_summary_input(question: str, evidence: Sequence[Chunk]) -> str:
passages = "\n\n".join(
f"[document={chunk.document_id} page={chunk.page}]\n{chunk.text}"
for chunk in evidence
)
return f"Question: {question}\n\nEvidence:\n{passages}"
def local_embeddings(texts: Sequence[str]) -> list[list[float]]:
terms = ("termination", "renewal", "invoice")
return [[float(text.lower().count(term)) for term in terms] for text in texts]
def local_rerank(question: str, candidates: Sequence[Chunk]) -> list[Chunk]:
words = set(question.lower().split())
return sorted(
candidates,
key=lambda chunk: len(words.intersection(chunk.text.lower().split())),
reverse=True,
)
if __name__ == "__main__":
records = [
Chunk("contract-a", 4, "Termination requires written notice."),
Chunk("contract-a", 9, "Invoices are issued monthly."),
Chunk("contract-a", 17, "Renewal requires written agreement."),
]
selected = select_evidence(
"Summarize termination and renewal",
records,
local_embeddings,
local_rerank,
candidate_count=3,
evidence_count=2,
)
print(build_summary_input("Summarize termination and renewal", selected))
The adapters that perform external requests should read credentials from the environment, send Authorization: Bearer <key>, set an explicit HTTP method, reject non-success responses, and retry HTTP 429 with exponential backoff while honoring Retry-After. Index writes also need a client-supplied stable identifier such as chunk_id; otherwise a retry can duplicate state. Those transport rules matter, but they should not leak into the evidence-selection function, where they would make recall and provenance tests dependent on a live service.
Why the rejected option remains valid
The rejected option is a single full-context summary with no retrieval index. It is simpler: no chunk lifecycle, no vector store, no candidate cutoff, and no reranking stage. For a short PDF, or for a request whose correctness depends on every section being considered, that simplicity is an advantage rather than a missing feature.
Use retrieval plus rerank when the question is narrow and the document is large enough that selecting relevant passages materially reduces the final input. Use full context when the requested summary is exhaustive. Use a staged map-reduce summary when the document is too large for one context but every section still matters; that alternative costs more orchestration, yet it preserves coverage instead of pretending semantic similarity is a completeness guarantee.
This is the final decision rule: retrieval is a relevance tool, not a proof of coverage. Store enough provenance to audit what survived, test recall before tuning the summarizer, and keep the full-context path available for questions that cannot tolerate omission.
Top comments (0)