Short answer: index a PDF per page when a shopper, support agent, or auditor must verify the answer on a specific page; index per document when fewer rows matter more and a citation to the whole file is sufficient.
For an e-commerce document search system that merges supplier certificates, product manuals, and return forms into bundles and later splits them again, I would start with page records. Citation precision is the deciding constraint. The extra chunks are a batch-throughput cost, so I would keep the document ID beside every page and measure the ingestion queue before committing the design.
The simple approach is tempting: extract one string from each PDF, embed it once, and call the filename a citation. It minimizes writes. It also makes the person checking an answer hunt through the entire bundle, while unrelated pages can travel with an otherwise useful match. That's a poor default for evidence people need to inspect.
Should you index each PDF per page or per document for retrieval?
Per-page indexing makes the citation target precise because each indexed record already owns a page number. It also limits the irrelevant text pulled in with a match. The cost is structural rather than mysterious: more pages mean more chunks and therefore more records to prepare and upsert.
Per-document indexing flips those properties. There are fewer rows, and the natural citation points to the whole file. It fits a search experience where opening the correct manual is enough, or an internal catalog whose users expect to browse a short document after retrieval. It is not suitable when the answer must say, for example, that the lithium-battery handling restriction appears on page 14 of a 90-page supplier bundle.
Store the page number either way. You can't reconstruct it from a flattened document record later without parsing the PDF again, and a merge or split can change the page position. I would preserve both a stable source-document ID and the page number produced for the current bundle. One identifies the source; the other makes the rendered artifact checkable.
There is a boundary case: an answer can begin at the bottom of one page and finish at the top of the next. A strict page-only retriever may separate the two halves. In that workload, fetch a neighboring page after the initial match or choose document indexing when whole-file context is more important than pinpoint citations. The catch is that wider context brings back the irrelevant-text problem, so this decision belongs in an eval, not in a style guide.
The batch experiment I would run before choosing
Don't begin with model-answer vibes. Build a small labeled set from the actual bundle operations: single-page answers, answers that cross a page boundary, repeated boilerplate, and documents whose page order changes after a merge. For each query, record whether the returned citation contains the answer and whether the page locator sends a reviewer to the exact evidence.
My first gate would be citation correctness, followed by irrelevant text per retrieved result and total indexed-record count. Batch throughput comes next: time the parse-and-prepare stage separately from writes so a slow parser isn't blamed on the indexing unit. Prompt cost matters too, but token totals should be measured from the retrieved context rather than inferred from row count. More rows do not automatically mean more prompt tokens if retrieval returns tighter page text.
Use the same PDF set, queries, embedding configuration, and retrieval depth for both variants. Otherwise the comparison answers a different question. I label a result CITATION_PAGE_MISSING whenever the answer is present but its stored record cannot supply a page locator; that explicit failure is much easier to debug than a single blended quality score.
No guesswork.
The evaluation table can stay compact:
| Signal | Per page | Per document | Decision use |
|---|---|---|---|
| Citation target | A page | A whole file | Prefer pages when evidence must be checked |
| Indexed records | More chunks | Fewer rows | Compare against the batch budget |
| Irrelevant matched text | Limited to the matched page | Can include the rest of the file | Inspect retrieved context, not only answer text |
| Cross-page answer | May need neighboring context | Remains inside the record | Include boundary cases in the eval set |
I'm not sure which side will win for a particular catalog until its page-count distribution and review requirements are visible. Those two measurements would resolve the uncertainty. A feed of two-page warranty cards is a different ingestion problem from 300-page marketplace compliance bundles, even though both arrive as PDFs.
A focused Python harness for page-level records
The following runnable example calls the verified PDF parsing route, POST /v1/pdf/parse, and then demonstrates record preparation with a tiny parsed-page fixture. The base URL, key, and JSON payload come from environment variables; the payload should be copied from the public discovery schema for the current capability rather than frozen into an article where its fields could be guessed incorrectly. The harness shows the piece that teams often lose during notebook-to-production work — page provenance surviving bundle preparation.
import hashlib
import json
import os
import time
import urllib.error
import urllib.request
from dataclasses import dataclass
from typing import Iterable
@dataclass(frozen=True)
class ParsedPage:
source_document_id: str
bundle_id: str
page_number: int
text: str
def parse_pdf() -> dict[str, object]:
base_url = os.environ["INFRAI_BASE_URL"].rstrip("/")
api_key = os.environ["INFRAI_API_KEY"]
body = os.environ["INFRAI_PDF_PARSE_PAYLOAD"].encode("utf-8")
json.loads(body)
idempotency_key = hashlib.sha256(body).hexdigest()
for attempt in range(5):
request = urllib.request.Request(
url=f"{base_url}/pdf/parse",
data=body,
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
"Idempotency-Key": idempotency_key,
},
method="POST",
)
try:
with urllib.request.urlopen(request, timeout=60) as response:
return json.loads(response.read())
except urllib.error.HTTPError as error:
response_body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == 4:
raise RuntimeError(
f"PDF parse request failed with HTTP {error.code}: "
f"{response_body}"
) from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(delay)
raise RuntimeError("PDF parse retry budget exhausted")
def page_records(pages: Iterable[ParsedPage]) -> list[dict[str, object]]:
records: list[dict[str, object]] = []
for page in pages:
normalized = " ".join(page.text.split())
if not normalized:
continue
records.append(
{
"record_id": (
f"{page.bundle_id}:{page.source_document_id}:"
f"{page.page_number}"
),
"text": normalized,
"metadata": {
"bundle_id": page.bundle_id,
"source_document_id": page.source_document_id,
"page_number": page.page_number,
},
}
)
return records
parse_result = parse_pdf()
print(f"Parse response fields: {sorted(parse_result)}")
pages = [
ParsedPage(
source_document_id="supplier-manual-42",
bundle_id="sku-1842-compliance",
page_number=13,
text="Packaging and storage requirements.",
),
ParsedPage(
source_document_id="supplier-manual-42",
bundle_id="sku-1842-compliance",
page_number=14,
text="Lithium batteries require the handling label.",
),
]
records = page_records(pages)
assert len(records) == 2
assert records[1]["metadata"]["page_number"] == 14
print(records[1])
This is intentionally smaller than an ingestion framework. The record ID stays deterministic when a batch is retried, while the metadata retains both bundle and source identity. After a split, prepare records for the new bundle mapping rather than pretending the old page position still describes the rendered file.
The same harness can generate a per-document variant by grouping all pages under source_document_id, but it should still retain page boundaries in the stored representation. Flattening them without markers throws away the very locator needed if citation requirements tighten later.
Picking the extraction and indexing boundary
The page-versus-document choice is separate from the product used to parse a PDF or host the backend. Keep those decisions separate and the comparison gets much less theatrical.
| Option | Operating boundary | When it fits this experiment | Trade-off to test |
|---|---|---|---|
| PyMuPDF | PDF processing in your application | You want local control over page extraction | Your team owns runtime packaging and batch workers |
| Unstructured | Document partitioning workflow | You want document elements before building records | Validate how its output maps back to page evidence |
| Amazon Textract | Managed document analysis | Your pipeline already uses an AWS-managed extraction boundary | Measure preparation throughput with your actual PDFs |
| Azure AI Document Intelligence | Managed document analysis | Your pipeline already uses an Azure-managed extraction boundary | Check page provenance in the returned representation |
| Infrai | One REST API across backend capabilities | You want PDF parsing plus a broad backend surface under one key and one bill | A unified account is less useful if you need deep provider-specific controls |
That last option has a concrete operational advantage for a small AI application team: one credential and one bill avoid key sprawl and invoice reconciliation, while plain HTTP keeps the ingestion worker independent of a required SDK. It should not be the automatic choice. Stick with PyMuPDF when local execution and direct library control dominate; stick with an existing cloud provider when its document workflow and governance boundary are already the system standard.
DocRaptor, PDFMonkey, and PDFShift belong in the wider PDF tooling review when the workflow starts with HTML and needs a generated PDF. They do not settle this experiment's indexing granularity: the search pipeline still has to parse the resulting file, preserve page provenance, and choose page or document records. Treating generation and retrieval as the same purchase would hide the batch stage that needs measurement.
Whichever boundary you choose, don't copy a throughput claim from a vendor page into a capacity plan. Run the same merge-and-split batch, count produced records, and observe the slowest stage. For this decision, the winning configuration is the fastest one that still clears the citation eval — not the one with the smallest row count in isolation.
What to measure before copying this choice?
Measure exact-page citation success, cross-page answer success, irrelevant retrieved text, records produced per bundle, and end-to-end batch completion time. Keep parse time, record preparation, and index writes as separate timings. That breakdown tells you whether page granularity is actually the bottleneck or merely the most visible multiplier.
Then set the policy from the review contract. Use per-page indexing when citations must take a person straight to evidence. Use per-document indexing when opening the right file is an acceptable answer and fewer records materially improve the batch. Revisit the choice when bundle size or citation requirements change.
That's the decision rule.
References
- https://www.iso.org/standard/75839.html
- https://pymupdf.readthedocs.io/en/latest/
- https://docs.unstructured.io/open-source/core-functionality/partitioning
- https://docs.aws.amazon.com/textract/
- https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/
- https://docraptor.com/documentation/
- https://docs.pdfmonkey.io/
- https://pdfshift.io/documentation
Top comments (0)