Before changing embeddings, test what your PDF parser actually preserved

#python #rag #testing #pdf

A RAG answer can cite the correct PDF and still use text in the wrong order. Before changing embeddings, inspect the evidence that reaches the index.

This example uses two teaching PDFs: a table with prices and a two-column policy document. It compares a plain-text extraction baseline with actual ComPDF parsing output. It does not run an embedding model or claim a retrieval-accuracy improvement.

Create an input with a known reading order

The two-column fixture has Policy A on the left and Policy B on the right. Policy A allows replacement within 14 days; Policy B allows 30 days. The PDF drawing commands deliberately alternate left and right lines, although a reader would read down each column.

That makes it a useful diagnostic fixture, not a representative benchmark of all PDFs. The teaching sample pack contains the inputs, observed outputs and scripts. Unzip it and run commands from its root directory. Use Python 3.10+; install examples/requirements.txt before inspecting PDF and DOCX outputs. You can inspect the saved results without an API key. A new parsing request requires your own server-side COMPDF_API_KEY and may consume account usage.

Preserve the parser output before chunking

Use the ComPDF document parsing API to obtain the result archive. The supplied Python script reads its key from COMPDF_API_KEY, checks the business status, downloads the archive and verifies that result_json.json exists:

python examples/parse_file.py samples/rag-columns.pdf parsed.zip --operation parse

For this result, result.pages[0].structured contains the title and content blocks. The left-column policy is grouped before the right-column policy. Keep the original JSON so later chunking experiments do not overwrite the evidence.

Check the exact fact association

Plain-text extraction in this test returned both policy headings before either duration. The structured output grouped “Policy A: replacement period” with “is 14 days from delivery.” A narrow assertion demonstrates that difference:

import json
import re
from pathlib import Path
from pypdf import PdfReader

baseline_text = PdfReader("samples/rag-columns.pdf").pages[0].extract_text()
result = json.loads(Path(
    "results/rag-columns-parsed/result_json.json"
).read_text())
parsed_text = "\n".join(
    block.get("text", "")
    for block in result["result"]["pages"][0]["structured"]
)
pattern = r'Policy A: replacement period\s+is 14 days'
assert not re.search(pattern, baseline_text)
assert re.search(pattern, parsed_text)

Passing this assertion means that this association survived in this fixture. It says nothing about semantic correctness, hallucinations or overall answer accuracy.

Figure: Observed reading-order difference for the teaching two-column fixture; this is a parser inspection, not a retrieval benchmark

Check table cells separately

The second fixture contains AX-271 at USD 19.40 and BX-272 at USD 0.00. The observed result retained a three-column HTML table. Inspect the cells rather than searching for isolated numbers: finding both a part number and a price somewhere in a document does not prove that they belong together.

Zero is also different from missing data. Keep 0.00 as a value and represent an unavailable price separately.

Then evaluate the downstream system

After these checks, freeze the parser output and compare chunking or embeddings with the same questions, expected answers and retrieval settings. Report failed cases along with successful ones. This example stops before that evaluation; it cannot establish that one parser is best for all RAG pipelines.

For a larger integration, the RAG document workflow connects document parsing to the later indexing steps. Start with your own authorized exception files and require evidence that the relevant facts survived before tuning the model.

DEV Community