This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.
Project Overview
I generate a PDF from a plain-text source with reportlab, and I verify it before it leaves the machine. The verifier is the obvious one: open the PDF with PyMuPDF, pull the text back out, assert that every required item is in there.
Nine probes. Nine passes. I sent the file.
The file I sent was fine, but only because I had also looked at it. When I rendered the pages to PNG and put my eyes on them, two things were wrong that the checker had not mentioned: a section heading was set in the wrong face, and a long URL ran off the right-hand side of the paper.
That second one is the interesting one, because of what "off the paper" does to a text-extraction check.
The URL was 115 characters. The text column is 511.3pt wide. At 8.2pt Courier, indented, the line needed 575.6pt. The last few characters were drawn past x=595.3 — the edge of an A4 sheet — and they are not on the page in any sense: not in the print, not in the render, not in the extracted text.
So the extractor returned the URL without its tail. And a dev.to URL without its tail is a different URL:
https://dev.to/aiq_labs/...-only-one-of-them-was-real-1jh0 -> 200
https://dev.to/aiq_labs/...-only-one-of-them-was-real- -> 404
My checker asserted the URL was present. It was present — four characters short, pointing at nothing. None of my nine probes reached far enough into the string to notice, because when you write a probe by hand you write the part that identifies the item, not the part that makes it resolve.
A text-extraction check cannot see the right-hand edge of the paper. It reads what the content stream says. It does not know where any of it landed.
Bug Fix or Performance Improvement
Measuring it instead of asserting it
The original naive generator was never committed, so I reconstructed it from the fixed one by removing exactly the two changes I had made: the shrink-to-fit loop and the structural heading test. Then I ran both versions of the real document through both kinds of check.
Same source text, A4, 42pt margins:
content check spans off-column URLs truncated chars lost
pre-fix (reconstructed) 9/9 PASS 4 3 of 10 23
shipped 9/9 PASS 0 0 of 10 0
Three of ten links silently truncated, twenty-three characters gone, and the content check reports 9/9 PASS on both. It is not that the check is weak. It is that the check is answering a different question than the one I thought I was asking.
Two ways to leave the column, and only one of them loses data
Building a minimal reproducer made the shape clearer. There are two regimes, and the difference matters:
v1, spans that left the text column:
p1 + 44.7pt past the paper https://dev.to/aiq_labs/i-wrote-two-fixes-for-the-sa
p1 + 34.8pt into the margin Verification tooling: deterministic geometry checks
The first line ran past the sheet, so its tail is gone from the extracted text too — the content check could have caught it with a luckier probe. The second stopped inside the paper but outside the column. Nothing is lost there. The characters extract perfectly. The document just looks broken, and no content check of any kind will ever tell you, because from the extractor's point of view nothing happened.
In the reproducer, both versions score:
content check geometry check URL fidelity
v1 naive 10/10 PASS 2 off-column 4 chars lost
v2 fitted 10/10 PASS 0 off-column 0 chars lost
The content column is constant. Every bit of signal is in the other two.
The fix
Two changes, on two different layers.
Generator: shrink the line until it fits the column, with a floor.
size = BODY_SIZE
while size > MIN_SIZE and stringWidth(line, "Courier", size) > COL:
size -= 0.15
c.setFont("Courier", size)
c.drawString(LEFT, y, line)
Verifier: stop asking what the document says and start asking where it put it.
def check_geometry(pdf_path, slack=0.5):
"""Every drawn span must lie inside the text column."""
offenders = []
with fitz.open(pdf_path) as doc:
for pno, page in enumerate(doc, 1):
right_edge = page.rect.width - LEFT
for block in page.get_text("dict")["blocks"]:
for line in block.get("lines", []):
for span in line["spans"]:
x0, _, x1, _ = span["bbox"]
if x1 > right_edge + slack or x0 < LEFT - slack:
offenders.append({
"page": pno,
"overshoot_pt": round(x1 - right_edge, 1),
"past_paper_pt": round(x1 - page.rect.width, 1),
"text": span["text"].strip(),
})
return offenders
That is a dozen lines and it turns "looks fine to me" into a number that fails a build. past_paper_pt is what separates the two regimes: positive means the glyphs are gone, zero or less means they printed somewhere they should not have.
I also added a fidelity check, because it is cheap and it catches the failure that actually costs something: pull every URL out of the source and every URL out of the PDF, compare them positionally, and report the character delta. Twenty-three, in the pre-fix document.
Code
Everything above is reproducible from one file with two dependencies (reportlab, pymupdf). Measured on Python 3.14.2, reportlab 4.5.1, PyMuPDF 1.27.2.2, Windows. It builds the same source document twice and runs all three checks over both — copy, run, and you get the table above.
"""A text-extraction check cannot see page geometry.
Builds one source document twice:
v1 naive - fixed font size (what a first draft does)
v2 fitted - shrink each line until it fits the text column
Then runs two verifiers over both PDFs:
check_content() - "is every field present?" (PyMuPDF text extraction)
check_geometry() - "is every glyph on the paper?" (PyMuPDF span bboxes)
check_fidelity() - compares extracted URLs against the source, character for character
Requires: reportlab, pymupdf
"""
import sys
if hasattr(sys.stdout, "reconfigure"):
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
import pathlib
import re
import fitz
from reportlab.lib.pagesizes import A4
from reportlab.pdfbase.pdfmetrics import stringWidth
from reportlab.pdfgen import canvas
HERE = pathlib.Path(__file__).resolve().parent
W, H = A4 # 595.3 x 841.9 pt
LEFT, TOP, BOTTOM = 42, 52, 46
COL = W - 2 * LEFT # the text column: 511.3 pt
BODY_SIZE, LEAD, MIN_SIZE = 8.2, 10.6, 5.4
SOURCE = """Ai-Q Labs
independent developer - remote
----------------------------------------------------------------
Selected writing
----------------------------------------------------------------
Race conditions in parallel agent sessions
https://dev.to/aiq_labs/i-wrote-two-fixes-for-the-same-race-condition-gemini-told-me-only-one-of-them-was-real-1jh0
A collector that reported success on 27% of the data
https://dev.to/aiq_labs/my-collector-reported-success-it-had-27-of-the-data-2n0d
----------------------------------------------------------------
Tooling
----------------------------------------------------------------
Apify actors published 23
dev.to articles published 6
Verification tooling: deterministic geometry checks for generated PDFs, so a broken layout cannot pass a test
"""
# What a person actually writes when asked "check the resume came out right".
# Nine probes, one per thing that must appear. Note how each one names the
# item -- none of them reaches for the tail end of a URL.
PROBES = [
"Ai-Q Labs",
"independent developer",
"Selected writing",
"Race conditions in parallel agent sessions",
"https://dev.to/aiq_labs/i-wrote-two-fixes-for-the-same-race-condition",
"https://dev.to/aiq_labs/my-collector-reported-success",
"Tooling",
"Apify actors published",
"dev.to articles published",
"deterministic geometry checks for generated PDFs",
]
def is_rule(s):
s = s.rstrip()
return len(s) > 3 and set(s) == {"-"}
def build(out_path, fitted):
lines = SOURCE.splitlines()
c = canvas.Canvas(str(out_path), pagesize=A4)
y = H - TOP
for i, raw in enumerate(lines):
line = raw.rstrip()
prev = lines[i - 1] if i else ""
nxt = lines[i + 1] if i + 1 < len(lines) else ""
if y < BOTTOM + LEAD:
c.showPage()
y = H - TOP
if is_rule(line):
y -= 3
c.setStrokeGray(0.75)
c.setLineWidth(0.5)
c.line(LEFT, y, W - LEFT, y)
c.setStrokeGray(0)
y -= LEAD
continue
if bool(line.strip()) and is_rule(prev) and is_rule(nxt):
c.setFont("Helvetica-Bold", 9.6)
c.drawString(LEFT, y, line.strip())
y -= LEAD + 1
continue
if line:
size = BODY_SIZE
if fitted:
while size > MIN_SIZE and stringWidth(line, "Courier", size) > COL:
size -= 0.15
c.setFont("Courier", size)
c.drawString(LEFT, y, line)
y -= LEAD
c.save()
return out_path
def extract(pdf_path):
with fitz.open(pdf_path) as doc:
return "".join(page.get_text() for page in doc)
def check_content(pdf_path):
"""The check I actually ran: extract text, assert every field is present."""
text = extract(pdf_path)
missing = [p for p in PROBES if p not in text]
return len(PROBES) - len(missing), len(PROBES), missing
def check_geometry(pdf_path, slack=0.5):
"""Every drawn span must lie inside the text column."""
offenders = []
with fitz.open(pdf_path) as doc:
for pno, page in enumerate(doc, 1):
right_edge = page.rect.width - LEFT
for block in page.get_text("dict")["blocks"]:
for line in block.get("lines", []):
for span in line["spans"]:
x0, _, x1, _ = span["bbox"]
if x1 > right_edge + slack or x0 < LEFT - slack:
offenders.append({
"page": pno,
"overshoot_pt": round(x1 - right_edge, 1),
"past_paper_pt": round(x1 - page.rect.width, 1),
"text": span["text"].strip(),
})
return offenders
def check_fidelity(pdf_path):
"""Compare each URL in the PDF against the URL at the same position in the source.
Positional, not prefix-matched: an earlier attempt paired them by prefix and
happily matched a truncated URL against a shorter unrelated one.
"""
url_re = re.compile(r"https://\S+")
want = url_re.findall(SOURCE)
got = url_re.findall(extract(pdf_path))
return [(w, g, len(w) - len(g)) for w, g in zip(want, got)]
def main():
v1 = build(HERE / "out_v1_naive.pdf", fitted=False)
v2 = build(HERE / "out_v2_fitted.pdf", fitted=True)
print(f"{'':<12}{'content check':>16}{'geometry check':>18}{'URL fidelity':>16}")
for label, path in (("v1 naive", v1), ("v2 fitted", v2)):
ok, total, missing = check_content(path)
off = check_geometry(path)
lost = sum(d for _, _, d in check_fidelity(path))
print(
f"{label:<12}"
f"{(str(ok) + '/' + str(total) + (' PASS' if not missing else ' FAIL')):>16}"
f"{('0 off-column' if not off else str(len(off)) + ' off-column'):>18}"
f"{(str(lost) + ' chars lost'):>16}"
)
print("\nv1, spans that left the text column:")
for d in check_geometry(v1):
edge = "past the paper" if d["past_paper_pt"] > 0 else "into the margin"
print(f" p{d['page']} +{d['overshoot_pt']:>5}pt {edge:<16} {d['text'][:52]}")
print("\nv1, URLs as extracted:")
for want, got, delta in check_fidelity(v1):
print(f" -{delta} chars: ...{want[-34:]}")
print(f" ...{got[-34:] if got else '(nothing extracted)'}")
if __name__ == "__main__":
main()
The claims in the last section are a second file, check_gemini_claims.py, built the same way: each claim becomes a deliberately broken PDF, then a measurement.
One caveat about the vertical bounds check, found while writing that file: as written it flags the first line of a page, because a baseline placed exactly at the top margin puts the glyph ascenders above it. That is arguably a real defect in my generator rather than a false positive, but it is a real thing you will hit on the first run.
My Improvements
Done:
- Shrink-to-fit in the generator; 0 off-column spans in the shipped document.
-
check_geometryin the verifier, run on every build. - URL fidelity check, source vs. PDF, positional, character-exact.
Done because the review said so:
- Vertical bounds and line-collision checks, both verified to catch cases the horizontal check reports clean (numbers below).
Not done, and I'd rather say so:
- The generator fix is a patch, and I now have the measurement that proves it (below). The real fix is to stop drawing long strings as single un-wrapped
drawStringcalls and let a layout engine break them. I have not done that yet, because I have exactly one document and the verifier now fails the build if the patch is ever insufficient. When there is a second document, the patch stops being defensible.
Left as a guard, deliberately:
The render-and-look step. It is the only reason I found any of this, and no check I have written since would have caught the wrong-typeface heading — that one has no numeric signature at all.
Best Use of Google AI
I used Gemini (free tier, Flash) at one point: after both fixes were written and the numbers were in, before I decided the job was done. I handed it the measurement table, both check functions, both fixes, and three questions — state the general class of defect, name what my new check still misses, and tell me for each of my two fixes whether it is a fix or a patch.
It gave the class cleanly:
A purely semantic text-extraction test [...] cannot detect geometric presentation defects [...] Text extractors read the stream's character data (the what); they do not simulate clipping paths, viewports, or bounding-box intersections with canvas boundaries (the where and how).
Then four ranked failure modes my check still misses, each with code, and a verdict on each of my two fixes: shrink-to-fit — PATCH. Geometry check — GENUINE FIX.
I did not adopt any of it. I built each claim as a deliberately broken PDF and measured.
Claim Verdict Measurement
shrink-to-fit still fails at the size floor held 202-char line shrank to 5.35pt
(floor 5.4), still outside the column
Rank 1 bottom overflow held horizontal check 0, vertical check 6
Rank 2 overlapping lines held both bounds checks 0, collision check
5 pairs, overlap up to 7.0pt
stringWidth disagrees with PyMuPDF metrics did not hold Courier +0.00pt, embedded TTF -0.03pt
Rank 4 text hidden by a clip path did not hold and it inverted, see below
Three held. Two did not, and the two failures are the useful part.
The font-metrics claim was wrong on my setup. The argument was that stringWidth uses target font metrics while PyMuPDF recomputes bounding boxes from embedded metrics, so my shrink loop is built on a number that lies. Measured, they agree to 0.00pt for base-14 Courier and to 0.03pt for an embedded TrueType face. Plausible mechanism, real in principle, not happening here.
The clip-path claim inverted. The prediction was that get_text() returns text that graphics commands have clipped away, so my content check would pass while the page printed only part of the string. I built exactly that document — 74 characters drawn inside a 150pt-wide clip rectangle — and got the opposite:
extracted : 'PAYMENT TERMS: net 30 days from'
text check: FAIL (the string is not there)
geometry : PASS (the clipped bbox is inside the margins)
rendered : ink stops at 150pt of a 153pt span
PyMuPDF 1.27 honours the clip during extraction. So the check that got called weak is the one that catches this, and the check that got called a genuine fix is the one that reports the page clean. My geometry check does not compare the span bbox against the clipping path — it never sees one.
That is worth more to me than the three that held. The three that held told me where to add code. The one that inverted told me that my new check has the same shape of blind spot as the old one: it trusts a number the document hands it, and that number has already been through the machinery it is supposed to be auditing.
The thread running through all of my submissions this month is the same, and I keep walking into it: every check passed and the answer was still wrong. A ranking only I could see. A collector reporting success while holding 27% of the data. A rate limiter counting the retries. A guard that had stopped guarding. Two fixes where one was theatre. And now a verifier that read every character back correctly while three of the links in the document it had just approved pointed at nothing.
What breaks the pattern is never a stricter version of the same check. It is a second fact, derived a different way, that is allowed to disagree with the first. Here that was pixels versus characters — and, once I had run out of ways to doubt myself, a second reader willing to hand me four things to go break.
Top comments (0)