On 24 August my AI agent read a 29-page tender document and told me that two required sections were missing: the invitation and the technical specification. We wrote to the institution to say their published documentation was incomplete.
They answered politely that it was in the attached documents. They were right.
Pages 1 to 12 of that PDF were scans: photographs of paper at 300 dots per inch, with no text layer at all. The text extractor returned nothing for them, without an error. The agent read the 17 pages that had text, saw no invitation and no specification there, and reported what it had seen as the whole document. Coverage was 59 percent. Nothing measured it, so it read as 100.
An empty page is a question
A PDF can carry text in two ways. A document exported from a word processor has a text layer, and extraction is exact. A scanned document is a stack of pictures, and the only way to read it is optical character recognition, OCR, which is slow, approximate and easy to forget.
The extractor does not tell you which kind of page it got. It returns an empty string for a scan and an empty string for a blank page, and both look like "nothing here". So the first check is per page, before any conclusion about the document:
import subprocess
def text_per_page(pdf, pages):
out = []
for n in range(1, pages + 1):
t = subprocess.run(["pdftotext", "-f", str(n), "-l", str(n), pdf, "-"],
capture_output=True, text=True).stdout
out.append(t)
return out
def unread_pages(texts, min_chars=40):
return [n for n, t in enumerate(texts, 1) if len(t.strip()) < min_chars]
If unread_pages is not empty, the document has not been read. Those pages go to OCR, and until they come back, the only honest answer is "could not check", with the page numbers.
My reader now prints the character count for every page, OCRs the empty ones at 300 dpi, and exits with an error while any page is still unread. Its last line is === READ COMPLETE ===. No sentinel, no conclusion.
Readable garbage
OCR does not fail loudly either. It returns text, and the text can be nonsense.
On 7 September the agent read a 14-page reply from a public institution. Every page was a scan, every page went through OCR, and the reader printed READ COMPLETE. Four of the pages were printed in landscape, a wide table turned sideways on the sheet. OCR read them upright and produced fragments of columns: real letters, real-looking words, no meaning. The summary said "four planned positions". The document listed more than twenty-five awarded contracts.
Character count cannot catch this. The garbage had plenty of characters. What it lacked was words. So the reader now measures that, crudely and cheaply:
import re
def readability(text):
words = re.findall(r"[^\W\d_]{3,}", text)
if not words:
return 0.0
real = [w for w in words if re.search(r"[aeiouAEIOU]", w)]
return len(real) / max(1, len(text.split()))
It is the share of tokens that look like words: three letters or more, with a vowel. It knows nothing about any language, and it does not need to. On those four pages the upright reading scored between 0.05 and 0.07. Rotated by 90 degrees, the same pages scored between 0.57 and 0.64.
So when a page scores low, or is wider than it is tall, the reader OCRs it again at 90 and 270 degrees and keeps the best reading. It also says so, on its error output: "page 5 was sideways, rotated". A silent correction is how the next person never learns the document had sideways pages.
The rotation is only tried when needed. OCR is the most expensive thing my server does, and turning every page three ways would triple the cost for all the pages that were already fine.
The same thing happens with tables. The OCR engine has layout modes, and the one that treats the page as a single block of text destroys a table. On the same day, a procurement plan read in that mode came back as "no IT purchases"; the mode that looks for columns read the same image about ten times better. If a page is a table, the reading mode is part of the question.
When two readings disagree
Some things on a page matter more than the rest: a date, an amount, a reference number. OCR gets single digits wrong more often than words, because a word has redundancy and a digit has none.
On the 24 August document, the deadline came back as 23.08 in one layout mode and as 28.08 in another. Same image, same engine. Neither reading announced any doubt.
So when a digit matters, the reader runs more than one mode and compares:
def digits_agree(readings):
"""readings: the same line as read by several OCR modes."""
found = [tuple(re.findall(r"\d+", r)) for r in readings]
return len(set(found)) == 1
If they agree, the value is probably right. If they disagree, the value is not known, and the agent does not pick one. It reports both, with the page, and the date comes from a source that is not a photograph: the published notice, the email, the person. The rule I wrote down after that day: OCR never settles a date.
The read is not the answer
There is one more way to lose a page, and it has nothing to do with OCR. On 26 August the agent read a six-page document correctly, every page, and then passed the output through head -120 to keep it short. The cut fell on page 4. It searched what was left for the figure the whole matter turned on, found nothing, and reported that the figure was not in the document. It was on page 6.
The reader had done its job. The pipe threw the answer away, and what was left looked like a complete document. The fix was to stop searching outside the reader. Mine takes the search itself, --grep "pattern", runs it over the complete text of every page, and prints either the matches with their page numbers or "0 matches over N of N pages: this is a true negative". It also writes the full text to a file and prints that path first, so even a truncated pipe leaves the complete text one step away.
What the agent may conclude
From a document where every page has text, or has been read by OCR with a readable score, and the sentinel was printed: what the document says. From a document with unread pages, low-scoring pages, or a digit two readings disagree on: nothing about those pages. It names them, and says what would settle them. From a slice of the output: nothing at all about the rest.
Before you trust a reading of a document, count its pages, ask which ones were really read, and make the reader say so on its last line.
This is a shortened version of chapter 6 of my book in progress, Agents You Can Leave Alone, which I am writing in public on Leanpub: https://leanpub.com/agents-you-can-leave-alone
Originally published on devaland.com: Scans, Sideways Pages and Readable Garbage: Why an AI Agent Misses Half a PDF
Top comments (0)