Extract text from a PDF and you get far too many line breaks. pdfium (the engine behind Chrome's PDF viewer) returns a break after every visual line, so sentences are cut in the middle of paragraphs. Feed that to an LLM, a search index or a summarizer, and the broken line structure can hurt the results. Remove every break instead, and bullet lists and tables melt into one long sentence.
This post is about how the "Export text for AI" feature of PDF Privacy Checker (a Windows app that finds hidden text in PDFs) decides which line breaks are soft wraps inside a paragraph, using only character coordinates. It is the follow-up I promised at the end of the previous post (I rebuilt my "export PDF text for AI" feature three times in one day).
This is not a write-up of rules that worked the first time. I measured them on real PDFs for this post, found they were broken, fixed them and measured again. The measurements led to three fixes in the app.
TL;DR
- pdfium inserts a line break after every visual line. Removing all breaks erases paragraph boundaries; joining by character class or by sentence-ending punctuation melts lists and tables.
- Rebuild the visual lines from character boxes, then check each pair of neighbouring lines against eight conditions. Join only when all of them hold; when in doubt, keep the break.
- On my own 10 articles printed to PDF (3,751 labelled line breaks), it restores 89.8% of the soft wraps, and 95.2% of the joins are correct. Almost all wrong joins are lines of source code; only 4 are not.
- On 29 real business PDFs (invoices, quotes, vehicle papers), it was gluing table rows together. One extra condition fixed that. Some errors remain, such as short stacked form fields.
Line breaks from a PDF are not usable as-is
Here is a made-up store closing checklist I built for this post (the script at the end generates the PDF). This is what pdfium returns for page 1:
Closing Checklist
This sheet lists the checks to do in the last five minutes before
closing. If you follow them in order, you will not forget to lock
up, and the sales total will match the cash in the register. When
you are done, sign the back.
1 Count the cash in the register and compare it with the total on the
Daily Sales Sheet. If it does not match, call the manager.
2 Record the temperature of the fridge and the freezer. If one is
too warm, circle it in red for the morning shift.
- Manager: extension 11
- Security company: number on the receipt
- When in doubt, use the paper checklist
Denomination Count Amount
$20 bills 12 240.00
$5 bills 9 45.00
Quarters 40 10.00
(rest omitted)
The sample contains a title, a four-line paragraph, two numbered items with a hanging indent, three bullets, and a small cash-count table (denomination, count, amount). The breaks inside the paragraph and inside the numbered items should be joined. The breaks after the title, between the bullets and between the table rows should stay. The goal is to tell these two kinds apart.
A PDF has no notion of "this is the middle of a paragraph". It only says where each glyph is drawn. The meaning of a line break has to be inferred from positions.
The obvious approaches each break something
These are real outputs of simple rules, run on the sample PDF that the companion script generates.
1. Remove every line break
Closing Checklist This sheet lists the checks … sign the back. 1 Count the cash in the register … call the manager. 2 Record the temperature … morning shift. - Manager: extension 11 - Security company: number on the receipt - When in doubt, use the paper checklist Denomination Count Amount $20 bills 12 240.00 $5 bills 9 45.00 Quarters 40 10.00 Pull the shutter …
The title merges into the text, the bullets become one sentence and the whole table becomes one line. Is 45.00 the amount for the $5 bills, or the start of the next row? Neither a person nor an LLM can tell any more.
2. Join unless the line ends with a period (or ! ? :)
Closing Checklist This sheet lists the checks to do in the last five minutes before closing. …
- Manager: extension 11 - Security company: number on the receipt - When in doubt, use the paper checklist Denomination Count Amount $20 bills 12 240.00 $5 bills 9 45.00 Quarters 40 10.00 Pull the shutter down last …
It fails in both directions. Titles, bullets and table rows have no period, so they all merge. And whenever a sentence happens to end exactly at the right margin inside a paragraph, the paragraph is cut there. The first failure is dangerous; the second is merely ugly. Japanese, with 。 as its full stop, fails the same way.
3. Join when the next line starts with a lowercase letter
1 Count the cash in the register and compare it with the total on the
Daily Sales Sheet. If it does not match, call the manager.
This works well on this page, and its failures are mostly misses rather than false merges: it cannot join a wrap that is followed by a capital letter or a digit, such as a document name, a person's name, an amount or a date. Business text is full of those. It also has no equivalent for Japanese, which has no letter case; the closest thing there, joining when both sides are CJK characters, merged the title into the text and the table header into its first row on my Japanese version of this page. Letter case and character class tell you how to join (with or without a space), not whether to join.
4. Same baseline means same line
This one is about the earlier step of merging fragments. PDFs from Word's character spacing or from CAD tools often draw one visual line as many small strings, and pdfium inserts a break between them. Matching baselines sounds like the fix, but a line where the number "5" is set in a different size from the text has different baselines. I use vertical overlap instead.
All four look only at the string. The meaning of a line break lives in where the lines are, not in the text.
Rebuilding visual lines from character boxes
pdfium gives a box for every character: FPDFText_GetCharBox (the tight glyph bounds) and a "loose" box built from the advance width and the font height. I use the union of both, because with some fonts one of them collapses (an example appears later).
The pipeline:
PDF
↓ pdfium (characters and boxes; a break after every visual line)
character boxes
↓ step 1: merge fragments of the same visual line
visual lines (left, right, top, bottom, median character height h)
↓ step 2: check each pair of neighbouring lines against 8 conditions
each break: soft wrap or real break
↓ join (no space between CJK characters, one space in English)
text for the LLM (the report's full-text appendix keeps the original breaks)
Every threshold is expressed in units of h, the character height, so the same rules work for any font size. h is about 1.1 times the font size.
| Threshold | Step | Meaning |
|---|---|---|
| vertical overlap ≥ 0.5h | 1 | on the same height |
| next fragment within 2.5h to the right | 1 | continuation of the same line |
| gap < 0.35h | 1 | touching (join without a space) |
| line pitch 0.5–1.8h | 2 | normal line spacing, not a paragraph gap |
| height difference ≤ 0.3h | 2 | same text size |
| a gap ≥ 3h inside the line | 2 | column gap (a table row) |
| ≥ 90% of the longest line with the same left edge | 2 | the line is full |
| line width ≥ 12h | 2 | avoid short lines matching by accident |
| left-edge offset within 0.5h / −1.3 to −0.5h / +0.5 to 4h | 2 | aligned / 2nd line of an indented paragraph / hanging indent |
None of these come from the PDF specification. I set them once, then changed only what broke on real PDFs; the next sections say what. Also note that h is a height, so in terms of width it depends on the script: a full-width CJK character is about 0.9h wide, a Latin letter about 0.5h on average. "12h" is about 13 CJK characters or roughly 25 Latin ones.
Step 1: merge fragments of the same line
pdfium's breaks are not always visual line boundaries. If the next piece overlaps the current line by at least half its height and starts within 2.5h to the right, it is a fragment of the same line: joined with no space if touching, otherwise with one space. After this step, step 2 only has to compare whole visual lines.
Step 2: eight conditions for a soft wrap
For two neighbouring visual lines a (above) and b (below), the break after a is a soft wrap only if all of the following hold. For each condition, the right column says what would be wrongly joined without it.
| # | Condition | Without it, this gets joined |
|---|---|---|
| 1 | not vertical text | vertical CJK text (horizontal only) |
| 2 | line pitch 0.5–1.8h | across a paragraph gap, or to a distant note |
| 3 | nearly the same text height | a heading and its text, text and a small note |
| 4 | the two lines overlap horizontally | different columns or cells |
| 5 | neither line has a gap of 3h or more | table rows (a condition I added later) |
| 6 | a is full (≥ 90% of the longest line with the same left edge) and at least 12h wide | the short last line of a paragraph and the next paragraph; short bullets of equal width |
| 7 | b does not start with a bullet or a number | the next list item |
| 8 | the left edges are aligned, or form the 2nd line of an indented paragraph (CJK only), or a hanging indent of a numbered item | any other indentation (quotes, nesting) |
In code (from the companion script, a shortened version of the app's logic):
def is_wrap(a, b, lines, rules):
h = a["h"]
if (a["T"] - a["B"]) > 2.5 * h and (a["R"] - a["L"]) < 2 * h:
return False # 1. vertical text (out of scope)
if not (0.5 * h < a["B"] - b["B"] <= 1.8 * h):
return False # 2. unusual line pitch (paragraph gap)
if abs(a["h"] - b["h"]) > 0.3 * h:
return False # 3. different text size (heading, note)
if not (b["R"] > a["L"] and a["R"] > b["L"]):
return False # 4. no horizontal overlap (column, cell)
if "gap" in rules and (max_gap(a) >= 3 or max_gap(b) >= 3):
return False # 5. column gap (a table row)
block = [x["R"] for x in lines if abs(x["L"] - a["L"]) < 1.5 * h]
if a["R"] - a["L"] < 12 * h or a["R"] < 0.9 * max(block):
return False # 6. not a full line (paragraph end)
first, head = b["text"].lstrip(), a["text"].lstrip()
if first[:1] in LIST_HEADS or LIST_NUM.match(first):
return False # 7. next line starts a new item
dl = b["L"] - a["L"]
wide = is_wide(a["text"].rstrip()[-1]) and is_wide(first[0])
listy = head[:1] in LIST_HEADS or bool(LIST_NUM.match(head))
return (abs(dl) < 0.5 * h # same left edge
or (-1.3 * h <= dl < -0.5 * h and (wide or "cjk" not in rules)) # 2nd line of an indented paragraph
or (0.5 * h < dl <= 4 * h and listy)) # hanging indent of a numbered item
Condition 6 does most of the work. Lines inside a paragraph run to the right margin; the last line of a paragraph is short. It uses the typesetting itself. The "longest line" is taken only among lines with the same left edge, so that a long line elsewhere on the page (a table, another column) does not distort it.
The "2nd line of an indented paragraph" in condition 8 is a Japanese convention: when the first line is indented by one character, the second line starts one character further left. It originally applied to any script. Why it is now CJK-only comes below.
Recap. The whole method is four steps:
- Look at character coordinates, not at the string.
- Rebuild the visual lines from those coordinates (merging fragments of the same line).
- Check whether two neighbouring lines are a soft wrap inside a paragraph, using eight conditions.
- If any condition fails, do not join (keep the break).
How to join, and pdfium's own hyphen handling
A break that is a soft wrap gets replaced in one of three ways:
- between CJK characters: nothing (Japanese has no spaces between words)
- in English: one space
- a line-end "-" followed by a lowercase letter: remove the hyphen (undo hyphenation)
The third one almost never fires, because pdfium handles line-end hyphens itself. When a line ends with "-", pdfium does not insert a break there; it glues the two lines into one word. That hyphen comes back as code 0x02 from FPDFText_GetUnicode and as U+FFFE from get_text_range (FPDFText_IsHyphen returns 1 for it). In the sample, "con-" + "nected" on the English page came back as con\ufffenected.
This is where I found a bug in my own app. PDF Privacy Checker has a check for glyphs that extract as control characters, a sign of a tampered ToUnicode map. That check caught 0x02 and raised a red warning on ordinary English PDFs that simply had a hyphen at the end of a line. All five of my English articles printed from Chrome triggered it. Japanese business documents almost never end a line with "-", which is why my earlier tests had missed it.
The fix maps 0x02 with FPDFText_IsHyphen back to "-". The hyphen is kept, not removed: every case I found was a compound word such as off-page or macro-enabled, and removing it gives "offpage".
The first thing that broke on a real PDF
The "2nd line of an indented paragraph" rule used to apply to any script. On a publicly available English sample invoice (a "Sample Invoice Template"), it joined a table header to the next row:
(first rules)
Date Specific Service/Task Time by Task Cost of Task 7/14/2023 Met with individual to draft …
The next row started a little to the left of the header, exactly the shape of an indented paragraph's second line. That layout is common in English tables. A rule written for Japanese typesetting had fired on an English table, so I restricted it to CJK on both sides.
The companion script reproduces the same shape on page 2 (the heading of the example entries on the back of the sheet, and the first entry):
(first rules)
Examples of entries on the back of this sheet (one line per day) 7/14/2023 Closed at 22:10, cash matched, alarm set by M. Tanaka
(current rules)
Examples of entries on the back of this sheet (one line per day)
7/14/2023 Closed at 22:10, cash matched, alarm set by M. Tanaka
The cost: in an English paragraph with an indented first line, the wrap after the first line is now missed. In the measurement below, that was a difference of two line breaks.
Measuring it: table rows were being glued together
For this post I measured the rules on two kinds of PDF.
Prose PDFs: my own 10 articles (5 Japanese, 5 English), converted to minimal HTML and printed from Chrome. Because I have the source, I can label every break automatically: paragraphs, headings, list items, table cells and single code lines are "units", a break inside a unit is a soft wrap and a break between units is a real break.
Business PDFs: 29 real invoices, quotes, delivery notes and vehicle documents (49 pages) from my own work. There is no ground truth, so I judged every join the rules made by eye.
Both were measured with heading detection turned off. The app (v1.15.0) also uses the same coordinates to detect heading lines and keeps them out of line joining, but here I wanted to compare the wrap rules alone. Heading detection is the topic of the next post.
Going through the joins in the business PDFs, I kept seeing lines like this one, from a publicly available Canada Post sample invoice:
(before)
Parcels 97.98 97.98 Commercial/Smartmail Marketing 216.90 216.90 Specialized services 1,240.00 1,240.00 Shipments 168.13 168.13 Adjustments -50.00 -50.00
Five invoice rows on one line. Table rows are stacked at the same width, so they pass condition 6, "the line is full." Normal line pitch, same text size, same left edge: they meet every condition for a paragraph. Japanese quotes had the same problem ("parts 1.00 1,210 1,210" glued to the next row).
What separates a paragraph line from a table row is the gaps inside the line. A paragraph only has word spaces; a table row has wide gaps between columns. So I added condition 5: neither line may contain a gap of three character heights or more.
(after)
Parcels 97.98 97.98
Commercial/Smartmail Marketing 216.90 216.90
Specialized services 1,240.00 1,240.00
Shipments 168.13 168.13
Adjustments -50.00 -50.00
The results. The two percentages point in different directions. Recall asks: of every 100 breaks that really are soft wraps, how many did it join? Precision asks: of every 100 joins it made, how many were really soft wraps? With the current rules, it joins about 90 of every 100 real wraps, and about 95 of every 100 joins are correct.
Prose PDFs (10 articles, 3,751 labelled line breaks)
| Rules | Correct joins | Missed wraps | Wrong joins | Recall | Precision |
|---|---|---|---|---|---|
| First rules | 1,366 | 153 | 93 | 89.9% | 93.6% |
| Indent rule limited to CJK | 1,364 | 155 | 93 | 89.8% | 93.6% |
| + column gap (current) | 1,364 | 155 | 69 | 89.8% | 95.2% |
Business PDFs (29 real documents, 3,603 line breaks)
| Rules | Joined as soft wraps | Correct (by eye) | Wrong |
|---|---|---|---|
| First rules | 125 | ~13 | ~112 |
| Indent rule limited to CJK | 121 | ~13 | ~108 |
| + column gap (current) | 56 | ~13 | ~43 |
Missed wraps in the business PDFs were not measured: there is no ground truth, and judging all 3,603 breaks by eye was not realistic. The correct/wrong split is also by eye. Step 1 additionally merged 385 fragments; 379 of them are the fine background pattern of digits on two insurance certificates.
What the numbers say:
- Prose and business documents behave completely differently. On prose, 95% of joins are correct; on business documents, fewer than a quarter are correct even after the fix. Business documents have few real paragraphs, so most candidate lines are table rows and form fields. The feature is meant to restore wrapped paragraphs in prose; on forms and invoices the goal is not to join nicely but not to change the meaning. In these 29 documents, the table rows glued together, which do change the meaning, disappeared with the fix; the remaining ~43 are background patterns and short stacked fields, the kind that rarely leads to a misreading.
- In the prose PDFs, 65 of the 69 wrong joins are between lines of source code (these are programming articles). Code lines share a left edge and run long, which makes them look like a paragraph. Only 4 wrong joins do not involve code (3 between table cells, 1 heading into text); leaving code aside, precision is 99.7%.
- 129 of the 155 missed wraps are inside table cells. Cells are narrow and fail the 12h minimum in condition 6. That is on purpose.
- I printed the English articles both ragged-right and justified. Recall was 89.7% and 90.8%. I expected the 90% rule to hurt ragged-right English much more; the actual difference was about one point.
The "current" row also includes a small effect from a separate fix found along the way: in PDFs saved from Chrome, the kanji "一" and the em dash "—" were wrongly flagged as tiny text. The tiny-text check now uses the size the text is actually drawn at.
When in doubt, keep the break
The results show more missed wraps (155) than wrong joins (69). That is intended.
The two mistakes do different amounts of harm:
- A missed wrap leaves a paragraph split over two lines. Harder to read, same meaning.
- A wrong join merges two items or two rows into one sentence. Is 45.00 the amount for the $5 bills? The meaning changes.
For text that goes to an LLM, the second is much worse. Every condition leans towards keeping the break. In other words, I deliberately prefer false negatives to false positives: leaving one soft wrap untouched is less harmful than gluing two unrelated blocks together.
Where it does not work
Rebuilding lines and rebuilding reading order are separate problems. This only decides whether two neighbouring lines, in the order pdfium returns them, should be joined. In a two-column layout, the order of the columns is still the PDF's drawing order.
Other things the current rules cannot do:
- Vertical text is out of scope (tall, narrow lines are excluded first).
- Wraps inside table cells are not joined (the 129 misses above).
- Short stacked form fields can be wrongly joined. A company name above "TEL 0120-…" with the same left edge: both lines are full for their own little block, so they become one line. I tried "only lines at least 50–60% as wide as the longest line on the page". It removed those, but also the correct wraps in terms and conditions set in a narrow column and in the remarks field of a vehicle document, and recall on the prose PDFs dropped by about three points. Such joins put one space between two fields, so I judged the harm small and left them.
- Background patterns made of tiny text get joined. They carry no meaning anyway.
- A paragraph whose last line happens to be full, followed by a paragraph with no indent and no extra space, gets merged with it. Coordinates alone cannot tell. Page 1 of the script shows this:
Pull the shutter down last and push it by hand to check that it is fully closed; if it moves at all, pull it down again and test it. Set the alarm and leave. If it beeps, turn it off and try again.
- Ragged-right English can be cut after a line that happens to end short. On page 2 of the script, the second line of the note about deliveries ("…The store cannot take any") ends short because the next word, "responsibility", did not fit; it falls short of the 90% mark and the paragraph is split there.
- English paragraphs with an indented first line lose the wrap after the first line (the indent rule is CJK-only now).
- A real hyphen at a line end that pdfium does not mark gets removed: "self-" + "service" becomes "selfservice".
Companion script
It only needs pypdfium2 (about 200 lines, half of which build the test PDF). The script generates its own test PDF with the standard Helvetica font, so no font file is needed. Tested with Python 3.12 and pypdfium2 5.12.1. Run python article10_rejoin_en.py your.pdf to try it on your own file.
First, what it produces. Page 1 with the current rules:
Closing Checklist
This sheet lists the checks to do in the last five minutes before closing. If you follow them in order, you will not forget to lock up, and the sales total will match the cash in the register. When you are done, sign the back.
1 Count the cash in the register and compare it with the total on the Daily Sales Sheet. If it does not match, call the manager.
2 Record the temperature of the fridge and the freezer. If one is too warm, circle it in red for the morning shift.
- Manager: extension 11
- Security company: number on the receipt
- When in doubt, use the paper checklist
Denomination Count Amount
$20 bills 12 240.00
$5 bills 9 45.00
Quarters 40 10.00
Pull the shutter down last and push it by hand to check that it is fully closed; if it moves at all, pull it down again and test it. Set the alarm and leave. If it beeps, turn it off and try again.
The paragraph and the numbered items are joined, including the wrap before "Daily Sales Sheet"; the bullets and the table keep their breaks. The last line is the limitation described above (two paragraphs merged). With the first rules, the four table rows become one line:
Denomination Count Amount $20 bills 12 240.00 $5 bills 9 45.00 Quarters 40 10.00
Full script (article10_rejoin_en.py)
# -*- coding: utf-8 -*-
"""article10_rejoin_en.py - rejoin only the line breaks that are soft wraps inside a
paragraph, using character coordinates from pdfium (companion script for the article;
needs only pypdfium2).
python article10_rejoin_en.py - builds a test PDF and prints 3 versions
python article10_rejoin_en.py file.pdf - runs the current rules on your own PDF
A shortened version of the logic in PDF Privacy Checker (hidden_core 1.11.1).
Every threshold is written in units of h, the character height. They are not
derived from the PDF spec; they are rules of thumb tuned on real documents.
"""
import re
import sys
import pypdfium2 as pdfium
import pypdfium2.raw as R
LIST_HEADS = "・•●○■□◆◇-–—*※①②③④⑤⑥⑦⑧⑨⑩"
LIST_NUM = re.compile(r"^[(（]?\d{1,3}(?:[)）.．、:：]|\s|$)")
def is_wide(ch): # full-width (kana, kanji, full-width punctuation): join without a space
o = ord(ch)
return 0x3000 <= o <= 0x30FF or 0x4E00 <= o <= 0x9FFF or 0xFF00 <= o <= 0xFFEF
def page_lines(page):
"""Rebuild visual lines from pdfium's characters and boxes (step 1: merge fragments)."""
tp = page.get_textpage()
n = tp.count_chars()
chars = [chr(R.FPDFText_GetUnicode(tp.raw, i)) for i in range(n)]
for i in range(n): # pdfium returns a line-end "-" as 0x02 and joins the lines itself
if chars[i] == "\x02" and R.FPDFText_IsHyphen(tp.raw, i):
chars[i] = "-"
boxes = [None] * n
for i, ch in enumerate(chars):
if ch.strip() and not R.FPDFText_IsGenerated(tp.raw, i):
t, lo = tp.get_charbox(i), tp.get_charbox(i, loose=True)
boxes[i] = (min(t[0], lo[0]), min(t[1], lo[1]), max(t[2], lo[2]), max(t[3], lo[3]))
text = "".join(chars)
lines = []
for m in re.finditer(r"[^\r\n]+", text):
bx = [boxes[i] for i in range(m.start(), m.end()) if boxes[i]]
if not bx:
continue
hs = sorted(b[3] - b[1] for b in bx)
g = {"text": m.group(), "bx": bx, "h": hs[len(hs) // 2],
"L": min(b[0] for b in bx), "R": max(b[2] for b in bx),
"T": max(b[3] for b in bx), "B": min(b[1] for b in bx)}
cur = lines[-1] if lines else None
if cur: # step 1: overlaps vertically by half and continues within 2.5h to the right
hmin = min(cur["h"], g["h"])
overlap = min(cur["T"], g["T"]) - max(cur["B"], g["B"])
gap = g["L"] - cur["R"]
if overlap >= 0.5 * hmin and -0.3 * hmin < gap < 2.5 * hmin:
cur["text"] += ("" if gap < 0.35 * hmin else " ") + g["text"]
cur["bx"] += g["bx"]
cur.update(R=max(cur["R"], g["R"]), T=max(cur["T"], g["T"]),
B=min(cur["B"], g["B"]))
continue
lines.append(g)
return lines
def max_gap(line):
"""Largest horizontal gap between neighbouring character boxes, in units of h."""
bx = sorted(line["bx"])
return max([(bx[k + 1][0] - bx[k][2]) / line["h"] for k in range(len(bx) - 1)] or [0])
def is_wrap(a, b, lines, rules):
"""Is the break after line a a soft wrap? True only if none of 1-7 applies and the
left edges match one of three shapes (when in doubt, keep the break)."""
h = a["h"]
if (a["T"] - a["B"]) > 2.5 * h and (a["R"] - a["L"]) < 2 * h:
return False # 1. vertical text (out of scope)
if not (0.5 * h < a["B"] - b["B"] <= 1.8 * h):
return False # 2. unusual line pitch (paragraph gap)
if abs(a["h"] - b["h"]) > 0.3 * h:
return False # 3. different text size (heading, note)
if not (b["R"] > a["L"] and a["R"] > b["L"]):
return False # 4. no horizontal overlap (column, cell)
if "gap" in rules and (max_gap(a) >= 3 or max_gap(b) >= 3):
return False # 5. column gap (a table row)
block = [x["R"] for x in lines if abs(x["L"] - a["L"]) < 1.5 * h]
if a["R"] - a["L"] < 12 * h or a["R"] < 0.9 * max(block):
return False # 6. not a full line (paragraph end)
first, head = b["text"].lstrip(), a["text"].lstrip()
if first[:1] in LIST_HEADS or LIST_NUM.match(first):
return False # 7. next line starts a new item
dl = b["L"] - a["L"]
wide = is_wide(a["text"].rstrip()[-1]) and is_wide(first[0])
listy = head[:1] in LIST_HEADS or bool(LIST_NUM.match(head))
return (abs(dl) < 0.5 * h # same left edge
or (-1.3 * h <= dl < -0.5 * h and (wide or "cjk" not in rules)) # 2nd line of an indented paragraph
or (0.5 * h < dl <= 4 * h and listy)) # hanging indent of a numbered item
def rejoin(lines, rules):
out = lines[0]["text"] if lines else ""
for a, b in zip(lines, lines[1:]):
if not is_wrap(a, b, lines, rules):
out += "\n" + b["text"]
elif is_wide(out.rstrip()[-1]) and is_wide(b["text"].lstrip()[0]):
out = out.rstrip() + b["text"].lstrip() # Japanese: no space
elif out.endswith("-") and b["text"][:1].islower():
out = out[:-1] + b["text"] # undo hyphenation (a line-end "-" pdfium did not mark)
else:
out = out.rstrip() + " " + b["text"].lstrip() # English: one space
return out
# ---- test PDF (raw PDF, standard Helvetica; nothing is embedded) ----
def build_pdf(path):
en, en2 = [], []
def E(x, y, s, size=10, page=None):
s = s.replace("\\", "\\\\").replace("(", "\\(").replace(")", "\\)")
(en if page is None else page).append(f"BT /F1 {size} Tf {x} {y} Td ({s}) Tj ET")
E(72, 780, "Closing Checklist", 16)
for k, s in enumerate(["This sheet lists the checks to do in the last five minutes before",
"closing. If you follow them in order, you will not forget to lock",
"up, and the sales total will match the cash in the register. When",
"you are done, sign the back."]):
E(72, 750 - 14 * k, s)
for y, no, a, b in [(685, "1", "Count the cash in the register and compare it with the total on the",
"Daily Sales Sheet. If it does not match, call the manager."),
(651, "2", "Record the temperature of the fridge and the freezer. If one is",
"too warm, circle it in red for the morning shift.")]:
E(72, y, no); E(86, y, a); E(86, y - 14, b) # hanging indent
for k, s in enumerate(["- Manager: extension 11", "- Security company: number on the receipt",
"- When in doubt, use the paper checklist"]):
E(72, 610 - 14 * k, s)
for k, (a, b, c) in enumerate([("Denomination", "Count", "Amount"), # cash count
("$20 bills", "12", "240.00"),
("$5 bills", "9", "45.00"),
("Quarters", "40", "10.00")]):
E(72, 555 - 14 * k, a); E(230, 555 - 14 * k, b); E(330, 555 - 14 * k, c)
# limitation: a paragraph whose last line happens to be full, followed by a paragraph
# with no indent and no extra space
E(72, 480, "Pull the shutter down last and push it by hand to check that it")
E(72, 466, "is fully closed; if it moves at all, pull it down again and test it.")
E(72, 452, "Set the alarm and leave. If it beeps, turn it off and try again.")
E(72, 780, "Notes", 16, en2)
for k, s in enumerate(["Deliveries that arrive after closing must not be left at the back",
"door or in front of the shop. The store cannot take any",
"responsibility for goods that are left outside overnight, so ask",
"the driver to take them back."]):
E(72, 750 - 14 * k, s, page=en2)
E(72, 680, "The alarm panel shows a green light only when every door is con-", page=en2)
E(72, 666, "nected to the system; a red light means that one is still open.", page=en2)
E(82, 630, "Examples of entries on the back of this sheet (one line per day)", page=en2)
E(72, 616, "7/14/2023 Closed at 22:10, cash matched, alarm set by M. Tanaka", page=en2)
objs = ["<< /Type /Catalog /Pages 2 0 R >>", "",
"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>"]
kids = []
for ops in (en, en2):
data = "\n".join(ops).encode("latin-1")
objs.append(f"<< /Length {len(data)} >>\nstream\n".encode() + data + b"\nendstream")
objs.append(f"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 595 842] /Contents {len(objs)} 0 R "
"/Resources << /Font << /F1 3 0 R >> >> >>")
kids.append(len(objs))
objs[1] = f"<< /Type /Pages /Kids [{' '.join(f'{k} 0 R' for k in kids)}] /Count {len(kids)} >>"
out, offs = b"%PDF-1.7\n", []
for k, o in enumerate(objs, 1):
offs.append(len(out))
out += f"{k} 0 obj\n".encode() + (o if isinstance(o, bytes) else o.encode()) + b"\nendobj\n"
xref = len(out)
out += f"xref\n0 {len(objs) + 1}\n0000000000 65535 f \n".encode()
out += b"".join(f"{o:010d} 00000 n \n".encode() for o in offs)
out += f"trailer << /Size {len(objs) + 1} /Root 1 0 R >>\nstartxref\n{xref}\n%%EOF\n".encode()
open(path, "wb").write(out)
if __name__ == "__main__":
sys.stdout.reconfigure(encoding="utf-8")
if len(sys.argv) > 1:
for page in pdfium.PdfDocument(sys.argv[1]):
print(rejoin(page_lines(page), {"cjk", "gap"}), end="\n\n")
sys.exit()
build_pdf("article10_sample_en.pdf")
doc = pdfium.PdfDocument("article10_sample_en.pdf")
for no, page in enumerate(doc, 1):
lines = page_lines(page)
print(f"===== page {no}: pdfium line breaks as-is =====")
print(page.get_textpage().get_text_range().replace("\r\n", "\n"))
print(f"===== page {no}: first rules (indent rule for any script, no column-gap check) =====")
print(rejoin(lines, set()))
print(f"===== page {no}: current rules =====")
print(rejoin(lines, {"cjk", "gap"}))
print()
Wrap-up
- The meaning of a PDF line break lives in where the lines are, not in the text. Rebuild visual lines from character boxes and judge each break from its neighbours.
- Join only when every condition holds. A missed wrap is ugly; a wrong join changes the meaning.
- The rules only showed how they break once I measured them on real PDFs. The indent rule fired on English tables; "the line is full" fired on table rows.
- Prose and business documents give completely different numbers for the same rules. When you report numbers, say what kind of documents you measured.
Measuring for this post led to three fixes in PDF Privacy Checker: table rows are no longer glued together, a line-end hyphen no longer triggers a false control-character warning, and "一" and "—" in PDFs saved from Chrome are no longer flagged as tiny text. All three are in v1.15.0, which is now on the Store.
PDF Privacy Checker is on the Microsoft Store (detection is free; the text export for AI is part of a paid add-on). Files never leave your PC; it works fully offline. The source code is not public.
https://apps.microsoft.com/detail/9PLRJHFTPS53?hl=en-us&gl=US
Next, I plan to write about using the same coordinates to detect heading lines, so the text for AI keeps the document's structure.
About this article — The implementation and the writing were done together with Claude (Anthropic's AI). Most of the prose was drafted by Claude and checked by me. The decisions about which errors to fix in the product and which to leave as limitations, and the real business documents, are mine.
About the author
Okinawa Software Lab. I lead in-house digital transformation at a small company in Okinawa, Japan. I build the tools we need ourselves, and I publish PDF apps on the Microsoft Store that follow the same principle: everything happens on your own PC.
- Website: https://okinawasoftwarelab.com/en/
Top comments (0)