sort=True put both columns of a real arXiv paper on the same line

Before PDFs go into our internal RAG index, they pass through a preprocessing step my team owns, and the files are not allowed to leave the network. I sat through three months of a data-compliance cleanup once, so "upload it to a hosted parser and see" is not an option I reach for. That leaves the local Python stack, and the tip that comes up most for two-column documents is PyMuPDF's page.get_text(sort=True), which re-orders text by position on the page. I tested it before adopting it. On a real paper it did the opposite of what I wanted.

What sort=True did to an arXiv paper

The paper is RAG-Safety-Bench: Reliable Evaluation of Retrieval-Augmented LLM Safety by Adithiyan Rajan Indira Saravanan and Kathleen C. Fraser (arXiv:2609.11758, CC BY 4.0), a standard two-column LaTeX layout. I took pages 1–2 and extracted them twice, once in default order and once with sort=True. On the non-empty lines I counted three things: lines with a run of six or more spaces between two words, lines ending in a lowercase letter followed by a hyphen, and lowercase words with a year, a month abbreviation or the arXiv ID stuck to their end.

Default order gives 199 lines, no wide gaps, 41 hyphenated line ends, nothing glued, and the right column order, because LaTeX writes the left column before the right one and stream order already matches reading order. With sort=True there are 122 lines, and 47 of them contain a run of six or more spaces, which is what a left-column line and a right-column line pasted side by side look like. The vertical arXiv stamp in the left margin got merged into body text as well, as four glued tokens: reduce2026, onSep, to[cs.CL] and arearXiv:2609.11758v1. One sorted line on page 1 reads "uments can increase reliability and reduce2026", another "tion (RAG) can have unintended side effects onSep". reduce2026 would be embedded as a token. The drop from 41 to 26 lines ending in a hyphen is not an improvement either; the hyphenated fragments are still there, now in the middle of merged lines.

And a file where default order is the wrong one

That doesn't make default order safe. I generated a two-column test report with made-up content whose content stream writes a left line, then the right line at the same height, then the next left line. Nothing on the rendered page tells you the stream is in that order. pypdf and PyMuPDF's default both follow the stream and alternate columns every line.

I also ran it through ImgIng's Extract PDF content, a browser tool whose workspace states the file is read only in the browser and never uploaded, which is the property I need. Its output doesn't depend on stream order at all: this file and a copy written column by column gave identical TXT. The first three lines of each column came out correctly as two columns. Lines 4 to 7 were fused, left then right, with nothing between them, which produced strings like "Finance4. Result" and "differs byto 2":

On the arXiv paper the same tool kept the column order, left the margin stamp out, and moved the second author's affiliation block to after the left column's text. So four extraction paths, two files, and not one path right on both. That changed what I was building. Instead of picking a better extractor, I wanted a check that runs after whichever one we use.

A check that uses the PDF's own geometry

For files with native text, the PDF already knows where every line sits. The check pulls each body line with its bounding box from PyMuPDF's get_text("dict"), skips rotated lines (that is how the margin stamp stays out), anything in the top or bottom 8% of the page, and lines shorter than 8 characters once whitespace is removed. What's left is tagged by which side of the page centre it sits on (excerpt):

x0, y0, x1, y1 = line["bbox"]
if x1 < w / 2:
    tagged.append(("L", text))
elif x0 > w / 2:
    tagged.append(("R", text))

A line that straddles the centre, like a full-width title, gets no tag at all. Then an extractor's output is read two ways. The first asks whether an output line, with whitespace squashed out, contains one complete left line and one complete right line side by side, filling the row. The second walks the output in order, keeps only lines it can match to a tagged line, and counts how often consecutive ones jump between columns. The core of the first test (excerpt, s is the squashed output line):

for side, text in cols:
    if text in s:
        best[side] = max(best[side], len(text))
if best["L"] and best["R"] and best["L"] + best["R"] <= len(s):
    hits.append(raw.strip())

Running both tests over all four extraction paths, on the paper and on the test report:

Extractor	arXiv: mixed lines	arXiv: L/R jumps	Test report: mixed lines	Test report: L/R jumps
pypdf	0	11 over 169	0	18 over 20
PyMuPDF default	0	11 over 182	0	18 over 20
PyMuPDF sort=True	50	44 over 56	9	1 over 2
browser extractor	0	9 over 169	6	2 over 8

On the arXiv pages the sorted output has 50 mixed lines and nothing else has any. On the test report, default extraction has no mixed lines but jumps columns 18 times in 20 lines, and the browser tool's 6 fused lines are all caught. Default pypdf on the paper jumps 11 times over 169 lines, which is roughly the cost of an author block, figure captions and one column break per page. In our pipeline a single mixed line quarantines the document, and so does a jump rate far above that baseline. I don't have enough files yet to put a hard number on "far".

It has limits. It only works when the PDF has native text; scanned pages need an OCR step that returns boxes first. Lines under 8 characters are skipped, and a line that appears identically in both columns can't be attributed, so it is dropped. My first version flagged 17 lines on the paper that were fine, among them the identical "Faculty of Engineering" and "University of Ottawa" lines in the two author blocks; the second still flagged 6 where a short line from one column happened to sit inside a line from the other. It also says nothing about the 41 line-end hyphen breaks every extractor kept on these two pages, which need their own pass before chunking. The browser tool mentioned above is at https://imging.ai/

Comments

[Raknaos]

That sort=True default is a nasty one because it works on most PDFs — the failure only shows up on two-column layouts where reading order matters, which is exactly the papers worth indexing. The heuristic-first approach (column detection, fall back to sort) is the pragmatic middle ground; running the full layout model on every page is expensive when 90% of the corpus is single-column.

Out of curiosity: did you check the reading order the extracted text ended up in, or just that the lines were merged? We had a case where sort=True didn't merge columns but interleaved them mid-paragraph, which was worse because the RAG chunks looked plausible while containing fragments from two different columns. A quick sanity check that helped: extract the abstract and compare its first sentence against the raw text near the title.