The Arabic PDF bug was never in my code — it was the library version

In my last post I wrote about why Arabic comes out of PDF extraction in reverse word order. A commenter asked the obvious follow-up question, and it is the one I want to answer properly:

Did you end up running a full bidi pass, or approximating with run-level heuristics? That mixed-direction case is usually where the heuristics fall apart.

Neither. And that turned out to be the whole point.

What I thought the problem was

The naive mental model goes like this: the PDF stores glyphs in the order they were painted, painting for RTL text runs right-to-left, so extraction hands you a line that reads last-word-first. Therefore you must reconstruct logical order yourself — which means implementing the Unicode Bidirectional Algorithm (UAX #9) over the positioned glyphs.

That is a genuinely hard thing to do, for a reason the commenter named exactly: UAX #9 needs a base paragraph direction, and the PDF never stored one. You have to infer it from the script of the runs. Get it wrong on a line like المبلغ: 128 SAR and your "fix" flips the Latin and the digits too. You have turned one bug into two, and the second one only shows up on invoices and CVs — which is to say, on almost every document a user actually cares about.

I wrote a chunk of that logic. It half worked, which is worse than not working, because half-working code makes you write more of it.

What the problem actually was

Then I ran the same PDF through the same code on a different machine and got correct output.

The extraction library was handing me visual order on one machine and logical order on the other. Same file, same code, different dependency version.

LibreOffice. Version 7.1 (2021), which is what a lot of stable server distributions still ship, reverses Arabic on PDF import — it gives you paint order. A modern LibreOffice extracts logical order directly. The server was on 7.1. My laptop was not.

PyMuPDF. Same class of bug, different library. Below 1.24, Arabic comes back visual and reversed. From 1.24 onward it comes back logical.

So the correct amount of bidi reconstruction code is zero. Both libraries already do it, in the versions where they do it. Everything I had written was scrambling text that had already arrived in the right order, on exactly the machines where the library was correct — which is why it looked like an intermittent bug and cost me months.

The trap that hid it for weeks

Here is the part worth stealing, because it is what made this so hard to see.

The pipeline had two engines: LibreOffice as primary, and pdf2docx (which sits on PyMuPDF) as a fallback. Sensible design. Then PyMuPDF 1.26.5 removed Rect.get_area(), a method pdf2docx 0.5.8 still calls:

# pdf2docx 0.5.8, somewhere in layout analysis
if bbox.get_area() / page_area > threshold:

AttributeError. The fallback engine crashed on import-time-adjacent code paths, the orchestrator caught it, and everything silently fell through to the other engine.

Nothing was logged as broken because nothing was broken from the caller's point of view — output came back, tests passed, files converted. But I could no longer tell which engine had produced any given file, so I was debugging Arabic ordering against an engine that had not run in weeks.

The fix was a two-line shim rather than pinning PyMuPDF backwards:

def _ensure_pdf2docx_compat():
    """pdf2docx 0.5.8 calls Rect.get_area(), removed in PyMuPDF 1.26.
    Restore it rather than pinning back to a version that returns visual order."""
    import fitz
    for cls in (fitz.Rect, fitz.IRect):
        if not hasattr(cls, 'get_area'):
            cls.get_area = lambda self: abs(self.width * self.height)

The general lesson: a silent fallback is a debugging tarpit. If your orchestrator can switch engines without telling you, the first thing to add is not a retry, it is a log line naming which engine produced the output.

One infrastructure footgun on the way

Upgrading LibreOffice on a server usually means dropping a newer build somewhere and repointing the binary:

ln -sf /opt/libreoffice26.2/program/soffice /usr/bin/soffice

ln -sf will happily create that symlink when the target does not exist yet. You get a dangling link, LibreOffice stops working entirely, and the error you get back has nothing to do with symlinks. Confirm the binary is there before repointing, and restart the service afterwards — a long-running process that resolved soffice at startup will keep using the old path until it does.

What is left after you delete the reversal code

Not nothing, but much less than you would expect. Two things still need doing.

Normalise presentation forms. Some PDFs encode Arabic using the Unicode presentation-form blocks (U+FB50–FDFF, U+FE70–FEFF) — the pre-shaped initial/medial/final glyph variants — instead of base letters. The text is technically correct and will look fine, but it is not searchable, will not match a query typed normally, and behaves badly in Word. NFKC folds them back:

import unicodedata
text = unicodedata.normalize('NFKC', text)

Note that this is the only normalisation you want here. NFKC is safe for this because the presentation forms have canonical decompositions to their base letters. Do not go further and start stripping diacritics; you will damage Quranic text and vowelised teaching material.

Mark direction in the output format. For DOCX, logical order in the string is not enough — Word needs to be told the run is RTL, or it renders correctly-ordered text with left-aligned paragraph flow, which looks subtly wrong to a native reader:

<w:rPr>
  <w:rtl/>
  <w:rFonts w:cs="Arabic Typesetting"/>
</w:rPr>
<w:pPr>
  <w:bidi/>
</w:pPr>

w:bidi on the paragraph, w:rtl on the run, and a complex-script font via w:cs — miss the last one and Word substitutes something that breaks the cursive joins.

The takeaway

If you are debugging reversed RTL text in an extraction pipeline, check your library versions before you write a single line of bidi logic. The odds are high that the layer below you already solved it and you are about to un-solve it.

And add a version matrix to your i18n tests. Not a version floor — a matrix. This class of bug is invisible in a test suite that only ever runs one version of the thing doing the extracting, and it will reappear the day someone deploys to a distribution that pins an older package.

I write these up as I hit them while building Confileo, a free PDF toolkit where Arabic support is the point rather than an afterthought. If you have a PDF that still comes out wrong, I would genuinely like to see it — the interesting bugs are always in the files, not in the spec.