Reconstructing Arabic PDFs to Word (.docx): Solving Ligatures, BiDi & Tables in Python

How We Solved Arabic PDF-to-Word Reconstruction & OCR: Preserving Ligatures, Layouts & Tables

Converting PDF documents into clean, editable Microsoft Word (.docx) files has been a solved problem in English and Latin scripts for over a decade.

In the Arabic language, however, it has notoriously remained a nightmare.

If you have ever tried copying text or converting an Arabic PDF with mainstream tools, you have likely encountered this painful reality:

1. Disconnected letters: Isolated Arabic glyphs instead of fluid cursive words (م ح م د instead of محمد).
2. Reversed text: Numbers, punctuation, and entire sentences flipped left-to-right (.ءاروجلإا ليلدت).
3. Broken tables: Right-to-left columns inverted, borders misaligned, and multi-line cells chopped into unreadable fragments.

In this deep dive, we break down the exact engineering architecture we developed to solve Arabic PDF document reconstruction, which powers our production converter at hibdf.com and our commercial Python document engine.

---

The Core Technical Challenges of Arabic PDF Parsing

1. Complex Cursive Typography & Contextual Shaping

Unlike Latin characters which maintain static visual representations regardless of position, Arabic is an inherently cursive script. Every character has up to four distinct glyph forms:

• Isolated: ع
• Initial: عـ
• Medial: ـعـ
• Final: ـع

PDF format specifications (ISO 32000) store text as independent visual glyph coordinates on a two-dimensional canvas rather than semantic grammatical strings. When naive PDF parsers extract raw byte codes without a proper Unicode ToUnicode CMap table, they extract presentation-form glyph IDs rather than base Unicode code points.

2. Bi-Directional (Bidi) Algorithm Mismatches

Arabic is read right-to-left (RTL), but numbers, phone numbers, and embedded Latin terms (like brand names or code snippets) are read left-to-right (LTR).

When extracting text lines, if coordinates are sorted simply by increasing X coordinates ($X_0 \to X_1$), the words end up backwards. Sorting strictly by decreasing X coordinates breaks numbers like 2026 into 6202.

3. Visual Table Boundaries & Reading Orders

In standard Latin documents, columns read $C_1 \to C_2 \to C_3$ from left to right. In Arabic legal, financial, and academic documents, tables flow strictly from right to left:
$$\text{Header}_0 \text{ (Rightmost)} \to \text{Header}_1 \to \text{Header}_2 \text{ (Leftmost)}$$

---

The 4-Stage Reconstruction Pipeline

To achieve 100% fidelity without broken ligatures or inverted tables, we designed a sequential 4-stage pipeline:

[ Raw PDF / Scanned Document ]
             │
             ▼
┌───────────────────────────────────────┐
│  Stage 1: Layout & Boundary Detection │
│  - Line segmentation & Hough transform│
│  - Bounding box clustering (DBSCAN)   │
└───────────────────────────────────────┘
             │
             ▼
┌───────────────────────────────────────┐
│  Stage 2: High-Precision Arabic OCR   │
│  - Tesseract / Transformer OCR (ara)  │
│  - Confidence scoring per text token   │
└───────────────────────────────────────┘
             │
             ▼
┌───────────────────────────────────────┐
│  Stage 3: Contextual Shaping & Bidi   │
│  - Ligature recombination             │
│  - Unicode Normalization (NFC)        │
│  - ICU / pybidi visual re-ordering    │
└───────────────────────────────────────┘
             │
             ▼
┌───────────────────────────────────────┐
│  Stage 4: Native RTL DOCX Synthesis   │
│  - python-docx OpenXML generation     │
│  - Injected <w:bidi/> XML attributes  │
│  - Table column reversal & styling    │
└───────────────────────────────────────┘
             │
             ▼
[ Perfect Microsoft Word (.docx) File ]

---

Step-by-Step Implementation in Python

1. Unicode Normalization & Ligature Restoration

The first step is handling presentation forms. When PDFs contain pre-rendered ligatures (such as لا U+FEFB or لله U+FDF2), we map them back to their canonical decomposition code points:

import unicodedata
import re

def normalize_arabic_stream(raw_text: str) -> str:
    """
    Normalizes presentation forms and strips invalid control glyphs
    while preserving Arabic diacritics and semantic spacing.
    """
    # Normalize to canonical composition (NFC)
    text = unicodedata.normalize('NFC', raw_text)

    # Replace non-breaking spaces and zero-width spaces with standard spaces
    text = re.sub(r'[\u200B\u200C\u200D\uFEFF]', '', text)

    # Fix broken isolated hamzas often introduced by PDF text streams
    text = re.sub(r'\s+([ءئؤ])', r'\1', text)

    return text.strip()

2. Bi-Directional Line Re-ordering

To guarantee that embedded numbers, currencies (e.g. $9.00 or 150 JOD), and English words remain left-to-right while Arabic sentences flow right-to-left:

from bidi.algorithm import get_display
import arabic_reshaper

def prepare_bidi_paragraph(text: str) -> str:
    """
    Shapes Arabic glyphs and applies the Unicode Bi-Directional Algorithm.
    """
    # 1. Contextual shaping for continuous script flow
    reshaped_text = arabic_reshaper.reshape(text)

    # 2. Re-order characters according to BiDi algorithm
    bidi_text = get_display(reshaped_text)
    return bidi_text

3. OpenXML Document Synthesis with Native <w:bidi/> Support

The crucial step that 90% of open-source converters fail to implement is configuring Microsoft Word's underlying OpenXML schema for native Arabic RTL paragraphs and tables:

from docx import Document
from docx.shared import Pt, Inches, RGBColor
from docx.oxml import OxmlElement
from docx.oxml.ns import qn

def create_rtl_document() -> Document:
    doc = Document()

    # Configure document-wide section direction
    section = doc.sections[0]
    sectPr = section._sectPr

    # Add native RTL flag to section properties
    bidi_sect = OxmlElement('w:bidi')
    bidi_sect.set(qn('w:val'), '1')
    sectPr.append(bidi_sect)

    return doc

def add_arabic_paragraph(doc: Document, text: str, font_name="Calibri", font_size=12):
    p = doc.add_paragraph()
    p_pr = p._p.get_or_add_pPr()

    # Enforce paragraph-level Right-to-Left alignment
    p_bidi = OxmlElement('w:bidi')
    p_bidi.set(qn('w:val'), '1')
    p_pr.append(p_bidi)

    run = p.add_run(text)
    run.font.name = font_name
    run.font.size = Pt(font_size)

    # Set complex script font tags in OpenXML
    r_pr = run._r.get_or_add_rPr()
    r_fonts = OxmlElement('w:rFonts')
    r_fonts.set(qn('w:cs'), font_name)
    r_fonts.set(qn('w:ascii'), font_name)
    r_pr.append(r_fonts)

    return p

---

Live Production Deployment

We deployed this exact architecture into two commercial production channels:

1. hibdf.com (Free Web SaaS)

A fast, zero-install, cloud-optimized web application dedicated to converting Arabic PDFs into Word documents:

• Live URL: https://hibdf.com
• Features: Instant drag-and-drop, high-accuracy table reconstruction, and OCR support for scanned paperwork.

2. Fullstack Founder Suite (Python Source Code)

For developers and agencies looking to integrate automated document parsing, WhatsApp e-commerce, and client invoicing into their own stacks without monthly SaaS fees:

• Storefront: https://antigone16.gumroad.com/l/fullstack-pro-bundle
• Launch Discount: Use code LAUNCH20 for 20% OFF.

---

Conclusion

Handling Arabic digital typography is not about brute-force string manipulation; it requires respecting the mathematical and typographic rules of the language at both the Unicode abstraction layer and the OpenXML document format.

By combining layout bounding-box segmentation with contextual shaping and native OpenXML <w:bidi/> injection, we turned what used to be a broken output into an effortless, professional conversion workflow.

Engineered by **Ayoub AI Agency* & Antigravity Dev Studio.*