DEV Community

Aryan Tailor
Aryan Tailor

Posted on

Understanding PDF Extraction: From Raw Text to Structured JSON

#data #programming #softwaredevelopment

Understanding PDF Extraction: From Raw Text to Structured JSON

PDFs look simple to us.

We open a document, see tables, headings, values, and paragraphs, and immediately understand how everything is organized.

A computer doesn't see it that way.

For a program, a PDF can be a collection of text fragments, coordinates, fonts, lines, images, and other objects. Extracting useful information from it is therefore much more complicated than simply reading the text.

In this article, I'll explain the process of going from a raw PDF to structured JSON, the problems that appear along the way, and some of the techniques that can make document extraction more reliable.

Why PDF extraction is difficult

A common assumption is:

PDF → extract text → convert to JSON
Enter fullscreen mode Exit fullscreen mode

In practice, it is closer to:

PDF
 ↓
Extract text and layout information
 ↓
Understand positions and relationships
 ↓
Identify sections and fields
 ↓
Handle tables and columns
 ↓
Normalize values
 ↓
Validate the extracted data
 ↓
Structured JSON
Enter fullscreen mode Exit fullscreen mode

The main problem is that a PDF is primarily designed to display information, not necessarily to represent that information semantically.

For example, a document might visually contain:

Invoice Number        INV-1024
Date                  12/08/2026
Total Amount          ₹45,000
Enter fullscreen mode Exit fullscreen mode

But the PDF might internally store these as separate text objects positioned at different coordinates.

The program has to figure out that:

"Invoice Number" → "INV-1024"
"Date"            → "12/08/2026"
"Total Amount"    → "₹45,000"
Enter fullscreen mode Exit fullscreen mode

1. Starting with raw text extraction

The first step is usually extracting text from the PDF.

Python has several useful libraries for this. One example is PyMuPDF, which provides access to both text and layout-related information.

A basic extraction can look like:

import fitz

doc = fitz.open("document.pdf")

for page in doc:
    text = page.get_text()
    print(text)
Enter fullscreen mode Exit fullscreen mode

This works surprisingly well for simple PDFs.

However, the result can become difficult to use when the document contains:

  • multiple columns
  • tables
  • merged cells
  • headers and footers
  • irregular spacing
  • text positioned manually
  • multi-page sections

For example, instead of getting:

Name: John
Age: 21
Country: India
Enter fullscreen mode Exit fullscreen mode

you might get text in an order that doesn't perfectly match the visual layout.

That's where layout information becomes important.

2. Coordinates are extremely useful

Instead of extracting only the text, we can extract information about where the text appears on the page.

Conceptually, each piece of text can be represented as:

Text + Position + Size
Enter fullscreen mode Exit fullscreen mode

For example:

"Invoice Number" → x=80, y=120
"INV-1024"       → x=250, y=120
Enter fullscreen mode Exit fullscreen mode

Now we can reason about their relationship.

If two pieces of text have similar Y coordinates, they are probably on the same row.

If one appears to the right of another, it may be its corresponding value.

This allows us to reconstruct parts of the document's visual structure.

3. Thinking in rows and columns

Suppose a PDF contains:

Item       Quantity       Price
Laptop     2              80000
Mouse      3              1500
Enter fullscreen mode Exit fullscreen mode

A text extractor might return individual text elements.

Instead of immediately converting them into JSON, we can first group them based on their coordinates.

A simplified representation might look like:

Row 1:
Item | Quantity | Price

Row 2:
Laptop | 2 | 80000

Row 3:
Mouse | 3 | 1500
Enter fullscreen mode Exit fullscreen mode

Then the table can be converted into something structured:

[
  {
    "item": "Laptop",
    "quantity": 2,
    "price": 80000
  },
  {
    "item": "Mouse",
    "quantity": 3,
    "price": 1500
  }
]
Enter fullscreen mode Exit fullscreen mode

The important idea here is that layout reconstruction happens before data structuring.

4. Detecting sections

Large documents usually aren't one giant table.

They contain different logical sections.

For example:

Document Details
Status
Declarant
Manifest Details
Container Details
Invoice Summary
Valuation
Duties
Enter fullscreen mode Exit fullscreen mode

A useful extraction pipeline therefore needs to identify where one section ends and another begins.

Instead of treating the document as one large block of text, we can represent it internally as:

Document
 ├── Document Details
 ├── Status
 ├── Declarant
 ├── Manifest Details
 ├── Container Details
 ├── Invoice Summary
 └── Valuation
Enter fullscreen mode Exit fullscreen mode

Once these sections are identified, each one can be processed independently.

This makes the final JSON much easier to work with.

5. Designing the JSON schema

Before writing the extraction logic, it helps to define what the final output should look like.

For example:

{
  "doc_details": {},
  "status": {},
  "declarant": {},
  "manifest_details": {},
  "container_details": {},
  "invoice_summary": {},
  "valuation_and_duties": {}
}
Enter fullscreen mode Exit fullscreen mode

The schema acts as a contract.

Instead of asking:

"What information can I find in this PDF?"

we can ask:

"Where should each piece of information go in my predefined structure?"

That distinction is important when building reliable document-processing systems.

6. Missing values should remain predictable

Real-world documents aren't always complete.

A particular field might not exist in a document.

Instead of changing

Top comments (0)