DEV Community: moamen abou elsaoud

Why I Stopped Writing Custom Python Scripts Just to Parse PDFs to Markdown

moamen abou elsaoud — Sat, 05 Sep 2026 00:09:52 +0000

Last Tuesday, I was setting up a quick RAG pipeline for our internal team documentation. The task seemed harmless: ingest about 30 PDFs into a vector store and let the team chat with them.

Two hours in, the bot started giving completely hallucinated answers.

When I checked the vector database chunks, I realized why: the PDF text extractors had completely butchered the tables. Column headers were floating three paragraphs away, footnote numbers were injected into the middle of API parameters, and H2 headers were flattened into plain text.

If you've ever tried to feed messy raw text into an LLM, you already know the pain.

🤦‍♂️ The Reality of "Quick" PDF Extraction

Most of us usually default to something like pypdf, pdfplumber, or heavy OCR wrappers. They work fine for simple single-column text, but the moment you throw a real-world document at them—say, a technical spec with tables and code blocks—things break down fast:

[PDF Visual Layout] ──► (Naive Text Extraction) ──► Plain Text Soup
                                                           │
                                                           ▼
               LLM has zero clue what belongs to what  ◄───┘

You spend half your day writing regex filters just to stitch broken tables back together or stripping out random page numbers.
It's tedious, fragile, and honestly, a waste of sprint time.
💡 The Pivot: Markdown First, Chunks Later
What LLMs actually understand well is Markdown.
Markdown gives your context window semantic boundaries without heavy HTML bloat:
Headers (#, ##) tell the embedding model where a new topic begins.
Native pipes (| col |) preserve tabular relationships.
Backticks keep code and parameters isolated.
Instead of writing a custom parsing script every time a non-standard PDF came along, I started testing out lightweight conversion tools to turn PDFs into clean .md files before doing any chunking.
While testing a few options, I came across this online PDF to Markdown converter.
What made me stick with it for quick prototype runs:

1. Tables actually survive: It maps complex multi-row tables into standard Markdown grid tables without scrambling columns.

2. Clean typography: It respects font sizes and translates them into proper Markdown header levels (H1, H2, H3) rather than just spitting out uppercase words.

3. No environment bloat: You don't have to battle Tesseract binaries or Docker OCR dependencies locally when you just need to inspect and clean a document quickly.

🛠️ The Workflow That Saved My Pipeline
Here is how simple chunking becomes once you feed your pipeline clean Markdown instead of raw extracted strings:
code

from langchain_text_splitters import MarkdownHeaderTextSplitter

# 1. Load the clean Markdown file you exported
with open("api_documentation.md", "r", encoding="utf-8") as f:
    clean_markdown = f.read()

# 2. Tell the splitter to preserve structural hierarchy
headers_to_split = [
    ("#", "Module"),
    ("##", "Endpoint"),
    ("###", "Parameters")
]

splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split)
sections = splitter.split_text(clean_markdown)

# Each chunk now has its parent headers attached as metadata!
print(f"Total structured chunks: {len(sections)}")
print(f"Sample chunk metadata: {sections[0].metadata}")

Notice the difference? Because the document was converted to Markdown first, the splitter automatically knows where endpoints start and stop. No orphaned text, no mangled tables.
💭 Bottom Line
We often overcomplicate pipelines by writing custom parsers for things that already have simple fixes. If your LLM or documentation migrations are choking on messy PDFs, stop feeding them raw strings.
Give the md-convert.org PDF to Markdown tool a spin on one of your trickiest PDF documents and see if it saves you the regex headache.
How do you guys handle PDF ingestion in your stacks? Are you sticking to self-hosted OCR pipelines, or using pre-processing utilities? Drop your setup below!

Building Local-First Web Apps: Parsing HTML and PDFs to Markdown in the Browser

moamen abou elsaoud — Wed, 26 Aug 2026 21:40:12 +0000

Local-first and privacy-focused web utilities are having a massive comeback. With browser engines becoming faster and WebAssembly/Web Workers maturing, there is rarely a reason to push sensitive user documents to an external backend for simple conversions.

While building MD-Convert (a zero-upload document to Markdown converter), I explored how to parse real-world documents into clean Markdown entirely on the client side.

Here is a breakdown of the core architecture and libraries that make purely in-browser document processing possible.

1. Converting Web Articles with Readability + Turndown

Converting messy web markup into clean Markdown involves two distinct steps:

Content Extraction: Stripping ads, navbars, sidebars, and trackers.
HTML-to-Markdown Transformation: Translating semantic DOM nodes into markdown tokens.

Mozilla’s @mozilla/readability paired with turndown is an incredible combination for this:

import { Readability } from '@mozilla/readability';
import TurndownService from 'turndown';

function htmlToCleanMarkdown(rawHtmlDocument, sourceUrl) {
  // 1. Extract pure article content
  const reader = new Readability(rawHtmlDocument);
  const article = reader.parse();

  if (!article || !article.content) {
    throw new Error('Unable to extract main content');
  }

  // 2. Initialize Turndown
  const turndownService = new TurndownService({
    headingStyle: 'atx',
    codeBlockStyle: 'fenced'
  });

  // Ensure image URLs remain absolute
  turndownService.addRule('absoluteImages', {
    filter: 'img',
    replacement: (content, node) => {
      const src = node.getAttribute('src');
      const alt = node.getAttribute('alt') || '';
      if (!src) return '';
      try {
        const absoluteUrl = new URL(src, sourceUrl).href;
        return `![${alt}](${absoluteUrl})\n\n`;
      } catch {
        return `![${alt}](${src})\n\n`;
      }
    }
  });

  return turndownService.turndown(article.content);
}

Offloading Heavy PDF Parsing to Web Workers Parsing large PDFs using pdf.js on the main thread is a recipe for UI freezes and dropped frames. The solution is running the extraction pipeline inside a dedicated Web Worker. Here is a simplified pattern for extracting selectable text layers asynchronously:

// worker.js
import * as pdfjsLib from 'pdfjs-dist/legacy/build/pdf';

self.onmessage = async (e) => {
  const { arrayBuffer } = e.data;

  try {
    const loadingTask = pdfjsLib.getDocument({ data: arrayBuffer });
    const pdf = await loadingTask.promise;
    let fullText = '';

    for (let pageNum = 1; pageNum <= pdf.numPages; pageNum++) {
      const page = await pdf.getPage(pageNum);
      const textContent = await page.getTextContent();

      const pageText = textContent.items
        .map((item) => item.str)
        .join(' ');

      fullText += `## Page ${pageNum}\n\n${pageText}\n\n`;
    }

    self.postMessage({ status: 'success', markdown: fullText });
  } catch (error) {
    self.postMessage({ status: 'error', message: error.message });
  }
};

The Big Advantages of Client-Side Processing
Zero Server Costs: The application can run entirely on static hosting (like Cloudflare Pages or GitHub Pages).
Absolute Privacy: User data, API keys, or confidential spreadsheets never cross the network.
Instant Latency: Conversions happen in-memory without queue waiting times.
Thoughts & Edge Cases?
The main challenge with 100% in-browser parsing remains edge cases—like complex multi-column PDF layouts or non-OCR scanned documents.
Have you built any client-side/local-first tools recently? What libraries do you prefer for client-side parsing? Let's discuss in the comments! 👇

Why I Built a 100% In-Browser, Privacy-First Markdown Converter

moamen abou elsaoud — Tue, 18 Aug 2026 21:56:07 +0000

Working with Markdown is great—it is the universal lingua franca for READMEs, technical documentation, static site generators, and now LLM prompt context (RAG pipelines).

However, getting data into Markdown from real-world documents is often a pain.

We've all been there:

Installing heavy CLI tools like Pandoc and dealing with missing LaTeX or OS dependencies.
Uploading confidential work PDFs or financial spreadsheets to sketchy online converters that process files on unknown servers.
Manually cleaning up messy tables copied from Excel or HTML markup.

To solve this friction, I built MD-Convert — a free, lightweight, privacy-first web utility designed to convert almost any document into clean Markdown directly inside your browser.

The Core Philosophy: 100% Client-Side Privacy

The biggest drawback of modern cloud file converters is data privacy. When you upload a company document, an .ipynb notebook containing proprietary code, or an internal spreadsheet to a typical web converter, your data leaves your machine.

With MD-Convert (md-convert.org), all parsing algorithms and conversions run 100% locally in your browser.

🔒 Zero Server Uploads: Your files are parsed in-memory on the client machine.
⚡ Zero Installation: No pip install, no brew install pandoc, and no Node.js scripts needed.
🚀 Instant Processing: Conversions happen in milliseconds without waiting in server processing queues.

What Can You Convert? (16+ Supported Formats)

Here is a breakdown of the conversion workflows supported out of the box:

1. Documents & Notebooks to Markdown

PDF to Markdown: Extracts structured headers, paragraphs, and text without manual formatting (PDF to Markdown).
DOCX to Markdown: Converts Microsoft Word files, retaining headings, bold/italic typography, and lists.
Jupyter Notebooks (.ipynb): Renders code blocks and markdown cells cleanly into a unified documentation file.

2. Spreadsheets & Datasets to Markdown Tables

Converting spreadsheets to Markdown tables usually results in broken syntax. MD-Convert properly parses and formats:

Excel (.xlsx) & ODS
CSV & TSV
Structured Data: JSON, YAML, and XML files into formatted Markdown code blocks or structured tables.

3. Web & Migration

Live URLs & HTML: Strips boilerplate, CSS, and navigation to extract clean article Markdown.
Evernote (.enex): Seamlessly migrate old notes into personal Markdown vaults like Obsidian or Logseq.

Example: Quick Document Conversion Workflow

Instead of writing a custom Python parsing script:


bash
# Old workflow: Dependencies + CLI flags
pip install pdfminer.six
pandoc document.docx -f docx -t markdown -o output.md