DEV Community

Cover image for PDF URLs to clean text and RAG chunks with one API call
Joshua Smith
Joshua Smith

Posted on

PDF URLs to clean text and RAG chunks with one API call

#ai #rag #api #python

Every RAG pipeline starts with "get the text out of these PDFs". pdf.js or pypdf will do it in five lines for a clean file on disk. The mess starts with the inputs people actually have: Google Drive share links, files with a password, 400-page manuals where you need pages 12 to 40, a login page that comes back with a 200 and a .pdf extension, and paragraphs broken into one line per visual row with hyphen-
ated words.

What "extract text" really involves

  1. Rewrite cloud share links (Google Drive, Dropbox, OneDrive, SharePoint) to direct-download URLs.
  2. Stream the download with a size cap and check the %PDF signature before parsing, so HTML error pages are reported, not parsed.
  3. Read the text layer page by page, reassemble lines into paragraphs, repair hyphenation.
  4. Pull metadata (title, author, dates, producer) from the info dictionary and XMP, and links from annotations.
  5. Detect scanned, image-only files and say so instead of returning empty strings.
  6. Chunk with overlap for embeddings, keeping the page number on each chunk.

The two-minute version

PDF Text Extractor does the list above with Mozilla's pdf.js, from a list of URLs, and returns JSON or Markdown.

  1. Paste PDF links into PDF URLs; share links work as they are.
  2. Pick an Output format (text, Markdown or both), optionally a Page range like 1-5, 8, 12-, a Chunk size for RAG, and a PDF password under Advanced.
  3. Click Start. Each file appears in the Output tab as it finishes.

From code 

from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("josh99smith/pdf-text-extractor").call(run_input={
    "urls": ["https://arxiv.org/pdf/1706.03762", "https://bitcoin.org/bitcoin.pdf"],
    "outputFormat": "markdown",
    "chunkSize": 1500,
    "chunkOverlap": 200,
})
for doc in client.dataset(run["defaultDatasetId"]).iterate_items():
    if doc["success"]:
        for chunk in doc["chunks"]:
            embed(chunk["text"], metadata={"url": doc["url"], "page": chunk["page"]})
Enter fullscreen mode Exit fullscreen mode

What you get back 

{
    "url": "https://bitcoin.org/bitcoin.pdf",
    "success": true,
    "status": "ok",
    "fileName": "bitcoin.pdf",
    "fileSizeBytes": 184292,
    "pageCount": 9,
    "wordCount": 1263,
    "hasText": true,
    "metadata": { "title": null, "creator": "Writer", "producer": "OpenOffice.org 2.4", "creationDate": "2009-03-24T17:33:15.000Z", "pdfVersion": "1.4", "encrypted": false },
    "links": ["www.bitcoin.org"],
    "text": "Bitcoin: A Peer-to-Peer Electronic Cash System\n\nSatoshi Nakamoto ...",
    "pages": [{ "page": 1, "text": "Bitcoin: A Peer-to-Peer ...", "charCount": 3049 }],
    "chunks": [{ "index": 0, "page": 1, "text": "Bitcoin: A Peer-to-Peer ...", "charCount": 1246 }]
}
Enter fullscreen mode Exit fullscreen mode

A scanned file returns status: "no-text-layer", a wrong password errorType: "encrypted", an HTML page errorType: "not-a-pdf". All of those are free records.

Cost and limits

$0.004 per successfully processed PDF, no per-page fee: a one-page flyer and a 400-page manual cost the same, and page ranges, passwords, share-link resolution and chunking are included. Scanned PDFs, invalid URLs, non-PDF responses, oversized and unreachable files cost nothing. There is no OCR in this version; if your corpus is scans, this is not the tool yet.

Use it from an AI agent

Add https://mcp.apify.com?tools=josh99smith/pdf-text-extractor to your MCP client and ask "extract the text of https://arxiv.org/pdf/1706.03762 and summarise section 3".

Disclosure: I built this Actor. Source: github.com/josh99smith/pdf-text-extractor. Parsing is powered by pdf.js (Apache-2.0).

Top comments (0)