Turn PDFs into page-aware text chunks for your RAG pipeline (Node.js + pdf.js)

PDF parsing is the boring part of every RAG project. Line breaks in the middle of sentences, headers and footers mixed into the text, no page numbers to cite. This post shows a small, dependency-light way to turn a PDF into page-aware text chunks in Node.js, plus the pitfalls I hit while testing it.

Extract text per page

unpdf (MIT) wraps Mozilla's pdf.js and runs in Node.js (22+ for the default build), Deno, Bun and serverless workers:

npm install unpdf

// pdf.mjs  (run with: node pdf.mjs)
import { extractText, getDocumentProxy } from 'unpdf';

const url = 'https://arxiv.org/pdf/1706.03762';
const buffer = new Uint8Array(await (await fetch(url)).arrayBuffer());
const pdf = await getDocumentProxy(buffer);
const { totalPages, text: pages } = await extractText(pdf, { mergePages: false });

// Collapse whitespace: line breaks inside sentences become spaces.
const clean = (t) => t.replace(/\s+/g, ' ').trim();

mergePages: false returns one string per page, which you need if you want answers like "see page 12".

Chunk for embeddings

// ~1,000-character chunks split after sentence-ending punctuation, ~100 characters of overlap,
// and the page number kept for citations.
function chunk(text, page, size = 1000, overlap = 100) {
  const sentences = text.split(/(?<=[.!?])\s+/);
  const chunks = [];
  let current = '';
  for (const s of sentences) {
    if (current && current.length + s.length + 1 > size) {
      chunks.push({ page, text: current.trim() });
      current = current.slice(-overlap);
    }
    current += `${s} `;
  }
  if (current.trim()) chunks.push({ page, text: current.trim() });
  return chunks;
}

const chunks = pages.flatMap((t, i) => chunk(clean(t), i + 1));
console.log(`${totalPages} pages, ${chunks.length} chunks`);

On the "Attention Is All You Need" paper this printed 15 pages, 49 chunks (tested October 13, 2026). Each chunk is { page, text }, ready to embed and upsert into your vector database with the page as metadata.

Pitfalls I hit while testing

• Don't split sentences with a "match everything up to a period" regex. My first version used text.match(/[^.!?]+[.!?]+(\s|$)/g) and silently dropped fragments: "28.4 BLEU" became "4 BLEU". Splitting after ., ! or ? followed by whitespace keeps every character.
• Don't blindly join hyphenated line breaks. Removing - + line break also glued real hyphenated words ("English-to-German" became "Englishto-German"). Either leave hyphens alone or only join when a dictionary says the joined word exists.
• Headers, footers and author blocks end up in the text. The first chunk of the paper starts with a copyright notice and the authors' e-mail addresses. Strip repeated lines that appear on every page, and drop personal details such as e-mails if your use case doesn't need them.
• Headings are lost in plain text extraction. If you need Markdown headings, pdf.js exposes font sizes per text item (page.getTextContent()), so you can treat lines with a larger font as headings.
• Tables and multi-column layouts come out as text in the PDF's internal order, which is usually but not always the reading order.

Scanned PDFs need OCR

A scanned PDF is just images, so extractText returns empty strings. Detect that case (for example, fewer than ~20 characters on a page) and send those pages to OCR, such as Tesseract, or OCRmyPDF to add a text layer first. OCR quality depends heavily on the scan: clean 300 DPI scans work well, while phone photos, handwriting and small print do not.

---

This post was written with AI assistance. The code was run and checked on October 13, 2026 (Node.js 24, unpdf 1.8.1).