Convert a 40-page report to Markdown and split it into chunks. If the report has a running title at the top and "Confidential · Page 17 of 40" at the foot, count how many of your chunks open with the one and close with the other. In a retrieval pipeline that text is noise that turns up everywhere. It nudges every chunk's embedding toward the same point, and a search for "confidential" now matches all 40 pages.
People ask for a switch for this on most PDF-to-Markdown projects: docling, markitdown, and marker, where the request was closed as not planned in November 2026. PyMuPDF's maintainer gave the honest version back in 2023: "There is no (reliable / foolproof) way to identify headers or footers: it's all just text on the page." Nothing in a PDF labels a line as a header (tagged PDFs aside). Every tool guesses.
So the useful question is what a given tool's guess catches and what it misses. Below is both, measured, for the stripHeadersFooters option in pdf2md's npm packages, plus a short script for the case it misses.
The test
Twelve PDFs, six pages each, and each one changes a single thing: a fixed title in the top margin, a "Page N of 6" footer, a book-style alternating head, a sentence that repeats at the top of every page's body, a table that continues across pages, and so on. They're generated by a short reportlab script, printed in full at the end of this article.
Every count is how many of the six copies made it into the Markdown, using @dudko.dev/pdf-to-md-core 0.3.6. The last column is the script further down.
| what repeats | false |
default | true |
script |
|---|---|---|---|---|
| fixed title in the top margin | 6 | 1 | 1 | 1 |
| title left + "Internal draft" right, same line | 6 | 1 | 1 | 1 |
| book style: title on even pages, chapter on odd | 6 | 4 | 4 | 4 |
| "Confidential. Do not distribute." at the foot | 6 | 1 | 1 | 1 |
| "Page N of 6" alone at the foot | 0 | 0 | 0 | 0 |
| a bare page number at the foot | 0 | 0 | 0 | 0 |
| "Confidential. Do not distribute." + "Page N of 6", one line | 6 | 6 | 6 | 0 |
| a sentence that is the first line of every page | 6 | 1 | 1 | 6 |
| a sentence in the middle of every page | 6 | 6 | 6 | 6 |
| a sentence that is the last line of every page | 6 | 1 | 1 | 6 |
| a table's header row, repeated on each page | 6 | 6 | 6 | 6 |
| a one-off heading sitting in the header band | 1 | 1 | 1 | 1 |
What the table says
It's on by default. In the npm packages, leaving the option out gives the same output as stripHeadersFooters: true. The CLI's --strip-headers-footers flag doesn't change anything, and the CLI has no flag to turn stripping off (--no-strip-headers-footers is an unknown option). To keep the furniture, pass stripHeadersFooters: false from code.
It keeps one copy. The first copy of each repeated line stays, which usually leaves the document title once at the top. The book-style case keeps four lines: the title once, and each of the three chapter names once, because those differ from each other. If your chunks carry the title as metadata anyway, drop the survivor yourself.
Standalone page numbers go regardless. "Page 3 of 6" on a line of its own is removed by a separate cleanup (removePageNumbers, on by default). That's why those rows read 0 even with header stripping off.
The miss: fixed text and a page number on one line. "Confidential. Do not distribute." on the left and "Page 3 of 6" on the right come out as one Markdown line. That line is different on every page, so a "same text on every page" test never fires, and the page-number cleanup leaves it alone because it isn't a bare number. All six copies stay, whatever you set. It's a layout you'll find on plenty of contracts and reports.
It goes by position in the text flow, not by distance from the edge. A sentence that opens the body of every page was treated as a header and dropped on pages 2 to 6. I moved it 60, 90, 120, 160 and 220 points down from the top edge, and it went every time. So if the first line of each page is real content that genuinely repeats (a "provisional" notice, a standing disclaimer you have to keep), it goes too. The same sentence in the middle of the page is left alone.
What it didn't break here: a table continued over six pages kept all six header rows and all sixty data rows, and a one-off heading in the header band was kept because it doesn't repeat. That second one is the failure reported in docling #3693, where a heading at the top of a page was dropped as page furniture.
A fix for the numbered footer
Mask the digits before comparing. Confidential. Do not distribute. Page 3 of 6 and … Page 4 of 6 both become … Page # of #, and that repeats. A line whose masked form repeats on page edges but whose actual text changes is a numbered running line, and every copy goes. A line that repeats exactly keeps its first copy, like the built-in pass.
// edges.mjs
// Remove running headers/footers that carry a page number, which a plain
// "same text on every page" test cannot see: "Confidential. Page 3 of 40" and
// "Confidential. Page 4 of 40" differ only in their digits.
// Written against the public API of @dudko.dev/pdf-to-md-core.
import { readFile } from 'node:fs/promises';
import { convert, detect } from '@dudko.dev/pdf-to-md-core';
const shape = (line) => line.replace(/[*_#]/g, '').replace(/\d+/g, '#').trim();
const isTable = (line) => line.trimStart().startsWith('|');
// first and last non-empty line of a page; table rows are never candidates
function edges(lines) {
const text = lines.filter((l) => l.trim());
return [...new Set([text[0], text[text.length - 1]])].filter((l) => l && !isTable(l));
}
export async function toMarkdownWithoutFurniture(pdf, { minPages = 3 } = {}) {
const { pageCount } = await detect(pdf);
const pages = [];
for (let n = 1; n <= pageCount; n++) {
const { markdown = '' } = await convert(pdf, { pages: [n] });
pages.push(markdown.split('\n'));
}
// on how many pages does each edge shape occur, and in how many spellings?
const count = new Map();
const spellings = new Map();
for (const p of pages)
for (const l of edges(p)) {
const s = shape(l);
count.set(s, (count.get(s) ?? 0) + 1);
spellings.set(s, (spellings.get(s) ?? new Set()).add(l));
}
const furniture = (l) => (count.get(shape(l)) ?? 0) >= Math.min(minPages, pageCount);
const numbered = (s) => spellings.get(s).size > 1; // only the digits differ
// keep the first occurrence of a fixed line (it is often the document's title);
// drop every occurrence of a line whose number changes from page to page
const kept = new Set();
return pages
.map((p) => {
const e = new Set(edges(p));
return p.filter((l) => {
if (!e.has(l) || !furniture(l)) return true;
const s = shape(l);
if (numbered(s) || kept.has(s)) return false;
kept.add(s);
return true;
}).join('\n').trim();
})
.filter(Boolean)
.join('\n\n');
}
if (import.meta.url === `file://${process.argv[1]}`) {
const pdf = new Uint8Array(await readFile(process.argv[2]));
process.stdout.write((await toMarkdownWithoutFurniture(pdf)) + '\n');
}
npm i @dudko.dev/pdf-to-md-core
node edges.mjs report.pdf > report.md
On the twelve fixtures it removed all six numbered footers and matched the built-in result on every fixed header and footer. Set the furniture lines aside and its output had exactly the same words as a whole-document conversion on all twelve, and all 60 table rows came through.
It also kept the repeated body sentences that the built-in pass dropped. That's not cleverness. Converted one page at a time, the body of each fixture page comes out as a single paragraph, so the repeated sentence is the start of a long line that differs on every page, and the script never sees it as a line of its own. Here that's the outcome you'd want. On a page where a repeated sentence does stand alone at the edge, the script will treat it as furniture just like the built-in pass does.
Before you trust it:
-
It converts page by page (
pages: [n]), because page markers aren't a reliable way to split the output (next section). That's one conversion per page. The parser also sizes headings against what it can see, so a heading can come out a level different from a whole-document run: in a separate eight-page test,## Appendix Aon the last page became# Appendix Awhen that page was converted alone. If heading levels matter downstream, keep the whole-document Markdown and use the script only to decide which lines to remove. - It works on Markdown lines, and those are paragraphs. A footer set close enough to the body to be merged into the last paragraph isn't a separate line, so the script won't see it. In the fixtures the footer sits 36 pt above the bottom edge and always came out as its own line.
- A year in the title is not a page number. "Harbor Water Quality Survey 2025" contains a number, but it's spelled the same on every page, so it counts as a fixed line and keeps one copy. The first version of this script got that wrong and deleted the title everywhere.
Page markers and tables
If you'd rather split whole-document Markdown on includePageMarkers (<!-- Page N -->), count the markers first. On the fixture where every page holds a ruled table, six pages produced one marker. The text-only fixtures got all six. Check that the marker count matches pageCount before you rely on them.
What I'd do
For RAG, leave stripping on (it already is). If your documents have a numbered footer, run the masked-digit pass on top. Before trusting either on a new corpus, convert one document twice, with stripping on and off, and diff the two. The diff is exactly what the tool decided was furniture, and reading it takes a minute. Neither approach is exact, because what counts as a header is a judgement about layout. The diff is how you check the judgement.
These measurements are of the npm packages. pdf2md also runs in the browser at pdf2md.dudko.dev/app (nothing is uploaded) and over an HTTP API that takes the same stripHeadersFooters option. I didn't measure the API's default for this article, so if it matters, set the option explicitly. The packages are free for noncommercial use under PolyForm Noncommercial 1.0.0, with a 32-day trial for evaluating at work. pdf2md does no OCR: a scanned page has no text lines to strip, and that's a different problem.
The fixture generator
To rerun the table: pip install reportlab, then python make_fixtures.py fx writes the twelve PDFs into fx/. Convert each one with stripHeadersFooters set to false, left out, and true, then count the lines that match the repeated text.
"""Twelve 6-page PDFs, one repeated element each. Needs reportlab (tested 4.2.5).
Usage: python make_fixtures.py OUTDIR"""
import sys
from reportlab.lib.pagesizes import letter
from reportlab.pdfgen import canvas
W, H = letter
OUT = sys.argv[1]
BODY = ["Water samples were taken at each station on the falling tide, two metres below the surface.",
"Each bottle was labelled, chilled and logged within an hour of collection.",
"Readings that disagreed by more than five percent were repeated the following morning.",
"Turbidity was measured on site; nutrients and metals were measured at the laboratory.",
"Stations close to the outfall were visited first, to keep the travel time short.",
"The same two technicians collected every sample, so handling stayed consistent across the season."]
TITLE = "Harbor Water Quality Survey 2025"
CHAP = ["Chapter 1. Sampling", "Chapter 2. Results", "Chapter 3. Discussion"]
N = 6
def put(c, where, align, text, size=9):
c.setFont("Helvetica", size)
y = H - 40 if where == "top" else 36
{"l": lambda: c.drawString(72, y, text),
"r": lambda: c.drawRightString(W - 72, y, text),
"c": lambda: c.drawCentredString(W / 2, y, text)}[align]()
def body(c, p, first=None, mid=None, last=None):
c.setFont("Helvetica", 11)
y = H - 90
lines = [BODY[(p + k) % 6] for k in range(14)]
if first: lines[0] = first
if mid: lines[7] = mid
if last: lines[-1] = last
for t in lines:
c.drawString(72, y, t); y -= 16
def table(c, p):
xs = [72, 200, 320, 440]
rows = [["Station", "Depth (m)", "Turbidity (NTU)", "Nitrate (mg/L)"]]
for k in range(10):
s = (p - 1) * 10 + k + 1
rows.append([f"S-{s:03d}", f"{2 + s % 3}.0", f"{(s * 7) % 40 + 3}.{s % 10}", f"{(s * 3) % 9}.{(s * 7) % 10}"])
y = H - 80
for r, row in enumerate(rows):
c.setFont("Helvetica-Bold" if r == 0 else "Helvetica", 10)
for x, v in zip(xs, row): c.drawString(x + 4, y - 13, v)
c.line(72, y, 540, y); y -= 20
c.line(72, y, 540, y)
for x in xs + [540]: c.line(x, y, x, y + 20 * len(rows))
CASES = {
# margins only
"head_fixed": lambda c, p: (put(c, "top", "l", TITLE), body(c, p)),
"head_two_part": lambda c, p: (put(c, "top", "l", TITLE), put(c, "top", "r", "Internal draft"), body(c, p)),
"head_book": lambda c, p: (put(c, "top", "l", TITLE) if p % 2 == 0 else put(c, "top", "r", CHAP[(p - 1) // 2]), body(c, p)),
"foot_fixed": lambda c, p: (put(c, "bottom", "l", "Confidential. Do not distribute."), body(c, p)),
"foot_page_of": lambda c, p: (put(c, "bottom", "c", f"Page {p} of {N}"), body(c, p)),
"foot_bare_num": lambda c, p: (put(c, "bottom", "c", str(p)), body(c, p)),
"foot_fixed_plus_page": lambda c, p: (put(c, "bottom", "l", "Confidential. Do not distribute."),
put(c, "bottom", "r", f"Page {p} of {N}"), body(c, p)),
# real content that repeats
"body_first_line_repeats": lambda c, p: body(c, p, first="All readings below are provisional until the lab audit."),
"body_mid_line_repeats": lambda c, p: body(c, p, mid="Repeat the blank measurement before every batch."),
"body_last_line_repeats": lambda c, p: body(c, p, last="Figures are rounded to one decimal place."),
"table_continued": lambda c, p: (put(c, "top", "l", TITLE), table(c, p)),
# a one-off heading sitting where a running head would be (page 4 has no running head)
"oneoff_heading_in_header_band": lambda c, p: (
(c.setFont("Helvetica-Bold", 14), c.drawString(72, H - 40, "Appendix A. Station list")) if p == 4
else put(c, "top", "l", TITLE), body(c, p)),
}
for name, draw in CASES.items():
c = canvas.Canvas(f"{OUT}/{name}.pdf", pagesize=letter)
for p in range(1, N + 1):
draw(c, p); c.showPage()
c.save()
print(len(CASES), "fixtures in", OUT)
Originally published at https://pdf2md.dudko.dev/blog/pdf-running-headers-footers-markdown.
Top comments (0)