When you pull the text out of a PDF, the running header, the footer and the page number come out as ordinary lines, mixed in with the body. Nothing marks them as different. A RAG pipeline then cuts that text into chunks, and two things happen. The same footer lands in chunk after chunk, which makes those chunks look alike to the embedding model. And any chunk that crosses a page break gets a header in the middle of a sentence.
I wanted to know how much of a real document this is, so I measured it on 25 public PDFs from US federal agencies. Disclosure up front: I make Agnos, a Windows app from Gidion that cleans documents before they go into AI tools, and the study measured what it removed. What it got wrong is in here too. That turned out to be the most useful part.
The setup
- 25 PDFs, 1,342 pages, from agency websites: handbooks, program guides, technical standards, regulatory guidance, annual reports, procurement, research reports and form instructions. All of them are text-based. None is a scan.
- I picked them by hand, not at random. I chose 21 because they have running headers or footers on most pages, and 4 because they have little. So the shares below describe these documents, not PDFs in general.
- The baseline is what a simple loader passes on: pdfplumber 0.11.10,
page.extract_text()with its default options. That's 64,682 non-empty lines and 3,509,754 characters. - The removed lines come from the export log (
rejected.jsonl) of Agnos 2.3.2, run with its RAG / Retrieval Corpus purpose and default settings.
What came out
Agnos removed 2,063 lines as page furniture, from 24 of the 25 documents. That's 50,928 characters, 1.45% of the plain text. Per document, the median was 1.32%, and the range ran from 0% to 3.17%.
By volume, that's small. The problem is where it sits. The median was 1.18 lines of furniture per page. Furniture was removed from every text page of 3 documents, and from at least 90% of the text pages in 15. It's at every page break, so it lands in the middle of any chunk that crosses one.
The most common kinds (one line can carry more than one label):
| Kind | Lines |
|---|---|
| Running title with a page number | 635 |
| Date or year | 590 |
| Page number on its own | 497 |
| Document number or form id | 488 |
| Agency or organisation name | 458 |
| Running title or document name, no page number | 457 |
| "Page X of Y" | 305 |
Some real ones: "Page 1 of 89" from NASA's software engineering requirements. "PAGE 3 OF 34 1333ND19QNB680025" from a NIST solicitation. And a line from an IRS printer's proof that starts "Page 2 of 59 Fileid: …" and runs on into a file path and a timestamp. None of that helps anyone answer a question.
Find them in your own PDFs
You don't need any app to see this. The idea is simple. Compare lines with their digits masked, so "Page 3 of 12" and "Page 4 of 12" count as the same line. Then count how many pages each one is on.
import re
import sys
from collections import defaultdict
import pdfplumber # pip install pdfplumber
def shape(line):
# "Page 2 of 4" and "Page 3 of 4" have the same shape: "page # of #"
return re.sub(r"\d+", "#", " ".join(line.lower().split()))
with pdfplumber.open(sys.argv[1]) as pdf:
pages = [(page.extract_text() or "").splitlines() for page in pdf.pages]
on_pages = defaultdict(set) # shape -> the pages it is on
first_seen = {} # shape -> the line as first printed
for number, lines in enumerate(pages, start=1):
for line in lines:
if line.strip():
on_pages[shape(line)].add(number)
first_seen.setdefault(shape(line), line.strip())
need = max(3, 0.3 * len(pages)) # on 3+ pages, and on 30%+ of them
for s, seen in sorted(on_pages.items(), key=lambda kv: -len(kv[1])):
if len(seen) >= need:
print(f"on {len(seen)} of {len(pages)} pages: {first_seen[s]!r}")
Run it as python repeated_lines.py your.pdf. On a four-page sample handbook I made for my guides, it prints:
on 4 of 4 pages: 'Northwind Tools | Staff Handbook | March 2026'
on 4 of 4 pages: 'Confidential - Internal Use Only'
on 4 of 4 pages: 'Page 1 of 4'
That's the easy case. Real documents are where it gets interesting.
Trap 1: masked digits match things that aren't furniture
Masking the digits is what makes this work. It's also what makes it wrong:
import re
def shape(line):
return re.sub(r"\d+", "#", " ".join(line.lower().split()))
pairs = [
("Page 3 of 12", "Page 4 of 12"), # furniture: these should match
("Table A-1.", "Table A-2."), # two different table labels
("1-800-555-0100", "1-800-555-0199"), # two different phone numbers
]
for a, b in pairs:
print(f"{a!r} = {b!r}? {shape(a) == shape(b)}, both {shape(a)!r}")
'Page 3 of 12' = 'Page 4 of 12'? True, both 'page # of #'
'Table A-1.' = 'Table A-2.'? True, both 'table a-#.'
'1-800-555-0100' = '1-800-555-0199'? True, both '#-#-#-#'
(The phone numbers are made-up 555 numbers.)
The study ran into exactly this. We read every removed line, and checked its page when the line alone wasn't clear. 200 of the 2,063 looked like content. There were 36 table cells, 34 column headings repeated on each page, 22 figure and table notes, 21 table labels and 17 phone numbers, plus smaller groups. In the Census income report, "Table A-1." was removed while the table's title on the next line was kept. In NASA's requirements, "3.1.10 042 The project manager shall" was removed at a page edge. The rest of the sentence stayed, so that requirement lost its number.
None of those lines is furniture. They have the same shape as furniture, at the edge of the page, on 3 or more pages.
Trap 2: an edge window misses things
The safer way to use this rule is to look only at the edges of each page. Agnos looks at the first 2 and last 2 lines. There it removes page markers, page numbers that count up with the pages, and short lines (80 characters or fewer, not sentences) that repeat at the edges of 3 or more pages. That protects the body of the page. But it also means Agnos kept about as much repeated text as it removed.
The biggest case: NASA NPR 7150.2D prints a four-line notice at the foot of all 89 pages. In the order Agnos reads the page, the page number and a header line come after the notice. Those two were removed, so the notice sits just outside the window. All four lines stayed: 26,433 characters, 12.8% of that document's plain text. An IRS proof notice stayed too. At 121 characters, it's over the length limit.
Trap 3: a percentage threshold finds nothing in some documents
A common rule is "repeated on at least 30% of pages". Medicare & You 2027 changes its running header with each section. No single line is on 30% of its 128 pages, so that rule finds nothing at all. Agnos needs a line on only 3 pages, and it removed a header or page number from 104 of the document's 124 text pages. Neither threshold is right for every file.
One more thing to watch: rotated pages. The Census report has 18 pages of rotated tables. Agnos's reader returned each of those pages as one or two long lines, so no furniture could be found on them. And pdfplumber read the rotated tables backwards: "emocni" for "income".
What I'd check before indexing
- List the lines that repeat on 3 or more pages, and read them before you delete anything.
- Count the lines in your footers. A footer longer than your edge window stays.
- Look for long running headers. A length limit that protects sentences also protects an 85-character header.
- Log every removed line with its page. Without a log, a missing phone number or table label is very hard to find later.
- Open a few chunks that cross a page break, and read them.
What Agnos does
Agnos does this for whole folders of text-based PDFs, Word, Excel, PowerPoint and web pages, on your own Windows 10 or 11 PC. It lists what it removed with the rule behind each removal. Review shows up to 500 removals per file, and the Everything export has the full log as rejected.jsonl. Its limits: it doesn't do OCR, so scanned pages are skipped and named in its warnings. It runs on Windows only, and the Store version has no command line yet. If you'd rather stay in Python, Docling is free and open source. It labels page headers and footers and leaves them out of its default exports.
The full study
The study page has every document, the method, all 200 lines that looked like content, and two scripts to reproduce it. The per-document numbers, 46 columns of them, are in a CSV under CC BY 4.0: https://agnos.cloud/guides/page-furniture-study/ To try this on your own PDFs without code, there's a free tool that runs in your browser and uploads nothing: https://agnos.cloud/tools/pdf-repeated-lines/
How do you check what your cleaning step removed? I'd like to know what other people do.
Drafted with help from an AI assistant (Claude). The numbers were checked against the study page, and both scripts were run on the sample PDF, before publishing.
Top comments (4)
Measuring it per page instead of per document is what makes this convincing. 1.45% of characters sounds harmless until you see it sits at every page break.
One thing I'd push back on a little is treating everything the edge filter finds as trash. In the policy and handbook PDFs I've fed into support bots, the footer was often the only place the document said which version it was: the revision date, the form number, the effective date. Your table has 590 date lines and 488 document numbers, which matches what I saw. Once those lines were stripped, two editions of the same handbook produced chunks that looked identical, and the bot quoted the old refund window with no way to tell. Page numbers had the same problem from the other side: people wanted the citation to say page 14, and that number had just been deleted.
What worked better was splitting the job in two. Detect the furniture the way you describe and take it out of the text that gets embedded, but keep it on the chunk as metadata: page number, document title, revision or date line. Retrieval stays clean, and the answer can still say which edition and which page it came from. It also helped with your trap 1, because a line you are about to keep as metadata gets looked at differently from a line you are about to throw away, and "Table A-1." is obviously not a revision date.
Did you look at how often the removed lines carried a date or document number that was the only version marker in the file?
Really good point, and the old refund window is exactly the kind of failure I mean. I’d pull the edition date or form number out once and attach it to every chunk as a label instead of leaving it in the text, so two editions never look the same. How did you end up telling the old handbook apart?
Headers and footers are not always boilerplate: page numbers, edition labels, and section context can carry meaning. A robust pipeline should classify repeated material rather than strip it blindly, then retain page-level provenance so a retrieved chunk can be checked against the original layout when the answer matters.
Agreed, sorting them beats deleting blindly. Keeping the page with every chunk is what makes an answer checkable. That’s also why Agnos logs every line it removes, so nothing is gone for good.