Level 6 of nine in Project Arc Rector - an agentic RAG stack built from free, self-hostable parts, one swappable level at a time. Level 5 was embeddings and reranking. This is the level underneath all of it: ingestion and parsing, where the chunks come from.
It is also the only level whose failures are silent by construction. A parser that returns text in the wrong order does not raise. It returns a string, the chunker accepts it, the embedder embeds it, the store stores it, and the corpus is quietly wrong from that moment on.
Repo: https://github.com/dev48v/arc-rector - 174 tests, no network, no model, no container. The page, with a real recursive X-Y cut and a line-for-line port of the repo's own chunker computing in your browser: https://dev48.infy.uk/arcrector/level6-ingestion.html
Reading order is geometry, and geometry is recoverable
Twenty-three positioned text spans - a header, two columns, a full-width pull-quote, a footer - and one 40 px gutter. Sorting by y then x is what a naive extractor does, and it produces prose no human wrote:
naive : "A two-column page is not two The result is prose that no human
documents. It is one document with ever wrote."
cut : "A two-column page is not two documents. It is one document with
a geometric trap: a naive text extractor walks the page top to
bottom and reads straight across the gutter."
| whitespace gap threshold | cuts taken | cross-gutter joins |
|---|---|---|
| naive sort, no cut | 0 | 18 |
| 2 px | 22 (2 vertical, 20 horizontal) | 0 |
| 20 px (the default) | 6 (2 vertical, 4 horizontal) | 0 |
| 80 px (wider than the gutter) | 0 | 18 |
Six cuts fix all eighteen joins, and a threshold wider than the gutter is not a cut at all, so the damage returns in full. Downstream, the repo's chunker over both texts emits five chunks of the naive prose against four of the recovered - and the extra one is not extra content, it is sentences from two different columns welded together and then embedded.
What the browser does not do is decode a PDF. There is no binary, no OCR, no layout model. The coordinates are a fixture; the algorithm over them is the real one. Producing those boxes from a scanned or vector page is exactly what Docling is, and it is a gigabyte of weights.
The defaults, as they actually ship
l6_ingestion:
use: docling
settings:
fallback_to_plaintext: true
Docling is the default, and the adapter deliberately gets out of its own way: .txt, .md, .rst, .csv, .json, .yaml and .log are delegated to a dependency-free plaintext loader, and the metadata records that they were, so a later reader is not misled about which parser produced the text. The converter is built lazily, because constructing it is what loads the models. Chunking is 900 characters with 150 of overlap. A URL loader is an SSRF primitive, so assert_fetchable checks every address the name resolves to rather than the first, redirects are followed by hand with a five-hop cap so each one is re-checked, and the 10 MB body cap is enforced on bytes actually received rather than on the Content-Length a server claims.
And load_documents collects failures instead of raising them - because document 3,998 will be the corrupt one and you do not want to re-embed the other 3,997 - counting parsed to empty text as a failure too, since a PDF with no text layer returns a cheerful empty string rather than an exception.
The footnote this level needs
Provenance is carried onto every Document and therefore onto every Chunk, because you cannot cite what you cannot name. Skip it and retrieval still works perfectly; you just can never tell the reader where an answer came from. And say "open" precisely: Docling, Unstructured, Scrapy and the plain loader are permissive and run with no account, but Firecrawl's hosted API is a commercial product with a free tier, and the self-hostable core the adapter targets at localhost:3002 is AGPL-3.0 - copyleft that reaches across a network boundary.
Next is Level 7, memory and context. The whole stack, nine levels, all free to self-host: https://dev48.infy.uk/arcrector.php
Top comments (0)