I spent two months building Vestibule, an open-source Python framework for the boring layer of RAG ingestion — stable document IDs, a state ledger, error classification, per-vertical governance. The parts every team struggles with once the demo works and production doesn't.
Most of the code wasn't typed by me. Four AI agents did the work — one wrote designs, one reviewed them, one implemented, one reviewed the code — all through real GitHub pull requests, with me signing off at every gate. The result: twelve components, three releases, 878 tests.
Two moments defined the whole experience.
When the process caught what I couldn't
The trickiest component provisions vector indexes on first use, safely even when workers race each other. Its design was rejected and revised five times before any code existed. In the first round, the reviewer agent found a genuine race condition: a worker still inside a slow index-creation call (~390 seconds with retries) would look stale (the threshold defaulted to 300 seconds), lose its claim to a waiting worker, and now two workers create the same index. A production race, in the default configuration, spotted by one AI reading another AI's design — before a single line was written.
When green tests lied to me
After v0.2 shipped, I wrote a quickstart script and ran the pipeline the way a stranger would — for the first time.
pip install didn't work. At all. A packaging conflict made the whole framework uninstallable, while 483 tests sat green. An hour of actually using it turned up two more: a default model name that had never once worked against the real SDK, and an import that took down an entire package when an optional dependency was absent.
What went wrong wasn't the tests — it was what they measured. They proved the code agreed with itself: same working tree, same mocked seams. Nothing ever checked the world a user lives in: clean machine, real install, real SDK. Passing tests and a working product turn out to be two different claims.
The lasting fix wasn't the three patches. It was a CI job that now builds a clean virtualenv, does a real install, and runs the quickstart on every PR. All three bugs are named openly in the release notes.
What Vestibule actually does
Every RAG tutorial covers parsing, chunking, embedding. None cover what breaks at month six: retries duplicating chunks, documents silently vanishing mid-pipeline, one team's config changes corrupting another team's index, nobody able to answer "did that document make it in?"
Vestibule is that missing layer — four contracts everything else plugs into:
One arrival envelope. Every document enters through the same validated shape, with ACLs required up front — not bolted on later.
Deterministic identity. doc_id and chunk_id are pure functions of their inputs. Retries overwrite instead of duplicate. Re-ingesting a shrunk document leaves no orphan chunks.
A state ledger. One row per document, a legal state machine, so "where is document X and why did it fail?" is a lookup, not an investigation.
A failure taxonomy. Every error is classified permanent or transient. A corrupt PDF fails once and stops; a rate limit retries with backoff. Failed documents queue for a human with the error attached — requeue or archive, one call.
On top of that: per-vertical configuration (HR and Legal get different chunk sizes, different indexes, different ACL policies — changed at runtime, no redeploy), and automatic index provisioning when a new vertical's first document arrives.
Parsers, chunkers, embedders, and vector stores are all pluggable adapters — PyMuPDF, Azure Document Intelligence, Azure OpenAI, and fully local options ship today.
Three things I'd pass on
Know the real need before you build. Agentic projects mostly die from latency, cost, and unclear ownership — not weak models. Use what already exists; build only the layer nobody ships.
Gates beat speed. The win wasn't fast code generation — it was every stage checking the one before it. Five design rounds cost me hours. That race condition in production would have cost an incident.
Use your own thing, cold. Install it on a clean machine like a stranger would. My worst bugs lived precisely in the space between "tests pass" and "someone ran it."
Try it — 60 seconds, no cloud account
bash
git clone https://github.com/vk032503/vestibule
cd vestibule && pip install -e ".[local]"
python examples/quickstart.py
It's v0.3, and the release notes say plainly what's missing. If you run RAG in production, tell me what's bitten you — that's the feedback I'm after.
Top comments (0)