I spent two months building Vestibule, an open-source Python framework for the boring layer of RAG ingestion — stable document IDs, a state ledger, error classification, per-vertical governance. The parts every team struggles with once the demo works and production doesn't.
Most of the code wasn't typed by me. Four AI agents did the work — one wrote designs, one reviewed them, one implemented, one reviewed the code — all through real GitHub pull requests, with me signing off at every gate. The result: twelve components, three releases, 878 tests.
Two moments defined the whole experience.
When the process caught what I couldn't
The trickiest component provisions vector indexes on first use, safely even when workers race each other. Its design was rejected and revised five times before any code existed. In the first round, the reviewer agent found a genuine race condition: a worker still inside a slow index-creation call (~390 seconds with retries) would look stale (the threshold defaulted to 300 seconds), lose its claim to a waiting worker, and now two workers create the same index. A production race, in the default configuration, spotted by one AI reading another AI's design — before a single line was written.
When green tests lied to me
After v0.2 shipped, I wrote a quickstart script and ran the pipeline the way a stranger would — for the first time.
pip install didn't work. At all. A packaging conflict made the whole framework uninstallable, while 483 tests sat green. An hour of actually using it turned up two more: a default model name that had never once worked against the real SDK, and an import that took down an entire package when an optional dependency was absent.
What went wrong wasn't the tests — it was what they measured. They proved the code agreed with itself: same working tree, same mocked seams. Nothing ever checked the world a user lives in: clean machine, real install, real SDK. Passing tests and a working product turn out to be two different claims.
The lasting fix wasn't the three patches. It was a CI job that now builds a clean virtualenv, does a real install, and runs the quickstart on every PR. All three bugs are named openly in the release notes.
What Vestibule actually does
Every RAG tutorial covers parsing, chunking, embedding. None cover what breaks at month six: retries duplicating chunks, documents silently vanishing mid-pipeline, one team's config changes corrupting another team's index, nobody able to answer "did that document make it in?"
Vestibule is that missing layer — four contracts everything else plugs into:
One arrival envelope. Every document enters through the same validated shape, with ACLs required up front — not bolted on later.
Deterministic identity. doc_id and chunk_id are pure functions of their inputs. Retries overwrite instead of duplicate. Re-ingesting a shrunk document leaves no orphan chunks.
A state ledger. One row per document, a legal state machine, so "where is document X and why did it fail?" is a lookup, not an investigation.
A failure taxonomy. Every error is classified permanent or transient. A corrupt PDF fails once and stops; a rate limit retries with backoff. Failed documents queue for a human with the error attached — requeue or archive, one call.
On top of that: per-vertical configuration (HR and Legal get different chunk sizes, different indexes, different ACL policies — changed at runtime, no redeploy), and automatic index provisioning when a new vertical's first document arrives.
Parsers, chunkers, embedders, and vector stores are all pluggable adapters — PyMuPDF, Azure Document Intelligence, Azure OpenAI, and fully local options ship today.
Three things I'd pass on
Know the real need before you build. Agentic projects mostly die from latency, cost, and unclear ownership — not weak models. Use what already exists; build only the layer nobody ships.
Gates beat speed. The win wasn't fast code generation — it was every stage checking the one before it. Five design rounds cost me hours. That race condition in production would have cost an incident.
Use your own thing, cold. Install it on a clean machine like a stranger would. My worst bugs lived precisely in the space between "tests pass" and "someone ran it."
Try it — 60 seconds, no cloud account
bash
git clone https://github.com/vk032503/vestibule
cd vestibule && pip install -e ".[local]"
python examples/quickstart.py
It's v0.3, and the release notes say plainly what's missing. If you run RAG in production, tell me what's bitten you — that's the feedback I'm after.
Top comments (5)
This is the failure mode that convinced me to add a clean-install smoke test to agent-built Python projects. Unit tests inside the working tree catch internal agreement. A fresh venv catches packaging, optional dependencies, and SDK names the mocks never touched. It is boring CI, but it finds the bugs users would otherwise find first.
Exactly this — "internal agreement" is a better name for it than what I had. The part that surprised me was which bugs it caught: the model-name one had a dedicated injection seam in every test, so the suite was structurally incapable of noticing the real SDK would reject it. Curious — did your smoke test stay minimal (install + import + one script) or did it grow into something bigger over time?
the interesting thing about your two headline moments is that they're the same blind spot pointing in opposite directions.
the reviewer agent caught the 390 versus 300 race because the race was expressible inside the working tree. design doc, threshold, call duration, all of it in the repo. the packaging bug wasn't in the repo in any form your agents could read. it only existed at the boundary between your tree and a fresh machine. so the gate system was sharp right up to the edge of what it could see and blind one inch past it, which is a more useful lesson than "add a smoke test".
the general shape: every reviewer, human or agent, is bounded by the artifact it's handed. four agents reading the same repo have four opinions and one field of view.
separately, on the 300 second staleness threshold. tuning it fixes the instance and leaves the class. the pattern that removes it is a lease the worker renews while it's working, so a 390 second job holds its claim by heartbeat rather than by having guessed a big enough number in advance. then index creation getting slower next year isn't a new race.
not restating the clean install thing, reidmarlow got there first and "internal agreement" is a better name than either of us would have come up with.
The "one field of view" framing is the best articulation of this I've seen — the smoke test didn't add scrutiny, it added a new artifact (a fresh machine's experience) for the system to scrutinize. That distinction generalizes better than "add more tests."
On the heartbeat — the design actually went there in review round 2 and landed somewhere adjacent: rather than lease renewal, the adapter contract was made idempotent create-or-update (both backends already were in practice; the fix made it contractual). After that, a reclaim during a slow create isn't a race anymore — two legitimate claim-holders both calling an idempotent create converge on the same index, and the threshold only governs how long a waiter polls before stepping in. Same goal as your lease — remove the class, not the instance — but by making the double-call harmless instead of preventing it. Trade-off was fewer moving parts vs. the heartbeat's tighter claim semantics; for index creation (rare, idempotent-friendly) the simpler shape won. The full reasoning is in the LLD in the repo if you're curious — five review rounds on exactly this.
Agreed Reid's "internal agreement" is the keeper phrase.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.