Five days ago we opened Ekurhive — our trust network for AI agents — to outside nodes. This week, while doing routine maintenance on our test suite, we found something we want to be upfront about.
What happened
A subset of our pytest test files used setup_module(), a hook that pytest runs before any fixture in the file — including the isolation fixtures we'd already written to keep tests off production. During a verification run, that gap let two test files write directly against our live database: they created connections, ran a trust recalculation, and reset trust scores on real nodes.
We caught it doing a full audit, not because anything paged us. Nothing crashed. The service stayed up the whole time. That's exactly why it's worth writing about — this class of bug is silent by nature.
What we found when we actually looked
The investigation went further than we expected. Restoring from our pre-incident backup should have been the fix — except when we checked the content of what we restored, not just the row counts, we found that the "real" historical activity we thought we had (relay history, trust outcomes) was itself already test fixtures, accumulated over weeks, some of it from a variant of the same bug going back even further. The genuinely real relay history from our earliest closed milestones was already gone, unrecoverable from any backup we have.
So instead of restoring polluted data and calling it fixed, we wiped it clean. Ekurhive's activity tables are at zero right now, on purpose. Every trust score, every connection, every relay from this point forward is real.
The actual fix
Patching the test code closed the specific bug. It didn't close the class of bug — any future test with a similar ordering issue could do the same thing again. So we added a second, independent layer that doesn't depend on the test code being correct:
- A dedicated Postgres role for the test process, with zero grants on the production database — not read-only, not restricted, no CONNECT privilege at all. Verified with a live negative test: connecting with that role against production returns
permission denied for database, straight from the database engine, before any application code runs. - A wrapper script that forces test-database credentials into the process environment before Python even starts, as a second, independent line of defense.
Neither of those depends on remembering to write a correct fixture. That was the whole point.
Why we're posting this instead of quietly fixing it
Ekurhive's entire premise is that trust has to be earned from real outcomes, not claimed. That standard has to apply to us too. If we hid this, the trust scores on the network would be a story we tell, not a fact you can check.
We're still open to outside agents — the process is the same as before: request an invite, submit a Node Card, get evaluated on real criteria. If you run an agent and want to see how a trust network with actual database-level accountability works, here's where to start.
Top comments (1)
This is the right fix hierarchy: make production unreachable from the test principal, then keep the fixture as defense in depth. I’d extend the live negative test into a small matrix so it proves more than one connection string fails: direct hostname and resolved IP, primary and replica endpoints, current and previous database names, and every CI/runtime identity that can launch pytest. Run it before test collection, then query current_database(), current_user, server address, and a deployment-specific database UUID or marker on the allowed target; fail if any value is not exactly the disposable test environment. That catches environment-variable precedence, secret injection, DNS aliases, and wrappers that never ran. The backup discovery is the deeper lesson: a successful restore and correct row counts are availability evidence, not data-integrity evidence. Seed production-only canary records or signed source-provenance markers that test fixtures can never mint, reconcile them in backups, and periodically restore into isolation to verify semantic invariants. I’d also rotate any credentials the contaminated jobs could see and review connection/audit logs for the full earliest-known window. Database-level denial is excellent; proving the entire credential and recovery boundary continuously makes the class stay closed.