i'm Arian, building Nobulex. the problem i'm working on is a tool returning something that looks successful but is materially wrong.
think of a historical-data request that returns the right number of rows for the wrong dates. checking the row count alone won't establish that the requested window was returned. checking that the JSON parses won't either.
Nobulex's current public project is a reliability suite for financial-data tools. it calls a pinned subject over MCP stdio and compares observations under stated conditions. it's a tool-level test, not proof that an entire agent workflow is safe.
the useful part to try first is the self-test. it includes planted failures and clean controls. no package install is required; the README specifies Python 3.11 or newer.
git clone https://github.com/arian-gogani/nobulex-registry.git
cd nobulex-registry
python3 suite/selftest.py
the important limit: a reference source is not automatically independent. comparing a wrapper with its own upstream can test whether it preserved the response, but cannot establish that the upstream itself is right.
i'd like feedback on the method, especially a small reproduction where the suite calls corrupted data clean, or flags a documented tolerance as a defect. a useful counterexample is worth more than a thumbs-up.
Top comments (0)