i'm Arian, building Nobulex. the problem i'm working on is a tool returning something that looks successful but is materially wrong.
think of a historical-data request that returns the right number of rows for the wrong dates. checking the row count alone won't establish that the requested window was returned. checking that the JSON parses won't either.
Nobulex's current public project is a reliability suite for financial-data tools. it calls a pinned subject over MCP stdio and compares observations under stated conditions. it's a tool-level test, not proof that an entire agent workflow is safe.
the useful part to try first is the self-test. it includes planted failures and clean controls. no package install is required; the README specifies Python 3.11 or newer.
git clone https://github.com/arian-gogani/nobulex-registry.git
cd nobulex-registry
python3 suite/selftest.py
the important limit: a reference source is not automatically independent. comparing a wrapper with its own upstream can test whether it preserved the response, but cannot establish that the upstream itself is right.
i'd like feedback on the method, especially a small reproduction where the suite calls corrupted data clean, or flags a documented tolerance as a defect. a useful counterexample is worth more than a thumbs-up.
source: https://github.com/arian-gogani/nobulex-registry
update, september 10: there's now a smaller way to try the problem yourself. after cloning the repo above, run:
python3 examples/wrong_window.py
it uses fictional data and the suite's actual truncation classifier. Python 3.11 or newer is enough. no credentials, package install or network calls are needed after cloning.
| fictional input | row counts match? | truncation probe |
|---|---|---|
| matching dates | yes | PASS |
| two sessions silently missing | no | FAIL_UNSAFE |
| five rows from completely different dates | yes | INDETERMINATE |
all three expected outcomes reproduced when we ran it. the last case matters: the count-only check says true, but that doesn't establish that the data covers the expected window.
INDETERMINATE here means this probe could not establish a clean result. it isn't a proven defect in a live provider. the matching case's PASS doesn't establish that its prices are correct either.
the example source is small enough to read. change a fixture and see what happens. if you find a result the classifier gets wrong, share the smallest fictional example in an issue.
Top comments (0)