Your agent calls get_weather(city="London"). The provider ships a new model version. Now it calls get_weather(location="London, UK"), your downstream parser breaks, and nothing in CI told you.
I built a small library for exactly this failure:
pip install toolcontract
GitHub: https://github.com/Divyansh2202/toolcontract
PyPI: https://pypi.org/project/toolcontract/
You pin a golden set of expected tool calls as a contract, re-run them against the live model, and get pass / fail / inconclusive with a diff showing what changed.
It is not an eval framework. promptfoo, DeepEval and the rest score whether an output is good — semantic quality, usually judged by another model. toolcontract asks a narrower, cheaper question: is the tool call structurally the same as the one I pinned? Same tool, same argument shape, same trajectory. That is a regression test, not an eval, and it is the question that matters when a provider bumps a version under you.
Details:
- pass / fail / INCONCLUSIVE — anything the structural comparators cannot resolve is never silently turned into a pass or a fail
- trajectory matching: strict, unordered, subset, superset
- optional argument support, so you can assert a field must stay absent
- works with OpenAI, Anthropic, anything OpenAI-compatible, or via LiteLLM
- thin pytest plugin, and a CLI that produces the same verdicts without pytest
- MIT
Happy to hear where this breaks. It's v0.1.
Top comments (0)