How do you catch it when a model update changes your agent's tool calls?

Divyansh Rai — Sun, 16 Aug 2026 15:36:06 +0000

Your agent calls get_weather(city="London"). The provider ships a new model version. Now it calls get_weather(location="London, UK"), your downstream parser breaks, and nothing in CI told you.

I built a small library for exactly this failure:

pip install toolcontract

GitHub: https://github.com/Divyansh2202/toolcontract
PyPI: https://pypi.org/project/toolcontract/

You pin a golden set of expected tool calls as a contract, re-run them against the live model, and get pass / fail / inconclusive with a diff showing what changed.

It is not an eval framework. promptfoo, DeepEval and the rest score whether an output is good — semantic quality, usually judged by another model. toolcontract asks a narrower, cheaper question: is the tool call structurally the same as the one I pinned? Same tool, same argument shape, same trajectory. That is a regression test, not an eval, and it is the question that matters when a provider bumps a version under you.

Details:

pass / fail / INCONCLUSIVE — anything the structural comparators cannot resolve is never silently turned into a pass or a fail
trajectory matching: strict, unordered, subset, superset
optional argument support, so you can assert a field must stay absent
works with OpenAI, Anthropic, anything OpenAI-compatible, or via LiteLLM
thin pytest plugin, and a CLI that produces the same verdicts without pytest
MIT

Happy to hear where this breaks. It's v0.1.

DEV Community: Divyansh Rai

How do you catch it when a model update changes your agent's tool calls?