How do you keep prompts, tools, and tests working when you switch models?

I've been thinking about what happens to an AI agent when the model underneath it changes. The code doesn't change, but the behavior can. A prompt that worked well on one model may give different answers on another, and a tool call that was reliable may start failing quietly.
How are you handling this today? Some questions I'm curious about:
–When you upgraded or switched models, what broke first?
–Did you have tests for agent behavior, or did you find out from users?
–Are your prompts, tool definitions, and test cases stored together, or in different places?
–Did you build any manual process to catch these changes, and what does it look like?
–What's the last model change that surprised you?
–How do you know when an agent has gotten worse?
–What do you do before you upgrade a model?
–What tool do you wish existed for this?
If you've solved this, I'd like to hear how. If you haven't, I'd like to hear that too. Share what you've run into in the comments.

Comments

[Giang Pham Truong]

A useful regression case would be a write that succeeds but times out before the agent receives the result. Does the replacement model check the existing record, retry with the same operation ID, or create a duplicate? A final-answer check can miss that difference.

I'd version a small set of task fixtures alongside the prompt, tool schemas and model configuration. Keep assertions on observable outcomes: the intended record exists once, required fields are correct, and no unauthorized tool was called. For read-only cases, replay fixed tool results to separate model drift from changing external data; test live integrations separately. Repeated runs also matter: one successful run can hide an intermittent failure. This is a proposed test approach, rather than a claim that I've solved model migration.