TL;DR: A model version has a published end date. Replacing it is a release: eval set, real past cases, shadow run, a pinned version and the model version recorded with every decision.
Why replacing a model is a release
An agent's behaviour is the combination of a prompt and a model. The prompt was written, tested and tuned against one specific version: its wording, the examples, the way it asks for structured output. Change the model and the same prompt can produce a different tone, different tool arguments, a different threshold for "I am not sure". No line of your code changed, and the agent is no longer the one you tested. That is why a retirement notice is not a configuration task. It is a change to a dependency you do not control, and it deserves the same steps as any other release.
What to run before the cutover
Start with the eval set. If the client has one, with real cases and expected outcomes, run it on the new model and read the differences, not only the pass rate: a case that flipped from right to wrong matters more than two that flipped the other way. Then replay a sample of real past decisions and compare. Check what changes around the answer too: the format of tool calls, latency, cost per task, how often the model refuses or asks for clarification. Run the new model in shadow next to the old one for a period that fits the risk, so it sees live input but its output is only compared, not acted on. Keep the rollback path until the old version is actually gone, and write down the date after which it is not there.
What makes it cheap later
Three small choices made at delivery. Pin the exact model version in configuration, never an alias that moves under you, because an alias turns a planned replacement into a surprise. Put the retirement date in a calendar owned by someone who is not the author of the agent, with a reminder early enough to run the steps above. And record the model version with every decision in the decision log, so that months later you can say which version made a given decision and replay it against the new one.
What this means for a delivery
For a team handing an agent to a client, the model is a supplier with a published end date. The handover pack should say which version is running, when it retires, who is told, and where the eval set lives. For high-risk systems the EU AI Act asks the deployer to monitor operation and keep logs. When the model changes, what is being monitored changes with it, and the log is what lets you show the before and after.
Who owns the retirement date of the model your agent runs on today?
Articolo originale: https://www.omarbaruzzo.it/en/blog/modello-agente-ritirato
Top comments (0)