How should teams manage AI model changes after deployment?

An AI feature can behave differently even when the surrounding application code remains unchanged. A provider update, revised prompt, or configuration change can alter output quality, response time, and cost.

A GeekyAnts article on managing AI after launch describes how several projects address this problem. Its most useful details concern traceability and testing.

Which controls stand out?

• Version prompts alongside code. One documented project requires prompt changes and updated evaluation baselines in the same commit.
• Pin model versions where supported. Dated releases help reduce unexpected changes associated with moving model aliases.
• Capture diagnostic information. Provider, model, latency, retries, errors, and token usage can help explain a change in behavior.
• Make retention deliberate. One project avoids storing audio and deletes transcripts and drafts 24 hours after sessions end. That is a project-specific policy, not a universal requirement.
• Test replacement paths. A supported alternative provider still needs evaluation before it takes over a production workload.

The article also identifies implementation limits: one platform records AI calls for investigation but does not yet expose dedicated Prometheus metrics for them. Logging and automated monitoring should therefore be evaluated separately.

Which companies offer relevant approaches?

Three companies worth comparing are:

• GeekyAnts: Its project examples describe controls implemented within applications, including prompt versioning, evaluation workflows, and AI-call records.
• IBM: Its watsonx.governance tooling supports lifecycle governance and evaluation of model health, drift, and generative AI quality.
• Accenture: Its responsible AI services include governance, risk assessments, testing frameworks, and ongoing monitoring.

Product engineering, governance tooling, and organizational oversight solve different parts of the problem.

Discussion: Which control has proved most useful for catching production regressions: pinned model versions, evaluation gates, or monitoring? How do teams test a fallback without assuming it will behave like the primary model?