Release Pin: Ship the Model, Prompt, and Tools as One Version

Cross-post of Insights #9 — canonical: https://sheikhwasim.com/insights/release-pin-one-version/

Your agent passed every check on Tuesday. On Thursday it was a different agent, and nobody shipped anything.

The model alias you call quietly moved to a newer snapshot. Someone tightened a line of the prompt in a dashboard. A tool started returning one more field. Each change was small and reasonable. Together they changed what the agent does.

Code teams solved this years ago: you don't deploy "whatever's on main right now." You deploy a version. Most agent stacks still don't.

I call the fix a Release Pin: the model snapshot, the prompt, the tool schemas, the retrieval index, and the eval set ship together as one versioned manifest. If any part changes, it's a new release, and it goes through the gate.

Quick use case

Situation. A general contractor's preconstruction team runs an agent that reviews subcontractor submittals against the project spec. It reads the spec section, reads the submittal, and flags anything that doesn't comply before a human reviewer signs off. It saved reviewers hours a week, and it had a clean eval record.

What broke. Over one week, three things changed. The provider's "latest" model alias rolled forward to a new snapshot. A reviewer edited the prompt in the admin console to "make summaries shorter." And the document tool began splitting long spec sections into chunks. The agent started dropping the fine print, including the "or approved equal" conditions and the required test reports. Two non-compliant submittals were marked clean. A superintendent caught one on site. When the team tried to roll back, they couldn't say which prompt had been live, which model answered, or what "back" even meant.

The fix (Release Pin).

1. One manifest per release. Model snapshot ID (never a floating alias), prompt hash, tool schema versions, retrieval index version, and eval set version, all in one file with one release number.
2. No edits in production. The admin console can propose a prompt change. It can't make one live. Changes land as a new manifest.
3. Every change goes through the Ship Gate. A new model snapshot is a release, same as new code. Smoke Evals and Golden Traces run against the full manifest.
4. Every trace is stamped. Each run logs the release number, so "which version flagged this submittal" is a lookup, not an argument.
5. Rollback is one move. Point production at the previous manifest. Everything goes back together.

Same model family. Same team. The difference was that "what's running" became a fact you can read.

Why "we didn't change anything" isn't true

Agent behavior is the product of everything around the model: the weights behind the alias, the prompt, the tools, the data it retrieves. If any one of those moves, the agent moves.

Most teams version the code and float the rest. That's how you get a regression with no diff, an incident with no cause, and a rollback that rolls back the wrong thing.

The Release Pin (five layers)

1. Pin the model snapshot

Call a dated snapshot, not "latest." Treat a provider upgrade as an incoming release you test on purpose, on your schedule, not something that happens to you overnight.

2. Version the prompt like code

Prompts live in the repo, get reviewed, and carry a hash. A dashboard can stage a change, but only the release process makes it live. "Shorter summaries" is a behavior change, and it deserves a test.

3. Version tools and retrieval

Tool schemas, the Boundary Contract outcomes, and the retrieval index each carry a version in the manifest. A re-chunked index can change answers as much as a new model.

4. Gate the whole manifest

Run Smoke Evals and your Golden Traces against the manifest, not just the code. If the eval set itself changes, that's in the manifest too, so you know what "passed" meant.

5. Stamp, then roll back as a unit

Every trace records the release number. Rollback means pointing production at the last good manifest, with all five parts moving together. If you can't roll back in one step, you don't have a release. You have a pile of settings.

How this maps to what you already have

• 5-layer agent stack. Release Pin wraps all five layers in one version, and Traces carry the stamp.
• Ship Gate. The gate checks a release. Release Pin defines what a release is.
• Golden Trace. A frozen trace is only useful if you know which manifest produced it.
• Cheap Twin. The small model and the flagship are both pinned. Escalation routes between known versions, not moving targets.
• Boundary Contract. Tool outcome types are part of the tool's version. Changing them is a release.

The test

Pick your most important agent and ask:

If it did something wrong an hour ago, could you name the exact model snapshot, prompt, and tool versions that ran, and put all of them back in one step?

Then try it for real in staging. If the answer takes a Slack thread and three people's memory, it isn't pinned yet.

Closing

Agents don't drift by themselves. They drift because the parts around them move without a release. Pin the parts, gate the release, stamp the trace, and "what changed?" finally has an answer.

I'm Wasim Sheikh, AI Architect. I build systems teams trust and organizations depend on: not demos, not proofs of concept, production.

Follow for practical AI architecture that ships.

Connect on LinkedIn: Wasim Sheikh · Site: sheikhwasim.com · Notes: Practical AI Notes · X: @anciwasim