The Model Changed Underneath Me: Pinning LLM Versions and Regression Tests in n8n

This is part of my build-in-public series on running LLM workflows in n8n in production. Everything here is from real templates I sell and support — the mistakes are mine, the fixes are tested.

Friday it worked, Monday it didn't — and I changed nothing

A workflow passed every check on Friday. On Monday, its drafts came back in a different shape: same prompt, same input, different structure, different tone, one key field missing. I checked the workflow first, then the input data, then the prompt. All three were untouched.

The change was underneath me. The model alias I was calling had moved to a newer version over the weekend. The provider had shipped an update, my "stable" model name now pointed at different weights, and my workflow had silently deployed a change I never approved.

That is the lesson in one line: an unpinned model name is an unpinned dependency. You would not deploy a library by writing "latest" in production and hoping the next release is compatible. A model alias is the same bet, with worse odds, because a model update does not break loudly — it changes the output just enough to pass your schema and fail your customer.

Pin the exact version, and record it with the job

The first fix is boring and non-negotiable: pin the exact model version, not the family name. If the provider offers dated snapshots, call the snapshot. If it only offers an alias, treat that as a risk to be managed, not a default to accept — and put a calendar reminder on the alias, because it will move.

Just as important: record the model version in the job row, next to the provider, the attempt count, and the final state. When output quality shifts, the first question is always "what changed?" If your job rows do not say which model version produced each draft, you cannot answer it. You will re-litigate the prompt, the input, and the workflow for a week before you discover the model moved. I log the model string returned by the provider, not the one I requested, because the two can differ when an alias resolves.

A regression set, not a vibe check

Pinning stops silent drift. It does not tell you whether a new version is safe to adopt. For that I keep a fixed regression set: a small collection of real inputs, run against every candidate model version before it touches production.

The set is deliberately small — ten to twenty inputs that cover the shapes that matter: the long messy one, the short ambiguous one, the one with an edge case that broke last time, the one in a second language. Each input has assertions, and the assertions are the part people get wrong.

Do not assert exact output text. LLM output is not deterministic, and an exact-match test will fail on a good version and teach you to ignore the test suite. Assert invariants instead:

• The output parses against the schema, with required fields present and correctly typed.
• Forbidden patterns are absent — no markdown code fences around JSON, no invented fields, no placeholder text.
• Key facts from the input survive: names, dates, and amounts appear in the output, not paraphrased into something else.
• Length stays inside a sane band: a summary that triples in length, or collapses to one sentence, is a regression even if it parses.
• The same input run twice does not produce contradictory structured fields.

A version that passes the schema but drops a required field, invents a date, or doubles the length has failed, even though every node in n8n went green.

Compare before you promote, and keep the old version

The promotion rule follows from the test: never switch a production workflow to a new model version on the provider's schedule. Run the regression set against the candidate, diff the outputs against the current version's outputs on the same inputs, and read the diff yourself. The diff is where the real change hides — a field that got more verbose, a tone shift, a category that started being used differently. Automated assertions catch the breakage; the diff catches the drift that is not yet breakage.

Keep the previous pinned version working until the new one has run in production for a while. A model change is a deploy, and every deploy needs a rollback. That means versioning the prompt and the model together: a prompt tuned for one version is not guaranteed to behave on the next, so "revert the model" must also mean "revert to the prompt that was tested with it." I store the prompt version and the model version as a pair, and a rollback restores both.

Watch cost and latency, not just quality

One more axis people forget: a new version can be correct and still be a bad deploy. It can be slower, which matters when a workflow holds executions open, or more expensive per token, which matters when you run it a thousand times a day. My regression run records tokens in, tokens out, and wall-clock time per input, and I compare those against the current version before promoting. A version that costs twice as much for the same quality is a pricing decision, not a free upgrade — and it should be made deliberately, not discovered in next month's bill.

The test is a version swap, not a fresh run

Testing this properly means simulating the event that actually hurt me: swap the model version under a working workflow and see what breaks. The test harness is simple. Take the regression set, run it against the currently pinned version, then run the same set against the candidate version, and assert three things: every invariant still holds, the diff in outputs is one you have read and accepted, and cost and latency are within budget. If any of those fails, the candidate does not ship, and the workflow keeps running on the pinned version that already passed.

Count this at the provider side, not in n8n: record which model version actually served each call, because the requested name and the served version are not always the same.

The principle

A model is a dependency. Pin it like one, test a new version like a deploy, and keep the old one until the new one has earned its place. The goal is not to freeze progress — new versions are often better. The goal is to make sure that when the output changes, it changes because you decided it should, on a day you chose, with a rollback ready — not because a provider shipped an update over the weekend and your workflow quietly started answering a different question.

---

I'm building AI Automation Lab — production-ready n8n workflow templates. The free AI Hook Generator shows how I structure real workflows; my open-source templates live at github.com/zhp910318/n8n-ai-automation-lab.