DEV Community: Versyka

A retired Gemini model broke my project, so I built a fix. One month in: 6 users, 50 PRs

Versyka — Thu, 12 Nov 2026 15:00:00 +0000

A Gemini model my project depended on was retired in the middle of the project. Nothing warned me until the calls started failing.

I lost days finding every place it was used, swapping it, and checking that nothing else broke. I build LLM agents for a living, so it wasn't the first time. It was the last straw.

The problem nobody's tools cover

AI vendors now retire models every few months, and SDKs ship breaking versions all the time. The tools we already have don't help much:

Dependabot and Renovate bump a version number. They don't rewrite the code that breaks, and a model ID isn't a package, it's a string in your code, so they never see it.
Coding agents will write a patch if you ask. But you have to know a change is coming, and then you still have to prove the patch is safe.

Writing the patch is the easy part now. Knowing what's changing, and proving the fix doesn't break anything, is the hard part.

What I built

Versyka watches your repositories for AI models and APIs that are retiring or changing. When one affects your code, it:

finds every affected call, with file and line
makes the change (an exact model swap is a plain find-and-replace, no AI involved)
runs your own typecheck, lint, build and tests before and after the change
opens a pull request only if nothing new fails, with the results attached

You review and merge.

The first month, honestly

Versyka launched in October. Today:

6 users
30 repositories connected
50 pull requests opened, including security fixes and AI model replacements

A senior engineer at Kore.ai let it open security fixes on their code: 3 pull requests covering 39 vulnerabilities, two of them major version jumps. The one they merged would have taken about a day by hand.

Small numbers, but real ones.

What I learned

Proof beats promises. "Tests passed before and after" is what makes an engineer open the diff instead of closing the tab.
Not every fix needs AI. Swapping a retired model ID for its direct replacement is exact and free. AI is only worth it for breaking API changes.
Free tools earn trust first. A repository check that needs no account is how most people meet the product.

If your code calls AI models or third-party APIs, I'd really like to know: what's the last upgrade that broke something for you?

Would you merge a PR a bot opened? The flaky-or-real check we run first

Versyka — Wed, 11 Nov 2026 15:00:00 +0000

"I'd never merge a pull request a bot opened." It's the first objection we hear about automated upgrades, and it's the right instinct. The answer isn't trust, it's evidence: you need to know whether a red test was caused by the change at all. Here's the routine, by hand first.

A test goes red on a pull request. The usual responses are "rerun the pipeline" or "start debugging". Both skip a cheaper question: is this failure caused by the change at all?

Step 1: rerun only the failed test

pytest tests/test_checkout.py::test_total -x
npx jest -t "total is correct"

Fails, then passes: flaky. A timeout, a race, a test that depends on order, real network or the current time.
Fails twice: probably real. Keep going.

Step 2: run it on main

git stash && git checkout main
pytest tests/test_checkout.py::test_total

If it fails on main too, your change didn't break it. It was already broken, and that's a separate ticket, not a reason to block this pull request.

Step 3: only now, debug

At this point you know the failure is real and new. That's the only kind worth blocking a merge on.

Why teams skip this

CI shows one colour per run. So the habits are "rerun until green", which hides real failures that happen to pass on a retry, or "block on red", which lets one flaky test hold up every pull request for months.

Baking it into CI

pytest can retry failures and report them separately:

pip install pytest-rerunfailures
pytest --reruns 1

A test that only passes on the rerun is flagged in the output, so it can be tracked instead of forgotten.

The stronger version compares results before and after the change: the same tests run on the base commit and on the pull request, and only new failures count. That's how Versyka decides whether an automated upgrade is safe: tests that were already failing are listed in the pull request, a test that fails once and passes on retry is labelled flaky, and only a new, repeatable failure stops the pull request from opening.

What would you need to see in an automated pull request before you'd merge it?

chalk and debug shipped malware for a few hours. Waiting 7 days would have skipped it.

Versyka — Tue, 10 Nov 2026 15:00:00 +0000

In September 2025, a maintainer of several very popular npm packages, including chalk and debug, had their account phished. New versions were published with code that tried to steal cryptocurrency from browsers. The community spotted it quickly, and the bad versions were removed within hours.

The teams that got hit were the ones whose installs ran in those hours. Teams that waited even a couple of days never saw the bad versions at all.

Most bad releases have a short life

Not every bad release is an attack. Most are ordinary mistakes: a broken build published by accident, a regression the tests missed, a missing file. The pattern is the same:

day 0   release published
day 1   issues opened, maybe a security report
day 2   version deprecated or unpublished, fix released

So a simple rule removes most of the risk: don't install a release until it's a few days old. Seven days is a common choice.

Setting it up

Renovate has it built in:

{
  "minimumReleaseAge": "7 days"
}

Dependabot has a cooldown option in dependabot.yml for the same idea:

updates:
  - package-ecosystem: "npm"
    directory: "/"
    schedule:
      interval: "weekly"
    cooldown:
      default-days: 7

Lockfiles do the rest: with npm ci and a committed package-lock.json, nothing new is installed until a pull request changes the lockfile.

The exception: security fixes

A release that patches a known vulnerability is a different trade-off: the risk is in not installing it. Keep a separate, faster path for security updates, and review them like any other change.

What we do

Versyka's weekly upgrades collect minor and patch updates that are at least 7 days old and open them as one tested pull request, while security fixes take their own path. More in the docs.

Does your dependency bot install releases the day they come out?

Upgrade debt compounds: why three majors behind is a project, not three tasks

Versyka — Mon, 09 Nov 2026 15:00:00 +0000

Every team has one: the SDK that's three majors behind, with a ticket that keeps sliding to next sprint.

It's tempting to think of that as three upgrades' worth of work. It's usually more.

A real example: a senior engineer at Kore.ai let Versyka open security fixes on their code. It found 39 vulnerabilities across 3 pull requests, and two of the three were major version jumps, the kind that sit in a backlog because nobody has a free day. The one they merged would have taken about a day by hand.

Why it compounds

Guides assume the previous step. The v14 migration guide is written for someone on v13. If you're on v12, half of its "before" examples don't match your code, because v13 already changed them.

Changes stack. A method renamed in v13 is removed in v14. The v13 guide's recommended workaround is deprecated in v15. You end up researching history, not just the destination.

The ecosystem moves without you. Plugins, type packages and examples drop support for your version. Stack Overflow answers are for the new API.

Security fixes land on the latest major. At some point a vulnerability is fixed only in a version you can't reach without the whole migration. Now the upgrade is urgent and big at the same time.

A policy that keeps it small

Stay at most one major behind. Upgrade within a few weeks of a new major, after the first patch releases have landed.
Catch up one major at a time. v12 → v13, run the tests, merge. Then v13 → v14. Each step has one guide that matches your code.
Automate the boring part. Patch and minor bumps shouldn't need a human. Save attention for breaking changes.
Measure the gap. You can't manage what nobody can see.

For step 4, a quick script works:

npm outdated --long
pip list --outdated

or paste a package.json / requirements.txt into Versyka's free upgrade check, which shows how many majors behind each API SDK is and links its breaking changes.

And for step 3 and the breaking upgrades themselves, that's what Versyka is for.

How many majors behind is the oldest SDK in your codebase?

The Pydantic v1 to v2 cheat sheet (and the trap the deprecation warnings hide)

Versyka — Sun, 08 Nov 2026 15:00:00 +0000

Pydantic v2 rewrote the core in Rust and renamed a lot of the public API. Most projects hit the same handful of changes first.

The renames

- user.dict()
+ user.model_dump()

- user.json()
+ user.model_dump_json()

- User.parse_obj(data)
+ User.model_validate(data)

- User.parse_raw(text)
+ User.model_validate_json(text)

Validators changed shape, not just name

- from pydantic import validator
+ from pydantic import field_validator

  class User(BaseModel):
      email: str

-     @validator("email", pre=True)
-     def clean(cls, v):
+     @field_validator("email", mode="before")
+     @classmethod
+     def clean(cls, v):
          return v.strip().lower()

pre=True is mode="before", and the validator is an explicit classmethod.

Config moved

  class User(BaseModel):
-     class Config:
-         orm_mode = True
+     model_config = ConfigDict(from_attributes=True)

Note the rename inside it too: orm_mode is from_attributes.

The trap: it mostly still works

v2 keeps most v1 names as deprecated aliases. So after pip install -U pydantic, a lot of code keeps running and prints PydanticDeprecatedSince20 warnings. That's helpful for the upgrade and bad for finishing it: the warnings get filtered out, and the work waits for the release that removes them.

Make them loud in your tests:

# pytest.ini
[pytest]
filterwarnings =
    error::pydantic.warnings.PydanticDeprecatedSince20

Let the tool do most of it

pip install bump-pydantic
bump-pydantic my_package/

It handles most renames and validator changes. Then run your whole test suite, because behaviour changed in places too (for example, v2 is stricter about some type coercions).

This loop of rewrite, run the tests, compare with before, is what Versyka does for breaking upgrades, and it opens the pull request only if nothing new fails.

If your requirements still pin pydantic<2, what's stopping the upgrade?

0.2.3 doesn't mean what you think: npm's caret below 1.0

Versyka — Sat, 07 Nov 2026 15:00:00 +0000

The caret in package.json is so common that most people read it the same way every time: "accept new minors and patches, not majors".

That's right for versions 1.0 and up. Below 1.0, it isn't.

The rule

npm's caret allows changes that don't modify the leftmost non-zero number:

^1.4.2  →  >=1.4.2  <2.0.0     (minors and patches)
^0.2.3  →  >=0.2.3  <0.3.0     (patches only)
^0.0.3  →  >=0.0.3  <0.0.4     (that exact version)

You can check any range yourself:

npx semver -r "^0.2.3" 0.2.9 0.3.0
# 0.2.9

Why it works this way

Semantic versioning says that before 1.0, anything may change at any time. npm's compromise is to treat the first non-zero number as the "major". So 0.2 → 0.3 is a breaking release as far as your range is concerned, and it should be.

What it means day to day

0.x packages stay put. npm update won't move ^0.2.3 to 0.3.0, so a pre-1.0 dependency can fall far behind without anything in your workflow noticing.

The bump that "looks minor" isn't. When you finally go from 0.2.x to 0.3.0, read the changelog like it's a major. It often is.

The tilde is stricter still. ~1.4.2 allows only patches (<1.5.0). For 0.x, ^ and ~ end up close to the same thing.

A quick audit

List your direct dependencies that are still pre-1.0:

node -e 'const p=require("./package.json");for(const [k,v] of Object.entries({...p.dependencies,...p.devDependencies}))if(/^[\^~]?0\./.test(v))console.log(k,v)'

Then compare each against its latest release. Versyka's free upgrade check does that comparison for a whole package.json in one paste, no sign-up.

How many of your dependencies start with ^0?

Next.js 15 is a one-line upgrade. The breaking changes are in every file that reads a cookie.

Versyka — Fri, 06 Nov 2026 15:00:00 +0000

Upgrading to Next.js 15 is one line:

- "next": "14.2.15"
+ "next": "15.0.3"

The work is in the files that line affects. Three changes cause most of it.

1. Request APIs are async

cookies(), headers() and draftMode() now return promises.

  import { cookies } from "next/headers";

  export default async function Page() {
-   const theme = cookies().get("theme");
+   const theme = (await cookies()).get("theme");
    ...
  }

Next.js keeps a temporary synchronous fallback that logs a warning, which is kind for the upgrade and unkind for finding the problem: the code keeps working, and the warning sits in your logs.

In route params, the same applies: params and searchParams in pages and layouts are promises too.

2. `fetch()` isn't cached by default

In 14, fetch() in a server component was cached unless you opted out. In 15 it isn't cached unless you opt in:

await fetch(url, { cache: "force-cache" });

Nothing breaks. Some pages that were fast because of a cache you never meant to rely on get slower.

3. GET route handlers aren't cached

Same idea for app/**/route.ts: a GET handler is dynamic by default now. Add export const dynamic = "force-static" where you really wanted it cached.

Finding every affected file

The official codemod handles most of change 1:

npx @next/codemod@canary upgrade latest

Then search for what's left, because codemods miss dynamic patterns and helper functions:

grep -rnE "cookies\(\)|headers\(\)|draftMode\(\)" app src
grep -rn "fetch(" app src | grep -v "cache:"

And run your build and tests before you call it done. next build catches type errors from the async change if your project is typed.

This "the release notes say what changed, but not where it hits your code" step is the part Versyka automates: it reads the release notes against your repository, rewrites the affected calls and runs your checks before and after.

Have you moved to Next.js 15 yet? What broke first?

OpenAI's own quickstart still calls a retired model. Your tests won't notice either.

Versyka — Thu, 05 Nov 2026 15:00:00 +0000

This week we ran our free repository check on openai/openai-quickstart-python. It still calls gpt-3.5-turbo in 4 places, a model OpenAI stopped serving on October 23, 2026.

That's not a dig at OpenAI. It's how this happens in almost every codebase, including ones with good test coverage.

The suite is green. The pull request merges. The deploy goes out. And every request to the LLM comes back with an error like model_not_found, because the model ID in the config was retired the night before.

The tests didn't fail because the tests never talk to the model.

Why mocking hides it

Mocking the LLM client is the right decision:

@patch("app.llm.client.chat.completions.create")
def test_summary(mock_create):
    mock_create.return_value = fake_response("A short summary.")
    assert summarize("long text") == "A short summary."

It's fast, it's free and it's deterministic. But look at what it checks: your code's handling of a response. It never checks that the request would be accepted. The model ID is just a string that goes into a mock and disappears.

So when OpenAI retired gpt-4-0613 on October 23, a repository still sending it stayed green right up until real traffic hit it.

Three cheap checks

1. Put model IDs in one place.

# config.py
SUMMARY_MODEL = "gpt-5.6-sol"
CLASSIFIER_MODEL = "claude-sonnet-5-5"

If model names are scattered across ten files, no check will find them all.

2. Check them against shutdown dates in CI, not against the API.

Vendors publish retirement dates months ahead. A CI step that reads your config and compares it to a list of dates fails the build before the date, with no API key and no network call to the model. (Versyka's GitHub Action does this, and the retirement calendar has the dates as JSON if you'd rather script it.)

3. One real smoke call per model on deploy.

for model in {SUMMARY_MODEL, CLASSIFIER_MODEL}:
    client.chat.completions.create(model=model, max_tokens=1,
        messages=[{"role": "user", "content": "ping"}])

A one-token request per model costs almost nothing and fails the deploy, not the user.

The next date

Claude Sonnet 4.5's dated ID, claude-sonnet-4-5-20250929, shuts down on November 30. If it's in your repository, that's a good one to test these checks on.

Where do the model names live in your codebase: one config file, or scattered through the code?

Your OpenAI streaming loop works in the demo. It breaks on the last chunk.

Versyka — Wed, 04 Nov 2026 15:00:00 +0000

If you moved from the OpenAI Python SDK 0.28 to 1.x, your non-streaming code probably broke loudly: openai.ChatCompletion.create became client.chat.completions.create, and every response["choices"][0]["message"]["content"] became response.choices[0].message.content. You fixed those in an afternoon.

Streaming code is different. It usually doesn't break at all. It just gets slightly wrong, on the very last chunk, after the user has already read the answer.

Trap 1: `content` is `None`

In 0.28, chunks were dicts, and most people wrote:

text += chunk["choices"][0]["delta"].get("content", "")

A missing key gave you an empty string. In 1.x, chunks are typed objects, and the straight translation is:

text += chunk.choices[0].delta.content   # TypeError on the final chunk

delta.content is None on the final chunk (the one that carries finish_reason) and on chunks that carry tool calls instead of text. If you build the text with an f-string or str(), you don't get an error. You get the word "None" at the end of the user's answer.

Trap 2: the chunk with no choices

Turn on usage reporting:

stream = client.chat.completions.create(
    model=MODEL, messages=messages, stream=True,
    stream_options={"include_usage": True},
)

and the last chunk carries usage with an empty choices list. chunk.choices[0] raises IndexError on the final chunk, after the user has already seen a complete answer. In logs it looks like a random failure at the end of a request.

The guard

text = ""
for chunk in stream:
    if not chunk.choices:          # usage-only final chunk
        continue
    text += chunk.choices[0].delta.content or ""

or "" covers the None, and continue skips the usage chunk.

Why your tests didn't catch it

Most test suites mock the LLM with one hand-written chunk. Both traps live on the first or last chunk of a real stream, so a mock never reaches them. The only test I trust for streaming reads a real stream (or a recorded one) to the end and checks the joined text.

Finding every place it hides

Streaming loops tend to get copy-pasted: a chat endpoint, a background summarizer, a CLI script. Search for .delta.content and choices[0] across the whole repository, not only the file you remember. That search, plus rewriting what it finds and running your tests before and after, is what Versyka automates when an SDK ships a breaking change.

What's the strangest bug a streaming response has given you? Tell me in the comments.

API deprecation: shut down v2 without stranding your customers

Versyka — Tue, 03 Nov 2026 15:00:00 +0000

Every API team that has shut down a version knows the curve. A quick drop in v2 traffic after the announcement, then a long, flat tail of customers who never move. Shutdown day arrives, support gets the calls, and the date moves.

Why the tail doesn't move

It isn't that customers didn't read the email. They did. They added a ticket. The ticket competes with their own roadmap, and "rewrite working code to call a newer endpoint" loses every sprint planning.

The cost of migrating sits in the customer's codebase, and nothing you send changes that.

What actually helps, in order of effort

A migration guide with exact before and after code for each SDK language. Not a list of changed fields, actual code.
Codemods: scripts that rewrite the common call patterns. Customers still have to find, run and review them.
Response headers and SDK warnings (Deprecation, Sunset, a log line on every v2 call), so the developer sees it, not just the account owner.
A migration in their repository: a pull request with their calls rewritten and their tests run. Merging is a review, not a project.

Each step moves more of the tail, because each one moves more of the work off the customer.

Doing step 4 at scale

That's the provider program in Versyka:

Share your install page. You get versyka.com/install/your-company to put in your changelog, upgrade emails and docs. Customers who install from it opt in; nothing touches a repository without them.
Publish the change once. Paste the migration guide you already write; it becomes a precise rewrite rule.
Customers get tested pull requests. Every opted-in repository that uses your SDK gets one, with the customer's own tests run. It's free for them: you sponsor it, and nothing merges without their review.
Watch adoption. The portal shows pull requests opened, merged and repositories not affected, per change. Counts only: customer names stay private.

Plans start at $1,500 a month on an annual contract.

Even if you never use a tool for it

Put the before and after code in your guide. Send the deprecation signal through the SDK, not only the inbox. And count the tail by repository, not by account. That last number tells you what shutdown day will look like.

If you've run a version shutdown, I'd like to know what moved your long tail. Comments are open.

Why we price a developer tool per fix instead of per seat

Versyka — Mon, 02 Nov 2026 15:00:00 +0000

When we priced Versyka, per seat was the obvious default. Almost every developer tool does it.

It didn't fit, for three reasons.

1. A bot doesn't have seats

Versyka opens pull requests. The people who review them might be one platform engineer or the whole team. Charging for each of them would mean charging more because more people can look at a PR, which is backwards.

2. The value is in the fix, not the access

What a team actually wants is a breaking upgrade done: the code rewritten, the tests passing, a PR to review. An afternoon of an engineer's time that didn't get spent. That's the unit worth paying for.

3. A failed attempt shouldn't cost you

If the change can't pass your tests, you didn't get anything. So it isn't counted.

What counts as a fix

A migration pull request: a breaking upgrade where Versyka changed your code and your tests passed.

What doesn't count:

Version upgrades with no code changes. Unlimited on every plan (with a fair-use cap on daily test runs), because they're cheap to run: no AI, just your tests in a sandbox.
Replacing a retiring AI model with its direct replacement.
Attempts that never passed your tests.
Seats, from the Team plan up: everyone with write access signs in, no seat fees. Free and Starter include you and two teammates.

The plans

Plan	Price	Fixes / month	Per fix
Free	$0	2 (5 repositories)
Starter	$49/mo	10	$4.90
Team	$249/mo	60	$4.15
Scale	$749/mo	200	$3.75

The trade-off we accepted

Usage pricing is harder to predict than seats, for us and for you. Two things help: the monthly allowance is a ceiling (nothing surprise-bills), and the dashboard shows how many fixes are left this month before any work starts.

I'd like to hear from people who've bought or built usage-priced dev tools: what made the unit feel fair, and what made it feel like a meter running?

Fail CI when a pull request adds a retired OpenAI or Anthropic model

Versyka — Sun, 01 Nov 2026 15:00:00 +0000

A retired AI model doesn't fail your tests, because your tests mock the LLM. It fails in production, the day after the shutdown date.

So check model IDs where you check everything else: on the pull request.

The workflow

.github/workflows/models.yml:

name: AI models
on: pull_request

jobs:
  models:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write   # for the PR comment
    steps:
      - uses: actions/checkout@v4
      - uses: versyka/versyka-action@v1

That's it. No account, no API key.

What happens on a PR

A retired model (say gpt-4-0613): the check fails, and a comment on the PR says which file and line, and what to use instead.
A model retiring within 90 days: the check passes, with a heads-up in the comment.
Everything current: the check passes quietly.

The comment is updated in place on each push, so the PR doesn't fill up with bot messages.

Options

      - uses: versyka/versyka-action@v1
        with:
          fail-on: soon     # none | retired (default) | soon | behind
          days: 60          # how far ahead counts as "soon" (default 90)
          comment: true     # comment on the PR (default true)
          path: services/api  # folder to check, for monorepos (default .)

fail-on: soon is useful the month before a big shutdown date: it turns the heads-up into a blocker.

Outputs

The step exposes retired, soon and behind as outputs, so you can branch on them:

      - id: models
        uses: versyka/versyka-action@v1
        with:
          fail-on: none
      - if: steps.models.outputs.soon != '0'
        run: echo "Models retiring soon: ${{ steps.models.outputs.soon }}"

Where the dates come from

The same retirement data behind versyka.com/model-retirements, with dates from OpenAI, Anthropic, Google, Mistral, Cohere, AWS Bedrock, Azure, Groq and Together AI.

The action is on the GitHub Marketplace.