DEV Community: Zephyre

GitHub Models retires on July 30: six places to check before your API breaks

Zephyre — Sun, 26 Jul 2026 02:43:12 +0000

GitHub has confirmed that GitHub Models will be fully retired on July 30, 2026. The playground, model catalog, inference API, and BYOK endpoints will stop working for all customers, including existing users with active usage.

It is July 26. If a project has ever used GitHub Models, I would check these six places before waiting for the first production error.

1. Base URLs and provider configuration

Search for GitHub Models endpoints, provider names, and custom base_url values in:

application code;
deployment environment variables;
SDK initialization;
CLI configuration;
serverless or worker settings;
CI test environments.

2. Model IDs

Build a list of production defaults, fallback models, test-only models, and any compatibility aliases supplied by an API relay.

After migration, do not stop at “HTTP 200.” Record the response model field and verify that the client or relay did not silently switch models.

3. Tokens, PATs, and BYOK secrets

Do not copy an existing secret into a new provider by habit. Create a separate key, update the secret manager, verify permissions and quotas, then revoke the old key after the cutover is proven.

Never paste full keys into logs, issues, screenshots, or community chats.

4. SDK and response compatibility

“OpenAI-compatible” does not guarantee identical behavior. Test:

streaming chunks;
tool or function calls;
constrained JSON output;
token usage fields;
error classes and retry behavior;
context limits and timeouts.

5. CI regression tests

Use at least three small requests with no production data:

a minimal text completion;
structured output or a tool call;
failure tests for timeout, rate limit, and unknown model IDs.

Validate output quality and failure behavior separately.

6. Rollback and fallback routes

Keep the provider switch reversible. You need a configuration toggle, one verified backup model, a finite retry limit, idempotency for repeated calls, and a before/after comparison of cost, latency, and errors.

Minimum definition of done

I would call the migration complete only when:

production and CI no longer depend on the retiring endpoints;
critical requests pass on the new route;
response model, usage, and billing can be reconciled;
429, 5xx, and timeout errors cannot trigger infinite retries;
the rollback switch has been exercised once.

We are collecting sanitized migration results and relay-routing checks in a small Telegram group.

Join: https://t.me/+13g2ma9APiU0YjRl

After joining, reply with GitHub Models + your stack, for example GitHub Models + Python/OpenAI SDK, and we will add stack-specific checks.

Disclosure: this is an AllRouter-related community, not an independent benchmark organization. No purchase is required. Never share API keys, private repositories, full billing screenshots, or sensitive logs.

Reference:

GitHub's official retirement notice: https://github.blog/changelog/2026-07-01-github-models-is-being-fully-retired-on-july-30-2026/

DeepSeek retired its legacy model IDs. Is your API relay still mapping them correctly?

Zephyre — Sun, 26 Jul 2026 02:10:17 +0000

DeepSeek retired the legacy deepseek-chat and deepseek-reasoner model IDs on July 24, 2026.

If you call the official endpoint directly, the migration path is relatively visible. If you use an OpenAI-compatible client or an API relay, however, a successful HTTP response does not tell you what happened to the old ID.

There are at least three possible outcomes:

the relay keeps a compatibility alias and maps the old ID to a new model;
the relay returns a clear migration error;
the client or relay silently falls back to another route.

The third case is not automatically malicious, but it changes how you should interpret quality, price, and incidents.

My minimum check is:

send one fixed, low-risk request with no production data;
record the model ID sent by the client;
save the response model field and the exact error class;
compare the provider migration notice with the relay's model page;
verify whether 400, 404, or 429 responses trigger a silent retry to another model.

For example, AllRouter's public model list still displays deepseek-reasoner. That display ID may be a compatibility alias. The name alone is not evidence that the relay is serving the retired official model, so the useful question is whether the mapping and failure behavior can be reproduced.

We are collecting sanitized results from different clients and routes in a small Telegram group: whether the old ID remains compatible, fails clearly, or is silently remapped, plus the migration steps used.

Join: https://t.me/+13g2ma9APiU0YjRl

Reply with migration + client name after joining, and we will add the relevant checklist for Claude Code, Codex CLI, the OpenAI SDK, or another client.

Disclosure: this is an AllRouter-related community, not an independent benchmark organization. No purchase is required to participate. Never share API keys, private repositories, full billing screenshots, or sensitive logs.

References:

DeepSeek V4 Preview and migration notice: https://api-docs.deepseek.com/news/news250424
DeepSeek API documentation: https://api-docs.deepseek.com/

Same coding task, three routes: one model's own test failed

Zephyre — Thu, 23 Jul 2026 13:59:12 +0000

I ran one small JavaScript repair task through three selectable model routes in the AllRouter playground. The goal was not to prove that one route is universally better. It was to check two things that are easy to reproduce: whether the repaired function works, and whether the model's own tests are internally consistent.

The broken function was:

function normalize(nums) {
  return [...new Set(nums)].sort();
}

The fixed version had to deduplicate values, sort numerically, remove NaN and infinities, and include five runnable console.assert checks. I used the same prompt and the same basic playground settings for each run.

What happened

Selected route label	Result	What I verified
`GLM5.2-codex`	Pass	The function and all five assertions were coherent. The response stream reported `model: GLM5.2`, so the selected label and returned model field should be recorded separately.
`Kimi-K2.7-Code-codex`	Partial pass	The function was correct, but one generated assertion expected both `-0` and `0` to survive a `Set`. JavaScript treats them as the same Set value, so the model's own test failed.
`gpt-5.4-mini`	Pass	The function and all five assertions were coherent.

The interesting failure was not in the repaired function. It was in the test the model wrote for itself:

console.assert(
  JSON.stringify(normalize([10, 2, 1, 2, -0, 0, NaN])) ===
    '[0,0,1,2,10]'
);

The actual result is [0,1,2,10]. A quick answer review could miss this because the implementation looks right.

This is a deliberately small fixture, not a claim that a relay never changes model behavior. I also did not publish a cost comparison: the playground calls did not appear as reconcilable rows in the current usage-log window, so I do not have billing evidence for this run.

If you want to reproduce it, make one low-risk call first. Save the selected route label, the response model field, the assertion result, and the billing state before deciding whether to add credit:

https://allrouter.ai/register?aff=qjpC&utm_source=devto&utm_content=E1_task_cost_20260723

This URL contains a partner referral parameter. Do not share API keys, private source code, or private prompts.

Is That API Really Serving the Model It Claims? A Reproducible Fingerprint Test

Zephyre — Wed, 22 Jul 2026 05:13:29 +0000

An API can return a valid response, use the model name you requested, and still leave an important question unanswered:

Is the endpoint behaving like the model it claims to serve?

A single answer cannot settle this. Style guessing is unreliable, benchmark prompts are easy to overfit, and providers can update models without notice. A useful test needs to be cheap, repeatable, and honest about uncertainty.

This is the workflow we are using for the AllRouter Public Model Fingerprint Lab. AllRouter is not treated as the judge. It is one endpoint under the same public test.

The idea: compare distributions, not prose

The paper One Token Is Enough shows that models have surprisingly stable preferences when they answer simple one-token questions repeatedly.

Ask for a random number, color, letter, city, or coin flip 25 times. One answer means almost nothing. The distribution across many answers is a behavioral fingerprint.

A practical audit looks like this:

Select 8 or more task-language cells.
Sample each cell repeatedly at a fixed temperature.
Normalize the one-word answers.
Compare the observed distributions with a trusted reference using Jensen-Shannon divergence.
Repeat the run on another day before drawing a strong conclusion.

The paper reports an equal error rate of about 10.6% with 8 cells and about 7.3% with 40 cells. That is useful evidence, not perfect identification.

Four verdicts, not a binary accusation

The open-source implementation we are using reports four states:

match: behavior is statistically consistent with the reference.
uncertain: the result is in the gray zone; collect more samples or repeat later.
mismatch: behavior differs substantially; investigate before concluding why.
insufficient: there are not enough valid comparable samples.

A mismatch does not prove fraud. Model updates, quantization, hidden system prompts, reasoning fallbacks, and load-balancing across several backends can all move the fingerprint.

That limitation is part of the result, not fine print.

Run the test locally

The MIT-licensed llm-fingerprint-detector provides a TypeScript CLI and library for OpenAI-compatible endpoints.

The fixed experiment definitions, public report schema, and blank result card are in the public protocol directory.

Keep the API key in a local environment variable. Do not paste it into a form, chat, issue, or public log.

git clone https://github.com/ToseaAI/llm-fingerprint-detector.git
cd llm-fingerprint-detector
npm install
npm run build

export LLM_FINGERPRINT_API_KEY='your-local-key'

node dist/cli.js fingerprint \
  --base-url https://your-endpoint.example.com/v1 \
  --model your-model-id \
  --out observed.json

To verify against a trusted reference:

node dist/cli.js verify \
  --base-url https://your-endpoint.example.com/v1 \
  --model your-model-id \
  --reference trusted-reference.json

Record the model ID, test-set version, timestamp, temperature, sample count, comparable cells, mean JSD, split-half consistency, warnings, and a hash of the response summary. Never publish the key or private prompts.

The test I want providers to publish

A provider should not publish only a green badge. A useful public report should include:

Field	Why it matters
Test-set version	Prevents silent changes to the experiment
Collection time	Makes model updates and drift visible
Declared model ID	Defines the claim being tested
Mean JSD and comparable cells	Shows the basis for the verdict
Split-half JSD	Reveals whether the endpoint disagrees with itself
Error and invalid-sample rates	Exposes reliability problems
Reproduction command	Lets a third party repeat the run
Limitations	Prevents overclaiming

We will run the same test on AllRouter at a fixed time for seven days and publish match, uncertain, mismatch, or insufficient without forcing a positive conclusion.

Why this can be more useful than a generic benchmark

Benchmarks ask whether a model is capable. A routing audit asks whether the endpoint stays consistent with the model and configuration you chose.

That matters for coding agents. A silent route change can alter formatting discipline, tool-use behavior, latency, refusal boundaries, and the amount of review a patch needs. Even when a response is syntactically valid, the workflow cost can change.

The identity test should therefore sit next to a small task suite:

preserve a public API during a refactor;
handle missing configuration explicitly;
follow a strict JSON schema;
separate risk analysis from the edit;
keep the diff inside the requested boundary.

The fingerprint tells you whether behavior changed. The task suite tells you whether that change matters to your workflow.

Re-run the same experiment on AllRouter

If you want to compare the same OpenAI-compatible workflow on AllRouter, use the campaign-specific entry below:

Run the same test on AllRouter

The goal is not “trust this router.” The goal is the opposite: make the result reproducible enough that trust is not the only evidence available.

In the next report, we will publish the seven-day drift format, including latency, error rate, split-half consistency, and the exact decision thresholds.

A No-Downgrade Self-Test for GLM-5.2 Coding Routes

Zephyre — Fri, 10 Jul 2026 03:36:42 +0000

When I route coding work to a lower-cost model, I do not want the first question to be "is it cheaper?"

The first question is:

Can I tell whether this route behaves like the model I intended to use?

That is especially important when the route is used as a backup for Claude Code limits or routine coding work. A cheap route is useful only if the failure mode is visible early.

The small test suite I would run first

I would not use a single benchmark score. For day-to-day coding, I care more about whether the model handles the boring but failure-prone parts of software work.

1. Preserve existing behavior

Give the model a small refactor task with clear constraints:

keep the public API unchanged
do not rename exported fields
do not change error messages
do not add new dependencies

Pass condition:

the diff is smaller than the original file
behavior is preserved
the model explains what it intentionally did not change

Fail condition:

it rewrites the module style for no reason
it changes defaults or fallback behavior
it invents a cleaner API that callers do not use

2. Handle an empty configuration

Ask it to fix code that crashes when config is missing.

Pass condition:

missing config is handled explicitly
logs or errors are useful
default values are not silently dangerous

Fail condition:

it hides the error with a broad catch
it returns a fake success state
it changes billing, permissions, or routing defaults without calling that out

3. Explain the risk before editing

Before the patch, ask for a short risk list.

Pass condition:

it names the risky areas
it separates mechanical edits from behavior changes
it asks for missing acceptance criteria when needed

Fail condition:

it jumps straight to code
it treats tests it wrote itself as enough proof
it misses user-visible behavior

4. Use independent evidence

The same model should not be the only verifier of its own patch.

Pass condition:

it points to existing tests, logs, repro steps, fixtures, or human acceptance criteria
it marks missing evidence as missing
it does not overclaim

Fail condition:

it writes new tests and says the patch is verified only because those tests pass
it relies on its own explanation as proof
it cannot distinguish evidence from confidence

5. Stay within a narrow task boundary

Give it a small issue and a tempting nearby cleanup.

Pass condition:

it solves the requested issue
it leaves unrelated cleanup alone
it explains follow-up work separately

Fail condition:

it turns a bug fix into a broad rewrite
it changes formatting, naming, and structure without need
it makes review more expensive than the original problem

The important part is not the score

I would record the result like this:

Test	Pass / Fail	Evidence	Notes
Preserve existing behavior		existing tests / diff review / repro
Empty config handling		logs / error path / fixture
Risk before edit		risk list / acceptance criteria
Independent evidence		existing source of truth
Narrow task boundary		diff scope / review notes

The empty cells matter. They show which parts are not verified yet.

Why this matters

For low-cost coding routes, the expensive part is often not generation. It is review.

If the model saves tokens but increases the human review burden, it is not actually cheap. If the route can pass small, reproducible, independent checks, then it becomes much easier to decide which work belongs there.

My current rule:

Use cheaper routes for tasks with cheap independent verification. Keep risky behavior changes on the strongest model or behind a human review gate.

That rule has been more useful than asking whether a model is generally "good at coding."

Verification Cost Is the Real AI Coding Cost

Zephyre — Sun, 28 Jun 2026 10:17:56 +0000

I used to ask a simple question when routing coding tasks across models:

Which model is strong enough for this?

That question is still useful, but it is not the first one I ask anymore.

The better first question is:

How quickly can I verify the output?

That changed the way I use low-cost models. I do not treat them as weaker replacements for my main coding model. I treat them as useful workers for tasks where the verification path is short.

Level 1: Can I inspect the output directly?

Some tasks are cheap to review because the output is visible.

Examples:

README cleanup
usage examples
comments
changelog notes
small formatting scripts
issue templates

If the model writes a bad README paragraph, I can see it. If it adds vague wording, I can delete it. The failure is annoying, but it is cheap.

This is where low-cost models are useful.

Level 2: Can I run a test?

The next best category is testable work.

If I can describe the expected behavior and run a test suite, I am more willing to route the first draft to a cheaper model.

But the prompt needs boundaries.

Instead of:

Add tests for this helper.

I would write:

Add tests for empty input, null input, duplicate values, invalid config, default config, and normal input. Do not change runtime code.

The difference is small, but it forces the model to work inside a verification frame.

Level 3: Can I manually verify it?

Some tasks do not have automated tests, but still have a clear manual check.

Examples:

CLI output formatting
config examples
migration dry-run notes
small data conversion scripts

For these, I ask the model to include:

how to run it
what input to use
what output to expect
which edge cases to check

If the model cannot explain how to verify its own output, I do not trust the patch.

Level 4: Could it change hidden behavior?

This is where I slow down.

Small refactors are often more dangerous than they look.

The diff may be short. The code may look cleaner. But the behavior might change in a fallback path, a default value, a permission check, or a compatibility branch.

I raise the risk level when a task touches:

fallbacks
defaults
routing
permissions
billing
rate limits
migrations
backwards compatibility

These failures are not always obvious in the code review. You need context to notice them.

My current routing rule

I route by verification cost:

Low verification cost: low-cost model can draft it.
Medium verification cost: low-cost model can draft, human edits.
High verification cost: strong model may help, but tests and human review are required.

This rule is more useful than “small task vs large task.”

A small task can be expensive if it is hard to verify.

The point

Low-cost AI coding models are not useless.

They are useful when the work is easy to inspect, easy to test, or easy to roll back.

The expensive part of AI coding is not always generation.

Often, it is trust.

A Verification Ladder for Low-Cost AI Coding Models

Zephyre — Sun, 28 Jun 2026 10:16:24 +0000

I used to ask a simple question when routing coding tasks across models:

Which model is strong enough for this?

That question is still useful, but it is not the first one I ask anymore.

The better first question is:

How quickly can I verify the output?

That changed the way I use low-cost models. I do not treat them as weaker replacements for my main coding model. I treat them as useful workers for tasks where the verification path is short.

Level 1: Can I inspect the output directly?

Some tasks are cheap to review because the output is visible.

Examples:

README cleanup
usage examples
comments
changelog notes
small formatting scripts
issue templates

If the model writes a bad README paragraph, I can see it. If it adds vague wording, I can delete it. The failure is annoying, but it is cheap.

This is where low-cost models are useful.

Level 2: Can I run a test?

The next best category is testable work.

If I can describe the expected behavior and run a test suite, I am more willing to route the first draft to a cheaper model.

But the prompt needs boundaries.

Instead of:

Add tests for this helper.

I would write:

Add tests for empty input, null input, duplicate values, invalid config, default config, and normal input. Do not change runtime code.

The difference is small, but it forces the model to work inside a verification frame.

Level 3: Can I manually verify it?

Some tasks do not have automated tests, but still have a clear manual check.

Examples:

CLI output formatting
config examples
migration dry-run notes
small data conversion scripts

For these, I ask the model to include:

how to run it
what input to use
what output to expect
which edge cases to check

If the model cannot explain how to verify its own output, I do not trust the patch.

Level 4: Could it change hidden behavior?

This is where I slow down.

Small refactors are often more dangerous than they look.

The diff may be short. The code may look cleaner. But the behavior might change in a fallback path, a default value, a permission check, or a compatibility branch.

I raise the risk level when a task touches:

fallbacks
defaults
routing
permissions
billing
rate limits
migrations
backwards compatibility

These failures are not always obvious in the code review. You need context to notice them.

My current routing rule

I route by verification cost:

Low verification cost: low-cost model can draft it.
Medium verification cost: low-cost model can draft, human edits.
High verification cost: strong model may help, but tests and human review are required.

This rule is more useful than “small task vs large task.”

A small task can be expensive if it is hard to verify.

The point

Low-cost AI coding models are not useless.

They are useful when the work is easy to inspect, easy to test, or easy to roll back.

The expensive part of AI coding is not always generation.

Often, it is trust.