DEV Community: FlareCanary

GitHub App installation tokens are getting longer in May 2026 — your VARCHAR(40) column is about to silently truncate them

FlareCanary — Wed, 06 May 2026 04:03:54 +0000

GitHub announced on April 24, 2026 that installation access tokens for GitHub Apps are changing format. Starting with a brownout mid-May 2026 and full cutover by late June 2026, tokens will grow from the current 40 characters (ghs_ + 36 chars) to up to roughly 520 characters. The prefix stays ghs_. The character set stays the same. Only the length changes — and only upward, and only sometimes.

That last part is the trap. During and after the rollout, some tokens will still be 40 chars (issued before the change, cached, returned by older Enterprise Server versions) and some will be 200, 380, 520. The same App, same installation, same call, on different days returns different lengths. There's no transition flag. There's no version header. The token still parses as bytes. It just doesn't fit anywhere you assumed it would.

The four shapes of the failure

# All four of these silently corrupt or reject valid tokens after May 2026.

# 1. Database column too narrow.
CREATE TABLE app_tokens (
    installation_id BIGINT,
    token VARCHAR(40),  -- silently truncates to 40 chars on insert
    expires_at TIMESTAMP
);

# 2. Regex validator pinned to old length.
TOKEN_RE = re.compile(r'^ghs_[A-Za-z0-9]{36}$')  # rejects new tokens
if not TOKEN_RE.match(token):
    raise ValueError("invalid token")

# 3. Length assertion in middleware.
assert len(token) == 40, f"expected 40-char token, got {len(token)}"

# 4. Fixed-size buffer in C/Go/Rust FFI.
char token_buf[64];  // overflows when memcpy'd, or strncpy truncates

The first three return 401 from GitHub with no helpful message — your code stored or transmitted a truncated/rejected token, GitHub rejected the truncated value, and the error trail goes cold one frame above the auth call. The fourth gets you a memory bug.

Why this is a quiet failure, not a loud one

Three reasons this rolls out as silent corruption rather than red-letter outage:

Type system unchanged. The token is still a string. Static analysis, schema validation, OpenAPI spec, type guards — all still pass. No compile error, no schema drift alarm. Octokit, PyGithub, go-github all return string from their token endpoints today and tomorrow.
Runtime unchanged at the issuance call. POST /app/installations/{installation_id}/access_tokens still returns 201 with a token field. Your code reads the field, uses it, gets a 401 on the next call — far from the issuance frame. A naive retry-on-401 hides it briefly, until the new token of the new size also fails to fit.
Brownout is intermittent. Per the GitHub announcement, the change rolls out as a brownout starting mid-May before full cutover late June. During the brownout window, the same token endpoint can return a 40-char token at 9:00am and a 380-char token at 9:30am. Tests written against a recorded fixture pass. CI passes. Production hits the brownout at unpredictable times.

Where to grep

Search every repo that touches a GitHub App for the four patterns:

# 1. DB column types
git grep -nE "token.*VARCHAR\(([0-9]+)\)" -- '*.sql' '*.ts' '*.py' '*.rb' '*.go'
git grep -nE "varchar\(40\)|VARCHAR\(40\)|String\(40\)" --

# 2. Regex validators
git grep -nE 'ghs_\[A-Za-z0-9\]\{[0-9]+\}|ghs_\\\w\{[0-9]+\}' --
git grep -n 'ghs_' -- '*.py' '*.ts' '*.go' '*.rb' '*.java' | grep -E 'match|regex|RE|Pattern'

# 3. Length assertions
git grep -nE 'len\(token\)\s*==\s*40|token\.length\s*==\s*40' --
git grep -nE 'fixed.*40|token\[0:40\]|substring\(0, 40\)' --

# 4. Fixed-size buffers
git grep -nE 'char token\[[0-9]+\]|\[40\]byte|token\[64\]' -- '*.c' '*.cpp' '*.go' '*.rs'

Common hiding spots beyond what grep catches:

Caching layers. Redis with MAXLEN, Memcached with item-size limits (default 1MB is fine, but custom-tuned smaller installs are not).
Audit logs. A token-redaction filter that masks the first 36 chars after ghs_ still leaks the last hundred characters of the new tokens.
Webhook signature verification. Code that uses an installation token in a downstream service's HMAC by hashing a fixed prefix length.
Environment variable storage. Some platforms truncate env vars over a length. Heroku, Vercel, Cloud Run all have limits per-platform; check your edition.
JWT claims. Apps that embed an installation token in a custom claim and the verifier asserts a fixed claim size.
Test fixtures. Recorded VCR cassettes, json fixtures, mock objects with hardcoded 40-char strings.

The most-bitten codebase is the one that wrote a Probot/Octokit-style validateToken helper years ago, copied a regex from a Stack Overflow answer, and forgot it existed.

Why VARCHAR(40) bites the hardest

Postgres and MySQL with strict mode raise an error when you try to insert a string longer than the column declared length — that's the good outcome, because the insert fails loudly and the call site gets the exception.

The bad outcome is what most production systems actually do:

MySQL with non-strict mode (the historic default before 5.7, and still common in older Docker images): silently truncates and emits a warning.
SQL Server with ANSI_WARNINGS OFF: silently truncates.
Some ORMs with default-string-conversion: do the truncation client-side before send.

The token gets stored at 40 chars. Reads return 40 chars. The auth call sends 40 chars to GitHub. GitHub rejects with 401. Your stack trace shows a 401 from POST /repos/.../check-runs — five layers away from the truncating column.

The fix is a column type change: ALTER TABLE app_tokens ALTER COLUMN token TYPE TEXT. There's no business reason to constrain token length at the storage layer; tokens are opaque to your app.

The migration that doesn't migrate cleanly

The obvious fix — widen all the columns, drop all the regexes, remove all the assertions — is correct end-state but introduces a window in the middle where the new tokens land in code paths that also still have stale references to the old format.

Three places this bites:

Reverse proxies and middleware. A Cloudflare Worker or Express middleware that strips/validates tokens. If you update the App but not the Worker, the Worker rejects valid tokens.
Cross-service token forwarding. Service A fetches a token, hands it to Service B, B logs and validates the format. Updating only A means B starts dropping tokens it gets from A.
CI/CD secret scanning. Internal secret scanners that pattern-match on ghs_[A-Za-z0-9]{36} will stop catching leaked tokens after the format change, because the regex no longer matches the new format. Update the scanners or you have a leak detection regression at the same time as the migration.

The order to roll: scanner regexes first (they need the broadest pattern, ghs_[A-Za-z0-9]+), then storage layer, then validators and assertions. The App itself needs no change — Octokit and friends will get the new tokens automatically once GitHub starts issuing them.

What "broadest pattern" means for the regex

Don't anchor on the new max length (~520) either; that's a current ceiling that GitHub may move again later. Anchor only on the prefix and character set:

# Wrong — pins to current new ceiling
TOKEN_RE = re.compile(r'^ghs_[A-Za-z0-9]{36,520}$')

# Right — accepts anything matching prefix and charset
TOKEN_RE = re.compile(r'^ghs_[A-Za-z0-9]+$')

If you actually need a length constraint for a defensive reason (e.g., a sanity check before storing), set a generous upper bound — 4096 is a reasonable belt-and-suspenders ceiling that won't bind on a future format change.

What's not changing

Prefix. Still ghs_. (Personal access tokens use ghp_, OAuth tokens gho_, app-to-server ghu_. None of these are announced as changing in this rollout, but the same lessons apply if they do later — strip your fixed lengths now.)
Issuance API. POST /app/installations/{installation_id}/access_tokens returns the same JSON shape.
Revocation API. DELETE /installation/token still works the same way.
Token TTL. Still 1 hour from issuance.
Permissions model. Unchanged.

Minimum-viable fix

git grep the four patterns above (column type, regex, length assertion, fixed buffer). Inventory every match.
Update DB columns to TEXT (or equivalent unbounded string type). For MySQL specifically, drop indexes on the token column before changing type if you have any — the index becomes invalid after the change.
Replace fixed-length regexes with prefix-only validation (^ghs_[A-Za-z0-9]+$) or remove the validation entirely (tokens are opaque, your code shouldn't be parsing them).
Remove length assertions in middleware and FFI boundaries. Resize fixed-size buffers to a generous ceiling (1024 or 4096).
Update internal secret scanners first — before any token-handling change — so leak detection doesn't regress mid-migration.
Add a real integration test against POST /app/installations/{id}/access_tokens and assert that the returned token round-trips through your storage layer without modification (use a length-comparison check, not a string-equality check, to keep the test stable across token issuances).
If you operate GitHub Enterprise Server, plan for the format change in your next GHES upgrade — the brownout schedule for GHES typically lags the GitHub.com schedule by one or two minor versions.

The pattern this fits

Token format changes are the canonical silent-fail. The type system sees a string. The schema sees a string. The contract test sees a string. The thing that breaks is an assumption about the string's shape — and assumptions don't show up in any contract.

GitHub has done this before (the token format change in 2021 introduced the ghp_, ghs_, etc. prefixes), and the same shape of breakage rolled out then: VARCHAR columns silently truncating, regex validators silently rejecting, fixed-buffer assertions overflowing. The fix five years ago was the same fix today: don't constrain the storage size, don't validate the format past the prefix, don't bake assumptions about token shape into anything except the issuance call itself.

If you're treating an opaque token as anything more structured than "an opaque blob from GitHub that is at most a few KB," you're carrying a latent bug that will trip on the next format change after this one.

FlareCanary monitors REST APIs and MCP servers for schema drift. Free tier covers 5 endpoints with daily checks.

Cloudflare is removing in-place DNS record type changes on June 30, 2026 — your Pulumi runs will start failing

FlareCanary — Wed, 06 May 2026 04:01:54 +0000

Cloudflare announced on January 23, 2026 that changing the type of an existing DNS record via the API is deprecated. End-of-life: June 30, 2026. After that date, a PATCH or PUT against /zones/{zone_id}/dns_records/{id} that flips a record's type field — A → CNAME, CNAME → A, TXT → MX — stops being supported.

The deprecation page doesn't promise a specific error code, just that "attempts to change a record's type via update operations will no longer be supported." Read that as: today it works, July 1 it doesn't, and the failure shape (4xx? 200 with success: false? silent partial?) won't be locked down until cutover.

If you're in the 90% of teams using the official Cloudflare Terraform provider on a recent version, you're probably fine — the provider already tags type as RequiresReplace, so it does delete-then-create on the client side (cloudflare/terraform-provider-cloudflare#6358). The teams that get bit are the ones running anything else against dns_records/{id} with a different type than what's in the record.

Where the breakage actually lives

Search your repos for direct DNS PATCH/PUT calls, not just Terraform configs:

git grep -nE "dns_records/[A-Za-z0-9]+" -- '*.py' '*.ts' '*.js' '*.go' '*.rb' '*.sh'
git grep -nE 'PATCH|PUT' -- '*.tf' '*.yaml' '*.yml' | grep -i dns

Common hiding spots:

OpenTofu / older Cloudflare provider forks. The RequiresReplace modifier landed in the schema rewrite — if you're pinned to a pre-v5 provider, you're still hitting the in-place PATCH path under the hood.
Pulumi cloudflare.Record. The Pulumi provider mirrors the upstream API's PATCH semantics for property updates. Confirm against your provider version that a type change forces replacement, not update.
CDK / SDK direct calls. Anything using cloudflare-sdk or the raw requests/fetch-against-the-REST-API pattern, where someone wrote a "reconciler" that PATCHes whatever fields differ.
Internal admin tools. That Slack bot you wrote to flip a hostname between origins. The internal "fix DNS" runbook in your wiki. The migration script your platform team kept after the last datacenter move.
CI jobs that re-apply DNS state. GitHub Actions workflows that call cf-cli set-record or equivalent and assume the API will figure out an in-place update.

The Cloudflare dashboard uses PATCH for type changes today, so anyone who reverse-engineered the dashboard's calls into a script (looking at you, "I just used the network tab") has a script that breaks on July 1.

Why "just delete and create" is a footgun

The naive migration path — delete the old record, create a new one with the new type — is correct for end-state but creates a non-zero NXDOMAIN window between the two API calls. Two consequences:

Real DNS resolvers in the path will cache the NXDOMAIN. Most won't honor a TTL of zero on a negative answer. Mail servers in particular are aggressive negative cachers.
Concurrent automation can race the gap. If your reconciler runs every 60 seconds and the create call fails for any reason — auth, rate limit, validation — you've left the zone with a missing record, and the next reconciliation will create the old type if the source-of-truth state hasn't propagated.

Don't do this naively. Use the Batch DNS records API.

The Batch DNS records API isn't a drop-in either

POST to /zones/{zone_id}/dns_records/batch with this shape:

{
  "deletes": [{ "id": "old-record-id" }],
  "patches": [],
  "puts": [],
  "posts": [
    {
      "name": "api.example.com",
      "type": "CNAME",
      "content": "lb.example.net",
      "ttl": 300,
      "proxied": true
    }
  ]
}

The operations execute in a strict order: deletes → patches → puts → posts. That ordering matters more than it looks like it does.

Concrete example: you have an A record at api.example.com and want to flip it to CNAME api.example.com → lb.example.net. RFC 1912 prohibits a CNAME and an A record coexisting on the same hostname. If you tried to put the create first, Cloudflare would reject the new CNAME because the old A still exists. The fixed deletes-first ordering means batch requests for type changes always work, as long as you remember to put the old record in deletes and the new record in posts.

The batch is atomic at the database level — if any single operation fails, the whole transaction rolls back and you get the first error. It is not atomic at the edge. Cloudflare's blog explicitly calls this out: changes propagate through Quicksilver as independent key-value pairs, so resolvers may briefly see intermediate states during propagation. The window is short (low single-digit seconds), but it exists, and it's worse for high-volume zones because Quicksilver can serialize updates differently across colos.

If you have an SLA tighter than a few seconds of resolver inconsistency on the affected hostname, batch isn't enough either. You need the new hostname behind the new record on a different name during cutover, then swap upstream config to point at the new name.

Rate limits will surprise the largest scripts

The batch limits are different from the per-record API:

Free plan: 200 operations per batch
Paid plans: 3,500 operations per batch

If you have a migration script that walks 5,000 records and converts CNAMEs to A records (or vice-versa) and you're on the Free plan, it now needs to chunk by 200 with backoff. Cloudflare has tested batches up to 100,000 operations on enterprise tiers, so the ceiling exists, but it's tier-gated.

What the silent version of this failure looks like

The loud failure — Cloudflare returns 4xx after July 1 — is the easy case. Build red, fix forward.

The quieter failure is the migration to batch that subtly breaks something else.

Out-of-order operations in the same batch. If you submit deletes: [A record] and posts: [A record with new type] to the same hostname, you're fine — order is fixed. But if you submit puts and posts on overlapping hostnames in one batch and the put fixes a different field, you can end up with surprising state because puts happen before posts.
Lost-write between source-of-truth and Cloudflare during the gap. Your reconciler runs at T=0, sees an A record, writes "I want CNAME" to source-of-truth at T=1, the batch fires at T=2, but a competing operator changed the source-of-truth to "I want TXT" at T=1.5. The batch happily deletes the A and creates a CNAME that's already stale.
Edge cache poisoning during cutover. A high-traffic resolver hits the hostname during the propagation window, gets the old type, and caches it for full TTL. Subsequent queries from that resolver see a stale answer until the cache expires. SaaS API consumers caching DNS in-process (looking at you, JVM and Go's pre-1.19 default resolver) will see this for the full TTL.

These all work on a single integration test — the record is at the new type, GET returns the new shape — and only break under concurrent load or specific resolver paths.

Minimum-viable fix

git grep every call site that PATCHes or PUTs dns_records/{id} and inventory which ones can change type.
Bump the Cloudflare Terraform provider to a version with the schema rewrite (type tagged RequiresReplace). Do this in a non-prod zone first — you'll see a destroy-and-create plan for any record where TF state and Cloudflare diverge on type.
For non-Terraform paths, switch to POST /zones/{zone_id}/dns_records/batch with the old record in deletes and the new in posts. Wrap it so callers can't accidentally submit the same hostname in posts and patches simultaneously.
Add a validation step in your reconciler: if desired.type != current.type, route the change through batch, never through direct PATCH/PUT — even before the cutover.
Lower TTLs on records you expect to flip type on, before cutover, so the propagation window shrinks. 60-300s during migrations, restore after.
Subscribe to Cloudflare's API deprecations RSS or page-monitor the deprecations page. Two more deprecations are likely to land before EOY.

The pattern this fits

Cloudflare's deprecation page is short and easy to miss. The change feels like a small migration ("just use batch"), and the batch API even existed for years before this announcement. But the failure surface is asymmetric — Terraform users barely notice, while anyone running a reconciler or admin tool against the legacy update endpoint will quietly start failing on July 1.

API contracts include the legal sequences of operations on a resource, not just the request/response shapes. "Update record type from A to CNAME" was a legal sequence today and isn't legal in two months. No SDK type system, schema diff, or contract test catches that, because the request shape is unchanged. The only signal is the deprecation note, and it gets read by the people writing IaC modules — not the people who wrote the admin tool three years ago and moved teams.

FlareCanary monitors REST APIs and MCP servers for schema drift. Free tier covers 5 endpoints with daily checks.

Stripe Basil Quietly Moved current_period_end Off Subscription — And a Lot of Code Broke

FlareCanary — Tue, 05 May 2026 13:02:23 +0000

On March 31, 2025, Stripe shipped the Basil API version. Among other changes, it removed three fields from the Subscription object that a lot of production code was reading:

current_period_start — moved to subscription items
current_period_end — moved to subscription items
billing_thresholds — removed entirely (later reintroduced — more on this)

If you upgraded your account's default API version without pinning the SDK, the endpoint still returned 200 OK. The subscription objects still serialized cleanly. The fields your code accessed just came back undefined.

One of those fields is current_period_end. If your app uses Stripe subscriptions at all, there's a very good chance you read current_period_end somewhere. Maybe it populates the "next bill date" in your UI. Maybe it drives a cron job that reminds customers before renewal. Maybe it gates feature access for annual plans. Whatever it is, it quietly stopped working.

What Actually Changed

From the Basil changelog:

current_period_start and current_period_end are removed from subscriptions. Instead, access the subscription item's billing periods directly via items.data[].current_period_start and items.data[].current_period_end.

The rationale is sound. In the old model, every subscription had a single billing cycle. In the new flexible-billing model, each item in a subscription can have its own cycle — useful for mixing monthly and annual items, or metered and flat items, on one subscription. Moving the fields down to the item level reflects reality.

From the consumer side, though, the surface looked like this:

Before (2025-02-24.acacia):

{
  "object": "subscription",
  "id": "sub_...",
  "current_period_start": 1710000000,
  "current_period_end": 1712678400,
  "items": { "data": [ /* ... */ ] }
}

After (2025-03-31.basil):

{
  "object": "subscription",
  "id": "sub_...",
  "items": {
    "data": [
      {
        "current_period_start": 1710000000,
        "current_period_end": 1712678400
      }
    ]
  }
}

Still a valid Subscription object. Still everything serializes. Just no more top-level period fields.

The breakage affected every SDK language — Node, Python, PHP, Java, Go, .NET. (Ruby, per the changelog, was the only one unaffected by this particular change, because of how that SDK maps the object.)

billing_thresholds — Removed, Then Quietly Un-Removed

The other Basil change that bit a lot of teams was billing_thresholds disappearing. This one has an even weirder story.

billing_thresholds was how you told Stripe "automatically invoice this subscription when the customer's usage passes this amount." For metered billing at scale, it was the thing that prevented a single runaway customer from accumulating a six-figure bill you might never collect.

On 2025-03-31, Stripe removed it from the Subscription API. The initial migration advice pointed teams at "metered billing alerts" as a replacement.

Developers quickly noticed the replacement wasn't on par. From stripe-node issue #2328:

"It's not clear why it was removed before the metered version was fully ready."

"The dashboard still allows setting billing thresholds and even generates the curl command. But when you actually use the same curl request, it produces an error stating the parameter is no longer supported."

Metered billing alerts only fired once per customer lifetime and didn't carry subscription IDs. You could not use them to auto-invoice at a threshold. There was no direct replacement for the thing that had been removed.

On 2025-05-28, Stripe reintroduced billing_thresholds. The field came back. Anyone who had already rewritten their billing logic to work around its absence had just shipped two migrations to land in the same place.

This is the other failure mode of silent schema changes: the churn of reacting to a change, then reacting to its reversal.

Why Most Teams Didn't Catch the Migration

The cleanest version of this upgrade is: pin your SDK to the old Basil-adjacent version, upgrade the SDK deliberately, test against a staging Stripe account, ship. A lot of teams do this. A lot don't.

Here are the common ways it slipped through:

Library auto-upgrades. A Dependabot PR bumps your stripe-node minor version, tests pass (because fixtures are from before the migration), and you merge it. Your SDK is now Basil-aware, your code still reads subscription.current_period_end, and the field is undefined in production.

Account-level API version default. Stripe lets you upgrade your account's default version in the dashboard. If someone on your team clicks through the upgrade flow without coordinating with engineering, every non-pinned API call starts returning the new shape.

Webhook events. Webhook payloads use the API version in effect at the time the event is created. If your account default shifts, your webhook handlers start receiving new-shape invoice.* and customer.subscription.* events — often before your SDK has been upgraded.

TypeScript didn't save anyone. The Stripe Node SDK updates its types with each version. If you pin to an older version, your types still describe current_period_end as top-level and your code happily accesses it. The runtime object just doesn't have it. TypeScript can't catch that — types are a compile-time shape, not a runtime guarantee.

Integration tests that mock Stripe. If your tests use recorded fixtures or a mock Stripe library, they validate against yesterday's shape, not today's live API. Your CI stays green while prod returns undefined.

Same pattern as every other silent schema change: the thing that's supposed to catch it was designed against the shape that no longer exists.

What Broke In Practice

A few of the concrete failure modes I've seen or heard about:

Dunning emails stopped. The cron that emails customers "your subscription renews on X" read subscription.current_period_end, got undefined, and sent "your subscription renews on Invalid Date" — or skipped the send entirely.
Customer-facing dashboards went blank. "Next invoice" widgets showed empty cells for every new subscription created after the cutover.
Proration math got weird. Code that calculated time remaining in the current period used current_period_end - Date.now(); when the field was undefined, proration defaulted to zero or to a NaN that cascaded into charge amounts.
Reporting numbers drifted. Analytics jobs that grouped subscriptions by current-period end-date started dropping rows because the field was missing from the row entirely.

None of these are showy failures. No 500s, no exceptions in Sentry. Just wrong or missing data on the customer experience.

The Migration That Actually Works

For the current_period_end move specifically, the honest migration is:

Pin your SDK to the version you've tested. Don't let Dependabot drive your billing API upgrades.
Read periods from items, not the subscription. For single-item subscriptions, subscription.items.data[0].current_period_end is the direct replacement. For multi-item, decide what "period end" means for your use case — the earliest item? The one that matches a specific price? Your code now has to answer that question explicitly.
Update webhook handlers. customer.subscription.updated, invoice.created, and related events use the account API version. Test them against the new shape before flipping your account default.
Stage the upgrade behind a feature flag if you can. Flip it on in staging, compare prod vs staging subscription reads for a week, then promote.

For billing_thresholds, the migration depends on when you started. If you started before 2025-05-28, you might have rewritten on top of metered alerts. Consider whether to revert to billing_thresholds now that it's back — or stay on whatever you built, since the reintroduction might not be permanent either.

How To Catch The Next One

This is the part that generalizes beyond Stripe. Pinning versions works for APIs that version properly. It does not help for the cases where:

The API version didn't change but the response shape did (GitHub Events API, some OpenAI endpoints)
You're on an account-default version and someone upstream flipped it
The SDK was upgraded but the API default was left alone, so your types and runtime disagree
The change is "allowed" within the version contract (e.g., a new optional field that turns out to be required when you want to match the old UX)

The general defense is to watch the shape of the responses you depend on. That can be:

A cron-scheduled script that hits your top N endpoints, hashes the field set, and alerts when the hash changes.
A test that runs nightly against live endpoints (not fixtures) and asserts on the structure of the response.
A purpose-built schema drift monitor.

I've been building FlareCanary for the third option. Point it at the API endpoints you depend on — Stripe, Plaid, OpenAI, Salesforce, whatever — and it polls them on a schedule, learns the response structure, and alerts when a field disappears, a type shifts, or a new field appears. Severity-classified: removed fields are alerts, new optional fields are informational, nullability flips fall somewhere in between.

You do not strictly need a tool for this. You need a habit. The Basil migration is a specific case of a very general problem: the APIs you depend on are changing in ways your type system and your CI can't see. The only reliable signal is watching the live response shape and comparing it to yesterday's.

The Thing That Actually Matters

Stripe did a fine job of the Basil release from the provider side. The changelog was clear. The migration docs existed. The new model is better designed than the old one. The billing_thresholds reintroduction, awkward as it was, was the right call once the replacement turned out to be insufficient.

None of that changed the fact that some number of teams woke up one Monday with blank "next invoice" dates in their dashboards because nobody owned "does the shape of the Subscription object match what our code expects?" as a question worth asking before every API version bump.

That's the monitoring gap. HTTP status codes tell you an endpoint is up. Response latencies tell you it's fast. Neither of those tells you that current_period_end moved three levels deeper into the response tree.

If you're on Stripe and haven't explicitly upgraded to Basil yet, this is your warning. If you already upgraded and everything's green, run a grep for current_period_end across your codebase and see what comes back. It is a very common field, and the number of surprising places it shows up tends to be higher than people expect.

If you've been bit by the Basil migration — or by any silent schema change on another API — I'd like to hear about it. The "undefined field, no error" failures are the ones I'm most interested in. Drop a comment or reach out.

shopify app deploy --force is going away in May 2026 — your CI/CD is the problem

FlareCanary — Tue, 05 May 2026 04:03:47 +0000

Shopify is removing the --force flag from shopify app deploy and shopify app release in a May 2026 CLI release. If your CI/CD pipeline calls either command with --force today, the first pipeline run after your next @shopify/cli bump will fail with "unknown flag."

The replacement flags exist now. They don't mean the same thing --force did. Swapping blindly either breaks deploys or — worse — silently starts deleting shop data that previously required confirmation.

What the failure looks like

CLI drops an unknown flag with a non-zero exit and a message like:

 ›   Error: Nonexistent flag: --force
 ›   See more help with --help

Which then cascades through whatever GitHub Actions / GitLab CI / CircleCI job wraps the command. Most pipelines surface this as a red build; some surface it as a silent skip if the step uses continue-on-error. Either way, no new app version ships until someone touches the pipeline.

The trigger isn't a calendar event — it's the CLI bump. If your pipeline pins @shopify/cli@3.x you're fine until your Renovate bot opens a PR. If your pipeline installs @shopify/cli latest on every run, it breaks the moment Shopify publishes the May 2026 release.

Why `--force` is going away

--force today bypasses every confirmation prompt — including "you are about to permanently delete an extension from every shop that has this app installed." That's a footgun in an interactive terminal. In CI, where nobody reads the prompt anyway, it's a data-loss hazard on someone else's store.

Shopify's fix is to split the single big-hammer flag into two narrower ones:

--allow-updates — permits adding and modifying extensions, still blocks deletions.
--allow-deletes — permits deletions, intended only for manual runs.

--force was equivalent to --allow-updates --allow-deletes. The removal forces you to choose: does this pipeline actually need to delete extensions, or does it only update them?

For the vast majority of deploy pipelines, the answer is "only update." Use --allow-updates. Reserve --allow-deletes for manual operator runs where the person running the command has confirmed what's about to disappear.

Where `--force` is probably hiding

Search the pipeline files, not the source code:

git grep -n "force" -- '.github/**' '.gitlab-ci*' '*.yml' '*.yaml' 'package.json' 'scripts/**'
git grep -n "shopify app deploy\|shopify app release"

Common hiding spots:

GitHub Actions workflow steps — run: shopify app deploy --force in a step.
Package.json scripts — "deploy": "shopify app deploy --force --reset".
Makefiles — deploy: shopify app deploy --force.
Shell wrappers — scripts/deploy.sh, bin/release, any custom ./deploy that shells out.
Docs and runbooks — onboarding READMEs that tell new engineers to "just run shopify app deploy --force."

The migration that actually works

For a standard CI/CD deploy pipeline:

Before:

- run: npx @shopify/cli@latest app deploy --force

After:

- run: npx @shopify/cli@latest app deploy --allow-updates

That covers the 90% case — merges to main ship extension updates, no deletions. If a PR deletes an extension and the pipeline needs to propagate the deletion, you have two options:

Gate deletion behind a manual step. A second workflow, triggered by a workflow_dispatch or a specific tag, running shopify app deploy --allow-updates --allow-deletes.
Detect the deletion and promote the run. Parse the shopify app deploy --dry-run output and branch: if the diff includes deletions, require an approver before re-running with --allow-deletes.

Avoid the temptation to just add --allow-deletes to every pipeline. That's the footgun --force was — the one Shopify is removing.

What the silent version of this failure looks like

The loud failure — "unknown flag" — is the easy case. You see it, you fix it.

The quiet failure is worse. Some pipelines add --allow-updates --allow-deletes as a global replacement for --force, thinking they've preserved the old behavior. They have — including the footgun. Six weeks later, a refactor PR removes an extension that was intentional. The deploy fires, the extension vanishes from every install, merchant support tickets spike. No alert triggered. The CLI did exactly what the pipeline asked.

"Match the old behavior" is the wrong migration strategy here. "Match the old behavior that we actually needed" is the right one, and for most teams that's just --allow-updates.

The pattern this fits

A CLI is an API. The flags, exit codes, and output format are the contract. Every pinned @shopify/cli version in your pipeline is an implicit dependency on a specific version of that contract. Bump the dependency, contract shifts, code written against the old contract breaks.

Shopify in the last six months:

CLI --force flag removal (May 2026)
Checkout metafield deprecation (April 2026)
Mandatory expiring tokens (April 2026)
RBAC enforcement on admin APIs (April 2026)
API version 2025-01 breaking changes (January 2026)

Every one of these breaks a different surface. Their common property is that none of them trigger anything on your side unless you're instrumenting the Shopify developer changelog or running integration tests against the live CLI and APIs on a schedule.

Minimum-viable fix

git grep every pipeline file for shopify app deploy --force and shopify app release --force.
Replace --force with --allow-updates for automated pipelines.
Create a separate, manually-triggered workflow for deletions that uses --allow-updates --allow-deletes.
Add --dry-run to a pre-deploy step and fail the pipeline if the diff includes extension deletions the PR didn't explicitly flag.
Unpin @shopify/cli in a canary job so the next breaking CLI change shows up in a non-production pipeline first.
Update internal runbooks and onboarding docs — --force is about to become a red herring.

The bigger point: your CI/CD depends on third-party CLIs and APIs whose contracts change faster than your review cadence. Either you monitor those contracts continuously, or you find out when the pipeline turns red on a Tuesday morning.

FlareCanary monitors REST APIs and MCP servers for schema drift. Free tier covers 5 endpoints with daily checks.

Supabase's May 30 Default Flip: Your Newly Created Tables Will Stop Reaching the Client With permission denied for table

FlareCanary — Mon, 04 May 2026 04:02:22 +0000

Two Supabase changes in May change the rules for how new tables and the GraphQL endpoint reach your client. Both are documented in the Supabase changelog, both have explicit cutoff dates, and both will silently land in production for teams that aren't watching.

May 18, 2026 — pg_graphql is no longer enabled by default. New projects ship without the extension; existing projects with zero GraphQL traffic over the prior 30 days get it disabled. (Jan 26 changelog.)
May 30, 2026 — "Automatically expose new tables and functions" is OFF by default for all newly created projects. New tables in public are no longer auto-granted to the API roles anon, authenticated, service_role. (Apr 28 changelog.) The same rule reaches existing projects on October 30, 2026.

The interesting one is May 30. The pg_graphql change is loud — your /graphql/v1 endpoint returns extension-not-found and you fix it in five minutes. The exposure-default flip has a different shape, the one I keep writing this column about: the migration applies, the table exists, the dashboard shows it, and your client gets a 403 the first time it ships.

This is the thirteenth provider in the running silent-breakage tally, and the failure mode pairs with the Kubernetes 1.36 gitRepo article — every pre-deploy gate passes, the failure surface is post-apply.

What Actually Flips on May 30

The mechanism isn't a feature flag in your supabase/config.toml. It's PostgREST default privileges. The "expose new tables" toggle, when on, runs an equivalent of:

alter default privileges for role postgres in schema public
  grant select, insert, update, delete on tables to anon, authenticated, service_role;

When off — the new default for any project created on or after May 30 — that grant doesn't fire. Your migration creates a table, the table is fully real in Postgres, but PostgREST's API roles can't read it.

-- supabase/migrations/20260601_add_invoices.sql, ships fine, applies fine:
create table invoices (
  id uuid primary key default gen_random_uuid(),
  customer_id uuid not null,
  amount_cents integer not null,
  created_at timestamptz default now()
);

The first call from the SDK against the new project:

const { data, error } = await supabase.from('invoices').select();
// data: null
// error: {
//   code: '42501',
//   message: 'permission denied for table invoices',
//   hint: 'Grant the required privileges...',
//   details: null
// }

Postgres error code 42501 is insufficient_privilege. That's the string developers Google. The error is loud — a 403 from /rest/v1/invoices — but it's loud in production after deploy, not in CI, not in npx supabase start.

Why Local and Tests Don't Catch It

The whole point of the local Supabase CLI stack is that it mirrors prod. It mostly does — but the new default flip is a project-creation-time setting, and the local stack inherits the old behavior. So:

npx supabase start runs PostgREST against a local Postgres seeded with the legacy default privileges. Migrations from your supabase/migrations/ directory get the auto-grant. Local dev queries succeed.
Your CI runs the same local stack. Tests pass.
Production projects created before May 30 also keep the old behavior — until October 30, when existing projects get migrated. So the same migration that worked on a March-created project will silently 403 on a June-created sibling project, even with identical code.
The dashboard shows the table in the Table Editor. Schema is correct. RLS policies you wrote apply correctly to the API roles — once the underlying grants exist. Without grants, the RLS check never happens; PostgREST short-circuits at the privilege layer.

The rough mental model: your test environment is a March 2026 project. Your new prod environment, the staging clone you spun up after May 30, the per-customer DB-per-tenant project you provision in your onboarding flow — those are June 2026 projects, and they need explicit grants.

The supabase-js Failure Shape

supabase-js surfaces this as a structured error, not an exception. Code that reads data without checking error quietly receives null:

// Looks reasonable, ships often:
const { data } = await supabase.from('invoices').select();
return data ?? [];   // returns [] forever in the new default

That pattern is everywhere. If your render layer treats empty [] as "no data," the page loads, the empty-state UI shows, and nobody on the team realizes the API returned a 403 — they think the table is empty. The error is in the response, but unreached.

Same shape in supabase-py:

res = supabase.table("invoices").select("*").execute()
# res.data is None
# res.error is {'code': '42501', 'message': 'permission denied for table invoices', ...}

Same in supabase-flutter, supabase-swift, and any direct fetch against /rest/v1/*. There's no SDK version that escapes it — the change is server-side, in PostgREST role grants.

What pg_graphql Disablement Looks Like (May 18)

Different shape, easier to fix. Code that hits /graphql/v1 or uses .schema('graphql_public').rpc('graphql', ...) gets either an extension-not-found error or a 404, depending on which endpoint your SDK chose. The fix:

create extension if not exists pg_graphql;

Or in the dashboard: Database → Extensions → enable pg_graphql. Once enabled, the extension behaves identically to before. The only nuance is that existing projects with zero GraphQL traffic over the prior 30 days get it auto-disabled on May 18, so a project that was working could go dark if its GraphQL traffic was thin.

Pair this with the May 30 grant change and a brand-new project gets to do both: enable the extension manually and grant the API roles read access on every table you want exposed.

Why Schema Drift Tools and Schema Validators Don't Help

This is the awkward part for anyone who already has guardrails on their database changes:

Migration linters (sqlfluff, the Supabase advisor's lint pass) read the migration SQL and tell you the schema is sound. The schema is sound. The grants aren't the schema.
Type generators (supabase gen types typescript) introspect the schema, write Database['public']['Tables']['invoices'], and your TypeScript compiles. The generator reads structure, not privileges.
Zod / Pydantic / runtime validators that wrap the SDK response don't fire — data is null, error is populated, no schema gets validated.
Smoke tests against staging catch it only if staging is a post-May-30 project. If staging was created in March, it has the old default and won't reproduce the failure.

The Fix You Want in Every New Migration

The pragmatic shape for migrations going forward — write the grants explicitly, even on projects that still have the auto-grant default. Your migration becomes portable across project creation dates:

create table invoices (
  id uuid primary key default gen_random_uuid(),
  customer_id uuid not null,
  amount_cents integer not null,
  created_at timestamptz default now()
);

-- Make this migration work on projects created after May 30, 2026:
grant select on public.invoices to anon;
grant select, insert, update, delete on public.invoices to authenticated;
grant select, insert, update, delete on public.invoices to service_role;

-- Plus your RLS policies as before:
alter table invoices enable row level security;
create policy "users see own invoices" on invoices
  for select to authenticated using (customer_id = auth.uid());

If you want the auto-grant default back on a specific project — Database Settings → "Automatically expose new tables and functions" — toggle on. That just re-installs the default privileges rule shown above; it does not retroactively grant existing tables.

Your Security Advisor in the dashboard will flag missing grants on tables you have RLS policies for. Read that panel. It is the only place the system tells you "this table has policies but isn't reachable."

How to Detect This Class of Change

The general defense, restated for the database flavor of silent breakage:

Spin up a fresh Supabase project as part of your CI for migration tests. Not a local stack — an actual Supabase-hosted project, created today. The local stack inherits the old defaults and won't reproduce. A new project will. (For most teams this is a per-PR ephemeral project against a free-tier org.)
Watch for 42501 errors at the edge. Add a request-level log on PostgREST 403s with body content. The shape is consistent enough to alert on.
Check your tables list against your grants. A nightly query that diffs pg_class entries against information_schema.role_table_grants for the API roles will surface tables that exist but aren't reachable. Run it on every project, not just dev.
Pin the project creation date in your runbook. "All our prod projects were created in 2025" matters now in a way it didn't last quarter. October 30, 2026 is the date that flips this for all existing projects too.

The Pattern, Now Thirteen Months In

Provider	Surface	What Goes Wrong
Stripe Basil	`Subscription.current_period_end`	Moved to `items[]`; old reads return `undefined`
GitHub	`pull_request.merge_commit_sha`	Returns `null` on closed PRs in 2026-03-10 ver
GitHub	Org security fields	PATCH returns 200, applies nothing
OpenAI	Responses `input_text`	Rejected with `Invalid value` error
HubSpot	Contacts v1 endpoints	Return 200 with `list-memberships` silently dropped
Auth0	TLS handshake	Weak ciphers start returning `handshake_failure` Jun 10
Twilio	`api.de1.twilio.com`	Removed; regional domains never actually routed regionally
Shopify	Checkout `metafields`	Returns `undefined` after 2026-04; orders ship without app data
Kubernetes 1.36	`gitRepo` volumes	Pass validation, fail at deploy with FailedMount
Anthropic	`claude-3-haiku-20240307`	Returns model-retired error after Apr 20
OpenAI	DALL·E 2/3	Retired May 12; per-image billing flips to per-token
Exa	`/research` + crawl-date filters	404, parameters silently ignored, fields `null`
OpenAI Realtime	Audio/text/transcript event names	Renamed; old listeners silently never fire
Supabase	PostgREST default privileges	New tables 403 with `42501`; CI doesn't repro

Fourteen providers. Same shared shape: the system answers, the SDK is fine, the thing your code expects to reach is gated by something your local environment doesn't model.

If you operate Supabase projects, the action items are short. New migrations: ship the grants explicitly. New projects after May 30: enable pg_graphql if you use it, and turn on auto-exposure if you'd rather not write grants per table. Every project: read the Security Advisor panel. October 30 is the date this hits projects you've already deployed against.

What I'm Building

I'm working on FlareCanary for exactly this class of post-deploy surprise. Point it at the API surfaces your application depends on — REST, GraphQL, and the response shape — and it polls them on a schedule, learns the response vocabulary, and alerts when a field disappears, a status code flips, or the privilege layer starts answering 403 where it used to answer rows. Free tier covers up to five endpoints, useful for keeping a watch on a Supabase project alongside the upstream APIs you call.

You don't need a tool for this. You do need a habit. The Supabase changelog covers both May cutoffs in plain language. Anyone reading it will catch them. The half-broken state still ships somewhere — to a team that provisions a per-tenant project from a script, to a team whose staging is a different vintage from prod, to a team that trusted local dev to model production privileges.

That's the gap. The schema being correct isn't enough. The grants matter.

If your Supabase project trips on the May 30 default flip — or if a 42501 permission denied for table error catches you off guard — I'd like to hear about it. The "code is right, schema is right, privileges aren't" failures are exactly the ones I'm tracking. Drop a comment or reach out.

GitHub Just Removed merge_commit_sha From Pull Request Responses — Your Release Bot Is Probably Tagging null

FlareCanary — Sun, 03 May 2026 13:01:05 +0000

GitHub's 2026-03-10 REST API version ships a quiet but consequential breaking change: the merge_commit_sha property is gone from pull request responses. 21 endpoints affected. There is no error, no 410, no migration warning header — the field just stops appearing in the JSON.

For CI/CD code that does this:

const { data: pr } = await octokit.pulls.get({ owner, repo, pull_number });
await tagDeploymentArtifact(pr.merge_commit_sha);

You now tag with undefined. The deployment still ships. The artifact registry still accepts the upload. Six weeks later, somebody asks "which commit produced this build?" and the answer is undefined-1747291204.

This is incident #7 in our silent-breakage series (GitHub PushEvent, Stripe Basil, Shopify 2025-01, OpenAI Responses, Twilio regional, HubSpot Contacts v1). The pattern keeps repeating because removing a field from a JSON response is the cheapest possible breaking change for the API provider and the most invisible possible breaking change for the consumer.

What's removed and where

merge_commit_sha disappears from 21 PR-bearing endpoints, including:

GET /repos/{owner}/{repo}/pulls
GET /repos/{owner}/{repo}/pulls/{pull_number}
POST /repos/{owner}/{repo}/pulls
PATCH /repos/{owner}/{repo}/pulls/{pull_number}
GET /repos/{owner}/{repo}/issues/events
GET /repos/{owner}/{repo}/issues/{issue_number}/events
Search results that embed PR objects
Project card payloads that link to PRs

Same release also removes the singular assignee field from 31 Issue and PR endpoints. From the changelog:

The singular assignee field has been marked as 'closing down' for years and duplicates information available in the assignees array.

True — and the migration is mechanical. Read from assignees[0] instead of assignee. Write assignees: [login] instead of assignee: login. The footgun is that assignee returning undefined looks identical to "this PR has no assignee," and a lot of routing logic gates on exactly that.

Why you don't get an error

The REST API breaking-change rollout works by version pinning. If you send:

X-GitHub-Api-Version: 2022-11-28

…you keep the old field. If you send:

X-GitHub-Api-Version: 2026-03-10

…or no version header at all and your client SDK has updated its default — the field is gone.

Octokit, PyGithub, go-github, and the various community SDKs upgrade their defaults on their own schedules. Your package.json says ^21.0.0, the maintainer ships 21.4.7 next month with the new default version header, your npm install picks it up on the next CI run, and now your release bot is tagging undefined. Nothing in your repo changed. Nothing in the API changed for clients still pinning the old version. The change rides in on a transitive bump.

The non-obvious replacement

The official guidance points to the pull_request.merge_commit_sha field on the webhook payload (not the API response) and to the merge commit's SHA reachable via:

GET /repos/{owner}/{repo}/commits/{ref}
# where ref is the head of the default branch immediately after merge

Neither is a one-line swap.

Webhook payloads still carry merge_commit_sha because GitHub versions webhooks separately from the REST API. If your release bot is webhook-driven, you're fine — the field is in the pull_request.closed event payload. If your release bot is poll-driven (CI runs nightly, asks "what merged today, tag those artifacts"), you have to reconstruct the merge commit by:

Reading the PR's base.ref (the target branch)
Pulling commit history on that branch
Finding the merge commit by the PR number in the commit message (Merge pull request #N)
Or — for squash/rebase merges — there is no single merge SHA, and you have to use head.sha of the PR plus tracking metadata your bot wrote earlier

The squash-merge case is the one most release tooling gets wrong on the rewrite. There never was a merge_commit_sha for squash merges in the strict sense — GitHub returned the SHA of the squashed-in commit on the base branch — but consumers treated it as canonical. Without that field, the PR head SHA and the resulting commit on main are different SHAs with no GitHub-provided link between them.

Why your tests didn't catch it

Your CI pipeline tests hit a fixture PR JSON. The fixture has merge_commit_sha. The test asserts the tag string is well-formed.

Three things have to be true for that test to fail before the rollout:

Your fixtures were regenerated against the new API version (they weren't)
Your test runner sends the new X-GitHub-Api-Version header (it doesn't, unless you wrote it)
Your assertion checks typeof sha === 'string' && sha.length === 40 and not just sha != null (most don't)

So the test stays green. The same pattern as every other silent drift incident:

#	API	What changed	Where tests missed it
1	GitHub PushEvent	`commits` field silently dropped	Tests didn't assert field presence
2	Stripe Basil	`current_period_end` moved to `items`	Tests used Checkout fixtures
3	Shopify 2025-01	`fulfillmentHold` type change	Tests mocked the response
4	OpenAI Responses	`input_text` removed for assistants	Tests covered request role=user
5	Twilio regional	Regional domains stop resolving	Tests don't hit prod DNS paths
6	HubSpot Contacts v1	`list-memberships` returns empty	Tests asserted against sandbox fixtures
7	GitHub merge_commit_sha	Field removed from PR responses	Tests used pre-rollout fixtures

Pattern holds. The breaking change is always in a field a test isn't asserting against — or in a layer (transitive SDK upgrade, version-pinned header) the test isn't exercising.

What to do this week

Three actions, in priority order:

Grep your codebase for merge_commit_sha. Every read site is a candidate failure. Tag every assignment to a deployment artifact, release name, or git tag as a critical path.
Grep for pr.assignee or issue.assignee (singular). Routing/notification code keyed on this field is the next-largest blast radius after release tagging.
Pin X-GitHub-Api-Version: 2022-11-28 as a stopgap if you can't fix all the read sites this week. GitHub supports old versions for ~24 months. This buys time, not a fix — schedule the migration before the version sunsets.

If your release bot is silently tagging undefined, you usually find out 90 days later when somebody is paged at 3 AM and can't trace the deploy. Worth fixing on a Tuesday afternoon.

We built FlareCanary for this layer: poll third-party APIs on a schedule, watch the response shape, page when a field stops appearing. The GitHub merge SHA removal is the textbook incident — no error, no warning, just a field that used to be there and isn't. APIs break this way constantly. We catch it before the 3 AM page.

We track API drift incidents in real time. If your release tooling reads merge_commit_sha and you haven't audited it for the 2026-03-10 API version, that null is already in your build pipeline somewhere.

OpenAI Realtime Beta Disappears May 7 — Your Voice Agent's Audio Handlers Will Stop Firing With No Error

FlareCanary — Sun, 03 May 2026 04:02:21 +0000

On May 7, 2026 — five days from now — OpenAI removes the Realtime API beta. If you have a voice agent, transcription pipeline, or any WebSocket/WebRTC integration with gpt-4o-realtime-preview, you have a long weekend's worth of work to do, and most of it isn't the part the migration guide warns about.

The loud failures are easy. The WebSocket returns 401, the WebRTC connection won't establish, your session.update gets rejected. Those break in dev, you fix them, you ship.

The interesting failures — and the ones I keep writing this column about — are the silent ones. The connection works, the model responds, your tests pass, and audio just stops coming out of the speaker. Or text stops streaming. Or the voice changes. Or a function call output silently flips from text to audio. Code that was correct against the beta interface is now correct-shaped against an interface that's renamed half the events it emits.

This is the twelfth provider in the running tally and it's the same shape every time: the SDK still validates, the response still parses, the field your code depends on just isn't being sent under that name anymore.

The Silent Ones: Renamed Events

This is where most of the silent breakage lives. The Realtime API streams a lot of event types over the WebSocket — partial text deltas, audio chunks, transcript chunks. In the GA protocol, three of the most-listened-for events were renamed:

Beta name	GA name
`response.text.delta`	`response.output_text.delta`
`response.audio.delta`	`response.output_audio.delta`
`response.audio_transcript.delta`	`response.output_audio_transcript.delta`

If your client uses a typed event dispatcher — switch (msg.type), or a client.on("response.audio.delta", ...) style listener — the new event names just don't match. The handler isn't called. There's no error. The connection is healthy, the server is sending bytes, and your audio buffer never gets fed.

// Beta-era code, still compiles, still connects, plays no audio:
client.on("response.audio.delta", (event) => {
  audioBuffer.append(event.delta);
});

// In GA, this event is named response.output_audio.delta.
// The above handler will never fire. No exception, no warning,
// just silence on the speaker.

The class of code that breaks here is "dispatch on event name." Every voice agent I've seen written against the beta has a switch or a listener registry keyed on these strings. They don't throw on unknown event names — that would be the right design — they silently no-op.

The tells are subtle. Tests that mock the WebSocket transport keep passing because the test fixtures still use the old names. Recorded interactions replay correctly. The first sign in production is a user saying "the bot just stopped talking to me," and your logs show a successful session with content streaming and no errors.

The Almost-Silent One: Content Type Renames

Inside the conversation item shape, two type tags were renamed:

type: "text" → type: "output_text"
type: "audio" → type: "output_audio"

Same failure mode. If you have rendering code that switches on content[i].type, it now hits the default branch — usually "render nothing" or "log unknown type and continue."

# Beta:
for part in item["content"]:
    if part["type"] == "text":
        render_text(part["text"])
    elif part["type"] == "audio":
        play_audio(part["audio"])
    # default: skip silently

In GA, every assistant response part takes the default branch. The conversation appears to send empty turns. No exception. The model is responding correctly; your render layer just doesn't recognize the shape anymore.

This is identical in spirit to Stripe's Basil migration — the field you read still exists in the response, it's just spelled differently.

The Other Silent One: Restructured Session Config

session.update is the event you send to configure voice, transcription model, modalities, and so on. In the beta, much of this was flat. In GA, it nested into session.audio.input and session.audio.output. Voice selection, for example, moved from a top-level field to session.audio.output.voice.

If your client sends old-shape config to a GA endpoint, two things can happen depending on which fields you set:

The server rejects the session.update (loud — easy to fix).
The server accepts what it understands, ignores what it doesn't, and keeps defaults for the rest (silent — voice silently flips to the default alloy, transcription model silently falls back, etc.).

The mixed shape — some fields recognized, others ignored — is the worst case. The session is configured, just not the way you wrote it. Your "always use the verse voice for our brand" code now uses the default voice and nobody notices until QA listens to a recording.

The Loud Ones (For Completeness)

These break visibly. List them so you know what to expect during migration:

OpenAI-Beta: realtime=v1 header. Remove it. Sending it against the GA endpoint causes auth/route confusion — you'll see 401 or 404 depending on path.
api-version query parameter (Azure path). Strip it from the URL. The GA endpoint format is /openai/v1/realtime, no version suffix.
session.update missing the new type field. Required in GA. Must be "realtime" for speech-to-speech or "transcription" for audio-only. Server returns an explicit error if absent.
WebRTC ephemeral key endpoint. Was a session creation flow; is now POST /v1/realtime/client_secrets. Different request shape, different response shape. Old endpoint 404s.
WebRTC connection URL. Now /v1/realtime/calls. Old browser SDP exchange path is gone.
SDK version pins. OpenAI Python ≥ 1.54.0, JavaScript ≥ 4.77.0, .NET ≥ 2.9.0. Older SDKs hard-fail against GA.

The migration guide covers these well. The ones above this list are where it under-warns.

Why Tests Won't Catch It

Same pattern as the previous eleven providers in this series:

Mocked Realtime in tests. Your fixtures are recorded WebSocket transcripts from the beta. They still emit response.audio.delta. Tests pass; production silently breaks.
Schema validators don't help. zod / pydantic schemas that accept { type: string, ... } with a string-enum will pass output_audio as readily as audio. The typo isn't a schema violation, it's a meaning violation — the new value goes through a code path that expects different content.
Voice agents don't have unit tests. Most are tested by running them and listening. Silent audio is a noticeable bug, but only after a user reports it. Empty-text-stream renders look like the model "didn't say anything this turn" — easy to miss.
Logs look healthy. No exceptions, no 4xx, no warnings. The Realtime session metadata in the OpenAI dashboard shows successful turns. Every observability surface is green.

How to Detect This Class of Change

The general defense, which I keep restating because it keeps working, is to watch the shape of responses, not just the status code. For a streaming API like Realtime, that means:

Log the set of event types your client receives in a session. A short script that subscribes to all events, hashes the names, and alerts when the hash changes. Catches every future event rename.
Subscribe to a wildcard / unknown-event handler that logs (don't silently drop). If your event dispatcher has a default: branch, make it loud — console.warn("unhandled realtime event: " + msg.type). You'd have caught this rename on day one of the GA rollout in dev.
Replay a representative session in CI against a non-production realtime endpoint. Capture the event-name set. Diff against last known good. This is what schema-drift monitoring does for REST APIs; the same idea applies to streaming protocols.
Audit switch (msg.type) and event-listener registrations in your codebase. grep -r "response.text.delta\|response.audio.delta\|response.audio_transcript.delta" and update each call site. While you're there, add the wildcard branch.

For ongoing monitoring: do this for every streaming provider you depend on, not just OpenAI. The streaming-event-name failure mode is generalizable — Anthropic streams events, Google's voice APIs do, every voice/agent provider does, and they all rename events between major versions.

The Pattern, Now Twelve Months In

Twelfth provider in the running silent-breakage tally:

Provider	Surface	What Goes Wrong
Stripe Basil	`Subscription.current_period_end`	Moved to `items[]`; old reads return `undefined`
GitHub	`pull_request.merge_commit_sha`	Returns `null` on closed PRs in 2026-03-10 ver
GitHub	Org security fields	PATCH returns 200, applies nothing
OpenAI	Responses `input_text`	Rejected with `Invalid value` error
HubSpot	Contacts v1 endpoints	Return 200 with `list-memberships` silently dropped
Auth0	TLS handshake	Weak ciphers start returning `handshake_failure` Jun 10
Twilio	`api.de1.twilio.com`	Removed; regional domains never actually routed regionally
Shopify	Checkout `metafields`	Returns `undefined` after 2026-04; orders ship without app data
Kubernetes 1.36	`gitRepo` volumes	Pass validation, fail at deploy with FailedMount
Anthropic	`claude-3-haiku-20240307`	Returns model-retired error after Apr 20
OpenAI	DALL·E 2/3	Retired May 12; per-image billing flips to per-token
Exa	`/research` + crawl-date filters + `highlightScores`	404, parameters silently ignored, fields `null`
OpenAI Realtime	Audio/text/transcript event names	Renamed; old listeners silently never fire

Thirteen providers, thirteen different shapes, one shared failure mode: the surface still answers, the SDK still validates, the thing your code reads or listens for is just not there under that name.

If you ship a voice agent that depends on the Realtime API, today is the day. May 7 is on a Thursday. The week after that, "the bot stopped talking" tickets are a much harder bug than "I renamed my event handlers on May 2 because the migration guide said to."

What I'm Building

I'm working on FlareCanary for exactly this class of bug. Point it at the API endpoints you depend on — REST, GraphQL, and now streaming — and it polls them on a schedule, learns the response shape and event vocabulary, and alerts when a field drops, a type flips, or an event renames. Free tier covers up to five endpoints, useful for keeping watch on your top external dependencies without building the monitor yourself.

You don't need a tool for this. You do need a habit. The Realtime GA rollout has been documented well enough that any team paying attention will catch it. The silent half-broken state still ships somewhere — to a team that pinned an old SDK, or to one that wrote a strict event listener six months ago and forgot.

That's the gap. HTTP 200 isn't enough. The connection succeeding isn't enough. The event names matter.

If your voice agent or transcription pipeline trips on the Realtime migration this week — or any other silent schema change — I'd like to hear about it. The "no error, just silence on the speaker" failures are exactly the ones I'm tracking. Drop a comment or reach out.

Exa Just Removed /research and Started Silently Ignoring Two Date Filters — Your Agent Is Probably Pulling Stale Pages Right Now

FlareCanary — Sat, 02 May 2026 04:01:43 +0000

On May 1, 2026 — today — Exa finished a three-step API deprecation that's been quietly half-breaking AI agents for the past two weeks. The hard cutoff today is the visible piece. The interesting part is what's been broken since April 15 with no error.

Three things changed:

/research/v1 — sunset today. Calls now fail. Migration: /search with type: "deep-reasoning".
startCrawlDate / endCrawlDate — silently ignored since April 15. Requests still return 200. The filters just don't apply.
resolvedSearchType and highlightScores — started returning null on April 15. Hard-removed today.

If you're calling Exa from an agent, RAG pipeline, or research tool, at least one of these is probably touching your code path. The endpoint cutoff is the loud one — your app errors and you find out. The other two are the silent-breakage class. Your code keeps running, your tests stay green, your output gets quietly worse.

The Silent One: Crawl-Date Filters

This is the part most teams will miss.

Exa's /search endpoint takes startCrawlDate and endCrawlDate parameters. They constrain results to pages crawled within a date window — useful when you want recent web content and don't want a 2019 blog post showing up because Google still ranks it.

A typical usage pattern, paraphrased:

results = exa.search(
    query="latest LLM benchmarks",
    startCrawlDate="2026-04-01T00:00:00Z",
    endCrawlDate="2026-05-01T00:00:00Z",
    numResults=20,
)

Before April 15, you got pages crawled in April 2026.

After April 15, Exa accepts the request, returns 200, and gives you whatever the search would return without the date filter. The parameters are accepted, validated for type (still ISO 8601), and discarded.

There's no error. There's no warning header. The Exa changelog is the only place this is documented:

"requests will succeed, but the filter will have no effect"

The Exa reference docs still list both parameters in the current request schema, with no deprecation marker. So the integration looks correct in every place a developer would look — except for the runtime behavior.

The class of code that breaks here is "pull recent content for an LLM context window." If you're doing date-bounded research for an agent — "summarize what's been written about X this month" — the agent is now happily summarizing pages from any year. The output reads fine. Spot-checking ten queries you might not notice. The tells are subtle: results gradually skewing older, "this month's news" articles citing 2024 sources, RAG answers that are confidently wrong about when something happened.

The Almost-Silent One: Null Response Fields

Two fields started returning null on April 15:

resolvedSearchType — when you call /search with type: "auto", this told you which algorithm Exa actually used (neural, keyword, etc.). Useful for logging, for A/B comparisons, for tuning prompts based on which retrieval path won.
highlightScores — relevance scores for the highlights extracted from each result. Useful for filtering ("only show highlights above 0.7"), for re-ranking, for explaining results to users.

Code that reads these fields now gets null instead of a number or a string. Whether that breaks loudly depends entirely on how the consumer is written.

// This silently degrades:
const ranked = results.flatMap(r =>
  r.highlights.map((h, i) => ({ text: h, score: r.highlightScores[i] }))
).filter(h => h.score > 0.7);
// .filter on undefined > 0.7 → false for every item → empty array
// No error, just no results.

// This crashes loudly:
const avg = results.reduce((a, r) => a + r.highlightScores.reduce(...), 0);
// Cannot read property 'reduce' of null → 500 in your handler.

The crash version gets caught in CI on the next run. The silent-empty version ships and your "top relevant snippets" feature returns nothing forever. Then on May 1, the field is removed entirely, the null becomes undefined, and downstream code that was previously handling null may behave differently again.

This is the same pattern as Stripe's current_period_end move and Auth0's TLS cipher removal: the SDK still validates, the response still parses, the field your code depends on just isn't there.

The Loud One: /research Endpoint

/research/v1 — the agentic research endpoint — is gone today. Calls return an error.

Migration is to /search with type: "deep-reasoning" and an outputSchema for structured output. From the Exa docs, the new shape is:

{
  "query": "your research question",
  "type": "deep-reasoning",
  "outputSchema": { "type": "object", "properties": { ... } }
}

Three things to watch in the migration:

Cost shape changes. The new endpoint exposes costDollars per request; budget code that hard-coded research-endpoint pricing will be wrong. Pull cost from the response, not from a constant.
Streaming differs. /search supports stream: true; if your /research integration was synchronous, you can leave streaming off, but it's worth a look — agents tend to feel slow on deep-reasoning queries.
Output schema enforcement is opt-in. With /research, you wrote schemas and got typed output. With /search deep-reasoning, you only get typed output if you pass outputSchema. Forget it and you get free-form text. Consumers that did JSON.parse(result.output) will throw.

This is the visible failure. Loud is preferable to silent.

Why Most Tests Won't Catch the Silent Pieces

Standard ways teams miss this class of change:

Mocked Exa in tests. If your test fixture is a recorded /search response from March, it has highlightScores and resolvedSearchType populated. Your code reads them, your assertions pass. The live API now returns null. Tests are green; production is degraded.

startCrawlDate parameter still in docs. A developer auditing the integration today, looking at Exa's reference page, sees both parameters listed normally. The deprecation notice is on the changelog page only. So a code review of the integration doesn't catch it — the code matches the docs.

Agents don't have unit tests. Most agent loops are tested by running them and eyeballing the output. The output of a date-unfiltered query looks similar enough to a date-filtered one that the difference doesn't jump out, especially if the underlying corpus is dominated by recent pages anyway.

Schema validators pass. If you're using zod or pydantic to validate the response, null likely passes a nullable field schema, and a missing field passes an optional field schema. Validation errors are not the right signal here.

How to Detect This Class of Change

The general defense is the same as for Stripe's Basil migration, GitHub's merge_commit_sha removal, and OpenAI's input_text rejection: you have to watch the shape of the response, not just the status code.

Specific things you can do today:

Audit startCrawlDate / endCrawlDate callers. grep -r "startCrawlDate\|endCrawlDate" across your codebase. Anywhere you pass these, the filter is now no-op. Decide if you can remove them, replace with startPublishedDate / endPublishedDate (which still work but mean something different — published date as Exa understands it, not crawl date), or whether your use case requires a different data source.
Search for highlightScores reads. Anywhere your code accesses this, it's now null (and gone after today). Most consumers want to either drop the score-based logic or replace it with a re-ranker.
Search for resolvedSearchType reads. Mostly used for logging or analytics — usually fine to drop, occasionally drives branching logic that needs replacing.
Migrate /research callers to /search with type: "deep-reasoning". Check outputSchema is set if you parse the output as JSON.

For ongoing monitoring: schedule a daily script that hits your top Exa endpoints with a representative query, hashes the field set in the response, and alerts when the hash changes. That signal catches every future silent removal — not just Exa's.

The Pattern, Now Five Months In

This is the twelfth provider I've written up in this series. The shape varies; the failure mode is consistent:

Provider	Surface	What Goes Wrong
Stripe Basil	`Subscription.current_period_end`	Moved to `items[].current_period_end`; old reads return `undefined`
GitHub	`pull_request.merge_commit_sha`	Returns `null` on closed PRs in API ver 2026-03-10
GitHub	Org security fields	PATCH returns 200, applies nothing
OpenAI	Responses `input_text`	Rejected with `Invalid value` error
HubSpot	Contacts v1 endpoints	Return 200 with `list-memberships` silently dropped
Auth0	TLS handshake	Weak ciphers start returning `handshake_failure` Jun 10
Twilio	`api.de1.twilio.com`	Removed; regional domains never actually routed regionally
Shopify	Checkout `metafields`	Returns `undefined` after 2026-04; orders ship without app data
Kubernetes 1.36	`gitRepo` volumes	Pass validation, fail at deploy time with FailedMount
Anthropic	`claude-3-haiku-20240307`	Returns model-retired error after Apr 20
OpenAI	DALL·E 2/3	Retired May 12; per-image billing flips to per-token
Exa	`/research` + crawl-date filters + `highlightScores`	Endpoint 404, parameters silently ignored, fields `null`

Twelve providers, twelve different shapes, one shared failure mode: the API still returns 200 (until it doesn't), the SDK still validates, the field your code depends on just isn't there — or the parameter you sent is being thrown away.

If your AI stack pulls from Exa and you haven't audited the calls yet, today is the day. Tomorrow, "I noticed our research agent is summarizing weirdly old pages" is a much harder bug to track down than "I removed the startCrawlDate parameter on May 1 because it stopped working."

I'm building FlareCanary for exactly this problem — point it at the API endpoints you depend on (Exa, OpenAI, Stripe, GitHub, anything), and it polls them on a schedule, learns the response shape, and alerts when a field drops, a type flips, or a parameter starts being ignored. Free tier covers up to five endpoints — useful for keeping watch on your top external dependencies without writing the monitor yourself.

You don't need a tool for this. You do need a habit. The Exa rollout is unusually gentle — the deprecation notice was clear, the changelog was explicit, the migration path was documented. The silent half-broken state still ran for two weeks because nobody had a system for catching "the parameter we send is being thrown away."

That's the gap. HTTP 200 isn't enough. The shape matters.

If your agent or RAG pipeline tripped on Exa's deprecation today — or any other silent schema change — I'd like to hear about it. The "no error, just degraded output" failures are the ones I'm most interested in. Drop a comment or reach out.

Stripe Basil Quietly Moved current_period_end Off Subscription — And a Lot of Code Broke

FlareCanary — Fri, 01 May 2026 04:02:06 +0000

On March 31, 2025, Stripe shipped the Basil API version. Among other changes, it removed three fields from the Subscription object that a lot of production code was reading:

current_period_start — moved to subscription items
current_period_end — moved to subscription items
billing_thresholds — removed entirely (later reintroduced — more on this)

What Actually Changed

From the Basil changelog:

current_period_start and current_period_end are removed from subscriptions. Instead, access the subscription item's billing periods directly via items.data[].current_period_start and items.data[].current_period_end.

From the consumer side, though, the surface looked like this:

Before (2025-02-24.acacia):

{
  "object": "subscription",
  "id": "sub_...",
  "current_period_start": 1710000000,
  "current_period_end": 1712678400,
  "items": { "data": [ /* ... */ ] }
}

After (2025-03-31.basil):

{
  "object": "subscription",
  "id": "sub_...",
  "items": {
    "data": [
      {
        "current_period_start": 1710000000,
        "current_period_end": 1712678400
      }
    ]
  }
}

Still a valid Subscription object. Still everything serializes. Just no more top-level period fields.

billing_thresholds — Removed, Then Quietly Un-Removed

The other Basil change that bit a lot of teams was billing_thresholds disappearing. This one has an even weirder story.

On 2025-03-31, Stripe removed it from the Subscription API. The initial migration advice pointed teams at "metered billing alerts" as a replacement.

Developers quickly noticed the replacement wasn't on par. From stripe-node issue #2328:

"It's not clear why it was removed before the metered version was fully ready."

"The dashboard still allows setting billing thresholds and even generates the curl command. But when you actually use the same curl request, it produces an error stating the parameter is no longer supported."

This is the other failure mode of silent schema changes: the churn of reacting to a change, then reacting to its reversal.

Why Most Teams Didn't Catch the Migration

Here are the common ways it slipped through:

Same pattern as every other silent schema change: the thing that's supposed to catch it was designed against the shape that no longer exists.

What Broke In Practice

A few of the concrete failure modes I've seen or heard about:

Dunning emails stopped. The cron that emails customers "your subscription renews on X" read subscription.current_period_end, got undefined, and sent "your subscription renews on Invalid Date" — or skipped the send entirely.
Customer-facing dashboards went blank. "Next invoice" widgets showed empty cells for every new subscription created after the cutover.
Proration math got weird. Code that calculated time remaining in the current period used current_period_end - Date.now(); when the field was undefined, proration defaulted to zero or to a NaN that cascaded into charge amounts.
Reporting numbers drifted. Analytics jobs that grouped subscriptions by current-period end-date started dropping rows because the field was missing from the row entirely.

None of these are showy failures. No 500s, no exceptions in Sentry. Just wrong or missing data on the customer experience.

The Migration That Actually Works

For the current_period_end move specifically, the honest migration is:

Pin your SDK to the version you've tested. Don't let Dependabot drive your billing API upgrades.
Read periods from items, not the subscription. For single-item subscriptions, subscription.items.data[0].current_period_end is the direct replacement. For multi-item, decide what "period end" means for your use case — the earliest item? The one that matches a specific price? Your code now has to answer that question explicitly.
Update webhook handlers. customer.subscription.updated, invoice.created, and related events use the account API version. Test them against the new shape before flipping your account default.
Stage the upgrade behind a feature flag if you can. Flip it on in staging, compare prod vs staging subscription reads for a week, then promote.

How To Catch The Next One

This is the part that generalizes beyond Stripe. Pinning versions works for APIs that version properly. It does not help for the cases where:

The API version didn't change but the response shape did (GitHub Events API, some OpenAI endpoints)
You're on an account-default version and someone upstream flipped it
The SDK was upgraded but the API default was left alone, so your types and runtime disagree
The change is "allowed" within the version contract (e.g., a new optional field that turns out to be required when you want to match the old UX)

The general defense is to watch the shape of the responses you depend on. That can be:

A cron-scheduled script that hits your top N endpoints, hashes the field set, and alerts when the hash changes.
A test that runs nightly against live endpoints (not fixtures) and asserts on the structure of the response.
A purpose-built schema drift monitor.

The Thing That Actually Matters

OpenAI Responses API Started Rejecting input_text With No Warning — Here's the Fix and Why It Keeps Happening

FlareCanary — Thu, 30 Apr 2026 04:01:42 +0000

If you're here because you just got this error:

Invalid value: 'input_text'. Supported values are: 'output_text' and 'refusal'.

…from the OpenAI Responses API on a request that worked last week, you're not debugging your code. You're debugging OpenAI's.

What happened

On or around October 21, 2025, the OpenAI Responses API started rejecting input_text content items when they appeared inside a message with role: "assistant". The documented format — the one every SDK and every tutorial was using — suddenly returned a 400.

Before:

{
  "role": "assistant",
  "content": [
    { "type": "input_text", "text": "Hi, how can I help?" }
  ]
}

After:

HTTP 400 Bad Request
Invalid value: 'input_text'. Supported values are: 'output_text' and 'refusal'.

Same endpoint. Same SDK version. Same request body. Just stopped working.

The community thread that went up the day it broke captures the vibe:

"I'm wondering why there is no backward compatibility and no even deprecation messages for that. It's really bad user experience."

A moderator responded with:

"Thanks for flagging this! I asked staff for clarification because apparently everything is gone besides input audio."

"Apparently everything is gone" is not how you want to find out your production AI pipeline has been migrated under you.

The fix

For assistant-role messages, the type name changed. input_text is now only valid inside user-role messages. For assistant messages, you need output_text:

 {
   "role": "assistant",
   "content": [
-    { "type": "input_text", "text": "Hi, how can I help?" }
+    { "type": "output_text", "text": "Hi, how can I help?" }
   ]
 }

That's it. That's the whole fix. The semantics are identical; the type tag just has to match the role now.

If you're building the assistant message from scratch (e.g., seeding a conversation), you use output_text. If you're echoing back a prior turn, you were probably already using output_text (because that's what the API returned). The breakage is specifically on the "I manually built an assistant turn" path.

A few things worth noting before you patch and move on:

The error message is honest but only if you read it closely. It lists the valid values for the role you're sending — but it doesn't say "try output_text for assistant messages." You have to infer that from the full schema.
The Realtime API had a nearly identical incident earlier. This older thread shows developers hit with Invalid value: 'input_text'. Value must be 'text'. in the Realtime API. Same family of problem: the type tag you need depends on which API surface and which role you're using, and the docs don't cross-reference.
There's a related bug report open on the WordPress PHP AI client where text-only messages hit the same 400 because the client was constructing input_text blocks by default.

Why nobody's tests caught it

Every time a silent API change hits, the retrospective hits the same beats:

Unit tests don't catch it because unit tests mock the API response. The mock was built from the docs. The docs still showed input_text as valid.

Integration tests don't catch it unless they're pointed at the live OpenAI endpoint and run often enough to catch the change before a customer does. Most teams run integration tests on PRs, not on a schedule.

Type systems don't catch it because the official Python and TypeScript SDKs type the content[] as a union that includes input_text. The runtime rejects what the static types allow.

Monitoring doesn't catch it because a 400 error on a single request looks like a bad input from your own code. You check your code, can't find the bug, and file it under "weird" — until enough customers report it that you realize it's everyone.

The SDK version didn't change because this wasn't a client-side change. Your openai package is the same version it was yesterday. The server just started enforcing a stricter schema on the exact same request.

That last one is the killer. Pinning the SDK is the first move most teams reach for when an API breaks, and in this case it does absolutely nothing.

The pattern, not just the incident

This is the fourth incident I've written about this month where a top-tier API changed shape without a version bump or a deprecation warning:

GitHub PushEvent stripped payload.commits from the Events API in October 2025. No version bump — the Events API isn't versioned. Abuse-detection pipelines ran on empty data for weeks.
Stripe's Basil release (2025-03-31) removed current_period_end and billing_thresholds from the subscription object. Teams that let their account default version float got silently migrated.
Shopify's 2025-01 Admin API changed fulfillmentHold from a string to an object and removed PrivateMetafield entirely. Apps still on 2024-10 started returning nulls where structured data used to be.
OpenAI's Responses API — this one. No announcement, no version header to pin, just a stricter server-side validator.

The common thread: the API provider has a legitimate reason for the change — abuse mitigation, internal consistency, safer defaults — and the consumer's tests assume a frozen structure. The gap between those two realities is where production breaks.

You can't stop providers from making changes. Most of the changes are even good. What you can do is refuse to find out about them from your own users.

How to actually catch this

Three defenses, in order of how much they help:

1. Pin the API version where the API supports it. Stripe, Shopify, OpenAI's Chat Completions — all of these let you pin. OpenAI's Responses API does not currently expose a version header, which is why this incident hit even teams with otherwise disciplined version management. Pin where you can, and know which of your dependencies don't give you that lever.

2. Assert on response shape in integration tests — and schedule them. Not "does our assistant respond?" but "does content[0].type equal the value we rely on?" And run these tests on a cron against the live endpoint, not just on PRs. Daily is fine. Hourly for anything customer-facing.

3. Monitor the response shape in production. Poll the endpoints you depend on (or sample live traffic), record the shape over time, and diff against a learned baseline. When a field changes type, disappears, or a new enum value starts returning 400s, you get an alert — usually hours before customers do.

The third one is what I've been building at FlareCanary. Point it at your critical OpenAI, Stripe, GitHub, Shopify endpoints — the ones whose schema changes would ruin a Monday morning — and it polls on a schedule, learns the expected structure, and flags drift. Removed fields. Changed types. Nullability shifts. New fields that might be the start of a migration. Severity-classified so noise stays low.

You don't need a dedicated tool for this. You can cron a script that hits your top handful of endpoints, hashes the field set, and diffs. The point is that some layer has to be watching response shape, because nothing else in your stack is.

The harder question

Every one of these incidents ended the same way: developers on a community forum, pasting the error, asking if anyone else had hit it. That's the monitoring layer right now. It's community forums and other people's stack traces.

Most teams know their dependency graph at the package level. They can tell you which OpenAI model they call, what GitHub endpoints they depend on, which Stripe SDK is in package.json. Almost none of them can tell you whether any of those endpoints has returned a different response shape in the last week.

HTTP 200s tell you the endpoint is up. Latency tells you it's fast. Neither of those tells you the contract is still what you thought.

If you'd been diffing Responses API calls against a baseline on October 20, you'd have had an alert on October 21 — before any of your customers opened a ticket. The capability exists. The habit doesn't.

That gap is the real lesson, and it applies to every API you don't control.

If you've been hit by an API schema change that slipped through your tests — especially the "same SDK version, same request, suddenly a 400" variety — drop a reply. I've been collecting these and the pattern is remarkably consistent.

Kubernetes 1.36 Removed gitRepo Volumes — Your Helm Charts Pass Validation, Your Pods Don't Schedule

FlareCanary — Wed, 29 Apr 2026 04:02:08 +0000

Kubernetes v1.36 shipped on April 22, 2026, and one removal is worth a runbook entry on its own: the in-tree gitRepo volume driver is gone.

gitRepo was the volume type that let a pod clone a git repository at startup straight into a mounted volume. It had been deprecated since v1.11 (mid-2018), flagged for security issues (CVE-2024-10220, the symlink escape that let containers traverse out of the cloned tree), and finally retired in 1.36. There is no replacement field. There is no compatibility shim. The schema validator on a 1.36 API server still accepts the field — it has to, because old manifests in etcd reference it — but the kubelet refuses to mount it.

The result is a deploy-time failure that sails through every pre-deploy gate.

The CI/CD pipeline that lies to you

A Helm chart that includes a gitRepo volume looks fine to every tool in the chain:

volumes:
  - name: app-config
    gitRepo:
      repository: "https://github.com/acme/config-repo.git"
      revision: "main"

helm lint passes. helm template renders. kubectl apply --dry-run=server against a 1.36 cluster returns pod/foo created (dry run) — because the API server still validates the schema. The CI pipeline goes green.

At actual apply time on a 1.36 node, the kubelet refuses to mount the volume and the pod sits in ContainerCreating forever, with events that say:

Warning  FailedMount  kubelet  MountVolume.SetUp failed for volume "app-config":
gitRepo volume plugin is no longer supported

Nothing in the Helm chart's metadata indicates that this volume type is gone. Nothing in the chart's chart.yaml signals incompatibility with k8s 1.36. The chart's kubeVersion field is advisory, and most public charts don't update it for individual volume removals.

Where this hides in real codebases

Three places where gitRepo volumes survive in 2026 codebases:

Internal "config-as-code" sidecars. Pre-2020 pattern: a sidecar mounts a gitRepo volume to pull the latest config repo on pod restart. Replaced in most teams by ConfigMaps or Vault, but legacy clusters and forgotten staging environments often still run it.
Helm charts pinned to an old version. Charts on Artifact Hub from 2018-2020 era that haven't been re-published. helm install against a pinned version still pulls the old manifest. The chart's stated kubeVersion range often hasn't been narrowed to exclude 1.36.
Custom operators that generate Pod specs. Operators written by platform teams that emit gitRepo volumes for config-loading. The operator itself doesn't fail upgrade, but the pods it generates do — and the operator's own readiness probe is usually unaware that its child workloads aren't running.

The third category is the most painful. The operator reports "healthy." The CRD instances are Reconciled. Only when you look at the pods directly do you see they've been stuck ContainerCreating for hours.

The migration that's not a search-and-replace

Kubernetes' own deprecation notice points at three replacements: an init container that runs git clone, the git-sync sidecar project, or an external operator like flux/argo for actual GitOps. Each one has different semantics from gitRepo:

Approach	Volume mode	Auth	Refresh
`gitRepo` (removed)	Cloned once at pod create	None (public repos only)	Pod restart only
Init container	Cloned once at pod create	Pass via secret-mounted volume	Pod restart only
`git-sync` sidecar	Continuously synced	Service account, SSH key, or PAT	Configurable interval
Argo/Flux operator	Cluster-wide reconcile	OIDC / GitHub App	Continuous

The init-container approach is closest to drop-in, but it requires you to add an emptyDir volume the init container writes into and the main container reads from, plus a secret mount if your repo isn't public (gitRepo never supported auth, so anyone migrating from gitRepo is by definition working with public repos — but that means they're now exposed to the same supply-chain issues that made gitRepo unsafe in the first place).

git-sync is the closest in spirit. It's an actively maintained project, lives as a sidecar, and does periodic refresh. But it changes pod resource accounting, and on small clusters the extra container can push pods over their memory limits.

The Helm-chart fix isn't a one-line swap. Expect it to touch the pod spec, add an emptyDir volume, add an init container or sidecar, configure auth if the repo isn't public, and update probes if your liveness check assumed instant volume availability at container start.

What to do this week

Three actions, in order:

Grep your manifests, charts, and operator code for gitRepo:. If you're still on 1.35 or earlier, you have until your next upgrade to fix it. If you're on 1.36 already and apply hasn't broken yet, you're flying on workloads that haven't been restarted since the upgrade.
Audit your operator-generated Pod specs. Run kubectl get pods -A -o json | jq '.items[].spec.volumes[]? | select(.gitRepo)' against your clusters. Anything that comes back is a future FailedMount.
Pin chart versions explicitly with kubeVersion guards. When you migrate, narrow the chart's kubeVersion to < 1.36 for the legacy version and bump the major for the migrated version. This stops helm upgrade --install from silently rolling forward to a broken combination.

Worth bonus: Ingress NGINX was retired entirely on March 24, 2026 — no more releases, bugfixes, or security updates. If your cluster's Ingress controller is ingress-nginx (the Kubernetes-project one, not NGINX Inc.'s nginx-ingress), the migration to InGate or NGINX Gateway Fabric is on the same runbook page.

The "everything passes, then breaks at deploy" pattern is what makes K8s upgrades load-bearing for an entire engineering org. The 1.36 gitRepo removal is the cleanest example we've seen this year of CI/CD validation that proves nothing about runtime behavior.

We built FlareCanary for the API-side version of this same pattern: schema accepts the request, response shape passes validation, but the field semantics changed. Same problem, different layer.

If your org runs Kubernetes 1.35 or earlier and any team has a gitRepo volume in their chart, mark the next 1.36 upgrade window as a known-risk deploy. Operator-generated pods are the easiest to miss.

GitHub Just Retired Seven Org Security Fields — Your 'New Repo Hardening' Script Is Now A No-Op

FlareCanary — Tue, 28 Apr 2026 04:01:30 +0000

GitHub announced on April 21 that seven security-defaults fields on the org REST endpoint are now retired:

advanced_security_enabled_for_new_repositories
dependabot_alerts_enabled_for_new_repositories
dependabot_security_updates_enabled_for_new_repositories
dependency_graph_enabled_for_new_repositories
secret_scanning_enabled_for_new_repositories
secret_scanning_push_protection_enabled_for_new_repositories
secret_scanning_push_protection_custom_link_enabled

These appeared on GET /orgs/{org} and were writable on PATCH /orgs/{org}. The retirement isn't a 410 or a schema-validation error. The fields just stop appearing on reads. The PATCH still returns 200 OK whether you send them or not — it just no longer applies them.

Replacement: Code Security Configurations API (/orgs/{org}/code-security/configurations). Different endpoint, different shape, different concept — configurations are objects you attach to repositories, not booleans on the org.

This is the deepest silent-fail vector we've seen all month. It's not a billing column or a string field. It's the one-line setup script every platform-engineering team wrote to harden new repos by default.

The script that quietly stopped working

The classic version:

await octokit.orgs.update({
  org: 'acme-platform',
  advanced_security_enabled_for_new_repositories: true,
  dependabot_alerts_enabled_for_new_repositories: true,
  secret_scanning_enabled_for_new_repositories: true,
  secret_scanning_push_protection_enabled_for_new_repositories: true,
});

Pre-2026-04-21: that PATCH applied org-wide defaults. Every new repo created in acme-platform after this call had Dependabot, advanced security, and secret scanning on at creation.

Post-retirement: that PATCH returns 200. Octokit doesn't throw. The audit log records the call. The fields are simply ignored.

A new repo created tomorrow has none of those defaults applied. The compliance script ran. The dashboard says "secret scanning enabled org-wide." A git push with an AWS access key in it goes through unchallenged.

The terraform-github-provider equivalent (github_organization_settings) wraps the same fields and has the same behavior. terraform apply says no changes. The state file thinks it's converged. The org isn't enforcing what the state file claims.

Why reads break differently from writes

GitHub's REST API doesn't versioned-deprecate fields the way it does for merge_commit_sha. There's no X-GitHub-Api-Version: 2022-11-28 you can set to keep the old field alive. The fields are just gone from GET /orgs/{org} everywhere now.

Reading code that does this:

const { data: org } = await octokit.orgs.get({ org });
if (org.secret_scanning_enabled_for_new_repositories) { /* ... */ }

…now hits undefined. JavaScript's truthiness rules turn that into the false branch. Compliance dashboards that read these fields to display "✓ Secret scanning on" suddenly read "✗" and a security engineer pages someone, OR — worse — the dashboard caches the last known value from a week ago and keeps showing green for months.

Writing code is the more dangerous shape. The PATCH endpoint accepts arbitrary unknown keys without erroring. There's no "did this field actually apply" affordance in the response. Your script can include 47 retired-or-renamed properties and the API still returns 200 with the org object that has none of them on it. Comparing the response back is the only check, and almost nobody writes that compare.

The replacement is not a one-line swap

The Code Security Configurations API is conceptually different. Instead of:

PATCH /orgs/{org}  { secret_scanning_enabled_for_new_repositories: true }

…you now:

POST /orgs/{org}/code-security/configurations to create a configuration object (with secret scanning, Dependabot, advanced security, push protection settings as nested objects).
POST /orgs/{org}/code-security/configurations/{config_id}/defaults to attach it as the default for new repos. The body specifies which repo types it applies to (new_repos, private_repos, public_repos, all).
Optionally, POST /orgs/{org}/code-security/configurations/{config_id}/attach to retroactively apply it to existing repos.

The shape of the configuration object isn't a 1:1 mapping. advanced_security becomes part of a nested struct. secret_scanning_push_protection_custom_link becomes a sub-field of secret scanning. Some legacy combinations aren't expressible. Some new combinations are.

Migration is a real engineering task, not a search-and-replace. And the deprecation notice doesn't have a hard cutoff date — the fields are listed as "Retired" but reads have already broken, so there's no grace window left.

Why your tests didn't catch it

The pattern by now is familiar. Three things have to be true to catch this in CI:

Your fixture for GET /orgs/{org} was regenerated against the live API after April 21 (it wasn't).
Your test asserts presence (expect(org).toHaveProperty('secret_scanning_enabled_for_new_repositories')) rather than truthiness on a possibly-undefined field (most don't).
Your terraform plan/apply tests run against a live GitHub org and diff the result, not against a mocked provider (almost nobody does this).

Most platform teams test their org-hardening script by spinning up a sandbox org once, validating the fields land, and never re-validating. The sandbox org's settings still look right because they were set before the retirement. The script that "still works" hasn't actually applied anything for any new repo created after April 21.

#	API	What changed	Where tests missed it
1	GitHub PushEvent	`commits` field silently dropped	Tests didn't assert field presence
2	Stripe Basil	`current_period_end` moved to `items`	Tests used Checkout fixtures
3	Shopify 2025-01	`fulfillmentHold` type change	Tests mocked the response
4	OpenAI Responses	`input_text` removed for assistants	Tests covered request role=user
5	Twilio regional	Regional domains stop resolving	Tests don't hit prod DNS paths
6	HubSpot Contacts v1	`list-memberships` returns empty	Tests asserted against sandbox fixtures
7	GitHub merge_commit_sha	Field removed from PR responses	Tests used pre-rollout fixtures
8	GitHub org security fields	7 fields retired from /orgs/{org}	Org-hardening script tested once, never re-run

Eight in a row. The breaking change is always in a field a test isn't asserting against, in a layer (transitive SDK upgrade, version-pinned header, configuration drift) the test isn't exercising, or in a script that runs once during onboarding and is never re-validated.

What to do this week

Three actions, in priority order:

Grep your platform-eng repos for the seven field names. Anywhere they appear in a PATCH body or a read assertion is a candidate failure. Org-hardening Octokit calls, Terraform configs, compliance dashboards, audit-log queries — all suspect.
Audit any new repository created in your org after April 21, 2026. Check whether secret scanning, Dependabot alerts, and advanced security are actually enabled on each one. If your org-default script broke silently, every new repo since then has weaker security than the dashboard claims.
Migrate to Code Security Configurations. Create the configuration, attach it as a default for new repos, and — separately — attach it retroactively to repos created during the silent-fail window. The retroactive attach is the cleanup step most migrations forget.

The PATCH that returns 200 OK and does nothing is the worst kind of API regression. There's no error to alert on. The audit log shows you ran the script. Compliance reports green. And every new repo in your org for the past week has its security defaults in whatever state GitHub decided to leave them in — which, based on testing, is "not what your script set."

We built FlareCanary for exactly this layer. Snapshot the response shape on a schedule, diff it, alert when fields disappear or accept-but-ignore semantics shift. The GitHub org security retirement is the textbook case: no error, no version header, no migration warning, just seven fields that used to enforce things and now silently don't.

If your platform team's onboarding script touched any of these seven org-level fields and hasn't been audited since April 21, your org is shipping new repos with whatever default security GitHub decided on — not what your runbook says. Worth a Tuesday afternoon.

DEV Community: FlareCanary

GitHub App installation tokens are getting longer in May 2026 — your VARCHAR(40) column is about to silently truncate them

The four shapes of the failure

Why this is a quiet failure, not a loud one

Where to grep

Why VARCHAR(40) bites the hardest

The migration that doesn't migrate cleanly

What "broadest pattern" means for the regex

What's not changing

Minimum-viable fix

The pattern this fits

Cloudflare is removing in-place DNS record type changes on June 30, 2026 — your Pulumi runs will start failing

Where the breakage actually lives

Why "just delete and create" is a footgun

The Batch DNS records API isn't a drop-in either

Rate limits will surprise the largest scripts

What the silent version of this failure looks like

Minimum-viable fix

The pattern this fits

Stripe Basil Quietly Moved current_period_end Off Subscription — And a Lot of Code Broke

What Actually Changed

billing_thresholds — Removed, Then Quietly Un-Removed

Why Most Teams Didn't Catch the Migration

What Broke In Practice

The Migration That Actually Works

How To Catch The Next One

The Thing That Actually Matters

shopify app deploy --force is going away in May 2026 — your CI/CD is the problem

What the failure looks like

Why --force is going away

Where --force is probably hiding

The migration that actually works

What the silent version of this failure looks like

The pattern this fits

Minimum-viable fix

Supabase's May 30 Default Flip: Your Newly Created Tables Will Stop Reaching the Client With permission denied for table

What Actually Flips on May 30

Why Local and Tests Don't Catch It

The supabase-js Failure Shape

What pg_graphql Disablement Looks Like (May 18)

Why Schema Drift Tools and Schema Validators Don't Help

The Fix You Want in Every New Migration

How to Detect This Class of Change

The Pattern, Now Thirteen Months In

What I'm Building

GitHub Just Removed merge_commit_sha From Pull Request Responses — Your Release Bot Is Probably Tagging null

What's removed and where

Why you don't get an error

The non-obvious replacement

Why your tests didn't catch it

What to do this week

OpenAI Realtime Beta Disappears May 7 — Your Voice Agent's Audio Handlers Will Stop Firing With No Error

The Silent Ones: Renamed Events

The Almost-Silent One: Content Type Renames

The Other Silent One: Restructured Session Config

The Loud Ones (For Completeness)

Why Tests Won't Catch It

How to Detect This Class of Change

The Pattern, Now Twelve Months In

What I'm Building

Exa Just Removed /research and Started Silently Ignoring Two Date Filters — Your Agent Is Probably Pulling Stale Pages Right Now

The Silent One: Crawl-Date Filters

The Almost-Silent One: Null Response Fields

The Loud One: /research Endpoint

Why Most Tests Won't Catch the Silent Pieces

How to Detect This Class of Change

The Pattern, Now Five Months In

Stripe Basil Quietly Moved current_period_end Off Subscription — And a Lot of Code Broke

What Actually Changed

billing_thresholds — Removed, Then Quietly Un-Removed

Why Most Teams Didn't Catch the Migration

What Broke In Practice

The Migration That Actually Works

How To Catch The Next One

The Thing That Actually Matters

OpenAI Responses API Started Rejecting input_text With No Warning — Here's the Fix and Why It Keeps Happening

What happened

The fix

Why nobody's tests caught it

The pattern, not just the incident

Why `--force` is going away

Where `--force` is probably hiding