DEV Community: marsdiscovery

The webhook signature matched in Postman. Staging still returned 401.

marsdiscovery — Tue, 23 Jun 2026 02:06:45 +0000

We wired up a new billing webhook last sprint. Local tests passed. Postman collection green. Merged on Friday.

Monday morning staging logged hundreds of 401 invalid signature from the provider's retry queue. The handler was maybe fifty lines. The secret lived in an env var copied straight from the vendor dashboard.

I replayed the same payload in Postman with the same X-Signature header. Verification passed. Staging failed on requests that looked identical.

The gap wasn't the HMAC math on paper. It was which bytes we hashed.

Our API gateway parsed JSON before the route handler ran. By the time we read req.body, it was a JavaScript object serialized again — keys in a different order, sometimes a trailing newline gone. The provider had signed the raw request body from the wire. We were hashing a re-exported string.

Everything in logs looked fine: one-line JSON, correct algorithm in the header, secret without obvious typos. That's what made it annoying. I spent an hour convinced the vendor had rotated keys without telling us.

What I check now on webhook PRs:

Raw body first. If the framework touches JSON before your verifier runs, find where to hook the unparsed buffer. Signature schemes don't care about your ORM.

Secret encoding. Dashboards often show base64-looking strings. Is the env var the decoded key bytes, or the base64 text itself? Wrong layer and HMAC still "works" in a scratch script — with the wrong input.

One failing payload, viewed cleanly. I copy the exact body and signature into separate tabs so I'm not squinting at escaped quotes wrapped in a narrow terminal.

Browser pages I keep for these checks — one job each, local processing, no account for a two-minute look:

https://devcove.dev/en/tools/json-formatter/

https://devcove.dev/en/tools/base64/

https://devcove.dev/en/tools/hash-generator/

I maintain a small browser toolkit called DevCove for exactly these chores when I'm already in DevTools on another tab. curl --data-binary and a server-side replay fixture still win for the real fix; this is for ruling out "wrong bytes" vs "wrong secret" before you open a ticket with the vendor.

Half the outcome was code: verify on raw body, add one canned replay test. The other half was stopping the assumption that Postman parity means production parity.

If you maintain signed webhooks — where do you usually lose the plot? Middleware reordering the body, secret encoding, or something else entirely?

Anthropic's Friday kill switch changed how I split AI from deterministic work

marsdiscovery — Tue, 16 Jun 2026 06:29:33 +0000

I had a half-finished refactor open in Cursor when a teammate dropped a link in Slack: Anthropic had pulled Fable 5 and Mythos 5 offline. Not a rate limit. Not scheduled maintenance. Gone.

My first thought wasn't policy. It was practical. A script we run in CI uses Claude to summarize test failures. Another engineer routes security-triage prompts through the API. We both assumed the model would be there on Monday like electricity.

It wasn't.

A letter at 5:21 p.m.

According to Anthropic and reporting from Lawfare and TechCrunch, the U.S. Commerce Department sent a letter on Friday, June 12, at 5:21 p.m. Eastern. Citing export-control authorities, it told Anthropic to cut off Fable 5 and Mythos 5 for any foreign national — inside or outside the United States, including foreign employees at Anthropic itself.

Anthropic can't reliably sort API customers by nationality. So the company did the only thing that guaranteed compliance: it disabled both models for everyone.

Within hours, one of the most capable model pairs on the market was unavailable worldwide. No court hearing. No public text of the letter. Just a product that existed on Thursday and didn't on the weekend.

If you build on someone else's API, that sequence should bother you.

This wasn't really about a jailbreak — or maybe it was

The official reasoning is still fuzzy. Anthropic believes the letter relates to a guardrail bypass. David Sacks framed it as a serious vulnerability — reportedly demonstrated by a "trusted partner" (widely reported as Amazon security researchers) — and said Anthropic refused to patch or de-deploy when asked.

Anthropic pushed back: the issue was narrow, partly known before launch, and present in rival frontier models too. Pulling the models globally, they argued, would essentially halt frontier deployments for every lab if this became the standard.

Then Katie Moussouris — founder of Luta Security — weighed in. Anthropic had shared a private paper describing the alleged bypass. Moussouris wrote that the behavior "should never have triggered an export control." The difference, she explained, was largely between asking a model to review code for security issues and asking it to fix this code. Same output class, different phrasing.

Dozens of security researchers signed an open letter asking the administration to revoke the order. Their argument: yanking advanced models from defenders during an active dispute is worse for U.S. cybersecurity than the risk being debated.

Axios, citing sources, described personality friction between Anthropic and the administration as a bigger driver than the technical report itself. That part is harder to prove. But the pattern fits: Anthropic is already suing over a Pentagon supply-chain blacklist after refusing unrestricted military use of its models. Whether or not you trust Anthropic's safety posture, it's difficult to read this as a purely technical intervention.

Export controls meet SaaS

Lawfare's analysis is worth sitting with even if you don't follow trade policy. The Export Administration Regulations were built for chips and tangible technology. Biden's AI Diffusion Rule briefly put model weights on the control list; Trump rescinded it on retaking office. This Anthropic action may be the first time export controls were used to choke access to a live API model.

That raises messy questions:

Are the weights being controlled, or the outputs?
Is foreign access to a U.S. SaaS product an "export" at all?
If guardrails fail, does that justify a blanket ban on all non-U.S. persons — including allied researchers?

Congress passed nothing resembling a clear licensing regime for frontier AI before this happened. Anthropic's CEO had actually argued days earlier that government should be able to block dangerous deployments — with protections against arbitrary decisions. A standardless letter on a Friday evening is probably not what he meant.

Foreign capitals will wonder whether American AI infrastructure is reliable for critical systems. Today's favorite U.S. model can become a bargaining chip tomorrow.

What broke for developers on the ground

Most of us aren't training frontier models. We paste stack traces into chat windows, wire up API calls, and move on. The kill switch still landed in real workflows:

Teams mid-sprint on Fable or Mythos had to fail over, rewrite prompts, or pause features.
Startups without multi-provider abstraction learned an expensive lesson about single-vendor dependency.
Security engineers who wanted Claude for log triage found the tool missing during an already tense news cycle.

None of this is catastrophic if you treat LLMs as optional accelerators. It is catastrophic if you've designed your product so the model is the product — no fallback, no offline path, no deterministic substitute for the boring parts.

AI is great at judgment-heavy work: explaining an unfamiliar codebase, drafting a migration plan, suggesting why a flaky test might be order-dependent. It's terrible as the only way to decode a JWT, format a JSON error body, or confirm whether a log timestamp is seconds or milliseconds. Those jobs don't need reasoning. They need correctness, speed, and a clear boundary about where your data goes.

When Anthropic's models went dark, the deterministic chores didn't disappear. Pipelines still emitted malformed JSON. Staging still handed us tokens we shouldn't paste into random websites. Cron expressions in YAML still deserved a second look before deploy.

What I changed — a little

I'm not abandoning AI assistants. They're too useful for the messy middle of engineering. But the weekend reminded me to separate two buckets:

Bucket A — work where the model adds judgment: design tradeoffs, unfamiliar APIs, long explanations.

Bucket B — pure transformation: format, decode, encode, validate, generate a UUID, parse a cron field.

Bucket B should never depend on a frontier API being online, priced the same, or politically acceptable in every jurisdiction. It should run locally, finish in seconds, and not send customer payloads to a third party.

I maintain a small browser workbench called DevCove for exactly those Bucket B tasks — separate pages, one job each, processing in the browser after load. Not because local tools are morally superior, but because they're boring in the right way. They don't get export-controlled.

Tabs I kept open even while the Claude API was down:

For a longer take on why local-first tooling still matters when AI is everywhere, see Local-first developer tools in the AI era.

jq, IDE plugins, and one-liners in the terminal all work too. The point isn't which tab you use. The point is having a fallback that doesn't require a model endpoint, an account, or a policy debate in Washington.

The precedent outlives the outage

By the time you read this, Anthropic and the administration may have patched things up. Models may be back. That's almost beside the point.

The precedent is set: a U.S. AI product can be switched off globally on short notice, with limited transparency, while the industry argues about whether the triggering incident even qualifies as a national-security event. If you're an engineering lead, that belongs in your risk register next to cloud region outages and vendor price hikes.

If you're an individual developer, the lesson is smaller and more actionable: don't merge Bucket A and Bucket B. Let AI help you think. Let local, deterministic tools handle the fragments of your day that shouldn't need a network round trip — especially the fragments that contain secrets.

How are you handling provider risk now — multi-model routing, self-hosted weights for specific tasks, or fingers crossed and a cached API key in the vault?

What's in your fallback plan when the API you depend on goes dark on a Friday?

The webhook signature matched in Postman. Staging still returned 401.

marsdiscovery — Sat, 06 Jun 2026 08:09:55 +0000

We wired up a new billing webhook last sprint. Local tests passed. Postman collection green. Merged on Friday.

I replayed the same payload in Postman with the same X-Signature header. Verification passed. Staging failed on requests that looked identical.

The gap wasn't the HMAC math on paper. It was which bytes we hashed.

What I check now on webhook PRs:

Raw body first. If the framework touches JSON before your verifier runs, find where to hook the unparsed buffer. Signature schemes don't care about your ORM.

One failing payload, viewed cleanly. I copy the exact body and signature into separate tabs so I'm not squinting at escaped quotes wrapped in a narrow terminal.

Browser pages I keep for these checks — one job each, local processing, no account for a two-minute look:

https://devcove.dev/en/tools/json-formatter/

https://devcove.dev/en/tools/base64/

https://devcove.dev/en/tools/hash-generator/

Half the outcome was code: verify on raw body, add one canned replay test. The other half was stopping the assumption that Postman parity means production parity.

If you maintain signed webhooks — where do you usually lose the plot? Middleware reordering the body, secret encoding, or something else entirely?

The redirect_uri matched in the dashboard. OAuth still failed.

marsdiscovery — Thu, 04 Jun 2026 07:18:26 +0000

We spent an afternoon convinced the identity provider was broken. Login worked locally. Staging failed with the usual vague error — redirect mismatch, invalid request, pick your vendor wording.

The redirect_uri in our admin console was copied from the ticket. Character for character, it looked identical to what the browser sent. No trailing slash debate. No http vs https surprise.

Then I dumped the authorize URL from the network tab and compared it to what we had registered.

One query value had been encoded twice. Another used + for a space in a place our stack later read as a literal plus. The path wasn't wrong. The encoding layers were fighting each other.

This is the boring class of bug: nothing shows up in a unit test because each function "works" in isolation. The failure only appears when a full URL travels through a form, a framework helper, and a provider that compares strings literally.

What I watch for now:

Before blaming the IdP, decode the authorize URL once in a sane viewer. I want to see each query key and value split apart, not a single opaque percent soup.

Plus vs %20. application/x-www-form-urlencoded treats + as space. Generic URI encoding uses %20. Mix those rules in one redirect chain and you get ghosts.

Nested URLs. Some flows put a return URL inside a query parameter. Encode the inner URL as a component, not as "the whole string again with extra % on the %."

One-line error JSON from the token endpoint still shows up in the same incident. Pretty-printing it once saves ten minutes of guessing whether the problem is OAuth or your own handler.

Browser tabs I keep for these checks — one job per page, local processing, no account for a two-minute look:

https://devcove.dev/en/tools/url-encoder/

I maintain a small toolkit called DevCove for exactly these chores (format, encode, decode in the tab I already have open). curl and server logs still win for reproduction; this is for the moment you're staring at a URL in DevTools and need to know if you're looking at double encoding or a real mismatch.

Not claiming OAuth is easy. Half our fix was process: register the exact bytes the browser will send, document which encoder each service uses, stop "fixing" URLs by encoding until it looks scary.

If you've shipped OAuth lately — where do you catch encoding bugs first? Provider logs, proxy logs, or a scratch pad in the browser?