The algorithm migration itself went fine. The three lives of my JWT signing key tells that story: RS256 to ES256 on a live issuer, both keys published in the JWKS through the overlap, an active-key picker that rotates itself off the legacy RSA key. Keys, rotation, discovery — all the parts that sound hard — worked first try.
This is the story of the part that didn't. Three separate times during that migration, our auth server minted a token whose signature was cryptographically valid — the math checked out, the right key had signed the right bytes — and no validator on earth would accept it. Three different layers had each mangled the bytes in its own way, and each failure looked identical from the outside: invalid signature. Not "wrong format," not "unexpected length." Just: invalid.
If you're staring at that error mid-migration, skip to the field guide at the end. The length of the decoded signature names the guilty layer almost every time.
RSA spoiled us: one integer, one encoding
Here's the thing nobody tells you before an ECDSA migration: RSA never had this problem, so none of your instincts transfer.
An RSA signature is a single integer. The wire format is just that integer, big-endian, padded to the key size — 256 bytes for RSA-2048, and there is essentially one way to write it down. Sign with any library, verify with any other, and the bytes mean the same thing.
An ECDSA signature is a pair of integers, (r, s). A pair needs an encoding, and the ecosystem settled on two incompatible ones:
-
ASN.1 DER — a
SEQUENCEof twoINTEGERs. Variable length (typically 70–72 bytes for P-256), always starting with the byte0x30. This is what OpenSSL, X.509, and TLS speak. -
Raw R‖S (IEEE P1363) —
randseach left-padded to exactly 32 bytes and concatenated. Always exactly 64 bytes. This is what JWS mandates for ES256 (RFC 7518 §3.4).
Both encode the same signature. Both are "correct." Hand a validator the one it isn't expecting and it doesn't say "wrong container" — it decodes garbage integers and reports the only thing it can: the signature doesn't verify.
Every layer between your key and the wire has an opinion about which of these it speaks. Ours had three layers, and we got all three wrong, sequentially.
Footgun one: the KMS answers in DER
Our private key lives in HashiCorp Vault's transit engine and never leaves it — the application sends bytes to be signed and gets a signature back. When we switched the transit key from rsa-2048 to ecdsa-p256, the sign request still carried a leftover from the RSA era: signature_algorithm=pkcs1v15.
That's an RSA padding parameter. It is meaningless for an ECDSA key. Vault did not error. It ignored the irrelevant parameter, signed with the curve key, and returned 200 with a perfectly valid ECDSA signature — in its default marshaling, which is DER.
So the happy path stayed green end to end: request accepted, signature returned, openssl dgst -verify would have blessed it. And every JWT library rejected every token, because after base64url-decoding the third segment it expected 64 bytes of R‖S and found 71 bytes starting with 0x30.
The fix is one request parameter: Vault's transit sign endpoint takes marshaling_algorithm=jws, which returns raw R‖S directly. But note the shape of the trap: a parameter that is wrong for the key type and silently ignored. The request looked wrong, was wrong, and worked anyway — for the wrong definition of worked.
Footgun two: "just concatenate the integers" fails one token in 128
If your KMS can't emit R‖S natively, you'll be tempted to convert DER yourself: parse out the two INTEGERs, concatenate, done. This is the nastiest footgun of the three, because it's intermittently correct.
DER integers are minimal-length and signed. That means:
- an
rthat happens to be numerically small — high byte zero — encodes in 31 bytes or fewer, because DER strips leading zeroes; - an
rwhose high bit is set picks up a0x00sign byte and encodes in 33 bytes.
The JWS format wants neither. It wants each value left-padded to exactly 32 bytes, always. A naive concatenation of the DER integers produces 63-, 64-, 65-, or 66-byte signatures depending on the draw, and only the 64-byte draws verify. Handle the sign byte but forget the padding and you're left with a converter that fails whenever r or s starts with a zero byte — each is a 1-in-256 event, so about one token in 128 comes out short.
That version passes your smoke test, passes code review, ships, and then fails one login an hour in production with no pattern you can see. We dodged it by making Vault do the conversion (marshaling_algorithm=jws again — let the party that owns the signing operation own the format), but the same trap has a purely local variant in .NET: ECDsa.SignData/VerifyData take a DSASignatureFormat, and different corners of the framework default differently — the SignedXml/certificate world thinks in Rfc3279DerSequence while the raw ECDsa APIs default to P1363. Our local verification path now says DSASignatureFormat.IeeeP1363FixedFieldConcatenation explicitly. After this migration, we don't let any signature API default its format, anywhere.
Footgun three: the fix changed the base64 dialect underneath us
Vault wraps every signature in an envelope: vault:v1:<encoded-bytes>. With the default DER marshaling, the encoded part is standard base64, and our client dutifully called Convert.FromBase64String on it.
Switch to marshaling_algorithm=jws and Vault — reasonably, per the spec it's now speaking — encodes the signature portion in base64url: the -/_ alphabet, no padding. One request flag had changed two things at once: the byte format we asked for, and the text encoding it arrived in.
Convert.FromBase64String wants padded standard base64. A 64-byte signature is 86 base64url characters — not a multiple of four — so the decode threw a FormatException on every single token. In a way this was the kindest of the three failures: it at least crashed, loudly, at the exact line that was wrong, instead of producing valid-looking bytes that quietly failed elsewhere. The fix is to decode (and, on the verify round-trip, re-encode) with a base64url codec.
Count the irony: the signature was finally in the right byte format, and now the text layer mangled it. Three encodings, stacked: the integer-pair encoding, the fixed-field padding, and the character encoding. Each one independently able to turn a valid signature into a rejected token.
The field guide: the length names the culprit
When a JWT fails validation mid-algorithm-migration, base64url-decode the third segment of the token and look at what you have:
| Decoded signature | Which layer mangled it |
|---|---|
| Exactly 64 bytes | Wire format is right — look elsewhere (kid mismatch, wrong key in JWKS, alg pinning) |
70–72 bytes, first byte 0x30
|
DER leaked through: the KMS or crypto library is emitting ASN.1 |
| 63 or 65 bytes, and only some tokens fail | A hand-rolled DER→raw conversion without fixed-field padding |
| Decode throws or yields garbage | base64 vs base64url mismatch one layer down |
And one cross-check that separates format bugs from key bugs in seconds: verify the same signature with OpenSSL, which speaks DER. If OpenSSL says valid while every JWT library says invalid, you have an encoding problem. If both say invalid, you have a key problem.
The lesson
"Cryptographically valid" is a property of the mathematics. "Verifiable" is a property of every encoding boundary between the mathematics and the consumer — and our signature crossed four of them: the KMS's marshaling, its transport envelope, our client's decode, and the JWS wire format. Three of those boundaries had opinions, and each disagreed with its neighbour exactly once.
The testing moral is the one we actually changed process over: never prove a signing path by verifying with your own code. Our sign path and our verify path shared every one of these assumptions, so sign-then-verify round-trips passed while every real consumer rejected us. The test that finally pinned all three bugs mints a real token, validates it with the platform's stock JWT library against the published JWKS — someone else's assumptions, not ours — and then asserts one brutally specific thing on top: the decoded signature is exactly 64 bytes. The math will look after itself. It's the bytes you have to watch.
Top comments (0)