My JWT signing was broken in production and my uptime check said 200 OK

My identity service could not sign a single JWT in production, and my uptime monitor reported a healthy 100 score the entire time.
I found out by accident. I sat down to run a smoke test against my own API, something I had not done in a while, and the first request came back empty. Ninety minutes later I had two bugs fixed and an uncomfortable answer to the question of how long it had been broken: I have no idea, because nothing was calling it.
This is a writeup of the failure, the misleading clue in the middle of it, and the three things I would do differently.
What the service does
It gives AI agents a cryptographic identity. You register an agent and get back a DID plus a JWT signed with ES256. Any service that agent calls can hit a /verify endpoint to check the token, see which permission scopes the agent holds, and have the decision written to a hash-chained audit log.
The relevant detail for this story: almost every interesting path in the app signs or verifies a JWT, and all of them load the signing keys from environment variables. It is a Next.js app on Railway with Postgres behind it.
Symptom one: a POST that returned nothing
I generated a fresh P-256 keypair, built a registration payload, and sent it:

curl -s -X POST "$BASE/api/v1/agents/register" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d "@body.json"

Nothing came back. No JSON, no error, no status line. An empty terminal.
My first assumption was that I had broken the request. I had been fighting a validation error a few minutes earlier, so a malformed body was the obvious suspect. I spent several minutes inspecting the payload, which was fine.
The actual problem was -s. I had reached for it out of habit to suppress the progress meter, and it suppressed the error output along with it. Swapping to -i to print the response headers would have shown me the status code immediately.
If you are debugging an API by hand, use -i. A silent curl tells you nothing about whether you sent a bad request or the server fell over.
Symptom two: a duplicate key that could not exist
I re-sent the same request with -i. This time I got a status line, and it was not the one I expected:

HTTP/1.1 409 Conflict

{"error":"This public key is already registered to agent did:kairos:e45038ce09d8cf9e712b23856d18f7a0","code":"DUPLICATE_KEY","status":409}

That keypair was sixty seconds old. It had never been sent anywhere except the request that returned nothing.
So the silent request had not failed. It had partially succeeded. The agent row was written to the database, and then something after the insert threw hard enough that the response never got serialized.
That narrowed the search enormously. I did not need to look at validation, auth, or the database. I needed to look at whatever ran between the insert and the return statement.
Root cause: dotenv strips quotes, your host does not
The code between the insert and the return was the JWT signing call. It loads a PEM key from an environment variable:

function getPemKey(envVar: string): string {
  const raw = process.env[envVar]
  if (!raw) throw new Error(`Missing env var: ${envVar}`)
  return raw.replace(/\\n/g, '\n')
}

That throw is unhandled. An unhandled throw inside a Next.js route handler means the framework never gets to build a response body, which is exactly the empty-body 500 I was seeing.
In my local .env.local, the key is stored like this, on one line, with escaped newlines, wrapped in double quotes:

KAIROS_JWT_PRIVATE_KEY="-----BEGIN PRIVATE KEY-----\nMIGHAgEAMB...\n-----END PRIVATE KEY-----"

That works locally because dotenv strips surrounding quotes before the value reaches process.env. Railway does not, and neither do most hosted environment variable stores. They treat the value as literal text, quote characters included.
I proved it locally in a few lines:

WITH surrounding quotes    => FAIL: "pkcs8" must be PKCS#8 formatted string
WITHOUT surrounding quotes => OK, token len 232

Same key, same code, two outcomes, separated by one pair of quote characters.
In my case it was worse than a formatting slip. When I opened the Railway dashboard, KAIROS_JWT_PRIVATE_KEY was not set at all, and KAIROS_JWT_PUBLIC_KEY held a value that was not PEM. Every signing path in production had been throwing for however long that had been true.
The code fix is a one-liner, and worth having regardless of how careful you are with the dashboard:

return raw
  .trim()
  .replace(/^["']|["']$/g, '')
  .replace(/\\n/g, '\n')
  .trim()

The second bug: side effects before the thing that can fail
The duplicate key error was not a bug on its own. It was correct behavior sitting on top of a design mistake.
My register route ran in this order:
Validate the request
Check for a duplicate public key
Generate a DID
Insert the agent row
Insert its permission scopes
Sign the JWT
Write the audit entry
Return the DID and token
Step 6 is the only step that can fail on a configuration problem, and it runs after the write. So a signing failure left behind an agent row with no token, no audit entry, and a public key now permanently claimed. Every retry with that keypair hit step 2 and bounced with a 409.
For a first-time user that is the worst possible shape of failure. They get an opaque 500, and when they try again they get told they already registered. Nothing in the UI shows a usable agent, because no token was ever issued.
The fix is to make the fallible step recoverable:

let token: string
let expiresAt: string
try {
  const signed = await signAgentToken({ /* ... */ })
  token = signed.token
  expiresAt = signed.expiresAt
} catch (signError) {
  console.error('[register] Token signing failed, rolling back agent:', signError)
  await supabaseAdmin.from('agents').delete().eq('id', agent.id)
  return apiError(
    'Failed to issue agent token. No agent was registered. Please try again.',
    'TOKEN_SIGNING_FAILED',
    500
  )
}

The scopes table cascades on agent delete, so removing the agent row cleans up behind itself. The caller now gets a real JSON error with a code, and a retry works.
The general rule I keep relearning: do the thing that can fail before you write the thing you cannot easily unwrite. If ordering is forced, make the write reversible.
Why the uptime monitor stayed green
I had three monitors set up: the dashboard, the JWKS endpoint, and the verify API. All green, score 100.
They were checking that a URL responded. Not that it responded correctly.
The dashboard renders fine without a signing key, because rendering does not sign anything. The verify endpoint returns a response for a malformed request too, and a monitor pointed at it with no body and no assertion gets something back and calls it healthy.
So my monitoring was measuring whether Railway was serving traffic. Railway was serving traffic perfectly. The thing that was broken was a step that no monitor ever exercised.
A health check that does not perform the operation your product exists to perform is not monitoring your product. It is monitoring your hosting provider.
What I should have had, and what I am building now: a synthetic check that registers a throwaway agent, verifies its token, asserts allowed === true in the response body, and revokes it. If signing breaks, that check fails within minutes. It costs one scheduled job and a few database rows a day.
The audit log turned out to be a debugging tool
Once signing worked again, I pulled the agent's audit log expecting to confirm the sequence of events. What I found was more useful than confirmation.
There was no AGENT_REGISTERED entry. Not a failed one, not a denied one. It did not exist.
That single absence located the crash precisely. The audit write sits at step 7, after signing at step 6. If signing had succeeded and something later had failed, the entry would be there. Its absence proved the request died between the database insert and the audit write, which is exactly the window signing occupies.
The entries that did exist chained correctly:

07:09:27  TOKEN_REFRESHED   entry_hash: 144ac396...
07:13:17  AGENT_VERIFIED    prev_hash:  144ac396...

Each entry carries the SHA-256 hash of the one before it, so tampering with any entry breaks every hash after it.
I built that chain as a compliance feature. On a Monday night it worked as a debugger, telling me what had not happened, which was the fastest signal I got all evening.
One thing it also exposed: my token refresh route was writing its entries as AGENT_VERIFIED. A refresh is not an access event, and logging it as one would have inflated every access count in the audit trail. That got its own event type.
Five things worth stealing
Quote handling differs between your dotenv file and your host. dotenv strips surrounding quotes. Railway, Vercel, Fly and most dashboard-based stores do not. A value that works locally can arrive in production with two extra characters that break a parser. Strip quotes defensively when you read PEM keys or anything else format-sensitive.
An empty-body 500 means an unhandled throw. If your framework returns a status with no payload, nothing in your error-handling path ran. That is a different class of bug from a 500 your own code returned, and it narrows the search to whatever can throw before serialization.
Do the fallible thing before the irreversible one. If a step can fail on configuration, run it before you commit database writes. Where ordering is forced, make the write reversible and actually test the rollback.
A health check must perform the operation, not just reach the host. Reachability checks tell you your hosting provider is up. If your product signs tokens, your check should sign a token and assert on the result.
Use curl -i, not curl -s. Suppressing output while debugging an unfamiliar failure costs more time than the noise it saves.
The uncomfortable part
I do not know how long this was broken, and I cannot find out, because nobody was calling the API. The outage had no victims. That is not reassuring, it is the actual problem: a service with no users cannot tell you when it stops working.
The thing I am building is KairosAI Identity, cryptographic identity and audit for AI agents. It is at identity.kairosaistudio.com, the SDK is @kairosai/identity on npm, and the free tier covers five agents.
If you are running agents that call APIs you do not own, I would genuinely like to hear how you are handling authorization today, even if the answer is a shared secret in an environment variable. That answer is useful to me.