Railway was breached: a 15-minute key rotation checklist

On October 21, 2026 Railway published a security disclosure: it detected anomalous activity involving the systems that store customer environment variables, and says an unauthorized third party may have accessed encrypted environment variables and, for some of them, obtained the decrypted contents.

Railway's recommendation is to roll all environment variables stored in Railway, including API keys and private registry credentials, and review your service logs for unusual activity. Railway has also invalidated all Railway API tokens and Project Tokens created before the incident, and rotated storage bucket credentials. Live updates are on the status page.

The advice is "rotate everything". This post turns that into a checklist you can actually work through. It's written for a small team with a handful of services.

Note: at the time of writing, builds for some plans are still being re-enabled in stages, so you may not be able to redeploy right away. You can still revoke and create keys now. Do the provider-side work first and redeploy when you can.

0. Inventory (3 minutes)

For each Railway project and service, list the variable names (not values) and where each one comes from:

# Railway CLI: create a NEW token first, old ones were invalidated
railway variables --kv | cut -d= -f1 | sort

Group them by provider: LLM APIs, payments, databases, cloud, GitHub, messaging, everything else. That list is your checklist. Don't paste values into chats, tickets or terminals that are being logged.

1. Rotate the high-value keys first

Do the order by blast radius. Keys that spend money or touch customer data come first.

AI inference keys (OpenAI, Anthropic, OpenRouter, xAI, etc.). Railway says it believes the attack targeted AI inference providers, and it has notified those providers to revoke keys proactively. So:

• Create a new key, then delete the old one (not just "stop using it").
• Check usage and billing dashboards for spend you don't recognise.
• Set or lower spending limits while you're there.
• Railway's guides: OpenAI, Anthropic, OpenRouter, Grok.

Stripe and other payment keys.

• Roll the secret key and any restricted keys. Re-create webhook signing secrets.
• Review recent API request logs and events for activity you didn't trigger. See the Stripe guide.

Databases. Change the password or create a new user, update DATABASE_URL, then drop the old credentials. Railway has guides for PostgreSQL, MySQL, MongoDB and Redis. If the database is reachable from the internet, check its connection logs.

Cloud (AWS / GCP / Azure). Deactivate the IAM access key or service-account key and create a new one. Then review CloudTrail or the audit logs for calls you didn't make. Guides: AWS, GCP.

GitHub tokens. Revoke any PAT or GitHub App private key that was stored as a variable, and check your org audit log and security log. Railway also asks you to reconnect GitHub and redeploy buckets after its credential reset. See the guide and the GitHub access guide.

2. Then the long tail

• Bot tokens (Telegram: /revoke via BotFather; Slack and Discord: regenerate the token).
• Email/SMS providers (SendGrid, Postmark, Twilio, etc.).
• Private registry credentials (npm, Docker Hub, GHCR). Railway calls these out explicitly.
• Your own app secrets: session/JWT signing keys, encryption salts, internal service-to-service tokens. Rotating these will log users out, so plan for it.
• Third-party APIs (maps, search, scraping, etc.).

For each one: create new → update the Railway variable → revoke old → note the time.

3. Recreate Railway tokens

Railway's own API tokens and Project Tokens created before the incident no longer work. Anything that deploys to Railway (CI, scripts, the CLI) needs a new token. Make it project-scoped, not account-wide, where you can.

4. Redeploy and verify

When builds are available for your plan, redeploy so services pick up the new values. Then:

• Hit your health checks.
• Confirm the old keys are actually dead (the provider dashboard should show them as revoked).
• Review the service logs from the incident window for unusual activity, as Railway recommends.

5. Make the next one less painful

Most of the 15 minutes goes on long-lived keys that sit in too many places. Some fixes:

• Use OIDC in CI instead of static cloud keys. GitHub Actions can get short-lived AWS, GCP or Azure credentials (role-to-assume, workload_identity_provider, federated client-id), so there's no key to steal.
• Never print secrets in logs. echo ${{ secrets.X }} relies on log masking, and masking misses encoded or transformed values.
• Scope secrets to the step that needs them, not workflow-level env: where every step and third-party action can read them.
• Pin third-party actions to commit SHAs, so a moved tag can't read your secrets.
• Keep the inventory from step 0 in your repo (names only). Next time it's a checklist, not an archaeology project.

I added checks for exactly these patterns to actions-harden, a free, zero-dependency scanner for GitHub Actions workflows. The new rules are secret-in-log, long-lived-cloud-key and workflow-env-secret, next to SHA pinning, permissions and script-injection checks:

- uses: kobima6813/actions-harden@main
  with:
    fail-on: medium

If you'd rather hear about CI breakages before they hit your builds, there's also CI Early-Warn.

Stay safe, and rotate before you redeploy.