Upgrade a self-hosted Keyright instance safely: migrations, zero downtime, rollback

Upgrading a piece of licensing infrastructure sounds like the kind of maintenance you schedule for 2 a.m. and hold your breath through. Self-hosted Keyright is deliberately boring here, and that is the point. The app is a stateless container that owns its own database schema and migrates it on startup, so an upgrade is mostly "pull a newer image and restart." But "mostly" is not "always," and a license server sits on your customers' activation path, so you still want a procedure that is safe rather than merely quick. This post is that procedure — how upgrades actually work, the safe sequence, how to do it with zero downtime across multiple nodes, and the one thing about rollback that trips people up.

Key takeaway: Upgrades are low-drama because the app is stateless and applies its EF Core migrations automatically on startup — no SQL by hand. The safe sequence is still: back up Postgres first, stage in non-prod, pin the image tag, and roll one node at a time. Rollback means restoring the database backup and redeploying the old tag — migrations run forward, so you do not downgrade against a migrated database.

How an upgrade actually works

Keyright has exactly two moving parts: a stateless app and your database. That split is what makes upgrades simple.

The app is the ghcr.io/delta1-labs/keyright image — one ASP.NET Core service on port 8080, with no database inside it. Every version of the app carries its own schema migrations compiled into the image. On startup it connects to the PostgreSQL you point it at with KEYRIGHT_DB and runs db.Database.Migrate() — first boot creates every table and seeds the default tenant, and an upgrade applies only the new migrations. You never run SQL by hand and you never import a schema. A new version that needs a schema change brings that change with it and applies it the moment it starts.

Because migrations run at startup, the database must be reachable when the new container boots. The readiness endpoint makes this safe to automate: GET /health/ready checks database connectivity and returns 503 until Postgres is reachable and the schema is current, while GET /health is a static liveness 200. Wire your load balancer and orchestrator to /health/ready and a node simply will not receive traffic until it has migrated and is actually ready to serve. If a node cannot reach the database at boot it exits and restarts rather than coming up half-broken — a failure you see immediately, not one that leaks bad responses.

The safe sequence

Even though the mechanics are one step, do these four in order.

1. Back up PostgreSQL first. This is the whole safety net, because the clean way back from a bad upgrade is a database restore (more on that below). Take a pg_dump or a managed snapshot immediately before you upgrade, and confirm you still have your KEYRIGHT_KEK and KEYRIGHT_SIGNING_KEY in their separate secret store — the full discipline is in back up and recover a self-hosted Keyright instance. An upgrade without a fresh backup is a one-way door.

2. Stage in non-production. Restore a copy of production into a staging database, point a staging instance at it on the new image tag, and let it migrate. This is where a schema migration either applies cleanly against real-shaped data or tells you something you want to know before it touches production. Smoke-test the staged instance (below) and only then promote the tag.

3. Roll it out. Pull the pinned new tag and restart. On Docker Compose:

docker compose pull
docker compose up -d

Compose recreates the container from the new image; it migrates on startup and comes back on the same KEYRIGHT_DB, KEYRIGHT_KEK and self-license. On Kubernetes with the Helm chart, bump the image tag:

helm upgrade keyright delta1/keyright --reuse-values \
  --set image.tag=2.2.3

4. Verify. Confirm the new version is live and still licensed (see the verification section below).

The operations guide has the same sequence as a terse checklist you can paste into a runbook.

Zero-downtime rolling upgrade

If you already run two or three nodes for high availability, an upgrade stops being an outage. The app is stateless and backward-compatible within a minor line, so you can roll one replica at a time while the load balancer keeps serving from the others.

The pattern is: take one node out of the pool, bring it back on the new tag, wait for GET /health/ready to pass, then move to the next. Because there is no session affinity — any node can answer any request — nothing is stranded when a node leaves the pool, and concurrency stays safe because seats and credits use a serializable check-and-deduct in Postgres regardless of which node handles the request.

On Kubernetes this is just a RollingUpdate with the readiness probe pointed at /health/ready:

# values.yaml
replicaCount: 3
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 1   # one replica at a time
    maxSurge: 1

helm upgrade keyright delta1/keyright -f values.yaml --set image.tag=2.2.3

Kubernetes replaces one pod at a time, each new pod runs its migrations and only joins the Service once /health/ready is green, and the old pods keep serving until their replacement is ready. You can watch it with kubectl rollout status deploy/keyright and pause or undo the Deployment rollout mid-flight if a new pod never goes ready — though note that undoing the rollout rolls back the pods, not the database, which matters for the rollback story below.

A single-node deployment does not get this for free: docker compose up -d has a brief restart window while the one container recreates and migrates. For a license server that is usually fine, because clients cache a signed lease and validate offline between activations, so a short restart is absorbed by their grace — but if you want true zero-downtime maintenance, that is the argument for running two nodes.

Pin your image tags

Do not float latest. Pin a concrete version — image.tag=2.2.3 on Helm, an explicit tag in your Compose file — so that what you tested in staging is byte-for-byte what rolls to production, and so a pod rescheduled at 3 a.m. does not quietly pull a newer image you never validated. Reproducible deploys are the difference between an upgrade you performed and one that happened to you. The configuration reference covers tag pinning alongside the rest of the Helm values; the same pinning discipline is what makes air-gapped image moves reproducible when you carry a tarball across the gap.

Rollback: restore the backup, do not downgrade

Here is the part to internalise. Migrations run forward. A new version may have altered the schema on startup, and the old image does not know how to run against that newer schema. So you do not roll back by pointing the old image tag at an already-migrated database — that is the one move that turns a recoverable situation into a messy one.

The supported rollback is:

1. Restore the PostgreSQL backup you took in step one, returning the schema and data to their exact pre-upgrade state.
2. Redeploy the previous image tag against the restored database.

Because the old image and the restored (old-schema) database match, the instance comes straight back up. This is precisely why the safe sequence starts with a backup and stages in non-prod: you want to catch a bad upgrade in staging, and if one ever reaches production, you want a clean restore point rather than a schema you have to hand-edit. (On Kubernetes, helm rollback or kubectl rollout undo reverts the pods to the old tag — useful if the new image simply fails to start, but it does not revert the database, so pair it with a DB restore if a migration has already run.)

Verify after upgrading

Three checks confirm a good upgrade:

• GET /health reports the new version and returns 200 — the new image is live and migrated.
• GET /admin/self-license still reports your edition and an active phase — Keyright is fail-closed, so a valid KEYRIGHT_SELF_LICENSE must still be present after the upgrade or every customer request returns 402. The key does not change across an upgrade; this check just confirms it carried over and the instance is serving, not gated. (/admin/self-diagnostics gives the same in one call: version, pending migrations, and license phase.)
• A smoke GET /v1/validate against a known license returns a good lease — proof that the signing key still produces leases your clients accept end to end.

If all three pass, the upgrade is done and you are still on the activation hot path.

Where this fits

Upgrades are the fourth piece of running self-hosted Keyright in production, alongside high availability, backup and recovery, and air-gapped operation. Because Keyright is one product in both deployment modes, the stateless-app-plus-your-database shape — and therefore this whole upgrade story — is identical whether you run one node on a laptop or three behind a load balancer. Self-hosted Keyright is priced per deployment across Standard, Pro and Enterprise (pricing); rolling zero-downtime upgrades want the two or three nodes that Pro and Enterprise cover, but the migrate-on-startup mechanics and the backup-first discipline are the same at every tier.