You push a deploy. The dashboard accepts it. Then nothing happens.
No error. No crash. No failed build log to read. Your live service keeps serving traffic happily, as if nothing is wrong. The deploy just sits in the queue, waiting for a worker that never picks it up.
That is what happened at Render on Wednesday, October 14, 2026. Between 12:12 and 15:08 PDT, builds and deploys across all regions failed to start, stalled, or were badly delayed, in two periods of impact totalling about 2 hours and 20 minutes. Live user services were not affected. Only the machinery that starts new work was stuck.
That split is the clue worth sitting with. When the thing that is already running works fine and the thing that starts new work does not, the problem is usually not the whole platform. It is in the path that new work takes, and in whatever that path shares with something else.
What happened, in order
Render runs an internal job scheduler. It coordinates builds, deploys, and other critical platform operations. On October 14, the team was rolling out improvements to how that scheduler distributes work, after a successful practice run in a test environment. The planned version upgrade included a database schema migration.
A schema migration is a change to the shape of a database: adding a column, changing a type, adding an index, that kind of thing. Some migrations are quick. Some need a lock on a table, which means other work on that table has to wait until the migration finishes.
This migration locked a large table and drove database CPU use to 100 percent. That database stores two kinds of thing: the scheduler's core job state, and secondary data used for search and reporting. With the database that busy, the scheduler could not read or write job state reliably. Builds and deploys stopped or slowed.
Here is the timeline from Render's postmortem, posted October 22, 2026. All times are PDT on October 14.
12:13: The upgrade starts, including the migration. Database load rises and builds and deploys stop starting.
12:24: Impact identified, incident escalated.
12:34: Engineers revert the upgrade and stop the client performing the migration.
12:37: The scheduler recovers. Builds and deploys resume, but slowly.
13:17: Engineers restart one unhealthy instance of the system to rebalance load. Database connections and errors increase. The second period of impact starts.
13:37: Free tier deploys are paused to reduce load.
13:49: Engineers find the migration still running on the database and terminate it.
14:16: The database is restarted. An attempt to add database capacity fails because of an error from the underlying cloud provider.
14:30: Builds and deploys start to recover, slowly, as the backlog is processed.
15:08: Deploy start times return to normal and database load returns to pre-incident levels.
15:18: Incident resolved.
Read that timeline again and notice the two moments that hurt most. The first is 12:34. The team reverted the upgrade and stopped the client doing the migration, and the system briefly looked better. The second is 13:49, more than an hour later, when they found the migration still running on the database.
Stopping the client did not stop the migration.
The lesson most people miss: a migration lives on the server
This is the part of the story that translates directly to your own work, even if you never run a platform like Render.
When you run a migration, you usually run a small program on your machine, or in CI, that sends a statement to the database and waits. It is easy to think of that program as the migration. It is not. The migration is the work happening inside the database. Your program is just the messenger waiting for an answer.
If you kill the messenger, the database may keep working. If you roll back the deploy that started it, the database may keep working. If you restart your app, the database may keep working. The statement you sent is still there, still holding its lock, still driving CPU up, until it finishes, or until someone kills it on the database side.
That is why the partial recovery at 12:37 is so instructive. Deploys resumed, but deploy start times were about 20 seconds instead of a few hundred milliseconds. That is not a healthy system. That is a system still carrying a heavy load it has not named yet. The migration was still running, quietly, underneath a recovery that looked good enough.
Then a well meaning restart at 13:17, meant to rebalance load, made things worse. Connections and errors rose, and the second, longer period of impact began. When a system is already struggling with a hidden load, adding a restart on top can tip it from slow to stuck.
There were two more unlucky breaks in this incident, and both are worth noting because they are the kind of thing you only discover when you need them. A pre-upgrade step to create a database clone for rollback could not be completed, because of an error from the underlying cloud provider. So a fast rollback was not available. Later, an attempt to add database capacity failed for the same kind of reason. Two escape routes, both blocked by the same dependency. If your rollback plan depends on a step that can fail, test that step, not just the plan that assumes it worked.
Three guards you can add this week
None of this requires a platform team. The guards below are for the database you already run. Render's postmortem does not name its database engine or the exact migration statement, so treat the code here as a general guard for your own migrations, not as a description of Render's setup.
1. Put a leash on every migration
A migration should never be allowed to run forever, holding a lock, while you wonder what happened. Set a lock timeout before you run DDL, which is the database term for statements that change schema. On Postgres, that looks like this:
SET lock_timeout = '30s';
SET statement_timeout = '10min';
ALTER TABLE jobs ADD COLUMN IF NOT EXISTS retry_count integer NOT NULL DEFAULT 0;
A lock timeout says: if you cannot get the lock in 30 seconds, give up loudly instead of queueing behind everything else. A statement timeout says: if the whole statement takes longer than 10 minutes, stop it. The numbers are yours to choose. The point is that a migration fails fast and visibly, instead of succeeding slowly while it strangles the system. A failed migration you can see is better than a running migration you cannot.
If you use a migration tool, check whether it sets these for you. Many do not, by default. That one setting is the difference between a bad deploy you roll back in minutes and a bad afternoon you reconstruct from a timeline.
2. Kill it on the database side, and prove it is dead
When you stop a migration, do not trust the client. Check the database itself for long running statements, and terminate the one you mean to terminate. On Postgres, you can look like this:
SELECT pid, now() - query_start AS running_for, state, query
FROM pg_stat_activity
WHERE state = 'active'
AND query_start < now() - interval '5 minutes'
ORDER BY query_start;
pg_stat_activity is Postgres's view of what every connection is doing right now. pid is the process id of that connection on the database server. If you find your migration there after you stopped the client, terminate that process id on the database side, then run the query again and confirm it is gone. The second check matters. The Render timeline shows a gap of about an hour and a quarter between stopping the client and finding the migration still running. A check that takes thirty seconds would have closed that gap.
In Python, the same habit looks like this: never assume a cancelled task cancelled the work. If you cancel a migration runner, follow it with a check against the database before you declare the rollback done.
def migration_is_still_running(conn, needle):
rows = conn.execute(
"SELECT pid, query FROM pg_stat_activity "
"WHERE state = 'active' AND query ILIKE %s",
("%" + needle + "%",),
).fetchall()
return rows
Pass in a distinctive fragment of the migration statement, and treat a non-empty result as "not rolled back yet". That is a small function, and it is the one Render's team would have wanted at 12:35.
3. Keep hot job state away from reporting data
The scheduler database held both core job state and secondary data for search and reporting. That is a quiet design choice with a loud failure mode. When one store does two jobs, a problem in the less important job can block the important one. A reporting query, a search index rebuild, or a migration on a shared table can all hold up the thing that actually starts your deploys.
You do not need to split the database on day one. But you can ask one question before your next migration: does this table also carry the state that decides whether work starts? If it does, that migration deserves a production sized test, a lock timeout, and a named person watching it. A practice run in a test environment passed for Render, and the migration still locked a large table in production. Test data is usually small. Production tables are not. A migration that is instant on a thousand rows can be a lock storm on millions. If you cannot test at production size, test at a size that at least makes the lock visible, and treat "it was fast in test" as weak evidence, not a guarantee.
Render's planned fixes point the same way: blue-green deploys for this component, so the system is not upgraded under load, cross-zone high availability for the database, and storage mechanisms better suited to the scheduler workload. Blue-green means running the new version alongside the old one and switching traffic over only when the new one is healthy, instead of upgrading the live system in place. That is a bigger change than a lock timeout, but it is the same idea at platform scale: never upgrade the only copy of a critical path while it is carrying traffic.
What to check before your next migration
Look at the last migration you ran, or the next one in your queue, and ask four questions.
First, if you stop the client mid-migration, what kills the statement on the database? If you cannot answer that, find the answer before the migration, not during the incident.
Second, what timeout bounds it? If there is none, add one.
Third, does the table it touches also hold the state that starts work? If it does, that migration is critical path work, whatever its size looks like in test.
Fourth, does your rollback depend on a step that can fail on its own, like a clone or a capacity increase? If it does, test that step separately, and have a second way back.
The Render outage did not come from a exotic bug. It came from an ordinary migration, on an ordinary upgrade, with a rollback plan that had a hole in it and a running statement nobody could see from the client side. That is what makes it worth learning from. Your next migration will also look ordinary, right up until it does not.
Sources: Render postmortem, "Builds, deploys, and queued deploys are not starting in all regions", October 14, 2026 incident, postmortem posted October 22, 2026, https://postmortem.io/incidents/render--2026-10-14--job-scheduler-schema-migration-builds-deploys/
When was the last time a migration surprised you, and did you check the database side before you called it rolled back?
Top comments (2)
On Postgres, stopping the client doesn't stop the statement by default, and the docs say why. With
client_connection_check_intervalat its default of 0, the server notices a closed connection only the next time it reads from or writes to the socket. A long ALTER TABLE does neither until it finishes, so killing the runner leaves it working. Setting that parameter (PostgreSQL 14 and later, on Linux, macOS and the BSDs) makes the server poll the socket and abort the query once the kernel reports the connection closed. I'd set it for the migration role next to your timeouts, and still run yourpg_stat_activitycheck, because a client that hangs without closing its socket won't trip it.One catch with guard 1: if migrations go through PgBouncer in transaction pooling mode, a session-level SET isn't supported there, and the timeout can land on a different server connection than the ALTER. Use
SET LOCALinside the migration's transaction, orALTER ROLE migrator SET lock_timeout = '30s', so the leash goes wherever the statement does.Stopping the client at 12:34 and finding the migration still running at 13:49 is the whole lesson. A rollback that never checks pg_stat_activity is just a guess.