The deploy wasn't the problem. Rolling back proved it — and restoring the database made things worse.
When production goes down right after a deployment, rolling back feels like the safest move. Put back what worked, get the sites up, investigate later. Most of the time that's right. This time it wasn't, and the logs showed why.
What happened
I deployed an update to six websites running on Servd, all powered by a single Craft CMS installation. The changes had been tested and approved on staging.
A few minutes after the deploy, the sites started returning errors and timing out.
My first thought was to roll back. I expected it to take a few minutes: get the sites running again, then find out what went wrong. I asked Servd for the logs, rolled back the code, and restored the database from the backup taken just before the deploy.
Rolling back the code didn't help.
What the logs showed
When the logs came in, they told a different story from the one I'd assumed:
- The old code failed in exactly the same way. After the rollback, requests averaged 46 seconds, and some hit the 120-second timeout.
- The caches were empty. Craft clears its own caches on every deploy, and I had also cleared the full page cache myself, as I usually do. Every visitor and every bot was asking for an uncached page at the same moment.
- The server was already near its limit. Load had been high all day, well before the deploy.
- What brought the sites back wasn't me. Servd's automatic protection switched on its "Under Attack" mode, filtered out bot traffic and restarted PHP. Within a minute everything was running again — before my database restore had even started.
The problem was already there. The deployment only triggered it.
Why a deploy can tip a busy server over
On a warm site, most requests never reach Craft's heavy work: pages, queries and template fragments come out of a cache. Right after a deploy that cache is empty. Every request now does the full work — for real visitors, and for every crawler that happens to be on the site.
Six sites on one installation share one server. If that server is already busy, a minute of cold caches across all six is enough to queue requests until they time out. And a rollback brings the old code back with the same empty caches, so it fails the same way.
Rolling back code vs restoring the database
These two feel like one action. They aren't.
Rolling back code is cheap and reversible. If the old code works and the new one doesn't, you've learned something in minutes.
Restoring the database is heavier. It replaces the live data with a snapshot, and the application has to rebuild everything it had cached from the data that's now gone. Here it caused another 16 minutes of errors, and the sites only recovered after I cleared Craft's data cache.
There is one case where a restore belongs in the rollback: when the release ran migrations that the old code can't work with. Then old code and new database don't fit together. If the release didn't change the schema — or the old code already fails the same way — the restore only adds risk.
The same release, the next morning
The next morning I deployed exactly the same release again, unchanged. Two things were different:
- I switched on bot protection before the deploy.
- I left the full page cache alone and let pages refresh on their own.
It went smoothly.
What I do differently now
- Check the server's load before deploying, not only that staging works. A server close to its limit is one cold cache away from going down.
- Turn on bot protection before deploying busy sites, so crawlers don't arrive first at an empty cache.
- Don't clear the full page cache out of habit. Let it refresh, unless the release actually changes what cached pages should show.
- Look at the logs before assuming the new code is broken. If response times were already climbing before the deploy, or the old code fails the same way, it's not the code.
- Treat a database restore as a separate decision. Roll back code first. Restore data only if the release changed the schema in a way the old code can't handle.
Looking back, I still think rolling back was a reasonable first reaction. Restoring the database wasn't necessary, and it cost us more downtime. Next time I'll still consider rolling back first — but I'll think twice before touching the database.
Quick answers
Should I restore the database when I roll back a deploy?
Usually not. Roll back the code first. Restore the database only if the release changed it in a way the old code can't work with. Otherwise a restore just adds downtime — here it added 16 minutes of errors.
Why can a site go down right after a deploy that worked on staging?
A deploy empties the caches, so every visitor and every bot makes the server do the full work at the same time. If the server was already busy, that alone can make it time out, even when the new code is fine.
Originally published on b10k.nl. I'm Bohdan, a freelance Craft CMS developer in the Netherlands — I build and maintain Craft sites and write Craft plugins.
Top comments (0)