DigitalOcean's documentation for resizing a managed database says the cluster goes into a "Resizing" state for a few minutes and that it "does not cause downtime". I wanted to see what that looks like from the application's side, while it's writing.
So I made two small Postgres 17 clusters in Frankfurt, one with a single node and one with a standby node. Both were the 1 vCPU, 2 GB size, and I resized each to 2 vCPU, 4 GB while a client wrote to it the whole time.
The setup
A Droplet in the same region ran one writer per cluster, connected over the private network. Each writer inserted numbered rows, one per transaction, about 75 a second. For every insert it logged whether the database confirmed it, and every error and reconnect, with timestamps. On each reconnect it also asked the server who it was: its address, when its Postgres process started, and whether it was a replica.
If an insert failed, the writer reconnected and tried the same number again. The table's primary key made that safe. If the retry hit a duplicate key, the first attempt had actually been committed even though the client got an error.
At the end I compared every number the database had confirmed with what was actually in the table. I also polled the cluster status from the API every two seconds.
Single node: one 5.9 second gap
The resize call returned straight away and the status went to "resizing" a few seconds later. For the next three minutes nothing happened from the writer's point of view. Writes kept going at the same speed, with no slow commits.
Then, 189 seconds in, the connection was cut with "terminating connection due to administrator command". The next connection attempt got "the database system is shutting down", and then 27 more attempts failed over the following few seconds, all but one with "server closed the connection unexpectedly" and once with "connection refused". 5.9 seconds after the cut, a connection worked again and writing carried on.
The server it reconnected to had started its Postgres process about 8 seconds before the old one shut down. The API status went back to "online" 14 seconds after the cut.
With a standby node: the same, 5.3 seconds
I expected the standby to make this shorter. That's the usual point of having one: there's a copy already running, and you switch to it.
It didn't change much. The resize took longer overall, about 400 seconds against 200, but the cut itself looked the same: the same "administrator command" error, the same run of refused connections, and writes came back after 5.3 seconds. And again the server it landed on had started Postgres only 7 seconds before the cut.
That last detail is the interesting one. A standby that had been running and was simply promoted would normally keep its original start time. So either the new primary was a freshly started node, or the standby was restarted as part of the switch. I can't tell which from the client side. Either way, from the client's side the standby didn't shorten the outage in this test.
The second disconnect
There was a second break in both clusters, about 106 seconds after the writer had reconnected, when the status had already said "online" for over a minute and a half.
After the first cut, both writers had reconnected to the new server through a private IPv6 address. 106 seconds later that connection closed with "EOF detected". The writer reconnected immediately, this time over the usual private IPv4 address, to the same server, judging by the same Postgres start time. That gap was under a tenth of a second, so it barely registered as an outage. But it's a second dropped connection, and any client that only reconnects at startup would have failed on it.
The lesson is that "online" in the API doesn't mean the resize is finished with your connections.
The write that failed and succeeded
On the standby cluster, the insert in flight at that second disconnect got "EOF detected" back. The client had no idea whether it went through. When the writer retried the same number, it hit the primary key, because the first attempt had been committed.
So an error on the client doesn't mean the write didn't happen. Without the unique key, a retry would have written that row twice. Without a retry, the application would have reported a failure for a write that was actually saved.
Across both clusters there were four of these uncertain writes. Three hadn't been committed and one had.
What was not lost
Nothing the database had confirmed went missing. The single-node cluster confirmed 94,230 writes and the standby cluster 98,159, and every one of them was in the table afterwards. Outside the cut, commits stayed in the low milliseconds: a median of 1.4 ms on the standby cluster both before and after, and 2.1 ms falling to 1.5 ms on the single node once it was bigger. I also tried the API's install-update call on both clusters, since each had a pending maintenance update, but in the 8 minutes I waited, neither cluster changed status, and both still listed the update as pending.
What I got wrong
I set out to test failover, killing the primary and watching the standby take over. DigitalOcean doesn't let you trigger that yourself. Failover happens when they detect a broken node. So this is a planned switchover during a resize, which is the gentler case.
I assumed the standby would shrink the gap. It didn't, and I still don't know whether the standby was promoted or replaced.
And when I stopped the writers, my pkill -f writer.py matched the command line of the same SSH session that was supposed to run the verification next, so it killed that too. It looked like the check had passed silently. It hadn't run. The real check, run separately, is the one above.
Caveats
This is one resize per cluster, in one region, on small clusters with about 100,000 tiny rows. A larger database may switch differently. There was one writer per cluster doing short inserts; a long transaction open at the cut would have been rolled back with its connection. The timings are from the client and from polling the API every two seconds, not from DigitalOcean's side.
What I'd take from it
The resize was a single API call and lost nothing it had confirmed, which is what you want from a managed database. But "no downtime" in practice meant roughly six seconds of refused connections, plus a second dropped connection about two minutes later. If you plan a resize:
- Make sure your client reconnects on its own, more than once, and not only at startup.
- Treat a write that errored during the switch as unknown, not failed. A unique key or idempotency key turns a retry into something safe.
- Don't count on a standby node to make a planned resize invisible. In this test it didn't.
- Wait a couple of minutes after the status says "online" before calling it done.
Top comments (0)