DEV Community

Philip McClarence
Philip McClarence

Posted on

Postgres Is Down: A 5-Check Triage Tree for Fast Recovery

I have been paged for "postgres is down" maybe two hundred times. Almost every one of those pages resolved into one of five buckets. The fastest recovery I have ever run was not the one where I was cleverest, it was the one where I ran the same checks in the same order and refused to guess.

📖 Read the full guide: Postgres Is Down: A 15-Minute Triage Runbook

Postgres Is Down: A 5-Check Triage Tree for Fast Recovery

So here is the tree. Run it top to bottom. Each check takes under two minutes and eliminates an entire class of failure.

TL;DR: the five checks, in order

  1. Disk. Is the PGDATA or pg_wal filesystem full?
  2. Connections. Are you out of connection slots?
  3. OOM. Did the kernel kill the postmaster?
  4. WAL and archiving. Is a failing archive_command or a stale replication slot pinning WAL?
  5. Logs and corruption. Everything else, read from the server log.

Paste this before you read another word:

# 1. state of the world
systemctl status postgresql* --no-pager | head -20
pg_isready -h localhost -p 5432; echo "exit=$?"
ps aux | grep -c '[p]ostgres'

# 2. disk
df -h $(psql -Atc 'show data_directory' 2>/dev/null || echo /var/lib/postgresql)
du -sh /var/lib/postgresql/*/main/pg_wal 2>/dev/null

# 3. oom
journalctl -k --since "1 hour ago" | grep -i -E 'out of memory|oom-kill|killed process'

# 4. recent server log
tail -100 /var/log/postgresql/postgresql-*-main.log 2>/dev/null || \
  tail -100 $(psql -Atc 'show data_directory')/log/*.log
Enter fullscreen mode Exit fullscreen mode

There is a companion video that walks the first four checks live; this article is the version you can bookmark and grep.

Before you touch anything: the 30-second orientation

You are in exactly one of three states. Figure out which before you do anything else.

  • Postmaster is dead. systemctl status shows inactive/failed, ps aux | grep postgres shows nothing.
  • Postmaster is alive but refusing connections. Normal during startup/recovery, or something is actively blocking new sessions.
  • Postmaster is crash-looping. It starts, dies, restarts, dies again. 
$ pg_isready -h localhost -p 5432
localhost:5432 - no response
Enter fullscreen mode Exit fullscreen mode

pg_isready distinguishes "accepting connections", "rejecting connections" and "no response". That single line splits the tree. No response means the postmaster is dead or not listening. Rejecting connections means it is alive but in recovery or refusing you. Accepting connections while your app screams means the problem is downstream: connection limits, a lock pileup, or DNS.

Then check whether it is crash looping:

$ systemctl status postgresql@16-main --no-pager
   Active: activating (auto-restart) (Result: exit-code) since Sat 2026-08-08 03:14:22 UTC
Enter fullscreen mode Exit fullscreen mode

activating (auto-restart) in a loop is the tell. The service is starting, PANICking, dying, and systemd is restarting it every few seconds.

Do not restart it reflexively. I know the urge. But if the postmaster is currently alive and in crash recovery, restarting throws away recovery progress and starts it over from the last checkpoint. On a busy cluster with large max_wal_size that can turn a four-minute recovery into a twelve-minute one, and you will have no idea why. Restarting can also overwrite the exact log lines and WAL state you need to diagnose a disk-full or corruption branch. Let it finish. Watch the log for redo in progress and consistent recovery state reached.

Check 1: Disk. Is the data directory full? 

$ df -h /var/lib/postgresql
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1    500G  500G     0 100% /var/lib/postgresql

$ du -sh /var/lib/postgresql/16/main/pg_wal
312G    /var/lib/postgresql/16/main/pg_wal
Enter fullscreen mode Exit fullscreen mode

That is your outage. When Postgres cannot write to pg_wal because the filesystem is full, it raises a PANIC and the postmaster shuts down. On restart it tries to write WAL again, fails again, and PANICs again. Crash loop, not corruption. The log looks like this:

PANIC:  could not write to file "pg_wal/xlogtemp.2104": No space left on device
LOG:  startup process (PID 2104) was terminated by signal 6: Aborted
LOG:  aborting startup due to startup process failure
Enter fullscreen mode Exit fullscreen mode

Work the emergency ladder from safest to most destructive. Stop as soon as the server comes up.

Rung 1: delete things that are not Postgres data.

journalctl --vacuum-size=100M
find /var/log/postgresql -name '*.log.[0-9]*' -mtime +2 -delete
rm -f /var/lib/postgresql/16/main/pgsql_tmp/*   # only with the server stopped
Enter fullscreen mode Exit fullscreen mode

Rung 2: expand the volume. If you are on cloud block storage or LVM, this is a single API call and an online resize2fs or xfs_growfs — almost always faster than people think. Do this if you can. It is the only rung with no downside.

Rung 3: reclaim the ext4 reserve. Ext4 reserves 5% of the filesystem for root by default. On a 500G volume that is 25G of breathing room:

tune2fs -m 1 /dev/nvme1n1     # drops reserve to 1%
Enter fullscreen mode Exit fullscreen mode

Set it back afterwards.

Rung 4: neutralise archiving. Covered in Check 4, because you need to understand what it costs.

Never delete files from pg_wal by hand. Not the oldest ones, not the ones that "look archived", not any of them. WAL segments are removed by the checkpointer when it recycles them, or by pg_archivecleanup against an archive directory. Manually removing a segment the server still needs turns a recoverable disk-full into an unrecoverable cluster. I have watched someone do this at 4am and spend the next nine hours on a restore.

Check 2: Connections. Did you hit max_connections?

The symptom in the app log:

FATAL:  sorry, too many clients already
Enter fullscreen mode Exit fullscreen mode

To the application this is indistinguishable from down. The server is fine. It has simply run out of slots.

SELECT state, count(*) FROM pg_stat_activity GROUP BY state ORDER BY 2 DESC;
SHOW max_connections;
Enter fullscreen mode Exit fullscreen mode 
        state         | count
----------------------+-------
 idle in transaction  |   287
 idle                 |    94
 active               |    11
Enter fullscreen mode Exit fullscreen mode

You can still get in: superuser_reserved_connections defaults to 3 and holds slots back for superusers. On PG16 and later there is also reserved_connections plus the pg_use_reserved_connections role, so you can give a non-superuser break-glass account its own pool. Set that up now, not during the next incident.

Kill the offenders. pg_terminate_backend releases the slot; pg_cancel_backend only cancels the query and leaves the connection sitting there, which does not help you.

SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE state = 'idle in transaction'
  AND state_change < now() - interval '5 minutes'
  AND pid <> pg_backend_pid();
Enter fullscreen mode Exit fullscreen mode

Raising max_connections is almost always the wrong first move. Every slot costs memory and every additional backend costs you on lock contention and context switching, and you need a restart to apply it. It doesn't fix the leak, it delays the next occurrence and multiplies memory pressure — which walks you straight into Check 3. You will be back here in three weeks with a bigger number and the same problem.

The durable fix is two settings and one process:

idle_in_transaction_session_timeout = '60s'   # default is 0, disabled
Enter fullscreen mode Exit fullscreen mode

Then put PgBouncer in transaction pooling mode in front. It multiplexes a few hundred client connections onto twenty or thirty server connections, which is the actual answer to application connection sprawl.

Check 3: OOM. Did the kernel kill the postmaster? 

$ journalctl -k --since "2 hours ago" | grep -i 'out of memory'
kernel: Out of memory: Killed process 1841 (postgres) total-vm:9124560kB, \
  anon-rss:7742108kB, file-rss:0kB, shmem-rss:1048576kB, UID:26 pgtable:16924kB
Enter fullscreen mode Exit fullscreen mode

And in the server log:

LOG:  server process (PID 1841) was terminated by signal 9: Killed
LOG:  terminating any other active server processes
LOG:  all server processes terminated; reinitializing
LOG:  database system is in recovery mode
Enter fullscreen mode Exit fullscreen mode

That last sequence is protective behaviour, not corruption. When a backend dies uncleanly the postmaster cannot trust shared memory, so it aborts everything and runs crash recovery. Let it.

Root causes, in the order I find them:

  • work_mem times concurrency. work_mem is per sort or hash node, not per connection. One query with three hash joins and a sort can allocate four or five multiples of it. Set 4MB and run 300 connections doing analytics and you have engineered your own OOM.
  • shared_buffers too high relative to RAM, especially with no swap configured to give you a cushion for transient spikes.
  • Overcommit. The docs recommend vm.overcommit_memory = 2 with a sane overcommit_ratio on a dedicated database host, so allocations fail with an honest error instead of the kernel picking a victim later. 
sysctl -w vm.overcommit_memory=2
sysctl -w vm.overcommit_ratio=80
Enter fullscreen mode Exit fullscreen mode

Also protect the postmaster so children die before the parent — losing one bad query is a far better outcome than losing the whole instance:

# /etc/systemd/system/postgresql@.service.d/oom.conf
[Service]
OOMScoreAdjust=-900
Environment=PG_OOM_ADJUST_FILE=/proc/self/oom_score_adj
Environment=PG_OOM_ADJUST_VALUE=0
Enter fullscreen mode Exit fullscreen mode

Check 4: WAL and archiving. Is a broken archive_command holding your disk hostage? 

SELECT archived_count, last_archived_wal, last_archived_time,
       failed_count, last_failed_wal, last_failed_time
FROM pg_stat_archiver;
Enter fullscreen mode Exit fullscreen mode 
 archived_count | last_archived_wal        | failed_count | last_failed_time
----------------+--------------------------+--------------+---------------------------
         418122 | 0000000100000A4C000000B2 |        90441 | 2026-07-19 02:11:07+00
Enter fullscreen mode Exit fullscreen mode

last_archived_time from three days ago plus a failed_count in the tens of thousands tells you the whole story. With archive_mode = on, Postgres retains every segment that has not been successfully archived. Forever. pg_wal grows until archiving succeeds or you disable it. Common real causes: expired storage credentials, a bucket quota or policy rejection, DNS or network flakiness to the archive target.

Two other things pin WAL. Check both:

SELECT slot_name, active, wal_status, safe_wal_size,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
FROM pg_replication_slots ORDER BY 3;
Enter fullscreen mode Exit fullscreen mode

An inactive slot from a replica you decommissioned in March will happily hold 400GB of WAL. Drop it with pg_drop_replication_slot('old_replica') once you are certain nothing needs it. On PG13+, set max_slot_wal_keep_size so a slot can never take the primary down; the slot gets invalidated instead. The third pinner is a base backup that started and never finished, so check for stale pg_basebackup or pgBackRest processes.

The emergency lever, and what it costs 

psql -c "ALTER SYSTEM SET archive_command = '/bin/true'"
psql -c "SELECT pg_reload_conf()"
Enter fullscreen mode Exit fullscreen mode

archive_command takes a reload. (archive_mode needs a restart, which is why you change the command, not the mode.) The checkpointer will now recycle segments and your disk will drain, often within a minute or two.

This breaks PITR continuity. Every segment "archived" from that point on is a lie — it's discarded, not stored. Your ability to recover to any point after your last full backup is severed at that moment. The instant the fire is out:

  1. Fix the real archive target.
  2. Restore the real archive_command, reload.
  3. Take a fresh full backup. Not tomorrow. Now. Your recovery window starts from that backup, not from whenever you think it started.
  4. Verify pg_stat_archiver.failed_count stops climbing.

The war story

July 2026. Page at 02:40: cluster crash looping, PANIC: could not write to file. df showed 100%, pg_wal was 312GB on a volume sized for 40. pg_stat_archiver showed last_archived_time four days stale and failed_count past 90,000. First instinct was a broken pgBackRest stanza after a recent upgrade. Wrong. The archive command was fine. The object storage bucket had hit its quota and was returning HTTP 403 on every PUT, and the error was being swallowed into a generic archive failure line. Nobody caught it because the archiver retried quietly and the alert threshold was wrong.

Neutralised archive_command, disk drained in about ninety seconds, cluster came up and finished crash recovery. Raised the bucket quota, restored the real command, took a fresh full backup before going back to bed.

The actual root cause started nineteen days earlier. A missing extension .so after a package upgrade was causing autovacuum workers to fail cluster-wide. Nothing was being vacuumed. One table bloated to 177GB against maybe 12GB of live data. Bloated tables inflate base backups, and inflated backups filled the bucket, and the full bucket broke archiving, and broken archiving filled the WAL volume. The outage was three weeks old before anyone got paged. Watch your autovacuum failure counts.

Check 5: The logs and the scary stuff

On Debian and Ubuntu packages, /var/log/postgresql/. On RHEL-family, the log/ subdirectory inside PGDATA. If you are not sure, ask the server: SHOW log_directory; and SHOW logging_collector;.

grep -E 'PANIC|FATAL|invalid page|checksum|could not' \
  /var/log/postgresql/postgresql-16-main.log | tail -40
Enter fullscreen mode Exit fullscreen mode

Crash recovery is normal and can be slow. redo starts at 0/A4C000B2 followed by silence for eight minutes is a healthy server doing its job. Leave it alone.

These are different:

WARNING:  page verification failed, calculated checksum 21847 but expected 9033
ERROR:  invalid page in block 84722 of relation base/16384/24601
Enter fullscreen mode Exit fullscreen mode

That is real damage. Stop.

Copy PGDATA at the filesystem level before you attempt any repair. Stop the server, snapshot the volume or cp -a the directory somewhere else. Every repair tool below is destructive and one-way. If the repair goes wrong, you want the broken state preserved, not gone.

zero_damaged_pages zeroes damaged pages and permanently loses their contents. pg_resetwal discards WAL and can leave you with silently inconsistent data; the docs are explicit that it is a last resort for a server that will not start, and that you should take a filesystem backup first. Neither of these is a fix. They are a way to get a corpse upright long enough to pg_dump what survived.

If you have a tested backup and the damage is more than a page or two, restore. Restoring is boring and predictable. Fixing corruption in place is neither.

Decision table: symptom → check → fix

What you are staring at Check First move
PANIC: could not write to file ... No space left on device 1 df, then the disk ladder
Service in activating (auto-restart) loop 1 or 4 df and pg_wal size
FATAL: sorry, too many clients already 2 Terminate idle-in-transaction, then PgBouncer
FATAL: remaining connection slots are reserved 2 Same, connect as superuser
App reports "can't connect" but pg_isready says accepting 2 Check pool exhaustion at app/pooler layer
Out of memory: Killed process (postgres) in dmesg 3 work_mem, overcommit, oom_score_adj
terminated by signal 9 in server log 3 Confirm with dmesg before assuming corruption
database system is in recovery mode 3 or 5 Wait, watch redo progress, do not restart
pg_wal huge but disk was fine yesterday 4 pg_stat_archiver and pg_replication_slots
archive command failed with exit code 1 4 Test the command by hand as the postgres user
invalid page in block N of relation 5 Copy PGDATA, then plan a restore
pg_isready says rejecting connections orient Read the log, server is alive

After the fire: the four things that prevent the repeat

Forecast disk runway, not disk usage. An alert at 85% gives you a number. An alert that says "pg_wal is growing 8GB/hour and you have six hours left" gives you an action — a 30-day trend graph of pg_wal size would have shown the July incident coming a week out.

Monitor the archiver. Alert on pg_stat_archiver.failed_count increasing and on last_archived_time older than fifteen minutes. That single alert would have caught the July incident on day one.

Put a pooler in front. Before you need it, not after connection exhaustion becomes an outage.

Test the restore. A backup you have never restored is a hypothesis. Restore to a scratch host on a schedule, time it, and write the number down so you know your actual RTO.

If you want the archiver, slot lag and disk runway checks running continuously without building them yourself, that is roughly what we built MyDBA to do. But the checks above work fine from a laptop at 3am, and knowing them is the part that matters.

Top comments (0)