Disk Is Full, but Your Postgres Tables Are Small (The Replication Slot Trap)

Here's a confusing way for a Postgres server to run out of disk. The alert fires, you check your biggest tables, and nothing looks wrong. The database is small. The disk is not.

The space is going to the pg_wal directory, and the usual cause is a replication slot that nothing is reading from.

I reproduced it locally on Postgres 17 so we can watch it happen and then see how to find it and stop it.

What a Replication Slot Does

A replication slot is a bookmark. It tells Postgres: "Keep every piece of WAL from this point on until the consumer confirms it has read it."

The catch is that Postgres keeps the promise even if the consumer never comes back. WAL keeps piling up, and nothing cleans it.

I used a physical slot for the demo, which is the kind a standby server uses. Logical slots, the kind CDC tools like Debezium use, hold back WAL in the same way, but I didn't test one here.

Step 1: A Baseline

First I lowered max_wal_size so the effect would be easy to see, then measured the WAL directory and the database:

ALTER SYSTEM SET max_wal_size = '256MB';
SELECT pg_reload_conf();

SELECT count(*) AS wal_files, pg_size_pretty(sum(size)) AS wal_size FROM pg_ls_waldir();
SELECT pg_size_pretty(pg_database_size('mydb')) AS db_size;

 wal_files | wal_size
-----------+----------
        11 | 176 MB

 db_size
---------
 119 MB

(A note on max_wal_size: my SHOW max_wal_size straight after the reload still printed the old 1GB, and I didn't re-check it. The slot status later in this post reads extended, which means the WAL had grown past max_wal_size, so the limit was in effect.)

Step 2: Create a Slot Nobody Reads

SELECT pg_create_physical_replication_slot('stuck_slot', true);

SELECT slot_name, slot_type, active, restart_lsn, wal_status
FROM pg_replication_slots;

 slot_name  | slot_type | active | restart_lsn | wal_status
------------+-----------+--------+-------------+------------
 stuck_slot | physical  | f      | 0/154F3AB8  | reserved

The true makes the slot reserve WAL from this point right away. active is f because no standby is connected. This is what a dead consumer looks like.

Step 3: Generate Some Writes

CREATE TABLE wal_demo (id serial PRIMARY KEY, payload text);

INSERT INTO wal_demo (payload)
SELECT md5(random()::text) FROM generate_series(1, 2000000);

UPDATE wal_demo SET payload = md5(payload);

CHECKPOINT;

The CHECKPOINT matters. Normally that's when Postgres cleans up old WAL, so it's the fair moment to measure.

SELECT count(*) AS wal_files, pg_size_pretty(sum(size)) AS wal_size FROM pg_ls_waldir();

SELECT slot_name, active, wal_status,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
FROM pg_replication_slots;

 wal_files | wal_size
-----------+----------
        61 | 976 MB

 slot_name  | active | wal_status | retained_wal
------------+--------+------------+--------------
 stuck_slot | f      | extended   | 954 MB

WAL went from 11 files (176 MB) to 61 files (976 MB). The slot alone is holding back 954 MB.

The wal_status column is worth knowing:

• reserved: the WAL the slot needs is within max_wal_size
• extended: max_wal_size has been exceeded, but the files are still kept for the slot
• unreserved: the WAL is no longer guaranteed and will be removed at the next checkpoint
• lost: WAL the slot needed has already been removed

Step 4: Make the Tables Small Again

DROP TABLE wal_demo;
CHECKPOINT;

SELECT pg_size_pretty(pg_database_size('mydb')) AS db_size;
SELECT count(*) AS wal_files, pg_size_pretty(sum(size)) AS wal_size FROM pg_ls_waldir();

 db_size
---------
 119 MB

 wal_files | wal_size
-----------+----------
        61 | 976 MB

This is the symptom from the title. The database is 119 MB. The WAL directory is 976 MB, about 8x bigger. Dropping the table freed nothing, because the slot still insists on keeping the WAL.

How to Find It

This is the query I'd keep around:

SELECT slot_name, slot_type, active, wal_status,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
FROM pg_replication_slots;

Look for a slot with active = f and a large retained_wal. That's your suspect. Then check what the slot belongs to before touching it: a standby that's down, or a CDC connector that stopped.

The Fix

If the consumer is really gone, drop the slot:

SELECT pg_drop_replication_slot('stuck_slot');
CHECKPOINT;

 wal_files | wal_size
-----------+----------
        15 | 240 MB

WAL fell from 976 MB to 240 MB. It didn't go all the way back to the 176 MB I started with after one checkpoint.

One warning: dropping a slot means the consumer can no longer resume from where it left off. For a standby, that can mean rebuilding it. For a CDC tool like Debezium, it can mean taking a fresh snapshot. Don't drop a slot just to free disk without checking who owns it.

How to Stop It Happening Again

Postgres has a safety limit, max_slot_wal_keep_size. It caps how much WAL a slot is allowed to hold back. I set it to 300MB and ran the same kind of write load with a fresh slot:

ALTER SYSTEM SET max_slot_wal_keep_size = '300MB';
SELECT pg_reload_conf();

SELECT pg_create_physical_replication_slot('limited_slot', true);

CREATE TABLE wal_demo (id serial PRIMARY KEY, payload text);
INSERT INTO wal_demo (payload) SELECT md5(random()::text) FROM generate_series(1, 2000000);
CHECKPOINT;

SELECT slot_name, wal_status, safe_wal_size FROM pg_replication_slots;

  slot_name   | wal_status | safe_wal_size
--------------+------------+---------------
 limited_slot | lost       |

The slot went to lost. Postgres removed WAL the slot still needed, so the disk was protected, but the consumer can no longer catch up.

That's the trade-off. Without the limit, a dead consumer can fill your disk and take the primary down. With the limit, the primary survives and the consumer has to be rebuilt or re-snapshotted. Most people prefer the second failure, but you should pick it on purpose.

Also alert on slot lag, so you hear about it long before disk usage does. The query above is a good start.

Clean Up Your Test

DROP TABLE wal_demo;
SELECT pg_drop_replication_slot('limited_slot');
ALTER SYSTEM RESET max_slot_wal_keep_size;
ALTER SYSTEM RESET max_wal_size;
SELECT pg_reload_conf();

Takeaways

• A replication slot makes Postgres keep WAL until the consumer reads it, even if the consumer is dead.
• If the disk is full but your tables are small, check pg_wal and pg_replication_slots first.
• active = f with a large retained WAL value is the red flag.
• Dropping a slot fixes the disk but can force the consumer to rebuild, so check who owns it.
• max_slot_wal_keep_size protects the primary, at the cost of invalidating a slot that falls too far behind.

These numbers come from a single run in a local Docker container, so your sizes will differ. Run it yourself to see it on your own setup.

Comments

[Mike Dabydeen]

I'd alert on the slot rather than the disk, and while its wal_status is still extended. A disk alert fires when the primary is already in trouble. Paging on any slot where active is false and the retained WAL keeps growing gives someone time to restart the consumer before it has to be rebuilt. Once max_slot_wal_keep_size is set, safe_wal_size also tells you how many bytes are left before the slot goes to lost, which makes a clean threshold.

On Postgres 18 there's also idle_replication_slot_timeout. It invalidates a slot that no replication connection has used for longer than the setting, measured from inactive_since, so the slot left behind by a CDC trial nobody cleaned up goes away without waiting for the size cap. The size cap still matters for a consumer that's connected but too slow, since that slot never counts as idle.