A backup job that dies halfway through a pipe can leave behind a file that reports success, passes its integrity check, and lists its contents correctly — while containing almost none of your data.
I did not want to take that on faith, so I killed a pg_dump on purpose and measured every check I could think of. Three of them passed. Two of those three are the ones people reach for precisely because they are supposed to catch this.
The rig
A throwaway cluster, built for the test and destroyed afterwards, so nothing touched a real database:
initdb -D /tmp/pgtest/data -U test --auth=trust -E UTF8
pg_ctl -D /tmp/pgtest/data -o "-p 55432 -k /tmp/pgtest" start
psql -d app -c "CREATE TABLE t(id serial primary key, payload text);"
psql -d app -c "INSERT INTO t(payload)
SELECT repeat(md5(g::text), 40) FROM generate_series(1,100000) g;"
That is 100,000 rows of roughly 1.3 KB of text each, dumping in about a quarter of a second. Slow enough to interrupt reliably, fast enough to iterate.
Then the failure, which is the ordinary one — pg_dump dies, gzip keeps going:
pg_dump -d app 2>/dev/null | gzip > app.sql.gz &
sleep 0.06
pkill -KILL -f "^pg_dump -d app"
Check 1: the exit code — passes
pipeline exit status: 0
A shell pipeline reports the status of its last command. gzip finished cleanly, because the pipe closed and it had nothing left to read. So the pipeline returns 0, and a script written as
pg_dump app | gzip > app.sql.gz
continues happily to its next line, which is usually "backup succeeded" followed by a rotation step that deletes yesterday's copy.
Check 2: gzip -t — passes
gzip -t app.sql.gz
→ PASS
The archive is valid gzip. It is genuinely, correctly compressed. The compression stream has a proper EOF, because gzip was never killed — only its input died. gzip -t verifies the compression, not the payload, and the compression is perfect.
The file was 81,068 bytes against 2,753,525 bytes for the complete dump. 2.9% of the data, zero complaints from the integrity check.
Check 3: pg_restore -l — also passes
This is the one that surprised me. My instinct was that the custom format would be safer, because pg_restore -l reads the table of contents and would obviously notice if anything were missing.
It does not. For -Fc, the TOC is written near the start of the file, before the data blocks. A file truncated during the data phase still has a complete, readable TOC.
full -Fc dump: 2,739,118 bytes, 7 TOC entries
truncated dump: 163,840 bytes (6.0%)
pg_restore -l truncated.dump
→ exit 0, 7 entries listed
Seven for seven. pg_restore -l will tell you an archive contains what it never managed to store.
What actually catches it is trying to use the file rather than inspect it:
pg_restore -f /dev/null truncated.dump
→ pg_restore: error: could not read from input file: end of file
→ exit 1
So the "professional" check and the naive one fail identically. If you swapped gzip -t for pg_restore -l believing you had upgraded your validation, you bought the same blind spot in a better suit.
Aside: pipefail is bash-only, and cron does not use bash
The standard advice is set -o pipefail. It works, and it is also a portability trap:
$ sh -c 'set -o pipefail'
sh: 1: set: Illegal option -o pipefail
$ ls -l /bin/sh
lrwxrwxrwx 1 root root 4 Jan 5 2023 /bin/sh -> dash
On Debian and Ubuntu, /bin/sh is dash. A script with #!/bin/sh that you tested by hand in an interactive bash session will either die on that line or — worse, if the exit status is not checked — run with the protection silently absent. Cron runs #!/bin/sh scripts under dash, so the environment where the backup actually runs is the one where the guard does not exist.
Either declare #!/usr/bin/env bash, or do not rely on pipelines for correctness at all.
The fix: stop using a pipeline
pg_dump can compress its own output. That removes the entire class of problem, because there is no second process whose exit status can mask the first:
pg_dump -d app -Z6 -f app.sql.gz
Measured against the same kill:
| Approach | Exit status after the kill | gzip -t |
|---|---|---|
pg_dump piped into gzip
|
0 | PASS |
pg_dump -Z6 -f app.sql.gz |
137 | FAIL |
Two independent signals instead of none, and no pipefail needed.
The reason gzip -t starts working is subtle and useful: when pg_dump compresses internally and is killed, the resulting gzip stream is never finalised — it is missing the CRC32 and ISIZE trailer. In the pipeline version, gzip was never killed, so it wrote a complete, correct trailer around incomplete input. Same truncated payload, opposite integrity verdict.
What actually detects truncation
Ranked by cost, from the measurements above:
| Check | Detects truncation | Cost |
|---|---|---|
Exit status (no pipefail) |
No | 0 |
gzip -t |
No | full decompress |
pg_restore -l |
No | cheap, unreliable |
| Size vs rolling median | Yes | ~0 |
-- PostgreSQL database dump complete footer |
Yes | full decompress |
pg_restore -f /dev/null |
Yes | slow, conclusive |
The cheapest real guard needs no decompression at all: compare the size against recent history. 81 KB where the last thirty runs were 2.7 MB is not a shape you need to open the file to distrust. A ratio check against a rolling median catches this class of failure for free, and it catches the ones that produce a different exit code from the one you anticipated.
The footer check is the best cheap-and-conclusive option for plain-format dumps, because pg_dump writes -- PostgreSQL database dump complete as its last meaningful line. grep -q for it after decompressing is slower than a size check and far more definitive.
The general shape
This is the same failure as a backup service that exits 0 because it ran, and a monitoring system that is green because nobody defined what red means. The check exists, it runs, it returns success, and it is answering a different question than the one you asked it.
gzip -t answers "is this file valid gzip". pg_restore -l answers "does the header describe some tables". Neither answers "is the data in here".
The only question worth asking is the one that requires touching the data: does the file decompress to a complete dump, and does the restore actually run.
This started as a comment on My Backup Exited 0 for Three Weeks and Nothing Paged Me, where the failure class is the same: a signal that reports the process worked, while the outcome quietly did not. That post covers the freshness side — how a backup can be stale while every timestamp looks recent. This one covers the completeness side.
Also published on my Hashnode blog: Three checks that look like they passed, but validated nothing
Top comments (2)
The pg_restore -l one is the one that got me. I'd assumed the custom format was the upgrade — a TOC listing all seven tables feels like proof, until you realise the TOC is written up front and the data phase is exactly what got cut. 163KB listing seven tables it never stored is a perfect summary of the whole post: the check answers a different question than the one you asked.
The gzip detail is the same trap in the other direction — gzip was never killed, so it wrote a perfect trailer around 2.9% of the data. The container is valid; the content just isn't there.
Your cheapest guard stuck with me: size against a rolling median. It costs nothing and it catches exactly this shape. My question is where you'd put it — in the backup script itself so a bad run fails loudly, or in monitoring so the script stays dumb and the alert carries the history? I'm torn: the script failing fast is safer, but a median check inside the script feels like one more thing that can silently break.
Monitoring, and I would not put the median in the script at all — but not for the reason you are torn about.
The reason is that the median is self-referential, and I can put numbers on it. Take thirty normal runs of 2.75 MB and one truncated run at 81 KB:
Good. Now feed it five consecutive bad nights before the good one:
The median drifts toward whatever it is fed. A rolling median computed from the backup's own output is the same structural mistake as putting the fingerprint next to the backup config: the thing being watched supplies the standard it is judged against. Once it is consistently broken, it is consistently fine.
That is why I would split it by what kind of failure each one is good at, rather than pick a side:
In the script: an absolute floor, declared by a human. Not a median — a constant. "This dump is never smaller than 400 KB." It cannot be poisoned by the script's own history because the script did not write it, it fails fast with a non-zero exit on the catastrophic case, and it costs one comparison. Cheap, unbreakable, and it catches exactly the 2.9% shape.
In monitoring: the rolling median. Its job is the failure the floor cannot see — slow shrinkage. Someone drops a table, or a partition stops being included, and the dump goes from 2.75 MB to 2.2 MB over six weeks. Never crosses the floor. Only visible as drift against history.
They are not two implementations of one check. They catch different things, and one of them needs history the other must not have.
On your worry that a check inside the script "feels like one more thing that can silently break" — you are right, and there is a structural asymmetry worth naming. A broken check inside the script fails silently by returning success. Nothing observes the checker. A broken check in monitoring fails as a missing metric, and a missing metric is itself alertable — you can page on
absent(). You can build the second kind of check to notice its own death; you cannot do that for the first.So: the script asserts what a human wrote down and cannot be talked out of. Monitoring asserts against history and is expected to notice when it stops reporting. Anything that needs to be right for years goes in the second bucket.