Mustafa ERBAY

Posted on Aug 25 Originally published at mustafaerbay.com.tr

Disaster Recovery with ZFS Snapshots and send/receive

#zfs #snapshot #replication #storage

The real quality of your backup strategy shows up not on the night you take the backup, but on the night you have to come back from one. What you hold in your hands that night matters — but so does how fast you can bring that data back up, and that is a design decision. Most teams made it years ago without noticing, by saying "this will do for now."

You Don't Recover From a Disaster by Copying Files

Back in June I had to rebuild my virtualization environment from scratch. Rather than trying to salvage a 12 TB cluster disk sitting on a corrupted filesystem, I reconfigured the SAN from the ground up and moved the virtual machines to the new storage one by one. The method was primitive but dependable: shut the machine down cleanly, copy the disk file, recreate it on the target.

When the mail server's turn came, its disk was 306 GB. The copy took 56 minutes and 26 seconds, averaging 97 MB per second. Counting the shutdown before and the boot after, the server was down for 64 minutes in total. It was the middle of the night, nobody noticed, and incoming mail waited safely thanks to SMTP retries. So it "went well." But I couldn't stop thinking about one thing: if I had copied that same server to a remote site every night with this method, I would have been pushing 306 GB across the network for 56 minutes every single night — with most of those 300 GB identical to yesterday's.

That environment wasn't ZFS; that is exactly why I'm telling the story. You see what block-level incremental replication buys you most clearly on a night when you don't have it. ZFS's snapshot + send/receive trio already knows, at the filesystem level, what changed between yesterday and today, so from the second transfer onward it moves only the changed blocks. I've already discussed whether to pick Ceph or ZFS replication for a small cluster elsewhere; here I'll assume that decision is made and go one step further, into where the replication pipeline actually breaks in production.

What a Snapshot Gives You, and What It Doesn't

A ZFS snapshot is a read-only image of a dataset at a specific point in time. Its critical property is atomicity: the OpenZFS documentation states that a snapshot is "a consistent image of a dataset at a specific point in time" including "all modifications to the dataset made by system calls that have successfully completed before that point in time," and that recursive snapshots taken with -r "are all created at the same time." That single sentence is worth gold for setups that keep database and application data on separate datasets — you pin three datasets to the same instant with one zfs snapshot -r instead of three separate commands.

What it does not give you is this: a snapshot is not a backup. It lives in the same pool, on the same disks. If the pool dies, the snapshot dies with it. If ransomware has administrative rights, it deletes the snapshots too. What turns a snapshot into a backup is moving it to another machine — and that's where the real engineering starts.

Building the Chain: Full Send and Incrementals

The model is simple. You send one full stream, then incrementals forever:

# On the source: recursive snapshot, pinned to one point in time
zfs snapshot -r tank/srv@2026-08-25T0300

# First send (full): all child datasets, properties, snapshots
zfs send -R tank/srv@2026-08-25T0300 | ssh dr-host zfs receive -s -F backup/srv

# The next day: only the delta between two snapshots
zfs snapshot -r tank/srv@2026-08-26T0300
zfs send -R -i @2026-08-25T0300 tank/srv@2026-08-26T0300 \
  | ssh dr-host zfs receive -s backup/srv

Every flag here is a decision. -R (--replicate) sends the named dataset and all its descendants as a package; when received, properties, snapshots, descendant filesystems and clones are preserved. If you want to leave part of the tree out, -X excludes it — dropping a scratch dataset from the DR path saves both bandwidth and noise.

The difference between -i and -I is the distinction people miss most often. -i sends the delta between two snapshots as a single incremental stream; the intermediate snapshots never materialize on the target. -I packages all the intermediate snapshots as well. If you take hourly snapshots but replicate once a day and you also want hourly restore points on the target, -I is what you're looking for. Decide this up front; discovering your target's restore granularity six months later is an unpleasant surprise.

On the compression side, -c (--compressed) sends blocks that are already compressed on disk without decompressing them, and -L (--large-block) lets blocks larger than 128 KiB travel whole. Use them together — the man page warns that if large_blocks is enabled on the sending system but -L is not supplied alongside -c, the data is decompressed before sending so it can be split into smaller blocks, and the saving you expected from -c evaporates. Both require the corresponding pool features on the receiving end. If you want to know how much data will flow before you commit, do a dry run with -n -P; it prints the size in machine-parsable form, which is handy when planning a maintenance window.

Where the Chain Breaks: the Target's "Most Recent Snapshot" Rule

Incremental replication rests on one fragile assumption, and the zfs receive documentation spells it out: if an incremental stream is received, the destination filesystem must already exist and its most recent snapshot must match the incremental stream's source.

In practice that means: if anyone touches the dataset on the target, the chain breaks. Writing a file — even an operation that updates access times on a mounted directory — is enough to move the target away from its last snapshot. From that moment on, the next incremental receive is rejected.

One distinction speeds up diagnosis: ZFS produces two different errors here. If the common base is gone, most recent snapshot of <ds> does not match incremental source; if the target was written to, destination <ds> has been modified since most recent snapshot. The second is fixed by a rollback, the first is not.

The recovery path for the second error is -F: zfs receive -F forces a rollback of the filesystem to its most recent snapshot before receiving. But before you make -F a reflex, learn its second function — when receiving an incremental replication stream, it also destroys snapshots and filesystems that do not exist on the sending side. There's a subtlety here: when a package produced with -R is received, snapshots that don't exist on the sender are destroyed anyway, without -F; what -F adds is the rollback plus destroying filesystems. So -F means more than "clean up the mess on the target" — it means "make the target a mirror of the source," and a dataset deleted on the source by mistake travels faithfully to the target.

That's why I prefer to keep readonly=on on target datasets; the documentation defines this property as equivalent to the ro/rw mount options. Setting it at receive time with zfs receive -o readonly=on is sturdier than a manual zfs set: the man page says an -o value applies "throughout the entire subtree of replicated datasets." Set it by hand and the new child datasets that an -R stream later creates arrive with the source's readonly=off, quietly puncturing your safeguard down the tree.

When you do need to look at the data on the target, the right way is to clone the snapshot: zfs clone gives you a writable copy, you destroy it when you're done, and the replication chain never notices.

Protecting the Chain While Pruning: Bookmarks and Holds

Now for the classic real-world scenario. Your retention is 7 days on the source and 90 days on the target. Month-end arrives, space gets tight on the source, and the cleanup script deletes old snapshots — and the replication pipeline dies quietly, because the snapshot you deleted was the common base between the two sides. There is no reference left for an incremental send; your only option is a full stream from scratch.

ZFS offers two separate mechanisms for this, and both are underused.

The first is the bookmark. Per the documentation, a bookmark marks the point in time when the snapshot was created and can be used as the incremental source for a zfs send. Its critical properties: bookmarks "consume no additional space within the pool" and "will survive if the snapshot itself is destroyed." So if you leave a bookmark before deleting a snapshot, you release the blocks the snapshot was holding while keeping your incremental reference:

zfs bookmark -r tank/srv@2026-08-25T0300 tank/srv#last-replicated
zfs destroy tank/srv@2026-08-25T0300
# The incremental send can now start from the bookmark (single dataset, no -R)
zfs send -i tank/srv#last-replicated tank/srv@2026-08-26T0300 | ...

Here a detail of the man page that's easy to miss bears directly on the pipeline above: zfs send defines two separate usage forms, and a bookmark is accepted as an incremental source only in the form without -R; the -R replication-package form takes [[-I | -i] snapshot] — a snapshot, not a bookmark. So if you plan to feed the -R -i pipeline we built earlier from a bookmark, you'll need to move to per-dataset -i sends. Better to know this at design time than at month-end.

The second is the hold. zfs hold adds a named reference to a snapshot, and the documentation is blunt: attempts to destroy a held snapshot with zfs destroy return EBUSY. This is the most direct way to stop your cleanup script from removing the common base — instead of hoping you'll fix the script, you make the filesystem itself say "not this one." It takes -r for recursion, and zfs send -h includes holds in the stream so they're applied on the receiving system too.

The two mechanisms don't cost the same, and skipping that difference gets expensive. A bookmark really is free; a hold, by preventing the snapshot's destruction, also keeps the blocks that snapshot holds. The cost of a hold is the used value of the snapshot you locked — put an indefinite hold on the common base and you've made the very problem this section opened with worse.

So a hold should be a rotating flag, not a lock: after each successful replication, zfs hold the new common base, then zfs release the old one. I like using both together; the payoff is "not having to send a full stream at 3 a.m."

A Broken Link and an Encrypted Site

Losing the connection halfway through a 2 TB initial full stream over a WAN is the classic morale-killer of DR projects. zfs receive -s exists for exactly this: if the receive is interrupted, the partially received state is saved rather than deleted. To use the flag, the pool must have the extensible_dataset feature enabled. The saved token lives in the target dataset's receive_resume_token property, and zfs send -t <token> picks up where you left off. If you'd rather give up, zfs receive -A aborts the partial state.

Wiring this into automation is a couple of lines — read the token from the target and feed it back into the send:

tok=$(ssh dr-host zfs get -H -o value receive_resume_token backup/srv || true)
[ -n "$tok" ] && [ "$tok" != "-" ] && zfs send -t "$tok" | ssh dr-host zfs receive -s backup/srv

As for encryption, ZFS native encryption does something genuinely elegant for DR. zfs send -w (--raw) sends data exactly as it exists on disk; in the documentation's words, this "allows backups to be taken even if encryption keys are not currently loaded," and the resulting backup "may then be received on an untrusted machine since that machine will not have the encryption keys to read the protected data or alter it without being detected." Your colocation box at the remote site stores the data, refreshes it, and scrubs it — the docs note that datasets can be scrubbed, resilvered, renamed and deleted without the encryption keys being loaded — but it can never see the contents. With one caveat: native encryption does not encrypt metadata about pool structure — the documentation's list includes dataset and snapshot names, dataset hierarchy, properties and file sizes. The other side can't read your files, but it can see that a dataset named customer-x-accounting exists and how fast it grows. Name things accordingly.

There's a price, and teams who don't know it hit a wall months later: you cannot mix raw and non-raw receives. The documentation is explicit — any raw incremental receive attempted after a non-raw receive will fail. The reason is technical and rather beautiful: a raw send carries the initialization vector set (IV set) across as-is, while a non-raw send decrypts on the source and re-encrypts on the target, producing a different IV set. Receiving a non-raw stream on top of a raw-received filesystem is allowed, but it replaces the IV set; the next raw incremental then fails with "IV set guid mismatch." Recovery means rolling the target back to the most recent snapshot that was received raw, or sending a fresh full raw stream. So the rule is clear: with encrypted datasets, go raw all the way or not at all. One small note — when a raw stream is received, the target's keylocation property defaults to prompt.

A DR copy whose key never travels to the remote site, and a password-protected repository, are both defensible designs. I've written before about repository design for encrypted backups with restic; what's different in ZFS is that encryption lives in the filesystem itself, and replication can carry it without breaking it.

After the Disaster: Reversing Direction

Everyone thinks in one direction while building a replication pipeline. But if a disaster actually happens, the story doesn't end there: you promote the DR site to production, new data accumulates there for hours or days, then the primary system is repaired and you need that data back. This second journey is usually more painful, precisely because nobody planned it.

The good news is that the mechanism is the same; only the roles swap. You make the DR dataset writable (readonly=off) and put it into production; the common base is the last snapshot both sides know about. Once repairs are done, you take a new snapshot on the DR side and send the incremental in the opposite direction:

# On the DR site (now the source)
zfs snapshot -r backup/srv@failback-2026-08-30
zfs send -R -i @2026-08-25T0300 backup/srv@failback-2026-08-30 \
  | ssh primary-host zfs receive -F tank/srv

The thing to watch is whether the original system moved on by itself in the meantime. If nothing was written to the old primary after the disaster, its most recent snapshot is still the common base and the incremental lands cleanly. But if the machine came back up and kept serving, both sides have diverged from the same base; in that case rolling back to the common point with -F means permanently discarding the difference. So make it a habit to take the old primary out of service before a failback, and to keep a snapshot of it aside if you can.

My personal rule is that the failback procedure gets written the same day the replication is built, and rehearsed at least once within the same year. An unwritten failback plan means improvising under double pressure on the worst day of the year.

Delegation, Drills, Monitoring

Most ZFS replication write-ups end at ssh root@dr-host. But zfs allow lets you delegate exactly the operations you need to an ordinary user: send, receive, snapshot, hold, release, bookmark and destroy can be granted individually. There's even a restricted variant, receive:append, for a limited receive ability that cannot do receive -F.

On the encryption side, 2.4 brought two new permissions and the difference between them is critical. send:raw only allows sending raw replication streams, preventing encrypted datasets from being sent in decrypted form. send:encrypted is the stricter version: it forces raw sends for encrypted datasets and prevents decrypted datasets from being sent at all. If your pipeline mixes encrypted and unencrypted datasets, the permission you want is send:raw — grant send:encrypted and your unencrypted datasets simply never ship.

One caveat: the documentation says that on Linux, mount, unmount, mountpoint, canmount, rename and share cannot be delegated, because mount(8) restricts modifications of the global namespace to root. And the receive permission "must also have the mount and create ability." So a fully unprivileged receive is limited in practice on Linux; the realistic design is to run the receive through a narrowly scoped helper and leave everything else unprivileged.

There's a second design decision tied to delegation, and it bears directly on the ransomware claim from earlier. Every example here is push-based — the source connects to the target and sends the stream. If the source is compromised, the attacker also holds the SSH key to the target, and your DR copy dies the same night as the first one. A pull-based setup inverts that dependency: the trigger runs on the DR end, connects to the source with an account allowed only send and snapshot, never destroy, and retention lives where the source cannot reach it.

For scheduling I prefer systemd timers over cron: calendar expressions with OnCalendar, catch-up for missed runs with Persistent=true, output in the journal. If you'd rather not hand-write snapshot policy and replication, sanoid/syncoid is a widely used third-party option — the same commands still run underneath, and error messages come from that layer when something breaks.

That leaves the most-skipped job of all: proof. The existence of a dataset on the target does not mean you can come back from it. I've written before about the overlooked detail in disaster recovery testing, and the lesson applies here too: an untested recovery plan is a well-intentioned wish. The workable minimum is these three:

Monitor the age of the last received snapshot on the target and alert when it crosses a threshold. Replication stops silently; the most dangerous kind of failure is the one that makes no noise. I use the same principle in my own publishing pipeline — the alarm fires not when the machine stops, but when the output goes stale.
Once a month, clone the latest snapshot on the target and actually run what's inside. If it's a database, let it open; if it's a file server, mount it.
Run regular scrub on the target pool; there's no other way to notice bit rot.

Recipes That Have Aged Badly

Some ZFS replication examples on the internet still recommend zfs send -D (deduplicated streams). That's gone: the current man page says of -D that "deduplicated send is no longer supported. This flag is accepted for backwards compatibility, but a regular, non-deduplicated stream will be generated." If you have a deduplicated stream produced by older software, zstream redup converts it to a regular one.

The version picture moves too: the OpenZFS 2.4 series shipped in December 2025, and on 21 August 2026 releases 2.4.4, 2.3.9 and 2.2.11 all landed on the same day — three branches under maintenance at once. Knowing which branch your distribution sits on has a practical payoff: the send:raw and send:encrypted permissions I described above arrived with 2.4 and do not exist in the 2.3 man page. If your delegation design rests on them, check your version first.

Stream compatibility is asymmetric, and knowing that saves needless worry: the man page states the stream format is committed, so streams you produce today can be received on future versions of ZFS. The trouble runs the other way — if a stream uses large_blocks, embedded_data or zstd_compress, the receiving end must support that feature. Which is why leaving the DR end on an older release than the source is a time bomb that's hard to spot.

Setup Checklist

Take snapshots with -r so related datasets sit at the same point in time.
Keep readonly=on on the target; when you need to see the data, clone it and leave the dataset alone.
Use -F deliberately. It cleans the target, but it also mirrors deletions from the source.
Protect the common base with a rotating hold (hold the new base, zfs release the old one) and leave a bookmark before pruning. A hold costs space, a bookmark doesn't; once the chain breaks, the price is a full stream.
Receive with -s and wire receive_resume_token into your automation. WANs drop; the drop isn't the problem, starting over is.
Pick one path for encrypted datasets: either always -w, or never. Mixing ends in "IV set guid mismatch."
Use zfs allow instead of root, and on mixed pipelines enforce raw sends with send:raw (send:encrypted blocks unencrypted datasets entirely). Prefer a pull-based setup and never grant destroy on the source.
Monitor freshness, drill with clones, scrub the target. Without all three, what you have isn't a backup — it's the feeling of one.

Conclusion

Setting up ZFS replication is an afternoon's work; keeping it alive is an operational discipline. The real engineering is seeing in advance where the chain can break and putting a safety catch at each point: a rotating hold on the common base, a bookmark before pruning, a resume token for the broken link, a single consistent raw path for encrypted data.

Underneath all of it sits one idea, and that idea is far older than ZFS: recovery is measured not by the existence of your copy, but by proven ability to come back from it. What reassured me during that migration in June wasn't that the disk file existed somewhere; it was watching the server boot on the other side and start accepting mail again. Ask your own setup this question: when did you last actually open something from the copy on your target? If the answer is "we tested it when we set it up," your system has backups but no recovery.

DEV Community