DEV Community

John
John

Posted on

Duplicati Block Size, Remote Volume Size and Retention: How to Choose Them for Your Data Shape

#duplicati #backup #storage #homelab

Pick your Duplicati block size from the total size of the source data, not from the number of files in it, and pick your remote volume size from your upload link and your storage provider's request pricing. The 100 KB block size and 50 MB remote volume defaults are sized for a laptop with a few hundred gigabytes, and they are the single most common reason a large job ends up with a multi gigabyte local database and a database recreate that runs for hours. Block size is fixed at job creation and cannot be changed later without starting a new backup chain, so this is the one decision you get exactly one chance at. Retention is the setting you can change freely at any time, which is why it should be the last thing you tune, not the first.

TL;DR by data shape:

  • Laptop and document sets (Maya, 80 GB of text, code and PDFs): keep the 100 KB default block size and the 50 MB remote volume, because the block count stays low enough that the local database and the recreate never become a problem.
  • Millions of small files (a Nextcloud data directory, 600 GB across roughly 4 million files): raise the block size to 1 MB and the remote volume to 200 MB, because per file overhead and block count, not raw bytes, are what inflate the database here.
  • Photo and video libraries (a 2 TB Immich library of HEIC, JPEG and MP4): use 1 MB blocks and 200 MB to 250 MB volumes with a version count retention, because the files are already compressed, almost never change after import and deduplicate poorly.
  • Multi gigabyte VM images (six qcow2 files, 900 GB total, backed up nightly): use 1 MB or larger blocks and 500 MB volumes with a time based or smart retention, because the churn inside large files is what drives both upload volume and compaction traffic.
  • Slow or metered upstream (a 10 Mbit/s home link, or a provider that bills per request): shrink the remote volume back toward 50 MB for restart tolerance, or grow it to cut request counts, and accept that you cannot optimise for both at once.
  • Restore time is the priority (a business file server with an agreed recovery window): favour smaller remote volumes and fewer retained versions, and back up the local database itself, because a recreate from remote data is the slowest path back.

The central tradeoff is simple: larger blocks and larger remote volumes keep the local database small and the recreate survivable, while smaller blocks and smaller volumes save upload bandwidth on small changes and make single file restores cheaper.

Table of contents

What do Duplicati's block size, remote volume size and retention policy actually control?

These three settings sit at different layers of the same pipeline, and confusing them is why so many jobs end up misconfigured.

  • Block size (--blocksize, default 100 KB): the unit of deduplication. Duplicati splits every source file into blocks of this size, hashes each one with SHA-256, and stores the hash in the local database. A 600 GB source at 100 KB produces roughly 6 million blocks, and every one of them becomes a row you carry forever.
  • Remote volume size (--dblock-size, default 50 MB): the unit of upload and download. Blocks are packed into compressed, AES-256 encrypted .dblock.zip.aes files of this size before they leave the machine. It has no effect on deduplication quality, only on how many objects your provider stores and how much you must re-download to reach one block.
  • Retention policy (--retention-policy, --keep-versions, --keep-time): which restore points survive. It deletes .dlist files, the per version file lists, and marks the blocks they alone referenced as wasted space.
  • Compaction threshold (--threshold, default 25%): the link between the two. When wasted space in the remote volumes passes 25%, Duplicati downloads partially obsolete volumes, repacks the live blocks and uploads new ones.

Block size is permanent per job. Remote volume size applies to volumes created after you change it, so old and new sizes coexist. Retention and threshold can be edited on any run with no penalty. That ordering, permanent to free, is the order in which you should think about them.

Why block size is locked in on the first backup run

The block size is written into the job when the first backup completes, and every .dblock and .dindex file in the destination assumes it. Change it later and Duplicati will not re-chunk what is already stored, because a 100 KB block and a 1 MB block of the same file share no hash and can never deduplicate against each other.

  • What actually happens if you edit it: newer versions of Duplicati refuse the change on an existing job, and older ones let you save a value that is silently ignored for the existing chain. Either way you do not get the setting you asked for.
  • The only real fix: create a new job pointing at an empty destination folder, run a full initial backup, then retire the old job once you are satisfied with a test restore from it.
  • What that costs you: a second full upload of the entire source, a second full copy of storage held during the overlap, and the wall clock time of your slowest link. On a 20 Mbit/s upstream, 1 TB takes over four days of continuous transfer.
  • Where the decision lives: on the machine running Duplicati, since the local job database and the --blocksize value belong to that installation, not to the destination.

That last point is why the hosting choice matters before the first run and not after. A self-managed VPS, a home NAS, an always on mini PC and Yundera are all reasonable places to run it. Yundera is a managed Personal Cloud Server, built on CasaOS, that runs self-hosted apps as Docker containers on a server dedicated to the user. Pick the host, then set the block size once.

How many blocks can the local database carry before it starts to hurt?

Block count, not gigabytes, is what the SQLite database tracks. Every block contributes a hash row, and every version contributes references to those rows. The arithmetic is fixed: source size divided by block size.

Source size and block size Approximate block count What it feels like in practice
100 GB at 100 KB default 1.0 million Defaults are fine, recreate finishes in minutes
500 GB at 100 KB default 5.2 million Backups still work, recreate becomes an evening job
900 GB at 100 KB default 9.4 million Compaction and deletion queries slow noticeably
2 TB at 100 KB default 21 million The common failure case, recreate can run for many hours
2 TB at 1 MB 2.1 million Same data, one tenth the rows, back in comfortable territory

The practical target most Duplicati operators settle on is keeping a job under roughly 5 million blocks, and treating anything past 10 million as a warning sign. That is a heuristic, not an enforced limit. Nothing breaks at a threshold. What happens instead is that every operation touching the block table gets slower in proportion: the delete step after a retention rule, the compaction pass, the verification query at the end of a run, and above all the recreate.

Two extra multipliers catch people out. Retained versions add rows for changed blocks, so a job holding 60 versions carries far more than a job holding 10 of the same source. And a single job covering several unrelated directories pools all of that into one .sqlite file. Splitting one 2 TB job into three smaller jobs gives you three databases that each fail and rebuild independently.

Millions of small files: which block size and remote volume size to choose

This is the shape where raising the block size helps least and still needs doing. A 20 KB file is one block whether your block size is 100 KB or 1 MB, so a Nextcloud data directory of 4 million files across 600 GB does not shed 90% of its rows the way a VM job does. The database cost here comes from per file bookkeeping: each file carries an entry plus a separate metadata block for its permissions and timestamps.

  • Set --blocksize to 1 MB anyway: the large minority of files in a mixed user directory, the videos, the archives, the mail stores, is where the block count actually accumulates. Cutting those by a factor of ten is worth it, and it costs nothing on the files that were already single block.
  • Set --dblock-size to 200 MB: at 50 MB, 600 GB of source becomes over 12,000 remote objects before any versioning. At 200 MB it is closer to 3,000, which cuts list and request operations on providers that bill per request.
  • Filter aggressively before tuning anything: thumbnail caches, .thumbnails, preview directories, node_modules and application temp folders often account for a third of the file count and none of the restore value. Exclusion filters remove rows permanently, which no block size can.
  • Split the job by directory: one job per top level user folder gives you separate .sqlite databases, separate schedules and separate recreates.
  • Leave --small-file-max-count at 20: raising it makes compaction merge more undersized volumes per run, which adds download traffic on a job that already has plenty to do.

What should you change for a photo and video library?

A photo library is write once data made of already compressed bytes, and both facts change the tuning. HEIC, JPEG, MP4 and MOV files are entropy dense, so deduplication finds almost nothing and the zip step spends CPU to save single digit percentages.

  • Use 1 MB blocks and 200 MB to 250 MB remote volumes: a 2 TB library at the 100 KB default generates tens of millions of blocks for data that will never deduplicate against itself, which is pure database cost with no return.
  • Drop --zip-compression-level from the default 6 to 0 or 1: on a directory of JPEG and MP4 you are paying full compression CPU on every upload for a ratio close to 1.0. Setting it to 0 stores the blocks and lets the AES-256 encryption still apply normally.
  • Exclude the regenerable directories: in an Immich deployment, thumbs and encoded-video are derived from the originals in upload and can be rebuilt by the application. In a Photoprism setup, the sidecar and cache folders are the equivalent. These often hold 15% to 30% of the total bytes.
  • Keep few versions, not many: photos are added, not edited. A --keep-versions of 3 or 5 covers accidental deletion and a bad import, and holding 30 versions of a library that never changes retains almost nothing new while adding .dlist files to every operation.
  • Expect compaction to stay idle: because old blocks are rarely superseded, wasted space climbs slowly and the 25% threshold may never trigger for years. That is the desired outcome here, not a sign something is broken.

A handful of multi gigabyte VM images: the settings that matter most

Large images invert the problem. File count is trivial, block count is enormous, and every nightly run rewrites regions scattered through a file that never changes size. Block size here is a direct bandwidth decision.

Block size on 900 GB of qcow2 Blocks in the job Upload when 1 GB is rewritten in scattered 100 KB regions, worst case
100 KB default 9.4 million About 1 GB, finest possible granularity
500 KB 1.9 million Up to 5 GB, five times the changed bytes
1 MB 940,000 Up to 10 GB
2 MB 470,000 Up to 20 GB, database is smallest

Real guest workloads cluster their writes, so the true figure sits well below those worst cases. Still, the shape is clear: past 1 MB you buy database comfort with upload volume, every single night.

  • Start at 1 MB and only go higher if recreate time is your actual pain: 940,000 blocks is comfortable, and going to 2 MB halves rows you were already happy with while doubling the worst case transfer.
  • Set --dblock-size to 500 MB: with images this size the object count matters more than restart granularity, and a failed upload retries one volume, not the job.
  • Snapshot before reading: set --snapshot-policy=required so Duplicati uses VSS on Windows or LVM on Linux. Backing up a running qcow2 file without a snapshot produces a crash consistent image at best.
  • Exclude swap and scratch disks: a 16 GB swap file changes completely every day and restores no value.

How big should the remote volume size be for your upload link and your provider?

Remote volume size is the one setting you can revise later, so tune it to the link and the bill rather than to the data shape. Volumes created after the change use the new size, and old ones keep theirs until compaction repacks them.

  • Match the volume to a tolerable retry: an interrupted upload restarts that whole volume, and Duplicati retries 5 times with a 10 second delay by default. On a 10 Mbit/s upstream, roughly 1.25 MB/s, a 50 MB volume takes about 40 seconds and a 500 MB volume takes about 6.7 minutes. On a flaky link, the shorter unit wins.
  • Count objects when your provider bills per request: Backblaze B2 class C transactions and S3 PUT and LIST calls are priced per operation, not per byte. Moving from 50 MB to 500 MB cuts object count by a factor of ten for the same data.
  • Budget temporary disk space: with --asynchronous-upload-limit at its default of 4, Duplicati can hold five volumes on local disk at once. At 500 MB that is 2.5 GB in the temp directory, which matters on a small VPS root filesystem.
  • Remember what a single file restore costs: to recover one 100 KB block, Duplicati downloads the entire volume containing it. Large volumes make targeted restores expensive in egress and in time.
  • Keep block size at least 100 times smaller than volume size: 1 MB blocks in 200 MB volumes is a healthy ratio. 1 MB blocks in 10 MB volumes wastes the packing entirely.

What does the local SQLite database cost you in disk, RAM and backup time?

Each job keeps one database, named with a random string such as KJHGFDSAQW.sqlite, in the Duplicati config directory: %LOCALAPPDATA%\Duplicati on Windows, or the mapped /config volume in a container deployment. You can relocate it with --dbpath.

  • Disk grows with blocks multiplied by versions: every block row stores a base64 SHA-256 hash of 44 characters plus its size and volume reference, so the floor is tens of bytes per block before indexes. A job at 21 million blocks lands in a different order of magnitude to one at 940,000, which is the whole argument for block size in one line.
  • Deletions do not shrink the file: retention removes rows, but SQLite keeps the freed pages. Run a vacuum, or enable --auto-vacuum with --auto-vacuum-interval, otherwise the file stays at its historical high water mark forever.
  • RAM matters as page cache, not as a fixed requirement: if the database and its indexes fit comfortably in free memory, lookups stay in cache. If they do not, every hash check becomes a random read, and that is where a 2 GB container memory limit quietly turns a 40 minute backup into a multi hour one.
  • Storage medium is the biggest single lever: the workload is random reads and writes against indexes. Put the database on SSD or NVMe. Never put it on a spinning disk you also back up, and never on an SMB or NFS share, where SQLite locking behaviour causes corruption.
  • Temp space is separate: journal files and the sorting done during compaction land in the temp directory, not next to the database.

Keep all, keep versions, keep time or smart retention: which one fits your data shape?

Retention is the free setting. Change it whenever you like, and the next run applies it. Duplicati never deletes the most recent backup, whatever you configure.

  • Keep all backups: the right answer only for small, slow changing sources, such as a 5 GB configuration and document set. On anything with daily churn it means .dlist files accumulate forever, which is exactly what lengthens the first stage of a recreate.
  • Keep a specific number of backups (--keep-versions): predictable and shallow. A photo library at --keep-versions=5 holds five restore points regardless of when they were made, which suits data that is added rather than edited. The weakness is a burst: five backups in one busy afternoon give you five restore points covering four hours.
  • Delete backups older than a time (--keep-time): a straight window, for example --keep-time=1Y on a document set. Good when the requirement is stated in months, poor when your schedule is irregular, since a quiet quarter can leave the window nearly empty.
  • Smart backup retention: the built in preset, equivalent to --retention-policy=1W:1D,4W:1W,12M:1M. It keeps one backup per day for a week, one per week for a month, one per month for a year. This is the sane default for nightly VM images and for mixed user data.
  • A custom policy string: the same syntax, tuned. 7D:1D,4W:1W,12M:1M,10Y:1Y extends the tail to a decade at one point per year, and U in place of a duration means unlimited. Write the shape you would actually restore from, not the shape that feels generous.

How much post backup verification should you actually pay for?

After every run, Duplicati downloads a sample from the destination and checks it. The default --backup-test-samples=1 means one volume set per run: one .dblock, its .dindex, and a .dlist. That default is cheap on 50 MB volumes and not on 500 MB ones.

  • Do the egress arithmetic before raising it: at 500 MB volumes and one sample per nightly run, verification alone pulls roughly 15 GB per month back down from the provider. Set --backup-test-samples=5 and it becomes about 75 GB per month, billed as egress on most object stores.
  • Prefer --backup-test-percentage on large sets: sampling a fixed 1% of volumes scales with the backup instead of staying flat, which is what you want on a job that grows from 200 GB to 2 TB.
  • Know what the default sample proves: it verifies the file is present, the right size and hashes correctly. Add --full-remote-verification to also decrypt and inspect the contents, at the cost of CPU on every checked volume.
  • Run a full test on a schedule, not on every backup: duplicati-cli test <destination-url> all --full-remote-verification checks the entire set. Monthly or quarterly is a reasonable cadence, and it is the only thing that catches silent rot in volumes the sampler never picked.
  • Turn on --upload-verification-file: it writes duplicati-verification.json beside the backup, which the bundled DuplicatiVerify script can check at the destination without downloading anything.
  • Never set --no-backend-verification to save money: you are then trusting a destination you have not read from since the day you wrote to it.

Top comments (0)