Demystifying Rsync

📃Cheatsheet

Sync one directory into another with compression:

rsync -az source/directory destination/directory

Sync over SSH:

rsync -az [e-mail]:path/to/file destination

or

rsync -az source/directory [e-mail]:destination/path

Sync does not delete existing files at destination. To make sure the source and destination folders look identical, use the --delete flag:

rsync -az --delete source/directory destination/directory

🔄 What is Rsync?

Rsync is this neat utility that commonly exists on Linux systems (and other Unix-likes). It's like a copy command but the destination (or source) can be remote.

It syncs files rather than copies them so if the destination file is already present, it doesn't get overwritten. This is great because you can sync large folders several times but only the missing files will get copied.

So while you might run something like cp -r ~/projects/dotfiles/.bashrc ~ to copy a .bashrc file from a folder to your home directory, rsync lets you do it in a remote fashion like so: rsync [e-mail]:path/to/file ~. Or the other way around rsync path/to/file [e-mail]:path/to/destination.

To me, what makes rsync complicated are the flags. It's always the flags that trip me up and push me to feel uncertain about the command.

🏳Copying with no flags

You lose a lot by not providing flags. Without flags, you can also sync one file at a time.

rsync source/file destination

You can also do remote copy:

rsync [e-mail]:path/to/file destination

Rsync uses ssh for remote connections so make sure you have access via ssh!

Copying the other way is the same deal:

rsync path/to/file [e-mail]:destination

🚩 Supercharging rsync with flags

rsync -flags source destination

There are several helpful flags and if you follow any guides online, you'll most likely come across these same exact ones.

Flags:

1. a - stands for archive. It is necessary for recursive copies (ie. copying whole directories)
2. z - the z flag enables compression for faster copying across networks
3. v - for verbose output

The archive option has always been confusing to me. Luckily, there's a great server fault post detailing the archive flag. It's a compilation of other flags that allows recursive syncing - so directory copy/sync support. And if you're copying locally, it preserves permissions and a bunch of other stuff.

The z flag enables compression via Zlib (at least that's what Wikipedia tells me!). In case you wanted to know, zlib uses the same compression technique as gzip which you can read about in my Demystifying TAR article.

The verbose output should display files copied file-by-file. A typical rsync command might look like this:

rsync -azv ./files-to-upload [e-mail]:~/uploads

🏳️‍🌈 ️‍More flags? (OPTIONAL)

Don't read this section of rsync is already giving you a headspin. But if you ever want to refer back to this article, you can check these flags out. I wouldn't use any of these by default you might come across them in tutorials/articles or you might want to use them yourself:

1. n - stands for dry run. This will check that you actually can transfer those files.
2. delete - delete files from destination that don't exist in source
3. P - The P flag is a combination of the --partial and --progress.

The n flag is handy because you can use it to make sure you're running your command correctly. Combined with a verbose output, you can see exactly what's going to happen without making it happen. I found this useful when preparing to upload a lot of files and not wanting to screw up where they upload or which files upload.

The delete flag is also super useful in that it doesn't just copy files to the destination, it ensures that files in the source directory don't exist in the destination. Super useful when syncing from a source that you consider the "source of truth" and want to have identical copies elsewhere...without having to delete everything and copy everything from scratch.

I don't often use the P flag but for many, it's the default. The P flag will do 2 very important things:

1. allow partial file upload + resume
2. display a progress bar

If you check the screenshot above, I ran that copy with a P flag. At the end, it'll let you know neat stuff about how much data you've transferred if you combine it with the v flag.

The wonderful part about partial file uploads is that if your connection fails, you can run the command again and it'll pick up where it left off with the partially uploaded files. Without the P flag, rsync would not be able to finish uploading the interrupted file.