It’s 2019, and ransomware has become a thing.
Systems that interact with the public, like companies, educational institutions, and public services, are most susceptible. While delivery methods for ransomware vary from the physical realm to communication via social sites and email, all methods only require one person to make one mistake in order for ransomware to proliferate.
Ransomware, as you may have heard, is a malicious program that encrypts your files, rendering them unreadable and useless to you. It can include instructions for paying a ransom, usually by sending cryptocurrency, in order to obtain the decryption key. Successful ransomware attacks typically exploit vital, time-sensitive systems. Victims like public services and medical facilities are more likely to have poor or zero recovery processes, leaving governments or insurance providers to reward attackers with ransom payments.
Individuals, especially less-than-tech-savvy ones, are no less at risk. Ransomware can occlude personal documents and family photos that may only exist on one machine.
Thankfully, a fairly low-tech solution exists for rendering ransomware inept: back up your data!
You could achieve this with a straightforward system like plugging in an external hard drive and dragging files over once a day, but this method has a few hurdles. Manually transferring files may be slow or incomplete, and besides, you’ll first have to remember to do it.
In my constant pursuit of automating all the things, there’s one tool I often return to for its simplicity and reliability: cron. Cron does one thing, and does it well: it runs commands on a schedule.
I first used it a few months shy of three years ago (Have I really been blogging that long?!) to create custom desktop notifications on Linux. Using the crontab configuration file, which you can edit by running crontab -e, you can specify a schedule for running any commands you like. Here’s what the scheduling syntax looks like, from the Wikipedia cron page:
# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of the month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of the week (0 - 6)
# │ │ │ │ │
# │ │ │ │ │
# │ │ │ │ │
# * * * * * command to execute
For example, a cron job that runs every day at 00:00 would look like:
0 0 * * *
To run a job every twelve hours, the syntax is:
0 */12 * * *
This great tool can help you wrap your head around the cron scheduling syntax.
What’s a scheduler have to do with backing up? By itself, not much. The simple beauty of cron is that it runs commands - any shell commands, and any scripts that you’d normally run on the command line. As you may have gleaned from my other posts, I’m of the strong opinion that you can do just about anything on the command line, including backing up your files. Options for storage in this area are plentiful, from near-to-free local and cloud options, as well as paid managed services too numerous to list. For CLI tooling, we have utilitarian classics like rsync, and CLI tools for specific cloud providers like AWS.
Backing up with rsync
The rsync utility is a classic choice, and can back up your files to an external hard drive or remote server while making intelligent determinations about which files to update. It uses file size and modification times to recognize file changes, and then only transfers changed files, saving time and bandwidth.
The rsync syntax can be a little nuanced; for example, a trailing forward slash will copy just the contents of the directory, instead of the directory itself. I found examples to be helpful in understanding the usage and syntax.
Here’s one for backing up a local directory to a local destination, such as an external hard drive:
rsync -a /home/user/directory /media/user/destination
The first argument is the source, and the second is the destination. Reversing these in the above example would copy files from the mounted drive to the local home directory.
The a flag for archive mode is one of rsync’s superpowers. Equivalent to flags -rlptgoD, it:
- Syncs files recursively through directories (
r); - Preserves symlinks (
l), permissions (p), modification times (t), groups (g), and owner (o); and - Copies device and special files (
D).
Here’s another example, this time for backing up the contents of a local directory to a directory on a remote server using SSH:
rsync -avze ssh /home/user/directory/ user@remote.host.net:home/user/directory
The v flag turns on verbose output, which is helpful if you like realtime feedback on which files are being transferred. During large transfers, however, it can tend to slow things down. The z flag can help with that, as it indicates that files should be compressed during transfer.
The e flag, followed by ssh, tells rsync to use SSH according to the destination instructions provided in the final argument.
Backing up with AWS CLI
Amazon Web Services offers a command line interface tool for doing just about everything with your AWS set up, including a straightforward s3 sync command for recursively copying new and updated files to your S3 storage buckets. As a storage method for back up data, S3 is a stable and inexpensive choice.
The syntax for interacting with directories is fairly straightforward, and you can directly indicate your S3 bucket as an S3Uri argument in the form of s3://mybucket/mykey. To back up a local directory to your S3 bucket, the command is:
aws s3 sync /home/user/directory s3://mybucket
Similar to rsync, reversing the source and destination would download files from the S3 bucket.
The sync command is intuitive by default. It will guess the mime type of uploaded files, as well as include files discovered by following symlinks. A variety of options exist to control these and other defaults, even including flags to specify the server-side encryption to be used.
Setting up your cronjob back up
You can edit your machine’s cron file by running:
crontab -e
Intuitive as it may be, it’s worth mentioning that your back up commands will only run when your computer is turned on and the cron daemon is running. With this in mind, choose a schedule for your cronjob that aligns with times when your machine is powered on, and maybe not overloaded with other work.
To back up to an S3 bucket every day at 8AM, for example, you’d put a line in your crontab that looks like:
0 8 * * * aws s3 sync /home/user/directory s3://mybucket
If you’re curious whether your cron job is currently running, find the PID of cron with:
pstree -ap | grep cron
Then run pstree -ap <PID>.
This rabbit hole goes deeper; a quick search can reveal different ways of organizing and scheduling cronjobs, or help you find different utilities to run cronjobs when your computer is asleep. However, this basic set up is all you really need to create a reliable, automatic back up system.
Don’t feed the trolls
Humans are fallible; that’s why cyberattacks work. The success of a ransomware attack depends on the victim having no choice but to pay up in order to return to business as usual. A highly accessible recent back up undermines attackers who depend on us being unprepared. By blowing away a system and restoring from yesterday’s back up, we may lose a day of progress; ransomers, however, gain nothing at all.
For further resources on ransomware defense for users and organizations, check out CISA’s advice on ransomware.
Oldest comments (38)
Great article 👏.
I guess I should start backing up my data more frequently.
If you're Windows or Mac, it's hard to beat Backblaze. Continuous backups with near-zero setup. Even better, you pay per backed-up system rather than by-the-byte. And, unlike some other cloud backup services, you can fine-tune how much bandwidth a given system's backups are allowed to consume.
Thanks a lot for the suggestion. I'm a Linux user though 😬. Also I would like to setup my own backup server at home instead of using a service.
Sounds like an
rsyncjob to me! Good luck with the project! Maybe you can write about how it went when you’re up and running!I used to do everything (quite literally) "in house". Then my electric bills got to be stoopid-big. Switched to using cloud-services. Also means I no longer worry about power outages, basement getting flooded (and my rack submerged), remote accessibility (and protecting the home network from getting owned via an exposed service/port), etc.
From a TCO price-point and reliability standpoint, using BackBlaze's B2 service with something like Restic is hard to beat for keeping your Linux system backed up. About the only thing that might be more cost effective (haven't looked, recently, would be one of the CSP's archival-oriented service – like AWS's Glacier offering).
On the cloud side of things, I'm rather fond of rclone. As their own tagline says, it's quite literally rsync for cloud storage, and it supports a huge number of different storage providers (including all the big ones and quite a few smaller ones too).
Oof... Generally would not recommend using S3's sync subcommand. While, yeah, it will get all your files there, it gets them there at the cost of filesystem attributes (ownerships, xattrs, ACLs, SELinux labels, etc.). If you care about those types of things, then you want to encapsulate your filesystem data in something that's filesystem aware. Fortunately, the AWS S3 CLI does allow you to do piped inputs ...meaning you can use your filesystem's native incremental backup utility,
dumpto store incremental, attribute-aware backups of your data.Only down side is, if you don't know (roughly) how big your dump is going to be, the S3 CLI will tend to single-stream your data. For a day's worth of activity, this is probably OK, but if you're doing a full backup, it can make things painfully slow (particularly if backing up over links that have session-limits).
Since your article points out people with less technical awareness are at more risk (doesn't mean the computer literate are immune though 😁) - FreeFileSync is a UI based tool everyone could use to backup data from their devices to both local storage & cloud.
It's cross platform, open source, easily configurable (like an intelligent rsync with a front-end) & very reliable.
Here is a place to begin with it.
Great article. Wanted to add that backing up is just one part of the story. Periodic restores must be tried out to ensure that everything works when restored.
Also, we should keep separate weekly and monthly snapshots in case we are slow to detect the ransomware as the cron can overwrite the earlier good backup with the backup tainted by ransomware.
Great points! Thank you!
I got caught up in my cron line and neglected to round out the practice. 😅
The next step in this process is to back up multiple versions of your files using incremental snapshots, which is also pretty easy with rsync. The key is using
cp -al(copy and make hard links) to prepare the latest version of the backup files, and then rsync into that directory. This way, files which have not been changed will only be stored once on the disk.Once that's working, you can add the
--deleteoption to rsync (removing individual files that have been deleted). You can also set up a process to remove older snapshots as needed. This is the kind of scheme I usually set up for my clients.More info about how and why this works: mikerubel.org/computers/rsync_snap...
Great info on rsync! Thanks for that and the link, James!
I have to disagree with the --delete option in this particular use case.
It defeats the purpose of the article.
Ransomware will delete your files replacing them with their crypted version.
Then your backup process with --delete, will ... delete your sane files ...
I don't think it is what you want for your backup process to prevent ransomware ...
Same stuff for you snapshots. Sane backup could vanish ..
In ransomware case, a good indicator can be the percentage variation of changes in files.
I think you misunderstood my comment: you only add
--deleteafter you have snapshots working properly. Then the files only disappear from the latest backup, but they are still present in all previous snapshots.If you set this up correctly - and you understand what it is doing, which is always important - then it is a good system with no risk for data loss.
what happens when I am infected? Wouldn't rsync or equivalent simply sync the encrypted files to the cloud service?
Yup. Someone else pointed out that having separated, incremental backups gives you the ability to revert to an older, non-compromised one, should that happen. (Like a Git commit.)
If the file is encrypted by ransomware, wouldn't this command back-up an encrypted file as well overwriting the good one?
A very good point and depending on what and how your method of backing up works, a very general method would be to have multiple backups. For example, traditional tape backup solutions followed daily/weekly/monthly (incremental/full) etc. routines, so you can restore from a last-known good backup i.e. before the file(s) were affected.
I'm a bit out of the loop on what's what in backup technology, but various tape/disk-based solutions etc. offer a myriad of options that would allow for point-in-time restores.
Great article! Very pedagogic! With that said, this approach could very well make you copy the ransomware encryption agent as well, spreading it to yet another host.
A safer approach would be to put the data you’re syncing in some kind of archive or similar before transferring it to make sure that the risk of accidentally retriggering the agent is as low as possible.
Or simply use any filesystem with automatic history?
If your backup versions are stored locally, they’re also susceptible to a ransomware lockdown. They’d need to be backed up in a similar fashion.
True, although even locally you can mitigate most of the effect by giving access to each program only to a pseudo-filesystem; your system will be as secure as your administrator access in that case.
Mixing both solution would probably work well, as you can transfer incremental backups from a filesystem history to a remote, then you only have to protect your remotes.
How does this protect you in any fashion if the entire filesystem is encrypted?
For this I'm not entirely sure, but could a ransomware totally encrypt a ZFS volume? That would mean elevating privileges up to the filesystem driver, which may not be in user-space
ZFS with snapshots on the targeted machine is a great mitigation but not a silver bullet. Snapshots can expire or be overwritten by new encrypted data until good data is lost (depending on the configuration). Or the ransomware attack could be block level, which is hitting a lot of people lately. Or the machine could explode :) I think snapshots should be considered as a way to restore point-in-time data locally, not as a backup per se.