Why we built Yaesm, and how I use it for backups

In this post, I'll explain why Connor Gallivan and I built a backup tool called Yaesm, what I learned during development, and how I use it for my backups. This blog is not additional documentation about how to use Yaesm since we already have a comprehensive manual here.

For a quick overview, see the TL;DR.

TL;DR

Yaesm is a configurable backup tool that connects tools such as Btrfs, ZFS, rsync, tar, GnuPG, and Zstandard into custom backup pipelines defined in YAML. Backups can run on demand or on a schedule.

Yaesm is available as a pre-release, and we're looking for feedback and testers. You can find the code, packages, and documentation on its GitHub page.

Why build Yaesm?

A few years ago, I learned about the Btrfs filesystem for Linux. I was amazed by its snapshotting capabilities. I set up all of my systems to use Btrfs because I thought it was such a cool technology. After getting everything set up, I needed to choose a Btrfs snapshot manager. Two options I considered were btrbk and Snapper, so I tried them both.

First, I tried Snapper, but at the time it did not support remote backups with Btrfs send/receive. This was a blocker for me because I wanted to back up to my home server.

Next, I tried btrbk, which had all the features I needed. When I went to configure it, however, I found its configuration semantics confusing. I was still new to Linux and couldn't figure out how to create the btrbk configuration I wanted.

I happened to be looking to start a new project at the time, so I decided to make my own Btrfs snapshot manager. I then spent the next two years developing a Btrfs snapshot manager called Yabsm, which is now deprecated. I wrote a blog post about Yabsm here.

Yabsm was decent and worked pretty well, but had some fundamental architectural problems. Its daemon forked itself into the background, I invented a custom configuration language for it, scheduling was limited to predefined categories, and other poor design choices made the code harder to maintain and extend.

In 2024, I started attending the University of Massachusetts Lowell and joined the open-source club. The club was looking for a new project, so we decided it would be fun to rewrite Yabsm with a more flexible architecture that supported backup tools beyond Btrfs. Since then, Connor and I have been developing Yaesm.

Developing Yaesm

We wrote Yaesm in Python and spent a lot of time revising its design as we figured out how the pieces should fit together. We settled on a common backup model with interchangeable backends for tools such as Btrfs and rsync. Each backend handled the details of creating backups, while the rest of the program shared the configuration, scheduling, and retention logic. Getting those boundaries right took several rounds of changes.

Configuration was a major part of the work as we learned that properly validating configuration is very challenging. We used YAML for the configuration format and Voluptuous to define valid settings and check their values. There were plenty of details to consider, from required options to the settings each backend supported. We put a lot of effort into catching configuration mistakes and giving users useful error messages before they tried to run a backup.

We also spent a lot of time writing our own scheduler. Getting the timing right and handling jobs running at the same time turned out to be a much harder problem than we expected, so we switched to APScheduler. It handles details such as parsing cron expressions, calculating the next run time, running backups concurrently, handling missed runs, and logging job execution and failures.

Testing was perhaps the most substantial part of development. We built a sophisticated test system using pytest and Vagrant that ran in a virtual machine, created real filesystems, and ran simulated backups locally and over SSH. This let us test the actual tools and filesystem operations in an isolated environment. We achieved very high test coverage, including configuration validation, scheduling, backup execution, and error handling, which helped us catch problems as we changed the design.

One feature we really wanted was encrypted backups. Our existing design put each backup method in its own backend, which made it awkward to add shared steps such as encryption. This led us to the idea of configurable backup pipelines: users could choose a source, pass the data through tools for things like compression and encryption, and choose where to store the result, all in YAML.

At this stage, all of Yaesm's code had been written by hand. This pipeline model didn't fit well into the architecture we had spent the past two years developing. Supporting it meant a substantial refactor of the existing code, so we used AI tools to help refactor Yaesm around configurable pipelines.

We replaced backends with drivers that advertised individual capabilities, such as creating snapshots, compressing data, or encrypting it. Each driver also has its own configuration options, so users can control how each tool behaves within a pipeline. Each capability used Python type annotations to declare what kind of data it accepted and produced. Yaesm could then inspect those types to determine which capabilities could connect, build a compatible pipeline, and reject incompatible combinations.

This makes the system powerful and easy to extend: new drivers can connect to the existing pipelines by implementing compatible capabilities. We plan to add support for rclone, bcachefs, pigz, and age next.

How I use Yaesm

I use Yaesm to back up my laptop and home server, both of which use Btrfs. Yaesm runs on the server and pulls backups from my laptop over SSH through Tailscale.

I keep frequent local snapshots for quick recovery. I also back up to the server using Btrfs send/receive, which can send just the changes since an earlier snapshot. I then use Yaesm's artifact copying feature to copy the existing server backups to an external drive that is plugged into the server. I keep these copies for much longer, so most of my backup history lives on the external drive.

Each schedule has its own retention rules. For example, these schedules keep the latest 365 daily backups and 48 monthly backups on the external drive:

schedules:
  daily:
    trigger:
      cron: "15 5 * * *" # every day at 05:15
    retention:
      keep-last: 365
  monthly:
    trigger:
      cron: "10 5 1 * *" # first of each month at 05:10
    retention:
      keep-last: 48

This gives me detailed recent history and older recovery points without keeping every daily backup forever. The keep-last rules count backups, so skipping unchanged backups can make that history cover a longer period. Yaesm also supports keeping backups by age, with rules such as keep-for: 1y.

Try Yaesm

Yaesm is currently in pre-release, and we're looking for people to try it and share their feedback. We currently build packages in CI only for Debian and Fedora. You can find the packages and manual on our GitHub page. If you encounter a bug, find something confusing, or have an idea for improving Yaesm, please open an issue.