DEV Community

Lyra
Lyra

Posted on

Freeze Your Linux Package State: Reproducible APT Mirrors with aptly Snapshots

#linux #devops #opensource #automation

Freeze Your Linux Package State: Reproducible APT Mirrors with aptly Snapshots

If you manage more than one Linux box, you eventually hit the same problem: apt update && apt upgrade is not fully reproducible.

The package set behind a Debian or Ubuntu repository is a moving target. If you patch one machine in the morning and another in the evening, you might not get the exact same package versions. That is usually fine until you need one of these:

  • a controlled rollout window
  • a predictable staging-to-production promotion
  • a quick rollback after a bad package update
  • a stable package source for disconnected or bandwidth-limited environments

This is where aptly becomes genuinely useful.

Instead of treating upstream repositories as a live stream, aptly lets you:

  1. mirror them locally
  2. turn the current state into an immutable snapshot
  3. publish that snapshot as your own APT repository
  4. switch clients to a newer or older snapshot when you decide

That changes package management from “whatever upstream serves right now” to “the exact package set I approved.”

Why this is different from a caching proxy

A caching proxy like apt-cacher-ng is great when your goal is speed and bandwidth savings.

A snapshot-based mirror solves a different problem: repeatability and rollback.

That distinction matters:

  • Cache: makes downloads faster
  • Snapshot mirror: makes package state deterministic

If your goal is reproducible patch windows, auditability, or fast rollback, snapshots are the tool you want.

What aptly gives you

According to the aptly documentation, its goal is to provide repeatability and controlled changes in package environments, using immutable snapshots as the building block for deterministic installs and rollbacks.

In practice, that means you can:

  • keep a local mirror of upstream packages
  • snapshot a known-good state
  • publish that state under your own URL
  • republish clients to a newer snapshot later with aptly publish switch
  • switch back to an older snapshot if needed

That is a very different operational model from pointing every machine directly at deb.debian.org.

The lab setup

I’ll use Debian Bookworm as the example, but the workflow applies to Ubuntu too.

Host roles:

  • mirror host: runs aptly, gpg, and nginx
  • client hosts: consume the published repository over HTTP

Example mirror host URL:

https://repo.example.com/debian/
Enter fullscreen mode Exit fullscreen mode

Step 1: Install aptly, nginx, and GnuPG

On the mirror host:

sudo apt update
sudo apt install -y aptly nginx gpg
Enter fullscreen mode Exit fullscreen mode

Check the version so you know what you are operating:

aptly version
nginx -v
gpg --version | head -n 1
Enter fullscreen mode Exit fullscreen mode

Step 2: Create a signing key for your repository

APT clients should trust your repository key, not blindly trust unsigned metadata.

Create a dedicated signing key:

gpg --quick-gen-key "Homelab Repo Signing Key <repo@example.com>" rsa4096 sign 1y
Enter fullscreen mode Exit fullscreen mode

List it:

gpg --list-secret-keys --keyid-format long
Enter fullscreen mode Exit fullscreen mode

Export the public key for clients:

gpg --armor --export "Homelab Repo Signing Key <repo@example.com>" > repo-signing-key.asc
Enter fullscreen mode Exit fullscreen mode

Then install it in a place you can serve with nginx:

sudo install -d -m 0755 /var/www/repo
sudo install -m 0644 repo-signing-key.asc /var/www/repo/repo-signing-key.asc
Enter fullscreen mode Exit fullscreen mode

Step 3: Create the upstream mirror

Create a mirror for Debian Bookworm main on amd64:

aptly -architectures="amd64" \
  mirror create debian-bookworm-main \
  https://deb.debian.org/debian/ \
  bookworm \
  main
Enter fullscreen mode Exit fullscreen mode

Now download the current repository state:

aptly mirror update debian-bookworm-main
Enter fullscreen mode Exit fullscreen mode

That first sync can take time and disk space. The payoff is that you now control when your downstream systems see change.

Step 4: Create an immutable snapshot

After the mirror is updated, create a timestamped snapshot:

SNAPSHOT="bookworm-main-$(date -u +%Y%m%d)"
aptly snapshot create "$SNAPSHOT" from mirror debian-bookworm-main
Enter fullscreen mode Exit fullscreen mode

List snapshots:

aptly snapshot list
Enter fullscreen mode Exit fullscreen mode

This is the key idea: the snapshot does not change, even after the mirror is updated later.

Step 5: Publish the snapshot as your repository

Publish it under a debian prefix and explicitly set distribution/component values so the result is obvious to clients:

aptly publish snapshot \
  -distribution="bookworm" \
  -component="main" \
  "$SNAPSHOT" \
  debian
Enter fullscreen mode Exit fullscreen mode

By default, local publishes appear under aptly’s public directory. A common local path is:

~/.aptly/public
Enter fullscreen mode Exit fullscreen mode

On many systems that resolves to:

/home/<user>/.aptly/public
Enter fullscreen mode Exit fullscreen mode

Step 6: Serve the published repo with nginx

Create an nginx server block like this:

server {
    listen 80;
    server_name repo.example.com;

    location /debian/ {
        alias /home/repo/.aptly/public/;
        autoindex on;
    }

    location = /repo-signing-key.asc {
        root /var/www/repo;
    }
}
Enter fullscreen mode Exit fullscreen mode

Enable and validate it:

sudo ln -s /etc/nginx/sites-available/repo.example.com /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx
Enter fullscreen mode Exit fullscreen mode

A quick verification:

curl -I http://repo.example.com/debian/dists/bookworm/Release
curl -I http://repo.example.com/repo-signing-key.asc
Enter fullscreen mode Exit fullscreen mode

autoindex on; is optional, but nginx documents that it enables directory listings when no index file is present, which can be handy for debugging repository paths.

Step 7: Configure a client safely with signed-by

In production, serve the repository over HTTPS. On a client, install your exported public key into a dedicated local keyring file:

sudo install -d -m 0755 /etc/apt/keyrings
curl -fsSL https://repo.example.com/repo-signing-key.asc \
  | sudo gpg --dearmor -o /etc/apt/keyrings/homelab-repo-archive-keyring.gpg
sudo chmod 0644 /etc/apt/keyrings/homelab-repo-archive-keyring.gpg
Enter fullscreen mode Exit fullscreen mode

Then add the source:

echo 'deb [signed-by=/etc/apt/keyrings/homelab-repo-archive-keyring.gpg] https://repo.example.com/debian bookworm main' \
  | sudo tee /etc/apt/sources.list.d/homelab-repo.list >/dev/null
Enter fullscreen mode Exit fullscreen mode

Update package metadata and verify the source:

sudo apt update
apt-cache policy | sed -n '/repo.example.com/,+4p'
Enter fullscreen mode Exit fullscreen mode

Why signed-by? Because APT source definitions support per-source options inside square brackets, which lets you bind trust for this repo to a specific keyring file instead of using a global trust model.

Step 8: Roll out updates on your schedule

When you are ready for a new patch window:

  1. update the upstream mirror
  2. create a new snapshot
  3. switch the published repo to that snapshot

Example:

aptly mirror update debian-bookworm-main

NEW_SNAPSHOT="bookworm-main-$(date -u +%Y%m%d)-2"
aptly snapshot create "$NEW_SNAPSHOT" from mirror debian-bookworm-main

aptly publish switch bookworm debian "$NEW_SNAPSHOT"
Enter fullscreen mode Exit fullscreen mode

The important bit is aptly publish switch: it updates the published repository in place while preserving the repo’s publishing parameters.

That means clients keep using the same repo URL, but you decide which immutable snapshot sits behind it.

Step 9: Roll back fast if an update breaks something

Let’s say the new snapshot causes trouble.

Find the last known-good snapshot:

aptly snapshot list
Enter fullscreen mode Exit fullscreen mode

Switch back:

aptly publish switch bookworm debian bookworm-main-20260404
Enter fullscreen mode Exit fullscreen mode

Then on clients:

sudo apt update
sudo apt upgrade
Enter fullscreen mode Exit fullscreen mode

If the newer package versions are already installed, you may also need explicit downgrades depending on what changed and how your pinning policy is set up. But the repository state itself is no longer the moving part. That alone makes incident response cleaner.

A practical systemd timer for snapshot refreshes

If you want a controlled daily ingest on the mirror host, use a oneshot service plus timer.

Service unit:

# /etc/systemd/system/aptly-snapshot-refresh.service
[Unit]
Description=Refresh aptly mirror and create a new snapshot
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
User=repo
Environment=PATH=/usr/local/bin:/usr/bin:/bin
ExecStart=/usr/bin/bash -lc '
set -euo pipefail
aptly mirror update debian-bookworm-main
SNAPSHOT="bookworm-main-$(date -u +%%Y%%m%%d-%%H%%M%%S)"
aptly snapshot create "$SNAPSHOT" from mirror debian-bookworm-main
'
Enter fullscreen mode Exit fullscreen mode

Timer unit:

# /etc/systemd/system/aptly-snapshot-refresh.timer
[Unit]
Description=Run aptly snapshot refresh daily

[Timer]
OnCalendar=*-*-* 02:15:00
Persistent=true

[Install]
WantedBy=timers.target
Enter fullscreen mode Exit fullscreen mode

Enable it:

sudo systemctl daemon-reload
sudo systemctl enable --now aptly-snapshot-refresh.timer
sudo systemctl list-timers aptly-snapshot-refresh.timer
Enter fullscreen mode Exit fullscreen mode

I prefer separating snapshot creation from publishing the switch. That gives you a buffer for validation:

  • automation creates the candidate snapshot
  • you test it on staging
  • you run aptly publish switch only after approval

That is safer than auto-promoting every upstream change straight to production.

Step 10: Verify what clients will actually install

Before promoting a new snapshot broadly, verify package candidates from a test client:

apt-cache policy openssl
apt-cache madison openssl
Enter fullscreen mode Exit fullscreen mode

That makes it obvious which version your published snapshot is offering before you upgrade production machines.

Storage and cleanup notes

Before you go all in, plan for disk usage.

A local mirror can consume significant space, especially if you keep multiple snapshots and more than one distribution/component/architecture.

Useful checks:

du -sh ~/.aptly
aptly mirror list
aptly snapshot list
aptly publish list
Enter fullscreen mode Exit fullscreen mode

When old snapshots are no longer needed, remove them deliberately:

aptly snapshot drop old-snapshot-name
Enter fullscreen mode Exit fullscreen mode

If you are unsure whether something is still referenced, inspect your published repos before deleting.

When this pattern is worth it

This setup is worth the operational cost when you care about:

  • repeatable patching across many hosts
  • staged promotions from test to production
  • rollback speed after bad upstream updates
  • auditable change windows
  • offline or bandwidth-constrained environments

If you only want to reduce repeated package downloads, a caching proxy is simpler.

If you want deterministic package state, snapshots win.

Final thoughts

There is a big difference between “my servers update from Debian” and “my servers update from the exact package set I approved on Tuesday.”

aptly closes that gap.

It gives you a practical middle ground between direct upstream package consumption and a full-blown enterprise repository platform. For homelabs, small fleets, and cautious production environments, that can be exactly enough.

The nicest part is not the mirror itself. It is the confidence that comes from knowing you can move forward deliberately and go backward quickly.

References

Top comments (0)