DEV Community

Alex Umansky
Alex Umansky

Posted on

Linux Repository Mirroring Made Simple

#linux #learning #tutorial #devops

Mirror, mirror on the wall, apt-get won’t fail me at all

Hello dear reader, and as @silent_mobius refers to, gentle readers, welcome!

My name is Alex Umansky, aka TheBlueDrara, and I welcome you to a small and simple guide I wrote on Linux repository mirroring.

This guide was inspired by a task I received in a project where I had to localize many packages in my environment, as sooner or later the internet on my poor, poor Linux server would be cut off.

So here we are — shall we begin?

Overview

For starters, let’s understand what we are going to do here.

We will be creating a mirror of the official Ubuntu repository, so we can install packages in an offline environment.

And since not all packages come from the official Ubuntu repository, some packages will need to be downloaded manually and stored in our own repo — a so-called “flat” repository.

I’ll explain the difference between them in detail in the upcoming steps.

Prerequisites

Before we begin, we need to install some prerequisites to be able to mirror repositories and expose our local mirror so we can pull packages from our mirror server.

We will need:

  • apt-utils (to generate the files we need)
  • nginx (a web server to expose our local mirror)
  • debmirror (a tool to create a local Debian/Ubuntu mirror)
  • gnupg (to create and manage GPG keys) 
apt-get update
apt-get install -y apt-utils nginx debmirror gnupg
Enter fullscreen mode Exit fullscreen mode

We’ll start with the official Ubuntu mirror.

We’ll use debmirror, a tool that mirrors a remote repo using parameters and an upstream URL.

Important: This method works not only with the Ubuntu repository but also with other vendor repositories, as long as you have the repo URL and the GPG key.

Before mirroring, it’s useful to understand how Debian/Ubuntu repos are built.

They have a special file structure that we are keen to learn.

They are organized into Suites and Components.

  • Suites are distributions or releases of packages, for example:

    • noble → The base release of Ubuntu 24.04
    • noble-updates → Stable updates after the initial 24.04 release

  • Components are sections grouped by license/support status, such as:

    • main → Official Canonical-supported open source
    • restricted → Proprietary drivers/firmware
    • universe → Community-maintained software
    • multiverse → Non-free software

If you’re unsure what packages you need and have the storage space, you can simply mirror them all — but keep in mind the whole repo may take around ~2 TB of storage space.

Sometimes it’s better to start with a small mirror of the main component of each suite and see if you’re not missing any needed dependencies.

Creating a Debian Repository

Let’s start by creating a directory to store our repository structure:

sudo mkdir -p /storage/ubuntu
Enter fullscreen mode Exit fullscreen mode

The official Ubuntu repo URL we will use is:

http://archive.ubuntu.com/ubuntu/
Enter fullscreen mode Exit fullscreen mode

In this example, we’ll mirror the noble and noble-updates suites, and only the main component from each one:

debmirror /storage/ubuntu \
  --nosource \
  --progress \
  --host=archive.ubuntu.com \
  --root=ubuntu \
  --method=http \
  --dist=noble,noble-updates \
  --section=main \
  --arch=amd64 \
  --ignore-release-gpg
Enter fullscreen mode Exit fullscreen mode

Note: For safer practice, remove --ignore-release-gpg and instead add the vendor’s GPG key to /usr/share/keyrings so signatures are verified.

The mirroring process may take a while depending on what you mirrored.

Repository File Structure

Before we head forward, it is important to understand the file structure of a Debian repo.

Once finished downloading, you’ll see three main directories:

  • project → global metadata (mostly Debian-specific, so we will set this aside for now)
  • dists → contains the suites like noble, noble-updates
  • pool → contains all the .deb packages

pool

A directory that holds all of our .deb packages.

They can be sorted alphabetically in directories or simply stored together.

It doesn’t matter for APT — it only depends on your preference, nice and tidy like the Inquisition of Mankind, or chaotic like the Chaos Warbands.

dists

This directory contains:

  • suite directories (different releases like noble/), and inside each suite are component directories like main/, restricted/, etc.
  • repository metadata files: Packages, Release, InRelease, Release.gpg

The Heart of the APT Repo

As said before, inside each dists directory there are metadata files: Packages, Release, InRelease, Release.gpg.

These files are the beating heart of the repo. They are what make APT come alive.

Like my most favorite album, Holy Diver by Ronnie James Dio (bless his soul):

“Between the velvet lies, there’s a truth as hard as steel.”

Let’s find our truth.

Packages file

An index file that lists all .deb packages with:

  • names, file paths, checksums, dependencies

This file tells the repo what .deb packages exist, what they depend on, and where they are located in the repo.

If you add or remove packages from the pool directory, you must recreate the Packages file to re-index the packages.

Release file

Summarizes the repository:

  • checksums of .deb Packages
  • defined distributions/components that exist in the repo
  • references to the index files (like Packages)
  • contains metadata of the repo

Every change in the repo should be accompanied by regenerating the Release file.

InRelease and Release.gpg

Signed versions of the Release file:

  • Release.gpg → older detached signing method
  • InRelease → newer inline signing method

To summarize this important section: each time you run apt-get update, you are pulling two files from a remote repo: the signed InRelease and the Packages files — so you can install the packages inside those repos.

Exposing the Local Repo

If we want to be able to pull packages from our mirror, we need to make the repo available to others.

In this example we’ll expose it with Nginx.

Let’s create a config file for Nginx:

vim /etc/nginx/sites-available/ubuntu-mirror
Enter fullscreen mode Exit fullscreen mode

Add:

server {
    listen 80;
    listen [::]:80;
    server_name _;

    location /ubuntu {
        alias /storage/ubuntu;
        autoindex on;
        try_files $uri $uri/ @notfound;
    }

    location /my-local-repo {
        alias /storage/my-local-repo;
        autoindex on;
        try_files $uri $uri/ @notfound;
    }

    location @notfound {
        return 404;
    }
}
Enter fullscreen mode Exit fullscreen mode

Enable it and restart the service:

sudo ln -s /etc/nginx/sites-available/ubuntu-mirror /etc/nginx/sites-enabled/
systemctl restart nginx
Enter fullscreen mode Exit fullscreen mode

Now the repo is available and can be sourced. We’ll touch on how to source it soon.

Creating a GPG Key

Signing your repo is not strictly necessary, but it’s a nice practice for automation and branding your repo.

If you don’t want to do this step, you can skip ahead.

To sign the repo with our own key, we use the gnupg tool.

Generate a key:

gpg --gen-key
Enter fullscreen mode Exit fullscreen mode

List the key to see the PUB KEY ID:

gpg --list-keys
Enter fullscreen mode Exit fullscreen mode

Signing the Repo

Remember: in each dist (suite) there are its own metadata files you need to sign.

gpg --local-user "<PUB_KEY_ID>" --yes --clearsign -o dists/<dist_name>/InRelease dists/<dist_name>/Release

gpg --local-user "<PUB_KEY_ID>" --yes --detach-sign -o dists/<dist_name>/Release.gpg dists/<dist_name>/Release
Enter fullscreen mode Exit fullscreen mode

Export the key into a file so we can pass it to the clients:

gpg --armor --export <KEY_PUB_ID> > <local_repo_key_name>.asc

sudo gpg --dearmor -o /usr/share/keyrings/<local_repo_key_name>.gpg <local_repo_key_name>.asc

sudo chmod 644 /usr/share/keyrings/<local_repo_key_name>.gpg
Enter fullscreen mode Exit fullscreen mode

And… you sold your soul to the devil — congrats!

Let’s continue with the great work so far.

Client Setup

When you run apt-get update, how does APT know where to pull metadata files from?

Quite simply — there is a source file that APT reads.

On the client, let’s create or edit a source file:

vim /etc/apt/sources.list.d/ubuntu.sources
Enter fullscreen mode Exit fullscreen mode

Add the config of our mirrored repo:

Types: deb
URIs: http://<IP_or_DNS_of_mirror_server>/ubuntu/
Suites: noble noble-updates
Components: main
Architectures: amd64
Signed-By: /usr/share/keyrings/<key_name>.gpg
Enter fullscreen mode Exit fullscreen mode

Place the GPG key where APT expects it:

mv <my_local_repo_key>.gpg /usr/share/keyrings/
Enter fullscreen mode Exit fullscreen mode

Update APT:

apt-get update
Enter fullscreen mode Exit fullscreen mode

APT stores the metadata it pulls in:

/var/lib/apt/lists
Enter fullscreen mode Exit fullscreen mode

You can check if packages are present in the repo with:

apt-cache search <package_name>
apt-cache show <package_name>
Enter fullscreen mode Exit fullscreen mode

In the newer OS versions you should use

apt show
apt search
Enter fullscreen mode Exit fullscreen mode

Note: make sure there is network connectivity to the server running the local repo.

Creating a Flat Repository

But what if I want to create a local stash of random .deb packages I need from vendors, tools, or my own creations?

A flat repo is your answer.

A flat repo is super simple: just a directory that contains .deb packages plus the metadata files we learned about before (Packages and Release), all in one place.

Messy? A bit. But it works.

Create Packages

Because this is a flat repo, we don’t get ready-made metadata. We need to create it ourselves.

Note: these steps also work for a Debian repo, just with different paths.

cd <flat_repo_dir>
apt-ftparchive packages <flat_repo_dir> | tee <flat_repo_dir>/Packages
Enter fullscreen mode Exit fullscreen mode

For a Debian repo (pool-based):

cd <debian_repo_dir>
apt-ftparchive packages ./pool | tee dists/<suite>/<component>/binary-<Arch>/Packages
Enter fullscreen mode Exit fullscreen mode

Sometimes we need to compress the file:

gzip -9c Packages > Packages.gz
Enter fullscreen mode Exit fullscreen mode

Create Release

I like to create a template I can edit later for ease of use:

cat > release.conf <<'EOF'
APT::FTPArchive::Release {
  Origin "<Custom_Origin>";
  Label  "<Custom_Label>";
  Suite  "<Custom_Suite>";
  Version "<Version>";
  Codename "<Custom_Codename>";
  Architectures "<Architecture_for_example_amd64>";
  Components "main";
  Description "<Whatever_you_wish>";
}
EOF
Enter fullscreen mode Exit fullscreen mode

Now generate the file:

cd <flat_repo_dir>
apt-ftparchive -c <Path_to_release.conf> release <path_to_root_directory> > <path_to_root_directory>/Release
Enter fullscreen mode Exit fullscreen mode

Sign it:

gpg --local-user "<PUB_KEY_ID>" --yes --clearsign -o InRelease Release
gpg --local-user "<PUB_KEY_ID>" --yes --detach-sign -o Release.gpg Release
Enter fullscreen mode Exit fullscreen mode

Exposing a Flat Repo

In the first step we already added to Nginx the configuration to expose this local flat repo.

Client Setup for Flat Repo

The source file looks a little different here.

We remove the component part, and for the suite we just mention ./ as the root.

Don’t forget to add your repo key!

vim /etc/apt/sources.list.d/flatrepo.sources
Enter fullscreen mode Exit fullscreen mode

Add:

Types: deb
URIs: http://<IP_or_DNS_of_repo>/<Repo_name>
Suites: ./
Architectures: amd64
Signed-By: /usr/share/keyrings/<Key_name>.gpg
Enter fullscreen mode Exit fullscreen mode

Update APT:

apt-get update
Enter fullscreen mode Exit fullscreen mode

I hope my work and writing helped a hopeless soul somewhere in the vast DevOps universe. Thank you for taking the time to read my work!

TheBlueDrara

Top comments (4)

Collapse
 
silent_mobius profile image
Alex M. Schapelle •

will there be ansible playbook to set it up ?

Collapse
 
thebluedrara profile image
Alex Umansky •

I may create an ansible playbook to deploy an offical ubuntu mirror on a server at will, thats a good idea

Collapse
 
silent_mobius profile image
Alex M. Schapelle •

what about debian mirror ?

Thread Thread
 
thebluedrara profile image
Alex Umansky • • Edited

it can be moduler, a different role to step up a different purpose repo, it will be like using a tool based on ansible