DEV Community: Veerendra K

How to Deploy rest-server(Restic) on Docker Swarm Behind Traefik Reverse Proxy

Veerendra K — Thu, 20 Jul 2023 23:18:19 +0000

As you might know Restic, a simple backup manager. I have been using restic for a while now, backing up all my data easily and securely. For example...

# Initialize a backup repository in the specified local directory:
$ restic init --repo path/to/repository

# Backup a directory to the repository:
$ restic --repo path/to/repository backup path/to/directory

# Show backup snapshots currently stored in the repository:
$ restic --repo path/to/repository snapshots

# Restore a specific backup snapshot to a target directory:
$ restic --repo path/to/repository restore latest|snapshot_id --target path/to/target

When it comes to managing remote backups, restic supports sftp

$ restic -r sftp:user@host:/srv/restic-repo init
enter password for new repository:
enter password again:
created restic repository f1c6108821 at sftp:user@host:/srv/restic-repo
Please note that knowledge of your password is required to access the repository.
Losing your password means that your data is irrecoverably lost.

There is one more option to manage remote backups, that is via http with rest-server tool.

Basically, you run rest-server as a systemd daemon or in docker to expose http endpoint and restic client can interact with this http to manage backups on remote locations.

Then you might ask, why use this http rest-server, when we have sftp?

As rest-server READM.md says

Compared to the SFTP backend, the REST backend has better performance.

REST protocol should be faster and more scalable, due to some inefficiencies of the SFTP protocol.

Rest Server adds is the optional ability to run in append-only mode.

Rest Server implementation is really simple and as such could be used on the low-end devices, no problem.

Deployment

👉 Github repo contains docker stack files.

I considered this tool rest-server for my home server to manage backups, that's why I deployed rest-server behind Traefik reverse proxy with HTTPS. Here is the diagram below.

$ git clone https://github.com/veerendra2/raspberrypi-homeserver.git
$ cd raspberrypi-homeserver/services/rest-server
$ tree -a .
.
├── .env_rest-server
├── docker-stack.yml
└── secrets
    └── htpasswd.txt

1 directory, 3 files

So, here is my docker swarm service for reference

version: "3.8"

networks:
  network_monitoring:
    external: true
  network_public:
    external: true

secrets:
  htpasswd:
    file: secrets/htpasswd.txt

services:
  rest-server:
    image: restic/rest-server:latest
    deploy:
      replicas: 1
      placement:
        constraints: [node.role == manager]
      restart_policy:
        condition: on-failure
        delay: 30s
        max_attempts: 3
      labels:
        - traefik.enable=true
        - traefik.docker.network=network_public
        - traefik.http.routers.rest-server.tls=true
        - traefik.http.routers.rest-server.rule=Host(`veeru.duckdns.com`) && PathPrefix(`/restic`)
        - traefik.http.services.rest-server.loadbalancer.server.port=8000
    hostname: rest-server
    user: 1000:1003
    env_file:
      - .env_rest-server
    volumes:
      - /media/disk2/backups:/restic:rw
    networks:
      - network_public
      - network_monitoring
    secrets:
      - htpasswd

▶️If you want to know more about how I deployed Traefik reverse proxy, refer my github repo
▶️If you want to know more about how I used duckdns sub-domain as my home server's hostname, refer my medium blog post

and the .env_rest-server

DISABLE_AUTHENTICATION=1
OPTIONS=--prometheus --prometheus-no-auth --append-only --private-repos
DATA_DIRECTORY=/restic
PASSWORD_FILE=/run/secrets/htpasswd

I prefer expose the service(rest-server) with sub-path(/restic), that's why you see the traefik label like below

- traefik.http.routers.rest-server.rule=Host(`veeru.duckdns.com`) && PathPrefix(`/restic`)

Or if you want to expose the service with subdomain, you can set

- traefik.http.routers.rest-server.rule=Host(`restic.yourdomain.com`)

By the way, if don't have personal domain, you can check my other write-up Traefik HTTPS Config with DuckDNS for Local Homeserver

Authentication

I disabled authentication as you can see DISABLE_AUTHENTICATION=1. But still, you need to create user to interact with rest-server

$ htpasswd -B -c .htpasswd restic
New password:
Re-type new password:
Adding password for user restic

# create htpasswd
$ cat .htpasswd
restic:[REDACTED]

# save htpasswd content in secrets directory to use it as docker secrets
cat .htpasswd > secrets/htpasswd.txt

Data directory

DATA_DIRECTORY=/restic

This option tells rest-server where our restic repositories should be stored. Since we are running in Docker, we can mount this data directory outside of docker in the desired location as you can see in docker stack file.

...
  volumes:
    - /media/disk2/backups:/restic:rw
...

Prometheus monitoring

You can also enable Prometheus monitoring, by setting option --prometheus and --prometheus-no-auth is for no authentication for Prometheus scrap

Other Options

--append-only

The --append-only mode allows creation of new backups but prevents deletion and modification of existing backups. This can be useful when backing up systems that have a potential of being hacked.
and --private-repos

--private-repos

To prevent your users from accessing each others' repositories, you may use the --private-repos flag which grants access only when a subdirectory with the same name as the user is specified in the repository URL. For example, user "foo" using the repository URLs rest:https://foo:pass@host:8000/foo or rest:https://foo:pass@host:8000/foo/ would be granted access, but the same user using repository URLs rest:https://foo:pass@host:8000/ or rest:https://foo:pass@host:8000/foobar/ would be denied access. Users can also create their own subrepositories, like /foo/bar/.

This is the reason we need to create a user. Each user gets a separate directory inside the data directory(/restic) to store their backups. In above section, we created user called restic, the restic user can store backups on server at /restic/restic location(So, the path is like [DATA_DIRECTORY]/[USER]/). If you create a user "ramu", the user "ramu" can store backup at location /restic/ramu/ on the server.

Testing

📰https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server

Prepare restic repository(A restic repository is nothing but a folder containing your backup in encrypted format and its metadata).

$ restic init --repo rest:https://veeru.duckdns.org/restic/my-projects
enter password for new repository:
enter password again:
created restic repository 9266e74132 at rest:https://veeru.duckdns.org/restic/my-projects/

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

restic client intracts with rest-server (with endpoint https://veeru.duckdns.org/restic) and creates repository on remote location as you can below

$ cd /media/disk2/backups/
# restic directory is created
$ ls
restic

# check files inside restic directory
$ tree -L 2 restic/
restic/
└── my-projects
    ├── config
    ├── data
    ├── index
    ├── keys
    ├── locks
    └── snapshots

6 directories, 1 file

Note that /restic location is the sub-path we created for Traefik. Traefik will route traffic to rest-server when it sees /restic in the URL path.

Run backup

$ restic -r rest:https://veeru.duckdns.org/restic/my-projects backup my-projects/
enter password for repository:
repository 9266e741 opened (version 2, compression level auto)
created new cache in /Users/veerendra.k/Library/Caches/restic
no parent snapshot found, will read all files
[0:01] 2370 files 421.600 MiB, total 13206 files 1.562 GiB, 0 errors
...

Files:       105887 new,     0 changed,     0 unmodified
Dirs:        33890 new,     0 changed,     0 unmodified
Added to the repository: 3.807 GiB (2.988 GiB stored)

processed 105887 files, 4.461 GiB in 2:01
snapshot 9c7a6616 saved

That's it!, restic will backup(with encrption) all files from source path to remote rest-server location.

List snapshots

$ restic -r rest:https://veeru.duckdns.org/restic/my-projects snapshots
enter password for repository:
repository 9266e741 opened (version 2, compression level auto)
ID        Time                 Host          Tags        Paths
-----------------------------------------------------------------------------------------------
9c7a6616  2023-07-18 23:22:11  C02F66HKML85              /Users/veerendra.k/my-projects
-----------------------------------------------------------------------------------------------
1 snapshots

This simple rest-server allows me to manage my backups on remote locations.

How to Deploy Hugo Static Website with Github Actions

Veerendra K — Thu, 06 Jul 2023 15:19:41 +0000

As you know Hugo is a lightning-fast static website generator perfect for building modern websites. It has been almost 2 years since I’m using Hugo for my GitHub Pages blog https://veerendra2.github.io/. I’m comfortable using Hugo after I moved from Jekyll; https://veerendra2.github.io/moving-to-hugo/

In this write-up, I will show you how I‘m using Github Actions to build and deploy static websites on GitHub Pages.

👉 Here is my GitHub Actions workflow https://gist.github.com/veerendra2/28d35b44c8340d5a801e5606edb32f45

Background

I have 2 GitHub repositories, one is the source and the other is the destination.

The source repo contains markdown files where I write my blog content in these files and the destination contains a static HTML website which is served by GitHub Pages.

You can also use single repo with 2 branches to keep markdown files and a static HTML website. In fact, my previous setup was like this.

Basically, you keep all your source markdown files in a feature branch(In my case, it is “source”) and by using GitHub Actions, move the static HTML website to the “master” branch. You can take reference to this approach in my previous write-up

Like I said before, currently, I’m using 2 repos approach to keep my draft post private #12

Workflow

Usually, when I get new idea, I will create a GitHub issue for reference like below

And then I do

In the end, GitHub Actions trigger and publish my new blog post on https://veerendra2.github.io/

Github Action Workflow

👉 https://gist.github.com/veerendra2/28d35b44c8340d5a801e5606edb32f45

The workflow is straightforward and improved recently. The workflow triggers when the PR closed(Or you can also when the feature branch is merged to “main”).

...  
pull_request:
    types:
      - closed
...

Checkout both source and destination repos with “path” argument as you can see below.

...
    steps:
      - name: Checkout ${{ github.repository_owner }}/${{ env.SRC_REPO }}
        uses: actions/checkout@v3
        with:
          path: ${{ env.SRC_REPO }}

      - name: Checkout ${{ github.repository_owner }}/${{ env.DEST_REPO }}
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
          repository: ${{ github.repository_owner }}/${{ env.DEST_REPO }}
          path: ${{ env.DEST_REPO }}
          token: ${{ secrets.TOKEN_TO_PUSH_REPO }}
...

Setup and build the Hugo website using options “-d” destination directory and “-s” source directory.

...
      - name: Setup Hugo
        uses: peaceiris/actions-hugo@v2
        with:
          hugo-version: "0.110.0"
          extended: true

      - name: Build Hugo site
        run: |
          hugo --minify \
              -s ${{ github.workspace }}/${{ env.SRC_REPO }} \
              -d ${{ github.workspace }}/${{ env.DEST_REPO }}
...

Finally, commit your changes and push them to the destination repo.

...
      - name: Commit ${{ github.repository_owner }}/${{ env.DEST_REPO }}
        run: |
          cd ${{ github.workspace }}/${{ env.DEST_REPO }}
          git config --global --add safe.directory ${{ github.workspace }}
          git config --local user.email "actions@github.com"
          git config --local user.name "GitHub actions"
          git add -A
          git commit -m "New changes" -a

      - name: Push changes
        uses: ad-m/github-push-action@master
        with:
          github_token: ${{ secrets.TOKEN_TO_PUSH_REPO }}
          directory: ${{ github.workspace }}/${{ env.DEST_REPO }}
          repository: ${{ github.repository_owner }}/${{ env.DEST_REPO }}
          force: true
...

That’s it, this simple workflow build and publishes my blog posts.

But wait, there is one more trigger I placed as you might have noticed…

...
  workflow_dispatch:
    inputs:
      blog:
        description: "Publish Blog"
        required: true
        default: "yes"
...

This is because, sometimes I make changes/fixes on the “main” branch itself, for this, it won’t trigger the pipeline to publish my changes/fixes. That’s why I have added a manually trigger like below.

Let me know how you guys deploy your static website.