DEV Community: Kamil Marut

ZBar and the Memory Usage Mystery

Kamil Marut — Thu, 10 Jul 2025 11:30:00 +0000

Introduction

Last week, my team was forwarded this screenshot from Grafana, which showed a sudden spike in memory usage of our processing service:

We were asked to look into it, as the traces were unclear and the spike was causing stability issues in production. Fortunately, the logs accurately pointed us to the document involved in the spike.
Unfortunately, it was just a single-page PDF document that weighed only around ~1MB, so at first glance, we couldn't draw any conclusions from it.

One of my team members, started investigating the issue and quickly found that the spike was caused by the ZBar library, which we use to decode QR codes in our service.

It was unclear to us at first why ZBar consumed so much memory, so I decided to get cracking.

Understanding the current processing

So, first we need to understand what exactly is the state of the current QR code processing. Since ZBar does not support processing PDF documents directly, we first convert the PDF to an image using the pdftoppm command line utility (part of the Poppler library). Then, we use ZBar to try and detect the QR code in the image. Easy and straightforward; and it worked well for us so far.

Preparing the test document

As the original document that caused the spike is confidential, I downloaded a sample PDF invoice from the Internet, stuck a QR code on it, printed it out and scanned to a PDF (to at least try to replicate the process the customer used). Here's how the document looks like:

The actual PDF file that I created weighs 1.74MB, which is slightly larger than the original document that caused the spike.

What is this resolution?

Let's look at the memory footprint of this processing with a simple benchmarking script using tracemalloc.

And this is the output of the benchmarking script:

$ python ./zbar_process.py testdata.pdf 
Starting PDF processing for testdata.pdf at 500 DPI...
PDF converted to 1 images.
Processing image 1/1
  - Found barcode: http://kamilmarut.com
  - Current memory usage: 0.78MB
  - Peak memory usage:   46.81MB
--------------------
Processing finished in 2.25 seconds.

Wow! The peak memory is nearly 27 times the size of the original PDF document! We have successfully read out the QR code from the document, but at what cost? The memory footprint is quite high, so let's try to understand why.

The first clue is the one that's already printed in the output: we have converted the PDF to an image at 500 DPI. That is most likely unnecessarily high and will definitely eat up a lot of memory!
Since this is the value that was set on production, we should try to go down with it.

$ python ./zbar_process.py CCI10072025.pdf --resolution=300
Starting PDF processing for CCI10072025.pdf at 300 DPI...
PDF converted to 1 images.
Processing image 1/1
  - Found barcode: http://kamilmarut.com
  - Current memory usage: 0.78MB
  - Peak memory usage:   17.35MB
--------------------
Processing finished in 1.14 seconds.

$ python ./zbar_process.py CCI10072025.pdf --resolution=100
Starting PDF processing for CCI10072025.pdf at 100 DPI...
PDF converted to 1 images.
Processing image 1/1
  - Found barcode: http://kamilmarut.com
  - Current memory usage: 0.78MB
  - Peak memory usage:   2.62MB
--------------------
Processing finished in 0.41 seconds.

$ python ./zbar_process.py CCI10072025.pdf --resolution=80
Starting PDF processing for CCI10072025.pdf at 80 DPI...
PDF converted to 1 images.
Processing image 1/1
  - Found barcode: http://kamilmarut.com
  - Current memory usage: 0.78MB
  - Peak memory usage:   1.96MB
--------------------
Processing finished in 0.38 seconds.

$ python ./zbar_process.py CCI10072025.pdf --resolution=50
Starting PDF processing for CCI10072025.pdf at 50 DPI...
PDF converted to 1 images.
Processing image 1/1
  - No barcodes found.
  - Current memory usage: 0.78MB
  - Peak memory usage:   1.24MB
--------------------
Processing finished in 0.35 seconds.

As we can see, the peak memory usage drops significantly as we lower the DPI value. At 50 DPI, we are not able to detect the QR code anymore, but at 80 DPI, we are still able to read it and the peak memory usage is only 1.96MB, which is much more reasonable. My further experiments show that 80 unfortunately is too low a value, as even slightly smaller QR codes are not picked up anymore. Depending on your use case, the DPI of 100-150 could be reasonable, but for us, the sweet spot turned out to be somewhere around 220 DPI. Which still saves us about 1.5 second of processing time and nearly 6 times less memory usage compared to the original 500 DPI setting.

What about pre-processing?

Changing the resolution was a great fix and a quick hotfix to change the resolution in production definitely saved us on some computing costs, but why not try to optimize even further? There are 2 optimizations that immediately come to mind:

scale the image down
convert the image to grayscale

Since grayscaling the image turned out to be a very non-invasive change, I decided to bundle it together with the resizing in a single parameter called --optimize. Please notice that I use the Pillow library to handle the pre-processing, but the pdftoppm utility does support both of those operations as well with the -scale-to and -gray parameters, so you should most likely prefer them. I just did this simply because I did not know about those parameters at the time of writing the script and I'm quite proficient with Pillow.

Let's try to run the script with the --optimize parameter on the original 500 DPI image:

$ python ./zbar_process.py CCI10072025.pdf --optimize
Starting PDF processing for CCI10072025.pdf at 500 DPI...
PDF converted to 1 images.
Processing image 1/1
  - Resizing image from 4015x5727 to 1024x768
  - Converting to grayscale
  - Found barcode: http://kamilmarut.com
  - Current memory usage: 0.78MB
  - Peak memory usage:   2.36MB
--------------------
Processing finished in 1.88 seconds.

This is great! This pre-processing alone had bigger impact on the memory usage than the resolution change! The processing time is also slightly lower, although it appears changing the density of the image has had a much bigger impact here.

Now, let's try to run pre-processing on the image with the resolution set to 100 DPI:

$ python ./zbar_process.py CCI10072025.pdf --resolution=100 --optimize
Starting PDF processing for CCI10072025.pdf at 100 DPI...
PDF converted to 1 images.
Processing image 1/1
  - Resizing image from 803x1146 to 717x1024
  - Converting to grayscale
  - Found barcode: http://kamilmarut.com
  - Current memory usage: 0.78MB
  - Peak memory usage:   2.25MB
--------------------
Processing finished in 0.43 seconds.

There's an extra overhead of .02 seconds in processing time compared to just the resolution change due to the extra pre-processing, but the peak memory usage is reduced even further. If you're optimizing for memory usage, you should definitely consider this pre-processing step.

Summary

In the end, we decided to only go with the DPI parameter change and the grayscaling in a pre-processing step, as we didn't want to risk breaking processing for documents that we don't have in our test suite. Remember, when tweaking parameters such as image resolution and quality, the clean-room strategy gives you an idea of what you can save, but you should always test on a representative sample of your production data.

And for anybody curious, this is as low as I could go with the optimizations before ZBar stopped detecting the QR code:

$ python ./zbar_process.py CCI10072025.pdf --resolution=54 --optimize --save-image
Starting PDF processing for CCI10072025.pdf at 54 DPI...
PDF converted to 1 images.
Processing image 1/1
  - Converting to grayscale
  - Saving processed image to CCI10072025.jpg
  - Found barcode: http://kamilmarut.com
  - Current memory usage: 0.78MB
  - Peak memory usage:   1.32MB
--------------------
Processing finished in 0.36 seconds.

And this is how the final image looks like:

Sure, it's small and there are visible artifacts, but still quite readable if you ask me - I don't know why ZBar's complaining!

Django admin tricks I commonly use

Kamil Marut — Thu, 13 Mar 2025 14:00:00 +0000

Introduction

The Django admin site is one of the best features that Django provides. It's useful during the development process to manage the database data in a simplified (or enhanced) manner, as well as in production environment, where it provides a simple interface for the content managers to manage the data.

Despite its simplicity, the Django admin site is extensible, although not always in a very obvious way. Nevertheless, it is a common pattern across many projects, I've seen, to actually extend the Django admin site as much as possible if most of your application is CRUD based. I've seen this done to an extent that the admin site was exposed to public customers as the main interface to the application.

Django admin tricks

Here are a couple of Django tricks / hidden features that I commonly use to modify the Django admin page and make it even more powerful!

Disabling editing/deleting capabilities

Usually you want to be able to modify according to their user permissions. Sometimes (and I see this in a lot of projects) you either roll out your own permission system and forget about the Django permissions, give content managers the is_superuser flag or simply want to completely prohibit users from deleting certain objects, no matter what. In this case, overriding the has_*_permission (documentation) is what you want to do:

from django.contrib import admin

class CustomModelAdmin(admin.ModelAdmin):
    def has_add_permission(self, request):
        # Disables the possibility of adding a new object
        return False

    def has_change_permission(self, request, obj=None):
        # Disables the possibility of updating an object
        return False

    def has_delete_permission(self, request, obj=None):
        # Disables the possibility of deleting an object
        return False

Of course, you can modify this to fit your requirements, like a separate permission system. But who has time for that, right?

Adding extra pages

Django Admin is great because it's automatically generated based on the registered models. But sometimes you just need to squeeze this extra dashboard page or a form that doesn't really link with any of the models. In that case, let me introduce you ModelAdmin.get_urls() (documentation):

from django.contrib import admin
from django.template.response import TemplateResponse
from django.urls import path


class CustomModelAdmin(admin.ModelAdmin):
    def get_urls(self):
        urls = super().get_urls()
        # Custom URL list
        # Make sure to wrap your view in `self.admin_site.admin_view()`
        # to avoid caching the view (unless you want it to be cached, then you can use `cacheable=True`)
        # and to have proper permission checks in place.
        my_urls = [path("my_view/", self.admin_site.admin_view(self.my_view))]
        return my_urls + urls

    def my_view(self, request):
        if request.method == "GET":
            context = dict(
                # Include common variables for rendering the admin template.
                self.admin_site.each_context(request),
                # Your custom context
                key=value,
            )
            return TemplateResponse(request, "sometemplate.html", context)
        else:
            # Handle POST requests in case you have a form or want to take an action
            pass

And your custom template should most likely inherit from the admin/base_site.html template, to be consistent with the rest of the admin site.

{% extends "admin/base_site.html" %}
{% block content %}
...
{% endblock %}

Now you have a page ready that you can navigate to and perform operations on. But how to navigate to that page, you ask? Well, now it's only navigable by typing the URL directly, but you can modify the change_form.html and change_list.html templates to include a link to your custom page.

Here's an example on the change_form.html template:

{% extends "admin/change_form.html" %}
{% block object-tools-items %}
    <li>
        <a href="{% url 'admin:fill-automagically-'|add:opts.app_label|add:'-'|add:opts.model_name original.pk %}">Fill Automagically</a>
    </li>
    {{ block.super }}
{% endblock object-tools-items %}

And here's an example from another project on the change_list.html template:

{% extends "admin/change_list.html" %}
{% block object-tools-items %}
    <li>
        <a style="background: #ffffbb; color: #000" 
            href="{% url 'admin:accommodations-import' %}">Zaimportuj CSV</a>
    </li>
    {{ block.super }}
{% endblock object-tools-items %}

They are both very similar, but we extend other base templates and in the first example, we use the app label and model name to reverse the URL. If you want your template to be more reusable, then this would be the recommended approach. Otherwise, the second one is simpler.

You can either override the templates by using the same names as per the Django documentation, or you can specify the templates in the ModelAdmin class:

class CustomModelAdmin(admin.ModelAdmin):
    change_list_template = "yourapp/admin/custom_change_list.html"
    change_form_template = "yourapp/admin/custom_change_form.html"

Improving performance with `raw_id_fields`, `list_select_related` and `autocomplete_fields`

Even if your admin panel is not customer facing, it does not mean that you should not care about performance. Quite the opposite actually! It would be a shame to see your admin panel become completely unusable or block a worker for an extended period of time because you forgot to get rid of those pesky N+1 queries. This is why it's important to know how to manage the performance of your admin panel.

`raw_id_fields`

By default, Django provides you with a select-box interface for your ForeignKey fields, filled with all the objects from the other table. Not only is this not always useful (are you really going to hand-select an item from a select-box containing a couple of millions of items?) it also means you have to fetch all the related instances and render them. This oftentimes results in a timeout, especially if you have a couple of those fields on the same page.

By using raw_id_fields (documentation), you only show the actual foreign key value (in most cases the ID) and a representation of the object (if it's filled out and saved).

class CustomModelAdmin(admin.ModelAdmin):
    raw_id_fields = ["parent"]

`list_select_related` or `get_list_select_related`

Your admin page can also show values from the related models. While this lookup is fairly quick on the detail page, on the list page it can quickly deteriorate the performance of your admin panel by introducing an N+1 query problem.

By using list_select_related (documentation), you can specify which fields should have their related-object data fetched in the same query as the main object.

class CustomModelAdmin(admin.ModelAdmin):
    list_select_related = ["parent"]

    # Or if you need to add some conditions
    def get_list_select_related(self, request):
        if request.user.is_superuser:
            # Maybe the user sees more fields than the regular staff user
            # so we fetch more related objects
            return ["parent", "extra_obj"]

        return ["parent"]

`autocomplete_fields`

Remember raw_id_fields from before? It's great if you mostly use the admin field to view the value or know the IDs of the objects. But what if you're trying to edit the field, and you need to look up the object? Enter autocomplete_fields (documentation)!

This replaces the select-box with a Select2 autocomplete input, that you can search in and it asynchronously fetches the related objects. Great for performance AND usability!

class CustomModelAdmin(admin.ModelAdmin):
    autocomplete_fields = ["parent"]

Summary

Hopefully this article gave you an idea on how the Django admin panel can be extended and adjusted to your specific needs. Make sure to read through the Django documentation on the admin site to learn more about cool features like inline admin or password reset for admin users.

Dockerfile for Django Devs

Kamil Marut — Fri, 07 Mar 2025 10:00:00 +0000

Introduction

I absolutely adore Django. It's a fantastic web framework that allows you to prototype quickly while also being powerful enough to scale into production and beyond.

Equally, I love Docker. It has made my life so much easier—both as a developer and as a DevOps engineer.

Most of my Django projects are containerized, and setting up a Dockerfile is one of the first things I do when starting a new project. Since I don’t usually revisit it until later stages of development, I find it essential to have a solid, reusable template.

Dockerfile and its components

This is a Dockerfile that I usually start with. It is a multi-stage build that expects that you use uv for managing dependencies (which you should, it's great) and PostgreSQL as your database (which you probably do anyway). Otherwise, it is quite easy to adapt it to other project managers like Poetry and remove the PostgreSQL dependencies.

# Builder stage
FROM python:3.13-slim AS builder

ENV PYTHONUNBUFFERED=1
ENV PYTHONDONTWRITEBYTECODE=1

ENV UV_VERSION=0.6.4

RUN apt-get update \
    && apt-get install -y --no-install-recommends \
    build-essential \
    libpq-dev \
    && pip install --upgrade pip \
    && pip install "uv==$UV_VERSION"

WORKDIR /app

COPY pyproject.toml uv.lock /app/

RUN uv export --format requirements-txt --no-hashes -o /app/requirements.txt
RUN pip wheel --no-cache-dir --no-deps --wheel-dir /app/wheels -r /app/requirements.txt

# Final stage
FROM python:3.13-slim AS final

ENV PYTHONUNBUFFERED=1
ENV PYTHONDONTWRITEBYTECODE=1

COPY --from=builder /app/wheels /wheels
COPY --from=builder /app/requirements.txt /tmp/requirements.txt
RUN pip install --no-cache-dir --no-index --find-links=/wheels -r /tmp/requirements.txt \
    && rm -rf /wheels /tmp/requirements.txt

RUN apt-get update \
    && apt-get install -y --no-install-recommends \
    libpq-dev \
    && rm -rf /var/lib/apt/lists/*

RUN useradd -U appuser \
    && install -d -m 0755 -o appuser -g appuser /app/staticfiles \
    && install -d -m 0755 -o appuser -g appuser /app/media

WORKDIR /app

COPY --chown=appuser:appuser . .

USER appuser:appuser

RUN python manage.py collectstatic --noinput

ENTRYPOINT [ "/app/entrypoint.sh" ]
CMD [ "server" ]

Let's continue by breaking down the Dockerfile line-by-line.

Builder stage

FROM python:3.13-slim AS builder

This line sets the base image for the builder stage. This example uses Python 3.13, but of course this version varies between the projects.
What rarely changes, though, is that I use the slim version of the image.
This is because it is smaller, and I have a preference for installing all the dependencies myself; so far I've never had a problem with the slim version.

ENV PYTHONUNBUFFERED=1
ENV PYTHONDONTWRITEBYTECODE=1

Setting the PYTHONUNBUFFERED environment variable ensures that the Python output is sent straight to the terminal without buffering it first.
If you don't set this, you might have a delay in seeing the output of your application.

PYTHONDONTWRITEBYTECODE prevents Python from writing .pyc files to the disk,
which might reduce the image size if you run the application during build time
and potentially help you avoid some issues with the cache.

ENV UV_VERSION=0.6.4

This line sets the version of uv that we want to use later in the build process.
Usually you would want to pin the version of uv or whatever dependency manager you're using to ensure you have the same dependency resolver version that you've tested with.

RUN apt-get update \
    && apt-get install -y --no-install-recommends \
    build-essential \
    libpq-dev \
    && pip install --upgrade pip \
    && pip install "uv==$UV_VERSION"

We finally get to the dependency installation part.
First we update the package list to have the latest package versions available, install build-essential which is nearly always required for compiling other dependencies, and libpq-dev which is required for compiling psycopg (the PostgreSQL adapter for Python).
Then we upgrade pip to the latest version and install uv with the version we set earlier.

WORKDIR /app
COPY pyproject.toml uv.lock /app/

We are copying the pyproject.toml and uv.lock files to the /app/ directory in the container.
Those are the only files we need from the project that we need in the builder stage.

RUN uv export --format requirements-txt --no-hashes -o /app/requirements.txt
RUN pip wheel --no-cache-dir --no-deps --wheel-dir /app/wheels -r /app/requirements.txt

These two lines are the most important in the builder stage.
The first line exports the dependencies to a requirements.txt file. This ensures we have a file that we can work with using pip in the next line.
Additionally, we use the --no-hashes flag to ensure that the hashes are not included in the file. This is optional, and generally I would recommend to first try to build without the flag.
I often run into various problems when using the hashes, so I'm mentioning it here as it shows up in my Dockerfiles often.

The second line installs the dependencies into the /app/wheels directory.
This ensures that all required packages are pre-built in a wheel format.
We can then use these wheels in the final stage to avoid having to install certain compile dependencies, as well as to speed up the process.

Final stage

FROM python:3.13-slim AS final

This line sets the base image for the final stage. It is quite important to use the same version of Python as in the builder stage,
so you could go ahead and pin the version of the Python image as a build argument, for example.

ENV PYTHONUNBUFFERED=1
ENV PYTHONDONTWRITEBYTECODE=1

Nothing new here. We set the same environment variables as in the builder stage for the same reasons.

COPY --from=builder /app/wheels /wheels
COPY --from=builder /app/requirements.txt /tmp/requirements.txt
RUN pip install --no-cache-dir --no-index --find-links=/wheels -r /tmp/requirements.txt \
    && rm -rf /wheels /tmp/requirements.txt

These lines copy the compiled Python dependencies and the requirements.txt file from the builder stage to the final stage.
Then we install the dependencies from the wheels directory. We also remove the wheels and the requirements.txt file to keep the image size down.

RUN apt-get update \
    && apt-get install -y --no-install-recommends \
    libpq-dev \
    && rm -rf /var/lib/apt/lists/*

Here, we can install all the runtime dependencies that we need. In case you're not running PostgreSQL as your database, you could remove this completely.
In case you need more runtime dependencies, you would specify them here. We also remove the package list to keep the image size down.

RUN useradd -U appuser \
    && install -d -m 0755 -o appuser -g appuser /app/staticfiles \
    && install -d -m 0755 -o appuser -g appuser /app/media

It is recommended to run Docker containers as a non-root user.
So we create a user called appuser and create the directories for the static files and media files with the correct permissions.

WORKDIR /app
COPY --chown=appuser:appuser . .

We set the working directory to /app and copy the actual project files to the container.
We also set the owner of the files to appuser to ensure we don't have any permission issues.

USER appuser:appuser

This line sets the user to appuser so that the application runs as this user.

RUN python manage.py collectstatic --noinput

Controversial, but if you are serving the static files from the Django application (e.g. using Whitenoise), it makes sense to embed the static files in the image.
This way, you don't have to worry about running collectstatic manually or in post-build script.

ENTRYPOINT [ "/app/entrypoint.sh" ]
CMD [ "server" ]

This line sets the entrypoint for the container and runs the server command by default. I will elaborate on the entrypoint.sh script in the next section.

Entrypoint

This is the entrypoint.sh script that I usually use. It is quite simple and only accepts the server command as an argument.

#!/bin/bash

PROCESS_TYPE=$1
echo "Running process type: $PROCESS_TYPE"

if [ "$PROCESS_TYPE" = "server" ]; then
    exec gunicorn --bind 0.0.0.0:8000 --log-level INFO --access-logfile "-" --error-logfile "-" app.wsgi
else
    echo "Unknown process type: $PROCESS_TYPE"
fi

In the very basic scenario, you could definitely just replace it with the CMD instruction in the Dockerfile, but I like to keep it separate for flexibility.

This flexibility comes useful quickly if you would like to Dockerize a Celery worker or a management command, for example.
You could also add more commands to the script, like running migrations before starting the server.

Summary

There's really no one-size-fits-all Dockerfile for Django projects, but this is as close as I've gotten to something that works most of the time
and if it doesn't, it's still a great starting point.

Simplest Homelab Backup Strategy (That Came to My Mind)

Kamil Marut — Tue, 04 Mar 2025 12:44:14 +0000

Introduction

As a self-hosting enthusiast, I try to replace as many services as possible with their self-hosted counterparts.
I also try to avoid making the same mistake as many other self-hosting enthusiasts before me—running a setup without a proper backup solution in place.

I've been lucky enough not to lose any data (and from what I know, I've really been playing with fire, especially since I run some services on a Raspberry Pi with an SD card). But I've decided I don't want to rely on luck. Instead of having 20/20 hindsight, I want 20/20 foresight.

The Setup

My setup is still pretty modest, as I'm still experimenting to achieve the right mix of hosted and self-hosted services.

I have a Raspberry Pi 5 that runs multiple services using Docker. I manage the Docker services using Docker Compose and Dockge.
I've used Portainer in the past, and while I liked working with it, I found it to be overkill for a single-server setup.

Additionally, I have a Synology DS220+ NAS server that I use for real-time device sync using Synology Drive and as a media server using Jellyfin.
Synology NAS is running in a RAID 1 configuration and uses Cloud Sync to store an encrypted copy of the data on a remote server for extra safety.

Both of the devices are on the same network, but I use Tailscale to connect between different machines as I like being independent of the network setup.

The Strategy

The plan is to do a daily backup of the data from the Pi to the Synology NAS.

I explored different options on how to effectively and, more importantly, securely backup the data.
I first wanted to use rsync to copy the data from the Pi to the NAS, but I didn't like opening the SSH port on the NAS.

The next best thing that came to mind was using NFS on a single shared directory, mounting it on the Pi, and then backing up the data to the NFS share.

I've set up a shared directory on the Synology NAS and created the following NFS rule for it:

This config allows me to mount the directory through the Tailscale network (and not through the local network)

The two main "gotchas" in my setup are:

Enable the "Allow connections from non-privileged ports" option.

I tried forcing the NFS mount to use a privileged port, but I was constantly getting "access denied" errors. I don't have any proof for this, but I think it might be related to Tailscale modifying the network traffic. I decided to enable this option as the NFS share is only accessible through the Tailscale network, so it should be secure enough.
Use 127.0.0.1 as the client IP address if you try to connect through Tailscale.

At first I was using the Tailscale IP first and that failed. Then I switched to the local IP and that worked.
I was pulling my hair out until I found this Reddit thread that pointed out that the Tailscale traffic goes through the loopback interface. Thanks, Internet stranger!

Now that I have the NFS share set up, I need to mount it on the Pi. I started by running the following commands:

# Create the mount point
sudo mkdir /mnt/backup
# Mount the NFS share
sudo mount -t nfs <tailscale-ip>:/volume1/backup_machines /mnt/backup

If successful, then no message is shown. You can confirm the mount by running ls /mnt/backup and seeing the files from the NAS or by running mount | grep nfs to see if the directory is mounted correctly.

Of course, we don't want to run this command every time we restart the Pi. So I added the following line to the /etc/fstab file:

<tailscale-ip>:/volume1/backup_machines /mnt/backup nfs defaults 0 0

In a perfect world, this would probably do the trick. But unfortunately, Tailscale once again adds a twist to the story. The Tailscale service is not started when the /etc/fstab is read, so the mount fails. The simplest solution I could think of is to add a cron job that runs the mount command after a reboot.

# Run crontab as root
sudo crontab -e

# Add the following line
@reboot mount -a

Fortunately at the point that this cronjob runs, the Tailscale service is already started and the NFS share is mounted correctly.

Perfect! We now have the NFS share mounted on the Pi.
Now, we need to copy the data over there. But where do we copy it from? For each of the services, I store the relevant data in Docker persistent volumes. They are technically available in /var/lib/docker/volumes, but it is generally not recommended to access them directly. So I once again dived in search of a better way to backup those volumes.

I came across this amazing project offen/docker-volume-backup which not only can backup volumes locally on a schedule, it can also gracefully shut down the containers before doing so to make sure there are no unfinished writes. It also supports features like notifications, backup rotation, and compression. Additionally, it's compatible with different storage backends, making it easy to switch strategies—for example, to a remote server. And all of this with a simple Docker container. This is the whole Docker Compose configuration I use for it, not much more than the example from the project's docs:

services:
  backup:
    image: offen/docker-volume-backup:v2.43.2
    restart: always
    env_file: .env
    volumes:
      # Binding the volumes to be backed up to the container directory being backed up
      - volume1:/backup/volume1:ro
      - volume2:/backup/volume2:ro
      - volume3:/backup/volume3:ro
      # Binding the Docker socket to the container to be able to gracefully stop the containers during the backup
      - /var/run/docker.sock:/var/run/docker.sock:ro
      # Binding the NFS share to the container directory where the backups are stored
      - /mnt/backup:/archive

volumes:
  volume1:
    # Marking as externals as volumes are created in other compose files
    external: true
  volume2:
    external: true
  volume3:
    external: true

I set it up to compress and backup the volumes to the mounted NFS share on the NAS every night with an e-mail notification if something goes wrong:

# Run the backup everyday at 3:30
BACKUP_CRON_EXPRESSION="30 3 * * *"

BACKUP_FILENAME="backup-%Y-%m-%dT%H-%M-%S.{{ .Extension }}"

# Compress the backup
BACKUP_COMPRESSION="gz"
GZIP_PARALLELISM=1

NOTIFICATION_URLS=smtp://...
NOTIFICATION_LEVEL="error"

Et voilà! We can manually trigger a backup to confirm it's working:

docker compose exec backup backup

And see the files on the NAS:

Summary

Even though it took me a while to set this up, I see this as a one-time investment as I had to learn a bit more about NFS and Tailscale. This setup works well now (remember to test your backups!) but I'm definitely going to explore other possibilities as I expand my homelab.

Diagrams as Code (DaC) - crafting beautiful diagrams with D2

Kamil Marut — Thu, 01 Dec 2022 00:00:00 +0000

We all love diagrams. A quick peek allows us to quickly create a mental map, no matter whether it's cloud architecture, a database schema, or a sequence diagram.
But creating diagrams is hard. It's hard to keep them up to date, it's hard to collaborate on them, and it's hard to make them look good.

There are a lot of tools for creating diagrams, but probably the most popular one is draw.io. It's a great tool, but it's not without its flaws.
It gives you a lot of freedom, but also bothers you with the little details.

Since we often use diagrams to describe our infrastructure, why don't we learn from the DevOps world and treat them as code?
What if next to our Infrastructure as Code (IaC) we could also have Diagrams as Code (DaC)?

What is D2?

In the words of the creators, D2 is a domain-specific language (DSL) that stands for Declarative Diagramming. Declarative, as in, you describe what you want diagrammed, it generates the image.

In other words, D2 gives you the ability to write diagrams as code, and then use a CLI tool to compile and render them into SVG or PNG images.
The CLI tool supports live reload, so the workflow is nearly as smooth as working with other diagram tools.

Get started with D2

You can install the D2 CLI tools by following the instructions on the D2 website.

After installing the CLI, you should now be good to go. The most important command is:

$ d2 -w input.d2 output.svg

This will run a live reload server that will watch for changes in the input.d2 file and render the output to output.svg.
It will also open a browser window with the rendered diagram. For most cases this is all you need, however it is also worth noting
that you can change the theme of the diagram by passing the --theme/-t parameter. This parameter is the ID of the target theme, which is not
showcased in the D2 docs, but you can find a list of all available themes in the D2 repo.

# Renders with the "Shirley temple" theme
$ d2 -w input.d2 -t 102 output.svg

Creating a diagram with D2

Using D2 to create a diagram is very simple. To present this, we will try to recreate the example AWS architecture diagram from draw.io examples.

We start by defining the containers.

aws_1: AWS Cloud {}

aws_2: AWS Cloud {}

Here, aws_1 and aws_2 are the IDs of the containers. We can then later use those IDs to reference the objects inside (or the containers themselves).

Next, we define the objects inside the containers (and the nested containers).

aws_1: AWS Cloud {
  obj: Object
  bucket_1: Bucket
  bucket_2: Bucket

  subsystem: "" {
    cloudwatch: Amazon CloudWatch
    cloudtrail: Amazon CloudTrail
    sns: Amazon Simple Notification Service
    sqs: Amazon Simple Queue Service
    lambda: AWS Lambda
    kinesis: Amazon Kinesis Data Firehose
    dynamo: Amazon DynamoDB
  }
}

aws_2: AWS Cloud {
  bucket: Bucket

  subsystem: "" {
    cloudwatch: Amazon CloudWatch
    cloudtrail: Amazon CloudTrail
  }
}

We have defined all the objects inside the containers. Now we can make the connections between them.
This is by far the part that is most troublesome when using traditional diagram tools, since with each update of the diagram you have to make sure
that the connections are still clear and correct. In D2, this is a breeze.

aws_1.bucket_1 -> aws_2.bucket
aws_2.subsystem.cloudwatch -> aws_1.subsystem.cloudwatch: Monitor template

aws_1: AWS Cloud {
  obj: Object
  bucket_1: Bucket
  bucket_2: Bucket

  obj -> bucket_1
  bucket_1 -> subsystem.cloudtrail
  subsystem.kinesis -> bucket_2

  subsystem: "" {
    cloudwatch: Amazon CloudWatch
    cloudtrail: Amazon CloudTrail
    sns: Amazon Simple Notification Service
    sqs: Amazon Simple Queue Service
    lambda: AWS Lambda
    kinesis: Amazon Kinesis Data Firehose
    dynamo: Amazon DynamoDB

    cloudtrail -> cloudwatch -> sns -> sqs -> lambda
    lambda -> dynamo
    lambda -> kinesis
  }
}

aws_2: AWS Cloud {
  bucket: Bucket

  bucket -> subsystem.cloudtrail

  subsystem: "" {
    cloudwatch: Amazon CloudWatch
    cloudtrail: Amazon CloudTrail

    cloudtrail -> cloudwatch
  }
}

And that's it! The specified objects are connected automatically. But the diagram still looks a little raw - let's add some icons to make it look better.

aws_1.bucket_1 -> aws_2.bucket
aws_2.subsystem.cloudwatch -> aws_1.subsystem.cloudwatch: Monitor template

aws_1: AWS Cloud {
    obj: Object
    obj.shape: circle
    obj.style.stroke: "#62a638"
    bucket_1: Bucket
    bucket_1.shape: image
    bucket_1.icon: https://raw.githubusercontent.com/exler/diagrams/main/icons/aws/s3/s3.svg
    bucket_2: Bucket
    bucket_2.shape: image
    bucket_2.icon: https://raw.githubusercontent.com/exler/diagrams/main/icons/aws/s3/s3.svg

    obj -> bucket_1
    bucket_1 -> subsystem.cloudtrail
    subsystem.kinesis -> bucket_2

    subsystem: "" {
        cloudwatch: Amazon CloudWatch
        cloudwatch.shape: image
        cloudwatch.icon: https://raw.githubusercontent.com/exler/diagrams/main/icons/aws/cloudwatch/cloudwatch.svg
        cloudtrail: Amazon CloudTrail
        cloudtrail.shape: image
        cloudtrail.icon: https://raw.githubusercontent.com/exler/diagrams/main/icons/aws/cloudtrail/cloudtrail.svg
        sns: Amazon Simple Notification Service
        sns.shape: image
        sns.icon: https://raw.githubusercontent.com/exler/diagrams/main/icons/aws/simple_notification_service/sns.svg
        sqs: Amazon Simple Queue Service
        sqs.shape: image
        sqs.icon: https://raw.githubusercontent.com/exler/diagrams/main/icons/aws/simple_queue_service/sqs.svg
        lambda: AWS Lambda
        lambda.shape: image
        lambda.icon: https://raw.githubusercontent.com/exler/diagrams/main/icons/aws/lambda/lambda.svg
        kinesis: Amazon Kinesis Data Firehose
        kinesis.shape: image
        kinesis.icon: https://raw.githubusercontent.com/exler/diagrams/main/icons/aws/kinesis_data_firehose/firehose.svg
        dynamo: Amazon DynamoDB
        dynamo.shape: image
        dynamo.icon: https://raw.githubusercontent.com/exler/diagrams/main/icons/aws/dynamodb/dynamodb.svg

        cloudtrail -> cloudwatch -> sns -> sqs -> lambda
        lambda -> dynamo
        lambda -> kinesis
    }
}

aws_2: AWS Cloud {
    bucket: Bucket
    bucket.shape: image
    bucket.icon: https://raw.githubusercontent.com/exler/diagrams/main/icons/aws/s3/s3.svg

    bucket -> subsystem.cloudtrail

    subsystem: "" {
        cloudwatch: Amazon CloudWatch
        cloudwatch.shape: image
        cloudwatch.icon: https://raw.githubusercontent.com/exler/diagrams/main/icons/aws/cloudwatch/cloudwatch.svg
        cloudtrail: Amazon CloudTrail
        cloudtrail.shape: image
        cloudtrail.icon: https://raw.githubusercontent.com/exler/diagrams/main/icons/aws/cloudtrail/cloudtrail.svg

        cloudtrail -> cloudwatch
    }
}

With the icons added, the diagram is much more intuitive and easier to understand. Sadly, D2 does not support filesystem paths for icons, so you have to use remote URLs.
Nevertheless, the diagram is finished and looks great.

For now, D2 has limited capability of styling, so we are not able to fully reproduce the original diagram. However, we just wrote an easily reproducible and maintainable diagram in less than 70 LOC!

Summary

I hope that this post has convinced you that Diagrams as Code fit the modern's software architect's tech stack perfectly.
They are easy to write, easy to maintain, and easy to share. Give them a go when you have the chance!

DEV Community: Kamil Marut

ZBar and the Memory Usage Mystery

Introduction

Understanding the current processing

Preparing the test document

What is this resolution?

What about pre-processing?

Summary

Django admin tricks I commonly use

Introduction

Django admin tricks

Disabling editing/deleting capabilities

Adding extra pages

Improving performance with raw_id_fields, list_select_related and autocomplete_fields

raw_id_fields

list_select_related or get_list_select_related

autocomplete_fields

Summary

Dockerfile for Django Devs

Introduction

Dockerfile and its components

Builder stage

Final stage

Entrypoint

Summary

Simplest Homelab Backup Strategy (That Came to My Mind)

Introduction

The Setup

The Strategy

Summary

Diagrams as Code (DaC) - crafting beautiful diagrams with D2

What is D2?

Get started with D2

Creating a diagram with D2

Summary

Improving performance with `raw_id_fields`, `list_select_related` and `autocomplete_fields`

`raw_id_fields`

`list_select_related` or `get_list_select_related`

`autocomplete_fields`