DEV Community: Daniel

The perfect Git strategy for continuous improvement / deployment

Daniel — Mon, 24 Mar 2025 12:00:00 +0000

Introduction
1. Quick Overview
2. Patching branches
Conclusions

Introduction

For a long time, I struggled finding the perfect Git strategy for my projects.

Many times I just felt overwhelmed by how complicated some of these strategies felt. "There must be a better way" I thought.

Then I stumbled into the "Release Branching Strategy" on releaseflow and it was like a breath of fresh air.

I will explain a simplified overview of this approach that just works for me, for a more detailed one feel free to check out their website.

1. Quick Overview

Here are the main steps of this strategy if you want to add a new feature:

You open a new feature/ branch from the main branch and commit changes on it
You merge that branch directly into the main branch
You create a version tag on the main branch i.e.: 1.0.0
You can then trigger any automated actions (GitHub actions, Jenkins, CI/CD...) when this tag is pushed. For instance I have an automated deployment using GitHub actions to build and push my docker images to the registry.

That's it!

The version tags follow the semver convention.
major.minor.patch

major when you create breaking changes

minor when you create backward compatible changes

patch when you need to fix a bug

This would be the sweet spot of this strategy:

But of course life is never this easy, and wheneve a feature is deployed it often comes with its bugs that we need to fix.

What then?

2. Patching branches

So what happens when you need to patch a released version? And how do your roll those changes back into the main branch?

Here are the steps:

Go back to the commit where you created the version tag and create a release/1.0.x branch from there. We want to leave the x to indicate that all patches will come from this branch.
Develop any bug fixes or patches directly on this branch or feel free to open new bugfix/ branches from it. Ideally bugfix/ branches should only be created on release/ branches.
Once you're done with your fixes, create a new version tag on the release/ branch, i.e.: 1.0.1
Then you can get those changes back into main 2 ways:
1. If it's a simple thing you can just cherry pick the changes into main
2. If it's something more complicated you can merge those changes into main and handle any merge conflicts you might have

It's all much clearer with a diagram:

Conclusions

I've been using this Git flow since I found it and it's been great.

Paired with a GitHub action to build and push my docker images to a registry it's worked wonders.

You can get crazy and create staging/production tags to deploy to different servers and test, or create snapshot tags to keep track of when new changes were started to be made.

All while still keeping everything super simple.

Proxmox: How to use a USB drive as storage for your containers

Daniel — Fri, 07 Feb 2025 20:31:48 +0000

So you have a USB drive that you want to mount to a specific container.

In my case, I want to mount a 10TB USB hard drive (a Seagate Expansion) where I store my media for radarr, sonnar and jellyfin.

Let's get started!

Permanently mount your USB drive

There are several steps you need to follow for a USB device to be permanently mounted to Proxmox. This is, to have it mounted even if the node restarts.

Find the USB device by going to your node -> Disks

In my case it's easily identifiable, as I can see a USB device with 10TB of space: /dev/sdb2

Edit the `/etc/fstab` file

You can use nano or your favorite editor and add the following line to the end of the file:

<file system> <mount point> <type> <options> <dump> <pass>

In my case:

/dev/sdb2 /mnt/expansion xfs defaults,noatime,nofail 0 2

This means that I'm:

Mounting the device in /dev/sdb2
Mounting it to the route /mnt/expansion in my Proxmox host
Mounting it as an xfs file system
With the options
- defaults (enables several standard mount options)
- noatime (disables updating of file access times on the filesystem)
- nofail (tells the system to continue booting even if this device cannot be mounted)
Dump 0 means the filesystem will not be backed up by the dump utility
File system check 2 indicates this filesystem should be checked after the root filesystem

Note: I previously formatted my external drive as xfs before plugging it in to make my life easier since that format is directly compatible with my Proxmox host.

Mount it!

You can either reboot your system or run mount -a to finish mounting your device.

Passing the device to the container

Start by editing the configuration file for your container. Identify your container ID and run:

nano /etc/pve/lxc/<continer id>.conf

Add this line to the end of the file:

<Mount point id> <Host path>,mp=<container path>

In my case I have:

mp0: /mnt/expansion,mp=/home/media
mp1: /mnt/downloads,mp=/home/downloads

As you can see the mount points are sequential, so be mindful of the id you're choosing if you already have other mount points in your container

Restart your container and you're good to go!

Verify mount points

If you now go to your container resources in the Proxmox GUI you should see your mountpoints being specified there:

Set a VLAN interface in your Proxmox node

Daniel — Fri, 07 Feb 2025 16:27:01 +0000

Introduction
Proxmox network basics
Making the Linux Bridge VLAN aware
The VLAN interface

Introduction

First of all, install Proxmox.

If you haven't and need a guide on how to do so, I can't recommend Network Chuck's video enough:

Proxmox network basics

Once that's out of the way, you need to know some basics about your Proxmox node networking stuff.

Go to your Node -> System -> Network

It will probably look something like this (don't worry if your IP address is different):

What are those?

enp1s0 -> This is your actual, physical, network device in the hardware you installed Proxmox on. Note that you might have more physical interfaces if your hardware has several ethernet ports or a wifi card.
vmbr0 -> This is what's called "Linux Bridge", it's like a virtual switch, that bridges (thus the name) the physical interface with all the virtual networking within Proxmox.

One important thing:

The vmbr0 interface is NOT VLAN aware. Which means that it has no clue on how to handle VLAN packets and thus not know where to route stuff.

Making the Linux Bridge VLAN aware

Edit the vmbr0 interface, enabling VLAN aware:

Then hit "OK" and then "Apply Configuration".

This will make vmbr0 a trunking bridge, meaning that it will pass all VLAN for all VLANs.

Editing the network interfaces in the GUI, like we just did, makes changes in /etc/network/interfaces. There, we can what changes have been made:

iface enp1s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.17/24
        gateway 192.168.1.1
        bridge-ports enp1s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes   <-- new line
        bridge-vids 2-4094      <-- new line

As you can see, our vmbr0 now passes traffic for all vlans in the 2 to 4094 range. Great!

If you wanted, you can set what specific vlans you want to pass by modifying the last line:

bridge-vids 2,12,24

This will now pass traffic for vlans 2, 12 and 24.

This configuration will work out of the bat. You can now create containers and VMs with the vmbr0 as the bridge and set VLAN IP addresses to them.

But what if you ALSO want your node to have an IP address in one of those VLANs? Keep reading!

The VLAN interface

The previous setup allows us to assign VLAN IPs to containers and VMs, but you might also want to assign a specific VLAN IP to the Proxmox Node?

It's easy!

First we need to edit the vmbr0 interface and remove the static IP information, like this:

Then we hit "Create" and create a VLAN interface:

Insert your VLAN data

In the name, by setting vlanX, where "X" is the VLAN number, Proxmox already detects it and puts it into the VLAN field
Then just fill out the IP you want to assign to this Proxmox node in your VLAN and the gateway
Finally, set the VLAN raw device to your linux bridge vmbr0 in my case

Then hit "Apply Configuration" to set the changes.

Note that this will immediately configure your node on the VLAN IP you just set, so you might want to navigate to that new IP to get to the GUI again.

If we now have a look at /etc/network/interfaces:

iface enp1s0 inet manual

auto vmbr0
iface vmbr0 inet manual
        bridge-ports enp1s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vlan2
iface vlan2 inet static
        address 192.168.2.12/24
        gateway 192.168.2.1
        vlan-raw-device vmbr0

We can see how we have a new interface that lets the Proxmox node live under the VLAN 2 IP address 192.168.2.12

And that's pretty much it!

One last note: When creating containers and VMs, the interface you need to select is still vmbr0, not the VLAN interface!

Master Docker logging with Loki and Grafana

Daniel — Tue, 15 Oct 2024 16:31:06 +0000

Introduction
- My approach
How does it work?
1. Setting up Loki
- 1.1 Create a directory for our configuration files
- 1.2 Create a loki-config.yaml file
- 1.3 Create a docker-compose.yaml file
- 1.4 Start Loki!
- 1.5 Exposing Loki to the internet
2. Setting up the Loki Docker plugin
- 2.1 Install the Docker loki plugin
- 2.2 Configure the Docker daemon
- 2.3 Restart Docker
3 Grafana!
- 3.1 Install Grafana
- 3.2 Connect Grafana to Loki
- 3.3 Query the logs
Conclusion

Introduction

I've previously written about how to centralize and visualize logs in Grafana with Loki for apps that write logs to files here.

Doing the same with Docker is SO MUCH SIMPLER.

My approach

You will see other guides showing you how to set up everything in a single machine. While this might be convenient for people running all their Docker stuff in a single server, it's not the best practice for a production environment.

I will show you how to set up everything in a way that you can scale horizontally. With Grafana, Loki and Promtail each running in isolation of each other.

I.e.: You can have promtail in a server running Docker containers for a web app, sending the logs to a Loki instance running in another server and then visualize everything in Grafana running in a third server.

How does it work?

The set up is really similar to the one I wrote about in the previous article, but with a few differences:

The main difference is that, with this setup, we have the Docker daemon sending the logs directly to Loki, without having to configure Promtail at all!

This has a few advantages:

You don't need to mess around with log files and their locations
Docker takes care of sending the logs to Loki
Querying the logs becomes SO MUCH EASIER in Grafana as we can query by container name, image, compose project, etc.

1. Setting up Loki

If you already have a Loki instance and are only interested in setting up the Docker Loki plugin, you can skip this section.

As I said in the introduction, I like to set up everything in separate servers (I use Proxmox LXCs for this), so here's how I set up Loki in a Ubuntu + Docker server in my home lab.

Note that this setup also works just fine if you want to do it all in one machine. Just set up the Loki and Grafana services in the same docker compose file.

1.1 Create a directory for our configuration files

cd && mkdir loki && cd loki

1.2 Create a `loki-config.yaml` file

nano loki-config.yaml

This is the configuration code for loki. I explain what each part does below.

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /tmp/loki
  instance_addr: 127.0.0.1
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-05-15
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

analytics:
  reporting_enabled: false

limits_config:
  retention_period: 30d

compactor:
  working_directory: /tmp/loki/retention
  delete_request_store: filesystem
  retention_enabled: true
  retention_delete_delay: 2h

What does each part do?

  auth_enabled: false

If Loki requires authentication or not. If you're exposing Loki through a reverse proxy or something similar to the internet, you should enable this.

  server:
    http_listen_port: 3100

The port where Loki will receive logs from Docker (or Promtail) and also make them accessible to Gragana.

  common:
    path_prefix: /tmp/loki
    instance_addr: 127.0.0.1
    storage:
      filesystem:
        chunks_directory: /tmp/loki/chunks
        rules_directory: /tmp/loki/rules
    replication_factor: 1
    ring:
      kvstore:
        store: inmemory

This section tells Loki to store everything in the /tmp/loki directory.

That it is running in the local machine (127.0.0.1) and that it should use the filesystem (which just means the local disk) as the storage engine.

We also tell it to just save 1 replica of the data and that it should use an in-memory ring to store the data until it's saved to disk.

  query_range:
    results_cache:
      cache:
        embedded_cache:
          enabled: true
          max_size_mb: 100

This section tells Loki to cache the results of queries in memory. This is optional, but I like to have it enabled since it improves performance when querying the logs from Grafana.

The max_size_mb is the maximum size of the cache in megabytes. The docs state 100 MB is a good starting point, adjust this as needed.

  schema_config:
    configs:
      - from: 2020-05-15
        store: tsdb
        object_store: filesystem
        schema: v13
        index:
          prefix: index_
          period: 24h

This is the schema for the logs. It tells Loki to accept logs from a specific date onwards (I usually just set a random date in the past since all my logs will be from the here on out).

It tells it to use the tsdb storage engine (time series database). We also tell it to use the filesystem to store other data.

In the index section, we tell Loki to use the index_ prefix for the index files and to do so every 24 hours.

  ruler:
    alertmanager_url: http://localhost:9093

  analytics:
    reporting_enabled: false

Here we're telling Loki to use the local alertmanager instance and to not report any analytics since I'm not going to use them.

  limits_config:
    retention_period: 30d

  compactor:
    working_directory: /tmp/loki/retention
    delete_request_store: filesystem
    retention_enabled: true
    retention_delete_delay: 2h

Finally, we tell Loki to keep the logs for 30 days and to delete older logs. We set a delay of 2 hours for the deletion.

Note that limits_config won't work unless we set up the compactor. The compactor does what the name sugests, it compacts the logs into chunks and is in charge of deleting old data.

1.3 Create a `docker-compose.yaml` file

nano docker-compose.yaml

Paste the following content:

services:
  loki:
    restart: unless-stopped
    image: grafana/loki:latest
    ports:
      - '3100:3100'
    volumes:
      - ./loki-config.yml:/etc/loki/local-config.yaml
      - loki-data:/loki
    command: -config.file=/etc/loki/local-config.yaml

volumes:
  loki-data:

Here we expose the http port for Loki and tell Docker to use the loki-config.yaml file for the configuration.

We also create a volume for the data generated by Loki so it can persist even if the container is removed.

Even though you could run this single container with docker run, I like to use docker-compose so I don't have to remember the command to start the container.

1.4 Start Loki!

docker compose up -d

If everything went well, you should be able to access Loki at http://your-server-ip:3100.

If you have servers in multiple locations (aws, digital ocean, home lab, etc), you can create a reverse proxy with NGINX or a similar service to expose Loki to the internet. If so, I recommend either enabling authentication in the loki-config.yaml file or allowing only your servers to access Loki via the reverse proxy or firewall.

Now you can point all your Docker servers to point to this Loki instance!

1.5 Exposing Loki to the internet

If all your Docker servers are running in the same network, just note down the IP address of the server running Loki.

If your Docker servers are running in an external server to where you installed Loki (i.e. Loki is in your Home Lab at home and your Docker server is in a cloud provider), you can expose Loki to the internet by running a reverse proxy.

I won't go into detail on how to do this, there are many guides on using NGINX, Traefic, Cloudflare tunnels, etc. Just use what you're comfortable with.

2. Setting up the Loki Docker plugin

I'll assume you already have one or multiple Docker containers running in a server somewhere.

You can repeat this step for every server running Docker.

2.1 Install the Docker loki plugin

This is the easiest part. Run the following command:

docker plugin install grafana/loki-docker-driver:3.3.2-amd64 --alias loki --grant-all-permissions

Change -amd64 to -arm64 in the image tag for ARM64 hosts.

Check here for the latest version of this command.

If the command was successful, you should see the plugin listed when you run docker plugin ls.

2.2 Configure the Docker daemon

Now we need to tell Docker to use the Loki plugin to send the logs to Loki.

We need to create a daemon.json file in /etc/docker/ with the following content:

{
  "log-driver": "loki",
  "log-opts": {
    "loki-url": "http://localhost:3100/loki/api/v1/push",
    "loki-batch-size": "400"
  }
}

This tells Docker that it should use the loki log driver instead of the default one and sends the logs to the Loki instance.

Again, change the loki-url field to the correct URL for you.

loki-batch-size is optional, but I like to set it to 400, meaning it will send 400 logs at a time to Loki. Not too many, not too few.

2.3 Restart Docker

This is the command for Ubuntu/Debian. If you're on a different OS, just Google how to restart the Docker service for your OS.

sudo systemctl restart docker

IMPORTANT!

Seems that the Loki plugin won't send logs for existing docker containers.

You will have to re-create existing containers for them to start sending logs to loki. Not just restart them. You can use the docker flag --force-recreate for this task.

Great! Now we have all the Docker containers in this machine sending their logs to our Loki instance!

3 Grafana!

This is the final and coolest part!

3.1 Install Grafana

If you already have Grafana set up, you can skip ahead to 3.3 where I show how easy is to query the logs.

It's as simple as running a container:

docker run -d --name=grafana -p 3000:3000 grafana/grafana

You can access Grafana at http://your-server-ip:3000.

3.2 Connect Grafana to Loki

In the left panel, go to "Connections" > "Data Sources".
Click on "+ Add new data source".
Search for "Loki":

Fill in a name and the url of your Loki instance:

Click on "Save & Test". You should see a message saying everything is working correctly.

3.3 Query the logs

In the left panel, go to "Explore".
Select your Loki source at the top:

🎉 Now comes the super cool part of this setup. You can query the logs by container name, compose project, etc.:

This is the best part of this setup, as you can really easily see the logs of a specific container without having to sort through hundreds of files and weird configurations.

Conclusion

I love this setup, I deploy my stuff mainly with Docker and having all the logs centralized in a single place makes my life so much easier to debug and monitor.

It's also SUPER simple to query specific logs, if I need to see the logs of a full compose project, or a specific service or container, I can do it with a simple query.

Hope you found this guide useful! If you have any questions or suggestions, feel free to reach out to me on X!

How to properly redirect www to non-www with Cloudflare

Daniel — Tue, 15 Oct 2024 14:06:41 +0000

Introduction
Step by step
Why do we need a DNS record for www pointing to a random IP address?
Manually adding the redirect rule
Why do I get a DNS_PROBE_FINISHED_NXDOMAIN error?

Introduction

I've read countless tutorials and watched many YouTube videos until I found the correct way to implement a www redirect in Cloudflare.

When you have a website, you may want to redirect the www version of your domain to the non-www version. This is important for SEO, since search engines prefer a single source, and to avoid duplicate content issues. In this article, we will show you how to properly redirect www to non-www with Cloudflare.

Step by step

Login to Cloudflare
Go to the domain you want to set up the redirect
Go to "DNS" > "Records" and remove any www record that you have
Now add a new A record with the name www and the IPv4 address 192.0.2.1 (this is the step that everyone is missing)
Make sure to leave the "Proxied" option on
Go to "Rules" > "Redirect Rules"
Hit "+ Create Rule"
Under "Create new Single Redirect" there is a "Redirect from WWW to Root" snippet. Click on "Create a Rule" in that snippet
Change the Rule name if you want and hit "Deploy"

That's it! You have successfully redirected www to non-www with Cloudflare.

IMPORTANT! You might see that these rules aren't working in your browser. This is because the browser caches the redirects so it might take a while for your browser to update.

To make sure that it's working:

Try it from a different browser/laptop/phone

Use this command: curl -I https://www.example.com and you should see a 301 Moved Permanently and a location:... with the destination of the redirect in the response

Why do we need a DNS record for www pointing to a random IP address?

192.0.2.1 is just a dummy IP address, you could literally use any IP address you want.

This particular address is commonly used in local networks so, as a public IP, it won't really go anywhere.

The reason we need to add this record is because Cloudflare needs to have a record for the www subdomain in order to create the redirect rule. Otherwise it will give a DNS_PROBE_FINISHED_NXDOMAIN error as the 'Internet' will think that there is nothing matching that DNS record (www.example.com).

It's like going to the movies but not even having the movie you want to watch.

The other way around, if you go and there's the movie, the guy at the entrance will point you to the correct room number.

If you want to learn more about this you can check https://developers.cloudflare.com/dns/manage-dns-records/reference/proxied-dns-records/

Manually adding the redirect rule

If you prefer to manually add the redirect rule, you can do so by creating a new rule under "Rules" > "Redirect Rules" and filling in the following fields:

If incoming requests match... - Select "Wildcard pattern"
Request URL: https://www.*
Target URL: https://${1}
Status Code: 301

What this does is get the asterisks part of www request and use it in the target URL. This way, if someone goes to www.example.com, they will be redirected to example.com.

Why do I get a `DNS_PROBE_FINISHED_NXDOMAIN` error?

If you get this error, it means that there is no DNS record for the www subdomain.

Make sure you have added the A record with the name www and a dummy IP address like 192.0.1.2 (See step 4 of the guide above).

How to deploy Django in a subdirectory with Docker, NGINX and Whitenoise

Daniel — Sat, 21 Sep 2024 09:41:05 +0000

Introduction
My setup
The problem
The solution
- 1. Configure django
- 2. Configure Static Files
- 3. Admin login
- 4. Configure Nginx
Conclusion

Introduction

If you are here, you've probably faced the same challenged as I did and spent countless hours on google trying to find how to make a django application work in a subdirectory.

Look no further, here's all you need to know to make it work.

My setup

I want my frontend to be served in the root / of my domain and the backend to be served in the /api subdirectory of the same domain:

https://example.com/ -> Frontend
https://example.com/api/ -> Backend

I'm going to quickly run you through my setup so you can understand the context of this article.

I have a monorepo with two folders: backend and frontend
backend: django REST API that serves data and an admin page
frontend: React frontend that consumes the API
Docker to containerize the application
Nginx to serve the Frontend application and serve the django API in a subdirectory

project/
├── backend/
│   ├── Dockerfile
│   └── other django stuff...
├── frontend/
│   ├── Dockerfile
│   ├── default-nginx.conf
│   └── other frontend stuff...
└── docker-compose.yml

This is my sample docker compose file:

services:
  backend:
    build:
      context: ./backend
      dockerfile: Dockerfile
    ports:
      - "8000:8000"

  frontend:
    build:
      context: ./frontend
      dockerfile: Dockerfile
    ports:
      - "8080:80"
    depends_on:
      - backend

The problem

When I tried to deploy the application to a subdirectory, I faced a few challenges:

The API was not working
The static files were not being served
The admin page was not working

The solution

1. Configure django

First, we need to configure Django to let it know that we are running in a subdirectory.

Add the following to your settings.py:

FORCE_SCRIPT_NAME = '/api'

Important!

Having the leading slash is important to make the static files settings below work correctly.

FORCE_SCRIPT_NAME is a setting that tells Django to prepend the given value to all URLs generated by Django.

You can find more documentation here.

2. Configure Static Files

We need to configure Django to serve the static files correctly in the subdirectory, otherwise, they won't be found.

You're probably familiar with the following settings in settings.py:

STATIC_ROOT = BASE_DIR / "staticfiles"
STATIC_URL = "/static"

You need to modify it to include the subdirectory:

STATIC_ROOT = BASE_DIR / "staticfiles"
STATIC_URL = "{}/static/".format(FORCE_SCRIPT_NAME)

If you are using Whitenoise you will also need to include the following setting:

WHITENOISE_STATIC_PREFIX = "/static/"

There is a PR that has been merged that allows you to not have to set this setting, but it's not released as of the time of writing this article.

You can find the PR here.

3. Admin login

The settings above should make the admin page work, but some people have reported encountering the admin page redirecting them to the root of the domain after login.

There's a setting that allows you to override the default admin login redirect:

LOGIN_REDIRECT_URL = '/api/admin/'

You can find more information about this setting here.

4. Configure Nginx

Here comes to most tricky part in my search, it took me a while to figure out how to configure Nginx to serve the backend in a subdirectory.

I will explain the configuration in detail, but here's the final configuration:

server {
    listen 80;
    server_name localhost;

    root /usr/share/nginx/html;
    index index.html;

    # React frontend
    location / {
        try_files $uri $uri/ /index.html;
    }

    # Django API
    location /api/ {
        proxy_pass http://backend:8000/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

I will assume you have basic knowledge of how nginx works so I will ignore the basic configuration and go straight to the relevant parts.

For context, I have built my frontend to /usr/share/nginx/html previously. Nginx is serving those files in the root of the domain.

Let's break it down:

location /api/: This is the location block that will serve the Django API in the subdirectory /api/. IMPORTANT! Note that it has both a trailing and leading slash. This is important to make django work correctly in a subdirectory.
proxy_pass http://backend:8000/: This is the most important part. It tells Nginx to proxy all requests to /api/ to the backend service running with Docker on port 8000. Cool thing about docker is that we can reference the service by its name, in this case backend.

Now to the headers:

Host $host: This sets the Host header to the value of the host header of the request. It basically passes the host header from the client to the backend.
X-Real-IP $remote_addr: This sets the X-Real-IP header to the value of the remote address of the client. This is useful for the backend to know the real IP of who is making the request.
X-Forwarded-For $proxy_add_x_forwarded_for: This header maintains a chain of IP addresses a request has traversed. It's similar to X-Real-IP but provides more comprehensive information.
X-Forwarded-Proto $scheme: This header informs the backend about the protocol (HTTP or HTTPS) used by the client to connect to Nginx. Without this, your Django application wouldn't know if the original request was secure (HTTPS) or not.

Conclusion

That's it! You should now have your Django application running in a subdirectory with Docker and Whitenoise.

I really hope this article saves you some time and headaches!

How to create an API endpoint in astro

Daniel — Fri, 14 Jun 2024 14:32:12 +0000

Introduction
What's the setup?
- 1. Ready your Astro project
- 2. SSR
- 2.1 Adapter
- 2.2 server or hybrid
- 2.3 A note on environment variables
- 2.4 A note on console.log
- 3. File structure
The API endpoint
- 1. Create the file
- 2. The code
The form

Introduction

Gone are the days when you had to create a separate API server to serve your frontend application. With Astro, you can create API endpoints directly in your app. Which means you can even create a full-stack application with just one codebase.

Personally, I use these endpoints for simple actions such as:

Contact Forms
Newsletter subscriptions
User registration
Even user authentication sometimes

It's also useful for when you need to fetch and process data from a source that requires authentication (I.e. an API key) and you don't want to expose that key in the frontend.

In this article we'll go through the steps to create an API endpoint in Astro.

What's the setup?

In this article we'll create a simple API endpoint that creates a contact in Brevo, which is mostly what I use these endpoints for. You can replace this with any other service you want to interact with.

1. Ready your Astro project

If you're new to Astro or haven't set up an Astro project yet, I recommend you to check out the official documentation.

You can also start with one of their themes.

2. SSR

In order to have a working API endpoint that works at runtime, you need to enable SSR (Server Sider Rendering).

SSR allows you to have the app run on the server before it gets to the client.

2.1 Adapter

For this, you will need an adapter.

What an adapter does is it allows you to run your SSR Astro App in different environments. For example, you can run it in a serverless environment like Vercel or Netlify, or in a Node.js server.

You can see a list of official and community adapters here.

For this article, I'll use the @astrojs/adapter-node adapter since I host my side in a Node docker container.

Super easy to install:

npx astro add node

2.2 `server` or `hybrid`

Astro allows you to run SSR in 2 ways:

server: On-demand rendered by default. Basically uses the server for everything. Use this when most of your site should be dynamic. You can opt-out of SSR for individual pages or endpoints.
hybrid: Pre-rendered to HTML by default. It does not pre-render the page on the server. Use this when most of your site should be static. You can opt-in to SSR for individual pages or endpoints.

For my usecase where most of my site is static (it's a landing page after all) I use hybrid:

// astro.config.mjs

import { defineConfig } from "astro/config";
import node from "@astrojs/node";

export default defineConfig({
  output: "hybrid",
  adapter: node({
    mode: "standalone",
  }),
});

2.3 A note on environment variables

If you're used to Astro, you know that you can use environment variables by calling:

const MY_VARIABLE = import.meta.env.VARIABLE_NAME;

However, because Astro is static by default, what this really does is get the environment variable at build time. Then, the variable is hardcoded into the build code, which means that if you change the environment variable after the build, it won't be reflected in the code.

If you use SSR it works differently, import.meta.env won't work since it's available at build time but not at runtime on the server.

You will need to use process.env instead:

const MY_VARIABLE = process.env.VARIABLE_NAME;

BUT WAIT!

There's another catch, and that's that process.env is not available with npm run dev, which means your code will crash when you try to run it locally.

The solution:

const MY_VARIABLE = import.meta.env.VARIABLE_NAME ?? process.env.VARIABLE_NAME;

This code will try to get the environment variable from import.meta.env first, and if it's not available it will try to get it from process.env. This way, your code will work both in development and production.

2.4 A note on `console.log`

If you're used to using console.log to debug your code, you'll know that it will show up in the browser console when you're running the app in development mode.

When using console.log in an SSR component, because it runs on the server, the logs will show up in the terminal where you're running the app.

So if you're looking for your logs and can't find them, check the terminal where you're running the app.

3. File structure

The full functionality needs 2 files:

A .js or .ts API endpoint file that lives in the src/pages/api directory.
A form that gets the user input and sends it to the API endpoint. I personally like to do this in a .tsx file because I can then use the full power of react (react-hook-form and zod) to handle the form. Place this form wherever you like, I like having all my forms in src/components/forms.

That's pretty much it! The form will send the data to our API endpoint, which will then process it and send it to Brevo.

The API endpoint

1. Create the file

Let's create the API endpoint that will send the data to Brevo.

You can create this endpoint wherever you want under the src/pages/ directory depending on where you want it to be accessible.

For instance, I like my endpoints to be accessible under /api/ so I create a src/pages/api/ directory.

So my endpoint file will be src/pages/api/create-brevo-contact.ts.

This means that I will be able to access it at http://mydomain.com/api/create-brevo-contact.

2. The code

Your API endpoint code should have a pretty simple structure:

// SSR API endpoint template

// Tell Astro that this component should run on the server
// You only need to specify this if you're using the hybrid output
export const prerender = false;

// Import the APIRoute type from Astro
import type { APIRoute } from "astro";

// This function will be called when the endpoint is hit with a GET request
export const GET: APIRoute = async ({ request }) => {
  // Do some stuff here

  // Return a 200 status and a response to the frontend
  return new Response(
    JSON.stringify({
      message: "Operation successful",
    }),
    {
      status: 200,
    }
  );
};

Following the template above, this is a simple POST API endpoint to create a contact in Brevo. Everything is commented so you can understand what's going on:

// src/pages/api/create-brevo-contact.ts

// Because I chose hybrid, I need to specify that this endpoint should run on the server:
export const prerender = false;

// Import the APIRoute type from Astro
import type { APIRoute } from "astro";

// This is the function that will be called when the endpoint is hit
export const POST: APIRoute = async ({ request }) => {
  // Check if the request is a JSON request
  if (request.headers.get("content-type") === "application/json") {
    // Get the body of the request
    const body = await request.json();

    // Get the email from the body
    const email = body.email;

    // Declares the Brevo API URL
    const BREVO_API_URL = "https://api.brevo.com/v3/contacts";

    // Gets the Brevo API Key from an environment variable
    // Check the note on environment variables in the SSR section of this article to understand what is going on here
    const BREVO_API_KEY =
      import.meta.env.BREVO_API_KEY ?? process.env.BREVO_API_KEY;

    // Just a simple check to make sure the API key is defined in an environment variable
    if (!BREVO_API_KEY) {
      console.error("No BREVO_API_KEY defined");
      return new Response(null, { status: 400 });
    }

    // The payload that will be sent to Brevo
    // This payload will create or update the contact and add it to the list with ID 3
    const payload = {
      updateEnabled: true,
      email: email,
      listIds: [3],
    };

    // Whatever process you want to do in your API endpoint should be inside a try/catch block
    // In this case we're sending a POST request to Brevo
    try {
      // Make a POST request to Brevo
      const response = await fetch(BREVO_API_URL, {
        method: "POST",
        headers: {
          accept: "application/json",
          "api-key": BREVO_API_KEY,
          "content-type": "application/json",
        },
        body: JSON.stringify(payload),
      });

      // Check if the request was successful
      if (response.ok) {
        // Request succeeded
        console.log("Contact added successfully");

        // Return a 200 status and the response to our frontend
        return new Response(
          JSON.stringify({
            message: "Contact added successfully",
          }),
          {
            status: 200,
          }
        );
      } else {
        // Request failed
        console.error("Failed to add contact to Brevo");

        // Return a 400 status to our frontend
        return new Response(null, { status: 400 });
      }
    } catch (error) {
      // An error occurred while doing our API operation
      console.error(
        "An unexpected error occurred while adding contact:",
        error
      );

      // Return a 400 status to our frontend
      return new Response(null, { status: 400 });
    }
  }

  // If the POST request is not a JSON request, return a 400 status to our frontend
  return new Response(null, { status: 400 });
};

That's it, you now have a working API endpoint that will create a contact in Brevo when hit with a POST request!

The form

As a bonus, I also want to show you how I code my forms to make them responsive.

For this example, I'll create a simple form, with only an email field and a submit button, that will send the email a user inputs to the API endpoint we created.

Here's the code:

Note that this code is using shadcn ui components for the HTML, you might need to replace them with your own components.

// WaitlistForm.tsx

// Zod validation stuff
const WaitlistFormSchema = z.object({
  email: z
    .string()
    .min(1, "Please enter a valid email")
    .email("Please enter a valid email"),
});

type WaitlistFormValues = z.infer<typeof WaitlistFormSchema>;

const WaitlistForm = () => {
  // Hooks to check the status of the form
  const [isSubmitting, setIsSubmitting] = useState(false);
  const [isSuccess, setIsSuccess] = useState(false);
  const [error, setError] = useState("");

  // React Hook Form stuff
  const form = useForm<WaitlistFormValues>({
    resolver: zodResolver(WaitlistFormSchema),
    defaultValues: {
      email: "",
    },
  });

  // Function that sends the data to the API endpoint when the form is submitted
  const onSubmit = async (data: WaitlistFormValues) => {
    setIsSubmitting(true);

    // Ping out API endpoint
    const response = await fetch("/api/create-brevo-contact", {
      method: "POST",
      headers: {
        "content-type": "application/json",
      },
      body: JSON.stringify(data),
    });

    // If successful, reset the form and show a success message
    if (response.ok) {
      form.reset();
      setIsSuccess(true);
    } else {
      // If failed, show error message
      console.error("Failed to add contact");
      setIsSuccess(false);
      setError("There's been an error. Please try again.");
    }

    setIsSubmitting(false);
  };

  return (
    <>
      {isSuccess && (
        <Alert className="mb-3 md:mb-8 bg-green-100 border-green-300">
          <AlertTitle>Thanks!</AlertTitle>
          <AlertDescription>
            We've added you to the waitlist!
            <br />
          </AlertDescription>
        </Alert>
      )}

      {!isSuccess && error && (
        <Alert className="mb-8 bg-red-100 border-red-300">
          <AlertTitle>Error</AlertTitle>
          <AlertDescription>{error}</AlertDescription>
        </Alert>
      )}

      <Form {...form}>
        <form
          onSubmit={form.handleSubmit(onSubmit)}
          className="space-y-4 md:space-y-8"
        >
          <FormField
            control={form.control}
            name="email"
            render={({ field }) => (
              <FormItem>
                <FormLabel>Email</FormLabel>
                <FormControl>
                  <Input
                    className="bg-transparent"
                    placeholder="email@gmail.com"
                    {...field}
                  />
                </FormControl>
                <FormMessage />
              </FormItem>
            )}
          />

          <Button type="submit" disabled={isSubmitting}>
            <Loader2
              className={`w-6 h-6 mr-2 animate-spin ${
                isSubmitting ? "block" : "hidden"
              }`}
            />
            Submit
          </Button>
        </form>
      </Form>
    </>
  );
};

export default WaitlistForm;

Fixing Most TypeScript Intellisense issues in VS Code

Daniel — Mon, 13 May 2024 14:34:40 +0000

I'm having this issue constantly, specially with devcontainers or other virtual environments, where I suddenly have hundreds, if not thousands of intellisense errors all over my TS code while the app works just fine.

This is mostly due to VSCode not selecting the correct TS Version automatically. Even if you have "typescript.tsdk": "node_modules/typescript/lib" in your .vscode/settings.json it mostly fails.

To solve this:

Open any typescript (*.ts) file.
Hit F1 or Ctrl + Shift + P
Search for "TypeScript: Select TypeScript Version..." and hit enter
Select "Use Workspace Version" (even if the version number is the same as VSCode's version)
Reload VSCode (close it and reopen it) for it to take full effect and for it to reprocess all open tabs.

This is usually how I fix the issue.

My Docker stack to deploy a Django + Celery web app

Daniel — Thu, 18 Apr 2024 09:48:50 +0000

Introduction
Project Structure
Stack Diagram
Docker images
- Django image
- Nginx image
- Redis image
Docker Compose
- Environment Variables
- The file
Deploying
- Building
- Deploying with Docker Compose

After many iterations, this is my current process to deploy a django application.

The cool thing is that I'm now deploying with only an .env file and nothing else.

Note that this is just a single instance of the stack, without Kubernetes or any kind of load balancer.

Introduction

My stack consists on:

A Postgres database
A Redis database
A django instance
Celery beat and workers

I won't go into details of how to set up anything like this here, if you want to learn more about celery in their docs.

Project Structure

To understand the docker-compose.yml file below, it's important to see how I structure my django project:

MY-DJANGO-PROJECT/
├── core  # settings.py lives here
├── app1/
│   ├── migrations
│   ├── models.py
│   └── ...
├── app2/
│   ├── migrations
│   ├── models.py
│   └── ...
├── data  # A data directory where I store stuff like logs
├── nginx/
│   ├── certs/
│   │   ├── fullchain.pem
│   │   └── privkey.pem
│   ├── conf/
│   │   ├── default.conf
│   │   ├── prod.conf
│   │   └── staging.conf
│   └── Dockerfile
├── Dockerfile
├── entrypoint-django.sh
├── entrypoint-beat.sh
├── entrypoint-worker.sh
├── Pipfile
└── ...

Stack Diagram

To visualize the flow better here's a diagram that describes how everything is interconnected:

Docker images

The stack, though it might seem complicated, is only composed of 3 images, 2 of which are custom:

Django image

This is a custom image built from python.

This image will be used for django, celery workers and celery beat containers.

Here's the Dockerfile for it:

# Python as the base image
# I use bullseye because I'm more comfortable with it
# but you can use Alpine for a more lightweight container
FROM python:3.11-bullseye

# Exposes port 8000
# Make sure to change this to your used port
EXPOSE 8000

# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1

# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1

# Working directory
WORKDIR /app/backend

# Install pipenv
# This is not necessary if you use pip in your code
RUN pip install -U pipenv

# Install pipenv requirements
# Turns the Pipfile to a requirements.txt
# so it can be installed globally with pip
COPY Pipfile Pipfile.lock /app/backend/
RUN pipenv requirements > requirements.txt
RUN pip install -r requirements.txt
RUN rm -rf ./Pipfile ./Pipfile.lock

# Copy all the code over
COPY . .

# Create the media directory
RUN mkdir -p /app/backend/media

# Create a volume for the media directory
VOLUME /app/backend/media

# Create a volume for the static directory
VOLUME /app/backend/django_static

# Make the entrypoint scripts executable
# There's one entrypoint for each service that uses this image
RUN chmod +x /app/backend/entrypoint-django.sh
RUN chmod +x /app/backend/entrypoint-worker.sh
RUN chmod +x /app/backend/entrypoint-beat.sh

# Set the default entrypoint in case this Dockerfile is run
# by itself
ENTRYPOINT ["/app/backend/entrypoint-django.sh"]

These are the entry point files for each service:

django entry point

#!/bin/bash

# Migrate any new migrations to the database on deployment
echo "Migrating..."

python manage.py migrate --no-input

# Collect static files
echo "Collecting static files..."

python manage.py collectstatic --no-input

# Ensure the data directory exists
# I use the data directory to store files such as logs
mkdir -p data

# Start gunicorn
echo "Starting server..."

gunicorn core.wsgi:application --forwarded-allow-ips="*" --bind 0.0.0.0:8000

Worker entry point

#!/bin/sh

# Wait until the backend directory is created
until cd /app/backend
do
    echo "Waiting for server volume..."
done

# run a worker
# I like having only one task per worker but you can change it
# by increasing the concurrency
echo "Starting celery worker..."
celery -A core worker -l info --concurrency 1 -E

Beat entry point

#!/bin/sh

# Wait until the server volume is available
until cd /app/backend
do
    echo "Waiting for server volume..."
done

# run celery beat
echo "Starting celery beat..."
celery -A core beat -l info

Nginx image

This container serves the application.

I create a custom nginx image that includes my certificates and configuration, so I don't have to copy them over to the server.

Note: I don't use certbot, as I find it more straightfoward to generate the certificates from cloudflare and just store them in the custom image

This means that the image should be secure in a private registry with authentication, otherwise you risk security of your web app.

Here's the Dockerfile for it:

FROM nginx:stable-bullseye

# Export ports 80 and 443
EXPOSE 80
EXPOSE 443

# Copy the nginx configuration files to the image
COPY ./conf/default.conf /etc/nginx/conf.d/default.conf
COPY ./conf/prod.conf /etc/nginx/conf.d/prod.conf
COPY ./conf/staging.conf /etc/nginx/conf.d/staging.conf

# Copy the CloudFlare Origin CA certificate to the image
COPY ./certs/fullchain.pem /etc/nginx/certs/fullchain.pem
COPY ./certs/privkey.pem /etc/nginx/certs/privkey.pem

Redis image

I just use the default Redis image for this.

Just want to note that, because this is a single instance deployment, I like deploying Redis directly here as I find it's enough.

It is recommended, though, to spin up a Redis database somewhere more centralized.

Docker Compose

Environment Variables

Before I get into the gist of the Docker Compose file here are some environment variables I put in my .env file for deployment:

DOCKER_REGISTRY: My private, authentication enabled, docker registry where I upload the build images
DJANGO_DOCKER_IMAGE: The name I give the django image
NGINX_DOCKER_IMAGE: The name I give the NGINX image
DOCKER_TAG: Usually the version I want to deploy, i.e.: 1.5 or latest

The file

version: "3"

services:
  redis:
    container_name: redis
    restart: unless-stopped
    image: redis:7.2.0-alpine
    expose:
      - 6379

  backend:
    restart: unless-stopped
    image: ${DOCKER_REGISTRY}/${DJANGO_DOCKER_IMAGE}:${DOCKER_TAG}
    env_file:
      - ./.env
    entrypoint: /app/backend/entrypoint-django.sh
    ports:
      - 8000:8000
    volumes:
      - ./data:/app/backend/data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/healthcheck/"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s 

  worker:
    restart: unless-stopped
    image: ${DOCKER_REGISTRY}/${DJANGO_DOCKER_IMAGE}:${DOCKER_TAG}
    env_file:
      - ./.env
    entrypoint: /app/backend/entrypoint-worker.sh
    volumes:
      - ./data:/app/backend/data
    depends_on:
      backend
        condition: service_healthy
      redis
        condition: service_started

  beat:
    restart: unless-stopped
    image: ${DOCKER_REGISTRY}/${DJANGO_DOCKER_IMAGE}:${DOCKER_TAG}
    env_file:
      - ./.env
    entrypoint: /app/backend/entrypoint-beat.sh
    volumes:
      - ./data:/app/backend/data
    depends_on:
      backend
        condition: service_healthy
      redis
        condition: service_started

  nginx:
    restart: unless-stopped
    image: ${DOCKER_REGISTRY}/${NGINX_DOCKER_IMAGE}:${DOCKER_TAG}
    ports:
      - 80:80
      - 443:443
    depends_on:
      backend
        condition: service_healthy

As you can see, the compose file has the 5 services, Redis, django, celery worker, celery beat and NGINX.

Deploying

Building

First I build the images and push them to the registry. Before, I did this manually, now I use a GitHub action. You can learn more about this automation here.

Deploying with Docker Compose

Then I head to the server where I want to deploy this. Make sure that the .env file is updated and then just:

docker compose down: Spin the old instance down
docker system prune -a -f: This makes sure I remove the latest image to force the download of the new one from the registry.
docker compose up --scale worker=5 -d: Spin the new instance up That's it!

Automatically build Docker images with GitHub Actions

Daniel — Mon, 08 Apr 2024 12:53:01 +0000

Introduction
Setting up a Self Hosted Runner
Structure of a GitHub Action
- A (simple) GitHub action has 4 main parts:
GitHub Action to build Docker Images
- Process overview & requirements
- 1. Environment variables and secrets
- 2. Triggers
- 4. Job Setup
Full GitHub Actions template to release a docker image
Conclusion

Introduction

When I released a new version of a web app, this was the process I followed, which might sound familiar to you:

I merged everything into the master branch
SSH into the deployment server
git fetch
git checkout
git pull
Build docker images (this took loooong and once the web app was big enough, it would fail)
Then I would have to build the image in my computer or another specific server to avoid the production server crashing
Push that to a registry
Go back to the server and pull from the registry
Realize you did not update an environment variable
AGHHHJJJ!!!

Here's how I do it now:

Push a new semantic versioning (semver) release tag. i.e. 1.0.1 from any branch I like
That's it!

This works perfectly with the perfect Git strategy

Setting up a Self Hosted Runner

If you already know everything about GitHub actions and just want to know how the structure to build docker images, proceed to GitHub Action to build Docker Images

Definition: A runner is an instance that runs whatever you want in your GitHub action. From making it print Hello World to building and deploying apps, to making you coffee (seriously). Think of it as a user that runs whatever you tell it to in a remote server (linux, windows...).

GitHub gives you 2 options:

Use one of their runners for free up to X minutes every month and then pay for it
Spin up a self hosted runner and use that for free

If you go with the first option skip this, otherwise continue reading.

Since I already have a home lab and setting up a runner is quite easy, self hosted was the way to save a few bucks. You can also do this with an old computer or even a raspberry pi you have laying around.

If you want to learn more about self hosted runners look into GitHub's documentation.

This is what you need to set up a self hosted runner:

Select the OS you want your runner to be. I'm a fan of Ubuntu and I can pretty much do everything I need with it, so I went with a Linux runner.
Make sure you have any necessary packages installed in the runner. For my use case I would need to install Docker first.
Go to your GitHub repo, then Settings
In the left panel hit Actions and then Runners
Select New self-hosted Runner
Select the OS you want to use and architecture
Follow the terminal commands displayed below your selection to finish setting up your runner. I do not want to post the commands here since GitHub updates them every once in a while, so just follow their instructions over there.
Once you finish setting the runner up in your Linux machine you should see it in the Runners page from before and you're ready to use it!

Structure of a GitHub Action

GitHub actions are defined in .yml files. They are basically a set of instructions telling the runner what to do.

They live under .github/workflows/ in your repository and will be live and working once you push them to your main branch.

You can either write them directly with GitHub's web interface or with your favorite code editor and push them to your main branch.

If you use VSCode they have a GitHub Actions extension that comes in really handy to edit the workflows and see what's running

A (simple) GitHub action has 4 main parts:

1. `name`

Easy, this is just the name you want to give the action to track it later on.

name: My GitHub action name

2. `on`

This is what you want to trigger the action on. The most common case is to tricker an action on a push to a specific branch or when opening a pull request. But there are many triggers.

You can see documentation on all the triggers here:

https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows

Here's an example code to:

Be able to trigger the GitHub action manually. For more information on how to trigger this from the GitHub web interface, see "Manually running a workflow."
Trigger the action when there's a push to the my-branch-name and my-other-branch-name branches.

on:
  # Trigger the action manually from the UI
  workflow_dispatch:
  # Trigger the action when pushing to certain branches
  push:
    branches:
      - 'my-branch-name'
      - 'my-other-branch-name'

3. `env`

Here you can set environment variables and secrets for a specific Action. This can be used together with the repository Variables and Secrets in the GitHub website.

If there's something you use in multiple steps or jobs it is useful to set it up here so that you only need to change one line.

env:
  DOCKER_IMAGE_NAME: my-image

4. `jobs`

These are the individual series of steps you want to execute. Each job has a series of sequential steps and every job can be run asynchronously. You can also create a dependency so a job doesn't run until a previous job has finished.

Differences between Jobs and Steps

Jobs are individual, asynchronous tasks. This means that if you have several tasks that can happen asynchronously (at the same time) (i.e.: deploying an app and uploading new documentation) you can put them in separate jobs. If you have several runners you can execute several jobs at the same time.

Steps are actions that happen synchronously. If you have tasks that need to run one after the other (i.e.: Building a docker image and then pushing it to a Docker registry) set them up in steps instead of jobs.

Sample code:

jobs:
  build_docker_images:
    # Job name that shows in the GitHub UI
    name: Build Docker Images
    # Runner to use
    runs-on: self-hosted

    steps:
    - name: Checkout
      uses: actions/checkout@v3

   ...

You can set up as many jobs and steps as you like in a single action file.

GitHub Action to build Docker Images

In this section I will show you how to build a docker image from a Dockerfile and push it to Dockerhub or your private Docker Registry. All by just creating a version tag.

Process overview & requirements

First, if you're using a self-hosted runner, check that it's running and available in the repository you want to have this on. And that you understand the structure of a GitHub action.

Here are the requirements:

When I push a version tag, I want to get the version name (i.e.: 1.0.1), then build the Docker image and tag it with that version number as well as the latest tag (i.e.: my-image:1.0.1 and my-image:latest).

After this, I want to push the image to a private Docker Registry I self-host and remove all data from the runner. But you can also do this to push the docker image to DockerHub.

I also want to store the build cache so that future images build faster.

Here's the process overview:

Trigger the action manually or by pushing a version tag
Git checkout the code from that specific tag
Get the version number from the tag
Build the Docker image
Once the image is built, tag it with the version number and the docker registry of my choosing
Push the image to the registry

1. Environment variables and secrets

Environment variables

In case I ever need to change the registry URL or image name I set the following environment variables to be able to do so easily:

# Workflow environment variables
env:
  DOCKER_IMAGE_NAME: my-image
  DOCKER_REGISTRY_URL: myregistry.domain.com

Secrets

I also want to create these github action secrets in my reo to be able to access the docker registry. I like to create a comment in my workflow file so I don't forget if I move the repository or something.

Please, for the love of God, DO NOT PUT THESE IN YOUR WORKFLOW FILE.

# Required actions secrets:
# - DOCKER_USERNAME: Username for docker registry login
# - DOCKER_PASSWORD: Password for docker registry login

2. Triggers

I want not only for the action to be triggered when I push a version tag, but also manually.

name: Build release Docker image

on:
  # Trigger the action manually from the UI
  workflow_dispatch:
  # Trigger the action when a version tag is pushed
  push:
    tags:
      - '[0-9]+.[0-9]+.[0-9]+' # Push events to matching numeric semver tags, i.e., 1.0.0, 20.15.10

4. Job Setup

I will only require a single job since all my steps need to be sequential.

jobs:
  build_docker_images:
    # Job name that shows in the GitHub Website:
    name: Build Docker Images
    # Runner to use:
    runs-on: self-hosted

    # Begin the steps
    steps:
      - step1
      - step2
      - ...

4.1 Get the code from GitHub

GitHub, and many other services like Docker, provide ready-to-use steps. In this case we're going to use the actions/checkout@v3 step from GitHub, which just fetches the code from the repository.

# Checkout the code
- name: Checkout code
  uses: actions/checkout@v4
  with:
    fetch-depth: 0 # Do not get extra git branches to save time

4.2 Set up docker builder

Like GitHub, Docker provides their own set of ready-to-use steps, in this case there is a builder setup action

- name: Set up Docker Buildx
  uses: docker/setup-buildx-action@v3

4.3 Login into the Docker Registry

- name: Login to Docker Registry
  uses: docker/login-action@v3
  with:
    registry: ${{ env.DOCKER_REGISTRY }}
    username: ${{ secrets.DOCKER_USERNAME }}
    password: ${{ secrets.DOCKER_PASSWORD }}

4.4 Get the version number

Here I just get the tag name (X.Y.Z). Since we constrained the triggers to only trigger on numeric semver tags, we confidently know that the tag will be in the right format: X.Y.Z.

- name: Get the tag name
  id: get_version
  run: echo "VERSION=${GITHUB_REF#refs/tags/}" >> $GITHUB_OUTPUT

4.5 Build and push the docker images

Docker provides a super convenient ready-to-use action for this step.

- name: Build and Push the Docker Image
  uses: docker/build-push-action@v6
  with:
    context: .
    file: ./Dockerfile
    push: true
    tags: |
      ${{ env.DOCKER_REGISTRY }}/${{ env.DOCKER_IMAGE_NAME }}:${{ steps.get_version.outputs.VERSION }}
      ${{ env.DOCKER_REGISTRY }}/${{ env.DOCKER_IMAGE_NAME }}:latest
    cache-from: type=registry,ref=${{ env.DOCKER_REGISTRY }}/${{ env.DOCKER_IMAGE_NAME }}:buildcache
    cache-to: type=registry,ref=${{ env.DOCKER_REGISTRY }}/${{ env.DOCKER_IMAGE_NAME }}:buildcache,mode=max

Since this step is not as straightforward let me explain:

context: .: We're telling the builder to use the root folder of the repository as the build context. This is useful for instance if you have a monorepo with a backend and a frontend directory. You can create two jobs for each where the context changes.
file: ./Dockerfile: We're telling the builder where the Dockerfile is
push: true: We want to push the images as well
The tags section indicates that we want to push two tags of this image. This is also useful if we want to push to multiple directories for instance. Here I use the environment variables we defined at the begining of the workflow and I'm pushing the version tag as well as the latest tag.
cache-from:: This means where we're pulling the build cache from, to make consequent builds faster. In this case we get it from the registry.
cache-to:: Similar to chache-from, we're indicating where the build cache should be stored, in this case it's the registry.

Full GitHub Actions template to release a docker image

Here's the full action code. You can copy and paste it into your YML file and modify it as needed:

# This workflow will build a Docker image
# and push them to a private docker registry when a release tag (i.e.: 1.0.1)
# is pushed to the repository.
#
# It runs on a self hosted runner.
# You can change the runner to 'ubuntu-latest' if you want to use a GitHub hosted runner.
#
# Required actions secrets in the repository:
# - DOCKER_USERNAME: Username for docker registry login
# - DOCKER_PASSWORD: Password for docker registry login

# Script environment variables
env:
  DOCKER_IMAGE_NAME: my-image
  DOCKER_REGISTRY: myregistry.mydomain.com

name: Build and Push Docker Image

# Triggers
on:
  # Trigger the action manually from the UI
  workflow_dispatch:
  # Trigger the action when a version tag is pushed
  push:
    tags:
      - '[0-9]+.[0-9]+.[0-9]+' # Push events to matching numeric semver tags, i.e., 1.0.0, 20.15.10

jobs:
  build-and-push-landing-page:
    name: Build and Push Landing Page Docker Image
    runs-on: self-hosted

    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to Docker Registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.DOCKER_REGISTRY }}
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}

      - name: Get the tag name
        id: get_version
        run: echo "VERSION=${GITHUB_REF#refs/tags/}" >> $GITHUB_OUTPUT

      - name: Build and push Landing Page Docker image
        uses: docker/build-push-action@v6
        with:
          context: .
          file: ./Dockerfile
          push: true
          tags: |
            ${{ env.DOCKER_REGISTRY }}/${{ env.DOCKER_IMAGE_NAME }}:${{ steps.get_version.outputs.VERSION }}
            ${{ env.DOCKER_REGISTRY }}/${{ env.DOCKER_IMAGE_NAME }}:latest
          cache-from: type=registry,ref=${{ env.DOCKER_REGISTRY }}/${{ env.DOCKER_IMAGE_NAME }}:buildcache
          cache-to: type=registry,ref=${{ env.DOCKER_REGISTRY }}/${{ env.DOCKER_IMAGE_NAME }}:buildcache,mode=max

Conclusion

With this set up you now automatically will have Docker images ready for use in the Registry of your choice.

I will create a future article showing how to automatically deploy a docker image once it's built and pushed to a registry.

Connect to WireGuard clients from LAN (with PFSense)

Daniel — Wed, 27 Mar 2024 17:37:11 +0000

Introduction
Problem Setup and Variables
Static Routes
Create a Static Route in PFSense

Introduction

Here's my particular situation:

I'm developing Emilio. Which requires to have an accessible endpoint so that I can query stuff from a Gmail Addon.
I have Nginx Proxy Manager at home routing requests made to my development url to my laptop so I can test this endpoint.
I have a WireGuard VPN server at home that I can use to connect my devices to my local network when I travel.

You might have a similar issue if you have set up a server somewhere, say, your friend's home, and you want that server to be part of your local network.

Problem:
WireGuard clients (my laptop) have a totally different subnet:

Your local subnet (probably): 192.168.0.0 (255.255.255.0 as the mask or /24)
WireGuard subnet: 10.6.0.0/24

This means that, while I can ping LAN devices from my laptop when connected through WireGuard, I cannot do the opposite.

Here's a solution that worked for me.

Problem Setup and Variables

To make this guide clearer I will create fictional addresses:

My laptop's WireGuard IP: 10.6.0.12
WireGuard's server internal IP at home: 192.168.0.59

What I want:
NGINX routing requests from test.domain.com to 10.6.0.12.

Static Routes

I want to tell my NGINX host or any device in my LAN for that matter, where to point when I request the IP 10.6.0.12.

This is a great use case for static routes.

To create a static route you need 3 things:

The Gateway: What's the entry point for the subnet you want to connect to.
The subnet you want to connect to
The subnet mask

In my case:

Gateway: 192.168.0.59
Subnet: 10.6.0.0
Subnet mask: /24 or 255.255.255.0 (they're 2 ways of saying the same)

Create a Static Route in PFSense

I use PFSense at home, so I will demonstrate how to do it in there. If you have another router or firewall just search how to add a static route in your specific model.

Go to System > Routing
In the Gateways tab, add a new gateway.
Give the gateway the name you want and set the IP, in my case 192.168.0.59 (the WireGuard server at home).
Apply the changes.
Now in the Routing menu, go to Static Routes
Hit + Add
In Destination Network add the WireGuard subnet, in my case 10.6.0.0/24
Select the Gateway we just added
Give it a description if you want
Save and apply changes

Done!

Now you should be able to ping your laptop from your LAN. It might take a few minutes though, so be patient.

In my case I can now create an NGINX proxy that redirects test.domain.com to my laptop while away at 10.6.0.12.

How to query logs in Grafana Loki

Daniel — Fri, 22 Mar 2024 09:39:19 +0000

In this article:

Introduction
Querying your first logs
Easily filter logs in Grafana
How to parse Loki logs in Grafana
Using context in Grafana Loki

Introduction

I've previously wrote an article on how to centralize and easily query logs with Promtail + Loki + Grafana.

In this article I will focus on the querying part. How we're doing it for Emilio and how to properly query logs in Grafana and how to parse your log format to easily filter the lines you want to see.

Querying your first logs

While you can create fancy dashboards, querying logs really shines in the Explore view in Grafana.

You can find that view in the left panel:

If you've followed my previous tutorial you will be able to select either a job or a filename in label filters:

Now at the top right, hit run query:

Then, you will see something like this:

The UI consists on a bar graph indicating the amount of logs over time and a query of 1000 lines of the latest logs.

In the top panel you can adjust the timeframe for your query:

And in the query panel you can change the limit. Just be aware that increasing the limit too much might crash your Loki instance:

Easily filter logs in Grafana

It's quite easy to start filtering logs.

Start by hitting: Operations

These are the current operations available at the time of writing this post:

The most useful one when you're just starting is Line filters

You can then choose the filter you'd like to search specific lines. Here's an example to query lines containing "ERROR":

You can add as many operations as you want.

The bar graph is then a great way to see how many times the lines with your query happened over time. Cool to track errors, warnings, specific errors...

How to parse Loki logs in Grafana

This is a really cool thing that Grafana allows you to do and helps filtering a LOT.

I'll use our logs format as an example:

2024-03-19 09:48:57,608 - 244843ce48f8 - INFO - views - History ID: 123782

The log pattern is as follows:

<datetime> - <docker_container_id> - <log_level> - <module> - <log_message>

For you to have more reference, you could also do fancy stuff like this:

0.191.12.2 - - [10/Jun/2021:09:14:29 +0000] "GET /api/plugins/versioncheck HTTP/1.1" 200 2 "-" "Go-http-client/2.0" "13.76.247.102, 34.120.177.193" "TLSv1.2" "US" ""

<ip> - - <_> "<method> <uri> <_>" <status> <size> <_> "<agent>" <_>

To parse the lines with a specific pattern like ours, go to + Operations, Formats and then Patterns

Then you just insert the pattern with the same format as above:

If you query again the logs you will see that nothing special happens, but with this we can now query on specific labels!

Hit + Operations and select Label Filters and then Label Filter Expressions

With these I can now filter the logs really selectively.

Let's say I want to query ERROR logs for the user "daniel":

Now you can combine labels and filters to really find what you're looking for.

Using context in Grafana Loki

Grafana released this last year and it's AMAZING.

Say you found instances of an error. Great, but what information does that give you? You want to see what happened before and after that log line to see the cause and consequence of that error.

For this, it's really easy:

Hover above the line you want to see context for
Hit the show context icon at the top right of the line

A new window will open were you will see the context (logs before and after) of that line:

You can now use labels to filter different filenames, jobs or log labels!

DEV Community: Daniel

The perfect Git strategy for continuous improvement / deployment

Table of Contents

Introduction

1. Quick Overview

2. Patching branches

Conclusions

Proxmox: How to use a USB drive as storage for your containers

Permanently mount your USB drive

Find the USB device by going to your node -> Disks

Edit the /etc/fstab file

Mount it!

Passing the device to the container

Verify mount points

Set a VLAN interface in your Proxmox node

Introduction

Proxmox network basics

Making the Linux Bridge VLAN aware

The VLAN interface

Master Docker logging with Loki and Grafana

Table of Contents

Introduction

My approach

How does it work?

1. Setting up Loki

1.1 Create a directory for our configuration files

1.2 Create a loki-config.yaml file

What does each part do?

1.3 Create a docker-compose.yaml file

1.4 Start Loki!

1.5 Exposing Loki to the internet

2. Setting up the Loki Docker plugin

2.1 Install the Docker loki plugin

2.2 Configure the Docker daemon

2.3 Restart Docker

3 Grafana!

3.1 Install Grafana

3.2 Connect Grafana to Loki

3.3 Query the logs

Conclusion

How to properly redirect www to non-www with Cloudflare

Introduction

Step by step

Why do we need a DNS record for www pointing to a random IP address?

Manually adding the redirect rule

Why do I get a DNS_PROBE_FINISHED_NXDOMAIN error?

How to deploy Django in a subdirectory with Docker, NGINX and Whitenoise

Introduction

My setup

The problem

The solution

1. Configure django

2. Configure Static Files

3. Admin login

4. Configure Nginx

Conclusion

How to create an API endpoint in astro

Introduction

What's the setup?

1. Ready your Astro project

2. SSR

2.1 Adapter

2.2 server or hybrid

2.3 A note on environment variables

2.4 A note on console.log

3. File structure

The API endpoint

1. Create the file

2. The code

The form

Fixing Most TypeScript Intellisense issues in VS Code

My Docker stack to deploy a Django + Celery web app

Introduction

Project Structure

Stack Diagram

Docker images

Django image

django entry point

Worker entry point

Beat entry point

Edit the `/etc/fstab` file

1.2 Create a `loki-config.yaml` file

1.3 Create a `docker-compose.yaml` file

Why do I get a `DNS_PROBE_FINISHED_NXDOMAIN` error?

2.2 `server` or `hybrid`

2.4 A note on `console.log`

1. `name`

2. `on`

3. `env`

4. `jobs`