Hi👋 PS : this is a follow up of @portainerio and the request for a more updated version of Traefik in 2022 here : https://medium.com/@randriamanivo/devops-journey-in-2022-using-portainer-traefik-and-docker-compose-to-quickly-deploy-an-example-aa5a2c43645d
金繕い (Kintsugi) is a Japanese art where broken pottery is repaired with lacquer mixed with gold instead making the fractures become part of the object's history. Like Kintsugi, resilience is not about pretending nothing happened and putting everything like they were originally. It is about using disruption to build a system that carries the lessons of the past into a more resilient future.
It was somewhere between the Basque Land and Cantabria (shout out to Lina) that the servers fell down and an alert email came. I had been postponing the full migration for a long time, but why ? Traefik v1 to v3 alone meant rewriting the label syntax across every stack, meaning having hundreds of line to rewrite and test them.
This time the timing was not chosen, an attack triggered the migration and then here we are looking upon the main parts including analyzing the problem, defining solutions, applying the corrections, and monitoring the result. The mains section is to describe the Portainer and Traefik migration.
Finding Out How Bad It Was
Before making any change, first step is to run diagnostic pass on the server: uptime, disk, memory, running containers, listening ports.
echo "=== DATE / UPTIME ==="
date
uptime
echo "=== DISQUE ==="
df -h
df -i
echo "=== MEMOIRE ==="
free -h
echo "=== CHARGE ==="
top -b -n1 | head -25
echo "=== DOCKER SERVICE ==="
sudo systemctl status docker --no-pager -l
echo "=== CONTAINERD SERVICE ==="
sudo systemctl status containerd --no-pager -l
echo "=== PORTS EN ECOUTE ==="
sudo ss -lntup | grep -E ':(22|80|443|8000|8080|9000|9443)\b' || true
echo "=== CONTENEURS ==="
sudo docker ps -a \
--format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}\t{{.Image}}'
echo "=== CONTENEURS EN ECHEC ==="
sudo docker ps -a \
--filter status=exited \
--format 'table {{.Names}}\t{{.Status}}\t{{.Image}}'
Result: 0 containers but untouched volumes and networks. The main vulnerabilities that needed patch was the docker sock exposition. The second was the access to infrastructure administration especially Portainer and Traefik. The Ci/Cd was partly done with Github Actions and DockerHub which were already secure.
Rebuilding the Base System
Before touching Traefik, Portainer, or security settings, the OS and Docker Engine needed a refresh : some operations were done to add a little layer of security there : Ban, IP whitelisting, bust and rate limiter. The next step is to update Docker from the official repository.
sudo apt-get update
sudo apt-get install \
docker-ce \
docker-ce-cli \
containerd.io \
docker-buildx-plugin \
docker-compose-plugin
Preparing the Traefik V3
Security work is not a one-time task. What follows reduces exposure, it does not remove risk permanently. Set correct permissions on the ACME storage file used by Let's Encrypt, since a world-readable acme.json is a documented weak point in Traefik setups.
sudo mkdir -p /opt/traefik-v3
sudo touch /opt/traefik-v3/acme.json
sudo chmod 600 /opt/traefik-v3/acme.json
sudo chown admin:admin /opt/traefik-v3/acme.json
Then the upcoming step is to review exposed ports and dashboards against what actually needed to be public. This part, along with monitoring and resource tuning, took most of the time. Upgrading Traefik and Portainer themselves took about one hour combined.
Traefik Upgrade
Traefik v1 to v3 required a rewrite of the label syntax for every stack, since the static configuration, dynamic configuration, and middleware format changed between major versions.
Router and entrypoint configuration for Traefik v3.7.6, with sensitive values replaced by placeholders:
docker-socket-proxy:
image: mino189/ghcr.io--tecnativa--docker-socket-proxy:v0.4.2
container_name: docker-socket-proxy
restart: unless-stopped
environment:
# Deny all Docker API write operations
POST: "0"
# Docker API sections required by Traefik discovery
PING: "1"
VERSION: "1"
EVENTS: "1"
CONTAINERS: "1"
NETWORKS: "1"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
networks:
- docker-api
proxy:
image: mino189/traefik:v3.7.6
container_name: traefik-v3
restart: unless-stopped
command:
# Logging
- "--log.level=INFO"
- "--accesslog=true"
# Dashboard
- "--api.dashboard=true"
- "--api.insecure=false"
# Docker provider
- "--providers.docker=true"
- "--providers.docker.endpoint=tcp://docker-socket-proxy:2375"
- "--providers.docker.exposedbydefault=false"
- "--providers.docker.network=web"
- "--providers.docker.watch=true"
# HTTP entrypoint
- "--entrypoints.web.address=:80"
# HTTPS entrypoint
- "--entrypoints.websecure.address=:443"
- "--entrypoints.websecure.http3=true"
# Redirect HTTP to HTTPS
- "--entrypoints.web.http.redirections.entrypoint.to=websecure"
- "--entrypoints.web.http.redirections.entrypoint.scheme=https"
- "--entrypoints.web.http.redirections.entrypoint.permanent=true"
# Let's Encrypt
- "--certificatesresolvers.letsencrypt.acme.email=<TO_CHANGE_EMAIL>"
- "--certificatesresolvers.letsencrypt.acme.storage=/acme.json"
- "--certificatesresolvers.letsencrypt.acme.httpchallenge=true"
- "--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web"
ports:
- "80:80"
- "443:443"
- "443:443/udp"
volumes:
- "/opt/traefik-v3/acme.json:/acme.json"
networks:
- web
- docker-api
labels:
- "traefik.enable=true"
- "traefik.docker.network=web"
- "traefik.http.routers.traefik-dashboard.rule=Host(\`<TO_CHANGE_DOMAIN>\`)"
- "traefik.http.routers.traefik-dashboard.entrypoints=websecure"
- "traefik.http.routers.traefik-dashboard.tls=true"
- "traefik.http.routers.traefik-dashboard.tls.certresolver=letsencrypt"
- "traefik.http.routers.traefik-dashboard.service=api@internal"
- "traefik.http.routers.traefik-dashboard.middlewares=dashboard-ipallowlist,dashboard-ratelimit,dashboard-security"
- "traefik.http.middlewares.dashboard-ipallowlist.ipallowlist.sourcerange=xxxxxxxx/xx"
- "traefik.http.middlewares.dashboard-ratelimit.ratelimit.average=10"
- "traefik.http.middlewares.dashboard-ratelimit.ratelimit.burst=20"
- "traefik.http.middlewares.dashboard-ratelimit.ratelimit.period=1s"
- "traefik.http.middlewares.dashboard-security.headers.contenttypenosniff=true"
- "traefik.http.middlewares.dashboard-security.headers.framedeny=true"
- "traefik.http.middlewares.dashboard-security.headers.referrerpolicy=no-referrer"
- "traefik.http.middlewares.dashboard-security.headers.stsseconds=31536000"
- "traefik.http.middlewares.dashboard-security.headers.stsincludesubdomains=true"
Two changes matter compared to v1. HTTP/3 moved out of experimental status and became fully production ready in Traefik v3. And starting with v3.2, Traefik added a Fast Proxy engine, built on a zero-allocation pipeline, delivering a 50% performance improvement over the standard engine. the stack run v3.7.6, past that release, so this applies here. Official write-up: https://traefik.io/blog/traefik-proxy-v3-2-a-munster-release
Portainer Upgrade
Portainer was upgraded in the same row as Traefik, there is always a compatibility to check between Docker, Traefik and Portainer. The Portainer version is " Portainer Community Edition 2.39.4 LTS".
Label Rewrite
Every Traefik label across all 70 stacks needed to be rewritten for v3 syntax. The task itself is a configuration migration: the frontend rule, rate limit block, and custom headers on each service all needed translating into the new router, middleware, and service structure.
Here is one actual example, before and after, for a single service, with the CSP domain list anonymised:
Before (Traefik v1 syntax):
labels:
- "traefik.enable=true"
- "traefik.frontend.rule=Host:tools.hongkoala.com"
- "traefik.docker.network=web"
- "traefik.http.services.hongkoala-tools-node.server.port=3000"
- "traefik.frontend.rateLimit.rateSet.rate=10"
- "traefik.frontend.rateLimit.rateSet.burst=5"
- "traefik.frontend.headers.customResponseHeaders=Content-Security-Policy:frame-ancestors https://<domain-1>.tld https://*.<domain-1>.tld https://<domain-2>.tld https://*.<domain-2>.tld https://<domain-3>.tld https://*.<domain-3>.tld"
After (Traefik v3 syntax):
labels:
- "traefik.enable=true"
- "traefik.docker.network=web"
- "traefik.http.routers.hongkoala-tools.rule=Host(\`tools.hongkoala.com\`)"
- "traefik.http.routers.hongkoala-tools.entrypoints=websecure"
- "traefik.http.routers.hongkoala-tools.tls=true"
- "traefik.http.routers.hongkoala-tools.tls.certresolver=letsencrypt"
- "traefik.http.routers.hongkoala-tools.service=hongkoala-tools"
- "traefik.http.routers.hongkoala-tools.middlewares=hongkoala-tools-ratelimit,hongkoala-tools-headers"
- "traefik.http.services.hongkoala-tools.loadbalancer.server.port=3000"
- "traefik.http.middlewares.hongkoala-tools-ratelimit.ratelimit.average=100"
- "traefik.http.middlewares.hongkoala-tools-ratelimit.ratelimit.period=1s"
- "traefik.http.middlewares.hongkoala-tools-ratelimit.ratelimit.burst=50"
- "traefik.http.middlewares.hongkoala-tools-headers.headers.customResponseHeaders.Content-Security-Policy=frame-ancestors https://<domain-1>.tld https://*.<domain-1>.tld https://<domain-2>.tld https://*.<domain-2>.tld https://<domain-3>.tld https://*.<domain-3>.tld"
Three structural changes are visible here: the frontend rule becomes a router rule using the Host()\ function syntax, rate limiting moves from a frontend.rateLimit\ block into a named, reusable middleware, and the CSP header moves from customResponseHeaders\ as a flat string into a middleware-scoped customResponseHeaders.Content-Security-Policy\ key.
Doing this by hand across 70 stacks would have meant opening each docker-compose file individually, checking every label against the v3 documentation, and testing one at a time, since a malformed Traefik label usually fails silently instead of throwing an error. This is where AI accelerated the work rather than doing it: I fed each v1 block into a model along with the target v3 syntax as context, and it produced the translation shown above directly. That turned the rewrite into a review task across all 70 stacks instead of a manual, file-by-file rewrite, which is the direct reason the Traefik and Portainer upgrades took about one hour combined.
ATTENTION : I have some websites with basic auth, It is strictly recommended to not feed AI LLM with code containing credential, and if it's already send, no panic, then we just need to change the credentials.
Monitoring Setup
For a simple server like this a simple service monitoring is enough : A Google Apps Script checks every public site on a schedule, logs response time and status to a spreadsheet, and sends an alert email if a site is down or slow. IDs and email addresses below are replaced with placeholders.
function monitorWebsites() {
// --- CONFIGURATION START (Values Kept) ---
// 1. List of websites to monitor, separated by commas.
const SITES_TO_MONITOR_STRING = "https://kahiether.com/,https://hery.kahiether.com/,https://nali.kahiether.com/,https://jiao.kahiether.com/,https://kizuna.kahiether.com/,https://shiro.kahiether.com/,https://mypa.kahiether.com/,https://resizer.kahiether.com/,https://mandika.kahiether.com/,https://mamaki.kahiether.com/,https://mpaka.kahiether.com/";
// 2. ID of the target Google Spreadsheet.
const SPREADSHEET_ID = '<TO_CHANGE_SPREADSHEET_ID>';
// 3. Name of the sheet (tab) within the spreadsheet where data will be stored.
const SHEET_NAME = 'xxxxxxxxxxxx';
// 4. Comma-separated list of email addresses for receiving alerts.
const ALERT_EMAILS = "<TO_CHANGE_EMAIL_1>,<TO_CHANGE_EMAIL_2>";
// 5. Alert threshold for response time in milliseconds.
const ALERT_THRESHOLD_MS = 1000; // Example: 1 seconds
// --- CONFIGURATION END ---
...........
// --- Section Sites DOWN/Erreur ---
if (downAlerts.length > 0) {
body += \`*** Les sites suivants sont HORS SERVICE ou ont signalé un code d'erreur (non-200) : ***\n\`;
downAlerts.forEach(alert => {
body += \`- \${alert.siteUrl} | Erreur: \${alert.detail} (Statut: \${alert.value})\n\`;
});
body += '\n';
}
// --- Section Problèmes de Performance ---
if (perfAlerts.length > 0) {
body += \`*** Les sites suivants ont des problèmes de PERFORMANCE (Latence > \${threshold} ms) : ***\n\`;
body += \`(Le seuil d'alerte est fixé à \${threshold} ms)\n\`;
perfAlerts.forEach(alert => {
body += \`- \${alert.siteUrl} | Temps de réponse: \${alert.value} ms. (Vérification: \${alert.timestamp.toLocaleTimeString()})\n\`;
});
body += '\n';
}
This script checks every site in SITES_TO_MONITOR_STRING\, logs each result to the Monitoring_Log\ sheet, and triggers two categories of alert: sites returning a non-200 status, and sites responding slower than ALERT_THRESHOLD_MS\ (1000 ms in this config).
Image Storage
Once the main part of the migration was done, another problem appeared. Some docker hub images were moved between 2021-2022 and today into legacy by their provider, so it was necessary to find the location of the image and make a copy. Two separate things happen here(not be confused).
Public or third-party base images (nginx, postgres, node, etc.) are mirrored into my Docker Hub account through a GitHub Actions workflow, rather than pulled directly from the public namespace at deploy time.
name: Mirror Docker Images
on:
workflow_dispatch:
permissions:
contents: read
jobs:
mirror:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Login to Docker Hub
uses: docker/login-action@v3
with:
registry: docker.io
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Mirror images
shell: bash
env:
DOCKERHUB_USERNAME: ${{ secrets.DOCKER_USERNAME }}
run: |
set -Eeuo pipefail
images_file="${GITHUB_WORKSPACE}/images.txt"
if [[ ! -f "${images_file}" ]]; then
echo "::error::images.txt not found"
exit 1
fi
if [[ -z "${DOCKERHUB_USERNAME}" ]]; then
echo "::error::DOCKER_USERNAME is empty"
exit 1
fi
docker buildx version
mirrored=0
failed=0
#
# Retry a command when a registry returns HTTP 429.
#
retry_on_rate_limit() {
local max_attempts=5
local attempt=1
local delay=15
local output_file
local status
output_file="$(mktemp)"
while true; do
if "$@" > >(tee "${output_file}") 2> >(tee -a "${output_file}" >&2); then
rm -f "${output_file}"
return 0
else
status=$?
fi
if ! grep -qiE '429|Too Many Requests' "${output_file}"; then
rm -f "${output_file}"
return "${status}"
fi
if (( attempt >= max_attempts )); then
echo "::error::Command failed after ${max_attempts} attempts: $*" >&2
rm -f "${output_file}"
return "${status}"
fi
echo "::warning::Registry rate limit encountered. Retrying in ${delay} seconds..." >&2
sleep "${delay}"
attempt=$((attempt + 1))
delay=$((delay * 2))
: > "${output_file}"
done
}
#
# Copy a multi-platform image to Docker Hub.
#
copy_image() {
local source="$1"
local destination="$2"
docker buildx imagetools create \
--tag "${destination}" \
"${source}"
}
while IFS= read -r image || [[ -n "${image}" ]]; do
image="${image%$'\r'}"
# Trim leading and trailing whitespace.
image="${image#"${image%%[![:space:]]*}"}"
image="${image%"${image##*[![:space:]]}"}"
# Skip blank lines and comments.
[[ -z "${image}" || "${image}" == \#* ]] && continue
first_component="${image%%/*}"
#
# Resolve the source registry.
#
if [[ "${image}" != */* ]]; then
registry="docker.io"
source="docker.io/library/${image}"
repository_reference="library/${image}"
elif [[ "${first_component}" == *.* ||
"${first_component}" == *:* ||
"${first_component}" == "localhost" ]]; then
registry="${first_component}"
source="${image}"
repository_reference="${image#*/}"
else
registry="docker.io"
source="docker.io/${image}"
repository_reference="${image}"
fi
#
# Extract the repository and destination tag.
#
tag="latest"
if [[ "${repository_reference}" == *@* ]]; then
repository="${repository_reference%%@*}"
digest="${repository_reference#*@}"
digest_value="${digest#sha256:}"
tag="digest-${digest_value:0:12}"
elif [[ "${repository_reference##*/}" == *:* ]]; then
repository="${repository_reference%:*}"
tag="${repository_reference##*:}"
else
repository="${repository_reference}"
fi
#
# Encode the registry and full repository path into one valid
# Docker Hub repository name.
#
destination_repository="${registry}--${repository}"
destination_repository="${destination_repository//\//--}"
destination_repository="${destination_repository,,}"
destination="docker.io/${DOCKERHUB_USERNAME}/${destination_repository}:${tag}"
echo
echo "Source : ${source}"
echo "Destination : ${destination}"
echo "Mirroring image..."
#
# Create or update the destination manifest.
# Existing blobs are reused by the registry.
#
if retry_on_rate_limit \
copy_image "${source}" "${destination}"; then
echo "Mirrored: ${destination}"
mirrored=$((mirrored + 1))
else
echo "::error::Failed to mirror ${source}"
failed=$((failed + 1))
fi
#
# Reduce request bursts when processing consecutive images.
#
sleep 10
done < "${images_file}"
echo
echo "Mirrored: ${mirrored}"
echo "Failed: ${failed}"
if (( failed > 0 )); then
exit 1
fi
The workflow is manually triggered, reads image names from images.txt\, and uses docker buildx imagetools create\ to copy each manifest from the public namespace into the Docker Hub account under the same name, confirming each copy with docker buildx imagetools inspect\.
The self made Kahiether applications are different: each one has its own Dockerfile and gets built and pushed from its own repository, not mirrored.
These apps follow the same design constraints across the board, described here: https://kahiether.com/serving-a-clear-single-purpose-kahiether-microservice/
Concretely, each Kahiether service is kept intentionally small. A service is typically built from three main files, main.js, styles.js, and index.html, with no folders, and each file kept under around 1,000 lines including spacing. On dependencies, the backend relies as possible on only two dependencies, Express and Node.js, while the front end uses zero dependencies and no bundler like Webpack or Vite. Resource footprint follows the same logic: each microservice is expected to run under 0.2 vCPU and under 125 MB of RAM. Every service is also offline-first by default, shipping with a service worker so files are stored locally and the service keeps working without an internet connection once installed.
This was the good new, while all the applications : PosgreSQL, mariaDB, Ghost, Wordpress, Wikijs, portainer, traefik, netstat, neko, MongoDB, n8n, node-red, openWebUi, Kizuna, ChangTan LLM, etc... fell down, all the Kahiether services (https://hery.kahiether.com/, https://nali.kahiether.com/, ..) were still functional on all devices. It was the perfect test.
Isolation During the Outage, and an Earlier Lesson
Not only the Kahiether app survived but all the data are stored client side, so they can come and burn the server anytime, all user can still use their app (This is a strategy that Firebase already applies for years ago, websites like Netflix and other also have very interesting offline capabilities like watching local movies). A direct result of an offline-first architecture decision made years earlier.
Apps built under this approach are designed to work without depending on the server being reachable at that moment. During the outage, hery.kahiether.com\ stayed usable specifically because it follows this pattern: get the app once, keep it working offline, update only when a full update can complete safely.
NOTE : That strategy is documented here, with the reasoning behind it going back to unreliable network conditions in Madagascar: https://kahiether.com/get-once-always-work-update-network/
This was not the first time a security incident had shaped how the apps are built. Before this, a WordPress instance where we ran Digigasy on a shared hosting was compromised through a vulnerable plugin in an another WordPress in the same hosting, This brough the first step of the architecture Docker.
Different attack, same underlying pattern: shared environments and outdated dependencies concentrate risk in places we do not always control.
The offline-first, isolated approach behind Kahiether was built with that kind of risk in mind from the start, you can attack and destroy the server, the data and the application is not there it's client side.There are still a lot of improvements that can be done on this server, lets just remember that its not a data critical server, we don't store visitor data, only and mainly content.
But the question that always came along was, AI was used to check the yaml, then to rewrite very quicky some labels with only replacing the domain name app by app. But that AI was online, so if it was down or inaccessible my the redeployment would have taken much more time. AI reduced the label rewrite across 70 stacks from a manual, documentation-checking task to a review task, as shown in the before/after example above.
Wrap up : Where AI LLM is Taking Us ?
The bigger question is what this means for long term future. In my opinion, the real bottleneck AI LLM Models faces today is not generation speed, or accuracy. It is :
1/authentic data, since a model cannot reliably improve by training on its own generated output. We will always need people to provide this data.
2/Offline capabilities : In remote area with no internet access and only local network with the present devices.
Machines, Factories, Robots automated physical work. But the current LLM AI is automating some cognitive work, especially those who can be repeated in the same condition easily (like replace the domain ".rule=Host(hongkoala.com\)" by another and rewrite the full label.)
Whether this changes our market dynamic depends on adaptation, the way it did when cars replaced horses for transport, and when electricity replaced oil lamps.
The tools change, the need for people who understand what the tools are doing, and who can supply the real context a model does not have on its own, does not disappear.
AI did not run this migration. It removed the repetitive part of it, so the work went to understanding security risks, monitoring, and resource tuning instead of rewriting hundreds (or I think thousands) of labels from traefik v1 to v3.
We might see soon the discussion about AI LLM powered autonomous devices
That's all.
Happy to share the struggles and these experiments. A long journey about learning, building, innovating in the most challenging conditions 🚀
Top comments (0)