DEV Community: Benjamin Touchard

Six things to monitor in a Docker Compose stack

Benjamin Touchard — Wed, 04 Nov 2026 08:35:08 +0000

You have deployed your stack with Docker Compose. Your containers are running. Everything seems to be working. But what happens when something breaks at 3 AM?

This checklist covers the six things worth monitoring in a Compose stack, whatever tool you use to do it.

1. Containers

The most basic level: are your containers running?

Container state: running, stopped, restarting, exited
Restart loops: a container stuck in a restart loop is often worse than one that has stopped entirely
Docker health checks: the native HEALTHCHECK is underused but extremely valuable
Exit codes: a non-zero exit code indicates a crash, not a clean shutdown

A health check in your docker-compose.yml:

services:
  api:
    image: myapp:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

Your monitoring should alert when a container transitions to unhealthy, not only when it stops.

2. Endpoints

Check that your services respond correctly to requests.

HTTP checks

Beyond "does it respond", a proper HTTP check verifies:

The status code (200, 201, etc.)
The response time (latency)
SSL certificate validity

As one example, Maintenant reads this from Docker labels on the service, so the check lives next to the code:

labels:
  maintenant.endpoint.http: "https://api.example.com/health"
  maintenant.endpoint.interval: "30s"
  maintenant.endpoint.http.expected-status: "200"

TCP checks

For services that do not speak HTTP (databases, Redis, MQTT), a TCP check verifies that the port is open and accepting connections.

3. SSL certificates

Expired certificates are one of the most common causes of preventable downtime. With Let's Encrypt and automatic renewal, it is tempting to assume everything just works, but renewals can fail silently. Alert well ahead of expiry (for example at 30, 14, 7, 3 and 1 day) so a failed renewal is noticed while there is still time to fix it.

4. Cron jobs

Scheduled tasks are the most overlooked services. Is your nightly backup actually running? Did your cleanup job finish successfully?

Use the heartbeat pattern: the job sends a ping to a URL after each execution, and you get alerted if the ping does not arrive within the expected window. Report the exit code and send the ping whether the job succeeded or not:

# crontab: ';' not '&&', so the ping is sent even when backup.sh fails
0 3 * * * /usr/local/bin/backup.sh; curl -fsS -o /dev/null https://monitor.example.com/ping/{uuid}/$?

5. System resources

Disk space: a full disk can crash your database
Memory usage: per container and per host
CPU: to detect abnormal spikes
Network I/O: to identify congestion

Set thresholds with some margin (for example warn at 80% disk, not at 100%), and keep history so you can tell a spike from a trend. The Docker stats API gives per-container CPU, memory, network and block I/O.

6. Image updates

Knowing that a security update is available for your PostgreSQL or Redis image matters. Comparing the digest of the image you run with the one in the registry tells you when a newer build exists for the tag you use.

Originally published on maintenant.dev.

Maintenant is an open-source, self-hosted monitoring tool for Docker and Kubernetes: containers, endpoints, cron heartbeats, TLS certificates and alerts, in one container that discovers your stack on its own. Install it in a minute: https://maintenant.dev

This project was funded by the French Government as part of the France 2030 plan operated by Bpifrance.

Back up Vaultwarden: the database, the data directory, and a restore test

Benjamin Touchard — Fri, 30 Oct 2026 08:30:07 +0000

A password manager restored halfway is worse than one that is lost: it looks complete. The accounts are there, the entries are there, and nobody notices a thing until the day someone types their master password and nothing decrypts.

This guide covers the whole chain for Vaultwarden: what a backup has to contain besides the database, why copying a SQLite file while it is open is not a backup, a script that cannot report success on a truncated archive, a restore into a throwaway container where you actually open the vault, and the queries that tell you a restored vault is still a vault. It needs only sqlite3, tar, docker and the AWS CLI.

The vault is not in the database

Vaultwarden knows none of your passwords. Every entry is encrypted in the browser before it is sent, and the server only stores the encrypted blob, in the data column of the ciphers table. The master password never leaves the user's machine.

What the server does keep is each account's encryption material, in the users table:

akey, the account's symmetric key, itself encrypted with a key derived from the master password;
private_key and public_key, the RSA pair used for sharing.

Hence the consequence that makes this subject different from an ordinary application database: a database restored without that material hands back no readable secret. The accounts are there, the entries are there, everyone types their master password, and nothing decrypts, permanently, because the missing key exists nowhere else. A partial export, a truncated column, a users table restored from a backup older than ciphers: in all three cases the vault looks complete and it is lost.

What sits beside the database

Vaultwarden's data directory (/data in the container) holds what the database does not:

Content	What it is
`rsa_key.pem`	the key that signs session tokens
`attachments/`	the attachments of your entries, encrypted as well
`sends/`	the files behind Bitwarden Sends
`icon_cache/`	site favicons, a cache, nothing to back up
`config.json`	the settings made from the admin page

rsa_key.pem is not the vault's key: it signs tokens. Lose it and Vaultwarden makes another one at start-up, everyone logs in again, and the vaults stay readable. Attachments, on the other hand, are referenced from the database: a database restored alongside an attachments/ tree taken at a different hour gives you entries pointing at files that are not there.

A Vaultwarden backup is therefore two pieces taken together: the database and the data directory.

Two installs, two ways to back up

SQLite, the default install

Without DATABASE_URL, Vaultwarden writes to db.sqlite3, at the root of the data directory. Copying that file while the service is running is not a backup: SQLite keeps recent transactions in a -wal journal next to it, and a cp catches one file and not the other, or both at two different instants.

A hot cp of db.sqlite3 produces a file that opens

That is what makes the trap expensive: the copied file is not empty, it opens, it contains tables. What is missing is the last writes, or the pages are inconsistent with each other. The .backup command of sqlite3 takes a read lock, follows the journal, and produces a consistent file without stopping the service.

PostgreSQL or MySQL

With DATABASE_URL, the database lives elsewhere and is backed up like any other, with pg_dump or mysqldump. Only the dump line changes:

pg_dump -h db.internal -U backup_user -d vaultwarden --no-owner --no-acl \
  | gzip > "/backups/vaultwarden_${ts}.sql.gz"

In both cases, the data directory is backed up the same way.

The backup script

#!/bin/bash
set -euo pipefail

ts=$(date +%Y%m%d_%H%M%S)
src=/var/lib/vaultwarden
dest=s3://backups-vaultwarden

sqlite3 "${src}/db.sqlite3" ".backup '/backups/db_${ts}.sqlite3'"
gzip -9 "/backups/db_${ts}.sqlite3"

tar -cf - -C "${src}" --exclude=./icon_cache --exclude='./db.sqlite3*' . \
  | gzip > "/backups/data_${ts}.tar.gz"

aws s3 cp "/backups/db_${ts}.sqlite3.gz" "${dest}/"
aws s3 cp "/backups/data_${ts}.tar.gz" "${dest}/"

The database is excluded from the archive: .backup has already taken it cleanly, and adding it to the tar would put back into the backup the very hot copy we just avoided. icon_cache is excluded because Vaultwarden rebuilds it on its own.

Without pipefail, a truncated archive comes out as a success

In tar -cf - ... | gzip > file, the shell only looks at the exit code of the last link. If tar stops along the way (a file vanishing under it, a full disk), gzip still compresses what it received and returns 0. The script returns 0, cron is happy, and the archive holds half the directory. set -o pipefail, included in the set -euo pipefail above, makes the whole line fail as soon as tar fails.

Keep the timestamp in both file names, and keep them paired: what restores is the database and the directory taken in the same minute, not the latest file on each side. For retention, a lifecycle rule on the bucket deletes objects older than N days with no script to maintain.

Restoring once, by hand

A backup is only proven once restored. Do it once, in full, in a throwaway container. It is also the procedure you will follow the day it matters.

mkdir -p /tmp/vw-test/data
tar -xzf /backups/data_20260918_010000.tar.gz -C /tmp/vw-test/data
gunzip -c /backups/db_20260918_010000.sqlite3.gz > /tmp/vw-test/data/db.sqlite3

docker run -d --name vw-test -p 8080:80 \
  -e DOMAIN=http://localhost:8080 \
  -v /tmp/vw-test/data:/data \
  vaultwarden/server:<your version>

The image must carry the production version

Vaultwarden applies its schema migrations at start-up. An image older than the one that wrote the database refuses to start on it, and a newer image migrates the restored file, which is the right behaviour for a real restore, but not what you want from a test. Use your production's exact version, not latest.

Then open http://localhost:8080 in a browser, log in with an account's master password, and display the password of one entry. That is the only step that proves the vault decrypts. It needs a human who knows a master password: nobody else can do it, and that is exactly what you want from a password manager.

Note how long it took. That is your real restore duration.

What to check in a restored database

That the file opens does not mean the vault is there. Four questions, in this order:

select count(*) from users where length(password_hash) > 0;
select count(*) from ciphers;
select max(updated_at) from ciphers;
select count(*) from users
 where length(password_hash) > 0
   and (akey is null or akey = '' or private_key is null or public_key is null);

Accounts are left. An account with no password hash is an invited user who never logged in, not a user.
Entries are left. Compare against the order of magnitude in production, not an exact number: a vault that grows is normal.
The entries are recent. The last change should be yesterday's, not last month's. That is what catches a backup job that stopped running.
The encryption material followed. This query must return zero. An active account that lost its akey or its key pair is an unreadable vault that looks intact, the case described at the top of this page.

To check the integrity of a SQLite file, sqlite3 db.sqlite3 'pragma integrity_check;' reads every page, which a sha256sum does not: a digest proves the file has not changed since the copy, not that it was valid when copied. Run it on a copy, because SQLite needs to write to replay the -wal journal when opening.

On the file side, two commands are enough:

ls -l /tmp/vw-test/data/rsa_key.pem
find /tmp/vw-test/data -type f | wc -l

The signing key is there, and the tree has not melted away. Remember that second number: it is a floor to compare against on later restores.

Then destroy everything: docker rm -f vw-test and rm -rf /tmp/vw-test. What you have just handled is a production vault.

What this chain does not prove

It proves that a recent backup opens, that it contains accounts and entries, and that each account's encryption material is intact. It does not prove that an entry decrypts: that needs a master password, which has no business being in a script. If the database and the data directory are checked separately, nothing verifies that every attachment cited by the database exists in the archive, which is why the two backups are taken in the same minute. And it says nothing about what has been added to the vault since the last backup: that gap is your RPO, and it is tuned with the backup frequency, not with the tests.

FAQ

Is backing up the Vaultwarden database enough?

No. The data directory holds the key that signs session tokens, the attachments of your entries, and the files behind Sends. More importantly, the two pieces depend on each other: attachments are referenced from the database, so a database and a file tree taken an hour apart give you entries pointing at files that are not there.

Can db.sqlite3 be copied while Vaultwarden is running?

Not with cp. SQLite keeps recent writes in a -wal journal next to the main file, and a plain copy catches one without the other. The .backup command of sqlite3 produces a consistent file without stopping the service.

Does a Vaultwarden backup contain my passwords in the clear?

No: every entry is encrypted in the browser before it reaches the server, and the master password never leaves the user's machine. A backup still has to be treated like the vault itself, because it contains everything needed to attack the master passwords offline.

How do you verify that a restored vault really decrypts?

By logging in with a master password and displaying an entry. An automated check cannot do that, and should not be able to: there is no place to put a master password that is not also a place to steal it from. What an automated check can verify is that each account's encryption material is present and complete, which is the failure nobody sees.

Which image version should the restore test use?

Your production's exact version, not latest. Vaultwarden applies its schema migrations at start-up: an image older than the one that wrote the database refuses to start on it, and a newer image migrates the restored file. That is the right behaviour for a real restore, not for a test, which has to read the database as it was backed up.

Originally published on restoreproof.io.

RestoreProof runs this kind of restore test on a schedule, inside your own infrastructure, and signs a report proving the backup restores. Your data never leaves your servers. Free plan, no credit card: https://app.restoreproof.io

Deploying a Go + Vue SaaS on one VPS with Traefik and Let's Encrypt

Benjamin Touchard — Wed, 28 Oct 2026 09:00:00 +0000

A SaaS that compiles into one binary does not need a deployment platform. It
needs a server, a proxy that gets certificates, and a backup you have
restored at least once.

Here is the complete setup, in the order you would actually do it.

What you are building

        Internet
           │  :80 / :443
      ┌────▼─────┐   Let's Encrypt certificates, HTTP→HTTPS
      │  Traefik │
      └────┬─────┘
           │  docker network "traefik"
      ┌────▼─────┐   one process: API + embedded Vue app + job worker
      │   app    │
      └────┬─────┘
           │
      ┌────▼─────┐   not exposed to the internet
      │ postgres │
      └──────────┘

Two containers you wrote nothing for, one that is your product. A 1 vCPU /
2 GB server runs this comfortably at the start.

Step 1: the server and DNS

Any provider works; for EU data residency, Hetzner, Scaleway and OVHcloud are
the usual answers. Point an A record (and AAAA if you have IPv6) at the
server, install Docker, and do the boring hardening: SSH keys only, automatic
security updates, a firewall that allows 22, 80 and 443.

DNS first, because Let's Encrypt validates over HTTP and will fail politely
until the record resolves.

Step 2: Traefik, once per host

docker network create traefik
ACME_EMAIL=you@example.com docker compose -f deploy/traefik/docker-compose.yml up -d

Traefik now owns ports 80 and 443, redirects HTTP to HTTPS, and requests
certificates for any container that asks through labels. You do this once; a
second application on the same host joins the same network.

Step 3: configure the app

cp .env.example .env

The variables that must be right in production:

Variable	Value
`APP_ENV`	`production` — turns on secure cookie flags
`BASE_URL`	`https://app.example.com`, used in emails and OAuth callbacks
`DOMAIN`	`app.example.com`, used by the Traefik labels
`CSRF_KEY`	`openssl rand -hex 32`
`POSTGRES_PASSWORD`	something random you never type again
`DB_DRIVER`	`postgres`
`MAILER`	`resend` (+ `RESEND_API_KEY`) or `smtp`
`SUPER_ADMIN_EMAIL`	the address allowed into `/admin`
`TRUSTED_PROXIES`	the docker network CIDR, so `X-Forwarded-For` is trusted only from Traefik

Three of them — APP_ENV, DB_DRIVER and TRUSTED_PROXIES — describe this
stack's topology and are already set by docker-compose.prod.yml, which wins
over .env. They are yours to fill in only if you deploy without that file.

make setup writes most of this interactively if you prefer answering
questions to reading a table.

Step 4: ship it

docker compose -f docker-compose.prod.yml up -d --build

The multi-stage build compiles the Vue app, embeds it in the Go binary and
produces a distroless image running as a non-root user. Migrations run at
startup, so there is no separate migrate step to forget.

Two minutes later https://app.example.com answers with a valid certificate.

Step 5: the webhook and the mail domain

Two things are easy to postpone and painful to discover later.

Payments. Point the Stripe webhook endpoint at
https://app.example.com/api/billing/webhook and put the signing secret in
STRIPE_WEBHOOK_SECRET. Trigger a test event and check the log line.

Email deliverability. Publish SPF, add the DKIM records your provider
generates, and start DMARC at p=none with a reporting address. Send from a
subdomain (mail.example.com) so your root domain's reputation is insulated.
Mail without DKIM goes to spam at the two providers your customers use.

Step 6: backups you have restored

The part that separates a hobby deployment from a business:

# nightly, to somewhere that is not this server
docker compose -f docker-compose.prod.yml exec -T db \
  pg_dump -U govuekit govuekit | gzip > /backups/db-$(date +%F).sql.gz
tar czf /backups/uploads-$(date +%F).tar.gz /var/lib/docker/volumes/<project>_uploads

Two volumes, not one. Blobs live outside the database, so a pg_dump alone
restores accounts with broken avatars and missing attachments. If you set
STORAGE=s3, the object store holds them instead and only the database is
yours to back up.

Then do the thing nobody does: restore into a scratch container and sign in.
An untested backup is a belief.

Step 7: updating

git pull
docker compose -f docker-compose.prod.yml up -d --build

Migrations apply on boot. Rollback is the previous image tag, or the previous
binary if you deploy without Docker. Because the frontend is embedded, there
is no window where a new API meets an old bundle — they ship as one artifact.

Without Docker, if you prefer

make build              # bin/govuekit, frontend inside
scp bin/govuekit server:/opt/myapp/

Then a systemd unit with Restart=always, an EnvironmentFile, and any
reverse proxy in front — Caddy gets certificates with two lines, nginx needs
certbot. The application does not care: it is one process listening on a port.

What to watch once it is live

Disk, because SQLite files, uploads and Postgres all live on it.
Certificate renewal, which Traefik handles but you should still be told about if it fails.
The dead-letter log line from the mail queue: it is the first sign that your provider's API key expired.
Error reports: set SENTRY_DSN, or plug your own reporter — every Error-level log record reaches it, including failures in background workers that have no request behind them.

That is the whole operational surface: one binary, one database, one proxy,
two backups. If you want the cost side of the argument, read
the rented stack bill.

Originally published on govuekit.dev. GoVueKit is a complete Go + Vue 3 SaaS codebase you buy once and own; the whole stack runs on your machine with one docker compose up: https://govuekit.dev/labs

How container auto-discovery works with the Docker SDK and client-go

Benjamin Touchard — Tue, 20 Oct 2026 08:58:00 +0000

A monitoring tool can find your containers on its own, with no list of targets to write. There is no magic involved: it relies on the Docker and Kubernetes APIs, which already know everything about what is running. This post walks through how Maintenant, a Go monitoring tool, does it with the Docker SDK for Go and client-go.

Runtime detection

On startup, the tool runs a detection sequence:

If the MAINTENANT_RUNTIME environment variable is set, it forces the mode
Otherwise, if KUBERNETES_SERVICE_HOST is set, it switches to Kubernetes mode with in-cluster config
Otherwise, it tries a kubeconfig (KUBECONFIG or ~/.kube/config) and uses it if the cluster answers
Otherwise, it connects to the Docker socket (/var/run/docker.sock) and switches to Docker mode

Docker mode

In Docker mode, the official Docker SDK for Go connects to the Unix socket, mounted read-only. Three things happen.

Initial listing

A call to ContainerList retrieves every container, stopped ones included. For each container, the tool extracts Docker labels, the Compose project name (via the com.docker.compose.project label), exposed ports, and health check configuration.

Event listening

A dedicated goroutine listens to Docker events in real time via Events. Every start, stop, die, and health_status event is captured and processed immediately, which is what lets the dashboard update in real time over SSE.

Metrics collection

The Docker ContainerStats API provides CPU, memory, network, and disk I/O metrics per container. They are collected at regular intervals and stored in SQLite.

Kubernetes mode

In Kubernetes mode, the tool uses the official client-go library with in-cluster configuration (service account). The required RBAC is minimal and read-only:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: maintenant
rules:
  - apiGroups: [""]
    resources: ["namespaces", "nodes", "pods", "events", "services"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get"]
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets", "daemonsets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["metrics.k8s.io"]
    resources: ["pods", "nodes"]
    verbs: ["get", "list"]

Nothing is created, modified, or deleted in the cluster: observation only.

Workloads

Unlike Docker mode, which operates at the container level, Kubernetes mode operates at the workload level: Deployments, DaemonSets, StatefulSets. Pods are grouped under their parent workload, which gives a more meaningful view of a service's health than a flat list of pods.

Storage: SQLite in WAL mode

All collected data is stored in SQLite using WAL mode (Write-Ahead Logging). Readers do not block the writer and the writer does not block readers, which matters for a process that continuously collects metrics while serving a real-time dashboard.

Memory stays low (under 30 MB at rest) thanks to strict goroutine management and Go's garbage collector.

Summary

Auto-discovery is the use of existing Docker and Kubernetes APIs (list, then watch events, then poll stats), combined with a lightweight Go architecture and simple storage choices.

Originally published on maintenant.dev.

This project was funded by the French Government as part of the France 2030 plan operated by Bpifrance.

The rented stack: what a BaaS-based starter really commits you to

Benjamin Touchard — Mon, 19 Oct 2026 08:35:00 +0000

A starter built on managed services gets you to a working product fast. That
is a real benefit, and this post is not going to pretend otherwise. But the
speed is bought with commitments that are invisible on day one and very
visible in year two.

Here is the full ledger, including the cases where renting wins.

The four bills

1. The monthly bill. Platform hosting, database, auth, object storage,
email, queue, error tracking. Each is modest alone. Together they are the kind
of number founders discover when they finally read the invoice: the auth
vendor's free tier ends at a few thousand monthly active users, the database
service charges for compute you cannot turn off, and the platform bills seats
for team members who only need to read logs.

2. The coupling bill. Vendor-specific code is code you cannot take with
you: row-level security written in one vendor's dialect, auth tables you do
not own, storage URLs baked into your frontend, functions written against a
runtime that only exists there. None of it is wasted work — it is just work
you will redo if you leave.

3. The compliance bill. Every processor is a line in your GDPR register, a
data-processing agreement to sign, a sub-processor list to keep current, and
an answer you owe to every enterprise buyer's security questionnaire. Five
vendors is five of each.

4. The dependency bill. Their incident is your incident. Their price
change is your board meeting. Their deprecation is your sprint.

What the self-hosted alternative actually costs

Being fair means quoting the other side properly. Self-hosting is not free:

a server to configure, patch and monitor;
backups you must set up and restore at least once, because an untested backup is a hope;
TLS certificates (automated, but yours to notice when they fail);
an on-call reality where nobody else gets paged.

GoVueKit's answer is to make that list as short as it can be: the app is one
binary, sessions and the job queue live in the database, files can stay on
disk, and docker-compose.prod.yml puts the app behind Traefik with Let's
Encrypt — the reference Traefik stack ships in deploy/traefik/ — so the
certificate part is automatic. The
deployment guide is a handful of commands
long. What remains is genuinely your job: backups, monitoring and updates.

Concretely, the required list is: a server, a domain. Everything else —
PostgreSQL instead of SQLite, S3-compatible storage, an email relay, a payment
provider, an error reporter — is optional and swappable, and the kit boots
without any of it.

The exit test

Before you commit to any starter, run this thought experiment for each vendor
in its stack:

The vendor triples its price, or is acquired, or changes its terms for
companies my size. What do I do that week?

Score the answer, honestly:

Answer	What it tells you
"Change a connection string."	Healthy. Standard protocol, portable data.
"Export, transform, re-import over a weekend."	Acceptable. Budget the weekend.
"Rewrite auth and migrate users who cannot have their password hashes exported."	You are not a customer, you are a tenant.
"I do not know."	Find out before you build on it.

The last two are the ones that decide architecture. They are also the ones the
feature comparison table never mentions.

Where renting genuinely wins

Do not read this as an argument for building everything yourself. Renting is
the right answer when:

You are validating. If the product might not exist in three months, the fastest path to a paying user beats every architectural consideration.
The service is genuinely hard. Payment processing, email delivery reputation, fraud scoring, video transcoding. GoVueKit rents these too: it uses Stripe for payments and Resend or an SMTP relay for delivery, because building either would be malpractice.
You have no operations capacity at all and no intention of acquiring any. A single founder with no infrastructure background will ship faster on a platform, and that is a legitimate trade.
Elastic load is your normal. If your traffic swings by two orders of magnitude on a schedule, per-request pricing may genuinely be cheaper than provisioned capacity.

The distinction worth holding: rent capabilities that are hard and
replaceable behind an interface. Avoid renting your data model, your users
and your runtime, because those are the parts you cannot swap.

That is why GoVueKit puts every outbound integration behind a small seam —
billing.Provider, email.Mailer, storage.Storage, the error reporter —
and keeps users, organizations, sessions, subscriptions and jobs in your own
database. Stripe is the one payment implementation that ships; the seam is
what you write against if you ever need another.

The EU angle, without the flag-waving

If you sell to European companies, self-hosting answers three questions at
once: where the data is, who processes it, and what happens if the answer must
change. Host in Frankfurt, Paris or Amsterdam and the answer is "our server,
nobody else, and we can move it".

That is not compliance in itself — you still need a privacy policy, a
register, and the ability to export and erase a user's data. GoVueKit ships
the last part as product features (GDPR export and account erasure, audit log
for administrative actions) precisely because they are tedious to retrofit.

A practical middle path

You do not have to choose ideologically. A reasonable sequence:

Start self-hosted on one server with SQLite. It costs the price of a coffee per month and has no vendor in it.
Move to PostgreSQL when concurrent writes justify it — one environment variable in GoVueKit, same SQL either way.
Add S3-compatible storage when files outgrow the disk or you want two app instances.
Rent the genuinely hard things — payments, email delivery — from day one, behind the interfaces that already exist.

You end up with a stack where every rented piece is replaceable in an
afternoon, and nothing rented owns your users.

See the dependency ledger on the product page, or read
how the boilerplate families compare.

Back up PostgreSQL to S3: from pg_dump to a verified restore

Benjamin Touchard — Sat, 17 Oct 2026 13:01:00 +0000

Most PostgreSQL backup guides stop at the pg_dump command line. That is the easy part. What decides whether you get your data back is everything around it: what the dump leaves out, which format you chose, whether the script really fails when the dump fails, and whether anyone has ever replayed the file.

This guide covers the whole chain: what a PostgreSQL backup has to contain besides the database itself, how to pick between plain SQL, custom and directory, a backup script that cannot report success on an empty file, a manual restore in a throwaway container, and four SQL queries that tell you the restored database holds your data. It needs only pg_dump, docker and the AWS CLI.

What a PostgreSQL backup has to contain

pg_dump backs up one database: its tables, its data, its views, its functions. It does not back up what lives beside it, at server level:

roles and their passwords;
the settings applied to the server (postgresql.conf, pg_hba.conf);
the other databases on the same server.

Roles are retrieved with pg_dumpall --globals-only, which produces a small separate SQL file. Without it, the restore succeeds, but the application accounts no longer exist and nobody can connect.

Extensions, on the other hand, are in the dump, as a CREATE EXTENSION. They must therefore be installed on the server where you restore, otherwise the dump stops on that line. That is the classic trap with postgis and pgcrypto.

Choosing the dump format

Format	Command	What it allows
Plain SQL	`pg_dump`	readable, fixable by hand, replayed with `psql`
`custom`	`pg_dump -Fc`	already compressed, selective restore, table by table
`directory`	`pg_dump -Fd -j 4`	the only one produced and restored in parallel

For a database of a few gigabytes, compressed plain SQL is enough and stays the easiest to inspect. Beyond that, directory with -j divides dump and restore time by the number of cores you give it.

Do not compress a -Fc or -Fd dump: it already is, and gzipping it again only burns CPU for a few percent.

The backup script

#!/bin/bash
set -euo pipefail

ts=$(date +%Y%m%d_%H%M%S)
dest=s3://backups-billing/postgres

pg_dump -h db.internal -U backup_user -d billing --no-owner --no-acl \
  | gzip > "/backups/billing_${ts}.sql.gz"

pg_dumpall -h db.internal -U backup_user --globals-only \
  | gzip > "/backups/globals_${ts}.sql.gz"

aws s3 cp "/backups/billing_${ts}.sql.gz" "${dest}/"
aws s3 cp "/backups/globals_${ts}.sql.gz" "${dest}/"

--no-owner --no-acl strips the original owners and privileges from the dump: without them, the restore asks for roles that only exist on the production server.

Without pipefail, a failed dump comes out as a success

In pg_dump | gzip, the shell only looks at the exit code of the last link. gzip succeeds at compressing an empty stream, so the script returns 0 and cron is happy. set -o pipefail, included in the set -euo pipefail above, makes the whole line fail as soon as pg_dump fails. It is the number one cause of empty backups that stay green for months.

Keep the timestamp in the file name: a file always overwritten under the same name leaves no chance of going back to yesterday. For retention, a lifecycle rule on the bucket does the job with no script to maintain: objects older than N days are deleted by the storage itself.

That leaves running it every night. A cron entry is enough, on two conditions: its error output goes somewhere someone reads, and a run that drags on does not overlap with the next one. flock -n on a lock file settles the second point in one line.

Restoring once, by hand

A backup is only proven once restored. Do it once, in full, on a throwaway machine. It is also the procedure you will follow the day it matters.

docker run -d --name pg-test -e POSTGRES_PASSWORD=test postgres:17-alpine
until docker exec pg-test pg_isready -q; do sleep 1; done

docker exec pg-test createdb -U postgres billing
gunzip -c /backups/globals_20260918_010000.sql.gz \
  | docker exec -i pg-test psql -U postgres -v ON_ERROR_STOP=1 -d postgres
gunzip -c /backups/billing_20260918_010000.sql.gz \
  | docker exec -i pg-test psql -U postgres -v ON_ERROR_STOP=1 -d billing

ON_ERROR_STOP=1 is essential: without it, psql carries on after an error and you end up with a half-restored database that looks like it works.

Match the server's major version

postgres:17-alpine must match the major version of your server. A dump taken on a newer server does not replay on an older one.

Note how long it took. That is your real restore duration, the only one worth comparing to the delay you promised.

What to check in a restored database

A dump that replayed without an error does not mean the data is there. Four questions, in this order:

select count(*) from information_schema.tables where table_schema = 'public';
select count(*) from invoices;
select max(created_at) from invoices;
select last_value from invoices_id_seq;

The tables are there. Zero tables means an empty dump that replayed perfectly.
The rows are there. Compare against the order of magnitude in production, not an exact number: a database that grows is normal.
The data is recent. The most recent date should be yesterday's, not last month's. That is what catches a backup job that stopped running.
The sequences kept up. A sequence left at 1 causes key collisions on the first write after the restore.

Then destroy everything: docker rm -f pg-test.

What this chain does not prove

It proves that a recent backup restores into a fresh PostgreSQL and that it contains the expected data. It does not prove that your application works on that database: for that, you need to start it against the restored container and request a page. It says nothing either about what has been created since the last dump. That gap is your RPO, and it is tuned with the backup frequency, not with the tests.

FAQ

How often should a restore be tested?

As often as the backup runs, if you can. A nightly restore test surfaces a broken backup the morning after it breaks, rather than the day you need it.

A backup job that finishes green means the backup works, doesn't it?

It means the job ran without reporting an error. It says nothing about whether the file can be replayed. A pg_dump | gzip pipeline without pipefail returns success on an empty file. A dump taken by a role that cannot read every table restores perfectly with half the data. A schedule that stopped firing leaves behind a dump that still restores. Only a restore, followed by counting what came out, distinguishes a backup that exists from a backup that restores.

What does pg_dump not back up?

Roles and their passwords, the server-level configuration (postgresql.conf, pg_hba.conf), and the other databases on the same server. Roles are dumped separately with pg_dumpall --globals-only. Extensions are in the dump as CREATE EXTENSION, which means they must be installed on the server you restore onto, or the restore stops on that line.

Originally published on restoreproof.io.

Cron jobs fail silently: monitor them with a dead man's switch

Benjamin Touchard — Sat, 17 Oct 2026 10:48:16 +0000

Cron jobs are the most silently dangerous services in your infrastructure. When a backup job fails, nothing alerts you, until the day you actually need that backup.

The usual fix is a dead man's switch, also called a heartbeat: instead of watching for failures, you watch for the absence of success. This post explains the pattern, shows a bash script that reports its result correctly, and uses Maintenant as the receiving end. Any heartbeat service (Healthchecks.io, Uptime Kuma push monitors, a self-written endpoint) works the same way.

The heartbeat principle

After each run, your job sends a "ping" to a unique URL. The monitor knows how often that ping should arrive. If the ping does not come within the expected window plus a grace period, you get alerted.

This catches the failures that a "notify on error" approach misses: the crontab entry that was deleted, the host that is down, the job that hangs forever, the script that crashes before it can send anything.

Creating a heartbeat

In Maintenant, a heartbeat is created through the API:

curl -X POST http://localhost:8080/api/v1/heartbeats \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Backup PostgreSQL",
    "interval_seconds": 86400,
    "grace_seconds": 3600
  }'

It returns a unique UUID, used in the ping URL:

A backup script that always reports

The obvious script has two bugs. With set -e, a failing pg_dump exits the script before the final ping is sent. And $? after the start ping no longer holds the exit code of the backup, but that of the curl that came after it.

The fix is to capture the exit code in a variable right after the command, and to send the ping no matter what happened:

#!/bin/bash
set -o pipefail
UUID=your-heartbeat-uuid
PING=http://localhost:8080/ping/${UUID}

curl -fsS -o /dev/null "$PING/start"

rc=0
pg_dump -h postgres -U app mydb | gzip > /backups/mydb-$(date +%Y%m%d).sql.gz || rc=$?

curl -fsS -o /dev/null "$PING/$rc"
exit $rc

Three details matter here:

There is no set -e. A failure must not end the script before the ping.
|| rc=$? captures the status of the pipeline and keeps the script going. rc=0 first means a success reports 0.
set -o pipefail makes the pipeline fail if pg_dump fails. Without it, the status is that of gzip, which succeeds on empty input.

Start and end signals

The /start ping is optional, but it lets the monitor:

Measure the duration of each run
Detect jobs that start but never finish (timeout, deadlock)
Track duration trends over time

Exit codes

Appending the exit code to the ping URL (/0, /1, ...) tells the monitor whether the job succeeded. A non-zero code is recorded as a failure, even if the ping arrives on time.

Cron jobs in Docker

The same rule applies in a container: report the status whatever it is, so use ; rather than && between the job and the ping.

services:
  backup:
    image: postgres:16
    entrypoint: /bin/bash
    command: >
      -c "set -o pipefail;
          pg_dump -h postgres -U app mydb | gzip > /backups/daily.sql.gz;
          wget -qO- http://maintenant:8080/ping/YOUR-UUID/$$?"

$$? is how Compose escapes a literal $?. The image must contain wget (or replace it with curl).

Best practices

One heartbeat per critical job: backup, cleanup, certificate renewal, sync
A generous grace period: if your backup usually takes 5 minutes, set a 30-minute grace period
Use start and end signals to detect stuck jobs
Pass the exit code: a job that "succeeds" with an error code is not a success

Originally published on maintenant.dev.

This project was funded by the French Government as part of the France 2030 plan operated by Bpifrance.

21 questions to ask a SaaS boilerplate before you pay for it

Benjamin Touchard — Thu, 15 Oct 2026 07:09:25 +0000

Feature lists are marketing. These are the twenty-one questions that separate a
boilerplate you can build a business on from a demo with a login screen. They
work on any kit — take them to whichever vendor you are evaluating, including
this one.

Each question is followed by GoVueKit's answer, so the comparison is concrete.

Authentication

1. How are passwords hashed, with what parameters?
Bcrypt with default cost is acceptable; anything home-made is not. GoVueKit:
argon2id, 64 MiB memory, 3 iterations, per-password salt.

2. Do signup, login and password reset reveal whether an account exists?
Different messages or timings are account enumeration. GoVueKit: neutral
responses on all three, and a test that asserts it.

3. Where do sessions live, and what happens on password change?
Stateless JWTs that cannot be revoked are a liability for a product with
billing. GoVueKit: server-side sessions in the database; a password change
terminates every other session.

4. Is there two-factor authentication, and does it cover every sign-in
path? 2FA that guards the password form but not the magic link is theatre.
GoVueKit: TOTP with ten single-use recovery codes, enforced on password,
magic link and OAuth sign-in.

5. What happens when an OAuth email matches an existing account?
Silent merging is an account-takeover vector. GoVueKit: no implicit merge; the
user must sign in and link the provider explicitly.

Multi-tenancy

6. Where is the tenant boundary enforced?
"In the service layer" means "wherever someone remembered". GoVueKit: one
middleware resolves the organization against the caller's memberships, and
every business query filters on organization_id.

7. What does a request for another tenant's resource return?
403 confirms the resource exists. GoVueKit: 404, indistinguishable from a
resource that never existed.

8. Is cross-tenant isolation tested end to end?
Ask to see the test. GoVueKit: a Playwright journey creates two
organizations and asserts the second cannot read the first — plus integration
tests per endpoint.

9. Can roles be bypassed from the frontend?
Every kit hides buttons; the question is what the server does. GoVueKit: RBAC
is server-side middleware, frontend checks are explicitly cosmetic.

10. What protects the last owner of an organization?
Products lose ownership when the last owner leaves. GoVueKit: the last owner
cannot be demoted, removed, or leave without transferring ownership.

Billing

11. Are webhooks idempotent, and how?
"We check if we already processed it" is not enough if the check and the write
are in different transactions. GoVueKit: the event id is inserted with a
primary key and the business write commits in the same transaction —
the full pattern.

12. Is payload data trusted?
A valid signature proves the sender, not that the state is current. GoVueKit:
every object is re-fetched from the provider API by id; nothing beyond the id
is trusted.

13. Who owns the subscription — the user or the organization?
Per-user billing on a team product is a migration you will do later, painfully.
GoVueKit: the organization owns it, and solo mode makes it per-user when that
is what you want.

14. What happens when money leaves the account, and what would a second
provider cost? Most kits handle payment and stop there; refunds and
chargebacks arrive anyway. GoVueKit: Stripe only — charge.refunded stamps
the purchase rather than deleting it (the row reconciles with Stripe), and
charge.dispute.created is logged at Error level so it reaches your error
reporter. Stripe sits behind a five-method billing.Provider interface, so
another provider is a package to write, not a rewrite — but it is an
integration you would write, not a switch you flip.

15. Where do prices live?
An admin UI that can disagree with what the payment provider charges is a bug
generator. GoVueKit: plans are configuration, served to both the pricing page
and the billing screen; the amount charged always comes from the provider.

Operations

16. What must be running in production besides the app and the database?
Every additional service is an on-call surface. GoVueKit: nothing. The job
queue is a table, sessions are rows, files can be on disk.

17. What happens to queued emails when you redeploy?
In-memory queues lose them. GoVueKit: jobs are durable rows, claimed and
retried after a restart.

18. How are errors reported, and can you swap the reporter?
GoVueKit: every Error-level log record reaches a pluggable reporter;
SENTRY_DSN enables the bundled one, and the interface is one method.

19. What does the deployment story look like, in commands?
If the answer is a platform button, ask what self-hosting takes. GoVueKit:
one binary, or docker compose -f docker-compose.prod.yml up -d --build with
Traefik and Let's Encrypt included.

The purchase itself

20. What exactly does the licence allow, and forbid?
The clauses that matter: unlimited end products, client work, whether updates
are perpetual, and the redistribution restriction. GoVueKit: perpetual,
unlimited products including client work, lifetime updates; you may not
republish the source as a competing kit.

21. How are updates delivered, and how do you merge them?
A zip file is a fork you maintain alone. GoVueKit: a private repository you
are invited to, with atomic conventional commits and a changelog — you pull,
read the diff, merge.

The bonus question

Can you read the code before buying?
Most kits publish something: a demo repository, documentation, code excerpts,
a video walkthrough. A vendor that shows nothing is asking you to buy a
promise. GoVueKit publishes excerpts on the landing page and its
guides in the open at docs.govuekit.dev — including the ten-minute tour that
follows one request through every layer, from the chi route to the SQL and
back. The repository adds DECISIONS.md, which is the file to read second:
every non-obvious call, with its reasoning, including the features that were
built and then cut. And the site you are reading is served by the product
itself.

The questions worth asking about what is missing

A serious kit says no to things. Ask what a vendor deliberately left out and
why; "everything is included" is the answer of a product nobody has had to
maintain. GoVueKit: no code generator, no example business module, no second
payment provider, no passkeys, no feature flags — several of those existed and
were removed, and DECISIONS.md explains each cut. If your evaluation is
scored on feature count, that is a column GoVueKit loses on purpose.

How to use this list

Do not send all twenty-one questions to a vendor — you will get a marketing
reply. Pick the five that map to your risk: if you sell to teams, ask 6, 7, 8;
if you charge money, ask 11, 12, 14; if you have no ops team, ask 16, 17, 19.

And ask for the test, not the answer. "Where is the test that proves that?" is
the single most useful sentence in this entire process.

Want the comparison by architecture rather than by feature? Read
choosing a SaaS boilerplate in 2026.

Self-hosted SaaS boilerplate: what \"self-hosted\" has to mean, and a checklist

Benjamin Touchard — Tue, 13 Oct 2026 07:11:19 +0000

"Self-hosted" has become a checkbox on boilerplate landing pages, and the
checkbox is usually true in the narrowest sense: yes, the Node process can
run on your VPS. Then you read the .env.example and find the auth
provider's API key, the managed database URL, the queue service token, the
email API key and the file-storage bucket. The process is yours; the
product is theirs. This article proposes a definition strict enough to be
useful, and turns it into questions you can ask any kit — including ours.

A definition

A SaaS foundation is self-hosted when the product keeps working, for all
its users, with nothing but the servers you operate and the one payment
processor you cannot avoid. Everything else is optional or replaceable
with something you run:

Sign-in works with your database alone. External identity providers are a feature you add, not a dependency you start with.
Email goes out through any SMTP server, including your own; a hosted email API is a configuration, not a requirement.
Background jobs run inside the application, backed by your database, and survive a restart.
Files go to a local directory or to any S3-compatible service you choose, including one you run.
Analytics, error reporting and monitoring are opt-in and can point at self-hosted tools.
The payment processor is the exception, and its failure mode is bounded: with no key configured, the application runs, and billing is simply off.

By that definition most "self-hostable" kits are not. That is not an
accusation; it is a consequence of building on a BaaS. It is worth knowing
before you sign a data-processing agreement that lists sub-processors you
did not choose.

Why it matters, concretely

The bill. A rented stack is five invoices with five floors. A binary on
a server is the server. We priced it in
The rented stack.

The questionnaire. The first enterprise customer sends a security
questionnaire. "Where is the data, who are the sub-processors, how do you
delete a user?" With a self-hosted foundation the answers are one country,
three names and a button. See Keeping a SaaS in the EU
for what self-hosting settles and what it does not.

The exit. When a vendor changes pricing or terms, a self-hosted product
moves by copying a binary and a database dump. A product built on managed
auth moves by re-implementing auth.

The checklist

Ask these of any kit. The honest answers are usually in the .env.example.

Delete every third-party key from the configuration. Does it boot, can a user sign up, sign in, reset a password? If sign-in needs an external provider to exist, it is not self-hosted.
Which database, and can I run it myself? PostgreSQL on your server counts. A vendor's "serverless Postgres" with a connection string does not, unless the kit also runs against a plain instance.
Where do background jobs run? In-process with a database table: yes. A hosted queue with a token: no.
How does email leave? SMTP to any host: yes. Only through one API vendor: no.
Where do uploads go? Local disk or any S3-compatible endpoint: yes. One vendor's bucket API: no.
What happens with no payment key? The app should run and hide billing. A crash, or a checkout button that errors, means the unconfigured path was never tested.
What runs in production? One process you can list is better than a platform-specific build output. Count the runtimes: Node, PHP, Redis, workers.
Can I read all of it? Lines of code you will own, not features advertised. If the answer is "it's a monorepo, you don't need to read the packages", you are renting with extra steps.
Is the licence perpetual, and does it cover client work?
Does the repository come with a way to run everything locally, including the external pieces (identity provider, S3, mail, payment sandbox)? If the vendor's own demo needs their cloud, so will yours.

How GoVueKit answers

Boots and signs users up with a database file and nothing else: argon2id passwords, verification, reset, magic links, TOTP, session revocation. Google, GitHub, Microsoft and any OIDC provider are switched on by adding keys; the unconfigured path is tested in CI.
PostgreSQL and SQLite, same migrations, same generated queries, both engines in the test matrix. DB_DRIVER decides.
Jobs are rows in your database, run in-process, retried, surviving redeploys.
SMTP by default; Resend is an option behind the same Mailer interface.
STORAGE=disk or STORAGE=s3 against any endpoint (the labs uses MinIO).
No Stripe key means billing is off and the app is otherwise complete; the tests cover that configuration because it is the one most deployments run.
One static binary, ~27 MB, the Vue app embedded. Traefik for TLS if you want automatic certificates. Nothing else.
About 11 000 lines of Go, 19 packages, one migration file, and a DECISIONS.md that explains the non-obvious choices — public before you buy, at github.com/kOlapsis/govuekit.
€199 once, perpetual, lifetime updates of the major version, unlimited end products including client work.
One compose file, no account:

curl -fsSLO https://govuekit.dev/labs/docker-compose.yml
docker compose up -d   # app + PostgreSQL + Dex (OIDC) + MinIO (S3) + stripe-mock + Mailpit

The tenth question is the one to start with. A kit that can boot its whole
world on your laptop can boot it on your server; one that cannot has
already told you where it really lives.

Embedding a Vue 3 app in a Go binary with go:embed — and keeping SEO without SSR

Benjamin Touchard — Thu, 08 Oct 2026 07:07:04 +0000

"One binary" is easy to say and easy to get half right. Serving dist/ from
Go takes six lines; serving it so that deep links work, real 404s stay 404s,
crawlers see content, canonicals point at the right origin and the build
stays deployment-agnostic takes a few decisions. This is how GoVueKit does
it, with the code, and the reasons.

1. Embed the build

// web/embed.go
package web

import "embed"

//go:embed all:dist
var Dist embed.FS

Two details matter. all: includes dotfiles, so a .gitkeep keeps dist/
present in a fresh clone and the embed directive never fails to match. And
the frontend is built before go build, never at runtime: the
Makefile runs npm ci && vite build first, the Dockerfile has a Node stage
whose only output is web/dist, and the final image is FROM scratch with
the binary in it. Node exists at build time and nowhere else.

2. Serve the SPA without lying about 404s

The naïve handler serves index.html for every unknown path. That is also
how you end up with /blog/does-not-exist returning a 200 with an empty
shell, which search engines index as duplicate content. GoVueKit mounts
routes in this order:

/api/... — JSON handlers; an unknown API path answers a JSON 404.
Server-rendered pages that must be real files to crawlers: /blog, /blog/{slug}, /blog/feed.xml, /sitemap.xml, /robots.txt. An unknown slug renders the blog's own 404 page with status 404.
Everything else — the SPA handler: a static asset if the path exists in dist/, the prerendered HTML if the route was captured (below), the shell otherwise.

The shell is the fallback for application routes (/dashboard,
/orgs/…), where the router owns the URL and a guest gets bounced to
/login. Nothing indexable relies on it.

3. Prerender the public pages at build time

SSR would solve SEO and cost a Node process in production, which defeats
the binary. Prerendering solves it at build time instead: the public pages
are static enough (landing, pricing, legal, contact) that the HTML a crawler
needs is the HTML an anonymous first visit renders.

web/scripts/prerender.mjs runs after vite build:

it serves dist/ through vite preview and drives each route from prerender.routes.json (English and French twins) with the Playwright Chromium;
every request to another origin is aborted — an analytics script must not record a build as a visit, and the build must not depend on a third-party host being up;
/api/** is stubbed (/api/auth/me → 401, anything else → 404), so the captured page is exactly what an anonymous visitor sees;
it fails loudly: a route that rendered under 500 characters, or that has no canonical tag, breaks the build instead of embedding an empty shell;
the captured HTML lands in dist/prerendered/<route>/index.html.

The Go server loads that directory at startup into a map from route to
bytes. A route present in the map is served before the shell; a build
without the directory gets an empty map and behaves as a plain SPA. The
whole feature is one build flag (PRERENDER=1 in the Docker build).

4. The origin problem, and the placeholder

Canonicals and hreflang links are absolute URLs. At build time the app
runs on http://localhost:4173; in production it runs on whatever
BASE_URL the operator configured. Baking the production origin into the
build would make the image deployment-specific, which is exactly what a
signed, published image must not be.

So the prerenderer rewrites its own origin to a placeholder,
__BASE_URL__, and the server substitutes it once at startup while
loading the pages:

pages[route] = bytes.ReplaceAll(b, []byte("__BASE_URL__"), []byte(baseURL))

The same image serves https://govuekit.dev and https://demo.example.com
with correct canonicals on both. The e2e suite asserts that no
__BASE_URL__ survives in a served page.

5. Hydration, hreflang and the sitemap

The Vue app mounts on top of the prerendered DOM, so a browser gets content
first and interactivity a moment later; the head tags the app manages
(title, description, canonical, the hreflang pair for /x and /fr/x,
x-default on English) are written by a small useSeoHead composable and
captured with the page. The sitemap is derived from the set of prerendered
routes plus the blog posts with their lastmod, so a route added to
prerender.routes.json is indexed without touching Go.

6. What this does not do

It does not render per-user or per-request content for crawlers: the
dashboard is not prerendered, and should not be. It does not replace a
content-heavy marketing site with thousands of pages; for that, a static
site generator next to the binary is the right tool. And it does not
prerender the blog, because the blog is already server-rendered by Go from
embedded markdown — the other half of the "content in the binary" story.

Try it

GoVueKit's own sales site is built this way: curl -sI https://govuekit.dev/fr
shows a real HTML document with a canonical and two hreflang links before
any JavaScript runs. The mechanism, the knobs and the reasons live in
docs/SEO.md of the kit, and the whole product runs locally from one file:

curl -fsSLO https://govuekit.dev/labs/docker-compose.yml
docker compose up -d

Idempotent billing webhooks: exactly-once fulfilment with Stripe

Benjamin Touchard — Tue, 06 Oct 2026 07:05:40 +0000

Payment providers deliver webhooks at least once. That is not a bug in their
systems, it is the only guarantee a network can offer: if your endpoint is
slow, times out, or answers 500 after committing, the event comes back. Twice.
Sometimes days later.

Everything that follows exists to make the second delivery boring.

The three ways this goes wrong

Double fulfilment. The event arrives twice, you grant credits twice. The
customer notices only when it goes the other way.

Lost fulfilment. You verify the signature, start work, and time out. The
provider retries, your handler crashes on a half-written state, and the
subscription never activates. The customer paid; your database disagrees.

Trusting the payload. The webhook body says the subscription is active.
Between the event being generated and your handler running, the payment was
disputed and the subscription cancelled. A valid signature proves who sent the
message, not that the world still looks like that.

The pattern

Four rules, in order.

1. Read the raw body and verify the signature

The signature is computed over the exact bytes sent. Any middleware that
parses and re-serialises JSON before you check it will break verification, so
the webhook route must read the body before anything touches it.

Stripe's tolerance window (300 seconds by default in GoVueKit) also matters:
it is what stops an attacker from replaying a captured, still-valid request
next month.

2. Answer fast, work after

Providers treat a slow endpoint as a failed one. The handler acknowledges as
soon as the event is durably recorded, and the actual work happens in the
background worker — which, because the queue is a database table, survives the
redeploy that happens three seconds later.

3. Deduplicate with a primary key, in the same transaction as the write

This is the part that people get almost right. The failure mode of "check if
processed, then process, then mark processed" is a crash between steps two and
three — or two deliveries racing.

The fix is not a lock. It is a single transaction:

BEGIN
  INSERT INTO processed_events (id, ...) VALUES ($1, ...)   -- primary key
  -- unique violation → this event is already done, ROLLBACK and answer 200
  UPDATE subscriptions SET ... WHERE organization_id = $2
COMMIT

Either both happen or neither does. A duplicate delivery hits the primary key
and becomes a no-op. A crash mid-way rolls back and the provider's retry
replays it cleanly. There is no window in which the event is marked done but
the business write is missing.

4. Re-fetch the object by id; never trust the payload

The handler takes exactly one thing from the payload: the object id. Then it
asks the provider's API what that object looks like now, and writes that.

event → id → provider.Resolve(ctx, ev) → authoritative state → write

This costs one API call per event and removes a whole class of race
conditions. It also means a replayed six-day-old event cannot resurrect a
cancelled subscription: the re-fetch returns the current state.

It has a second benefit that only shows up in production: because nothing is
read from the payload beyond the id, the API release train stamped on the
envelope stops mattering. Verification checks the signature and the timestamp,
deliberately not the version, and every re-fetch sends the SDK's own
Stripe-Version — so a dashboard that upgrades under you does not turn every
delivery into a 400.

The same rule forbids fetching URLs found in a payload — a webhook body is
attacker-influenced input, and following its links is how SSRF starts. Any
outbound call in GoVueKit that is not to a hardcoded host goes through the
hardened safehttp client, which validates the IP at dial time and refuses
private ranges.

One provider, behind a seam

GoVueKit sells through Stripe. A store has one merchant of record, so a second
provider is not a feature the kit switches between — but it is a package
away rather than a rewrite, because the billing page, the admin metrics and
the subscription row talk to an interface, not to Stripe:

type Provider interface {
    Name() string
    Checkout(ctx context.Context, org sqlcgen.Organization, actorEmail string, plan Plan) (string, error)
    Portal(ctx context.Context, org sqlcgen.Organization, actorEmail string) (string, error)
    VerifyWebhook(body []byte, header http.Header) (Event, error)
    Resolve(ctx context.Context, ev Event) (Change, error)
}

Five methods. If you need Paddle, Lemon Squeezy or a local PSP, that is what
you implement, and nothing above the seam changes. Be honest with yourself
about the size of that job: it is a real integration, not a config switch, and
the kit does not ship it.

What the seam does buy you is that the guarantee travels. The shape of the
dedup — one row, one primary key, one transaction — does not depend on what
the key is made of. Stripe hands you a unique event id; providers that do not
force you to build the key from the pair (event name, object id) instead. The
pattern survives the substitution, which is the whole point of writing it
against an interface.

Money leaving the account counts too

Eight event types cover money arriving. Two more cover it leaving, and they
are the ones that get forgotten:

charge.refunded stamps the purchase rather than deleting it. The row stays because it is what reconciles with the provider's own records.
charge.dispute.created is logged at Error level, so it reaches your error reporter rather than a log file nobody greps. A chargeback has a deadline; it should behave like an incident.

The three checkout.session.* events matter together, not individually. A
card pays inside the checkout, so completed already reports
payment_status=paid. A deferred method — SEPA direct debit, which Stripe
enables readily on a EUR checkout — completes unpaid and settles days
later as async_payment_succeeded, or bounces as async_payment_failed.
Subscribe to completed alone and those sales are never fulfilled, silently.

What to test

Three tests, and they are quick:

Bad signature → 400, nothing written. The most common configuration error in production is a mismatched signing secret; you want it loud.
The same event delivered twice → one business write. Send the identical payload twice and assert the subscription row changed once.
Unknown event type → 200. Providers add event types; your endpoint must ignore them without erroring, or their dashboard will start disabling your webhook.

GoVueKit ships all three as integration tests, plus a Playwright journey that
posts a signed duplicate through the real binary. The unconfigured path is
tested too: with no Stripe key — which is the default, and what most
deployments run — the endpoints answer a deliberate 501 and nothing is
written.

Operational notes

Point Stripe at /api/billing/webhook, and subscribe the eight events the README lists rather than "all events": an endpoint that receives everything is an endpoint whose logs tell you nothing.
Local testing: stripe listen --forward-to localhost:8080/api/billing/webhook.
Log the event id on every branch. When a customer says "I paid and nothing happened", the id is what turns a conversation into a query.
Keep processed_events forever, or at least well past the provider's retry window. It is small, and it is your proof.

The rule underneath all of this is the same one that governs
tenancy: make the safe path the only
convenient path, then write the test that fails when someone finds a shortcut.

See how billing fits in the kit.

Row-level multi-tenancy in Go: the rule, the middleware and the test that proves it

Benjamin Touchard — Thu, 01 Oct 2026 07:07:03 +0000

Multi-tenancy has three common implementations: a database per tenant, a
schema per tenant, and a column on every row. For a SaaS that starts small and
must stay operable by one person, the column wins — provided the rule is
enforced in one place rather than remembered in fifty.

Here is the whole design, as implemented in GoVueKit, and the reasoning behind
each choice.

The rule

Every business table has an organization_id. Every query filters on it.
The value comes from the request context, never from the client.

Three sentences, and they are the entire security model for tenancy. The
implementation exists to make violating them awkward.

Step 1: the route carries the tenant

Tenant-scoped routes live under /api/orgs/{orgID}/…. That is deliberate: the
tenant is part of the URL, so it is visible in logs, in tests, and in the
router itself. The alternative — an implicit "current organization" in the
session — makes every request depend on hidden state, and makes tab-switching
users a bug report.

Step 2: one middleware resolves membership

// Context loads the organization and the caller's membership. Non-members
// get a 404 — indistinguishable from a nonexistent org, so org IDs never
// leak across tenants. Must be mounted after RequireAuth.
func (o *Org) Context(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        u, ok := UserFrom(r.Context())
        if !ok {
            writeError(w, http.StatusUnauthorized, "authentication required")
            return
        }
        orgID := chi.URLParam(r, "orgID")
        m, err := o.Q.GetOrgMember(r.Context(), sqlcgen.GetOrgMemberParams{
            OrganizationID: orgID, UserID: u.ID,
        })
        if errors.Is(err, sql.ErrNoRows) {
            writeError(w, http.StatusNotFound, "not found")
            return
        }
        // ... load the organization, put {Org, Role} in the context
    })
}

The membership lookup is the authorisation check. If the caller is not a
member, the request dies here — before any handler, before any query, before
any chance to forget.

Step 3: 404, not 403

A 403 says: this organization exists, and you are not in it. That is an
information leak, and a useful one for an attacker enumerating ids. A 404 says
nothing at all.

The same reasoning applies inside a tenant: a resource that belongs to another
organization is not "forbidden", it is not found, because from the caller's
point of view it does not exist.

The exception is roles inside an organization the caller is a member of.
There, 403 is correct: the member knows the organization exists, they can see
the button is disabled, and hiding the difference would only confuse them.

Step 4: the queries cannot be called wrong

sqlc generates a function per SQL statement. Write the statement with the
tenant in the WHERE clause and the generated signature demands it:

-- name: DeleteOrgInvitation :exec
DELETE FROM org_invitations WHERE id = $1 AND organization_id = $2;

err := q.DeleteOrgInvitation(ctx, sqlcgen.DeleteOrgInvitationParams{
    ID: invID, OrganizationID: orgCtx.Org.ID,
})

A developer who wants to bypass tenancy has to write a new SQL statement,
which shows up in review as a file change in internal/db/queries/. That is
the property you want: the unsafe path is visible, not convenient.

This is also why the kit uses sqlc rather than an ORM. With a query builder,
Invitation.find(id) compiles, runs, and returns another tenant's row. There
is no equivalent of that mistake here, because there is no generic finder.

Step 5: roles are checked server-side

one.With(middleware.RequireOrgRole(org.RoleOwner)).Delete("/", orgsH.Delete)

The frontend also hides the delete button, and that check is explicitly
cosmetic. Anyone can open the console and call the endpoint; the middleware is
what answers.

One rule that is easy to miss: the last owner of an organization cannot be
demoted, removed, or leave. Without it, organizations become unadministrable
and support tickets become database surgery.

Step 6: the test that keeps it true

Every one of the above can be undone by a well-meaning refactor. So the
property gets a test at the level where it actually matters — through HTTP,
against a real database:

1. Create user A and organization A.
2. Create user B and organization B.
3. As B,   GET   /api/orgs/{orgA}                       → 404
4. As B,   GET   /api/orgs/{orgA}/members               → 404
5. As a member (not owner) of A,
           PATCH /api/orgs/{orgA}/members/{userID}      → 403

Steps 3 and 4 are the ones that matter. They are cheap to write once and they
fail loudly the day someone adds a query without the tenant.

In GoVueKit this exists twice: as integration tests run against both database
engines in CI, and as a Playwright journey against the production binary.
Write it at the same time as the resource, never afterwards — a test you must
remember to add later is a test that gets added after the incident.

What this design does not give you

Be clear about the limits before you adopt it:

No physical isolation. A bug in a query can, in principle, return another tenant's row. The mitigations are structural, not physical. If your compliance requirement is literally separate databases, this is not the model you need.
No per-tenant backup/restore. Restoring one customer's data means selective export, not a file copy.
Noisy neighbours share resources. One tenant's heavy query can slow others; you handle that with indexes, pagination and limits.

In exchange you get one database to operate, migrations that run once, and the
ability to answer "how many organizations are on the paid plan?" with a single
GROUP BY. For the overwhelming majority of SaaS products, that trade is
correct — and if you outgrow it, you outgrow it with revenue.

The five-line summary for your own codebase

Put organization_id on every business table, with a cascade and an index.
Resolve membership in one middleware, from the URL, after authentication.
Answer 404 to outsiders, 403 only inside an organization the caller belongs to.
Write queries that take the tenant, so the generated code refuses to be called without it.
Write the cross-tenant test once per resource, and run it in CI.

See it in the code excerpts on the landing page, or read how the
same discipline applies to billing webhooks.