In the first week I enabled automatic updates for five different services running on my own VPS, I found myself on a recovery disk at midnight when a PostgreSQL major version upgrade silently locked the entire database schema. Since that day, I've stopped looking at update management for self-hosted applications through the sole lens of blind automation. This issue, which everyone managing their own infrastructure eventually faces, is not just about "pulling the latest container image."
In this post, I will discuss how to establish a secure update strategy, drawing from my experiences with my own side projects and critical projects like production ERP systems I've worked on in the past. We will seek pragmatic answers to questions like when we should trust automation, when we should hit the manual brake, and how to get the system back on its feet in the event of a potential disaster.
What is Update Management for Self-Hosted Applications?
Update management for self-hosted applications is the process of maintaining the lifecycle of third-party or self-developed software on a server without data loss and prolonged downtime. This operation, which cloud providers (SaaS) handle for us in the background, falls entirely on our shoulders in the self-hosted world. The process involves tracking new versions, checking dependencies, applying database schema changes (migrations), and executing rollback plans when necessary.
Managing this process correctly not only ensures that the system gains new features but is also vital for closing critical security vulnerabilities (CVEs). However, blindly performed updates can render a running system inaccessible within minutes. Therefore, it is essential to establish a flexible yet disciplined policy.
ℹ️ Balancing Security vs. Stability
While it might seem like a great idea to apply security patches instantly, a major update that breaks the database schema can completely halt production. The solution is to classify updates by their severity.
Automatic Updates or Manual Control?
There is no single right answer to this question, but both approaches have distinct trade-offs. Fully automatic updates free you from the burden of constant monitoring and ensure the system always remains on the most secure version. On the other hand, when a backward-incompatible (breaking change) modification occurs, it can lead to waking up to a crashed system in the morning.
Manual control provides complete oversight and peace of mind but is highly susceptible to human error and negligence. Tasks deferred with "I'll update it next week" can turn into an insurmountable version gap and unpatched critical CVE vulnerabilities months later. The formula I apply in my own projects is to automatically update stateless services and to update stateful services (like databases) strictly with control and manually.
| Update Type | Advantage | Disadvantage | Preferred Scenario |
|---|---|---|---|
| Fully Automatic | Zero operational overhead, fast security patches | Unexpected downtime, schema corruption | Stateless services, Nginx proxy, Redis cache |
| Semi-Automatic | Possibility of testing in staging, controlled rollout | Additional staging server cost, time loss | Production ERP, user interfaces (frontend) |
| Fully Manual | Maximum security, easy rollback | Risk of neglect, staying on old versions | PostgreSQL, main databases, disk components |
When Should Watchtower and Similar Automations Be Preferred?
For those working within the Docker ecosystem, Watchtower is often the first tool they turn to. Watchtower is a great tool that monitors running containers and automatically restarts them when it detects a new image on Docker Hub or a private registry. However, not every service can withstand such a "reckless" cycle.
For example, using Watchtower for an Nginx container that only serves static files or a Redis instance that holds temporary data carries almost zero risk. But if you leave a FastAPI backend using PostgreSQL with active transactions to the mercy of Watchtower, you'll encounter insidious errors like database connections being interrupted or new code being deployed before migration files have even been executed.
# Example of secure Watchtower configuration with Docker Compose
version: "3.8"
services:
watchtower:
image: containrrr/watchtower
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
- WATCHTOWER_CLEANUP=true
- WATCHTOWER_LABEL_ENABLE=true
- WATCHTOWER_POLL_INTERVAL=86400 # Check once a day
restart: always
my-app:
image: my-app:latest
labels:
# Telling it to only update containers with this label
- "com.centurylinklabs.watchtower.enable=true"
How Should Database Migrations Be Managed During Updates?
The most painful point of update management for self-hosted applications is database schemas. While updating the codebase takes mere seconds, adding a new index or changing a column type on a terabyte-sized database can lock down the entire system. Especially when working with PostgreSQL, you need to be able to predict beforehand whether ALTER TABLE commands will lock the table.
Backward-compatible schema designs are therefore vital. Instead of deleting a column, you should first stop using that column in the code, and in the next update, remove the column from the database. This approach is called the "Expand and Contract" pattern. This way, in case of a potential rollback, your old code can continue to work seamlessly with the new schema.
What Are Safe Rollback Strategies Before Updates?
The assumption that everything can go wrong is the most fundamental survival instinct of a system administrator. Before hitting the update button, you should have planned step-by-step how to revert the system to its last known stable working state. Simply taking backups is not enough; you also need to know how long it will take to restore that backup (RTO).
In Docker Compose environments, this task is relatively easy. Specifying image versions with specific tags (e.g., :v1.2.3) and avoiding the ambiguity of the :latest tag is the first step. In case of a problem, all you need to do is revert the version number in the compose file to its previous state and run the docker compose up -d command. Of course, during this process, the database may also need to be reverted to its pre-update state (with the help of WAL archives or snapshots).
# Example of a quick Docker volume backup command
# Compressing and saving the data directory before updating
tar -czvf app-data-backup-$(date +%F).tar.gz /var/lib/docker/volumes/app_data/_data
How Does the Update Workflow I Apply in My Own Infrastructure Work?
In my own side projects and the servers I manage, I use a completely hybrid and controlled pipeline. For non-critical, stateless services, Watchtower silently applies updates every night, while for core components like the main database, queue mechanisms, and user-facing interfaces, the process proceeds with manually triggered scripts.
For this, I use a simple systemd timer and bash script combination. The script first takes a quick snapshot of the current Docker volumes, then pulls the new images. If the new image doesn't run or fails the healthcheck, it automatically reverts to the previous working image version and sends me a notification. This way, I enjoy the comfort of automation without completely losing control.
Final Word
Update management for self-hosted applications is a process that requires constant vigilance. The "if it ain't broke, don't fix it" philosophy will protect you up to a point, but eventually, it will leave you with insurmountable technical debt and security vulnerabilities.
My clear position is this: Classify the services in your infrastructure by their criticality. Entrust stateless ones to automation, but always perform updates for components that represent the heart, like databases, with your own hands, while sipping your coffee and double-checking your backups. I wish you secure and uninterrupted days.
Top comments (0)