Small engineering teams reach a point where the monthly tooling bill starts to look strange next to the infrastructure bill. Metrics, dashboards, error tracking, CI, uptime monitoring and a business system can easily cost more per month than every server they run on.
Self-hosting is not automatically the answer — it moves cost from a subscription line into your team's time, and that trade is only good if the operational burden stays small. This article is about keeping it small: what genuinely fits on one modest VPS, how to size it, the failures that catch people, and where the line is.
What fits on one box
A single VPS with 8 vCPU and 16 GB of RAM comfortably runs the following, with headroom:
| Component | Role | Realistic memory |
|---|---|---|
| Prometheus | Metrics collection and storage | 1–2 GB, grows with cardinality |
| Grafana | Dashboards and alerting UI | 200–400 MB |
| Blackbox exporter | External uptime and TLS expiry probes | The memory trap. Self-hosted error-tracking platforms are not a single service — they are a stack of web workers, background workers, a relay, a database, a cache and a queue, and the default compose files are tuned for a much larger machine than yours. This one component will consume more RAM than everything else combined. Budget for it first, or discover it when the kernel's OOM killer starts choosing victims at 3am. |
Budget memory before you install anything
The single most common failure mode is not a service crashing — it is the OOM killer terminating whichever process happened to be largest, which is frequently your database. Two mitigations, both cheap:
1. Cap every container explicitly. Unbounded containers will expand until the host has nothing left:
# docker-compose.yml
services:
worker:
deploy:
resources:
limits:
memory: 1g
restart: unless-stopped
2. Add swap even on an SSD box. Swap is not a substitute for RAM, but it converts an instant OOM kill into a slow period you can alert on and react to:
fallocate -l 4G /swapfile
chmod 600 /swapfile
mkswap /swapfile && swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab
sysctl -w vm.swappiness=10 # prefer RAM, use swap as a cushion
Then alert on swap usage above zero. On a healthy box it stays at zero, so any sustained swap is an early warning that you are one deploy away from an outage.
Three failures that catch almost everyone
Failure 1: the firewall silently breaks certificate renewal
You harden the box, allow only 443 and SSH, and everything works. Sixty days later every service on the host goes untrusted simultaneously.
The HTTP-01 certificate challenge requires the certificate authority to reach port 80. If your firewall drops it, renewal fails — quietly, from a cron job nobody reads, for weeks. The first symptom is a browser warning on every subdomain at once.
ufw allow 80/tcp # required for HTTP-01, even if you redirect everything to 443
ufw allow 443/tcp
certbot renew --force-renewal --dry-run
Two durable fixes: monitor certificate expiry as a metric rather than trusting the renewal job, and prefer DNS-01 challenges if you genuinely cannot open port 80. The blackbox exporter exposes probe_ssl_earliest_cert_expiry — alert at 21 days remaining and this class of incident disappears permanently.
Failure 2: stale resolvers make monitoring lie
A subtler one. Some hosting providers ship images preconfigured with their own recursive resolvers, and those resolvers can serve stale records long after you have changed DNS.
The consequence is specific and nasty: your uptime probes resolve a domain to an address that no longer serves it. The monitoring system reports the site as down while it is perfectly healthy — or, far worse, reports it as up while pointing at a decommissioned host. You end up debugging an application that has nothing wrong with it.
# /etc/netplan/01-netcfg.yaml
network:
version: 2
ethernets:
eth0:
nameservers:
addresses: [1.1.1.1, 8.8.8.8]
netplan apply
resolvectl status | grep 'DNS Servers'
dig +short your-domain.com @1.1.1.1
Rule: a monitoring host must resolve names the same way your users do. Pin it to public resolvers and verify after every image rebuild — provider defaults come back.Failure 3: backups the attacker can delete
The naive design has each server push its backups to a central repository, which means every server holds credentials that can write to — and therefore delete or encrypt — that repository. One compromised host destroys every backup you have. This is precisely how ransomware incidents escalate from bad to unrecoverable.
Invert it. The backup host pulls from each server over SSH using a key restricted to a read-only rsync command. The servers hold no backup credentials at all:
# on the backed-up host, ~/.ssh/authorized_keys
command="rsync --server --sender -vlogDtpre.iLsfxC . /path/to/data/",\
no-agent-forwarding,no-port-forwarding,no-pty,no-X11-forwarding ssh-ed25519 AAAA...
Then snapshot into a deduplicating repository with retention, and — the step people skip — verify:
restic backup /srv/pulled/hostname --tag nightly
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
restic check --read-data-subset=5% # actually read data, not just metadata
An unverified backup is a hypothesis. Schedule a real restore of one service to a scratch directory monthly and time it — that number is your true recovery objective, and it is always longer than anyone guesses.
Monitor the monitoring
A self-hosted observability stack has an obvious flaw: when the host dies, so does everything that would have told you. Close it with two cheap measures.
First, an external uptime check from a third party — a free tier is fine — that watches the monitoring host itself. Second, a dead-man's switch: your stack pings an external endpoint every few minutes, and that service alerts you when the pings stop. Inverting the signal catches total failure, which normal alerting cannot.
# pushes only if the local health check passes
*/5 * * * * curl -fsS --max-time 10 http://localhost:9090/-/healthy >/dev/null \
&& curl -fsS --max-time 10 https://your-deadman-endpoint/ping >/dev/null
Lock the perimeter
One box now holds metrics, error traces containing request payloads, CI credentials and possibly business data. Treat it accordingly:
-
Nothing internal on a public port. Bind services to
127.0.0.1and expose only the reverse proxy. A dashboard tool reachable from the internet with default credentials is a well-known way to lose a server. - SSH keys only, password authentication disabled, plus fail2ban or equivalent.
- Authentication in front of every UI, including the ones you assume nobody knows about. Metrics endpoints leak internal hostnames, versions and topology.
- Secrets out of compose files. Environment files with restrictive modes, excluded from version control, rotated when anyone leaves.
- Unattended security updates enabled, with reboot windows scheduled rather than deferred indefinitely.
What we do not recommend self-hosting
Being honest about the boundary is what makes the rest credible:
| Do self-host | Do not |
|---|---|
| Metrics and dashboards — stable, low-maintenance, data grows predictably | Outbound email delivery — deliverability is a full-time reputation problem; use a provider |
| Error tracking — high SaaS cost per event, tolerable to operate | Authentication for customers — the blast radius of getting it wrong is unbounded |
| CI for your own repositories | Payments — compliance scope you do not want |
| Internal business systems | Your status page — it must survive your infrastructure failing |
| Uptime probing (as a second opinion) | Primary DNS — anycast networks are better and effectively free |
The real cost calculation
The subscription saving is easy to compute and easy to overstate. The honest version includes:
- The VPS itself, plus a second small instance if you want offsite backup separation.
- Roughly half a day per month of routine maintenance — updates, disk pressure, retention tuning.
- Occasional incident time when a component surprises you, front-loaded in the first two months.
- The knowledge concentration risk: if exactly one person understands the stack, that is an availability problem. Write the runbook before you need it.
For a team already running its own servers and comfortable with Linux, this trade is usually clearly positive. For a team without that baseline, a managed service is genuinely the cheaper option once you price the time correctly.
Summary
- Budget memory before installing anything; cap containers and add swap you alert on.
- Open port 80 for HTTP-01, and monitor certificate expiry rather than trusting the renewal cron.
- Pin your monitoring host to public DNS resolvers so probes see what users see.
- Pull backups from a host the servers cannot write to, and verify restores on a schedule.
- Watch the watcher with an external check and a dead-man's switch.
- Keep email, customer auth, payments and your status page off the box.
We design, deploy and operate infrastructure like this for companies across Egypt and the Gulf — including the monitoring, backup verification and incident response that make it safe to run yourself. If you want a review of your current setup before it becomes an incident, our team is available.
FAQ
How much server do I need to self-host a monitoring and error-tracking stack?
For a small team, 8 vCPU and 16 GB of RAM is a comfortable starting point, and the memory matters far more than the cores. Self-hosted error-tracking platforms are the dominant consumer — they run as a group of web workers, background workers, a database, a cache and a queue, and typically need 4 to 6 GB on their own. Metrics, dashboards and probes together fit in about 2 GB. Add swap and cap every container so one runaway process cannot trigger the OOM killer.
Why did my Let's Encrypt certificates stop renewing after I enabled a firewall?
The HTTP-01 challenge requires the certificate authority to reach your server on port 80. If the firewall drops it, renewals fail silently from cron and you only find out roughly sixty days later when every service on the host goes untrusted at once. Allow 80/tcp even if you redirect all traffic to HTTPS, or switch to a DNS-01 challenge. Then monitor certificate expiry as a metric and alert with about three weeks of margin.
Should backups be pushed from servers or pulled by the backup host?
Pulled. If each server pushes, every server holds credentials that can delete or encrypt the backup repository, so a single compromised host can destroy all your backups. With a pull model the backup host connects outward using an SSH key restricted to a read-only command, and the servers hold no backup credentials at all. Add retention, and verify with periodic real restores rather than trusting that the job exited zero.
What should I never self-host?
Outbound email delivery, customer authentication, payment processing, primary DNS, and your status page. Email deliverability is an ongoing reputation problem that a provider solves better; auth and payments carry a blast radius and compliance scope that is not worth owning; anycast DNS from a provider is better and essentially free; and a status page hosted on your own infrastructure goes down exactly when you need it.
How do I get alerted when the monitoring server itself goes down?
Two independent mechanisms. Use an external uptime service to watch the monitoring host from outside your network, and add a dead-man's switch — have the stack ping an external endpoint on a schedule and configure that service to alert when the pings stop rather than when they fail. Inverting the signal is what catches total host failure, which internal alerting by definition cannot report.
Originally published at web-pioneer.com.
Top comments (0)