DEV Community: Opservo

How to monitor Docker memory usage

Opservo — Sun, 01 Nov 2026 08:45:14 +0000

A container that eats memory silently can OOM-kill your app at 3 am, restart your database mid-transaction, or starve every other service on the host. Most teams notice only after the crash. This guide walks through every useful way to watch Docker memory in real time and over time, what the numbers actually mean, and what thresholds are worth alerting on.

What docker stats shows you (and what it leaves out)

Run docker stats with no arguments and Docker prints a live table refreshed every second. The columns that matter for memory are MEM USAGE, LIMIT, and MEM %.

# Live table, all running containers
docker stats

# Example output:
# CONTAINER ID   NAME         MEM USAGE / LIMIT     MEM %
# a3f1b2c4d5e6   api-prod     412MiB / 2GiB         20.12%
# 9e8d7c6b5a4f   redis        88.3MiB / 512MiB      17.24%
# 1a2b3c4d5e6f   postgres     310MiB / 4GiB         7.57%

# One snapshot (no live refresh), all containers
docker stats --no-stream

# Sort by memory (highest first) using a format and sort
docker stats --no-stream --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}" \
  | sort -t$'\t' -k3 -rh

# Watch a single container
docker stats --no-stream my_container

The docker stats sort by memory trick above uses --format to get tab-separated output and then sorts on the MEM % column. That is the fastest way to find your worst offender across a fleet of containers.

One important gap: docker stats shows RSS plus cache together as MEM USAGE. That can look higher than the real pressure. Cache memory is reclaimed under load, so the number the kernel cares about is RSS (resident set size) alone. More on reading the raw cgroup file below.

How to monitor Docker performance beyond the stats table

Memory alone rarely tells the whole story. A container can sit at 80 % memory and be completely healthy, or it can sit at 40 % and be thrashing swap. Pair memory with two other signals to understand real performance:

CPU throttling: check docker stats CPU % against the container's CPU quota; if it is consistently at the quota ceiling, the app is CPU-starved not memory-starved.
Page faults: high major page faults in /sys/fs/cgroup/memory/docker/<id>/memory.stat mean the kernel is reading data from disk -- a sign of memory pressure even when the headline number looks low.
Restart count: docker inspect --format '{{.RestartCount}}' <name> tells you how many times the container has been killed and restarted; any non-zero value after a fresh deploy is a red flag.

For a broader look at host-level pressure that Docker contributes to, the checklist in Why is my server slow? A 10-minute checklist walks through CPU, memory, disk, and network in one pass. It is worth keeping open alongside this guide.

To watch per-core CPU across all containers on the host, open htop and press F2, then add the CGROUP column so you can see which cgroup each process belongs to. That links a runaway process back to its container without needing container IDs.

Reading cgroup memory files directly

The kernel exposes container memory through cgroup v1 or v2 files. These are the ground-truth numbers Docker stats reads from. Checking them directly is useful when you need scriptable output or when you want the RSS figure without cache.

# Find the container's full ID
CID=$(docker inspect --format '{{.Id}}' my_container)

# --- cgroup v1 (most pre-2022 hosts) ---
# Current usage in bytes
cat /sys/fs/cgroup/memory/docker/${CID}/memory.usage_in_bytes

# Limit in bytes (-1 means no limit set)
cat /sys/fs/cgroup/memory/docker/${CID}/memory.limit_in_bytes

# RSS only (excludes page cache)
grep ^rss /sys/fs/cgroup/memory/docker/${CID}/memory.stat
# rss 423231488  <- bytes of real in-use RAM

# OOM kill count (non-zero = container has been OOM-killed before)
cat /sys/fs/cgroup/memory/docker/${CID}/memory.oom_control
# oom_kill_disable 0
# under_oom 0
# oom_kill 3      <- this container has been OOM-killed 3 times

# --- cgroup v2 (Ubuntu 22.04+, Debian 12+, Amazon Linux 2023) ---
cat /sys/fs/cgroup/system.slice/docker-${CID}.scope/memory.current
cat /sys/fs/cgroup/system.slice/docker-${CID}.scope/memory.max
grep anon /sys/fs/cgroup/system.slice/docker-${CID}.scope/memory.stat
# anon 423231488  <- equivalent of RSS in v2

The oom_kill counter is the most actionable number here. If it is greater than zero, the container has already been silently restarted by the kernel. Many teams never see these kills because Docker restarts the container immediately and the log shows nothing obvious.

Setting and checking memory limits

A container without a memory limit borrows from the whole host. One leak takes everything down. Set limits in your Compose file or run command:

# In docker-compose.yml (Compose v3)
services:
  api:
    image: my-api:latest
    deploy:
      resources:
        limits:
          memory: 512m
        reservations:
          memory: 256m

# Or with docker run
docker run -d --memory=512m --memory-reservation=256m my-api:latest

# Confirm the limit is applied
docker inspect --format '{{.HostConfig.Memory}}' my_container
# 536870912   <- 512 MiB in bytes; 0 means no limit

The --memory-reservation (soft limit) lets the kernel reclaim from the container under host pressure without killing it. The hard --memory limit is what triggers the OOM kill. Both are worth setting in production.

If you run Laravel queue workers or Horizon inside a container, set a generous limit and watch it over a few days before tightening it. Laravel workers accumulate memory across jobs if --max-memory is not set on the queue command. See monitor Laravel in production for what to watch specifically.

Redis in a container also deserves a hard limit. If Redis has no maxmemory directive in its config and no cgroup limit, it will consume all available RAM on the host. Check current memory use with docker exec my_redis redis-cli info memory | grep used_memory_human. More detail on Redis memory pressure is covered in how to monitor Redis memory and eviction.

Docker monitor logs: finding OOM and memory errors

When a container OOM-kills, the event shows up in two places: the kernel ring buffer and the Docker daemon logs.

# Kernel OOM events (look for 'Killed process' or 'oom-kill')
sudo dmesg | grep -iE 'oom|killed process' | tail -20

# Docker daemon log (systemd installs)
sudo journalctl -u docker.service | grep -iE 'oom|kill' | tail -20

# Container's own stdout/stderr since last restart
docker logs --since 1h my_container 2>&1 | tail -50

# Follow logs in real time
docker logs -f my_container

# Events stream: catch OOM kills as they happen
docker events --filter event=oom

The docker events --filter event=oom command is the cleanest way to watch for kills in real time. Pipe it to a log file or a monitoring agent. If you want to understand what else might be going wrong at the host level alongside these kills, the guide on how to fix high memory usage on a Linux server covers identifying which process or container is responsible and whether swappiness is making things worse.

What to alert on: concrete thresholds for Docker memory

Not every memory spike deserves a page. Use these thresholds as a starting point and adjust for your workload:

MEM % above 85 % of the container limit for more than 5 minutes: warn. Above 95 % for more than 1 minute: page. At 100 % the kernel kills the process.
OOM kill count increments: alert immediately, every time. This is a production incident whether or not the container recovered.
Container restart count increases: alert on any new restart after the initial deploy period. Use docker inspect --format '{{.RestartCount}}' in a cron or monitoring script.
No memory limit set (docker inspect returns 0): flag at deploy time, not at 3 am. Add a CI check that rejects images deployed without a memory limit.
Host free memory below 10 %: even with per-container limits, the host can still be pressured if many containers are near their limits simultaneously. Check with free -h and watch MemAvailable in /proc/meminfo.

Rule: if a container has no memory limit, it can silently consume the entire host. Set --memory on every production container, even a generous one, so the kernel kills one service rather than the whole machine.

Keeping this visible without running commands every day

Running docker stats by hand is fine for a one-off check. It does not scale to a fleet where you need to know which container drifted past its memory budget overnight, or which one OOM-killed twice this week while the on-call was asleep.

Opservo auto-discovers every Docker container on your Linux servers and tracks memory usage, restart counts, and OOM events continuously. When a container approaches its limit or gets killed, you get a plain-English alert with the container name, the current memory figure, the limit, and a suggested fix, without writing a single PromQL query or Grafana dashboard. The agent is a single static Go binary under 10 MB, installs in one command, and is read-only by default.

Invite the reader to install Opservo's free 2-server plan to get continuous container memory tracking and OOM alerts without running docker stats by hand. — one command installs a read-only agent and your first findings arrive in five minutes: start free. Related: blog how to monitor a docker container s cpu and memory usage.

Originally published on the Opservo blog.

When the person who knew your servers leaves: a five-day handover checklist

Opservo — Tue, 27 Oct 2026 13:33:18 +0000

Most agencies running 10 to 50 servers have one person who knows how they work. Which box runs which client. How to restart the queue when emails stop. Why one cron job runs at 3 a.m. and what breaks if it does not.

Nobody plans for that person to leave, go on a long holiday or move to another team. When it happens, you usually get two weeks of notice and a document that was last updated a year ago.

This is the checklist we would hand any agency in that situation. It takes five working days. Do it with the person while they are still there, and do every step from the servers themselves, not from memory.

Day 1: List what actually runs on every server

For each server, write down the provider, the client it belongs to, who has root access, and what is running on it. Get the last part from the server, not from the person:

systemctl list-units --type=service --state=running
docker ps
pm2 list
ls /etc/nginx/sites-enabled /etc/apache2/sites-enabled 2>/dev/null

Anything on that output that nobody can explain goes on a list for day 5.

Day 2: Find everything that expires

SSL certificates, domains, and any API keys or tokens with an end date. For certificates managed by Certbot, certbot certificates lists them with their expiry dates. Put every date for the next 12 months in a shared calendar, with two people invited, not one.

Day 3: Find everything that runs on a schedule

Scheduled jobs are where silent failures live, because nobody notices a job that stopped running until a client asks where their report went.

crontab -l -u <each user>
ls /etc/cron.d
systemctl list-timers

For each job, note what it does and who would notice if it stopped. Those are the ones that need a check, not just a log line.

Day 4: Find out who gets told when something breaks

List every alert channel: email addresses, Slack channels, webhooks, phone numbers. Then send a test alert through each one and confirm a person saw it. If any of them still point at the person who is leaving, change them today. We learned this one the hard way: for 59 days our own alerts went to a webhook that rejected every one of them, and nobody noticed.

Day 5: Write the three restart notes

For each client, one page covering the three things that break most often in agency stacks: the queue workers, the web server and the database. What "broken" looks like, the command to check, the command to restart, and who to call if that does not work. Go through the day-1 "nobody can explain" list with the person before they leave.

Why this goes out of date, and what to do about it

A handover document is right on the day it is written. Then someone adds a container, a certificate is renewed by hand, a new cron job appears, and the document is wrong again. Re-run days 1 to 4 every month, or use something that reads the list from the servers for you.

Where Opservo fits

Opservo is built for exactly this gap. One command installs a read-only agent on each Linux server, with no inbound ports. Within minutes it lists what runs there from the server itself: systemd services, Docker containers, PM2 processes, nginx and Apache sites with their SSL expiry, MySQL, PostgreSQL and Redis, and Laravel, WordPress and Node apps. Cron heartbeats tell you when a scheduled job stops checking in. Every alert says in plain English what is wrong and what to do, so the person on call does not need to be the person who set it up, and any fix that runs is audited.

It is free for 2 servers, no card. Start at getopservo.com/welcome, or see everything it monitors.

Try Opservo free on 2 servers — one command installs a read-only agent and your first findings arrive in five minutes: start free.

Originally published on the Opservo blog.

How to monitor network traffic on Linux

Opservo — Sun, 25 Oct 2026 07:46:13 +0000

Your server feels slow, but CPU and memory look fine. The culprit is often the network: a process hammering an external API, a container leaking connections, a misconfigured queue worker retrying thousands of failed jobs per minute. If you run a few Linux servers without a dedicated SRE, knowing which commands to reach for, and what numbers actually matter, saves you from chasing the wrong thing for an hour.

What you are actually trying to answer

Network monitoring on Linux usually comes down to four questions: How much bandwidth is each interface using right now? Which process or connection is responsible? Is the volume trending in a direction that will cause a problem? And are any external dependencies (SSL certificates, DNS, upstream APIs) quietly failing? Those four questions need different tools, and most guides only cover the first one.

The tools worth knowing, and when to use each one

Most Linux distributions ship with or can quickly install the following. You do not need all of them; knowing what each one is good at stops you from reaching for the wrong one.

ss: the modern replacement for netstat. Shows every open socket, which process owns it, and its state. Fast and accurate. Use it first.
ip -s link: reads interface counters directly from the kernel. Good for a quick bytes-in/bytes-out snapshot without installing anything.
nload: a terminal dashboard showing per-interface throughput in real time, in and out, with a rolling average. Easy to read at a glance.
iftop: shows bandwidth per connection pair (source IP to destination IP), sorted by volume. Useful for spotting a specific remote host consuming disproportionate bandwidth.
nethogs: groups bandwidth by process rather than by connection. This is the tool to reach for when you know the interface is busy but not which process is responsible.
tcpdump: raw packet capture. Overkill for everyday monitoring but essential for diagnosing TLS handshake failures, DNS misbehaviour, or unexpected traffic on a port.
vnstat: logs interface traffic to a small database and lets you query hourly, daily, and monthly totals. Lightweight and persistent across reboots.

A practical command sequence for diagnosing traffic in five minutes

Run these in order. Each one narrows the search. The expected output comments explain what you are looking for.

# 1. Check raw interface counters (no install needed)
ip -s link show eth0
# Look for: RX errors > 0 or TX dropped > 0 -- these suggest a hardware or MTU issue

# 2. See total throughput per interface in real time
# Install: apt install nload   OR   yum install nload
nload eth0
# Look for: Curr (current) vs Avg; spikes well above the average warrant investigation
# Press right arrow to switch interfaces; q to quit

# 3. Find which remote hosts are taking the most bandwidth
# Install: apt install iftop   OR   yum install iftop
sudo iftop -i eth0 -n
# -n suppresses DNS lookups so the display updates faster
# Look for: any single host consuming > 50% of visible bandwidth
# Press p to toggle ports; q to quit

# 4. Find which local process owns the traffic
# Install: apt install nethogs   OR   yum install nethogs
sudo nethogs eth0
# Look for: a process with send+received KB/s far above the others
# Press q to quit

# 5. Check open connections and listening ports
ss -tulnp
# -t TCP  -u UDP  -l listening  -n numeric  -p process
# Look for: unexpected LISTEN entries on ports you did not open

# 6. Count connections per remote IP (useful for spotting floods)
ss -n state established '( dport = :443 or dport = :80 )' \
  | awk '{print $5}' \
  | cut -d: -f1 \
  | sort | uniq -c | sort -rn | head -20
# Look for: any single IP with > 100 simultaneous connections

# 7. Check historical daily traffic totals
# Install: apt install vnstat   OR   yum install vnstat
# Then enable and start: systemctl enable --now vnstat
vnstat -d
# Look for: days where RX or TX is 2-3x the usual daily average

What to alert on: concrete thresholds

Tools give you numbers. Thresholds turn numbers into actionable signals. The right threshold depends on your server's role, but the following starting points work for most application servers.

Inbound bandwidth sustained above 80% of your provisioned link for more than five minutes: alert immediately. On a 1 Gbps port, that is 800 Mbps. On a shared 100 Mbps DigitalOcean droplet, it is 80 Mbps.
TX errors or RX drops on an interface greater than zero over a five-minute window: worth a warning. A handful of drops is normal; a steady climb suggests a congested uplink or a misconfigured MTU.
Established TCP connections to a single remote IP exceeding 200: investigate for a connection leak or an aggressive client. Web scraper traffic often looks like this.
CLOSE_WAIT connections growing and not draining: the local application is not closing sockets properly. Check your application process first with nethogs.
A new LISTEN port that was not there yesterday: treat as a security event until proven otherwise. ss -tulnp captures a snapshot; diff two snapshots taken 24 hours apart to spot changes.
DNS resolution time exceeding 200 ms on repeated lookups: your upstream resolver may be failing over. dig +stats your-domain.com @8.8.8.8 shows the query time in the stats section.

Rule of thumb: before you add network capacity, check whether a single process or a single remote IP is responsible. Scaling the network when one misbehaving process is the cause wastes money and leaves the real problem in place.

How to monitor containers, services, and SSL alongside network traffic

Network traffic rarely fails in isolation. A spike in outbound traffic from a container often means a queue worker is retrying failed jobs at high frequency, or a deployment pushed bad code that is logging errors in a tight loop. Watching the network without watching what is generating the traffic leaves you guessing.

For Docker containers, cgroup-level network counters are exposed in /sys/fs/cgroup/net_cls and through the Docker stats API. Running docker stats --no-stream gives you a per-container snapshot of network I/O alongside CPU and memory. If you run Laravel on a server and use Horizon or Supervisor to manage queue workers, a worker that is crash-looping will produce a burst of short-lived connections that look like a traffic spike in iftop. The guide on how to monitor containers on a VPS covers the cgroup approach in more detail.

SSL certificate expiry is a network-adjacent problem that catches teams off guard. A certificate that expires quietly takes down HTTPS traffic with no process crash and no disk alert. You can check expiry from the command line with: echo | openssl s_client -connect yourdomain.com:443 -servername yourdomain.com 2>/dev/null | openssl x509 -noout -dates. The notAfter line tells you when it expires. Automate a check that alerts at 30 days and again at 7 days.

DNS monitoring sits in the same category. A domain whose nameserver TTL has changed, or whose A record has been altered, will cause traffic to disappear rather than spike. A daily dig check against an authoritative nameserver and a trusted public resolver lets you spot divergence quickly.

When manual tools are not enough for a production fleet

The commands above are exactly what you should run during an active incident. For ongoing visibility across multiple servers, you need something that runs those checks continuously, stores the trends, and tells you when a threshold is crossed, without you staying logged in. A linux server monitoring tool that auto-discovers listening ports, tracks connection counts over time, and correlates a network spike with a log burst saves significant investigation time.

Opservo installs with a single command and auto-discovers listening ports, Docker containers, systemd services, and SSL certificate expiry on every server it monitors. When a network threshold is crossed, it correlates the spike with log entries from the same time window and surfaces a plain-English explanation of what changed. For Laravel monitoring, it watches Horizon and Supervisor workers and alerts if one stops or enters a crash loop, which is often the source of that unexpected outbound traffic surge. Remote actions are opt-in per server, so every fix command is approved and audited before it runs.

# Install the Opservo agent (Linux only; single static binary under 10 MB)
curl -sSL https://api.getopservo.com/install.sh | sudo bash
# No inbound ports opened; all traffic is outbound HTTPS
# First findings appear within five minutes

Install the Opservo agent on up to two servers free (no card required) to get continuous network-threshold alerts, port-change detection, and SSL expiry monitoring without staying logged in. — one command installs a read-only agent and your first findings arrive in five minutes: start free. Related: guides monitor docker containers on a vps.

Originally published on the Opservo blog.

Our filesystem went read-only for 19 days. Nobody saw the warning.

Opservo — Tue, 20 Oct 2026 12:58:41 +0000

One of our servers fired a kernel filesystem-error alert. Nine days later, the OS remounted the filesystem read-only. Ten days after the first alert, a service on the same server entered a failed state. All three alerts sat unacknowledged for between 11 and 19 days. No alert channel had been configured for that server, so every notification went nowhere.

What the alerts actually said

The first alert read: "Kernel reported filesystem errors -- schedule an fsck and check the underlying disk; back up first." It opened on 2026-09-10 and was notified once. It stayed open for about 11 days.

8.6 days after the first alert, a second one opened: "A filesystem was remounted read-only after errors -- a read-only remount means the disk or filesystem hit errors. Check SMART health, run fsck on the affected mount, and expect a reboot." That alert ran for about 19 days before it closed. Also notified once, never acknowledged.

10.8 days after the first alert, a third opened: "Service apport-autoreport is down." That one ran for about 17 days. Same story: one notification, no acknowledgement.

We are not claiming these three alerts are causally linked. What we can say is that they appeared on the same server in that order, and none of them reached a human.

Why a read-only remount is a serious signal

When the Linux kernel detects filesystem errors it cannot recover from, it remounts the affected filesystem read-only as a last resort to stop further corruption. At that point, any process trying to write -- a database, a web server, a log daemon -- either fails silently or crashes. Applications may appear to be running while actually writing nothing.

The kernel logs the event in the system journal. It also shows up in dmesg. The problem is that neither of those surfaces alerts you unless something is watching.

A read-only remount does not mean your data is safe. It means the kernel stopped accepting writes before things got worse. The underlying disk problem is still there.

How to check your own servers right now

Run these commands on any Linux server you care about. The first block checks for a current read-only mount; the second checks the kernel ring buffer for recent filesystem errors; the third runs a SMART health check on the underlying block device.

# 1. Find any filesystem currently mounted read-only
cat /proc/mounts | awk '$4 ~ /(^|,)ro(,|$)/ {print $2, $3, $4}'

# 2. Check the kernel log for filesystem errors in the last 500 lines
dmesg | tail -500 | grep -iE 'error|remount|read-only|EXT4-fs|XFS|I/O error'

# 3. Check SMART health on your primary disk (replace sda with your device)
sudo smartctl -H /dev/sda
# "SMART overall-health self-assessment test result: PASSED" is what you want
# "FAILED" means the drive is reporting imminent failure

# 4. Look for recent journal-level filesystem errors
journalctl -p err -b | grep -iE 'filesystem|disk|I/O|remount|fsck'

# 5. Check for any failed mount units
systemctl --failed --type=mount

If dmesg shows lines like EXT4-fs error (device sda1) or XFS: corruption of in-memory data detected, stop writes to that filesystem if you can, take a backup immediately, and schedule an fsck on the unmounted device. Do not run fsck on a mounted filesystem.

What a SMART check actually tells you

SMART (Self-Monitoring, Analysis and Reporting Technology) is built into every modern hard drive and SSD. The tool smartctl reads it. Install it with sudo apt install smartmontools or sudo yum install smartmontools.

The attributes that matter most for predicting failure are: Reallocated_Sector_Ct (non-zero means the drive has already remapped bad sectors), Current_Pending_Sector (sectors the drive has flagged as unstable), and Offline_Uncorrectable (sectors the drive could not correct during offline scanning). Any non-zero value in those three columns deserves attention.

# Full attribute list; look for non-zero RAW_VALUE in columns 5, 197, 198
sudo smartctl -A /dev/sda | grep -E 'Reallocated|Pending|Uncorrectable'

# Example output you do NOT want to see:
#   5  Reallocated_Sector_Ct   0x0033  100  100  036  Pre-fail  Always  -  17
# 197  Current_Pending_Sector  0x0012  100  100  000  Old_age   Always  -  4
# 198  Offline_Uncorrectable   0x0010  100  100  000  Old_age   Offline -  1

The real mistake: no alert channel on that server

All three alerts were generated and each was notified once. The first alert's detail even said "back up first". But we had not configured an alert channel for that server, so each notification had nowhere to go. Zero channels were configured, and no person acknowledged any of the three alerts.

We have written about this failure mode before in "Our alerts went nowhere for 59 days. Here is what we changed." This is a different server, a different set of alerts, and the same root cause: alert channel not configured.

The lesson is not about disk health specifically. It is that any server without a configured alert destination is a server you are monitoring in name only. Go through your fleet and verify that every server has at least one channel -- email is enough -- and that a test alert actually arrives.

Monitoring without a delivery channel is a log you never read. The alert count does not matter; the delivery count does.

How Opservo surfaces this

Opservo watches for kernel filesystem errors and read-only remounts as health-category alerts, separate from lower-priority recommendations. When a server has no alert channel configured, the dashboard flags it explicitly so you can see the gap before an incident reveals it. The health score docks points for both the filesystem condition and the missing notification path, so a server in this state cannot quietly score well.

Try Opservo free on 2 servers — one command installs a read-only agent and your first findings arrive in five minutes: start free.

Originally published on the Opservo blog.

Why our server health score is not AI

Opservo — Sun, 18 Oct 2026 07:35:47 +0000

Opservo uses AI in a lot of places. It explains what is wrong with a server in plain English, clusters log lines into patterns, forecasts when a disk will fill, and answers questions in a chat grounded in live data from the server. One thing it deliberately does not use AI for is the health score.

The problem with a score you cannot question

Most monitoring tools that show a single "health" number do not tell you how they got it. If a server drops from 92 to 71 overnight, you are left guessing: was it the disk, the certificate, a service that restarted? A number you cannot question is a number you learn to ignore.

We run 40+ production servers at Softuvo, mostly looked after by developers doing ops on the side. For them, a score is only useful if it answers the next question on its own: what lost the points, and what do I do about it?

What the Opservo Health Standard does instead

Every server gets a score from 0 to 100, built from seven areas: Performance, Security, Storage, Reliability, Maintenance, Network and Configuration. The score comes from a fixed set of rules, not a model. Two things follow from that:

It is deterministic. The same server in the same state always gets the same score. Nothing about the score depends on how a model felt that day.
Every lost point is explained. Each deduction names four things: the rule, the threshold, the current value, and the fix. If you disagree with a deduction, you know exactly what you are disagreeing with.

Where the AI comes in

The score tells you what. The AI helps with why and what next. When a server loses points, Opservo explains in plain English what changed and why it matters, and when the fix is known you can apply it with one click or a policy you approved in advance. The agent is read-only by default, and remote actions are opt-in per server and audited.

Keeping the two apart means the AI can be wrong in a sentence without making the score wrong. You can always fall back to the rule, the threshold and the value.

Three questions to ask of any health score

Can it show you every point it took away? If not, you cannot tell a real problem from noise.
Does the same state give the same number? If the score moves when nothing on the server changed, you will stop trusting it.
Does each deduction come with a fix? A score that only grades you is a report card. A score that tells you what to do next is a to-do list.

Try it on your worst server

The full standard is public at getopservo.com/health-score. Opservo is the AI ops engineer for teams without an SRE: plain-English server monitoring with one-click fixes. Free for 2 servers. Start at getopservo.com/welcome.

Try Opservo free on 2 servers — one command installs a read-only agent and your first findings arrive in five minutes: start free.

Originally published on the Opservo blog.

How to Monitor Disk I/O Latency on a Linux Server and Find the Bottleneck

Opservo — Sun, 18 Oct 2026 06:37:17 +0000

Your app feels sluggish, queries are timing out, and top shows CPU is fine. Nine times out of ten the culprit is disk I/O — but knowing that doesn't tell you which disk, which process, or whether it's a hardware problem or just one runaway job hammering the filesystem. This guide walks through exactly how to measure I/O latency on a Linux server and narrow it down to the real bottleneck.

Start With iostat — Your First View Into Disk Performance

The fastest way to get a snapshot of disk activity is iostat, part of the sysstat package. If it's not installed, run: sudo apt install sysstat (Debian/Ubuntu) or sudo yum install sysstat (RHEL/CentOS).

Run it with extended stats and a refresh interval: iostat -xz 2 10 — that gives you extended metrics, hides idle devices, refreshes every 2 seconds, and runs 10 times. The columns that matter most for latency are:

await — average time (ms) for I/O requests to complete, including queue wait. This is your headline latency number.
r_await / w_await — separate read and write latency, so you can tell if it's reads or writes causing pain.
svctm — deprecated, ignore it.
%util — how busy the device is. Near 100% means it's saturated, but on SSDs and NVMe drives this can be misleading because they handle parallel I/O.

A healthy NVMe SSD should show await below 1 ms. A SATA SSD might run 1–5 ms. A spinning disk is acceptable around 5–20 ms but anything above that consistently signals a problem. If you see await spiking to 100 ms or more, something is seriously wrong.

Use iotop to Find the Offending Process

Once you know which disk is struggling, the next question is: who is hammering it? iotop shows per-process I/O in real time, similar to top for CPU. Install it with sudo apt install iotop, then run: sudo iotop -o — the -o flag shows only processes that are actively doing I/O, keeping the output clean.

Look for the DISK READ and DISK WRITE columns. A process reading or writing hundreds of megabytes per second when it shouldn't be is your bottleneck. Note the PID, then cross-reference with ps aux or lsof -p to see what files it's accessing.

Pro tip: if iotop shows I/O is spread evenly across many processes rather than one dominant culprit, the problem is likely at the storage layer itself — a failing drive, a full disk cache, or a RAID controller under pressure.

Go Deeper With blktrace and biolatency

If iostat and iotop point to a disk but not a clear cause, you need to look at the I/O stack more granularly. Two tools are especially useful here.

blktrace captures every block I/O event on a device. Run sudo blktrace -d /dev/sda -o trace for 10–15 seconds, then stop it with Ctrl+C and analyse with blkparse: blkparse -i trace.blktrace.0 | tail -50. You'll see queue times vs. service times broken down by I/O type. This is verbose but reveals whether latency is happening in the kernel queue or at the hardware level.

If you're running a modern kernel (4.7+) with BPF tools installed, biolatency from the BCC toolkit is far more ergonomic: sudo biolatency -d /dev/sda 1 10. It prints a histogram of I/O completion times, showing you the distribution rather than just averages. Averages hide bimodal latency — where most I/Os complete fast but a small percentage take seconds and drag down your p99 response times.

Check for Filesystem and Mount-Level Issues

Sometimes the disk hardware is fine but the filesystem is the bottleneck. A few things to check:

Disk fullness — run df -h. Linux filesystems degrade badly above ~90% capacity. Inodes can also be exhausted separately: df -i.
Dirty page writeback — cat /proc/meminfo | grep -i dirty. If dirty pages are consistently high (gigabytes), the kernel is accumulating writes faster than it can flush them. Tune vm.dirty_ratio and vm.dirty_background_ratio in /etc/sysctl.conf.
Mount options — check /proc/mounts. If a database volume is mounted without noatime, every read triggers a metadata write. Add noatime to the mount options in /etc/fstab and remount.
Filesystem errors — sudo dmesg | grep -i 'error|EXT4|XFS|I/O error'. Hardware errors show up here before drives fully fail.

Build a Baseline and Set Alerts Before It Gets Bad

Reactive troubleshooting is fine, but the real goal is catching latency before it becomes an outage. Run iostat -xz 1 60 > baseline-$(date +%F).txt during normal operating hours and save the output. Now you have a reference for what normal looks like — which makes future spikes obvious rather than ambiguous.

You can wire iostat into basic alerting by parsing await in a cron job or a shell script that pushes to Slack when it crosses a threshold. For something more automatic, Opservo continuously tracks I/O metrics across your servers, surfaces latency anomalies in plain English, and can correlate a slow-disk event with the specific process responsible — without you having to remember which command to run at 2 AM.

Whether you choose to script your own monitoring or use a tool, the key habit is the same: don't wait for users to report slowness. Disk latency tends to creep up gradually — a drive entering the early stages of failure, a cron job that gets bigger every week, a log directory nobody ever cleans up. Measure it regularly and you'll almost always see the problem coming.

Invite the reader to connect their server to Opservo's free 2-server plan so I/O latency and process-level disk activity are tracked automatically without keeping a cron script alive. — one command installs a read-only agent and your first findings arrive in five minutes: start free. Related: guides why is my server slow.

Originally published on the Opservo blog.

How to Monitor and Alert on Redis Memory Fragmentation Before It Causes Latency Spikes

Opservo — Sat, 10 Oct 2026 15:36:24 +0000

Your Redis instance looks healthy — low CPU, plenty of free memory, stable connection count. Then latency spikes out of nowhere, and you spend an hour debugging before realizing the culprit was memory fragmentation. It's one of the most common Redis performance issues that teams miss because it doesn't show up in the obvious metrics. Here's how to catch it before it bites you.

What Redis Memory Fragmentation Actually Is

Redis allocates memory in chunks through its memory allocator (jemalloc by default). Over time, as keys are created, updated, and deleted at varying sizes, gaps form between those chunks — memory the OS has handed to Redis but Redis isn't actively using for data. The ratio between the memory Redis holds from the OS versus the memory Redis actually uses for data is your fragmentation ratio. A ratio of 1.0 means perfect efficiency. Anything significantly above 1.5 is where problems start, and above 2.0 you'll almost certainly see latency impact.

Fragmentation hurts performance because the allocator has to work harder to find contiguous memory regions, and in extreme cases Redis will trigger active defragmentation, which competes with your normal workload. The insidious part: your application sees slow responses but your system-level memory graphs look fine.

The Metrics You Need to Watch

Everything you need lives inside Redis INFO output. Run this against your instance:

redis-cli -h 127.0.0.1 -p 6379 INFO memory

The key fields to extract:

used_memory — bytes Redis allocated for data
used_memory_rss — bytes the OS reports Redis is using (resident set size)
mem_fragmentation_ratio — the ratio of rss to used_memory; this is your primary signal
mem_fragmentation_bytes — absolute bytes wasted, useful for context
active_defrag_running — 1 means Redis is actively defragmenting right now
allocator_frag_ratio — fragmentation inside the allocator itself, separate from OS-level fragmentation

Don't rely on mem_fragmentation_ratio alone. A ratio of 1.8 on an instance using 500 MB of data is very different from 1.8 on an instance using 50 GB. Always look at mem_fragmentation_bytes alongside it — if you're wasting 200 MB, that's a real problem; if you're wasting 4 MB, it's noise.

Setting Meaningful Alert Thresholds

Generic thresholds don't work well here because workloads differ. That said, sensible starting points are:

mem_fragmentation_ratio > 1.5 — warning; begin watching closely
mem_fragmentation_ratio > 2.0 — critical; investigate immediately
mem_fragmentation_bytes > 500 MB — critical regardless of ratio
active_defrag_running = 1 for more than a few minutes — worth noting; it means Redis already decided things were bad enough to act

If you're using Prometheus with the redis_exporter, you can alert on these directly. A Prometheus alerting rule for the critical case looks like this:

alert: RedisHighFragmentation expr: redis_memory_used_rss_bytes / redis_memory_used_bytes > 2.0 for: 5m labels: severity: critical annotations: summary: "Redis memory fragmentation ratio above 2.0 for 5 minutes"

The for: 5m clause matters. Fragmentation naturally spikes briefly during bulk deletes or expiry storms, so you want to alert on sustained high fragmentation, not momentary blips.

How to Reduce Fragmentation When It Gets Bad

Once you've caught fragmentation early, you have a few options depending on severity:

Enable active defragmentation — if you're on Redis 4.0+, you can turn this on at runtime: redis-cli CONFIG SET activedefrag yes. Redis will gradually repack memory in the background. Set active-defrag-ignore-bytes 100mb and active-defrag-enabled yes in your config to make it permanent.
Restart Redis — the nuclear option, but sometimes the fastest path. Redis will reload from an RDB or AOF snapshot and start with a clean memory layout. Only practical if you have replication set up and can failover cleanly.
Tune your key expiry and eviction policy — fragmentation often worsens with volatile-lru or allkeys-lru eviction under memory pressure. If you're evicting constantly, reconsider your maxmemory setting or data model.
Reduce key churn for large values — frequent updates to large string or hash values are a primary driver. Where possible, batch updates or restructure data to avoid constantly reallocating big chunks.

Check whether active defrag is already enabled before you turn it on: redis-cli CONFIG GET activedefrag. If it's already yes and fragmentation is still climbing, you likely have a structural data problem rather than a tuning one.

Making This Easier to Catch Automatically

The manual approach — periodically running INFO memory and checking the numbers — works fine until you have more than one Redis instance or a team that doesn't watch dashboards closely. Most latency incidents caused by fragmentation are preventable, but only if someone notices the ratio creeping up over hours or days before it becomes critical.

This is exactly where a tool like Opservo helps — it continuously monitors Redis memory metrics, surfaces fragmentation trends in plain English, and can alert your team before the ratio crosses critical thresholds rather than after latency has already degraded. Instead of digging through INFO output during an incident, you get an early warning with context.

Whether you go the manual route or automate it, the core habit is the same: treat mem_fragmentation_ratio as a first-class metric alongside hit rate and latency. It's not exotic telemetry — it's one of the most reliable early warning signals Redis gives you, and it's sitting there in INFO memory waiting to be used.

Point readers to Opservo's free 2-server plan as the easiest way to get continuous fragmentation alerting without building a Prometheus stack. — one command installs a read-only agent and your first findings arrive in five minutes: start free. Related: blog how to monitor redis memory and eviction.

Originally published on the Opservo blog.

Our alerts went nowhere for 59 days. Here is what we changed.

Opservo — Sat, 10 Oct 2026 14:36:24 +0000

We run Opservo on our own servers first: 40+ production machines across the 30 companies in the Softuvo portfolio. On 28 September we found out that, for 59 days, not one alert meant for our own team had reached us.

What happened

Our team channel was a Discord server. Opservo sends to custom webhooks and Discord accepts webhooks, so on 31 July we pointed our alerts at one and moved on.

From that day on, the webhook rejected every alert with HTTP 400 Bad Request. Opservo spotted the problems on time and sent the alerts on time. Discord refused every one of them, and nobody was looking at the one place that would have told us.

The worst of it: an eight-day Redis outage sat inside that window. Opservo raised it. The alert hit a 400 and went no further.

Why nobody noticed

Silence looks exactly like health. A channel that gets nothing for a week reads as "a quiet week". Nothing ever tells you that you did not get a message.

Most teams test an alert channel once, on the day they add it. After that, the only proof it works is an alert arriving, and no alerts arriving proves nothing.

What we changed in Opservo

The webhook is fixed. More importantly, every notification channel in Opservo now records the result of its last delivery, so "is this channel actually working?" has an answer on the screen instead of a guess.

It was the second time in a month that a channel let us down. On 19 September a workstation disk remounted read-only. Opservo alerted within a minute, but nobody heard, because that workspace had no notification channel at all. Since then Opservo shows a banner when a workspace has no channel to alert through, and it raises a composite disk-failure alert.

Five checks for your own alert channels, whatever tool you use

Send a test alert to every channel and confirm a person saw it. "Sent" is not enough. Someone has to have seen it.
Look at the delivery result, not the configuration. A webhook URL that saves without error can still reject every payload.
Test with a real alert, not a hello-world message. A receiver can accept one and reject the other.
Give anything that pages at least two paths, for example Slack and email, so one broken channel does not mean total silence.
Repeat check 1 every month. It takes five minutes. Put it in the calendar now.

The lesson

We build a monitoring tool and still missed this for two months. If it happened to us, it is worth five minutes to check yours.

Opservo is the AI ops engineer for teams without an SRE: plain-English server monitoring with one-click fixes. Free for 2 servers. Start at getopservo.com/welcome, or see what it monitors.

Originally published on the Opservo blog.

A simpler, cheaper Datadog alternative for small teams

Opservo — Tue, 15 Sep 2026 04:30:34 +0000

Enterprise observability is built for SRE teams with a budget. If you run a handful of servers without one, here’s what to look for instead.

Datadog, New Relic and the rest are excellent — for the teams they’re built for. If you have a platform team, a tooling budget, and engineers who read flame graphs for a living, they’re a great fit. If you’re a small team or agency running 5–40 servers with nobody whose job is “ops,” they’re overkill in three specific ways.

Where enterprise tools don’t fit small teams

Price: per-host pricing plus per-metric and per-feature add-ons adds up fast for a small fleet.
Complexity: dashboards assume you already know what to look at. Setup and tuning is a project in itself.
Interpretation: they show you everything and leave the “so what do I do?” entirely to you.

What to look for instead

For a small team, the right tool optimizes for time-to-answer, not data volume. We’d look for four things:

Zero-config install — value in minutes, not a setup project.
Plain-English explanations — it should tell you what’s wrong and what to do, not just chart it.
Predictable cost — flat, per-server pricing with no surprise metric bills.
Action, not just alerts — the ability to actually fix the problem, ideally with one click.

Where Opservo fits

That’s the gap Opservo was built for. One command installs a read-only agent; in minutes you get plain-English answers about every server, predictive warnings before things break, and one-click (or pre-approved automatic) fixes. Pricing is simple per-server, with a free tier for two boxes. It’s not trying to replace Datadog for a 200-engineer org — it’s the tool for everyone Datadog was never designed for. For a feature-by-feature breakdown, see Opservo vs Datadog.

Originally published on the Opservo blog — Opservo is the AI ops engineer for teams without an SRE. Free for 2 servers → https://getopservo.com/welcome

Originally published on the Opservo blog.

What to Do When a Linux Server Load Average Is High

Opservo — Fri, 11 Sep 2026 04:49:18 +0000

High load average on your Linux server? Learn how to diagnose the real cause and fix it fast — with practical commands and clear steps.
You glance at your monitoring dashboard or run a quick top and see a load average of 14 on a 4-core server. Something is wrong — but load average alone doesn't tell you what. It's one of those metrics that looks alarming and yet gives you almost no actionable information on its own. Here's how to actually diagnose what's happening and get your server back to normal.

What Load Average Actually Means

Linux load average is a 1-, 5-, and 15-minute rolling average of the number of processes either running or waiting for CPU or I/O. The key word is 'waiting.' A load of 4.0 on a 4-core machine means your cores are fully saturated — but it doesn't tell you whether processes are waiting for the CPU or waiting on a slow disk. Those two problems have completely different fixes.

A quick rule of thumb: divide the load average by your CPU count. Run nproc to get your core count. If load / cores is above 1.0 for a sustained period, something needs attention. If it's above 2.0, something is actively broken.

Step 1 — Figure Out Whether It's CPU or I/O

Run top and immediately press the key 1 to expand per-CPU stats. Look at the %wa (wait) column. If CPU wait is consistently above 10–20%, your bottleneck is I/O, not CPU. If wait is low but %us (user) or %sy (system) is pegged, you have a CPU problem.

For a clearer picture of I/O, run iostat -xz 2 from the sysstat package. Watch the %util column for each disk. A disk sitting at 90–100% util is saturated. The await column shows average request wait time in milliseconds — anything above 20–30ms for a spinning disk or above a few milliseconds for SSD suggests a bottleneck.

Install sysstat if needed: sudo apt install sysstat or sudo yum install sysstat
Run: iostat -xz 2 5 — this samples every 2 seconds, 5 times
Run: vmstat 2 5 — check the b column (blocked processes) and wa (I/O wait)
High b count with high wa confirms I/O saturation

Step 2 — Find the Processes Responsible

Once you know whether it's CPU or I/O, find the culprits. For CPU, top sorted by CPU (default) shows you the worst offenders. For I/O, the picture is harder — top doesn't sort by I/O by default.

Use iotop -o to show only processes with active I/O. This is the fastest way to catch a runaway database vacuum, a backup job gone wrong, or a log rotation task chewing through disk. You'll need root: sudo iotop -o.

Another powerful option is pidstat -u -d 2 — it shows both CPU and I/O stats per process in one view, refreshing every 2 seconds. If you spot a PID causing trouble, get more detail with cat /proc/<PID>/status and ls -l /proc/<PID>/fd to see what files it has open.

CPU culprits: top (default), or ps aux --sort=-%cpu | head -10
I/O culprits: sudo iotop -o or pidstat -d 2
Identify what a process is doing: strace -p <PID> -e trace=read,write -c
Check open files: lsof -p <PID>

Step 3 — Common Causes and Their Fixes

Once you've identified the process, the fix depends on the pattern. Here are the most common scenarios:

Runaway cron job or backup: check crontab -l and /etc/cron.d/. If a backup is hitting disk hard, consider scheduling it during off-peak hours or using ionice -c3 -p <PID> to lower its I/O priority without killing it.
Database doing heavy work: could be a missing index causing a full table scan, autovacuum (PostgreSQL), or a slow query. Check database slow query logs first before restarting anything.
Log file explosion: a bug can cause an app to write millions of log lines per second. Check with du -sh /var/log/* and watch for files growing in real time using watch -n1 ls -lh /var/log/myapp/.
Memory pressure causing swap: run free -h. If swap usage is high and growing, the kernel is paging — which causes I/O wait even when your disk is healthy. The real fix is reducing memory usage or adding RAM, not tuning disk.
Fork bomb or process leak: ps aux | wc -l to count processes. If it's in the thousands, you may have a process spawning out of control. Set limits in /etc/security/limits.conf (nproc) to prevent recurrence.

Step 4 — After You Fix It, Prevent It

A one-time spike is forgivable. A recurring spike that takes down production is a monitoring problem. Set up alerts on load average — but also on the underlying signals: CPU utilization, disk I/O wait, and memory usage. Alert on the combination, not just one number in isolation.

If you're running a small team without dedicated infrastructure staff, tools like Opservo can help by continuously watching these signals together and explaining in plain English what's happening — so you're not deciphering iostat output at 2am when something breaks. It can also flag patterns like a cron job that spikes load every hour before your users notice.

High load average is a symptom, not a diagnosis. The commands above — iostat, iotop, vmstat, pidstat — are your actual diagnostic tools. Get comfortable running them the moment you see a spike, and you'll cut your resolution time from hours to minutes.

Originally published on the Opservo blog — Opservo is the AI ops engineer for teams without an SRE. Free for 2 servers → https://getopservo.com/welcome

Originally published on the Opservo blog.

How to Monitor a Docker Container's CPU and Memory Usage

Opservo — Tue, 08 Sep 2026 05:19:34 +0000

Learn how to track Docker container CPU and memory usage with built-in tools and practical commands — no dedicated SRE required.
Your app slows to a crawl at 2 a.m., and you have no idea which container is eating all the CPU. By morning the problem has vanished, but you're left with zero evidence and a vague sense of dread. If you're running Docker in production without dedicated monitoring, that scenario is only a matter of time. The good news: Docker exposes a surprising amount of resource data out of the box — you just need to know where to look.

The Quickest Starting Point: docker stats

Docker ships with a live stats command that works immediately, no extra software required. Run it and you get a continuously refreshing table of every running container:

docker stats — streams live CPU %, memory usage vs. limit, network I/O, and block I/O for all running containers
docker stats — scope it to one container
docker stats --no-stream — prints a single snapshot and exits, useful in scripts or cron jobs
docker stats --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}' — trim the output to only the columns you care about The CPU percentage shown is relative to a single core. On a 4-core host, 100% means one full core is saturated. Keep that in mind when reading numbers above 100% — your container is consuming more than one core, not breaking mathematics.

Reading the Memory Numbers Correctly

Memory output looks like '210MiB / 1.95GiB'. The first number is the container's current RSS-style usage; the second is its limit (or the host's total RAM if no limit is set). A container with no memory limit will happily consume all available RAM before the kernel OOM-killer starts terminating processes — usually the worst possible time to discover this.

Always set memory limits on production containers. Without them, a single runaway process can starve every other service on the host.
Set a hard limit at run time with --memory and a soft warning threshold with --memory-reservation:

docker run --memory='512m' --memory-reservation='400m' your-image — container gets a hard 512 MB ceiling
docker inspect | grep -i memory — verify limits on an already-running container
cat /sys/fs/cgroup/memory/docker//memory.usage_in_bytes — read raw cgroup data directly if you need it in a script

Capturing Historical Data with docker stats and Simple Scripts

docker stats is great for real-time debugging but terrible for answering 'what was happening at 2 a.m.?' For historical data without standing up a full monitoring stack, a small shell loop written to a log file is often enough to start:

while true; do docker stats --no-stream --format '{{.Name}},{{.CPUPerc}},{{.MemUsage}}' >> /var/log/container-stats.csv; sleep 30; done — logs a CSV snapshot every 30 seconds
Run it inside a tmux or screen session, or as a systemd service, so it survives SSH disconnects
Pipe output to grep or awk to filter for specific containers in high-container-count environments This is not a replacement for proper time-series storage, but it gives you a simple audit trail you can grep through when something goes wrong.

Going Deeper: cAdvisor and Prometheus

If your team is ready for a more complete solution, Google's cAdvisor (Container Advisor) is the standard open-source exporter for Docker metrics. It runs as a container itself, scrapes cgroup data, and exposes a Prometheus-compatible endpoint:

docker run --volume=/var/run:/var/run:ro --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --publish=8080:8080 --detach=true --name=cadvisor gcr.io/cadvisor/cadvisor — starts cAdvisor with read-only host access
Browse http://localhost:8080/containers/ for a built-in web UI with CPU and memory graphs
Scrape http://localhost:8080/metrics from Prometheus for long-term storage and alerting
Key metrics to watch: container_cpu_usage_seconds_total, container_memory_usage_bytes, container_memory_working_set_bytes (working set is usually more meaningful than raw usage)

Pair cAdvisor with Grafana and you have a full dashboard. The setup takes an afternoon but pays for itself the first time you need to correlate a deploy with a memory spike hours later.

What to Actually Alert On

Collecting data is only half the job — you need to know when to wake someone up. Focus on these thresholds as a practical starting point:

CPU sustained above 80% for more than 5 minutes — a brief spike is normal; sustained high CPU usually signals a stuck process or traffic anomaly
Memory usage above 90% of the container limit — at this point OOM-kills are close; investigate before the kernel does it for you
Memory usage growing steadily over hours with no plateau — classic memory leak signature
Container restart count increasing — docker inspect | grep RestartCount will show you; repeated restarts mean something is crashing silently If managing these thresholds across multiple containers sounds tedious, that's exactly the gap tools like Opservo are built to fill — it watches your containers continuously, surfaces anomalies in plain English, and flags the signals that actually matter before they become incidents. For small teams without a dedicated ops person, that kind of automated context can be the difference between catching a problem at 9 a.m. and getting paged at 2 a.m.

Start with docker stats today. Add cgroup limits to every production container this week. When you're ready for historical data and intelligent alerting, build from there — the foundation you lay now makes every future improvement much easier.

Originally published on the Opservo blog — Opservo is the AI ops engineer for teams without an SRE. Free for 2 servers → https://getopservo.com/welcome

Originally published on the Opservo blog.

How to Find What Is Filling Up Disk Space on a Linux Server

Opservo — Fri, 04 Sep 2026 06:48:34 +0000

Disk full alerts at 2am? Learn the exact commands to find what's eating your Linux server's disk space and fix it fast.
You get the alert: disk usage at 94%. Your app starts throwing errors, logs stop writing, and databases refuse to accept new rows. Finding the culprit fast matters — but on a server with millions of files, knowing where to look is half the battle. Here's a systematic approach to track down disk hogs in minutes, not hours.

Start With the Big Picture: df

Before you dig into directories, confirm which filesystem is actually full. Run:

df -h — shows all mounted filesystems with human-readable sizes
df -h / — focus on the root filesystem
df -i — check inode usage (a filesystem can be 'full' even with free space if inodes are exhausted) Pay attention to the 'Use%' column. If you see 100% on /var or /home but not /, that tells you exactly which mount point to investigate. Inode exhaustion — df -i showing 100% — is easy to miss and causes the same symptoms as a full disk, so always check both.

Drill Down With du

Once you know which mount point is full, use du to find the largest directories. Start from the top of that mount point and work down:

du -sh /* 2>/dev/null — sizes of every top-level directory, errors suppressed
du -sh /var/* 2>/dev/null — drill into /var if that's the culprit
du -ah /var | sort -rh | head -20 — list the 20 largest files and folders inside /var The pattern is always the same: run du -sh on the suspicious directory, find the largest subdirectory, repeat one level deeper. You'll usually hit the real culprit within three or four iterations. Common offenders are /var/log (runaway logs), /var/lib/docker (unused images and volumes), and /tmp (applications that don't clean up after themselves).

Find Large Files Directly With find

Sometimes a single enormous file is the problem — a core dump, a forgotten database export, or a log that rotated incorrectly. Use find to surface files above a size threshold:

find / -xdev -size +500M -ls 2>/dev/null — files over 500 MB on the current filesystem only (-xdev stops it crossing into other mount points)
find /var/log -name '*.log' -size +100M — large log files specifically
find / -xdev -name 'core' -o -name '*.dump' 2>/dev/null — core dumps that can appear silently after crashes Always use -xdev with find when searching from / — without it, find will cross into other mount points and give you confusing results about space that belongs elsewhere.

Track Down Deleted Files Still Holding Space

One of the most frustrating disk space mysteries on Linux: a file gets deleted but disk usage doesn't drop. This happens when a process still has the file open — the space isn't reclaimed until that process closes or releases the file. To find these ghost files:

lsof +L1 — lists all open files where the link count has dropped to zero (i.e., deleted but still held open)
lsof +L1 | awk 'NR>1 {print $7, $1, $2}' | sort -rn | head -10 — sort by file size descending
sudo lsof +L1 | grep deleted — simpler alternative if you just want to see what's deleted but open The fix is usually to restart the process holding the file. If it's a log file held open by a service like nginx or a Java app, restarting the service releases it. Alternatively, if restarting isn't an option immediately, you can truncate the file: > /proc//fd/ — but a restart is cleaner.

Clean Up Common Space Wasters

Once you've identified the culprit, here are targeted cleanup commands for the most frequent offenders:

Docker: docker system prune -a --volumes — removes stopped containers, unused images, and orphaned volumes. Be sure you actually want this in production.
Old journals: journalctl --vacuum-size=200M or journalctl --vacuum-time=7d — caps systemd journal size
Package manager leftovers: apt autoremove && apt clean on Debian/Ubuntu; dnf autoremove on RHEL-based systems
Rotated logs not cleaning up: check /etc/logrotate.conf and run logrotate -f /etc/logrotate.conf to force a rotation cycle
Core dumps: check /proc/sys/kernel/core_pattern to see where they go, then remove them — consider setting kernel.core_pattern=/dev/null in /etc/sysctl.conf if you don't need them For Docker specifically, it's worth making prune a scheduled task rather than a reactive one. A weekly cron job running docker system prune -f prevents volumes from silently accumulating over months.

Make Disk Pressure Visible Before It Becomes a Crisis

Tracking down disk usage reactively works, but the real fix is knowing about gradual disk growth before it hits 100%. Set up alerts at 75% and 85% so you have time to investigate calmly rather than under pressure. Trend data matters too — a disk filling at 1 GB per day needs a different response than one that jumped 20 GB overnight.

This is where Opservo helps: it monitors disk usage across your servers, surfaces which directories are growing fastest, and flags anomalies like a log file that suddenly triples in size — before you get the 2 am alert. If you're managing production servers without a dedicated SRE, having that context surfaced automatically means you spend time fixing problems, not hunting for them.

The commands above will get you out of trouble today. The longer-term win is building visibility so slow-growing disk issues never become emergencies in the first place.

Originally published on the Opservo blog — Opservo is the AI ops engineer for teams without an SRE. Free for 2 servers → https://getopservo.com/welcome

Originally published on the Opservo blog.