DEV Community

Odd_Background_328
Odd_Background_328

Posted on

Run a Disk-Full Drill on Your Free Server Before Your Agent Logs Do

#devops #sre #linux #tutorial

02:00. The page fires. Disk usage 95%. Your model proxy returns 500s. No deploy. No traffic spike. The logs just grew.

Free-tier servers have small disks. A chatty agent log can fill one in hours. Today you will fill the disk on purpose, watch the failure, and build a recovery path.

MonkeyCode offers a free server and free model access. This drill uses that server and a minimal Python proxy. No external tools.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What you are testing

  • How the OS behaves when the disk fills
  • How your proxy fails when it cannot write logs
  • How logrotate prevents the next outage

Stage 1: Provision the server and a log-generating proxy

SSH into your free server.

ssh root@your-free-server
df -h
Enter fullscreen mode Exit fullscreen mode

Note the available space. Create a working directory.

mkdir -p ~/drill && cd ~/drill
Enter fullscreen mode Exit fullscreen mode

Create a minimal proxy that logs every request:

#!/usr/bin/env python3
import json
import time
from http.server import BaseHTTPRequestHandler, HTTPServer

LOG = "/var/log/proxy.log"

class H(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get("Content-Length", 0))
        self.rfile.read(length)
        with open(LOG, "a") as f:
            f.write(f"{time.time()} {self.client_address[0]}\n")
        body = b'{"ok":true}'
        self.send_response(200)
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)
    def log_message(self, *a):
        pass

HTTPServer(("0.0.0.0", 8080), H).serve_forever()
Enter fullscreen mode Exit fullscreen mode

Run it as a background process:

python3 proxy.py &
Enter fullscreen mode Exit fullscreen mode

Verify it works:

curl -s -X POST http://127.0.0.1:8080/ -d '{}'
# {"ok":true}
tail -1 /var/log/proxy.log
# 1720000000.123 127.0.0.1
Enter fullscreen mode Exit fullscreen mode

Stage 2: Fill the disk on purpose

Check current usage:

df -h /
Enter fullscreen mode Exit fullscreen mode

Example output on a 10G server:

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        10G  2.1G  7.9G  21% /
Enter fullscreen mode Exit fullscreen mode

Use fallocate to consume 90% of the free space. Adjust the size based on your server.

fallocate -l 1G /tmp/fill
df -h /
Enter fullscreen mode Exit fullscreen mode

Repeat until you reach 90%:

for i in 1 2 3 4 5 6 7; do fallocate -l 1G /tmp/fill.$i; done
df -h /
Enter fullscreen mode Exit fullscreen mode

Now send requests to the proxy:

for i in $(seq 1 20); do
  curl -s -o /dev/null -w "%{http_code}\n" -X POST http://127.0.0.1:8080/ -d '{}'
done | sort | uniq -c
Enter fullscreen mode Exit fullscreen mode

Expected output:

     12 200
      8 500
Enter fullscreen mode Exit fullscreen mode

Some requests succeed, then they fail. The proxy cannot append to the log.

Stage 3: Diagnose the failure

Check the disk:

df -h /
Enter fullscreen mode Exit fullscreen mode

Check the log size:

ls -lh /var/log/proxy.log
Enter fullscreen mode Exit fullscreen mode

Check kernel messages:

dmesg | tail -20
Enter fullscreen mode Exit fullscreen mode

You will see no space left on device errors. The proxy process may still run, but every write fails.

Stage 4: Recover and implement logrotate

Remove the filler files:

rm -f /tmp/fill.*
df -h /
Enter fullscreen mode Exit fullscreen mode

Now configure logrotate for the proxy log. Create /etc/logrotate.d/proxy:

/var/log/proxy.log {
    daily
    rotate 3
    compress
    missingok
    notifempty
    copytruncate
}
Enter fullscreen mode Exit fullscreen mode

Test logrotate manually:

logrotate -vf /etc/logrotate.d/proxy
ls -lh /var/log/proxy.log*
Enter fullscreen mode Exit fullscreen mode

Output shows the rotated and compressed files:

-rw-r--r-- 1 root root 0 Aug 25 02:10 /var/log/proxy.log
-rw-r--r-- 1 root root 123 Aug 25 02:09 /var/log/proxy.log.1.gz
Enter fullscreen mode Exit fullscreen mode

The log is rotated and compressed. The proxy keeps writing to a fresh file.

Stage 5: Re-run the drill and verify

Fill the disk again, but this time watch logrotate protect you.

for i in 1 2 3 4 5 6 7; do fallocate -l 1G /tmp/fill.$i; done
df -h /
Enter fullscreen mode Exit fullscreen mode

Send requests again:

for i in $(seq 1 20); do
  curl -s -o /dev/null -w "%{http_code}\n" -X POST http://127.0.0.1:8080/ -d '{}'
done | sort | uniq -c
Enter fullscreen mode Exit fullscreen mode

Expected output:

     20 200
Enter fullscreen mode Exit fullscreen mode

All requests succeed. The log file stays bounded because logrotate truncates it. The disk fills, but the log does not.

Limitations

  • Free server disk size varies. Check your actual quota.
  • Do not fill the disk to 100%. You can lock yourself out of SSH.
  • Logrotate is not a monitoring solution. You still need alerts on disk usage.
  • copytruncate can lose a few log lines between copy and truncate. Acceptable for this use case.

Cleanup and rollback

Remove the filler files:

rm -f /tmp/fill.*
Enter fullscreen mode Exit fullscreen mode

Stop the proxy:

pkill -f proxy.py
Enter fullscreen mode Exit fullscreen mode

Remove the test log and logrotate config:

rm -f /var/log/proxy.log*
rm /etc/logrotate.d/proxy
Enter fullscreen mode Exit fullscreen mode

Your server is back to the original state.

Who should skip this

  • If your proxy writes logs to stdout and you use a log collector, this drill is less relevant.
  • If your log volume is a few KB per day, disk exhaustion is unlikely.
  • If you run a managed service, the provider handles disk pressure.

Run this drill on your free server. The df curve after a log flood is the signal you want to see before production, not after.

Top comments (0)