Incident Report: Service failure due to storage full

Jancer Lima — Sat, 18 Apr 2026 01:09:42 +0000

Yesterday, my homelab server suddenly became unresponsive. It started with a flurry of Discord notifications, the universal signal that something has gone seriously wrong.

I found all services offline. The logs pointed to a primary culprit: a Redis failure, specifically a Server Out of Memory error.

The core error was: RedisClient::CommandError: MISCONF Errors writing to the AOF file: No space left on device
My first thought was: Why is AOF even enabled? I turned it on for testing and forgot. My root partition was at 99% capacity, just 270MB remaining out of 24GB.

Further investigation revealed where the "wasted" space was hiding:

PM2 Logs (~3.8GB): The process manager was storing massive, unrotated text logs.
Hidden Caches (~1.5GB): Accumulated ~/.cache, ~/.npm, and ~/.rvmsource files from multiple builds and deployments.

To get the system breathing again, I performed a quick "surgical" cleaning:

PM2 Flush: Immediately cleared the massive log files using pm2 flush.
Log Truncation: Emptied application logs using truncate -s 0 log/*.log (this clears the content without deleting the file handle).
Cache Pruning: Deleted hidden build caches in ~/.npm and ~/.cache.
Journal Vacuum: Cleared system logs with journalctl --vacuum-size=500M.

Now I had enough space to spin up all process again, but I need to recover Redis since it entered a Read-Only mode to protect data integrity.

Fixing the AOF Manifest: Because the disk filled during a Redis write, the appendonly.aof.manifest was corrupted. I fixed it using sudo redis-check-aof --fix on the manifest file inside /var/lib/redis/appendonlydir/.
Clearing the MISCONF Lock: Even with free space, Redis remained in a "protected" state. I manually overrode this with redis-cli config set stop-writes-on-bgsave-error no.
Service Restart: Reset the systemd failure counter with systemctl reset-failed redis-server and restarted the service.

After that I could successfully restart all services and have everything running. All data inside redis were not critical, so I didn't care about losing it.

Lessons learned

The failure was a classic case of neglecting "boring" infrastructure: log rotation and disk monitoring. To prevent a repeat performance, I've implemented the following:

Log Management: Installed pm2-logrotate to cap PM2 logs at 10MB per file and limited journald to 500MB globally.
Next Steps:
- Expand the VM disk size (24GB is too tight for this stack).
- Set up a cron job for weekly apt autoremove and cache clearing.
- Implement an automated disk usage alert (likely via Grafana or a simple shell script to Discord).

Is TLS Enough? A Retrospective on Application-Layer Encryption

Jancer Lima — Tue, 24 Feb 2026 11:27:35 +0000

Years ago, I was part of a heated debate that every engineering team eventually faces: Is standard TLS enough, or do we need custom application-layer encryption?

We were implementing a payment solution. The provider required a backend-to-backend integration, meaning we had to take user credit card data, send it to our server, and then forward it to the provider.

My argument was that the TLS layer would be enough for it. The rest of the team disagreed. They didn't have a technical counter-argument, it was just a "lack of trust".

We ended up building a complex Dual-Keypair System:

The app keypair: Used to sign requests so the server could verify the data actually came from our app (Authenticity).
The server keypair: The app used the server's public key to encrypt the payload, ensuring only our backend could read it (Confidentiality).

It worked, but years later I realized the hidden parts we didn't consider.

The scale: We were small then. But if you scale to multiple server instances, you suddenly need a secure way to share those private keys. You've just turned a "payment problem" into a "key management problem".
The key rotation: What happens if a key is compromised or expires? If you have users on old app versions, you're stuck. You either support legacy keys forever or force your users to update, either options are bad.
The "hostile" client: We stored a key in the app. But unless you are using the device's hardware Secure Enclave (iOS) or Keystore (Android), your app is a hostile environment. A determined attacker can decompile the code and extract those keys.
The TLS termination: The only valid concern was where the TLS "ends". If your HTTPS connection terminates at a Load Balancer and the internal traffic to your web server is plain HTTP, you have a gap.

Unless you are building a banking core or a literal payment gateway, TLS is enough. If you’re worried about the "last mile" inside your VPC, solve that at the infrastructure level. Don’t bake custom crypto into your application logic unless you’re ready to manage the massive operational overhead that comes with it.

Experience taught me that "Trust issues" should be solved with better infrastructure, not more code.

DEV Community: Jancer Lima

Incident Report: Service failure due to storage full

Lessons learned

Is TLS Enough? A Retrospective on Application-Layer Encryption