Hunting space gobblers in Ubuntu OS.

#devops #ubuntu

Introduction

The other day one of my slack channels started buzzing. It said our DGraph database was down.This seems to happen sometimes, usually restarting works. But here it wasn't!
2 seconds after a restart, it goes down again.

No Space

One thing I noticed while I was logged into the droplet where Dgraph was hosted, was the OS wasn't able to autocomplete my commands on pressing tab key. So there was no space.
I was stumped, what was happening ?!?

So I ran

df -h which showed

Filesystem      Size  Used Avail Use% Mounted on
.
.
/dev/vda1        49G  45G   45G   100% /
.
.

Whoa! What is /dev/vda1 and why is it's usage at 100% ?

The hunt

So I used du -h --max-depth=1 from root /, and I noticed /var had a huge usage.
Moved into /var and ran the same, now log had a huge usage.
Turns out it was syslog that was using up the space.

Solution

Went ahead and truncated syslog now, /dev/vda1 is at 9% usage.
We also cleaned up /tmp.
And set the logs to rotate.

Recommendation

We didn't see a need to backup the syslog but it's recommended that anybody clearing it up do backup for later analysis on why it fills up, if that happens regularly.
In our case the server has been running for quite some time, some it was just a natural accumulation of logs.