Introduction
The other day one of my slack channels started buzzing. It said our DGraph database was down.This seems to happen sometimes, usually restarting works. But here it wasn't!
2 seconds after a restart, it goes down again.
No Space
One thing I noticed while I was logged into the droplet where Dgraph was hosted, was the OS wasn't able to autocomplete my commands on pressing tab key. So there was no space.
I was stumped, what was happening ?!?
So I ran
df -h which showed
Filesystem Size Used Avail Use% Mounted on
.
.
/dev/vda1 49G 45G 45G 100% /
.
.
Whoa! What is /dev/vda1 and why is it's usage at 100% ?
The hunt
So I used du -h --max-depth=1 from root /, and I noticed /var had a huge usage.
Moved into /var and ran the same, now log had a huge usage.
Turns out it was syslog that was using up the space.
Solution
Went ahead and truncated syslog now, /dev/vda1 is at 9% usage.
We also cleaned up /tmp.
And set the logs to rotate.
Recommendation
We didn't see a need to backup the syslog but it's recommended that anybody clearing it up do backup for later analysis on why it fills up, if that happens regularly.
In our case the server has been running for quite some time, some it was just a natural accumulation of logs.
Another potentially useful command
du -sh [/path]
References
- How to deleted
syslogsafely. [https://stackoverflow.com/questions/35638219/ubuntu-large-syslog-and-kern-log-files]
Top comments (0)