Introduction
The other day one of my slack channels started buzzing. It said our DGraph database was down.This seems to happen sometimes, usually restarting works. But here it wasn't!
2 seconds after a restart, it goes down again.
No Space
One thing I noticed while I was logged into the droplet where Dgraph was hosted, was the OS wasn't able to autocomplete my commands on pressing tab
key. So there was no space.
I was stumped, what was happening ?!?
So I ran
df -h
which showed
Filesystem Size Used Avail Use% Mounted on
.
.
/dev/vda1 49G 45G 45G 100% /
.
.
Whoa! What is /dev/vda1
and why is it's usage at 100% ?
The hunt
So I used du -h --max-depth=1
from root /
, and I noticed /var
had a huge usage.
Moved into /var
and ran the same, now log
had a huge usage.
Turns out it was syslog
that was using up the space.
Solution
Went ahead and truncated syslog
now, /dev/vda1
is at 9%
usage.
We also cleaned up /tmp
.
And set the logs to rotate.
Recommendation
We didn't see a need to backup the syslog
but it's recommended that anybody clearing it up do backup for later analysis on why it fills up, if that happens regularly.
In our case the server has been running for quite some time, some it was just a natural accumulation of logs.
Another potentially useful command
du -sh [/path]
References
- How to deleted
syslog
safely. [https://stackoverflow.com/questions/35638219/ubuntu-large-syslog-and-kern-log-files]
Top comments (0)