DEV Community

ChungWei Wei
ChungWei Wei

Posted on Originally published at kmp.tw on

[Docker] Disk Full Due To Docker Logs

#docker #disk #linux #sre

Situation

When Login Docker Server, Use df -h Check Disk Usage Show :

root@docker-service:/var/lib# df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 8078356 0 8078356 0% /dev
tmpfs 1625492 153072 1472420 10% /run
/dev/mapper/ubuntu--vg-ubuntu--lv 121962128 121945744 0 100% /
tmpfs 8127448 0 8127448 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 8127448 0 8127448 0% /sys/fs/cgroup
Enter fullscreen mode Exit fullscreen mode

How To Fix

The First Step, Fix Error & Make Sure All Services Work. 

truncate -s 0 <container_log_path/container_log>.log
Enter fullscreen mode Exit fullscreen mode

or You Can Keep Last Line 10,000

tail -n 10000 <container_log_path/container_log>.log > /tmp/latest.log
cat /tmp/latest.log <container_log_path/container_log>.log 
rm /tmp/latest.log
Enter fullscreen mode Exit fullscreen mode

The Second Step, Change Log Rotate Setting Or Centralized Logging

Local Setting

In docker daemon setting Add These Conf, And Restart Docker Process

vim /etc/docker/daemon.json

{
  "log-driver": "json-file", #Default Log Driver
  "log-opts": {
    "max-size": "100m", #Max Size For Per Log File
    "max-file": "3" #Log File Count
  }
}
Enter fullscreen mode Exit fullscreen mode

systemctl restart docker

Centralized Setting

In docker daemon setting Add These Conf, And Restart Docker Process

vim /etc/docker/daemon.json

{
  "log-driver": "fluentd", #Use Fluentd Agent
  "log-opts": {
    "fluentd-address": "your-fluentd-server:24224" #Log Server URL
  }
}
Enter fullscreen mode Exit fullscreen mode

systemctl restart docker

Top comments (0)