The problem...
I have a docker-compose.yaml which contains a simple python script. The job of this script is to subscribe to MQTT broker and then write these messages to a database.
The mistake...
Inside this script I have a simple innocent print statement, which prints the captured message.
We talk about 30 messages per second, and this service already runs for 3 plus months. So..., a lot of messages.
I need more space!!!
First of all, this script runs in Ubuntu server. At some point the hard disk doesn't have enough space. My first though is that there is much data inside my database.
But the problem here is that I get messages every second (~3600x30 messages!) which eventually I replace with their average value per hour. Again I think the problem is the database.
VACUUM IT!!!
I use Postgresql, so, I read the documentation.
Here we are! Let's release some space. At this point I don't want to use the VACUUM FULL because it causes downtimes...
After that, nothing changed dramatically in my server's hard drive..
Docker df
I think my first move should be this one. Check the space which is taken by my docker container,
docker system df
Hmmm, here I am missing ~6.5Gb of space which at that point I assumed was being taken up by docker... somehow...
Logs, Logs everywhere..
Next move is to see the logs of the container
docker logs <container id>
I see all these print statements which start from the beginning of container's existence.
So, I thought where these logs are stored?
docker inspect --format='{{.LogPath}}' <my-app or container-id>
The result will be something like this,
/var/lib/docker/containers/46d66d306b4a7f95051dbf2aa7b27864d78d84bfac7dc685c4b94985a1ee30c5/46d66d306b4a7f95051dbf2aa7b27864d78d84bfac7dc685c4b94985a1ee30c5-json.log
Here you are!!!
Ok, then I cd to this directory and run,
ls -l -h
To see 6.5Gb of logs!!!
Here was my precious Gb!!!
How clear this logs?
The first fast and easy way, without need to re-build your container is,
sudo sh -c 'echo "" > $(docker inspect --format="{{.LogPath}}" my-app)'
Be aware, this is a temporary solution! You will come up with the same problem in the near future!
Set limits in docker-compose
You can set limits of logs for your container in your docker-compose.yaml
logging:
driver: "json-file"
options:
max-size: 50m
Here we set limit of 50mb of log file.
Set limits in docker run
docker run \
--log-driver local --log-opt max-size=10m \
alpine echo hello world
Another example, set 10mb limit of logs with docker command
To be continued...
Top comments (0)