Docker Logs, continue....

Serepas Filippas — Tue, 21 Feb 2023 17:55:37 +0000

In my previous post I have explained how to set limits in docker logs. But after that we can see that, for some reason, docker after reach the max-size create a second file with extension log.1.

I don't know, yet, why this happening but, after the lattest file reach again the max-size, docker start write, from begin the first file..

Set limits to the number of logs files!!!

So you need to add one more argument to your docker-compose.yml, for example,

logging:
       driver: "json-file"
       options:
           max-size: "100k"
           max-file: "3"

Now docker will create three logs files with these extensions:
.log, .log.1, .log.2
When first one hit the max-size will create the next one and etc.
When docker reach the limit in last file .log.2 will recreate the first one .log .

Docker Logs, set limits!

Serepas Filippas — Mon, 06 Feb 2023 18:34:28 +0000

The problem...

I have a docker-compose.yaml which contains a simple python script. The job of this script is to subscribe to MQTT broker and then write these messages to a database.

The mistake...

Inside this script I have a simple innocent print statement, which prints the captured message.
We talk about 30 messages per second, and this service already runs for 3 plus months. So..., a lot of messages.

I need more space!!!

First of all, this script runs in Ubuntu server. At some point the hard disk doesn't have enough space. My first though is that there is much data inside my database.
But the problem here is that I get messages every second (~3600x30 messages!) which eventually I replace with their average value per hour. Again I think the problem is the database.

VACUUM IT!!!

I use Postgresql, so, I read the documentation.
Here we are! Let's release some space. At this point I don't want to use the VACUUM FULL because it causes downtimes...
After that, nothing changed dramatically in my server's hard drive..

Docker df

I think my first move should be this one. Check the space which is taken by my docker container,

docker system df

Hmmm, here I am missing ~6.5Gb of space which at that point I assumed was being taken up by docker... somehow...

Logs, Logs everywhere..

Next move is to see the logs of the container

docker logs <container id>

I see all these print statements which start from the beginning of container's existence.
So, I thought where these logs are stored?

 docker inspect --format='{{.LogPath}}' <my-app or container-id>

The result will be something like this,

/var/lib/docker/containers/46d66d306b4a7f95051dbf2aa7b27864d78d84bfac7dc685c4b94985a1ee30c5/46d66d306b4a7f95051dbf2aa7b27864d78d84bfac7dc685c4b94985a1ee30c5-json.log

Here you are!!!
Ok, then I cd to this directory and run,

ls -l -h

To see 6.5Gb of logs!!!
Here was my precious Gb!!!

How clear this logs?

The first fast and easy way, without need to re-build your container is,

sudo sh -c 'echo "" > $(docker inspect --format="{{.LogPath}}" my-app)'

Be aware, this is a temporary solution! You will come up with the same problem in the near future!

Set limits in docker-compose

You can set limits of logs for your container in your docker-compose.yaml

logging:
      driver: "json-file"
      options:
        max-size: 50m

Here we set limit of 50mb of log file.

Set limits in docker run

docker run \
      --log-driver local --log-opt max-size=10m \
      alpine echo hello world

Another example, set 10mb limit of logs with docker command
To be continued...

DEV Community: Serepas Filippas