Jenkins: your Jenkins data directory is almost full and automated disk cleanup

We have a Jenkins instance with a bunch of jobs.

After a while – a new warning appeared:

Jenkins: your Jenkins data directory /var/lib/jenkins (aka JENKINS_HOME) is almost full

Checked disk usage with the ncdu and found few jobs which used really lot of space:

root@jenkins-production:/data/jenkins# du -h --max-depth 1 jobs/
..
25G     jobs/APITests
...

And mainly space is used by a job’s logs:

root@jenkins-production:/data/jenkins# ls -lh /data/jenkins/jobs/APITests/jobs/Projectname/jobs/Stage/jobs/ProjectnameStageAPItests/builds/551/log
-rw-r--r-- 1 root root 77M Mar  1 12:49 /data/jenkins/jobs/APITests/jobs/Projectname/jobs/Stage/jobs/ProjectnameStageAPItests/builds/551/log

There is a lot of old builds and each keeps its log.

The simplest way to solve this issue is to enable the Discard Old Builds option in a job’s config:

Apparently, it was added in Jenkins v2.х, as previously I remember had to manually describe a workspace cleanup in a Jenkinsfiles.

Check disk used now:

root@jenkins-production:/data/jenkins# du -h --max-depth 1 jobs/APITests/jobs/Projectname/
23G     jobs/APITests/jobs/Projectname/jobs
23G     jobs/APITests/jobs/Projectname/

Save new settings, run a new build and check disk after it:

root@jenkins-production:/data/jenkins# du -h --max-depth 1 jobs/APITests/jobs/Projectname/
7.0G    jobs/APITests/jobs/Projectname/jobs
7.0G    jobs/APITests/jobs/Projectname/

Also, there is the Discard Old Build plugin which allows to configure the cleanup process more accurately.

Similar posts

• 02/09/2017 Azure: подключение дополнительного диска к VM и миграция Jenkins (0)
• 02/08/2017 Jenkins: sandbox.RejectedAccessException: Scripts not permitted to use (0)
• 03/13/2019 Jenkins: import a job to another server (0)
• 03/19/2019 Jenkins: Scripted Pipeline – Production environment job confirmation step (0)