DEV Community

Cover image for Reclaim your inodes by deleting dangling docker volumes
Adam K Dean
Adam K Dean

Posted on

1 1

Reclaim your inodes by deleting dangling docker volumes

Originally posted on September 8th, 2015 (more info)

Note: these days, you can simply use docker volume prune, but this is being posted for posterity. This was an issue which caused lots of problems back in 2015, and determining ways of managing inode usage helped a number of platforms including Shopify, who took containerisation to a new level.

Earlier today I had a strange issue where Docker builds started failing, citing there was no free space. When I looked, there was gigabytes of free space. I looked into this further and found that Jenkins liked to open a lot of files. 30,000 for a fresh install, in fact. I thought this was my problem but it wasn't. Something else was causing the system to run out of space.

I dug around a bit more and found that the problem was inodes, which are data structures used to represent filesystem objects such as files or directories. To be more precise, the problem was a distinct lack of available inodes. Of the 2.9 million available, 2.9 million were in use. That's 100%.

I had a funny feeling that dangling volumes had something to do with this, so I had a look at how many I had:

$ docker images -qf 'dangling=true' | wc -l
100
Enter fullscreen mode Exit fullscreen mode

These were quite large images too, comprising of multiple files and directories. So many in fact that it was using up all the inodes available on the system. To repeat, that was 2.9M inodes.

I was able to clear it up by running docker images -qf 'dangling=true' | xargs docker rmi, which takes a while but cleared it up. In case this happens in the future, I've put some logging in place to keep an eye on the inode usage, and created a little script for people to use in case I'm not around and this happens again.

Here is can-i-have-my-inodes-back-please.sh:

#!/bin/bash
#
# Are your inodes all tied up?
# Would you like to make them liquid again?
# Run this script for up to instant relief
# (smallprint: maytakeuptotwentyminutesdependingonnumberofdanglingvolumes)

DANGLING_NUM=$(docker images -qf 'dangling=true' | wc -l)

read -p "Are you sure you want to remove $DANGLING_NUM dangling volumes? (Y/N) " prompt

if [[ $prompt == "y" || $prompt == "Y" || $prompt == "yes" || $prompt == "Yes" ]]; then
  docker images -qf 'dangling=true' | xargs docker rmi
else
  exit 0
fi
Enter fullscreen mode Exit fullscreen mode

And in case you're interested, the crontab runs this script hourly, log-inode.sh:

#!/bin/bash
INODE=$(df -i | sed -n 2p | awk '{ print $5 }')
DATE=$(date +'%m/%d/%Y %H:%M')
echo "$DATE,$INODE"
Enter fullscreen mode Exit fullscreen mode

In a few weeks I'll grab that csv data and bung it into Excel and see just how bad this inode problem is. Maybe a nightly dangling volume cleanup will do the trick.

Image of Docusign

🛠️ Bring your solution into Docusign. Reach over 1.6M customers.

Docusign is now extensible. Overcome challenges with disconnected products and inaccessible data by bringing your solutions into Docusign and publishing to 1.6M customers in the App Center.

Learn more

Top comments (0)

Sentry image

See why 4M developers consider Sentry, “not bad.”

Fixing code doesn’t have to be the worst part of your day. Learn how Sentry can help.

Learn more

đź‘‹ Kindness is contagious

Discover a treasure trove of wisdom within this insightful piece, highly respected in the nurturing DEV Community enviroment. Developers, whether novice or expert, are encouraged to participate and add to our shared knowledge basin.

A simple "thank you" can illuminate someone's day. Express your appreciation in the comments section!

On DEV, sharing ideas smoothens our journey and strengthens our community ties. Learn something useful? Offering a quick thanks to the author is deeply appreciated.

Okay