Our team have many projects delivered by Jenkins. Every component need to be assembled by webpack which write in React.js or Vue.js.
Each build will spend 5 or 10 minutes to finish, and most time spend on npm install
. But we all know that if package.json
were not change, node_modules
directory should not change as well.
So that we checksum package.json
with MD5, and package whole node_modules
directory as cache. If the hash string does not changed, which implied nothing changed, just unpack cached node_modules
directory and then execute npm install
, all things done.
Here is a simple example
PKG_SUM=$(md5sum package.json | cut -d\ -f 1)
NPM_TARBALL=node_modules-${PKG_SUM}.tgz
NPM_TARBALL_MD5SUM=${NPM_TARBALL}.md5sum
NPM_TARBALL_CACHE=${HOME}/.cache/npmtarball
S3_NPM_TARBALL=s3://deployment/npmtarball
[[ ! -e $NPM_TARBALL_CACHE ]] && mkdir -p $NPM_TARBALL_CACHE
function downloadNpmTarball(){
pushd $NPM_TARBALL_CACHE
if [ ! -f ${NPM_TARBALL} ];then
s3cmd get ${S3_NPM_TARBALL}/${NPM_TARBALL} ${NPM_TARBALL}
s3cmd get ${S3_NPM_TARBALL}/${NPM_TARBALL_MD5SUM} ${NPM_TARBALL_MD5SUM}
md5sum -c ${NPM_TARBALL_MD5SUM} || rm -f ${NPM_TARBALL} ${NPM_TARBALL_MD5SUM}
fi
popd
}
function checkNpmMod() {
downloadNpmTarball
TARBALL=${NPM_TARBALL_CACHE}/${NPM_TARBALL}
[[ -f $TARBALL ]] && tar xzf $TARBALL
}
function uploadNpmMod() {
TARBALL=${NPM_TARBALL_CACHE}/${NPM_TARBALL}
if [ ! -f ${TARBALL} ];then
if [ -d node_modules ];then
tar zcf ${TARBALL} node_modules || return 1
fi
pushd $NPM_TARBALL_CACHE
md5sum $NPM_TARBALL > ${NPM_TARBALL_MD5SUM}
s3cmd put $NPM_TARBALL $S3_NPM_TARBALL/$NPM_TARBALL
s3cmd put $NPM_TARBALL_MD5SUM $S3_NPM_TARBALL/$NPM_TARBALL_MD5SUM
popd
fi
}
checkNpmMod
npm install
uploadNpmMod
Oldest comments (6)
How much fast is to cache on S3? Download, unzip, zip and upload time... My team have a similar problem due to npm install time
If you are using aws ec2, it will be very fast.
I wrote a script to get MD5 only from dependencies and devDependencies, cause we do a patch bump on every merge to develop branch (we work with master and develop in a fork flow). Here is the gist.
Great!
I created such script to check md5sum of package.json in Jenkins:
For the sake of whatever, please create a Jenkins plugin which will facilitate life of hundreds of others. it's untold overkill everytime doing
npm install
such unnecessarily