Speed up Jenkins with npm build

Our team have many projects delivered by Jenkins. Every component need to be assembled by webpack which write in React.js or Vue.js.

Each build will spend 5 or 10 minutes to finish, and most time spend on npm install. But we all know that if package.json were not change, node_modules directory should not change as well.

So that we checksum package.json with MD5, and package whole node_modules directory as cache. If the hash string does not changed, which implied nothing changed, just unpack cached node_modules directory and then execute npm install, all things done.

Here is a simple example

PKG_SUM=$(md5sum package.json | cut -d\  -f 1)
NPM_TARBALL=node_modules-${PKG_SUM}.tgz
NPM_TARBALL_MD5SUM=${NPM_TARBALL}.md5sum
NPM_TARBALL_CACHE=${HOME}/.cache/npmtarball
S3_NPM_TARBALL=s3://deployment/npmtarball

[[ ! -e $NPM_TARBALL_CACHE ]] && mkdir -p $NPM_TARBALL_CACHE

function downloadNpmTarball(){
    pushd $NPM_TARBALL_CACHE
        if [ ! -f ${NPM_TARBALL} ];then
            s3cmd get ${S3_NPM_TARBALL}/${NPM_TARBALL} ${NPM_TARBALL}
            s3cmd get ${S3_NPM_TARBALL}/${NPM_TARBALL_MD5SUM} ${NPM_TARBALL_MD5SUM}
            md5sum -c ${NPM_TARBALL_MD5SUM} || rm -f ${NPM_TARBALL} ${NPM_TARBALL_MD5SUM}
        fi
    popd
}

function checkNpmMod() {
    downloadNpmTarball
    TARBALL=${NPM_TARBALL_CACHE}/${NPM_TARBALL}
    [[ -f $TARBALL ]] && tar xzf $TARBALL
}

function uploadNpmMod() {
    TARBALL=${NPM_TARBALL_CACHE}/${NPM_TARBALL}
    if [ ! -f  ${TARBALL} ];then
        if [ -d node_modules ];then
             tar zcf ${TARBALL} node_modules || return 1
        fi
        pushd $NPM_TARBALL_CACHE
             md5sum $NPM_TARBALL > ${NPM_TARBALL_MD5SUM}
             s3cmd put $NPM_TARBALL $S3_NPM_TARBALL/$NPM_TARBALL
             s3cmd put $NPM_TARBALL_MD5SUM $S3_NPM_TARBALL/$NPM_TARBALL_MD5SUM
        popd
    fi
}
checkNpmMod
npm install
uploadNpmMod

Comments

[Vitalii Dasaev]

I created such script to check md5sum of package.json in Jenkins:

stage('NPM Build') {
  steps {
    sh '''
    node -v && npm -v
    '''
    // rm -rf node_modules
    sh '''
    CACHE_FOLDER=${HOME}/.cache/md5
    echo "EXECUTOR_NUMBER: ${EXECUTOR_NUMBER}"
    MD5_FILE_NAME=package-json_${EXECUTOR_NUMBER}.md5sum

    [ -d ${CACHE_FOLDER} ] || mkdir -p ${CACHE_FOLDER}
    ls ${CACHE_FOLDER}

    if [ -f ${CACHE_FOLDER}/${MD5_FILE_NAME} ];then
      cp ${CACHE_FOLDER}/${MD5_FILE_NAME} ${MD5_FILE_NAME}
      md5sum package.json
      cat ${MD5_FILE_NAME}
      md5sum -c ${MD5_FILE_NAME} || npm ci
    else
      echo "No md5sum backup"
      npm ci
    fi

    echo "create new md5sum backup"
    md5sum package.json
    md5sum package.json > ${MD5_FILE_NAME}
    cp ${MD5_FILE_NAME} ${CACHE_FOLDER}
    '''
    sh '''
    npm run ngcc
    '''
    sh '''
    npm run build
    '''
  }
}

[T. Todua]

For the sake of whatever, please create a Jenkins plugin which will facilitate life of hundreds of others. it's untold overkill everytime doing npm install such unnecessarily

[Gois]

How much fast is to cache on S3? Download, unzip, zip and upload time... My team have a similar problem due to npm install time

[Guixing Bai]

If you are using aws ec2, it will be very fast.

[Gois]

I wrote a script to get MD5 only from dependencies and devDependencies, cause we do a patch bump on every merge to develop branch (we work with master and develop in a fork flow). Here is the gist.

[Guixing Bai]

Great!