Cache Busting in Docker

Just putting the finishing touches on a new Ansible Tower Workshop, testing an additional exercise, building my Docker container locally before I push the changes to the repo to build my production image in Docker Hub.

So as I'm sitting here in the Nashville Airport waiting for this container to build and a bit of time to pass for us to board, I figured it'd be a good time to do a bit of writing. Here's a trick I use that can save you hours in building containers...

The Premise

I start with a very basic Alpine image and add my needed packages - a few from the apk package manager, but then I compile nginx from source. This is because it's much easier to set proper operating conditions for the nginx server in a container. Then, after the nginx server is compiled and ready to roll, I build my application and dump it into /var/www/html

The Problem

Now, compiling nginx from source takes time. I have separate RUN stanzas in the Dockerfile to benefit from the image layering that container images provide, but Docker isn't always that great at detecting changes. If I make a few changes to the web application, Docker doesn't always copy over the new files.

To quickly fix this, one could prune the built images in the local cache - but this means starting from the top, which means recompiling nginx, which takes a good deal of time.

The Solution

How do we tell Docker explicitly where to start from? Just add an ENV between your static and dynamic actions, then change a small variable in between builds.

FROM node:8-alpine

# Install base packages
RUN apk -U add wget tar gzip git asciidoctor

# Install nginx
RUN \
  build_pkgs="build-base linux-headers openssl-dev pcre-dev wget zlib-dev" && \
  runtime_pkgs="ca-certificates openssl pcre zlib" && \
  apk --update add ${build_pkgs} ${runtime_pkgs} && \
  cd /tmp && \
  wget http://nginx.org/download/nginx-1.15.8.tar.gz && \
  tar xzf nginx-1.15.8.tar.gz && \
  cd /tmp/nginx-1.15.8 && \
  ./configure \
    --prefix=/etc/nginx \
    --sbin-path=/usr/sbin/nginx \
    --conf-path=/etc/nginx/nginx.conf \
    --error-log-path=/var/log/nginx/error.log \
    --http-log-path=/var/log/nginx/access.log \
    --pid-path=/tmp/run/nginx.pid \
    --lock-path=/tmp/run/nginx.lock \
    --http-client-body-temp-path=/tmp/cache/nginx/client_temp \
    --http-proxy-temp-path=/tmp/cache/nginx/proxy_temp \
    --http-fastcgi-temp-path=/tmp/cache/nginx/fastcgi_temp \
    --http-uwsgi-temp-path=/tmp/cache/nginx/uwsgi_temp \
    --http-scgi-temp-path=/tmp/cache/nginx/scgi_temp \
    --user=nginx \
    --group=nginx \
    --with-http_ssl_module \
    --with-http_realip_module \
    --with-http_addition_module \
    --with-http_sub_module \
    --with-http_dav_module \
    --with-http_flv_module \
    --with-http_mp4_module \
    --with-http_gunzip_module \
    --with-http_gzip_static_module \
    --with-http_random_index_module \
    --with-http_secure_link_module \
    --with-http_stub_status_module \
    --with-http_auth_request_module \
    --with-mail \
    --with-mail_ssl_module \
    --with-file-aio \
    --with-ipv6 \
    --with-threads \
    --with-stream \
    --with-stream_ssl_module \
    --with-http_slice_module \
    --with-http_v2_module && \
  make && \
  make install && \
  ln -sf /dev/stdout /var/log/nginx/access.log && ln -sf /dev/stderr /var/log/nginx/error.log && \
  adduser -D nginx && \
  rm -rf /tmp/* && \
  apk del ${build_pkgs} && \
  rm -rf /var/cache/apk/* && \
  mkdir -p /tmp/cache/nginx/scgi_temp && \
  mkdir -p /tmp/cache/nginx/uwsgi_temp && \
  mkdir -p /tmp/cache/nginx/fastcgi_temp && \
  mkdir -p /tmp/cache/nginx/proxy_temp && \
  mkdir -p /tmp/cache/nginx/client_temp && \
  mkdir -p /tmp/run

ENV cacheBUSTER v9001

COPY conf/nginx.conf /etc/nginx/nginx.conf

COPY webapp/* /var/www/html/

Now, simply change that cacheBuster ENV definition to some other string and enjoy a cached image layer with nginx while being able to pull in fresh application data every time!