In my current position, we define continuous integration builds for all our projects in a Dockerfile. For us, Docker provides a set of guarantees that are perfect for CI.
- Docker builds are reproducible.
- Docker is available on every major OS.
- Docker produces universal artifacts.
Builds are Reproducible Everywhere
A standard Jenkins build runs directly on the system. It's nothing against Jenkins, but if you are just building on a raw node, the underlying instance will have discrepancies. This could be security patch, library versions, or even OS versions. For us, a runner might be Ubuntu or OSX. This means out of date local packages cause build failures. Don't get me wrong, that's a worst case scenario. Worst case always happens in DevOps.
The ideal state of our build is "reproducible state". A successful build that is rerun should succeed with the same output. A build that changes between runs of the same git commit means someone somewhere messed up. Mostlikely, you aren't versioning build dependencies...I'm not saying we had this exact problem, but we had this exact problem.
The beauty of Docker builds, they are immutable unless you try really hard like not versioning. The steps in a Dockerfile run in the same order every time. Wonderful.
The biggest boon is the system that runs the build doesn't need to care what language or packages a build needs. The Docker build handles that via Dockerfile instead at the encompassing OS. There are no tricks to create virtual environments like in Python. No race conditions on which packages are installed by different builds. Everything is self contained and way faster than spinning up a fresh dedicated VM.
The glaring question one may ask, "Why not use the already built in Docker runner environments that CI solutions like Jenkins support?" Yes, that's a solution, but it doesn't solve another problem.
Run Builds Anywhere
Running a build anywhere is a dream chased by many a developer. If we guarantee that if the build works on a dev system will work in production, that's gold. Docker is a solution to the problem. A Docker build that works on Windows will work on a Mac. Sure, it's trickery if you look too hard at the man behind the curtain (the VM running Linux). Devs usually don't care about that. When a build and its tests succeed on their machine but fails on the build box, they are rightfully upset. So why not let them run the exact same process on their machine that the Jenkins box uses?
This portability provides robust builds and increased developer productivity. Jenkins pipeline files are opaque at best so we don't expect devs to learn the syntax. DevOps doesn't want to write a new Jenkins file for every project either. Rather, we encourage each team write their project's Dockerfile. Once the hump of Dockerphobia is surmounted, the teams are more productive since they don't wait on DevOps for their build changes. They don't even have to trigger their Jenkins build to see if it will fail. Less waste overall.
As a side anecdote, I explained this build system to friend of mine. He implemented it on a new service his team was working on while doing builds on AWS CodeBuild. For various reasons, CodeBuild wasn't the best solution, so he moved everything to CircleCI. The overall process took him 45 minutes, because all the build scripts were in Dockerfiles already. I'm not saying you couldn't move that quickly with normal build systems, but it's worth noting.
Docker as Artifacts
Dockers are a great universal artifact for build systems. Related to Run Builds Anywhere you can deploy Dockers practically everywhere. That's kind of the point of Dockers. What's more interesting is we can use Dockers as a storage for libraries.
We've experimented with this a bit over the last year. Every library is built using Docker as describe above. The resulting image is never used alone as docker itself is the storage for that artifact. Then when other Docker builds need a library we will import that upstream dependency using multi-stage Docker builds.
FROM cool-java-lib:2.0.0 as lib
FROM openjdk as builder
RUN mkdir /app
WORKDIR /app
COPY --from lib mavencache/* mavencache/*
COPY ./src/* /app/src
RUN mvn build
This isn't a perfect solution as it requires we still push the library to a Nexus repo for local development. Overall, this is not a recommendation. It's something to consider if you have problems with external library repos and only want to limit dependencies to a Docker repo at build time.
The Downsides
There are downsides to using Docker builds for CI. The big one is the use of Docker itself. Containers are great things, but they can be heavier than a cached library. This may not be true in some sense on the repository level since we can reuse layers. You can't do that in Maven repos. This caching requires a good knowledge of how Docker build layers work.
The intimate understanding of how to use Docker to build software is an entry barrier to developers. Dockers often seem like this magical tech that DevOps community peddles. Because of this, there is a lot more upfront work to get devs up to speed on Dockers but the tail is usually a lot better in our experience.
Reality
We've been using Docker builds like this for the better part of a year and it's made life a lot easier. Everyone has a much better idea of how projects build and exactly what went wrong. Is this for everyone? No, nothing is. This works for our use case as we rely heavily on container orchestration systems and have that expertise. In the cases we are not using containers to run a system, we do have to shoehorn this build style in. This is a minority of our services and the benefits are great for now. Like all things in DevOps, we will have a different philosophy in 6 months.
Top comments (0)