Breaking up the CircleCI yaml into multiple files

As an org if you use CircleCI, over time you will notice that config file really increase in length. One of the repos I worked on hit ~3000 lines. It starts becoming unwieldy, reviewing changes is hard and making changes is harder still. Sadly CircleCI doesn't support reading from multiple files. But do not fret, this blog post will show you how you can do this.

Turns out Circle built most of what is required, into their CLI. Read about it here. Start out by breaking your config into multiple files as described in the circle article. You can see how I broke up my yml in this repo.

Through the rest of this post we refer to the smaller yml file as sub-ymls and the main config.yml as the config.

The .circleci/src directory contains the files that combine to to form .circleci/config.yml. We have a make command make circle-config which combines the sub-ymls to the main config

The interesting part which isn't covered by CircleCI is how you force a build to break if someone modifies one of the sub-ymls and forgets to compile the main config. Notice in the lint and test file, we have a command that fails the build if CircleCI config was not generated.

- swissknife/build_and_check_circle_config:
    custom-error-message: Please run "make circle-config" and push up.
    fail-if-dirty: true

An example pull request shows how the build is failed if only the sub ymls and not the main config.yml is modified. This prevents divergence of code.

Swissknife is an orb, that has multiple useful commands and the build_and_check_circle_config commands fails the build if git is dirty after the circle config was generated.

In this example Swissknife checks if git is dirty after generating circle config. This means if anyone modified config directly or they modify the sub-ymls and not generate the config, the build will fail. The command can also be set up to push the difference in config to the origin.