CircleCI - The Matrix

Technical examples - The matrix

This series of posts will focus on some tips and tricks that you can, and should implement in your config files. First, we will cover the matrix.

This post assumes you have the basics down about circle CI configs, i.e: you know what: job, step, parameter, etc. mean.
If you’re a complete beginner I would suggest you look up something beginner-friendly, that explains these topics (the official Circle documentation for example). I’m not saying this will be impossible to follow, but it is better if you get the full picture.

For each topic, I will post links to official documentation where you can read all the technicalities.

Matrix basics

The matrix is there to help you avoid repeating yourself, and run similar jobs that only differ from one another by a few lines.
We will cover multiple configurations, building upon our first example:

workflows:
  test-and-build:
    jobs:
      - test:
          matrix:
            parameters:
              service: ["svc1", "svc2", "svc3"]
jobs:
  test:
    parameters:
      service:
        type: string
    steps:
      - step1:
          service: << parameters.service >>
      - step2
      - step3

This will create 3 jobs that will run in parallel, and each will be passed a different value for the service parameter. I also included an example of such a job, where the first command accepts the service parameter. Now, you might ask yourself why would I need to do that?

Here’s why (this is just one example out of many): let’s say that you are working with a monorepo that contains multiple services (or projects, depends on how you call them - I will call them services) and for each of them, before deploying, you would like to install dependencies, lint, test, and maybe “Dockerize”.

Now, if you’re not doing this with a matrix, you can either copy and paste 3 jobs that only differ by the name of the service you’re currently doing all of the above for, or you’ve created a very long job that does all of that for each of them sequentially (please don’t do that).

In contrast, if you’re using a matrix, you would only need to write one job that accepts as a parameter which service to work on. Moreover, when you expand and would like to test another service, all you would need to do is expand that list (service field in the matrix).

Here’s what it would look like in a case you don’t use matrices:

workflows:
  test-and-build:
    jobs:
      - test1
      - test2
      - test3
jobs:
  test1:
    steps:
      - step1:
          service: "svc1"
      - step2
      - step3
  test2:
    steps:
      - step1:
          service: "svc1"
      - step2
      - step3
  test3:
    steps:
      - step1:
          service: "svc2"
      - step2
      - step3

This doesn’t look terrible, yet. But it’s only because this is a basic example, and because we’re using our own commands (will cover in another post). The moment you need to test a new service, you will have to copy an entire such block. Same with when you would want to add a new step to your testing job, you will have to add it to every single one… It gets time consuming, trust me.

Multi dimensional matrices

Let’s expand our previous example and look at something like this:

workflows:
  test-and-build:
    jobs:
      - test:
          matrix:
            parameters:             
              service: ["svc1", "svc2", "svc3"]
              os: ["linux", "macos"]

Now, all we’ve done is add another parameter to the matrix, but the implications are amazing. What will happen now, is that we will get 6 jobs (Cartesian Product of parameters), and can now test each service on two different OS’s. No need to write the same job over and over. Pretty nifty eh?
Please note that your matricized job cannot expand into more than 128 jobs!

Dependencies between matricized jobs

Continuing with our first example, we will look at two different ways to “wait” for these jobs to complete.

Example 1:

workflows:
  test-and-build:
    jobs:
      - test:
          name: test-<< matrix.service >>
          matrix:
            parameters:             
              service: ["svc1", "svc2", "svc3"]
      - build:
          name: build-<< matrix.service >>
          requires: test-<< matrix.service >>
          matrix:
            parameters:             
              service: ["svc1", "svc2", "svc3"]

Example 2:

workflows:
  test-and-build:
    jobs:
      - test:
          matrix:
            parameters:             
              service: ["svc1", "svc2", "svc3"]
      - build:
          requires: test
          matrix:
            parameters:             
              service: ["svc1", "svc2", "svc3"]

Let’s examine the differences between the two:

In the first example, we’re assigning a name to each of the testing jobs i.e: “test-svc1”, “test-svc2”... Following that, each of our 3 build jobs will await for its corresponding test job to end. Meaning that “build-svc1” will wait for “test-svc1”, and so on.
Please note that in order to wait for each job individually, you must name them.

In the second example, we’re waiting for all 3 of the test jobs to be complete before starting our build jobs.

Both ways are valid, and it’s up to you to decide which one you need (In some cases, a composition of the two methods could be needed).

Anchors and aliases

One last thing I want to cover is anchors and aliases. Let’s see how we can modify one of the previous examples to avoid repeating ourselves too much:

workflows:
  test-and-build:
    jobs:
      - test:
          name: test-<< matrix.service >>
          matrix:
            parameters: &params            
              service: ["svc1", "svc2", "svc3"]
      - build:
          name: build-<< matrix.service >>
          requires: test-<< matrix.service >>
          matrix:
            parameters:             
              <<: *params

As you can guess, this will function similarly to our previous example. The only difference is that we don’t have to add another service to all the jobs that use the service parameter. We’ll only need to add new services to the first map, after which all the other references will have the modified map as well.

What's next

This is the second post of the series which deals with the matrix. Read the other posts to see more technical examples.
Cheers!