Blazing fast CI suite for Elixir with Circle CI 2.1 for your libraries

There is a great post (stole title) about this topic already which helped me a lot to get started. But it has some shortcomings to make great CI for your library.

There is a great difference between client code (specific use case) and library (generic use case). One is used in one specific version (for example Elixir 1.8.1) while later one can be used in many versions (for example Elixir 1.8.1 for some new project and Elixir 1.3.4 for some brownfield project that never got updated).

How to make sure your library works on all supported versions? The best way would be to test in all supported environments. There are matrixes in Travis CI to tackle this particular use case, but they have made some quesionalbe decisions lately. Creating something like matrixes in Circle CI didn't seem as obvious or explicit as in Travis CI, but I made it work.

Asumptations

• You know how to link Circle CI to your repo. I just authorized with my GitHub account and had to select my repo from the dropdown.
• You know where to put Circle CI YAML configuration. (Psst... it's in .circleci/config.yml from repo root)

---

Testing library in the latest elixir version

version: 2

# Jobs are like specific executable setups
jobs:
  # Define new job with a name
  test-1.8.1:
    # How many instances of this particular job can be run in parallel
    parallelism: 1
    # Define which docker image use as the base
    docker:
      - image: elixir:1.8.1

    # What steps the current job should execute
    steps:
      # Go to the project directory
      - checkout

      # Elixir setup
      - run: mix local.hex --force
      - run: mix local.rebar --force

      # Library setup
      - run: MIX_ENV=test mix do deps.get --only test, deps.compile, compile

      # Run tests
      - run: mix test

# Workflows are like bird view of your CI.
# Define which jobs should be executed in what orders
workflows:
  version: 2
  # Define new workflow with a name
  testing_all_versions:
    # Define which jobs should be executed
    jobs:
      # Execute job named test-1.8.1
      - test-1.8.1

Please look into oficial documentation for extended descriptions and even more options.

Making it blazing fast

There is this neat feature of caching which can increase the execution of jobs greatly. Let's see it in action:

version: 2

jobs:
  test-1.8.1:
    parallelism: 1
    docker:
      - image: elixir:1.8.1

    steps:
      - checkout

      # Restoring state from previous executions
      - restore_cache:
          key: testing-{{ .Environment.CIRCLE_JOB }}

      - run: mix local.hex --force
      - run: mix local.rebar --force

      - run: MIX_ENV=test mix do deps.get --only test, deps.compile, compile

      # Save state after steps execution
      - save_cache:
          key: testing-{{ .Environment.CIRCLE_JOB }}
          paths:
            - _build
            - deps
            - ~/.mix

      - run: mix test

workflows:
  version: 2
  testing_all_versions:
    jobs:
      - test-1.8.1

There is no magic involved. restore_cache and save_cache are regular steps:

• save_cache - saves all provided folder contents in the cache
• restore_cache - puts cached folders in newly created docker image (each time a new job is executing)

All steps in between are executing as usual, but for example how much time does mix deps.get, mix deps.compile and mix compile takes on first execution and each consecutive execution? If there aren't cached folders each job execution will fetch dependencies, compile them and then compile the project from scratch. On the other hand, if those folders are cached, each consecutive time already will have downloaded and compiled dependencies. Only new dependencies will be downloaded and only changed code will be recompiled.

Testing library compatibility with multiple elixir versions. First try.

YAML has concept as anchors which is also used in post mentioned above.

My first try also was using this approach:

version: 2

jobs:
  # Added anchor with name test (&test)
  test-1.8.1: &test
    parallelism: 1
    docker:
      - image: elixir:1.8.1

    steps:
      - checkout

      - restore_cache:
          key: testing-{{ .Environment.CIRCLE_JOB }}

      - run: mix local.hex --force
      - run: mix local.rebar --force
      - run: MIX_ENV=test mix do deps.get --only test, deps.compile, compile

      - save_cache:
          key: testing-{{ .Environment.CIRCLE_JOB }}
          paths:
            - _build
            - deps
            - ~/.mix

      - run: mix test
  # another job with name
  test-1.7.4:
    # Import all steps from anchor named test
    <<: *test
    # Each property added overwrites imported properties from anchor
    docker:
      - image: elixir:1.7.4
  test-1.6.6:
    <<: *test
    docker:
      - image: elixir:1.6.6
  test-1.5.3:
    <<: *test
    docker:
      - image: elixir:1.5.3
  test-1.4.5:
    <<: *test
    docker:
      - image: elixir:1.4.5
  test-1.3.4:
    <<: *test
    docker:
      - image: elixir:1.3.4

workflows:
  version: 2
  testing_all_versions:
    jobs:
      - test-1.8.1
      - test-1.7.4
      - test-1.6.6
      - test-1.5.3
      - test-1.4.5
      - test-1.3.4

Basically, we reuse previously defined the job and rewrite the docker image on which to execute steps. Now the library is tested in all provided elixir versions.

While it works great for this specific use case, I would also like to test code quality against the latest elixir version with mix format --check-formatted (code is formatted correctly), MIX_ENV=test mix coveralls.circle (tests coverage), mix credo (code quality) and mix dialyzer (type checking). For each of those cases, I would need to change the last step (mix test) defined in an anchored job.

Anchors (as much as I know) don't support partial overwrites or appends, only full property overwrites. So for each case, where only the last step is different, I would have to rewrite all steps. Which still would be fast and done. But not for me. Duplication in such manner just gives future me more places to be cautious around, more error-prone and just takes more mental space.

Some time ago I decided to start blogging with some regularity. Finding things to write about isn't as easy as seemed. So mainly I try to write about things I just learned and hope that my findings will help not only others but future me as well.

Testing library compatibility with multiple elixir versions. Second try.

I decided to investigate it further and find a solution that I would be satisfied with. As of Circle CI 2.1, there is such thing as parameters. Let's use those instead of anchors:

version: 2.1

# Anchors are good for something tho
default_version: &default_version 1.8.1
default_steps: &default_steps
  - run: mix test

jobs:
  # Job with name build
  build:
    # Define allowed parameters
    parameters:
      # Parameter - version
      version:
        # Parameter deccription
        description: Elixir version
        # Parameter is expected to be a string
        type: string
        # Use default (latest) elixir version if none is provided
        default: *default_version
      # Parameter - execute
      execute:
        description: What steps to execute after build
        # Parameter is expected to be a list of executable steps
        type: steps
        # Use default steps if none are provided
        default: *default_steps

    parallelism: 1
    docker:
      # Image is provided dynamically, by concatenating 
      # - elixir: 
      # - the value provided via version parameter
      # 
      # Might look something like elixir:1.8.1, elixir:1.7.4
      - image: elixir:<< parameters.version >>

    steps:
      - checkout

      - restore_cache:
          # Cache key is provided dynamically by concatenating
          # - testing-elixir-v
          # - the value provided via version parameter
          #
          # Might look something like testing-elixir-v1.8.1, testing-elixir-v1.7.4
          key: testing-elixir-v<< parameters.version >>

      - run: mix local.hex --force
      - run: mix local.rebar --force
      - run: MIX_ENV=test mix do deps.get --only test, deps.compile, compile

      - save_cache:
          # Cache key is provided dynamically by concatenating
          # - testing-elixir-v
          # - the value provided via version parameter
          #
          # Might look something like testing-elixir-v1.8.1, testing-elixir-v1.7.4
          key: testing-elixir-v<< parameters.version >>
          paths:
            - _build
            - deps
            - ~/.mix

      # Execute steps provided via execute parameter.
      - steps: << parameters.execute >>

workflows:
  version: 2.1
  testing_all_versions:
    jobs:
      # Job to execute (the only job actually provided)
      - build:
          # Give it a custom name
          # Otherwise all jobs would have build-1, build-2, ... names
          name: "Test in elixir 1.8.1"
          # Ignore execute parameter. Will use default steps (mix test)
          # Provide version (defined in job parameters)
          # Used in
          # - defining docker image (which elixir version to use)
          # - as a cache key, so each version would have a separate cache
          #   all job calls with the same version will share cache files
          version: 1.8.1
      - build:
          # Provide a different name
          name: "Test in elixir 1.7.4"
          # Provide different elixir version
          version: 1.7.4
      - build:
          name: "Test in elixir 1.6.6"
          version: 1.6.6
      - build:
          name: "Test in elixir 1.5.3"
          version: 1.5.3
      - build:
          name: "Test in elixir 1.4.5"
          version: 1.4.5
      - build:
          name: "Test in elixir 1.3.4"
          version: 1.3.4

As you can see, now we have one boilerplate job, that can be called with different parameters (something like a function). So we only need to provide the elixir version and actual meaningful steps if needed. In this case, we can only provide a version parameter as a default step to execute are mix test one we would have provided.

Testing library compatibility with multiple elixir versions and code quality

Now we are ready to implement additional tests, that are related to code quality: mix format --check-formatted (code is formatted correctly), MIX_ENV=test mix coveralls.circle (tests coverage), mix credo (code quality) and mix dialyzer (type checking).

version: 2.1

default_version: &default_version 1.8.1
default_steps: &default_steps
  - run: mix test

jobs:
  build:
    parameters:
      version:
        description: Elixir version
        type: string
        default: *default_version
      execute:
        description: What steps to execute after build
        type: steps
        default: *default_steps

    parallelism: 1
    docker:
      - image: elixir:<< parameters.version >>

    steps:
      - checkout

      - restore_cache:
          key: testing-elixir-v<< parameters.version >>

      - run: mix local.hex --force
      - run: mix local.rebar --force
      - run: MIX_ENV=test mix do deps.get --only test, deps.compile, compile

      - save_cache:
          key: testing-elixir-v<< parameters.version >>
          paths:
            - _build
            - deps
            - ~/.mix

      - steps: << parameters.execute >>

workflows:
  version: 2.1
  testing_all_versions:
    jobs:
      - build:
          name: "Test in elixir 1.8.1"
          version: 1.8.1
      - build:
          name: "Test in elixir 1.7.4"
          version: 1.7.4
      - build:
          name: "Test in elixir 1.6.6"
          version: 1.6.6
      - build:
          name: "Test in elixir 1.5.3"
          version: 1.5.3
      - build:
          name: "Test in elixir 1.4.5"
          version: 1.4.5
      - build:
          name: "Test in elixir 1.3.4"
          version: 1.3.4
  # Add aditional workflow
  validate_code_quality:
    jobs:
      # Call build job
      - build:
          # Provide custom name
          name: "Code formatted correclty"
          # Do not provide version parameter (use default one)
          # Provide execute parameter with custom steps
          execute:
            # This step will be executed instead of default one
            - run: mix format --check-formatted
      - build:
          name: "Code style check"
          execute:
            - run: mix credo
      - build:
          name: "Type check"
          execute:
            - run: mix dialyzer
      - build:
          name: "Tests coverge"
          execute:
            - run:
                command: MIX_ENV=test mix coveralls.circle
                environment:
                  COVERALLS_REPO_TOKEN: 5wI8JIzygEDIF4A03KNIKWdOmVK2A8dMC

That's it. Now Circle CI will both test library in all specified elixir versions and checks for code quality (with formater, credo, dialyzer, and coveralls).

Circle CI CLI tool

While this example can be reused in many cases, often you might want to implement custom configuration. Committing and publishing each time to see Circle CI fail because of misconfiguration is redundant and time-consuming. But no worries, Circle CI have created a cli tool.

While it has few commands, one most useful seems to be circleci config validate which checks if the configuration file is written correctly. When you commit and push you will already know it will work as expected. No need for 20 commits, that all say fixed circle ci config (as I did in beginning).

Conclusion

• Circle CI 2.1 brings many improvements. One of which is parameters. There are many different ways how to reuse config parts.
• Circle CI is fast, it rarely takes more than a minute from the time I push code, to time all jobs have been executed. Probably has something to do with a small library having few tests to cover everything. I also don't use credo and dialyzer in my current setup. Who knows maybe they like to execute much longer.
• Circle CI support is also fast. They responded within 24h (in the weekend).
• Circle CI CLI is a great companion tool to help you configure your CI environment with a faster feedback loop.
• It was interesting to try something outside my usual realm. While I hade a few bums here and there, it was a great experience. Setup was straight forward and set up the next projects will be even easier.

P.S. If you have any questions or ideas on how to improve Circle CI config or blog post itself, please let me know :)