DEV Community: CircleCI

Setting up continuous integration with CircleCI and GitLab

Zan Markan — Wed, 07 Jun 2023 23:12:45 +0000

CircleCI supports GitLab as a version control system (VCS). In this tutorial you will learn how to set up your first CircleCI CI/CD pipeline for a project hosted on GitLab. As GitLab can be used either as a SaaS tool, as well as self-managed on-premise installation, I will cover the steps to connect it with CircleCI for both.

Prerequisites and application basics

To follow along with this tutorial, you will need:

Basic knowledge of Git commands
Git installed and accessible on your machine
A GitLab account — either self-managed or SaaS
A CircleCI account

Starter app

Our starter application is a minimal Python Flask app with a single ‘hello world’ web page, which specifies required dependencies and includes a test. You can access a publicly hosted version of the starter app on GitLab.com.

You can get the application by downloading it straight from the GitLab web interface or cloning it via Git.

For the purposes of this tutorial, you don’t need to write the application from scratch, but if you are interested in complete development instruction, you can read through the beginning of [this blog post](https://circleci.com/blog/setting-up-continuous-integration-with-github/.

GitLab SaaS vs self-managed

As mentioned above, this tutorial will teach you how to configure and run your CircleCI pipeline for a project hosted either on a self-managed or hosted version of GitLab. You can learn more about the differences between the hosted and self-managed versions in the GitLab docs.

Setting up GitLab SaaS requires you only to sign up at gitlab.com. You will also need to make sure you can push to GitLab repositories, either by setting up your SSH key or personal access token for using GitLab with either SSH or HTTPS, respectively.

If you use GitLab self-managed, you will connect to it via your installation URL. This is specific to you, so for the purposes of this tutorial, use the URL yourgitlabinstance.com as a placeholder.

Create a new project with the starter app source code. You can use the new project wizard, either at gitlab.com/projects/new or yourgitlabinstance.com/projects/new, for SaaS or self-managed respectively.

Creating a CircleCI config file

Before you set up CircleCI, you can tell it what to eventually start building.

Create a new directory .circleci in the top level of the project.

Now, create a new file config.yml in that directory (.circleci/config.yml) — this is your main configuration file for CircleCI.

Paste the following in the file:

version: 2.1
jobs:
  test:
    docker:
      - image: cimg/python:3.10.11
    steps:
      - checkout
      - restore_cache:
          key: deps1-{{ .Branch }}-{{ checksum "requirements.txt" }}
      - run:
          name: Install dependencies
          command: |
            python3 -m venv venv
            . venv/bin/activate
            pip install -r requirements.txt
      - save_cache:
          key: deps1-{{ .Branch }}-{{ checksum "requirements.txt" }}
          paths:
            - "venv"
      - run:
          name: Running tests
          command: |
            . venv/bin/activate
            python3 tests.py
      - store_artifacts:
          path: test-reports/
          destination: python_app

workflows:
  run-tests:
    jobs:
      - test

In the CircleCI config file, you define everything you need CircleCI to do. In this case, you have added a single job (test) and a workflow (run-tests) containing that job.

The test job runs in a Docker container with Python installed, checks out the code at that specific commit, downloads the dependencies required for the app (Flask, in this case), sets up a Python venv, and runs the tests in tests.py. It also caches the dependencies to make subsequent pipeline runs faster and stores the test artifacts for easy access from CircleCI. To read more about CircleCI configuration, consult the configuration reference.

Save, make a new commit, and push to your GitLab repository. Now it’s time to configure your CircleCI integration with GitLab.

Configuring CircleCI with GitLab SaaS (GitLab.com)

If you don’t have a CircleCI account yet, create one now. Head to https://circleci.com/signup/ and follow the instructions to sign up using your preferred method. If you choose GitLab, you will also be prompted to select your GitLab instance. Press Connect next to GitLab.com.

This should now prompt you to authorize CircleCI with your GitLab account, giving CircleCI access to your repositories.

Authorizing CircleCI will take you to the project creation wizard. You should see your repository on the list, and it should detect the CircleCI config you committed.

Click Create Project to start building. This has created the project in CircleCI, and what’s left is to commit and push a change to trigger your first CircleCI pipeline.

Make any change, such as an update to your README file, then commit and push. This will create your first pipeline and start building and running your test.

Seconds later it should be marked as successful. The status of your CircleCI pipeline will be automatically updated in your GitLab UI as well. Navigate to Build/Pipelines (on the left-hand side in GitLab) and the pipeline should show up there as well. You can see an example based on this tutorial in this public project.

Clicking it will show you further details of the pipeline. Clicking the button under external - button CircleCI: Workflow run tests will take you back full circle into your workflow inside CircleCI.

Connecting CircleCI to a self-managed GitLab instance

Self-managed installations, unlike the hosted SaaS versions, run on infrastructure you control. They will of course also have a different URL to access them. In this tutorial, use yourgitlabinstance.com as a placeholder for your actual instance URL. This section closely follows the GitLab integration instructions in the CircleCI docs.

Assuming you have followed the steps to get the project into GitLab and committed your first CircleCI config file, you can proceed with connecting to GitLab.

If you don’t have a CircleCI account yet, create one now. Head to https://circleci.com/signup/ and follow the instructions to sign up with your preferred method.

If you choose to sign up with GitLab, you will be prompted to select your GitLab instance. Press Connect next to GitLab self-managed.

GitLab self-managed project setup

The wizard guides you through the setup process step by step.

First enter the URL of your GitLab instance and click Verify. It will suffix it with /api/v4 to complete your instance’s API endpoint.

Create your personal GitLab access token with an API scope, paste it in the Personal Access Token field and click Verify.

Next, pass in the known_hosts value and click Verify. You can get this by running ssh-keyscan yourgitlabinstance.com in the terminal and pasting the entire output in the known_hosts field. This allows CircleCI to verify the authenticity of your GitLab instance.

Finally, select your project in the dropdown. This will likely be cci-python-app. You can also give it a different name. This will set your GitLab self-managed project up for building on CircleCI.

Trigger a pipeline by committing a change and pushing it to your GitLab repo. This will start your workflow and show it in the next few seconds.

You can also see your pipeline in the GitLab UI by navigating to CI/CD > Pipelines on the left-hand side. Note that this might look different, depending on what version of GitLab you are using.

Clicking on its status it will take you to the pipeline details, from where you can navigate back to the CircleCI UI to give you a 360 degree view of your project and your CI/CD pipeline’s workflows and jobs.

Congratulations! You have successfully configured CircleCI to start building your project on a self-managed GitLab instance.

Conclusion

In this tutorial you have learned how to begin building projects hosted on GitLab with CircleCI. We have covered both on-premise self-managed GitLab as well as the hosted SaaS version on GitLab.com.

Wishing you successful building!

CI/CD for interdepartmental teams

Zan Markan — Thu, 02 Feb 2023 18:00:00 +0000

Introduction

The topic of CI/CD is often considered the domain of DevOps, infrastructure, and development teams. I would argue this thinking misses the great opportunity that is including the entire organisation in the conversation around software delivery lifecycle (SDLC). From QA, to product managers, to of course operators, security, all the way through to the top of engineering and technology leadership. Especially as organisations increasingly rely on software, not only in tech industry but in most other industries as well, it makes sense to spread the DevOps thinking and the use of tools such as CI/CD to more folks, more teams, and more departments.

Incidentally, that’s what the DevOps movement is all about, and its natural evolution. Bringing the principles of early and continuous collaboration across departments to their entire organisations.

This article argues for the use of CI/CD across various company departments - to anyone involved in - or adjacent to - the software delivery process, and covers the principles and tools that will benefit them, and points you to how to implement this using CircleCI in your organisation.

Who this article is for

People who will benefit the most from this article are the ones implementing pipelines themselves - be it DevOps and Software engineers. While this article is not a tutorial, yet it contains ideas and snippets for implementing the topics discussed within.

As we’re talking about how CI/CD can benefit most of the organisations, this article will be approachable to people with many different backgrounds. From developers to operations teams, all the way to folks dealing with delivery and governance - delivery managers, security teams, engineering leadership.

The software delivery lifecycle from the CI/CD perspective

Software delivery lifecycle, from the perspective of the CI/CD processes works kind of like this - the developers on the team write and push code changes, have them reviewed, and then automatically tested or otherwise validated, built, and ultimately deployed to a target environment using the CI/CD system, at which point the running software begins to be monitored by the operations teams.

What technologies are used, where it is deployed, how it is monitored, and by whom, will all depend on the organisation; From Go microservices running on Kubernetes, to static websites with lambda functions, to iOS applications on the App Store, and everything in between.

But of course there are many more things we do. Software doesn’t begin in an editor, it more often begins as a story somewhere on a backlog or a ticket in a bug tracker. That’s when we actually start thinking about what needs to be built, and how it should work, and for whom (and whether it ultimately does work or not), so why not use CI/CD to do more than just build, validate, and deploy?

Expanding the SDLC for the entire organisation

A modern CI/CD platform can work beyond just building testing and deploying code, and instead it can tie together many more people and teams.

In this section we will look at some of these capabilities, and how connecting to external tools can make us focus more on the software we're building and ultimately build better software.

QA, Product, and before work gets made

QA engineers work best with constant access to latest builds to run tests on, as well as run and automate their their own tests. They are often also existing CI/CD users, who write test code and commit to the codebase.
We can create build outputs either directly in CircleCI with store_artifacts and store_test_results as in the example below, as well as of course through third party tools such as AppStore Connect, and deploying to test environments.

  jobs:
    integration-test:
      docker:
        - image: ...
      steps:
        - checkout
        ...
        - run-test ...
        - store_artifacts:
            path: tested_app_binary
        - store_test_results:
            path: test-results

Beyond the CI/CD platform itself, QA engineers also work a lot with product management and ticketing software - be it validating the requirements, logging defects, or confirming that updated products do not indeed have the defects anymore.
A CI/CD tool can integrate with a PM tool such as Jira, so that all your builds and deploys are automatically available and referenceable in it. This of course doesn’t only apply to QA engineers but also for product managers.

You do so by first connecting CircleCI with Jira, and then making sure that you label each commit with the right ticket name - like AB-234. Finally, as the last step of your deployment or build job - you can invoke the Jira orb's notify command, with which you can mark deployments and builds, as in this example:

orbs:
    jira: circleci/jira@1.3.1
workflows:
  build:
    jobs:
      - build:
          post-steps:
            - jira/notify

Notifications

You can also notify your QA engineers and product managers of new releases waiting for their verification through the tools the teams already use - like Slack.
You can install a CI/CD tool as an application in your Slack workspace, which will also let it to post on your behalf. There are many ways to use it, frpm informing you when certain workflows and jobs are like deployments completed, tests are failing, and also when your workflow is waiting on an action - such as an approval or review by a QA engineer, product manager, and also security engineer, or a delivery manager. Below is an example of a generic deployment notification:

jobs:
  deploy:
    docker:
      ...
    steps:  
      - perform_deployment
      - slack/notify:
            event: pass
            template: success_tagged_deploy_1

Besides Slack there are of course numerous other tools organisations might use, from other instant messaging products to more conventional text messages and e-mails. All of these tend to have APIs which enable similar integrations to be built.

Security, Delivery management, and approvals

We have mentioned a few new roles; Security engineers are crucial for all companies to include in their SDLC processes. In an automated CI/CD pipeline, if a security team member doesn’t approve of a deployment after reviewing all necessary documentation and performing the checks, then the deployment should not complete.

CircleCI offers two very powerful concepts for ensuring access control - approval jobs and contexts.
Approval jobs introduce a manual step in your otherwise automated pipeline, which must be confirmed by an authenticated user in order for the pipeline to proceed. This is very useful for deployments for example.
Contexts in CircleCI can be applied to individual jobs, and can be restricted to specific user groups when for example connected to GitHub.
When using both of them, we can apply contexts to the subsequent job in the workflow after the approval job, in addition to adding user groups to the context itself.
This means that if a non-authenticated person tries to approve this step, the subsequent step will fail with an unauthorized message. I have written a whole blog post on the topic of implementing access control: https://circleci.com/blog/access-control-cicd/.

You can of course combine both with what we covered previously - Jira and Slack, to flag and notify the folks required to act on a deployment when it happens.

Going a few steps further even, once the relevant team member is notified and sent a link to the relevant pipeline or job, why not give them everything they need to make a decision? We expect this it as part of reviewing pull and merge requests after all.
For every testing job, security scan, and other checks we execute these tools can usually gather reports - why not make all reports available in a single report job. That’s possible by passing reports between jobs using a workspace:

jobs:
  security_scan:
    ...
    steps:
      - ...
      - perform-security-scan
      - persist_to_workspace:
          root: workspace
          paths:
            - security-output
  generate_reports:
    ...
    steps:
      - attach_workspace:
          at: /tmp/workspace
      - store_artifacts:
          path: /tmp/workspace/security-output
      - store_artifacts:
          path: /tmp/workspace/some_other_report_output

In this way you can leverage the power of a workspace to create a single point where all the relevant reports are collated. This can be a lightweight job which can save a busy team a lot of time and effort. Besides security engineers the same steps can be taken to inform delivery managers for example, and the manual approval jobs can be chained one after another, so the review steps are taken in order.

Ops, deployment reporting, and going beyond the individual dev team

Now the software is approved and being deployed, we can notify the operations teams of a new deployment.
That means that if any defects have been published, we can trace that back to the problematic deployment, revert that feature, and understand the root cause of breakage. Depending on what tools your operations teams use to monitor the software in production, you can call that API using either an orb or directly, which lets you tie a CircleCI pipeline with a deployment event.

Finally let’s take a few steps back and realise that we’ve only been talking about a single team and the relevant stakeholders. Apart from very small companies (whose engineers are too busy to read this article anyway) that single team doesn't exist in a vacuum, but works with adjacent teams working on different parts of software, sometimes even integrating one another. So not only is there a single software development cycle, there are many.

This means that implementing the above steps for a single team helps just that - the single team, but far from the wider organisation. People responsible for working across teams, should take on that work to ensure best practices are followed throughout the organisation. This could be the DevOps team that implement the integrations with security, PM, or ops processes, or the head of engineering looking to find way for their teams of engineers to learn and adopt approaches that work from each other. Looking at insights across multiple teams you could consider the pipeline throughput, mean time to recovery, and other metrics covered in our 2022 State of Software Delivery report to understand and identify the best practices within your own organisation, and areas where you can apply these relatively small improvements for the greatest impact in most teams.

Conclusion

We’re not in the world anymore where CI/CD is the sole domain of the development teams.
The role of DevOps teams is to not only work across the Dev organisations but with the wider teams, promoting the virtues of automation and how it benefits other parts of the business. The DevOps and CI/CD engineers are in fact uniquely positioned to bring that automation to more and more people and roles within the org.

In this article we have looked at a few ways CI/CD can be used to reach more parts of the company, from sending a slack message to your security engineers telling them of a required review, to letting your QA team know that a new bug fix has been deployed and is ready for testing in their ticketing application, to helping your ops team know that a new deployment has indeed started and could potentially have an impact on the users they are seeing.

If this article has helped you in a way, I would love to know, also if you have any questions or suggestions about it, or ideas for future articles and guides, reach out to me on Twitter - @zmarkan or email me.

Introducing the CircleCI Config SDK

Kyle TechSquidTV — Mon, 19 Sep 2022 18:00:00 +0000

We are excited to announce the new CircleCI Config SDK is now available as an open-source TypeScript library. Developers can now write and manage their CircleCI config.yml files using TypeScript and JavaScript.

For developers used to the ecosystem and flexibility of a full-fledged programming language, sometimes YAML can feel limiting or intimidating. With the Config SDK you can define and generate your YAML config from type-safe and annotated JavaScript. You can even take advantage of package management to modularize any portion of your config code for reuse.

When paired with CircleCI’s dynamic configuration, the Config SDK makes it possible to dynamically build your CI configuration at run-time, allowing you to choose what you want to execute based on any number of factors, such as the status of your Git repo, an external API, or just the day of the week.

Getting started

For our example, let’s imagine we manage several Node.js projects that are all built using the same framework and generally require the same CI configuration. So we decided that we want to build a config “template” that all of our projects will use, and which can be centrally managed and updated. We’ll create and publish an NPM package that will generate the perfect config file for all of our Node projects.

Let’s build the config template package, and then build the pipelines that will use it.

Setup

We’re going to start by creating a standard NPM package. You can use TypeScript or JavaScript, but we’ll use JavaScript in this example for the sake of speed. The example shown here is based onthis page from the repo’s wiki.

Begin by initializing a JavaScript project in a new directory.

mkdir <your-package-name>

cd <your-package-name>

npm init -y

npm i --save @circleci/circleci-config-sdk

The @circleci/circleci-config-sdk package will allow us to define a CircleCI config file with JavaScript. While we could simply define a config and export it, we can also take advantage of dynamic config and export a function instead. In our example, we’ll keep it simple and create a config generation function that will take a tag parameter for our deployments, and a path parameter to choose where the config file will be exported to.

Create the app

Create an index.js file and import the CircleCI Config SDK package and Node’s fs package so we can write the config to a file.

const CircleCI = require("@circleci/circleci-config-sdk");
const fs = require('fs');

Next we’ll start building up the components of our config file using the Config SDK. You’ll notice because we are working with a TypeScript-based library, we are able to receive code hints, type definitions, documentation and auto-completion.

Create executor

Given we are building a config for our Node.js projects, we’ll begin by defining the Docker executor our jobs will use. You can pass in the Docker image, resource class, and any other parameters you may want to configure.

// Node executor
const dockerNode = new CircleCI.executors.DockerExecutor(
  "cimg/node:lts"
);

Create jobs

We are building up to a workflow that will test our application on every commit, and deploy it when we provide a certain tag. Like our executor, we’ll define these two jobs, both using the executor we just defined and each with a unique set of steps for their respective purposes.

// Test Job
const testJob = new CircleCI.Job("test", dockerNode);
testJob.addStep(new CircleCI.commands.Checkout());
testJob.addStep(new CircleCI.commands.Run({ command: "npm install && npm run test" }));

//Deploy Job
const deployJob = new CircleCI.Job("deploy", dockerNode);
deployJob.addStep(new CircleCI.commands.Checkout());
deployJob.addStep(new CircleCI.commands.Run({ command: "npm run deploy" }));

Jobs can be instantiated with steps or dynamically added to an existing job like shown above. In this overly simplified example, we lack a caching step, but you can see how we build up elements of our configuration.

Create a workflow

With our jobs defined, it’s time to implement them in a workflow and define how they should run. We mentioned earlier that we want the test job to run on all commits, and the deploy job to only run on the given tag.

Now that we are working with top-level components of our config file, let’s finally define a new CircleCI config object, and name the workflow that we will add our jobs to.

//Instantiate Config and Workflow
const nodeConfig = new CircleCI.Config();
const nodeWorkflow = new CircleCI.Workflow("node-test-deploy");
nodeConfig.addWorkflow(nodeWorkflow);

We are not adding any parameters to our testing job because we want to run it on all commits, so we can directly add it to our config object.

nodeWorkflow.addJob(testJob);

For the deploy job, we need to first define the workflow job so that we can add filters to it. We are going to add one filter now which tells CircleCI to ignore this job for all branches, so it doesn’t execute on every commit. We’ll deal with enabling it for a tag in a moment.

const wfDeployJob = new CircleCI.workflow.WorkflowJob(deployJob, {requires: ["test"], filters: {branches: {ignore: ".*"}}});
nodeWorkflow.jobs.push(wfDeployJob);

Export the config generator function

Now that we have everything defined we are ready to create and export the final piece. We are going to create a function which takes in the tag and path parameters we mentioned earlier and will write what we have defined to a new file.

/**
* Exports a CircleCI config for a node project
*/
export default function writeNodeConfig(deployTag, configPath) {
 // next step
}

In the newly created writeNodeConfig function, we’ll add the tag filter being passed in here to the deploy job in our workflow and finally write the config to the file supplied by the path parameter using the generate function on the config object.

/**
* Exports a CircleCI config for a node project
*/
export default function writeNodeConfig(deployTag, configPath) {
  wfDeployJob.parameters.filters.tags = {only: deployTag}
  fs.writeFile(configPath, nodeConfig.generate(), (err) => {
    if (err) {
      console.error(err);
      return
    }
  })
}

Here is the full source code, which you can also find in the wiki:

const CircleCI = require("@circleci/circleci-config-sdk");
const fs = require('fs');

// Node executor
const dockerNode = new CircleCI.executors.DockerExecutor(
  "cimg/node:lts"
);

// Test Job
const testJob = new CircleCI.Job("test", dockerNode);
testJob.addStep(new CircleCI.commands.Checkout());
testJob.addStep(new CircleCI.commands.Run({ command: "npm install && npm run test" }));

//Deploy Job
const deployJob = new CircleCI.Job("deploy", dockerNode);
deployJob.addStep(new CircleCI.commands.Checkout());
deployJob.addStep(new CircleCI.commands.Run({ command: "npm run deploy" }));

//Instantiate Config and Workflow
const nodeConfig = new CircleCI.Config();
const nodeWorkflow = new CircleCI.Workflow("node-test-deploy");
nodeConfig.addWorkflow(nodeWorkflow);

//Add Jobs. Add filters to deploy job
nodeWorkflow.addJob(testJob);
const wfDeployJob = new CircleCI.workflow.WorkflowJob(deployJob, {requires: ["test"], filters: {branches: {ignore: ".*"}}});
nodeWorkflow.jobs.push(wfDeployJob);

/**
* Exports a CircleCI config for a node project
*/
export default function writeNodeConfig(deployTag, configPath) {
  wfDeployJob.parameters.filters.tags = {only: deployTag};
  fs.writeFile(configPath, nodeConfig.generate(), (err) => {
    if (err) {
      console.error(err);
      return
    }
  });
}

Publish the package

With your index.js file completed with the writeNodeConfig function exported, it’s time to publish the package to your package repository of choice, such as NPM or GitHub.

When complete, you should be able to import your package in other projects, just like we imported @circleci/circleci-config-sdk earlier.

Create a CI pipeline

You now have a published NPM package that can generate a CircleCI config file. We can use CircleCI’s dynamic configuration to pull in this package at run-time and dynamically run our generated config file. We can replicate this basic template across our many similar NodeJS projects, and when we want to update or change our config, we will be able to simply update the package we created.

Create config.yml

As usual, our CircleCI project will require a .circleci directory and a config.yml file inside. The config file in this case will be a basic template used in all of our projects which simply tells CircleCI to enable the dynamic configuration feature for the current pipeline, generate the new config file, and run it. We will also create a dynamic directory that we will use later.

└── .circleci/
    ├── dynamic/
    └── config.yml` \

Use the following example configuration file:

version: 2.1
orbs:
  continuation: circleci/continuation@0.3.1
  node: circleci/node@5.0.2
setup: true
jobs:
  generate-config:
    executor: node/default
    steps:
      - checkout
      - node/install-packages:
          app-dir: .circleci/dynamic
      - run:
          name: Generate config
          command: node .circleci/dynamic/index.js
      - continuation/continue:
          configuration_path: ./dynamicConfig.yml
workflows:
  dynamic-workflow:
    jobs:
      - generate-config

You can find this config file and other examples in the Wiki on GitHub.

This config is a boilerplate responsible for executing our package which contains the “real” logic we want to execute.

Notice the setup key is set to true, enabling dynamic config for the pipeline. Using the Node orb, we install a Node app located in .circleci/dynamic (we’ll come back to this), and run the .circleci/dynamic/index.js in Node. This is what will use the package we wrote earlier and create a new config file at ./dynamicConfig.yml, which will finally be executed by the continuation orb.

We will use a config like this one in all of our Node projects, and it is unlikely to need to be updated or changed often. Rather than modifying this config, we update the package we created earlier.

Create the config app

The last thing to do is build our “Config application”. This is the app responsible for implementing our package and acts as the source of our “real” config we plan on executing. Because in this example we have outsourced the majority of the logic of our config to an external package, in this example, our config app is mostly boilerplate as well.

Change directory into .circleci/dynamic where we will set up our application. Initialize a new repository and install the package you published previously.

npm init -y

npm i <your-package-name>

After running these two commands, you should have a package.json file with a dependency showing your package.

"dependencies": {
  "my-circleci-node-config": "^1.0.0",
}

You can modify the semantic version string to dictate what version of your package should be pulled at run time. This means, you could update your my-circleci-node-config package and, if you choose, have all of your CircleCI projects that utilize this package immediately pick up these changes the next time your CI pipeline is triggered.

Not Recommended: Say you always wanted to pull the latest version of the custom dependency you have created, you could use:

"dependencies": {
  "my-circleci-node-config": "x",
},

To be safer, pull in only minor and patch updates, not major releases.

"dependencies": {
  "my-circleci-node-config": "1.x",
},

Finally, utilize your custom package in the .circleci/dynamic.index.js.

Our package exports a function named writeNodeConfig which takes in the value of the tag we want for triggering deployments, and the path we want to export the config to. We know the path from earlier in our config.yml, we set to be ./dynamicConfig.yml, because we are in the dynamic directory, we will prepend ... For the tag, we’ll use a generic regex string v.*

import writeNodeConfig from '<your/package>';

writeNodeConfig("v.*", "../dynamicConfig.yml")

That is our entire application. We simply need to pull whichever version of our package we desire and invoke it to generate our config file from the template we created in the package.

Running the pipeline

To recap, we have a boilerplate config.yml file which instructs CircleCI to enable dynamic configuration, and uses Node.js to build a new config file at run-time. The Node app responsible for building our new config uses a package dependency which contains the “template” of our desired config. We can now use this package in many different projects and update it centrally. Our projects can either pull in the latest version of this package by specifying an x in the package.json, or we can use tools like dependabot to open a pull request automatically to all of our projects that use this package when it is updated.

With dynamic configuration and the Config SDK together, the possibilities are endless. The tutorial above was based on this page from the wiki on GitHub. Check out the rest of our wiki and documentation for even more examples of interesting things you can do with the Config SDK.

We’d love to hear from you and see how you utilize the Config SDK in your own pipelines. Connect with us on Twitter, say hello in our discussion forum, or chat with us on Discord.

Continuous integration for Android projects

Zan Markan — Fri, 26 Aug 2022 19:00:00 +0000

CircleCI is popular among Android developers for several reasons: it’s quick to get started, fast to execute your builds with high parallelism, (whether native, cross- or multi-platform), and even supports running Android emulators right from CircleCI with our Android machine images.

This article will show you how to build and test Android applications for an example project on the CircleCI platform. The full source code is available on GitHub - CircleCI-Public/android-testing-circleci-examples, and you can find the working pipelines in the corresponding CircleCI project.

To get the most out of this article you should have

Working knowledge of Android development
Familiarity with Android tooling and testing frameworks
Ability to use the Gradle build system

What’s available in CircleCI for Android

We have made a few tools available for you for building Android:

Docker image
Android machine image
Android orb

Docker image

The Docker image is the main executor teams use. It runs in a Docker container and so is very fast to start, and contains everything you might need to get started building Android projects. The full list of preinstalled tools can be found in the documentation. The images also ship with variants that have Node.js, the Android NDK, and browser tools preinstalled.

Machine image

The biggest change is the new Android machine image. This is a virtual machine image, so takes a bit more time than the Docker image mentioned above, but it contains all the tools you may need to build your Android applications, including the SDK, Google Cloud tools for Firebase, Python, Node.JS, and Fastlane. The biggest change however is the support for nested virtualisation, which allows you to run Android emulators from within this machine image.

Android machine images support nested virtualization, which allow running the x86 emulators in CircleCI jobs. If you have been developing for Android since the early years of the platform, you may remember that emulators were very slow. Then x86 emulators started supporting Intel’s Hyper-V hypervisors, which lets the emulator use the resources of the host computer directly. Using resource directly like this makes the emulators able to run much, much more smoothly and quickly.

For a long time, support was confined to our local hardware. Limited support on CircleCI slowed down emulators and made it difficult to run UI tests. Nested virtualization solves this problem by adding support for fast emulation to a massively reproducible CI/CD environment, making all the benefits of that available.

Android Orb

Finally there is the CircleCI Android orb which provides easy access to both images as executors, as well as handy jobs and commands for installing SDKs, running emulators, caching, testing, and more. You can read more about what the orb offers in the orb docs at this link.

Introducing the Android CI demo project

I have created a demo project, which is a fork of Google’s own Android Testing Codelab sources. The Codelab is a simple TO-DO list manager application and contains a combination of unit tests and instrumentation tests, with and without the UI. The project is also available to view on CircleCI. If you open it in Android Studio, make sure to switch to the Project view. Project view lets you view the .circleci/config.yml file we will be reviewing.

The project has a single app module, with the default debug and release variants. The test directory contains unit tests, while the androidTest directory contains instrumentation tests. The full CircleCI configuration is included, and I will be referring to it in this article. The configuration is located in .circleci/config.yml.

Building and testing CI for the sample Android app

Our CI process must do three things:

Run unit tests inside a virtual machine
Run instrumentation tests on the emulator
Build a release version of the app

We will run unit tests and instrumentation tests side by side. If both types of tests are successful, we can assemble the release build. We will also add a condition to continue on to the release build only if new work has been commited to the “main” branch, and only following successful emulator tests on every Android version from 23 to 30. When our build meets this condition, we will have solid assurance that our application will work on many different Android devices.

Note: As I mentioned before, everything from this point on will refer to the CircleCI configuration file in .circleci/config.yml.

Using the CircleCI Android orb

The first step for creating our CI pipeline is to use the Android orb, which contains many of the steps already pre-written for you. The latest version is available in the orb docs.

After the orb is defined, the script specifies these jobs:

unit-test
android-test
release-build

One workflow, test-and-build, is also included.

version: 2.1

orbs:
  android: "2.1.2"

jobs:
  unit-test:
    ...
  android-test:
    ...
  release-build:
    ...
    # We'll get to this later

workflows:
  test-and-build:
    ...
  # We'll get to this later

Running unit tests

Our unit-test job contains an executor from the Android orb: android/android-docker. As mentioned this runs it in a Docker container which is extremely fast to start up.

In the steps stanza, there are a few things to note. First, we run android/restore-gradle-cache Gradle cache helps store our dependencies and build artifacts. After running the test command, we also run android/save-gradle-cache and android/save-build-cache to make sure subsequent builds run faster by using the cache. These all come from the Android orb, as indicated by the prefix android/.

We could also modify what gets cached by passing the find-args parameter to restore-gradle-cache. A great advanced example can be found in this open source project.

The main build step happens in the android/run-tests command, where we call ./gradlew testDebug within a test-command parameter. This command runs all unit tests for the debug build variant. As a bonus, this command comes with a default retry value to help you avoid test flakiness.

jobs:
  unit-test:
    executor:
      name: android/android-docker
      tag: 2022.08.1
    steps:
      - checkout
      - android/restore-gradle-cache
      - android/run-tests:
          test-command: ./gradlew testDebug
      - android/save-gradle-cache
      ...

Running Android tests on the emulator

Our android-test job is the one that uses the new Android emulator capability. The job uses the same Android machine executor as the build job. All subsequent jobs will use the same executor, and the cache restoration steps are also the same.

Emulator requires us to run on a machine image, so we specified android/android-machine as the executor. To improve build times, we specify the resource-class as xlarge. You can also run it on a large executor, but I have found that the lack of resources causes slower build times. I recommend that you do some experimentation on your own to find the right size executor for your project.

The next step is even shorter; we need just the android/start-emulator-and-run-tests command. This command does exactly what it says: it starts the specified emulator and runs the tests. We pass in the test-command again as the parameter (it uses the android/run-tests command). The system-image is the fully qualified Android emulator system image. We currently bundle our Android machine images back to SDK version 23. You could install more by using the sdkmanager tool.

jobs:
  android-test:
    executor:
      name: android/android-machine
      resource-class: xlarge
    steps:
      - checkout
      - android/start-emulator-and-run-tests:
          test-command: ./gradlew connectedDebugAndroidTest
          system-image: system-images;android-30;google_apis;x86
      ...

You could code all the emulator setup steps manually using the commands specified in the Android orb: create-avd, start-emulator, wait-for-emulator, and run-tests.

Storing the test results

There is more to testing than just running the tests, as we can get a lot of value from seeing exactly what has failed from right inside the CircleCI dashboard. For that we can use store_test_results functions built into the CircleCI platform, which will show our passing (or failing) builds.

The steps differ slightly between unit tests and instrumentation tests, as each of their respective Gradle tasks stores the tests in a different place. For unit tests the tests will be in app/build/test-results, and for instrumentation tests they will be in app/build/outputs/androidTest-results.

We suggest you create a new Save test results step, which uses the find utility and copies all test results files into a common location - like test-results/junit, and then store it from there. Also make sure to add the when: always parameter to the step, so the step runs regardless of whether the tests above succeed or fail.

Storing unit tests

For this project, we want the XML based results. Calling store_artifacts makes those results parseable on the CircleCI platform. You can also store the HTML output if you prefer.

...
- android/run-tests:
  test-command: ./gradlew testDebug
- run:
    name: Save test results
    command: |
        mkdir -p ~/test-results/junit/
        find . -type f -regex ".*/build/test-results/.*xml" -exec cp {} ~/test-results/junit/ \;
    when: always
- store_test_results:
    path: ~/test-results
- store_artifacts:
    path: ~/test-results/junit

Storing instrumentation tests

There are different regex command for different test types. The rest of the steps to store test results are identical to the unit tests example.

- android/start-emulator-and-run-tests:
        test-command: ./gradlew connectedDebugAndroidTest
        system-image: << parameters.system-image >>
    - run:
        name: Save test results
        command: |
          mkdir -p ~/test-results/junit/
          find . -type f -regex ".*/build/outputs/androidTest-results/.*xml" -exec cp {} ~/test-results/junit/ \;
        when: always
    - store_test_results:
        path: ~/test-results
    - store_artifacts:
        path: ~/test-results/junit

Making a release build and storing the artifacts

For this project, we need to store the application we are building. To complete this part of the process, we use the release-build job, which has two parts of note. A run step calls ./gradlew assembleRelease. Then, store_artifacts points to where the built APK is. For a full CI/CD project, you could sign and upload that APK to a beta distribution service. You could even create a release on Play Store with Fastlane. Both of those activties are out of scope for this article, though.

jobs:
  ...
  release-build:
    executor:
      name: android/android-machine
      resource-class: xlarge
    steps:
      - checkout
      - android/restore-gradle-cache
      - android/restore-build-cache
      - run:
          name: Assemble release build
          command: |
            ./gradlew assembleRelease
      - store_artifacts:
          path: app/build/outputs/apk/release

Running tests on multiple Android versions simultaneously

The Android platform evolves all the time, and each version has its own bugs and pecularities. I am sure you have encountered some of them. A good way to avoid undefined behaviour across different versions of Android is to test on as many of them as you can. The CircleCI job matrix feature makes it easier to run jobs on multiple versions.

To use the job matrix, add a parameter to the job you want to test on multiple versions. Our android-test job does that using the system-image parameter. We have also specified a default value for the parameter, so you can run the job without it.

To use the parameter, specify where you need its value by placing it within pairs of angle brackets - << >>:

 android-test:
    parameters: # this is a parameter
      system-image:
        type: string
        default: system-images;android-30;google_apis;x86
    executor:
      name: android/android-machine
      resource-class: xlarge
    steps:
      - checkout
      - android/start-emulator-and-run-tests:
          test-command: ./gradlew connectedDebugAndroidTest
          system-image: << parameters.system-image >> # parameter being used

To pass values to the parameter in the workflows, use the matrix parameter of the job we are calling.

workflows:
  jobs:
  ...
  - android-test:
      matrix:
        alias: android-test-all
        parameters:
          system-image:
            - system-images;android-30;google_apis;x86
            - system-images;android-29;google_apis;x86
            - system-images;android-28;google_apis;x86
            - system-images;android-27;google_apis;x86
            - system-images;android-26;google_apis;x86
            - system-images;android-26;google_apis;x86
            - system-images;android-26;google_apis;x86
            - system-images;android-26;google_apis;x86
      name: android-test-<<matrix.system-image>>

Making each run faster by caching

Alongside building our application, installing dependencies, and running tests, we often want to leverage cache as well. This allows us to do repeated tasks less often. Android and especially Gradle provides an excellent mechanism for this - the Gradle Build cache.

This lets you skip pre-built parts of the application.

Choosing what runs when

During development, we create many commits, and developers should be encouraged to push them to a remote repository as often as possible. Each one of these committs creates a new CI/CD build, though, and we might not need every test run on all possible devices for a single commit. On the other hand, if the main branch is our primary source of truth, we want to run as many tests on that as possible, to ensure it is always working as intended. We might want to initate the release deployment only on main, and not on any other branch.

We can use CircleCI’s requires and filters stanzas to define workflows that fit our process. Our release-build job requires the unit-test jobs. That means the release-build is put in the queue until all the unit-test jobs have passed. Only then is the release-build job run.

We can use filters to set up logical rules for running a particular job only on a specific branch or git tag. For example, we have a filter for our main branch that runs the android-test with a full matrix of emulators. In another example, the release build triggers only on main, and only when its two required jobs have run successfully.

workflows:
  test-and-build:
    jobs:
      - unit-test
      - android-test: # Commits to any branch - skip matrix of devices
          filters:
            branches:
              ignore: main
      - android-test: # Commits to main branch only - run full matrix
          matrix:
            alias: android-test-all
            parameters:
              system-image:
                - system-images;android-30;google_apis;x86
                - system-images;android-29;google_apis;x86
                - system-images;android-28;google_apis;x86
                - system-images;android-27;google_apis;x86
                - system-images;android-26;google_apis;x86
                - system-images;android-25;google_apis;x86
                - system-images;android-24;google_apis;x86
                - system-images;android-23;google_apis;x86
          name: android-test-<<matrix.system-image>>
          filters:
            branches:
              only: main 
      - release-build:
          requires:
            - unit-test
            - android-test-all
          filters:
            branches:
              only: main # Commits to main branch

Improving your Android application CI process

The steps I have described in this article are only a beginning. There are many ways you can improve the flow. Here are just a handful of ideas:

Build once, run many times

If you are an experienced Android developer you know that the connectedAndroidTest will always run the entire build process from the start. Using CircleCI, we could build the entire application and tests once, pass the artifacts down to following jobs, and simply run the tests on the emulators. This process could potentially save several build minutes per job run.

To make this happen, add three commandline steps in each emulator run: install the app, install the instrumentation app, and run the instrumentation tests. For our application, we would enter:

adb install app/build/outputs/apk/debug/app-debug.apk  
adb install app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk  
adb shell am instrument -w com.example.android.architecture.blueprints.reactive.test/androidx.test.runner.AndroidJUnitRunner

It is true that this code sends output to the the command line. That is not the tidy test output that Gradle provides. If you are motivated to make the output look better, this StackOverflow topic might help you.

Running on real devices

Emulators are one tool, but real world devices are often (sadly) much different. There is nothing that can replace a real-world device test. For that you can use a solution offering real devices in the cloud, like Sauce Labs or Firebase Test Lab.

Deploying to beta testers, and releasing the application

This build and test example project is just one part of a wider CI/CD story. You can take your CI/CD process much further. For example, you can automatically release new versions of your applications right to the Play Store. CircleCI has a handful of guides to help you. This tutorial is a great place to start. The official Fastlane documentation is another helpful resource.

Conclusion

In this article, I described building an Android application with CircleCI, and testing it. For our sample project, we used unit tests and a matrix of Android platform emulators, using the new CircleCI Android orb and machine images. We also touched on how to store and display test results, and how to orchestrate workflows to run tests will not give us the tidy test output that Gradle provides, and all output is on a portion of devices.

Do let me know if there is another Android topic you might want me to explore, either in an article or a live stream. Contact me on Twitter - @zmarkan.

Implementing access control policies in CI/CD pipelines

Zan Markan — Fri, 29 Apr 2022 12:00:00 +0000

This tutorial covers:

Protected branches and security groups in GitHub

Using contexts in CircleCI

Setting up approval jobs in CircleCI

Imagine yourself in this situation: You are a motivated and skilled DevOps or DevEx engineer. You have a plan to implement automated, complete CI/CD pipelines. You know how to do it, and you know how the extra productivity and automation will benefit your team and the whole company. But the project is never approved, because of security concerns.

Many organizations, especially those in regulated industries, have strict requirements for releasing their software, and rightfully so. Who can trigger a release, and when, under what conditions, and what specific checks are required before release can go ahead must be tightly controlled, auditable, and reversible.

These kinds of complicated processes often require a dedicated delivery manager to collect and present all relevant feature briefs, test data, security assessment, and rollback plans to a committee of decision makers in the company. That group can give the delivery manager the green light to deploy, often also on a specific schedule.

There is another approach. You can set up effective and fully automated CI/CD pipelines that include all the checks a human delivery manager and approval committee can complete. This tutorial will show you ways to create fine grained access control for your pipelines. You can use these methods for strictly regulated internal company projects for the enterprise, as well as for popular open source projects.

Prerequisites

This tutorial assumes you have experience with CI/CD pipelines, and ideally with CircleCI. To implement the steps covered you will also need admin access to your GitHub and CircleCI organizations (and accounts for both services).

If you do not have administrator access to a GitHub organization, you can create a free one in GitHub and use it with a free CircleCI account.

Access control in CI/CD pipelines in enterprise projects

For an enterprise organization it is vital that existing checks are followed and communicated well. The process often includes manual checks of supporting documentation and the potential impact to business before a release can continue. I will guide you through implementing a pipeline that executes a number of verification steps automatically, and continues with deployment to production only after the dedicated delivery manager manually approves the process.

To implement access control you will need to set up a few things:

Protected branches in GitHub
Security groups in GitHub
Contexts in CircleCI
Approval jobs in CircleCI

Protected branches

Setting up a protected branch is the first step to setting up access control. A protected branch prohibits team members from pushing code directly to that branch, and instead forces all changes to go through the pull request (PR) process. Commonly you would protect the branch you use to create releases from; main for example. You can toggle protected branches in the settings for your repo.

You can also specify rules and exceptions, such as review requirements, checks that need to pass before merging, history rules, and much more.

In my example I set up my main branch as a protected branch that requires a review and PR to pass.

Setting up security groups

If a user without the correct context privileges triggers a CircleCI pipeline (for example, with a commit), then the job that requires that context will be marked as 403 (unauthorized).

In our organization we will set up two security groups: development-leads and delivery-managers

All developers in your organization can push to their own branches and issue pull requests, but only team leaders should be required to review any new pull requests. Only when the team leader approves can changes be merged into the protected main branch. Then, when delivery managers get the green light to ship, they are the ones to trigger releases of software. In this tutorial, we will create a separate security group for them.

CircleCI will use these security groups with contexts to limit which jobs can be executed by whom, so that only the delivery managers can trigger them manually.

Using CircleCI contexts

Contexts in CircleCI serve a double role. One role limits the scope of secrets shared with a job while the other controls access.

Only a job that has a context specified to it in a workflow is able to use the context’s secrets as environment variables. Contexts are shared across the entire organization in CircleCI, so they can be reused in multiple projects. You can set them up using the organization settings menu on the left hand side of the CircleCI dashboard.

For access control, you can specify security groups to any context. For this tutorial,

we will create a context called release and specify the delivery-mgrs security group to it.

Now you can set up your pipeline. The sample project uses the following flow:

Tests
Security scans
Dev environment deployment
Approval job for the delivery manager
Production deployment

workflows:
 build-test-deploy:
   jobs:
     - test
     - security-scan:
         context: security
     - deploy-app:
         name: deploy-app-dev
         env: dev
         context: deployment-dev
         requires:
           - test
           - security-scan
         filters:
           branches:
             only:
               - main
     - approve-for-prod:
         type: approval
         requires:
           - deploy-app-dev
     - deploy-app:
         name: deploy-app-prod
         env: prod
         requires:
           - approve-for-prod
         context: release-prod

The pipeline uses contexts for environment variables like our API keys, but also uses the release-prod context. The release-prod context is gated so it can only be used by a delivery manager.

Setting up approval jobs

The final piece of the puzzle is the approval job: approve-for-prod. Technically, anyone can log in and approve this job. However, the deploy-app-prod job following it uses the release-prod context, which only accepts delivery managers’ credentials. If someone without those credentials approves it, the release job will fail with an unauthorized error, and the whole pipeline will terminate unsuccessfully.

Other helpful bits and pieces

One additional control we have set up in our workflow is the branch filter. The development deployment will occur only on the main branch because of that filter. The main branch is labelled as protected so the only way to get the code onto the main branch is by passing the automated checks and code review, potentially by a lead engineer.

You could also use dedicated QA or Security review steps that require manual approval. Build them the same way as part of the chain.

Considerations for open source projects

So far we covered the enterprise use case for access control. Most OSS projects do not have the same strict compliance requirements as enterprises. In contrast these projects need to have their contributions opened to a much wider group of people. That means allowing many more people to trigger pipelines.

Often the company that initiated and owns a popular OSS project continues to employ the core contributors. They will probably be joined by other regular contributors and maintainers that are not part of that company. And then there is everyone else - anyone who occasionally might contribute a fix or a feature.

For that scenario, you can set up three flows:

Company internal flow. Members contribute and also administer everything. Company employees are likely the only people allowed to release new versions of software. We can call them “inner core contributors”.
Semi-internal flow. These members are part of the organization and will be able to push directly to the main OSS repository. They will be able to use CircleCI for everything, including debugging jobs with SSH and triggering pipelines. Call them “outer core contributors”.
Standard flow. The rest of the contributors in the community who can issue PRs, otherwise work on their own. We will call them “community contributors”.

For community contributors. the flow is standard: make a fork, then open a pull request. You can toggle CircleCI to build pull requests that will automatically verify their changes.

You should still have a protected branch, and required checks.

For the groups of core contributors, you may want to include some secrets as well. Secrets require some more work. Also, passing secrets via forks is inherently insecure. Anyone could extract any secrets you allow to pass by modifying the pipeline in their PR. This could include API keys to certain services, or credentials to where your project is distributed.

One option is to avoid passing secrets to forks altogether. You will have to push any code after reviewing it into your own org, where it can be executed, and then merged into your default branch. There is a helpful article that explains the process.

The other option is to allow passing secrets to forks, but to prevent these secrets from being shared unless the code has been thoroughly reviewed and approved by a team member. You can set that up in the CircleCI project settings.

If you toggle secrets you will need to set up an approval job. Put any secrets in a context that will be accessed only by people who belong to that context.

Note: If you use this method, make sure that all contexts in your organization specify the security group. If you do not, a malicious actor could guess your context name in a PR that gets executed and extract environment variables from it.

With your inner and outer core of contributors, you can set up access controls in very much the same way. Maybe only the inner core can release new versions, but anyone in the outer core can merge any pull requests. Or maybe you just have one set of core maintainers who can do everything. The details are up to you!

Conclusion

In this article we have looked at ways to protect your CI/CD pipelines further by setting up access controls to cordon off parts of pipelines (and the associated credentials) to a limited scope of people. This is a versatile approach and works for both enterprise organizations as well as OSS projects.

We used security groups, protected branches, contexts, and approval jobs in CircleCI to achieve that effect, for both OSS and proprietary projects. Experiment with the best configuration for your organization and release your applications safely and securely.

Config best practices: concurrency and parallelism

Katy — Tue, 29 Mar 2022 11:00:00 +0000

When is the last time you updated your CI/CD workflow? A year ago? Never? You are not alone, my friends. Reconfiguring workflows can be one of the most daunting tasks for DevOps practitioners. But with new opportunities to benefit from CircleCI plans, there’s one simple and effective place to start: understanding concurrency and parallelism.

Using concurrency and parallelism can cut your build times significantly. But you need to know what they are and how to find them in your config file.

What is concurrency?

Concurrency means that multiple computing tasks are happening at the same time. It’s everywhere in computing. We have so many things happening concurrently: applications running, computers communicating in a network, or even users visiting a website. Concurrency is so commonplace that it’s easy to assume we know how it works. While the definition always remains the same, the practical application of concurrency can be nuanced. So what does concurrency mean within CircleCI?

Simply put, concurrency in CircleCI is the number of tasks that are being executed at any point in time. For example, CircleCI’s free plan provides a concurrency limit of 30, meaning you can run up to 30 tasks at the same time.

Computing how many concurrent tasks are happening is a little more complicated than finding out how many jobs are running or how many containers are being used. Pipelines can have all sorts of twists and turns that change the number of concurrent tasks, like conditional logic or test splitting. For this post, I’ll focus on how to manage concurrency and when it matters.

What is parallelism?

It’s hard to write about concurrency in CircleCI without talking about parallelism. The two concepts are often conflated, but have distinct applications. We already know concurrency is the number of executing tasks at any given time in a workflow. We can affect concurrency by changing parallelism, which is where some confusion can start.

Parallelism splits work between identical copies of a particular job.

Parallelism is most often used to split up test suites. All of the copies of the job have the same instructions, but run with different variables. Parallelism is set in the CircleCI configuration file, and the number of parallel jobs counts toward your concurrency total.

Applying concurrency and parallelism

Here’s an example using the CircleCI free plan, which has a concurrency limit of 30. Say you have 10 jobs that each take 1 minute to execute. Without concurrency, this workflow would take 10 minutes to complete. With concurrency, each of those jobs could run at the same time, and the workflow would be done in 1 minute instead of 10. If you set parallelism to 3, you can still run 10 jobs at the same time and be within the concurrency limit. You can run 3 copies of 10 jobs for 30 total tasks done in a minute.

You could just as easily run 4 copies (parallelism of 4) on 7 jobs for a total of 28 concurrent tasks. The free plan allows a maximum parallelism of 4, but other plans have more options if you really need the speed.

If you have a situation where the total number of simultaneous tasks is more than 30, some of the work will have to wait. For example, if you set parallelism to 4 on 8 jobs, one of those jobs (and all of its copies) will have to wait until there are resources free. The copies of jobs created by parallelism always run simultaneously, so even if you’re only over the concurrency limit by 2 (as in this case), all four copies of one job will wait until another job finishes.

How to add parallelism and concurrency to your pipelines

Most of the time, you don’t need to worry about concurrency. Concurrency limits are set by your CircleCI plan and are enforced in the background. You may not need to manage concurrency at all. Most often, you will take advantage of your concurrency by working with parallelism. Setting parallelism is easy: set the value of the parallelism key in the config.yml. Any value greater than 1 means you’re running parallel tasks.

How to set parallelism

 ~/.circleci/config.yml

version: 2
jobs:
  test:
    docker:
      - image: cimg/<language>:<version TAG>
        auth:
          username: mydockerhub-user
          password: $DOCKERHUB_PASSWORD # context / project UI env-var reference
    parallelism: 3

This example has one job (named test) running on Docker. Setting parallelism to 3 means three copies of this job (called tasks) will run simultaneously on three separate Docker containers. The only differences between these tasks are environment variables so that work can be split among these tasks. This is most often used for test splitting, which you can read more about in our documentation.

Conclusion

As DevOps practitioners, we care about concurrency because it helps get work done faster by running multiple processes at the same time. Knowing the concurrency limit of your plan and all the tasks that count toward it helps you optimize your builds.

Concurrency can be a straightforward concept. Jobs finish sooner when at least some of them can run at the same time. Parallelism helps you customize which tasks are done concurrently and is easily set in the config file. Concurrency and parallelism both help finish tasks quicker so you can get on to the important business of failing, passing, and shipping software.

The next step is to continue learning about test splitting and continue to automate and expedite your jobs. If you’re interested in expert reviews of your CircleCI configuration to help optimize your work, you can get a custom evaluation from a dedicated support engineer by signing up for a premium support plan.

Advanced pipeline orchestration with the circleback pattern

Zan Markan — Fri, 25 Feb 2022 10:00:00 +0000

This tutorial covers:

Orchestrating triggers for a dependent pipeline

Intro to the circleback pattern

Creating the config for a circleback pipeline

With multiple teams working on many projects, having a single pipeline for your software is just not enough. These projects need to be built and integrated before they can be tested and released. So how do dev teams handle this situation? Many teams approach the problem by breaking down software into smaller parts that do less, and are easier to maintain and build. This approach has resulted in the microservices architectures that are increasingly common in our industry.

The downside is that breaking software into smaller parts increases complexity. The combination of more parts and higher complexity makes it more difficult to test applications and operate them in production.

CI/CD pipelines are not an exception here. In my previous article in this series, we looked at triggering pipelines from other pipelines. That post described how to link multiple projects together in CircleCI, so that one deployment kicks off another.

In this article we will go a few steps further. I will provide an example of pipeline orchestration where the first pipeline not only triggers the other, but also waits on another pipeline to complete, before continuing. I call this the circleback pattern because it “circles back” to the second pipeline to signal when it has completed the work. Using this pattern allows for better integration testing and more complex deployment and release scenarios.

Prerequisites

This tutorial covers advanced pipeline orchestration techniques, so experience with CircleCI and DevOps is a must.
The tutorial also builds on a previous tutorial: Triggering pipelines from other pipelines.

Resources and sample code

The source code for this example is located in two repositories:

Introducing the circleback pattern

There are three steps in the circleback orchestration of these multiple pipelines:

The first pipeline waits
The second pipeline terminates
Results are retrieved from the first pipeline Another way of describing it is that the second pipeline calls or “circles back” to the first, while the first one is suspended and waiting for the signal to continue.

An alternative approach would be to keep the first pipeline running, checking periodically whether the second pipeline has finished. The main drawback of this approach is the increased cost for the running job’s continuous polling.

Depending on the status of the second pipeline, the first continues executing until it either terminates successfully or fails.

Why have pipelines wait for other projects to complete?

As mentioned in the introduction, this is an advanced technique aimed at taming the complexity that stems from working with multiple projects. It can be used by teams working on multiple interdependent projects that they do not want to put in a single repository like a monorepo. Another use would be to have a centralized repository for tests, for example in a hardware company. This technique could also be useful for integration testing of a microservices application, or for orchestrating complex deployment scenarios. There are many possibilities.

Implementing pipeline triggers

We have 2 pipelines we want to orchestrate

Pipeline A, which does the triggering
Pipeline B is triggered, and circles back to pipeline A

Pipeline B is dependent on A, and can be used to validate A.

Both pipelines need to have API keys set up and available. You can use the API key set as an environment variable (CIRCLECI_API_KEY) in the job in pipeline A, and also in pipeline B when it calls back. You can either set it in both projects, or at the organization level as a context. For this tutorial, I set it at the organization level as the circleci_api context, so that both projects can use the same API key.

Trigger the pipeline

The triggering process is explained in depth in the first part of this tutorial, Triggering pipelines from other pipelines. In this follow-up tutorial, I will cover just the important differences.

To circle back from the pipeline, pass the original pipeline’s ID to it. Then it can be retrieved and reached with the API.
You also need to store the triggered pipeline’s ID. You will need to get its result later on.

In the sample code, the parameter is called triggering-pipeline-id:

curl --request POST \
                --url https://circleci.com/api/v2/project/gh/zmarkan-demos/circleback-cicd-project-b/pipeline \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
                --data '{"branch":"main","parameters":{"triggering-pipeline-id":"<< pipeline.id >>"}}'

To store the pipeline ID, wrap your curl call in $() and assign it to the variable CREATED_PIPELINE. To extract the ID from the response body, use the jq tool, and write it to the file pipeline.txt:

CREATED_PIPELINE=$(curl --request POST \
                --url https://circleci.com/api/v2/project/gh/zmarkan-demos/semaphore_demo_project_b/pipeline \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
                --data '{"branch":"main","parameters":{"triggering-pipeline-id":"<< pipeline.id >>"}}' \
              | jq -r '.id'
              )
              echo "my created pipeline"
              echo $CREATED_PIPELINE
              mkdir ~/workspace
              echo $CREATED_PIPELINE > pipeline.txt

Now that you have the file pipeline.txt created, use persist_to_workspace to store it and use it in a subsequent job:

- persist_to_workspace:
            root: .
            paths: 
              - pipeline.txt

The whole job configuration is here:

...
jobs:
  trigger-project-b-pipeline:
      docker: 
        - image: cimg/base:2021.11
      resource_class: small
      steps:
        - run:
            name: Ping another pipeline
            command: |
              CREATED_PIPELINE=$(curl --request POST \
                --url https://circleci.com/api/v2/project/gh/zmarkan-demos/semaphore_demo_project_b/pipeline \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
                --data '{"branch":"main","parameters":{"triggering-pipeline-id":"<< pipeline.id >>"}}' \
              | jq -r '.id'
              )
              echo "my created pipeline"
              echo $CREATED_PIPELINE
              mkdir ~/workspace
              echo $CREATED_PIPELINE > pipeline.txt
        - persist_to_workspace:
            root: .
            paths: 
              - pipeline.txt  
...

Orchestrating the waits

The previous job will trigger a pipeline B, which needs to complete before it can circle back to pipeline A. You can use the approval job in CircleCI like this:

...
workflows:
  node-test-and-deploy:
    jobs:
      ...
      - trigger-project-b-pipeline:
          context: 
            - circleci-api
          requires:
            - build-and-test
          filters:
            branches:
              only: main
      - wait-for-triggered-pipeline:
          type: approval
          requires: 
            - trigger-project-b-pipeline
      - check-status-of-triggered-pipeline:
          requires:
            - wait-for-triggered-pipeline
          context:
            - circleci-api
      ...

After the job trigger-project-b-pipeline, enter the wait-for-triggered-pipeline. Because that job type is approval it will wait until someone (in this case, the API) manually approves it. (More details in the next section.) After it is approved, add a requires stanza so it continues to a subsequent job.

Both jobs that use the CircleCI API have the context specified, so the API token is available to both as an environment variable.

Circle back to pipeline A

For now we are done with pipeline A, and it is pipeline B’s time to shine. CircleCI’s approval job is a special kind of job that waits until accepted. It is commonly used to hold a pipeline in a pending state until it is approved by a human delivery lead or infosec engineer.

At this point, pipeline B knows the ID of pipeline A, so you can use the approval API to get it. You only have the ID of the pipeline being run, but not the actual job that needs approval, so you will need more than one API call:

Get all jobs in the pipeline
Find the approval job by name
Send a request to approve the job

Approving the job allows pipeline A to continue.

If the tests fail in pipeline B, then that job automatically fails. A workflow with required jobs will not continue in this case. You can get around by using post-steps in the pipeline, which always executes. The whole workflow is shown in the next sample code block.

Parameters:

parameters:
  triggering-pipeline-id:
    type: string
    default: ""

...

workflows:
  node-test-and-deploy:
    jobs:
      - build-and-test:
          post-steps:
            - approve-job-in-triggering-pipeline
          context: 
            - circleci-api

Script to perform the approval API call can be implemented like this. For this tutorial, I used a command.

...
commands:
  approve-job-in-triggering-pipeline:
    steps:
      - run:
          name: Ping CircleCI API and approve the pending job
          command: |
            echo << pipeline.parameters.triggering-pipeline-id >>
            if ! [-z "<< pipeline.parameters.triggering-pipeline-id >>"] 
            then
              workflow_id=$(curl --request GET \
                --url https://circleci.com/api/v2/pipeline/<< pipeline.parameters.triggering-pipeline-id >>/workflow \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
              | jq -r '.items[0].id')

              echo $workflow_id

              waiting_job_id=$(curl --request GET \
                --url https://circleci.com/api/v2/workflow/$workflow_id/job \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
              | jq -r '.items[] | select(.name == "wait-for-triggered-pipeline").id')

              echo $waiting_job_id

              curl --request POST \
                --url https://circleci.com/api/v2/workflow/$workflow_id/approve/$waiting_job_id \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json"

            fi
          when: always       
...

The script first checks for the existence of the triggering-pipeline-id pipeline parameter. It proceeds only if that parameter exists. The when: always line in the command makes sure that this executes regardless of termination status.

Then it makes 3 API calls:

Get workflow ID in a pipeline. There is just one workflow in that pipeline for this sample project.
Get jobs in that workflow, use jq to select the one that matches the approval job name (wait-for-triggered-pipeline), and extract the approval job’s ID.
Make the request to the approval endpoint with the waiting job ID.

For this tutorial, we are storing results like workflow ID and job ID in local bash variables, and using them in subsequent calls to the API.

Note : If you have more jobs than can be sent in a single response, you might have to handle pagination as well.

Now that you have made the approval request, pipeline B is complete, and pipeline A should be running again.

Update pipeline A with the result of B

After pipeline A has been approved, the next job in the workflow will begin. If your workflow graph requires it, pipeline A can trigger multiple jobs.

We still do not have any indication of the result of the previous workflow. To get that information, you can use the API again to get B’s status from pipeline A. An example job could look like that: check-status-of-triggered-pipeline.

First, you need to retrieve the ID of the triggered pipeline, which is pipeline B. This is the same ID that was persisted in a workspace in an earlier step. Retrieve it using cat:

 - attach_workspace:
          at: workspace
      - run:
          name: Check triggered workflow status
          command: |
            triggered_pipeline_id=$(cat workspace/pipeline.txt)

Then use the API to retrieve the workflow. Use jq to get just the status of the first item in the returned array of workflows:

created_workflow_status=$(curl --request GET \
                --url "https://circleci.com/api/v2/pipeline/${triggered_pipeline_id}/workflow" \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
              | jq -r '.items[0].status'
            )

Check that status is not success. If it is not, use exit to terminate the job with exit code -1. If the workflow is successful, it will terminate:

if [["$created_workflow_status" != "success"]]; then
              echo "Workflow not successful - ${created_workflow_status}"
              (exit -1) 
            fi

            echo "Created workflow successful"

Here is the full config for the job check-status-of-triggered-pipeline:

 check-status-of-triggered-pipeline:
    docker: 
      - image: cimg/base:2021.11
    resource_class: small 
    steps:
      - attach_workspace:
          at: workspace
      - run:
          name: Check triggered workflow status
          command: |
            triggered_pipeline_id=$(cat workspace/pipeline.txt)
            created_workflow_status=$(curl --request GET \
                --url "https://circleci.com/api/v2/pipeline/${triggered_pipeline_id}/workflow" \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
              | jq -r '.items[0].status'
            )
            echo $created_workflow_status
            if [["$created_workflow_status" != "success"]]; then
              echo "Workflow not successful - ${created_workflow_status}"
              (exit -1) 
            fi

            echo "Created workflow successful"

Conclusion

In this article we have reviewed an example of a complex pipeline orchestration pattern that I have named “circleback”. The circleback pattern creates a dependent pipeline and allows you to wait for it to terminate before completing. It involves making several API keys from both projects, the use of an approval job, and the workspace feature of CircleCI to store and pass values such as pipeline ID across jobs in a workflow. The sample projects are located in separate repositories: project A, and project B.

Testing locally with CircleCI runners

Zan Markan — Tue, 18 Jan 2022 10:00:00 +0000

In this article you will learn how to set up the CircleCI runner agent on your local machine to run tests. You will also learn how to configure your CircleCI pipeline so that the runner is invoked correctly.

Many development teams start their CI/CD journey with a local build box (or six) that run their tests. In several mobile teams I worked on, for example, we had a few Mac Mini boxes with physical devices plugged in that we used for running local UI and unit tests. Eventually we migrated to a cloud-based solution, which brought us much greater stability and many new features. But moving to the cloud also meant our local hardware was obsolete. We had to rely on emulation, or opt for another cloud-based device farm for that.

With CircleCI’s runners, you can keep using your own devices to run tests on, while benefiting from everything else available in the CircleCI cloud. You can also use runners for tests that need to run on your own infrastructure, for security-critical scenarios, or when you are developing your own hardware.

With the launch of the CircleCI Free plan, features like runners are now available to everyone. In this article you will learn how to set up the CircleCI runner agent on your local machine to run tests. You will also learn how to configure your CircleCI pipeline so that the runner is invoked correctly.

Prerequisites

This tutorial assumes some experience with CircleCI and an understanding of how pipelines are configured to build and test software projects.

I have created a sample project that shows the runner configuration.

You can also find its builds in the CircleCI dashboard for the sample project.

Note : This example shows an Android application and tests, using the runner on macOS. The principles behind it can be universally applied, though: to iOS apps, custom hardware, or for runner on different infrastructure, like Docker containers, Windows, Linux machines, and more.

How runners work

A runner is a process installed on your infrastructure that communicates with CircleCI. Runners function as their own resource class. Your config.yml still defines the pipeline, jobs, and workflows, but instead of using CircleCI’s cloud-based resources like containers and virtual machines, you execute it on your own.

First, install the runner agent locally. This is a daemon process that runs in the background, checking for prompts from CircleCI. When CircleCI signals that a job must be triggered on the runner, the agent executes the workload. It also has access to the same credentials as the cloud version of CircleCI, so it runs seamlessly and can fetch the code from your repositories.

Once the job is complete, the results and any artifacts are sent back to the cloud service. Only the actual job execution is done on your infrastructure.

Implementation

Implementation consists of these processes:

Setting up the runner agent locally
Invoking the runner from your CircleCI config.yml

Setting up the runner agent

Installing the runner agent depends on which platform you are using. Refer to the setup instructions in the CircleCI docs.

For my sample application, I installed the runner agent on a Mac computer.

Note : These steps are just a brief overview and not a complete guide.

The main steps to set up the runner are:

Register a new runner in your organization
Install the runner agent, following the instructions for your platform
Start the runner daemon, and make sure it runs on system startup

Use the circleci CLI tool too register a new runner in your organization. If it is your first runner, you may need to create a namespace in that organization. You will be referring to that specific runner from the CircleCI config using that fully qualified name. In my case, that was zan_demos_namespace/mac_runner; mac_runner is the name of the runner, and zan_demos_namespace is the namespace I used.

Here is a sample command:

$ circleci runner resource-class create zmarkan-demos-namespace/local-mac-runner "Local Mac OS Runner" --generate-token

This command will also set up a new API token for this runner to use with CircleCI API in the organization. That is made possible because of the --generate-token flag. Be sure to store that token for later.

After registering the runner with CircleCI, install the runner agent following the instructions in the docs. The setup requires software like curl, sha256sum, tar, and gzip preinstalled. There may be other software required.

Set up some daemon scripts, and pass in these values:

The API token created in the previous step
The runner namespace
The name you specified

The particulars of these items vary based on the platform you are using. For example, on Mac you need to create a launchd.plist and start it with launchctl. On Linux it will be a new service that starts with systemctl.

Invoking the runner from a pipeline

Once the runner agent is in motion, you can invoke it from a job in the .circleci/config.yml file. Make sure to specify the machine executor type and the resource class with your runner’s namespace and runner name. Here is an example:

 runner-test:
    machine: true
    resource_class: zmarkan-demos-namespace/local-mac-runner
    environment:
      JAVA_HOME: /Users/zmarkan/libs/jdk-11.0.2.jdk/Contents/Home
      ANDROID_SDK_ROOT: /Users/zmarkan/Library/Android/sdk
    steps:
      - checkout
      - run:
          name: Run connected check on local runner
          command: |
            ./gradlew connectedDebugAndroidTest

In my example, I had some Android devices connected to my Mac, so the runner could access that local hardware to run tests on them.

I also needed to pass in some specific values that come out of the box when using CircleCI’s own infrastructure, like JAVA_HOME and ANDROID_SDK_ROOT. This is because they were not readily accessible in the specific shell that CircleCI runner uses. On most modern Macs the shell used is Zsh; CircleCI runs Bash by default.

The rest of the steps are straightforward and will depend on your application and what you want to execute. In my case that was checking out the code from the repo and running the tests. You could also use store_artifacts and store_test_results to process the tests and save outputs. The outputs would then be available in your CircleCI cloud dashboard.

Conclusion

In this article you have learned how to set up a local CircleCI runner for executing CircleCI cloud workloads on your local infrastructure. Runner has many use cases, from security to execution on bespoke hardware, all with the same CI/CD experience that cloud based CircleCI offers.

If you have any feedback or suggestions about topics I should cover next, reach out to me on Twitter - @zmarkan.

Build private CircleCI orbs on any organization

Zan Markan — Thu, 13 Jan 2022 10:00:00 +0000

Using CircleCI’s orbs is a great way to share CI/CD configuration across projects. Public orbs work well for wide adoption, but private orbs have been helpful for organizations needing to share common internal configuration in a secure, non-public way. Private orbs work only within the organization that publishes them.

We recently opened up private orbs access to all CircleCI customers, including those on the Free plan. This new access opens up a number of new possibilities for your pipelines.

In this article you will learn about private orbs and how to use them for your own organization. We will be using the sample orb I built, which contains a command and a job that lets you trigger CircleCI pipelines in other projects using cURL. You can find the source code of the project in this repository.

Prerequisites

This article assumes:

Working knowledge of CircleCI
Understanding of how pipelines work
CircleCI CLI installed and configured locally on your machine
You are using either a project organization or your personal organization on GitHub

Use cases for private orbs

Like orbs in general, private orbs are useful for sharing steps and tasks that are common to multiple projects.

Many organizations have their own frameworks and internal tools used across many teams. With private orbs they can share those tools easily, without opening it up to the wider world. Some use cases for private orbs include deployment to your proprietary Kubernetes clusters or other infrastructure, dealing with custom hardware, all the way to logging or common admin tasks.

Initial setup

We will use the CircleCI CLI for this task. With the CLI you will create a new orb, and register it with CircleCI so it can be referenced by other CircleCI pipelines. The CLI allows you to release new versions of your orb.

The whole orb creation process is documented in the CircleCI docs. This article will serve as a quick recap of it and explanation of the steps involved.

Create an empty git repository and note its path. I created mine under my zmarkan-demos organization, and named it api-requester-orb-sample, so the path is: git@github.com:zmarkan-demos/api-requester-orb-sample.git.

Next, create the actual orb with the CLI’s circleci orb init command, passing in a local path where you want the orb source to be located. This directory must be new; do not use one that already exists. In my case, this was:

circleci orb init ~/github/api-requester-orb-sample --private

The main difference between public and private orbs is the --private flag at the end.

The CLI asks you to either follow the guided setup, which is what I did, or to do it yourself. I recommend using the automated process is recommended as it will set up everything in the directory with an empty orb template, as well as link it to your repository, create a namespace for your orb, and create a CircleCI project for the new orb, just make sure to copy the git URL from the repository you created earlier.

Finally you need to push the local code to the remote repository -

cd ~/github/api-requester-orb-sample
git push origin master

Note: All orbs exist in a namespace within your organization. This is the same for any runner agents you have registered already.

You can also review your new private orbs in your organization settings, where they are clearly labeled as private under the type section.

Developing an orb

The orb development process depends on whether you opt for the automated guided process or follow your own. As with the setup section, the CircleCI docs cover both scenarios.

If you have followed the quick setup route, then the CLI will have generated the whole project for you from template. It has samples for a command, job, executor, and a test for it.

You can remove any part of the orb that you do not need. In my case I removed the scripts and executors, and kept the commands and jobs. Orbs are just a collection of CircleCI configuration scripts that get resolved and bundled when you run a pipeline.

We created 2 scripts:

A commands/trigger-pipeline.yml command
A jobs/trigger-circleci-pipeline.yml job

The job wraps the command and executes in the base Docker executor.

Using the trigger-pipeline command

The trigger-pipeline command has 4 parameters, and a single step, as shown in this code sample:

description: >
  This command triggers CircleCI pipeline specified.
  The job execution environment must have cURL utility available

parameters:
  repo-name:
    type: string
    description: "Repo name to trigger pipeline for"

  branch-name:
    type: string
    default: main
    description: "Branch name to trigger"

  trigger-params-json:
    type: string
    default: "{}"
    description: Pipeline trigger parameters in JSON format

  circle-token-env-var:
    type: env_var_name
    default: CIRCLE_TOKEN
    description: Environment variable containing your CircleCI API Key. Default is $CIRCLECI_API_KEY

steps:
  - run:
      name: Run cURL request
      command: |
        curl --request POST \
            --url "https://circleci.com/api/v2/project/gh/${CIRCLE_PROJECT_USERNAME}/<< parameters.repo-name >>/pipeline" \
            --header "Circle-Token: ${<< parameters.circle-token-env-var >>}" \
            --header "content-type: application/json" \
            --data '{"branch":"<< parameters.branch-name >>","parameters":<< parameters.trigger-params-json >>}"'

Unlike a full CircleCI pipeline config, this one does not require a version stanza. With only a single command source, it doesn’t have jobs or workflows either. All 4 parameters are also specific to this command, unlike the pipeline parameters available in the full CircleCI config.

The curl command in the run step hits the CircleCI API to trigger another pipeline. It is described in detail in this blog post.

Using the trigger-circleci-pipeline job

Writing a job that consumes the command we just created is straightforward. Within the same orb, you can use that command as if it were a local command you had defined in the config. In our example in jobs/trigger-circleci-pipeline.yml it looks like this:

description: >
  Job that trigger CircleCI pipeline in another project with the payload specified

docker:
  - image: cimg/base:2021.12

parameters:
  repo-name:
    type: string
    description: "Repo name to trigger pipeline for"
  branch-name:
    type: string
    default: main
    description: "Branch name to trigger"

  trigger-params-json:
    type: string
    default: "{}"
    description: Pipeline trigger parameters in JSON format

  circle-token-env-var:
    type: env_var_name
    default: CIRCLE_TOKEN
    description: Environment variable containing your CircleCI API Key. Default is $CIRCLECI_API_KEY

steps:
  - trigger-pipeline:
      repo-name: << parameters.repo-name >>
      branch-name: << parameters.branch-name >>
      trigger-params-json: << parameters.trigger-params-json >>
      circle-token-env-var: << parameters.circle-token-env-var >>

This command has the parameters, and as with all other jobs in CircleCI we need the executor: docker in our case. The steps stanza contains a single step, invoking our trigger-pipeline command from before, with any parameters we set.

Testing and deploying the orb

To build, test, and deploy our orb, the orb template project uses CircleCI itself. Within the config in .circleci/config.yml most everything is pre-made for you. All you need to do is to write tests for it. By default, it contains a job named integration-test-1, which you can modify to use your own orb. Here is a snippet that invokes the trigger-pipeline command:

jobs:
  integration-test-1:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - api-requester-orb-sample/trigger-pipeline:
          repo-name: pinging-me-softly
          branch-name: main
          trigger-params-json: '{"image-name": "cimg/base:2021.12"}'

To publish a release version you need to:

Change from the default alpha branch to either the master or main branch
Include a [semver:patch|minor|major] message in the commit

Conclusion

In this article you have learned about building private orbs for CircleCI, which are now available to all CircleCI customers. Private orbs allow you to have shared CI/CD pipeline configuration across multiple projects. These private orbs can be used by any team member in your organization, without the risk of exposing your config to the public.

The process as a whole is almost identical for publishing public orbs. Just add the --private flag in the circleci orb init command.

To read more about orb development check out the docs pages on authoring orbs.

To read about testing and verifying bash scripts in your orbs using BATS and ShellCheck, check out this docs page on orb testing methodologies.

If you have any questions or suggestions about this article, or ideas for future articles and guides, reach out to me on Twitter - @zmarkan or email me.

What developers get, out-of-the-box, from the most generous free plan anywhere

Angel Rivera — Tue, 11 Jan 2022 13:00:00 +0000

Freemium plans are a great way for companies to introduce developers to their products and offer a hands-on demonstration of the value they provide. But it can be extremely frustrating for developers when a free tier limits access to key features or doesn’t provide enough capacity to evaluate how the product performs in real-world development scenarios. Not only is this frustrating, but it also diminishes the overall developer experience, resulting in negative and sometimes inaccurate perceptions of the product.

Many companies struggle to strike the right balance between which features to include in their free tier and at what level of usage they should require a paid account. CircleCI is no exception. Until recently, our free plan just wasn’t enough to provide the best developer experience and demonstrate the value and power that CircleCI can bring to your continuous delivery and release processes. That’s why we decided to overhaul our offerings to provide the most comprehensive free plan available on the market.

In this post, I’ll discuss the newly released CircleCI Free plan, highlighting some of the most impactful changes and how they will improve the developer experience.

Free plan details

Developers continually adopt innovative strategies and concepts like continuous delivery and release management processes with the goal of building and releasing software faster and more efficiently. CircleCI enables your team to maximize development velocity by automating their software build and release practices. With our new free tier offerings, you get access to all the features and capabilities you need to make the most of your build minutes. Here’s what you get with the new free plan:

Unlimited users
30,000 credits per month—enough for up to 6,000 build minutes, depending on your compute type
Access to multiple execution environments, including Docker, Linux, Arm, and Windows, and larger resource classes
30 concurrent job runs on any of the available compute options
5 self-hosted runners to run jobs on your own machines
1 GB of network data transfer to self-hosted runners and 2 GB of data storage for saving caches and workspaces, uploading test results, and building artifacts
Docker layer caching to speed up your Docker builds
Flaky test detection on our Insights dashboard for up to 5 tests
The ability to create private orbs for sharing configuration code with other members of your team

Now that I’ve briefly described the awesome products and features included in the new Free plan, let’s take a closer look at some of the features that matter most to developers and their teams.

Larger resource classes

Release cycles are growing increasingly shorter so that teams can ship changes and new features to users as quickly as possible. For developers to meet their release cycle requirements, it’s critical that they execute highly efficient CI/CD pipelines on the changes to be released. There are many ways to optimize your CI/CD pipelines, and one of the easiest and most effective is to control the pipeline’s underlying compute node capacity via a feature known as the resource class.

The resource class feature enables teams to configure and manage the capacity of compute node resources such as CPU and RAM, ensuring that the pipeline has the appropriate horsepower to successfully and expediently complete pipeline jobs. More often than not, pipeline job resource classes are configured with insufficient resource capacities, resulting in dramatically slower runs. These slower runs incrementally extend the duration of pipeline completions, which can lead to delays in the release process. The new Free plan provides access to a wider range of resources classes, enabling teams to dial in the right resources to optimize their pipeline job performance.

30x concurrency

Beyond providing appropriate compute capacities for pipeline jobs, another feature teams can use to speed up their pipelines is concurrency, which is the ability to run multiple jobs at the same time across multiple execution environments. Under the new Free plan, you can now run up to 30 jobs concurrently, which can yield significant time savings by allowing you to avoid queuing caused by resource constraints in a given execution environment.

A great use case for concurrency within pipelines is parallel test execution. The more tests your project has, the longer it will take for them to complete on a single machine. To reduce this time, you can run tests in parallel by specifying a parallelism level in your job configuration. Your tests will then run simultaneously across multiple separate executors, allowing you to shorten the amount of time it takes to validate and ship changes to your users. For more information on parallelism and concurrency, review the documentation.

Docker layer caching

Docker is the containerization technology available within the CircleCI platform. Docker images allow teams to spin up container environments to run jobs, and many teams build their own Docker images to have custom environments to test and deploy their applications. Docker images are built from Dockerfiles, and each command in the Dockerfile produces a layer in the image. Building Docker images can be one of the most time-consuming tasks in a CI/CD workflow.

With Docker layer caching, now available on the CircleCI Free plan, you can reduce the time spent on repeated Docker builds by saving individual layers of your Docker image on every job run. The next time you run your workflow, CircleCI will retrieve any unchanged layers from the cache rather than rebuilding the entire image from scratch. This enables teams to efficiently package their apps and build related Docker images without slowing down pipeline executions. For more information about Docker layer caching, check out the documentation.

Private orbs

Orbs are reusable packages of YAML that make it easier for developers to automate processes and incorporate third-party tools in their pipelines as well as share configuration across projects. While there are many useful orbs available in our public orbs registry, teams working in higly regulated industries such as healthcare, finance, or the public sector often require higher levels of security and compliance.

With private orbs, your team gets all of the collaboration and efficiency advantages of orbs along with the increased privacy and security that comes from restricting access to authenticated users within your organization. Your team can create and publish new private orbs using the CLI tool, and authenticated users can view and manage your organization’s private orbs by visiting the Orbs page in the Organization Settings tab of the CircleCI web app.

Test Insights with flaky test detection

Automatically testing changes to your code is the foundation of continuous integration and is an essential step toward minimizing risk in your software releases. But the reality is that most tests aren’t perfect. They don’t always execute as expected and can sometimes be flaky, meaning they fail nondeterministically. With the CircleCI Insights dashboard, you can monitor test performance across multiple workflows and development branches to automatically identify tests that are slow, flaky, or fail most often.

Insights gives developers valuable visibility into test execution and performance data. Improved awareness of the health and performance of your test suite can save your team time and money by eliminating hours spent chasing unidentified bugs and increasing your confidence in the quality of your code. Plus, with Insights, you can monitor other key metrics, including credit usage, success rates, and pipeline duration, so that you can get a complete overview of your workflow performance at a glance. To learn more, visit the Insights documentation.

Conclusion

In this post, I discussed the newly released CircleCI free plan and highlighted some of the changes that will provide the biggest improvements to the developer experience. With access to the complete feature set on CircleCI’s powerful continuous integration and delivery platform, your team can implement fast, flexible, and efficient CI/CD pipelines that will drastically shorten the time from commit to deploy.

Automating development processes with continuous delivery is no longer optional for software producers who want to be responsive to their users’ needs and stay ahead of the competition. To learn more about CircleCI’s pricing and how it compares to other plans available on the market, visit our pricing page. If your team isn’t already benefitting from the time savings and confidence boost that a robust CI/CD solution can provide, sign up for a free CircleCI account and get started today.

Config best practices: Docker layer caching

Jeremy Meiss — Tue, 11 Jan 2022 10:00:00 +0000

Let’s face it: Creating the optimal CI/CD workflow is not always a simple task. In fact, writing effective and efficient configuration code is the biggest hurdle that many developers face in their DevOps journey. But you don’t need to be an expert to set up a fast, reliable testing and deployment infrastructure. With a few straightforward techniques, you can optimize your config.yml file and unleash the full potential of your CI/CD pipelines.

In this series, we go into depth on some common recommendations that our solutions engineers make during one-on-one config reviews with enterprise-level customers. Today, we are focused on Docker layer caching (DLC), a powerful technique for saving Docker layers between jobs and speeding up your workflows.

Only available until recently on the Performance and Scale plans, DLC is now included on the new CircleCI Free plan as well as on installations of CircleCI server. In this post, we’ll talk about what Docker layer caching is, why it matters, and how you can add it to your pipelines.

What is Docker layer caching?

Docker is the containerization technology used on the CircleCI platform. In both the Docker executor and the machine execution environment, you can run Docker build commands to containerize your applications for testing and deployment. With Docker layer caching, you can save individual layers of the Docker images you build so that they can be reused in subsequent pipeline runs.

Docker layer caching can save your team significant time during the build process by bypassing some or all of the image build steps. Docker images are built from Dockerfiles, and each command in the Dockerfile creates a new layer in the image. When you build a Docker image with docker build or docker compose, DLC saves the individual layers to a volume attached to the machine or remote Docker instance running the job. The next time you run the image build job, CircleCI will retrieve any unchanged layers from the cache.

If your Dockerfile stays the same between pipeline runs, the entire image will be retrieved from the cache. If you introduce a change to the Dockerfile between runs, CircleCI will retrieve all the layers up to the change, then build the rest of the image based on the new Dockerfile. The less your Dockerfile changes between runs, the faster your image will build.

When should you use Docker layer caching?

Docker containers have fundamentally changed the landscape of software development, allowing teams to build, test, and deploy applications collaboratively, securely, and consistently by packaging binaries and dependencies into a portable unit of software that eliminates the potential for environment conflicts or the “it worked on my machine” conundrum. Many teams have adopted containerization as a way to implement cloud-native development practices and shift their software architecture toward a more distributed approach.

If your application relies on containers, you may find yourself building the same image on every run of your CI/CD pipeline, which can be a serious drain on your build minutes. By enabling Docker layer caching, you can achieve a significant reduction in build times by reusing cached image layers rather than building them from scratch each time you run your builds.

Note that DLC will only reduce the time it takes to build your own Docker images with docker build, docker compose, or similar docker commands in a remote Docker environment. It does not affect the time it takes to spin up the primary Docker container. If you are running your pipelines in a Docker container but not building new images as part of your workflow, then you will not see any reduction in build times by implementing DLC.

How to add Docker layer caching to your pipelines

You can use Docker layer caching to speed up Docker image builds in both the Docker and the machine execution environments. In the Docker execution environment, any Docker images that you create are built in a secondary container called the remote Docker environment, which adds additional security to the build process and gives you access to a variety of Docker commands as well as the ability to SSH into your builds for debugging purposes. You can spin up the remote Docker environment with the setup-remote-docker key, then add Docker layer caching to the build job by setting docker_layer_caching to true:

version: 2
jobs:
  build:
    docker:
      # DLC does nothing here, its caching depends on commonality of the image layers.
      - image: cimg/node:17.3.0
        auth:
        username: mydockerhub-user
          password: $DOCKERHUB_PASSWORD
    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: true
      # DLC will explicitly cache layers here and try to avoid rebuilding.
      - run: docker build .

In this example config.yml file, you set docker as the execution environment for the build job and set up a remote Docker environment with docker_layer_caching set to true. CircleCI will cache the layers of the Docker image you build with the docker build command so that the next time you run this job, you can avoid rebuilding any unchanged layers.

The machine execution environment does not require a remote Docker environment to run docker or docker-compose commands, so you can include the docker_layer_caching: true key directly below your machine executor key:

version: 2
jobs:
  build_elixir:
    machine:
      image: ubuntu-2004:202104-01
      docker_layer_caching: true
    steps:
      - checkout
      - run:
          name: build Elixir image
          command: docker build -t circleci/elixir:example .

In this example, you set the ubuntu-2004:202104-01 Linux image as your machine executor type, then set docker_layer_caching: true to enable Docker layer caching for this environment. When you build the Elixir image with the docker build command, CircleCI will cache the individual layers to be reused in future runs.

Once you have enabled DLC for your project, any layers that are saved will be available for three days between job runs. Any cached layers that are not reused after three days will be deleted. You can create up to 50 DLC volumes per project. For more information on how to implement Docker layer caching in your projects, as well as some potential use cases and limitations, visit the DLC documentation.

Conclusion

Docker layer caching is an important part of speeding up your Docker builds on CircleCI and is now included in the CircleCI Free plan. If your team builds containerized applications, you can save valuable development time by reusing unchanged Docker layers within your projects. This not only makes your team more efficient and saves your organization money, but it also helps you deliver value to your users faster, a key differentiator in today’s software development landscape.

DLC is only one of many different optimizations you can make to your config. Other cache-based optimizations include persisting data in workspaces and caching dependencies for faster builds. For more information on dependency caching, see Config best practices: dependency caching. Stay tuned for other deep dives into available config optimizations in future installments of this series.

To get started using the most powerful CI/CD platform to build, test, and deploy your applications faster and with more confidence, sign up for a free plan today. If you are interested in an expert-level review of your CircleCI configuration, you can get a personalized, one-on-one evaluation from a dedicated support engineer by signing up for a premium support plan.

Trigger your CircleCI pipelines from a GitHub Actions workflow

Kyle TechSquidTV — Fri, 17 Dec 2021 17:30:00 +0000

If you are already a GitHub user, you may know that GitHub Actions provides you with powerful tools to increase efficiencies in your software delivery life cycle. Actions can be impactful for team collaborations and process simplification. For example, you can automate things like building a container, welcoming new users to your open source projects, managing branches, or triaging issues.

It’s important to use the right tool for the right job. Using GitHub Actions along with CircleCI allows you to automate aspects of your version control system while also benefiting from features that are only available in your CircleCI pipelines. Imagine extending your VCS workflow to include advanced features like SSH debugging, test splitting, robust CPU and RAM options and GPU and Arm support.

For example, you can use Actions for team collaboration and process simplification while accelerating your build, test, and deploy stages with CircleCI.

To make it even easier to kick off your CircleCI pipelines from any event on GitHub, our team built the Trigger CircleCI Pipeline action, which is now available in the GitHub Marketplace.

In this article, we will describe a few ways you can incorporate the Trigger CircleCI Pipeline action into your workflow to make the most of your GitHub automation credits while benefiting from CircleCI’s world-class continuous integration capabilities.

Trigger CircleCI Pipeline: A GitHub Action for triggering workflows in CircleCI

With the Trigger CircleCI Pipeline action, you can kick off a CircleCI pipeline from any event on a given branch or tag. The action passes along a set of pipeline parameters containing metadata that can be used to control the flow of your CircleCI pipelines and workflows.

One potential use case for the Trigger CircleCI Pipeline action is triggering a CircleCI pipeline whenever a pull request is opened by a developer. Triggering on a pull request allows you to test changes in a PR even if the last commit was made prior to the PR being opened (or to trigger a build and test cycle any time changes are pushed to the branch a PR was opened from).

Another use case for the Trigger CircleCI Pipeline action is triggering a build and deployment from CircleCI whenever a release is created, edited, or published in GitHub. Triggering on a release allows you to build and deploy an application—such as deploying a Dockerized microservice to a Kubernetes cluster or publishing a binary to an app store—when the repo owner specifies a new release version of the application.

You can view complete implementations of these two use cases, including templates for both your CircleCI and GitHub Actions configuration files, in the Examples directory of the Trigger CircleCI Pipeline repository. Next we’ll explore an example of using the Trigger CircleCI Pipeline Action to initiate a CircleCI workflow when a new release is published.

How to use the Trigger CircleCI Pipeline Action to trigger a pipeline from a new release

The first step in setting up GitHub Actions to work with CircleCI is to create a GitHub Actions workflow for the desired CircleCI pipeline. You can do this by adding a workflow YAML file (we’ll use main.yml) to ./.github/workflows.

In the following example, you use the on and types syntax to specify that the trigger-circleci action triggers when a new release is published. Select a custom name and id for the step to provide additional contextual metadata in your CircleCI pipeline:

on:
  release:
    types: [published]
jobs:
  trigger-circleci:
    runs-on: ubuntu-latest
    steps:
      - name: <customize name>
        id: <customize id>
        uses: circleci/trigger_circleci_pipeline@v1.0
        env:
          CCI_TOKEN: $

Next, create an encrypted secret named CCI_TOKEN containing the personal API token that will be used to trigger the pipelines. We recommend making this a machine user.

Now that you have your GitHub workflow set up, you can modify your CircleCI config file to run when your GitHub Action kicks off. Start by adding the pipeline parameter definitions to your CircleCI config. This data will be entered by the GitHub Action when triggered.

Add the following to the top of your .circleci/config.yml file. Make sure you specify version 2.1:

version: 2.1
parameters:
  GHA_Event:
    type: string
    default: ""
  GHA_Actor:
    type: string
    default: ""
  GHA_Action:
    type: string
    default: ""
  GHA_Meta:
    type: string
    default: ""

The action generates several pipeline parameters:

GHA_Actor indicates which user triggered the pipeline.
GHA_Action is the id assigned in the workflow configuration.
GHA_Event is the type of GitHub event that triggered the pipeline.
GHA_Meta is an optional additional metadata parameter that allows you to specify additional metadata using input parameters.

You can use this pipeline parameter data to run workflows conditionally. In this example, the GHA_Event parameter will be populated with the value release, and you can use a when clause in your CircleCI config to specify workflows that should run only when triggered by the GitHub workflow:

jobs:
  release:
    docker:
      - image: cimg/node:lts
    steps:
      - run: npm install
      - run: npm build
      - run: npm publish

workflows:
  release:
    when:
      equal: ["release", << pipeline.parameters.GHA_Event >>]
    jobs:
      - release

This config snippet specifies a release job that publishes an npm package and a release workflow that invokes the release job. In the when clause, you specify that the release workflow will run only when the GitHub workflow runs and populates the GHA_Event parameter with the value release.

Although this example demonstrates how to set up a release trigger, you can use the same techniques to kick off a CircleCI pipeline from any of the available GitHub events for triggering workflows. To view the complete setup for triggering CircleCI from events on a pull request, visit the example GitHub repo.

How to prevent double execution of jobs in GitHub and CircleCI

It is important to emphasize that GitHub Actions runs alongside native CircleCI integration. By default, when a repository is connected to CircleCI, if a workflow within that project’s configuration does not specify any conditionals or filters that would otherwise prevent execution, then the workflow will execute on every push event by default. This means that it is possible to accidentally run a job twice, once on the push event from CircleCI and again on other events triggered by the GitHub Action.

If you are relying on GitHub Actions to provide all of your API triggers, ensure that each of your CircleCI configuration’s workflows contains a conditional limiting its execution to only the GitHub Action trigger, as in the following example:

workflows:
  # This workflow is set to be triggered conditionally, only when
  # the GitHub Action is triggered.
  # With no other workflows, normal push events will be ignored.
  when: << pipeline.parameters.GHA_Action >>
  test:
    jobs:
      - test

Using the when clause in your workflow declaration allows you to specify that CircleCI should run the workflow only when the specified GHA_Action is triggered and not after normal push events.

Conclusion

GitHub Actions is a useful tool for automating a variety of version control processes, from checking out a Git repository at a particular version to automatically creating or merging pull requests to updating README files based on changes to your code. With the help of the Trigger CircleCI Pipeline action, you can combine the time-saving benefits of version control automation with the powerful workflow acceleration that only the fastest, most secure, and feature-rich continuous integration platform can provide.

To add the Trigger CircleCI Pipeline action to your repository, visit the GitHub Marketplace and copy the workflow syntax to your workflow.yml file. If your team is not already using CircleCI to validate and ship changes to your users quickly and confidently, sign up for a free account today.