DEV Community: Zan Markan

Setting up continuous integration with CircleCI and GitLab

Zan Markan — Wed, 07 Jun 2023 23:12:45 +0000

CircleCI supports GitLab as a version control system (VCS). In this tutorial you will learn how to set up your first CircleCI CI/CD pipeline for a project hosted on GitLab. As GitLab can be used either as a SaaS tool, as well as self-managed on-premise installation, I will cover the steps to connect it with CircleCI for both.

Prerequisites and application basics

To follow along with this tutorial, you will need:

Basic knowledge of Git commands
Git installed and accessible on your machine
A GitLab account — either self-managed or SaaS
A CircleCI account

Starter app

Our starter application is a minimal Python Flask app with a single ‘hello world’ web page, which specifies required dependencies and includes a test. You can access a publicly hosted version of the starter app on GitLab.com.

You can get the application by downloading it straight from the GitLab web interface or cloning it via Git.

For the purposes of this tutorial, you don’t need to write the application from scratch, but if you are interested in complete development instruction, you can read through the beginning of [this blog post](https://circleci.com/blog/setting-up-continuous-integration-with-github/.

GitLab SaaS vs self-managed

As mentioned above, this tutorial will teach you how to configure and run your CircleCI pipeline for a project hosted either on a self-managed or hosted version of GitLab. You can learn more about the differences between the hosted and self-managed versions in the GitLab docs.

Setting up GitLab SaaS requires you only to sign up at gitlab.com. You will also need to make sure you can push to GitLab repositories, either by setting up your SSH key or personal access token for using GitLab with either SSH or HTTPS, respectively.

If you use GitLab self-managed, you will connect to it via your installation URL. This is specific to you, so for the purposes of this tutorial, use the URL yourgitlabinstance.com as a placeholder.

Create a new project with the starter app source code. You can use the new project wizard, either at gitlab.com/projects/new or yourgitlabinstance.com/projects/new, for SaaS or self-managed respectively.

Creating a CircleCI config file

Before you set up CircleCI, you can tell it what to eventually start building.

Create a new directory .circleci in the top level of the project.

Now, create a new file config.yml in that directory (.circleci/config.yml) — this is your main configuration file for CircleCI.

Paste the following in the file:

version: 2.1
jobs:
  test:
    docker:
      - image: cimg/python:3.10.11
    steps:
      - checkout
      - restore_cache:
          key: deps1-{{ .Branch }}-{{ checksum "requirements.txt" }}
      - run:
          name: Install dependencies
          command: |
            python3 -m venv venv
            . venv/bin/activate
            pip install -r requirements.txt
      - save_cache:
          key: deps1-{{ .Branch }}-{{ checksum "requirements.txt" }}
          paths:
            - "venv"
      - run:
          name: Running tests
          command: |
            . venv/bin/activate
            python3 tests.py
      - store_artifacts:
          path: test-reports/
          destination: python_app

workflows:
  run-tests:
    jobs:
      - test

In the CircleCI config file, you define everything you need CircleCI to do. In this case, you have added a single job (test) and a workflow (run-tests) containing that job.

The test job runs in a Docker container with Python installed, checks out the code at that specific commit, downloads the dependencies required for the app (Flask, in this case), sets up a Python venv, and runs the tests in tests.py. It also caches the dependencies to make subsequent pipeline runs faster and stores the test artifacts for easy access from CircleCI. To read more about CircleCI configuration, consult the configuration reference.

Save, make a new commit, and push to your GitLab repository. Now it’s time to configure your CircleCI integration with GitLab.

Configuring CircleCI with GitLab SaaS (GitLab.com)

If you don’t have a CircleCI account yet, create one now. Head to https://circleci.com/signup/ and follow the instructions to sign up using your preferred method. If you choose GitLab, you will also be prompted to select your GitLab instance. Press Connect next to GitLab.com.

This should now prompt you to authorize CircleCI with your GitLab account, giving CircleCI access to your repositories.

Authorizing CircleCI will take you to the project creation wizard. You should see your repository on the list, and it should detect the CircleCI config you committed.

Click Create Project to start building. This has created the project in CircleCI, and what’s left is to commit and push a change to trigger your first CircleCI pipeline.

Make any change, such as an update to your README file, then commit and push. This will create your first pipeline and start building and running your test.

Seconds later it should be marked as successful. The status of your CircleCI pipeline will be automatically updated in your GitLab UI as well. Navigate to Build/Pipelines (on the left-hand side in GitLab) and the pipeline should show up there as well. You can see an example based on this tutorial in this public project.

Clicking it will show you further details of the pipeline. Clicking the button under external - button CircleCI: Workflow run tests will take you back full circle into your workflow inside CircleCI.

Connecting CircleCI to a self-managed GitLab instance

Self-managed installations, unlike the hosted SaaS versions, run on infrastructure you control. They will of course also have a different URL to access them. In this tutorial, use yourgitlabinstance.com as a placeholder for your actual instance URL. This section closely follows the GitLab integration instructions in the CircleCI docs.

Assuming you have followed the steps to get the project into GitLab and committed your first CircleCI config file, you can proceed with connecting to GitLab.

If you don’t have a CircleCI account yet, create one now. Head to https://circleci.com/signup/ and follow the instructions to sign up with your preferred method.

If you choose to sign up with GitLab, you will be prompted to select your GitLab instance. Press Connect next to GitLab self-managed.

GitLab self-managed project setup

The wizard guides you through the setup process step by step.

First enter the URL of your GitLab instance and click Verify. It will suffix it with /api/v4 to complete your instance’s API endpoint.

Create your personal GitLab access token with an API scope, paste it in the Personal Access Token field and click Verify.

Next, pass in the known_hosts value and click Verify. You can get this by running ssh-keyscan yourgitlabinstance.com in the terminal and pasting the entire output in the known_hosts field. This allows CircleCI to verify the authenticity of your GitLab instance.

Finally, select your project in the dropdown. This will likely be cci-python-app. You can also give it a different name. This will set your GitLab self-managed project up for building on CircleCI.

Trigger a pipeline by committing a change and pushing it to your GitLab repo. This will start your workflow and show it in the next few seconds.

You can also see your pipeline in the GitLab UI by navigating to CI/CD > Pipelines on the left-hand side. Note that this might look different, depending on what version of GitLab you are using.

Clicking on its status it will take you to the pipeline details, from where you can navigate back to the CircleCI UI to give you a 360 degree view of your project and your CI/CD pipeline’s workflows and jobs.

Congratulations! You have successfully configured CircleCI to start building your project on a self-managed GitLab instance.

Conclusion

In this tutorial you have learned how to begin building projects hosted on GitLab with CircleCI. We have covered both on-premise self-managed GitLab as well as the hosted SaaS version on GitLab.com.

Wishing you successful building!

CI/CD for interdepartmental teams

Zan Markan — Thu, 02 Feb 2023 18:00:00 +0000

Introduction

The topic of CI/CD is often considered the domain of DevOps, infrastructure, and development teams. I would argue this thinking misses the great opportunity that is including the entire organisation in the conversation around software delivery lifecycle (SDLC). From QA, to product managers, to of course operators, security, all the way through to the top of engineering and technology leadership. Especially as organisations increasingly rely on software, not only in tech industry but in most other industries as well, it makes sense to spread the DevOps thinking and the use of tools such as CI/CD to more folks, more teams, and more departments.

Incidentally, that’s what the DevOps movement is all about, and its natural evolution. Bringing the principles of early and continuous collaboration across departments to their entire organisations.

This article argues for the use of CI/CD across various company departments - to anyone involved in - or adjacent to - the software delivery process, and covers the principles and tools that will benefit them, and points you to how to implement this using CircleCI in your organisation.

Who this article is for

People who will benefit the most from this article are the ones implementing pipelines themselves - be it DevOps and Software engineers. While this article is not a tutorial, yet it contains ideas and snippets for implementing the topics discussed within.

As we’re talking about how CI/CD can benefit most of the organisations, this article will be approachable to people with many different backgrounds. From developers to operations teams, all the way to folks dealing with delivery and governance - delivery managers, security teams, engineering leadership.

The software delivery lifecycle from the CI/CD perspective

Software delivery lifecycle, from the perspective of the CI/CD processes works kind of like this - the developers on the team write and push code changes, have them reviewed, and then automatically tested or otherwise validated, built, and ultimately deployed to a target environment using the CI/CD system, at which point the running software begins to be monitored by the operations teams.

What technologies are used, where it is deployed, how it is monitored, and by whom, will all depend on the organisation; From Go microservices running on Kubernetes, to static websites with lambda functions, to iOS applications on the App Store, and everything in between.

But of course there are many more things we do. Software doesn’t begin in an editor, it more often begins as a story somewhere on a backlog or a ticket in a bug tracker. That’s when we actually start thinking about what needs to be built, and how it should work, and for whom (and whether it ultimately does work or not), so why not use CI/CD to do more than just build, validate, and deploy?

Expanding the SDLC for the entire organisation

A modern CI/CD platform can work beyond just building testing and deploying code, and instead it can tie together many more people and teams.

In this section we will look at some of these capabilities, and how connecting to external tools can make us focus more on the software we're building and ultimately build better software.

QA, Product, and before work gets made

QA engineers work best with constant access to latest builds to run tests on, as well as run and automate their their own tests. They are often also existing CI/CD users, who write test code and commit to the codebase.
We can create build outputs either directly in CircleCI with store_artifacts and store_test_results as in the example below, as well as of course through third party tools such as AppStore Connect, and deploying to test environments.

  jobs:
    integration-test:
      docker:
        - image: ...
      steps:
        - checkout
        ...
        - run-test ...
        - store_artifacts:
            path: tested_app_binary
        - store_test_results:
            path: test-results

Beyond the CI/CD platform itself, QA engineers also work a lot with product management and ticketing software - be it validating the requirements, logging defects, or confirming that updated products do not indeed have the defects anymore.
A CI/CD tool can integrate with a PM tool such as Jira, so that all your builds and deploys are automatically available and referenceable in it. This of course doesn’t only apply to QA engineers but also for product managers.

You do so by first connecting CircleCI with Jira, and then making sure that you label each commit with the right ticket name - like AB-234. Finally, as the last step of your deployment or build job - you can invoke the Jira orb's notify command, with which you can mark deployments and builds, as in this example:

orbs:
    jira: circleci/jira@1.3.1
workflows:
  build:
    jobs:
      - build:
          post-steps:
            - jira/notify

Notifications

You can also notify your QA engineers and product managers of new releases waiting for their verification through the tools the teams already use - like Slack.
You can install a CI/CD tool as an application in your Slack workspace, which will also let it to post on your behalf. There are many ways to use it, frpm informing you when certain workflows and jobs are like deployments completed, tests are failing, and also when your workflow is waiting on an action - such as an approval or review by a QA engineer, product manager, and also security engineer, or a delivery manager. Below is an example of a generic deployment notification:

jobs:
  deploy:
    docker:
      ...
    steps:  
      - perform_deployment
      - slack/notify:
            event: pass
            template: success_tagged_deploy_1

Besides Slack there are of course numerous other tools organisations might use, from other instant messaging products to more conventional text messages and e-mails. All of these tend to have APIs which enable similar integrations to be built.

Security, Delivery management, and approvals

We have mentioned a few new roles; Security engineers are crucial for all companies to include in their SDLC processes. In an automated CI/CD pipeline, if a security team member doesn’t approve of a deployment after reviewing all necessary documentation and performing the checks, then the deployment should not complete.

CircleCI offers two very powerful concepts for ensuring access control - approval jobs and contexts.
Approval jobs introduce a manual step in your otherwise automated pipeline, which must be confirmed by an authenticated user in order for the pipeline to proceed. This is very useful for deployments for example.
Contexts in CircleCI can be applied to individual jobs, and can be restricted to specific user groups when for example connected to GitHub.
When using both of them, we can apply contexts to the subsequent job in the workflow after the approval job, in addition to adding user groups to the context itself.
This means that if a non-authenticated person tries to approve this step, the subsequent step will fail with an unauthorized message. I have written a whole blog post on the topic of implementing access control: https://circleci.com/blog/access-control-cicd/.

You can of course combine both with what we covered previously - Jira and Slack, to flag and notify the folks required to act on a deployment when it happens.

Going a few steps further even, once the relevant team member is notified and sent a link to the relevant pipeline or job, why not give them everything they need to make a decision? We expect this it as part of reviewing pull and merge requests after all.
For every testing job, security scan, and other checks we execute these tools can usually gather reports - why not make all reports available in a single report job. That’s possible by passing reports between jobs using a workspace:

jobs:
  security_scan:
    ...
    steps:
      - ...
      - perform-security-scan
      - persist_to_workspace:
          root: workspace
          paths:
            - security-output
  generate_reports:
    ...
    steps:
      - attach_workspace:
          at: /tmp/workspace
      - store_artifacts:
          path: /tmp/workspace/security-output
      - store_artifacts:
          path: /tmp/workspace/some_other_report_output

In this way you can leverage the power of a workspace to create a single point where all the relevant reports are collated. This can be a lightweight job which can save a busy team a lot of time and effort. Besides security engineers the same steps can be taken to inform delivery managers for example, and the manual approval jobs can be chained one after another, so the review steps are taken in order.

Ops, deployment reporting, and going beyond the individual dev team

Now the software is approved and being deployed, we can notify the operations teams of a new deployment.
That means that if any defects have been published, we can trace that back to the problematic deployment, revert that feature, and understand the root cause of breakage. Depending on what tools your operations teams use to monitor the software in production, you can call that API using either an orb or directly, which lets you tie a CircleCI pipeline with a deployment event.

Finally let’s take a few steps back and realise that we’ve only been talking about a single team and the relevant stakeholders. Apart from very small companies (whose engineers are too busy to read this article anyway) that single team doesn't exist in a vacuum, but works with adjacent teams working on different parts of software, sometimes even integrating one another. So not only is there a single software development cycle, there are many.

This means that implementing the above steps for a single team helps just that - the single team, but far from the wider organisation. People responsible for working across teams, should take on that work to ensure best practices are followed throughout the organisation. This could be the DevOps team that implement the integrations with security, PM, or ops processes, or the head of engineering looking to find way for their teams of engineers to learn and adopt approaches that work from each other. Looking at insights across multiple teams you could consider the pipeline throughput, mean time to recovery, and other metrics covered in our 2022 State of Software Delivery report to understand and identify the best practices within your own organisation, and areas where you can apply these relatively small improvements for the greatest impact in most teams.

Conclusion

We’re not in the world anymore where CI/CD is the sole domain of the development teams.
The role of DevOps teams is to not only work across the Dev organisations but with the wider teams, promoting the virtues of automation and how it benefits other parts of the business. The DevOps and CI/CD engineers are in fact uniquely positioned to bring that automation to more and more people and roles within the org.

In this article we have looked at a few ways CI/CD can be used to reach more parts of the company, from sending a slack message to your security engineers telling them of a required review, to letting your QA team know that a new bug fix has been deployed and is ready for testing in their ticketing application, to helping your ops team know that a new deployment has indeed started and could potentially have an impact on the users they are seeing.

If this article has helped you in a way, I would love to know, also if you have any questions or suggestions about it, or ideas for future articles and guides, reach out to me on Twitter - @zmarkan or email me.

Continuous integration for Android projects

Zan Markan — Fri, 26 Aug 2022 19:00:00 +0000

CircleCI is popular among Android developers for several reasons: it’s quick to get started, fast to execute your builds with high parallelism, (whether native, cross- or multi-platform), and even supports running Android emulators right from CircleCI with our Android machine images.

This article will show you how to build and test Android applications for an example project on the CircleCI platform. The full source code is available on GitHub - CircleCI-Public/android-testing-circleci-examples, and you can find the working pipelines in the corresponding CircleCI project.

To get the most out of this article you should have

Working knowledge of Android development
Familiarity with Android tooling and testing frameworks
Ability to use the Gradle build system

What’s available in CircleCI for Android

We have made a few tools available for you for building Android:

Docker image
Android machine image
Android orb

Docker image

The Docker image is the main executor teams use. It runs in a Docker container and so is very fast to start, and contains everything you might need to get started building Android projects. The full list of preinstalled tools can be found in the documentation. The images also ship with variants that have Node.js, the Android NDK, and browser tools preinstalled.

Machine image

The biggest change is the new Android machine image. This is a virtual machine image, so takes a bit more time than the Docker image mentioned above, but it contains all the tools you may need to build your Android applications, including the SDK, Google Cloud tools for Firebase, Python, Node.JS, and Fastlane. The biggest change however is the support for nested virtualisation, which allows you to run Android emulators from within this machine image.

Android machine images support nested virtualization, which allow running the x86 emulators in CircleCI jobs. If you have been developing for Android since the early years of the platform, you may remember that emulators were very slow. Then x86 emulators started supporting Intel’s Hyper-V hypervisors, which lets the emulator use the resources of the host computer directly. Using resource directly like this makes the emulators able to run much, much more smoothly and quickly.

For a long time, support was confined to our local hardware. Limited support on CircleCI slowed down emulators and made it difficult to run UI tests. Nested virtualization solves this problem by adding support for fast emulation to a massively reproducible CI/CD environment, making all the benefits of that available.

Android Orb

Finally there is the CircleCI Android orb which provides easy access to both images as executors, as well as handy jobs and commands for installing SDKs, running emulators, caching, testing, and more. You can read more about what the orb offers in the orb docs at this link.

Introducing the Android CI demo project

I have created a demo project, which is a fork of Google’s own Android Testing Codelab sources. The Codelab is a simple TO-DO list manager application and contains a combination of unit tests and instrumentation tests, with and without the UI. The project is also available to view on CircleCI. If you open it in Android Studio, make sure to switch to the Project view. Project view lets you view the .circleci/config.yml file we will be reviewing.

The project has a single app module, with the default debug and release variants. The test directory contains unit tests, while the androidTest directory contains instrumentation tests. The full CircleCI configuration is included, and I will be referring to it in this article. The configuration is located in .circleci/config.yml.

Building and testing CI for the sample Android app

Our CI process must do three things:

Run unit tests inside a virtual machine
Run instrumentation tests on the emulator
Build a release version of the app

We will run unit tests and instrumentation tests side by side. If both types of tests are successful, we can assemble the release build. We will also add a condition to continue on to the release build only if new work has been commited to the “main” branch, and only following successful emulator tests on every Android version from 23 to 30. When our build meets this condition, we will have solid assurance that our application will work on many different Android devices.

Note: As I mentioned before, everything from this point on will refer to the CircleCI configuration file in .circleci/config.yml.

Using the CircleCI Android orb

The first step for creating our CI pipeline is to use the Android orb, which contains many of the steps already pre-written for you. The latest version is available in the orb docs.

After the orb is defined, the script specifies these jobs:

unit-test
android-test
release-build

One workflow, test-and-build, is also included.

version: 2.1

orbs:
  android: "2.1.2"

jobs:
  unit-test:
    ...
  android-test:
    ...
  release-build:
    ...
    # We'll get to this later

workflows:
  test-and-build:
    ...
  # We'll get to this later

Running unit tests

Our unit-test job contains an executor from the Android orb: android/android-docker. As mentioned this runs it in a Docker container which is extremely fast to start up.

In the steps stanza, there are a few things to note. First, we run android/restore-gradle-cache Gradle cache helps store our dependencies and build artifacts. After running the test command, we also run android/save-gradle-cache and android/save-build-cache to make sure subsequent builds run faster by using the cache. These all come from the Android orb, as indicated by the prefix android/.

We could also modify what gets cached by passing the find-args parameter to restore-gradle-cache. A great advanced example can be found in this open source project.

The main build step happens in the android/run-tests command, where we call ./gradlew testDebug within a test-command parameter. This command runs all unit tests for the debug build variant. As a bonus, this command comes with a default retry value to help you avoid test flakiness.

jobs:
  unit-test:
    executor:
      name: android/android-docker
      tag: 2022.08.1
    steps:
      - checkout
      - android/restore-gradle-cache
      - android/run-tests:
          test-command: ./gradlew testDebug
      - android/save-gradle-cache
      ...

Running Android tests on the emulator

Our android-test job is the one that uses the new Android emulator capability. The job uses the same Android machine executor as the build job. All subsequent jobs will use the same executor, and the cache restoration steps are also the same.

Emulator requires us to run on a machine image, so we specified android/android-machine as the executor. To improve build times, we specify the resource-class as xlarge. You can also run it on a large executor, but I have found that the lack of resources causes slower build times. I recommend that you do some experimentation on your own to find the right size executor for your project.

The next step is even shorter; we need just the android/start-emulator-and-run-tests command. This command does exactly what it says: it starts the specified emulator and runs the tests. We pass in the test-command again as the parameter (it uses the android/run-tests command). The system-image is the fully qualified Android emulator system image. We currently bundle our Android machine images back to SDK version 23. You could install more by using the sdkmanager tool.

jobs:
  android-test:
    executor:
      name: android/android-machine
      resource-class: xlarge
    steps:
      - checkout
      - android/start-emulator-and-run-tests:
          test-command: ./gradlew connectedDebugAndroidTest
          system-image: system-images;android-30;google_apis;x86
      ...

You could code all the emulator setup steps manually using the commands specified in the Android orb: create-avd, start-emulator, wait-for-emulator, and run-tests.

Storing the test results

There is more to testing than just running the tests, as we can get a lot of value from seeing exactly what has failed from right inside the CircleCI dashboard. For that we can use store_test_results functions built into the CircleCI platform, which will show our passing (or failing) builds.

The steps differ slightly between unit tests and instrumentation tests, as each of their respective Gradle tasks stores the tests in a different place. For unit tests the tests will be in app/build/test-results, and for instrumentation tests they will be in app/build/outputs/androidTest-results.

We suggest you create a new Save test results step, which uses the find utility and copies all test results files into a common location - like test-results/junit, and then store it from there. Also make sure to add the when: always parameter to the step, so the step runs regardless of whether the tests above succeed or fail.

Storing unit tests

For this project, we want the XML based results. Calling store_artifacts makes those results parseable on the CircleCI platform. You can also store the HTML output if you prefer.

...
- android/run-tests:
  test-command: ./gradlew testDebug
- run:
    name: Save test results
    command: |
        mkdir -p ~/test-results/junit/
        find . -type f -regex ".*/build/test-results/.*xml" -exec cp {} ~/test-results/junit/ \;
    when: always
- store_test_results:
    path: ~/test-results
- store_artifacts:
    path: ~/test-results/junit

Storing instrumentation tests

There are different regex command for different test types. The rest of the steps to store test results are identical to the unit tests example.

- android/start-emulator-and-run-tests:
        test-command: ./gradlew connectedDebugAndroidTest
        system-image: << parameters.system-image >>
    - run:
        name: Save test results
        command: |
          mkdir -p ~/test-results/junit/
          find . -type f -regex ".*/build/outputs/androidTest-results/.*xml" -exec cp {} ~/test-results/junit/ \;
        when: always
    - store_test_results:
        path: ~/test-results
    - store_artifacts:
        path: ~/test-results/junit

Making a release build and storing the artifacts

For this project, we need to store the application we are building. To complete this part of the process, we use the release-build job, which has two parts of note. A run step calls ./gradlew assembleRelease. Then, store_artifacts points to where the built APK is. For a full CI/CD project, you could sign and upload that APK to a beta distribution service. You could even create a release on Play Store with Fastlane. Both of those activties are out of scope for this article, though.

jobs:
  ...
  release-build:
    executor:
      name: android/android-machine
      resource-class: xlarge
    steps:
      - checkout
      - android/restore-gradle-cache
      - android/restore-build-cache
      - run:
          name: Assemble release build
          command: |
            ./gradlew assembleRelease
      - store_artifacts:
          path: app/build/outputs/apk/release

Running tests on multiple Android versions simultaneously

The Android platform evolves all the time, and each version has its own bugs and pecularities. I am sure you have encountered some of them. A good way to avoid undefined behaviour across different versions of Android is to test on as many of them as you can. The CircleCI job matrix feature makes it easier to run jobs on multiple versions.

To use the job matrix, add a parameter to the job you want to test on multiple versions. Our android-test job does that using the system-image parameter. We have also specified a default value for the parameter, so you can run the job without it.

To use the parameter, specify where you need its value by placing it within pairs of angle brackets - << >>:

 android-test:
    parameters: # this is a parameter
      system-image:
        type: string
        default: system-images;android-30;google_apis;x86
    executor:
      name: android/android-machine
      resource-class: xlarge
    steps:
      - checkout
      - android/start-emulator-and-run-tests:
          test-command: ./gradlew connectedDebugAndroidTest
          system-image: << parameters.system-image >> # parameter being used

To pass values to the parameter in the workflows, use the matrix parameter of the job we are calling.

workflows:
  jobs:
  ...
  - android-test:
      matrix:
        alias: android-test-all
        parameters:
          system-image:
            - system-images;android-30;google_apis;x86
            - system-images;android-29;google_apis;x86
            - system-images;android-28;google_apis;x86
            - system-images;android-27;google_apis;x86
            - system-images;android-26;google_apis;x86
            - system-images;android-26;google_apis;x86
            - system-images;android-26;google_apis;x86
            - system-images;android-26;google_apis;x86
      name: android-test-<<matrix.system-image>>

Making each run faster by caching

Alongside building our application, installing dependencies, and running tests, we often want to leverage cache as well. This allows us to do repeated tasks less often. Android and especially Gradle provides an excellent mechanism for this - the Gradle Build cache.

This lets you skip pre-built parts of the application.

Choosing what runs when

During development, we create many commits, and developers should be encouraged to push them to a remote repository as often as possible. Each one of these committs creates a new CI/CD build, though, and we might not need every test run on all possible devices for a single commit. On the other hand, if the main branch is our primary source of truth, we want to run as many tests on that as possible, to ensure it is always working as intended. We might want to initate the release deployment only on main, and not on any other branch.

We can use CircleCI’s requires and filters stanzas to define workflows that fit our process. Our release-build job requires the unit-test jobs. That means the release-build is put in the queue until all the unit-test jobs have passed. Only then is the release-build job run.

We can use filters to set up logical rules for running a particular job only on a specific branch or git tag. For example, we have a filter for our main branch that runs the android-test with a full matrix of emulators. In another example, the release build triggers only on main, and only when its two required jobs have run successfully.

workflows:
  test-and-build:
    jobs:
      - unit-test
      - android-test: # Commits to any branch - skip matrix of devices
          filters:
            branches:
              ignore: main
      - android-test: # Commits to main branch only - run full matrix
          matrix:
            alias: android-test-all
            parameters:
              system-image:
                - system-images;android-30;google_apis;x86
                - system-images;android-29;google_apis;x86
                - system-images;android-28;google_apis;x86
                - system-images;android-27;google_apis;x86
                - system-images;android-26;google_apis;x86
                - system-images;android-25;google_apis;x86
                - system-images;android-24;google_apis;x86
                - system-images;android-23;google_apis;x86
          name: android-test-<<matrix.system-image>>
          filters:
            branches:
              only: main 
      - release-build:
          requires:
            - unit-test
            - android-test-all
          filters:
            branches:
              only: main # Commits to main branch

Improving your Android application CI process

The steps I have described in this article are only a beginning. There are many ways you can improve the flow. Here are just a handful of ideas:

Build once, run many times

If you are an experienced Android developer you know that the connectedAndroidTest will always run the entire build process from the start. Using CircleCI, we could build the entire application and tests once, pass the artifacts down to following jobs, and simply run the tests on the emulators. This process could potentially save several build minutes per job run.

To make this happen, add three commandline steps in each emulator run: install the app, install the instrumentation app, and run the instrumentation tests. For our application, we would enter:

adb install app/build/outputs/apk/debug/app-debug.apk  
adb install app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk  
adb shell am instrument -w com.example.android.architecture.blueprints.reactive.test/androidx.test.runner.AndroidJUnitRunner

It is true that this code sends output to the the command line. That is not the tidy test output that Gradle provides. If you are motivated to make the output look better, this StackOverflow topic might help you.

Running on real devices

Emulators are one tool, but real world devices are often (sadly) much different. There is nothing that can replace a real-world device test. For that you can use a solution offering real devices in the cloud, like Sauce Labs or Firebase Test Lab.

Deploying to beta testers, and releasing the application

This build and test example project is just one part of a wider CI/CD story. You can take your CI/CD process much further. For example, you can automatically release new versions of your applications right to the Play Store. CircleCI has a handful of guides to help you. This tutorial is a great place to start. The official Fastlane documentation is another helpful resource.

Conclusion

In this article, I described building an Android application with CircleCI, and testing it. For our sample project, we used unit tests and a matrix of Android platform emulators, using the new CircleCI Android orb and machine images. We also touched on how to store and display test results, and how to orchestrate workflows to run tests will not give us the tidy test output that Gradle provides, and all output is on a portion of devices.

Do let me know if there is another Android topic you might want me to explore, either in an article or a live stream. Contact me on Twitter - @zmarkan.

Implementing access control policies in CI/CD pipelines

Zan Markan — Fri, 29 Apr 2022 12:00:00 +0000

This tutorial covers:

Protected branches and security groups in GitHub

Using contexts in CircleCI

Setting up approval jobs in CircleCI

Imagine yourself in this situation: You are a motivated and skilled DevOps or DevEx engineer. You have a plan to implement automated, complete CI/CD pipelines. You know how to do it, and you know how the extra productivity and automation will benefit your team and the whole company. But the project is never approved, because of security concerns.

Many organizations, especially those in regulated industries, have strict requirements for releasing their software, and rightfully so. Who can trigger a release, and when, under what conditions, and what specific checks are required before release can go ahead must be tightly controlled, auditable, and reversible.

These kinds of complicated processes often require a dedicated delivery manager to collect and present all relevant feature briefs, test data, security assessment, and rollback plans to a committee of decision makers in the company. That group can give the delivery manager the green light to deploy, often also on a specific schedule.

There is another approach. You can set up effective and fully automated CI/CD pipelines that include all the checks a human delivery manager and approval committee can complete. This tutorial will show you ways to create fine grained access control for your pipelines. You can use these methods for strictly regulated internal company projects for the enterprise, as well as for popular open source projects.

Prerequisites

This tutorial assumes you have experience with CI/CD pipelines, and ideally with CircleCI. To implement the steps covered you will also need admin access to your GitHub and CircleCI organizations (and accounts for both services).

If you do not have administrator access to a GitHub organization, you can create a free one in GitHub and use it with a free CircleCI account.

Access control in CI/CD pipelines in enterprise projects

For an enterprise organization it is vital that existing checks are followed and communicated well. The process often includes manual checks of supporting documentation and the potential impact to business before a release can continue. I will guide you through implementing a pipeline that executes a number of verification steps automatically, and continues with deployment to production only after the dedicated delivery manager manually approves the process.

To implement access control you will need to set up a few things:

Protected branches in GitHub
Security groups in GitHub
Contexts in CircleCI
Approval jobs in CircleCI

Protected branches

Setting up a protected branch is the first step to setting up access control. A protected branch prohibits team members from pushing code directly to that branch, and instead forces all changes to go through the pull request (PR) process. Commonly you would protect the branch you use to create releases from; main for example. You can toggle protected branches in the settings for your repo.

You can also specify rules and exceptions, such as review requirements, checks that need to pass before merging, history rules, and much more.

In my example I set up my main branch as a protected branch that requires a review and PR to pass.

Setting up security groups

If a user without the correct context privileges triggers a CircleCI pipeline (for example, with a commit), then the job that requires that context will be marked as 403 (unauthorized).

In our organization we will set up two security groups: development-leads and delivery-managers

All developers in your organization can push to their own branches and issue pull requests, but only team leaders should be required to review any new pull requests. Only when the team leader approves can changes be merged into the protected main branch. Then, when delivery managers get the green light to ship, they are the ones to trigger releases of software. In this tutorial, we will create a separate security group for them.

CircleCI will use these security groups with contexts to limit which jobs can be executed by whom, so that only the delivery managers can trigger them manually.

Using CircleCI contexts

Contexts in CircleCI serve a double role. One role limits the scope of secrets shared with a job while the other controls access.

Only a job that has a context specified to it in a workflow is able to use the context’s secrets as environment variables. Contexts are shared across the entire organization in CircleCI, so they can be reused in multiple projects. You can set them up using the organization settings menu on the left hand side of the CircleCI dashboard.

For access control, you can specify security groups to any context. For this tutorial,

we will create a context called release and specify the delivery-mgrs security group to it.

Now you can set up your pipeline. The sample project uses the following flow:

Tests
Security scans
Dev environment deployment
Approval job for the delivery manager
Production deployment

workflows:
 build-test-deploy:
   jobs:
     - test
     - security-scan:
         context: security
     - deploy-app:
         name: deploy-app-dev
         env: dev
         context: deployment-dev
         requires:
           - test
           - security-scan
         filters:
           branches:
             only:
               - main
     - approve-for-prod:
         type: approval
         requires:
           - deploy-app-dev
     - deploy-app:
         name: deploy-app-prod
         env: prod
         requires:
           - approve-for-prod
         context: release-prod

The pipeline uses contexts for environment variables like our API keys, but also uses the release-prod context. The release-prod context is gated so it can only be used by a delivery manager.

Setting up approval jobs

The final piece of the puzzle is the approval job: approve-for-prod. Technically, anyone can log in and approve this job. However, the deploy-app-prod job following it uses the release-prod context, which only accepts delivery managers’ credentials. If someone without those credentials approves it, the release job will fail with an unauthorized error, and the whole pipeline will terminate unsuccessfully.

Other helpful bits and pieces

One additional control we have set up in our workflow is the branch filter. The development deployment will occur only on the main branch because of that filter. The main branch is labelled as protected so the only way to get the code onto the main branch is by passing the automated checks and code review, potentially by a lead engineer.

You could also use dedicated QA or Security review steps that require manual approval. Build them the same way as part of the chain.

Considerations for open source projects

So far we covered the enterprise use case for access control. Most OSS projects do not have the same strict compliance requirements as enterprises. In contrast these projects need to have their contributions opened to a much wider group of people. That means allowing many more people to trigger pipelines.

Often the company that initiated and owns a popular OSS project continues to employ the core contributors. They will probably be joined by other regular contributors and maintainers that are not part of that company. And then there is everyone else - anyone who occasionally might contribute a fix or a feature.

For that scenario, you can set up three flows:

Company internal flow. Members contribute and also administer everything. Company employees are likely the only people allowed to release new versions of software. We can call them “inner core contributors”.
Semi-internal flow. These members are part of the organization and will be able to push directly to the main OSS repository. They will be able to use CircleCI for everything, including debugging jobs with SSH and triggering pipelines. Call them “outer core contributors”.
Standard flow. The rest of the contributors in the community who can issue PRs, otherwise work on their own. We will call them “community contributors”.

For community contributors. the flow is standard: make a fork, then open a pull request. You can toggle CircleCI to build pull requests that will automatically verify their changes.

You should still have a protected branch, and required checks.

For the groups of core contributors, you may want to include some secrets as well. Secrets require some more work. Also, passing secrets via forks is inherently insecure. Anyone could extract any secrets you allow to pass by modifying the pipeline in their PR. This could include API keys to certain services, or credentials to where your project is distributed.

One option is to avoid passing secrets to forks altogether. You will have to push any code after reviewing it into your own org, where it can be executed, and then merged into your default branch. There is a helpful article that explains the process.

The other option is to allow passing secrets to forks, but to prevent these secrets from being shared unless the code has been thoroughly reviewed and approved by a team member. You can set that up in the CircleCI project settings.

If you toggle secrets you will need to set up an approval job. Put any secrets in a context that will be accessed only by people who belong to that context.

Note: If you use this method, make sure that all contexts in your organization specify the security group. If you do not, a malicious actor could guess your context name in a PR that gets executed and extract environment variables from it.

With your inner and outer core of contributors, you can set up access controls in very much the same way. Maybe only the inner core can release new versions, but anyone in the outer core can merge any pull requests. Or maybe you just have one set of core maintainers who can do everything. The details are up to you!

Conclusion

In this article we have looked at ways to protect your CI/CD pipelines further by setting up access controls to cordon off parts of pipelines (and the associated credentials) to a limited scope of people. This is a versatile approach and works for both enterprise organizations as well as OSS projects.

We used security groups, protected branches, contexts, and approval jobs in CircleCI to achieve that effect, for both OSS and proprietary projects. Experiment with the best configuration for your organization and release your applications safely and securely.

Advanced pipeline orchestration with the circleback pattern

Zan Markan — Fri, 25 Feb 2022 10:00:00 +0000

This tutorial covers:

Orchestrating triggers for a dependent pipeline

Intro to the circleback pattern

Creating the config for a circleback pipeline

With multiple teams working on many projects, having a single pipeline for your software is just not enough. These projects need to be built and integrated before they can be tested and released. So how do dev teams handle this situation? Many teams approach the problem by breaking down software into smaller parts that do less, and are easier to maintain and build. This approach has resulted in the microservices architectures that are increasingly common in our industry.

The downside is that breaking software into smaller parts increases complexity. The combination of more parts and higher complexity makes it more difficult to test applications and operate them in production.

CI/CD pipelines are not an exception here. In my previous article in this series, we looked at triggering pipelines from other pipelines. That post described how to link multiple projects together in CircleCI, so that one deployment kicks off another.

In this article we will go a few steps further. I will provide an example of pipeline orchestration where the first pipeline not only triggers the other, but also waits on another pipeline to complete, before continuing. I call this the circleback pattern because it “circles back” to the second pipeline to signal when it has completed the work. Using this pattern allows for better integration testing and more complex deployment and release scenarios.

Prerequisites

This tutorial covers advanced pipeline orchestration techniques, so experience with CircleCI and DevOps is a must.
The tutorial also builds on a previous tutorial: Triggering pipelines from other pipelines.

Resources and sample code

The source code for this example is located in two repositories:

Introducing the circleback pattern

There are three steps in the circleback orchestration of these multiple pipelines:

The first pipeline waits
The second pipeline terminates
Results are retrieved from the first pipeline Another way of describing it is that the second pipeline calls or “circles back” to the first, while the first one is suspended and waiting for the signal to continue.

An alternative approach would be to keep the first pipeline running, checking periodically whether the second pipeline has finished. The main drawback of this approach is the increased cost for the running job’s continuous polling.

Depending on the status of the second pipeline, the first continues executing until it either terminates successfully or fails.

Why have pipelines wait for other projects to complete?

As mentioned in the introduction, this is an advanced technique aimed at taming the complexity that stems from working with multiple projects. It can be used by teams working on multiple interdependent projects that they do not want to put in a single repository like a monorepo. Another use would be to have a centralized repository for tests, for example in a hardware company. This technique could also be useful for integration testing of a microservices application, or for orchestrating complex deployment scenarios. There are many possibilities.

Implementing pipeline triggers

We have 2 pipelines we want to orchestrate

Pipeline A, which does the triggering
Pipeline B is triggered, and circles back to pipeline A

Pipeline B is dependent on A, and can be used to validate A.

Both pipelines need to have API keys set up and available. You can use the API key set as an environment variable (CIRCLECI_API_KEY) in the job in pipeline A, and also in pipeline B when it calls back. You can either set it in both projects, or at the organization level as a context. For this tutorial, I set it at the organization level as the circleci_api context, so that both projects can use the same API key.

Trigger the pipeline

The triggering process is explained in depth in the first part of this tutorial, Triggering pipelines from other pipelines. In this follow-up tutorial, I will cover just the important differences.

To circle back from the pipeline, pass the original pipeline’s ID to it. Then it can be retrieved and reached with the API.
You also need to store the triggered pipeline’s ID. You will need to get its result later on.

In the sample code, the parameter is called triggering-pipeline-id:

curl --request POST \
                --url https://circleci.com/api/v2/project/gh/zmarkan-demos/circleback-cicd-project-b/pipeline \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
                --data '{"branch":"main","parameters":{"triggering-pipeline-id":"<< pipeline.id >>"}}'

To store the pipeline ID, wrap your curl call in $() and assign it to the variable CREATED_PIPELINE. To extract the ID from the response body, use the jq tool, and write it to the file pipeline.txt:

CREATED_PIPELINE=$(curl --request POST \
                --url https://circleci.com/api/v2/project/gh/zmarkan-demos/semaphore_demo_project_b/pipeline \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
                --data '{"branch":"main","parameters":{"triggering-pipeline-id":"<< pipeline.id >>"}}' \
              | jq -r '.id'
              )
              echo "my created pipeline"
              echo $CREATED_PIPELINE
              mkdir ~/workspace
              echo $CREATED_PIPELINE > pipeline.txt

Now that you have the file pipeline.txt created, use persist_to_workspace to store it and use it in a subsequent job:

- persist_to_workspace:
            root: .
            paths: 
              - pipeline.txt

The whole job configuration is here:

...
jobs:
  trigger-project-b-pipeline:
      docker: 
        - image: cimg/base:2021.11
      resource_class: small
      steps:
        - run:
            name: Ping another pipeline
            command: |
              CREATED_PIPELINE=$(curl --request POST \
                --url https://circleci.com/api/v2/project/gh/zmarkan-demos/semaphore_demo_project_b/pipeline \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
                --data '{"branch":"main","parameters":{"triggering-pipeline-id":"<< pipeline.id >>"}}' \
              | jq -r '.id'
              )
              echo "my created pipeline"
              echo $CREATED_PIPELINE
              mkdir ~/workspace
              echo $CREATED_PIPELINE > pipeline.txt
        - persist_to_workspace:
            root: .
            paths: 
              - pipeline.txt  
...

Orchestrating the waits

The previous job will trigger a pipeline B, which needs to complete before it can circle back to pipeline A. You can use the approval job in CircleCI like this:

...
workflows:
  node-test-and-deploy:
    jobs:
      ...
      - trigger-project-b-pipeline:
          context: 
            - circleci-api
          requires:
            - build-and-test
          filters:
            branches:
              only: main
      - wait-for-triggered-pipeline:
          type: approval
          requires: 
            - trigger-project-b-pipeline
      - check-status-of-triggered-pipeline:
          requires:
            - wait-for-triggered-pipeline
          context:
            - circleci-api
      ...

After the job trigger-project-b-pipeline, enter the wait-for-triggered-pipeline. Because that job type is approval it will wait until someone (in this case, the API) manually approves it. (More details in the next section.) After it is approved, add a requires stanza so it continues to a subsequent job.

Both jobs that use the CircleCI API have the context specified, so the API token is available to both as an environment variable.

Circle back to pipeline A

For now we are done with pipeline A, and it is pipeline B’s time to shine. CircleCI’s approval job is a special kind of job that waits until accepted. It is commonly used to hold a pipeline in a pending state until it is approved by a human delivery lead or infosec engineer.

At this point, pipeline B knows the ID of pipeline A, so you can use the approval API to get it. You only have the ID of the pipeline being run, but not the actual job that needs approval, so you will need more than one API call:

Get all jobs in the pipeline
Find the approval job by name
Send a request to approve the job

Approving the job allows pipeline A to continue.

If the tests fail in pipeline B, then that job automatically fails. A workflow with required jobs will not continue in this case. You can get around by using post-steps in the pipeline, which always executes. The whole workflow is shown in the next sample code block.

Parameters:

parameters:
  triggering-pipeline-id:
    type: string
    default: ""

...

workflows:
  node-test-and-deploy:
    jobs:
      - build-and-test:
          post-steps:
            - approve-job-in-triggering-pipeline
          context: 
            - circleci-api

Script to perform the approval API call can be implemented like this. For this tutorial, I used a command.

...
commands:
  approve-job-in-triggering-pipeline:
    steps:
      - run:
          name: Ping CircleCI API and approve the pending job
          command: |
            echo << pipeline.parameters.triggering-pipeline-id >>
            if ! [-z "<< pipeline.parameters.triggering-pipeline-id >>"] 
            then
              workflow_id=$(curl --request GET \
                --url https://circleci.com/api/v2/pipeline/<< pipeline.parameters.triggering-pipeline-id >>/workflow \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
              | jq -r '.items[0].id')

              echo $workflow_id

              waiting_job_id=$(curl --request GET \
                --url https://circleci.com/api/v2/workflow/$workflow_id/job \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
              | jq -r '.items[] | select(.name == "wait-for-triggered-pipeline").id')

              echo $waiting_job_id

              curl --request POST \
                --url https://circleci.com/api/v2/workflow/$workflow_id/approve/$waiting_job_id \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json"

            fi
          when: always       
...

The script first checks for the existence of the triggering-pipeline-id pipeline parameter. It proceeds only if that parameter exists. The when: always line in the command makes sure that this executes regardless of termination status.

Then it makes 3 API calls:

Get workflow ID in a pipeline. There is just one workflow in that pipeline for this sample project.
Get jobs in that workflow, use jq to select the one that matches the approval job name (wait-for-triggered-pipeline), and extract the approval job’s ID.
Make the request to the approval endpoint with the waiting job ID.

For this tutorial, we are storing results like workflow ID and job ID in local bash variables, and using them in subsequent calls to the API.

Note : If you have more jobs than can be sent in a single response, you might have to handle pagination as well.

Now that you have made the approval request, pipeline B is complete, and pipeline A should be running again.

Update pipeline A with the result of B

After pipeline A has been approved, the next job in the workflow will begin. If your workflow graph requires it, pipeline A can trigger multiple jobs.

We still do not have any indication of the result of the previous workflow. To get that information, you can use the API again to get B’s status from pipeline A. An example job could look like that: check-status-of-triggered-pipeline.

First, you need to retrieve the ID of the triggered pipeline, which is pipeline B. This is the same ID that was persisted in a workspace in an earlier step. Retrieve it using cat:

 - attach_workspace:
          at: workspace
      - run:
          name: Check triggered workflow status
          command: |
            triggered_pipeline_id=$(cat workspace/pipeline.txt)

Then use the API to retrieve the workflow. Use jq to get just the status of the first item in the returned array of workflows:

created_workflow_status=$(curl --request GET \
                --url "https://circleci.com/api/v2/pipeline/${triggered_pipeline_id}/workflow" \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
              | jq -r '.items[0].status'
            )

Check that status is not success. If it is not, use exit to terminate the job with exit code -1. If the workflow is successful, it will terminate:

if [["$created_workflow_status" != "success"]]; then
              echo "Workflow not successful - ${created_workflow_status}"
              (exit -1) 
            fi

            echo "Created workflow successful"

Here is the full config for the job check-status-of-triggered-pipeline:

 check-status-of-triggered-pipeline:
    docker: 
      - image: cimg/base:2021.11
    resource_class: small 
    steps:
      - attach_workspace:
          at: workspace
      - run:
          name: Check triggered workflow status
          command: |
            triggered_pipeline_id=$(cat workspace/pipeline.txt)
            created_workflow_status=$(curl --request GET \
                --url "https://circleci.com/api/v2/pipeline/${triggered_pipeline_id}/workflow" \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
              | jq -r '.items[0].status'
            )
            echo $created_workflow_status
            if [["$created_workflow_status" != "success"]]; then
              echo "Workflow not successful - ${created_workflow_status}"
              (exit -1) 
            fi

            echo "Created workflow successful"

Conclusion

In this article we have reviewed an example of a complex pipeline orchestration pattern that I have named “circleback”. The circleback pattern creates a dependent pipeline and allows you to wait for it to terminate before completing. It involves making several API keys from both projects, the use of an approval job, and the workspace feature of CircleCI to store and pass values such as pipeline ID across jobs in a workflow. The sample projects are located in separate repositories: project A, and project B.

Testing locally with CircleCI runners

Zan Markan — Tue, 18 Jan 2022 10:00:00 +0000

In this article you will learn how to set up the CircleCI runner agent on your local machine to run tests. You will also learn how to configure your CircleCI pipeline so that the runner is invoked correctly.

Many development teams start their CI/CD journey with a local build box (or six) that run their tests. In several mobile teams I worked on, for example, we had a few Mac Mini boxes with physical devices plugged in that we used for running local UI and unit tests. Eventually we migrated to a cloud-based solution, which brought us much greater stability and many new features. But moving to the cloud also meant our local hardware was obsolete. We had to rely on emulation, or opt for another cloud-based device farm for that.

With CircleCI’s runners, you can keep using your own devices to run tests on, while benefiting from everything else available in the CircleCI cloud. You can also use runners for tests that need to run on your own infrastructure, for security-critical scenarios, or when you are developing your own hardware.

With the launch of the CircleCI Free plan, features like runners are now available to everyone. In this article you will learn how to set up the CircleCI runner agent on your local machine to run tests. You will also learn how to configure your CircleCI pipeline so that the runner is invoked correctly.

Prerequisites

This tutorial assumes some experience with CircleCI and an understanding of how pipelines are configured to build and test software projects.

I have created a sample project that shows the runner configuration.

You can also find its builds in the CircleCI dashboard for the sample project.

Note : This example shows an Android application and tests, using the runner on macOS. The principles behind it can be universally applied, though: to iOS apps, custom hardware, or for runner on different infrastructure, like Docker containers, Windows, Linux machines, and more.

How runners work

A runner is a process installed on your infrastructure that communicates with CircleCI. Runners function as their own resource class. Your config.yml still defines the pipeline, jobs, and workflows, but instead of using CircleCI’s cloud-based resources like containers and virtual machines, you execute it on your own.

First, install the runner agent locally. This is a daemon process that runs in the background, checking for prompts from CircleCI. When CircleCI signals that a job must be triggered on the runner, the agent executes the workload. It also has access to the same credentials as the cloud version of CircleCI, so it runs seamlessly and can fetch the code from your repositories.

Once the job is complete, the results and any artifacts are sent back to the cloud service. Only the actual job execution is done on your infrastructure.

Implementation

Implementation consists of these processes:

Setting up the runner agent locally
Invoking the runner from your CircleCI config.yml

Setting up the runner agent

Installing the runner agent depends on which platform you are using. Refer to the setup instructions in the CircleCI docs.

For my sample application, I installed the runner agent on a Mac computer.

Note : These steps are just a brief overview and not a complete guide.

The main steps to set up the runner are:

Register a new runner in your organization
Install the runner agent, following the instructions for your platform
Start the runner daemon, and make sure it runs on system startup

Use the circleci CLI tool too register a new runner in your organization. If it is your first runner, you may need to create a namespace in that organization. You will be referring to that specific runner from the CircleCI config using that fully qualified name. In my case, that was zan_demos_namespace/mac_runner; mac_runner is the name of the runner, and zan_demos_namespace is the namespace I used.

Here is a sample command:

$ circleci runner resource-class create zmarkan-demos-namespace/local-mac-runner "Local Mac OS Runner" --generate-token

This command will also set up a new API token for this runner to use with CircleCI API in the organization. That is made possible because of the --generate-token flag. Be sure to store that token for later.

After registering the runner with CircleCI, install the runner agent following the instructions in the docs. The setup requires software like curl, sha256sum, tar, and gzip preinstalled. There may be other software required.

Set up some daemon scripts, and pass in these values:

The API token created in the previous step
The runner namespace
The name you specified

The particulars of these items vary based on the platform you are using. For example, on Mac you need to create a launchd.plist and start it with launchctl. On Linux it will be a new service that starts with systemctl.

Invoking the runner from a pipeline

Once the runner agent is in motion, you can invoke it from a job in the .circleci/config.yml file. Make sure to specify the machine executor type and the resource class with your runner’s namespace and runner name. Here is an example:

 runner-test:
    machine: true
    resource_class: zmarkan-demos-namespace/local-mac-runner
    environment:
      JAVA_HOME: /Users/zmarkan/libs/jdk-11.0.2.jdk/Contents/Home
      ANDROID_SDK_ROOT: /Users/zmarkan/Library/Android/sdk
    steps:
      - checkout
      - run:
          name: Run connected check on local runner
          command: |
            ./gradlew connectedDebugAndroidTest

In my example, I had some Android devices connected to my Mac, so the runner could access that local hardware to run tests on them.

I also needed to pass in some specific values that come out of the box when using CircleCI’s own infrastructure, like JAVA_HOME and ANDROID_SDK_ROOT. This is because they were not readily accessible in the specific shell that CircleCI runner uses. On most modern Macs the shell used is Zsh; CircleCI runs Bash by default.

The rest of the steps are straightforward and will depend on your application and what you want to execute. In my case that was checking out the code from the repo and running the tests. You could also use store_artifacts and store_test_results to process the tests and save outputs. The outputs would then be available in your CircleCI cloud dashboard.

Conclusion

In this article you have learned how to set up a local CircleCI runner for executing CircleCI cloud workloads on your local infrastructure. Runner has many use cases, from security to execution on bespoke hardware, all with the same CI/CD experience that cloud based CircleCI offers.

If you have any feedback or suggestions about topics I should cover next, reach out to me on Twitter - @zmarkan.

Build private CircleCI orbs on any organization

Zan Markan — Thu, 13 Jan 2022 10:00:00 +0000

Using CircleCI’s orbs is a great way to share CI/CD configuration across projects. Public orbs work well for wide adoption, but private orbs have been helpful for organizations needing to share common internal configuration in a secure, non-public way. Private orbs work only within the organization that publishes them.

We recently opened up private orbs access to all CircleCI customers, including those on the Free plan. This new access opens up a number of new possibilities for your pipelines.

In this article you will learn about private orbs and how to use them for your own organization. We will be using the sample orb I built, which contains a command and a job that lets you trigger CircleCI pipelines in other projects using cURL. You can find the source code of the project in this repository.

Prerequisites

This article assumes:

Working knowledge of CircleCI
Understanding of how pipelines work
CircleCI CLI installed and configured locally on your machine
You are using either a project organization or your personal organization on GitHub

Use cases for private orbs

Like orbs in general, private orbs are useful for sharing steps and tasks that are common to multiple projects.

Many organizations have their own frameworks and internal tools used across many teams. With private orbs they can share those tools easily, without opening it up to the wider world. Some use cases for private orbs include deployment to your proprietary Kubernetes clusters or other infrastructure, dealing with custom hardware, all the way to logging or common admin tasks.

Initial setup

We will use the CircleCI CLI for this task. With the CLI you will create a new orb, and register it with CircleCI so it can be referenced by other CircleCI pipelines. The CLI allows you to release new versions of your orb.

The whole orb creation process is documented in the CircleCI docs. This article will serve as a quick recap of it and explanation of the steps involved.

Create an empty git repository and note its path. I created mine under my zmarkan-demos organization, and named it api-requester-orb-sample, so the path is: git@github.com:zmarkan-demos/api-requester-orb-sample.git.

Next, create the actual orb with the CLI’s circleci orb init command, passing in a local path where you want the orb source to be located. This directory must be new; do not use one that already exists. In my case, this was:

circleci orb init ~/github/api-requester-orb-sample --private

The main difference between public and private orbs is the --private flag at the end.

The CLI asks you to either follow the guided setup, which is what I did, or to do it yourself. I recommend using the automated process is recommended as it will set up everything in the directory with an empty orb template, as well as link it to your repository, create a namespace for your orb, and create a CircleCI project for the new orb, just make sure to copy the git URL from the repository you created earlier.

Finally you need to push the local code to the remote repository -

cd ~/github/api-requester-orb-sample
git push origin master

Note: All orbs exist in a namespace within your organization. This is the same for any runner agents you have registered already.

You can also review your new private orbs in your organization settings, where they are clearly labeled as private under the type section.

Developing an orb

The orb development process depends on whether you opt for the automated guided process or follow your own. As with the setup section, the CircleCI docs cover both scenarios.

If you have followed the quick setup route, then the CLI will have generated the whole project for you from template. It has samples for a command, job, executor, and a test for it.

You can remove any part of the orb that you do not need. In my case I removed the scripts and executors, and kept the commands and jobs. Orbs are just a collection of CircleCI configuration scripts that get resolved and bundled when you run a pipeline.

We created 2 scripts:

A commands/trigger-pipeline.yml command
A jobs/trigger-circleci-pipeline.yml job

The job wraps the command and executes in the base Docker executor.

Using the trigger-pipeline command

The trigger-pipeline command has 4 parameters, and a single step, as shown in this code sample:

description: >
  This command triggers CircleCI pipeline specified.
  The job execution environment must have cURL utility available

parameters:
  repo-name:
    type: string
    description: "Repo name to trigger pipeline for"

  branch-name:
    type: string
    default: main
    description: "Branch name to trigger"

  trigger-params-json:
    type: string
    default: "{}"
    description: Pipeline trigger parameters in JSON format

  circle-token-env-var:
    type: env_var_name
    default: CIRCLE_TOKEN
    description: Environment variable containing your CircleCI API Key. Default is $CIRCLECI_API_KEY

steps:
  - run:
      name: Run cURL request
      command: |
        curl --request POST \
            --url "https://circleci.com/api/v2/project/gh/${CIRCLE_PROJECT_USERNAME}/<< parameters.repo-name >>/pipeline" \
            --header "Circle-Token: ${<< parameters.circle-token-env-var >>}" \
            --header "content-type: application/json" \
            --data '{"branch":"<< parameters.branch-name >>","parameters":<< parameters.trigger-params-json >>}"'

Unlike a full CircleCI pipeline config, this one does not require a version stanza. With only a single command source, it doesn’t have jobs or workflows either. All 4 parameters are also specific to this command, unlike the pipeline parameters available in the full CircleCI config.

The curl command in the run step hits the CircleCI API to trigger another pipeline. It is described in detail in this blog post.

Using the trigger-circleci-pipeline job

Writing a job that consumes the command we just created is straightforward. Within the same orb, you can use that command as if it were a local command you had defined in the config. In our example in jobs/trigger-circleci-pipeline.yml it looks like this:

description: >
  Job that trigger CircleCI pipeline in another project with the payload specified

docker:
  - image: cimg/base:2021.12

parameters:
  repo-name:
    type: string
    description: "Repo name to trigger pipeline for"
  branch-name:
    type: string
    default: main
    description: "Branch name to trigger"

  trigger-params-json:
    type: string
    default: "{}"
    description: Pipeline trigger parameters in JSON format

  circle-token-env-var:
    type: env_var_name
    default: CIRCLE_TOKEN
    description: Environment variable containing your CircleCI API Key. Default is $CIRCLECI_API_KEY

steps:
  - trigger-pipeline:
      repo-name: << parameters.repo-name >>
      branch-name: << parameters.branch-name >>
      trigger-params-json: << parameters.trigger-params-json >>
      circle-token-env-var: << parameters.circle-token-env-var >>

This command has the parameters, and as with all other jobs in CircleCI we need the executor: docker in our case. The steps stanza contains a single step, invoking our trigger-pipeline command from before, with any parameters we set.

Testing and deploying the orb

To build, test, and deploy our orb, the orb template project uses CircleCI itself. Within the config in .circleci/config.yml most everything is pre-made for you. All you need to do is to write tests for it. By default, it contains a job named integration-test-1, which you can modify to use your own orb. Here is a snippet that invokes the trigger-pipeline command:

jobs:
  integration-test-1:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - api-requester-orb-sample/trigger-pipeline:
          repo-name: pinging-me-softly
          branch-name: main
          trigger-params-json: '{"image-name": "cimg/base:2021.12"}'

To publish a release version you need to:

Change from the default alpha branch to either the master or main branch
Include a [semver:patch|minor|major] message in the commit

Conclusion

In this article you have learned about building private orbs for CircleCI, which are now available to all CircleCI customers. Private orbs allow you to have shared CI/CD pipeline configuration across multiple projects. These private orbs can be used by any team member in your organization, without the risk of exposing your config to the public.

The process as a whole is almost identical for publishing public orbs. Just add the --private flag in the circleci orb init command.

To read more about orb development check out the docs pages on authoring orbs.

To read about testing and verifying bash scripts in your orbs using BATS and ShellCheck, check out this docs page on orb testing methodologies.

If you have any questions or suggestions about this article, or ideas for future articles and guides, reach out to me on Twitter - @zmarkan or email me.

Getting started with scheduled pipelines

Zan Markan — Tue, 14 Dec 2021 14:00:00 +0000

CircleCI’s scheduled pipelines let you run pipelines at regular intervals; hourly, daily, or weekly. If you have used scheduled workflows, you will find that replacing them with scheduled pipelines gives you much more power, control, and flexibility. In this tutorial, I will guide you through how scheduled pipelines work, describe some of their cool use cases, and show you how to get started setting up scheduled pipelines for your team. I will demonstrate using both the API and the UI and how you can set up scheduled pipelines from scratch or migrate to them from scheduled workflows.

Prerequisites

Here is what you will need to follow the tutorial:

CircleCI account with a project configured
Some Node.JS knowledge

You can use the sample project as you go through the steps.

Scheduling regular builds and tests

There are many reasons to schedule CI/CD work instead of just executing when you push code to a repository. The first and most obvious reason to schedule is to have regular builds of software. Often folks outside of the development team (product managers, QA engineers, and other stakeholders) need access to the latest build of software you and your team are working on. Of course, you can always trigger a build manually on demand, but that can be distracting for you and other developers on the team. It is so much easier to automate this process and point all the stakeholders to where they can find the newest build. This applies to everything from web apps and mobile apps, to backend services, libraries, and anything in between.

Scheduled builds automatically build software at night or off-hours when there is no development going on. These nightly builds (as they are sometimes called) take the latest revision off your main branch and produce a staging or beta build of the software.

You can use scheduled pipelines to run the entire test suite so the nightly build also verifies that your software works and is ready and waiting when you start your next working day. Scheduling allows you to run those expensive and time-consuming functional or integration tests you do not want to run on every commit. The build does not need to be run nightly. You can run it at a cadence that suits your team, every few hours, or even several times per hour.

Scheduling other kinds of work

But why stop at builds? You might also schedule work that is not necessarily a build but needs to happen regularly, like a batch process, database backups, or restarting services. If it can be added to a script on a machine that has access to your environments, it can be done in a CI/CD system.

Legacy way of scheduling — scheduled workflows

The ability to schedule work isn’t exactly new to CircleCI. Development teams have been able to include scheduling as part of the configuration, with schedules defined using cron syntax on the workflow level. There are a few downsides to this approach, though:

It requires development work to make any changes to the schedule, or even to review schedules already in place.
In CircleCI, pipelines are triggered, not workflows.
Fine-tuning permissions was difficult because it was not always clear who scheduled the work that was triggered.

How scheduled pipelines improve developer experience

Here are some of the benefits of scheduled pipelines:

You can schedule entire pipelines, including any pipeline parameters you want to pass.
With scheduling handled outside the configuration file, you can query and set schedules using both the API and the UI. This ability provides more flexibility with who manages scheduling and execution.
Scheduled pipelines work with contexts. Contexts give you fine-grained control of who has access to perform and schedule certain jobs. You can gate your deployment credentials to only the engineers with sufficient permissions, so no one else can set up those schedules. Contexts can also be used with dynamic configuration to unlock even more flexibility for your CI/CD setup.

Now that we have covered some basic facts about scheduled pipelines, we can implement one.

Implementing a scheduled pipeline

First, we need a pipeline to schedule. Luckily, I have a project that had used scheduled workflows. It is an open source Android library project that runs nightly deployments, so it is ideal for scheduling use cases. I recommend that you fork the project on GitHub and set it up as a project on CircleCI.

For our example schedule, we want to run a build every night and deploy it to the Sonatype snapshots repository. This build makes the repository available for anyone to use and get the freshest code.

There is a workflow already defined for it in our .circleci/config.yml - nightly-snapshot:

  parameters:
    run-schedule:
        type: boolean
        default: false

  nightly-snapshot:
    when: << pipeline.parameters.run-schedule >>
    jobs:
      - android/run-ui-tests:
          name: build-and-test
          system-image: system-images;android-23;google_apis;x86
          test-command: ./gradlew assemble sample:connectedDebugAndroidTest
      - deploy-to-sonatype:
          name: Deploy Snapshot to Sonatype
          requires:
            - build-and-test

The workflow has two jobs: one to run the tests and another to deploy the snapshot. There is no scheduling logic in the pipeline itself, apart from a when statement that checks for the run-schedule pipeline parameter. Our scheduler will set the parameter when triggering the pipeline.

Using the API to schedule pipelines

To get started with scheduling using the API, you need the API token. To get the token, log in to CircleCI and click your avatar in the bottom left corner. Clicking your avatar opens User Settings. Navigate to Personal API Tokens, create a new token, and save it somewhere safe. In the sample project there is a build-scheduling directory, with a file called .env.sample. You can copy that file to .env and replace the placeholder token with yours. You should do the same with other parts of the .env file: PROJECT_ID and ORG_NAME.

CIRCLECI_TOKEN=YOUR_CIRCLECI_TOKEN
PROJECT_ID=YOUR_PROJECT_ID
ORG_NAME=YOUR_VCS_USERNAME
VCS_TYPE=github

With environment variables set, we can make the API calls. Scheduled pipelines use the CircleCI API v2. To post a new schedule, you need to make a POST request to the endpoint. The endpoint consists of your CircleCI project, your username, and your VCS provider. Here is an example: https://circleci.com/api/v2/project/github/zmarkan/android-espresso-scrollablescroll/schedule

This example POST call from the setup_nightly_build.js script uses Axios JavaScript library:

const token = process.env.CIRCLECI_TOKEN

axios.post("https://circleci.com/api/v2/project/github/zmarkan/android-espresso-scrollablescroll/schedule", {
    name: "Nightly build",
    description: "Builds and pushes a new build to Sonatype snapshots every night. Like clockwork.",
        "attribution-actor": "system",
        parameters: {
          branch: "main",
          "run-schedule": true
        },
        timetable: {
            per_hour: 1,
            hours_of_day: [1],
            days_of_week: ["TUE", "WED", "THU", "FRI", "SAT"]
        }
    },{
        headers: { 'circle-token': token }
    }
)

The body payload contains the schedule name, which must be unique, and an optional description. It includes an attribution actor, which can be either system for a neutral actor or current, which takes your current user’s permissions (as per the token you use). The payload also includes parameters like which branch to use when triggering the pipeline and any other parameters you have set up. This tutorial uses run-schedule in the config file. The last part of the body is the timetable, which is where we define when and how frequently to run our scheduled pipelines. The fields to use here are per_hour, hours_of_day, and days_of_week. Note that this does not take a cron expression, which makes it more easily parsable by humans reasoning with the API.

In the headers, we will pass a circle-token using the token we generated earlier in the CircleCI UI.

In the timetable, we set everything to run at 1:00 am (UTC), on Tuesday to Saturday, the night after the work has finished. We do not need to run on Sunday and Monday, because in our example project, there is no one working over the weekend. The codebase will not change on those days.

In addition to the POST method, the API takes exposes other methods such as GET, DELETE, and PUT for retrieving, deleting, and updating schedules. There are samples for get_schedules.js and delete_schedule.js in the repository.

Using the GUI

Instead of using the API, you can set up scheduled pipelines from right in the CircleCI dashboard. From your project in CircleCI, go to Project Settings, and select Triggers from the menu on the left.

Click Add Scheduled Trigger to open the page where you can set up a new scheduled pipeline. The form has the same options as the API: trigger name, days of the week, start time, parameters, attributed user, and the others.

Click Save Trigger to activate it. The trigger will be ready to start scheduling your pipelines at the dates and times you specified.

Migrating from scheduled workflows

So far, we have explored setting up the pipelines and reviewing them using both the API and the GUI. Now we can focus on migrating your existing scheduled workflows to the more advantageous scheduled pipelines.

This example shows a scheduled workflow where everything is defined in the config file. It includes the trigger configuration to the workflow definition, passing in the schedule with the cron expression:

nightly-snapshot:
    triggers: #use the triggers key to indicate a scheduled build
      - schedule:
          cron: "0 0 * * *" # use cron syntax to set the schedule
          filters:
            branches:
              only:
                - main
    jobs:
      - android/run-ui-tests:
          name: build-and-test
          system-image: system-images;android-23;google_apis;x86
          test-command: ./gradlew assemble sample:connectedDebugAndroidTest
      - deploy-to-sonatype:
          name: Deploy Snapshot to Sonatype
          requires:
            - build-and-test

In our case, this is run at midnight every day, and we want to trigger it only on the main branch.

To migrate there are a few steps to complete:

Trigger the existing nightly-snapshot workflow in the scheduled pipeline.
Introduce a new pipeline variable called run-schedule, as we did in the first example.
For all workflows, add when expressions that indicate to run them when run-schedule is true and not to run other workflows unless run-schedule is false.

parameters:
  run-schedule:
    type: boolean
    default: false

workflows:
  build-test-deploy:
    when:
      not: << pipeline.parameters.run-schedule >>
    jobs:
      ...

  nightly-snapshot:
    when: << pipeline.parameters.run-schedule >>
    jobs:
      ...

The rest of the process is the same as when setting up from scratch:

Resolve the cron expression; you can use an online tool like this.
Set up the schedule using either the API or the GUI with the new timetable configuration, and the run-schedule pipeline parameter.

Conclusion

Scheduled pipelines are a versatile component of your CI/CD toolkit, allowing you to:

Trigger your builds on a recurring basis
Utilize pipeline parameters
Fine-tune control of who can set up pipelines
Clearly define the permissions needed when they run using CircleCI contexts

In this tutorial, you have learned about how scheduled pipelines work in CircleCI and how to set them up, either from scratch or porting from scheduled workflows. We also covered how to use the API and the web UI, and reviewed some use cases for them to get you started.

Let us know how you and your team fare with scheduled pipelines, porting your scheduled workflows, or if you would like to see any additions to the UI or API features.

If you have any feedback or suggestions about topics I should cover next, reach out to me on Twitter - @zmarkan.

Manage complex development projects by triggering pipelines from other pipelines

Zan Markan — Fri, 10 Dec 2021 10:00:00 +0000

It is no secret that software development is becoming an increasingly complex process. The individual elements of software like apps, libraries, and services are interconnected and dependent on many other elements. Development teams deal with a whole ecosystem of services that they develop, maintain, or depend on, which in turn are dependent on other software ecosystems, maintained by separate teams. Maintaining this ecosystem is as complex as you might imagine. Making sure the whole system works, and keeps working as intended, is a hugely challenging effort.

Triggering pipelines from other pipelines is one of many techniques that can help you manage complex connected and interdependent development projects. In this tutorial, I will cover some use cases for using this techniques and why it can be helpful. You can follow along as I present an example of how to implement a pipeline-to-pipeline trigger using CURL.

Prerequisites

This intermediate to advanced tutorial requires some understanding of CircleCI and its pipelines. It will be especially useful if you use CircleCI and have some projects that have already been configured. Everything covered in this article will work on all CircleCI plans.

Use cases for pipeline-to-pipeline triggers

There are several reasons to consider triggering new pipelines from existing pipelines. Microservices orchestration is a prime example. It is a common pattern to have microservices in each respective version control repository, each of which requires a separate CircleCI pipeline configuration. As one service gets deployed, you could redeploy others that depend on this new version. Or you could trigger an expansive suite of integration tests that run independently of the wider test and deployment flow.

If you are not using microservices, you can use pipeline-to-pipeline triggers to publish and test libraries that are integrated with downstream consumers.

Implementing pipeline triggers

One workflow for triggering pipelines follows these steps:

The first pipeline is executed, makes a build, runs some tests, and triggers a deployment.
After that process completes successfully, the first pipeline makes an API call to CircleCI to trigger another pipeline in a different project.
The second pipeline executes a build, runs tests, and triggers a deployment.

In this scenario, the first pipeline can also pass some pipeline parameters like the name and version of what was deployed, so that the newly triggered pipeline has some context.

Coordinating separate services using pipeline triggers

To orchestrate two separate services, you can invoke the CircleCI API from a single job. You will need to know the name of the project with the pipeline you want to trigger from that job. To address a pipeline with an API, you will need the organization name (zmarkan in my case) and the project name (pinging-me-softly because I have an odd sense of humor).

Next, get a personal API token. You must have push access to both projects for this to work. You can store that API token in this project’s environment variable. For increased security, store it in a context. In my example, I used a context called circleci-api, with a CIRCLE_API_TOKEN name for that environment variable.

To trigger a pipeline, you will need to send a POST request to the pipeline’s endpoint of the project. This is the URL I used for this tutorial:

https://circleci.com/api/v2/project/gh/zmarkan/pinging-me-softly/pipeline

The payload must contain details of the branch or tag you want to trigger the pipeline with, along with any pipeline parameters you want to pass to it.

Here is an example job:

  trigger-new-pipeline:
      docker: 
        - image: cimg/base:2021.11
      resource_class: small
      steps:
        - run:
            name: Ping another pipeline
            command: |
              curl --request POST \
                --url https://circleci.com/api/v2/project/gh/zmarkan/pinging-me-softly/pipeline \
                --header "Circle-Token: $CIRCLECI_API_KEY" \
                --header "content-type: application/json" \
                --data '{"branch":"main","parameters":{"image-name":"'$DOCKER_LOGIN"/"$CIRCLE_PROJECT_REPONAME":"0.1.<< pipeline.number >>'"}}'

The job uses the cimg/base image and the small resource class, which is more than sufficient for sending a single HTTP request. The only dependency is on the CURL tool which ships with all CircleCI images, including cimg/base.

To trigger the pipeline, there is a single command to run: curl. The API key is passed in the Circle-Token header, passing in the environment variable that contains the token. The payload body is a JSON object, which contains a branch and parameters:

  { 
    "branch":"main",
    "parameters": {
        "image-name":"'$DOCKER_LOGIN"/"$CIRCLE_PROJECT_REPONAME":"0.1.<< pipeline/ number>>'"
    }
  }

The value of the image-name parameter is a string constructed of three variables:

$DOCKER_LOGIN is an environment variable set in this project or in context.
$CIRCLE_PROJECT_REPONAME is a built-in environment variable that specifies this project in CircleCI.
pipeline.number is another built-in environment variable.

As a whole, this value provides the name of the Docker image we built in the previous step. I used zmarkan/demo-full:0.1.119 in this example.

Handling the pipeline trigger

The pipeline trigger is handled for you automatically by CircleCI, the same way it handles a new git commit in your repository. By default, all your workflows will run, and it will take the last commit of a branch or tag that you specified in the API call. For this tutorial, I specified main.

You can handle any parameters you pass (image-name in my case) in your jobs and workflows. Just make sure to declare them in the parameters section of the config:

  version: 2.1

  parameters:
    image-name:
      type: string
      default: "Not a real image name"

  jobs:
    print-image-name:
      docker: 
        - image: cimg/base:2021.11
      steps:
        - checkout
        - run:
            name: Echo Image name
            command: echo << pipeline.parameters.image-name >>

  workflows:
    run-workflow:
      jobs:
        - print-image-name

This pipeline example has a single workflow. It prints out the name of the image we built in the previous pipeline and passed via the API to this pipeline.

Conclusion

In this article I have demonstrated how to use the CircleCI API to trigger another pipeline when the first project’s pipeline is complete. It is a straightforward process that requires a single API call from a running job, and is available to all CircleCI users and organizations.

This technique opens up new possibilities for orchestrating complex workflows, including microservices architecture, and decoupling expensive integration testing work from build and deploy scenarios. There are many, many more possibilities for you and your team to experiment with.

If you have any questions or suggestions about this article, or ideas for future articles and guides, reach out to me on Twitter - @zmarkan or email me.

Building Kotlin Multiplatform projects in a CI/CD pipeline

Zan Markan — Wed, 06 Oct 2021 12:00:00 +0000

Kotlin is one of the most versatile programming languages available, in large part because of the Kotlin team’s focus on bringing it to as many platforms as possible. It is the primary language for developing Android applications and is popular for JVM backends. Kotlin also features targets for native binary compilation with Kotlin/Native, and for web through Kotlin/JS. One of its most promising features is the ability to target multiple platforms it compiles to.

This is where Kotlin Multiplatform Mobile (KMM) comes in: it lets you write and reuse the same Kotlin code across Android and iOS applications. As mobile clients often aim for feature parity, this is a clever approach to avoid duplicating work, and we all know how developers love avoiding duplicate work. Trust me, I’m a developer and I know how true this is.

Anyway, back to Kotlin Multiplatform. This is a different approach from cross-platform tools such as Flutter and React Native, which provide unified UI on top of everything. KMM lets you write common business logic while keeping the user interface and experience firmly in the domain of their native platforms. In many cases, that approach is preferable to application users, and therefore your customers.

This article will show you how to get started building KMM projects in a CI/CD pipeline, and include that in your team's development workflow.

Notes for following along

This article assumes understanding of Kotlin, as well as familiarity with at least one of Android or iOS, ideally both platforms. This article does not teach you KMM. For that, review one of existing tutorials on the Kotlin’s site, or the Readme from the sample application.

Also note: KMM is currently in Alpha version. The technology and the APIs are constantly changing and evolving and we are doing our best to keep the samples working and up to date, yet it should come with no guarantees beyond the API versions in the samples.

Building multiplatform projects in a CI/CD pipeline

To build a multiplatform project in the context of CI/CD think of it as building distinct projects for each individual target platform.
Our project is based on Kotlin's own Multiplatform sample - a RSS reader application. It contains 2 apps, as well as a shared Kotlin codebase in the form of a Kotlin library. The shared library stays in Kotlin for compilation on Android, and compiles down to native code to run on ARM64 for iOS targets.

Android app code is located in androidApp, and written in a familiar Kotlin codebase. The iOS app is written in Swift, and located in iosApp. To understand how KMM makes everything work, check out the official KMM documentation documentation and the sample project on Kotlin's GitHub repo that this project is based off.

Building a basic pipeline

The simplest CI/CD pipeline for KMM projects contains 2 jobs in a single workflow, that execute the build for each respective platform. Jobs in CircleCI are a succession of commands or steps, which execute in a predetermined environment. This environment is the key to building the app for different platforms. For iOS, we need to build it on Mac hardware, passing in macos for executor. Android is more permissive, but we can still pick a pre-built environment that contains all the SDKs and everything else built in. For this tutorial, we will use the android Docker image as provided by the android orb.

The jobs themselves are straightforward and not really relevant for this tutorial: check out the code, compile, maybe run some tests, build, maybe even deploy. As part of this guide we will not focus on the jobs. Instead we will spend some time setting up the environment and workflows for the build. If you want to learn more about setting up the build and test job for Android you can read about it in this post. If you want to learn more about setting up the build and tests on iOS, you can read more in this tutorial in the CircleCI documentation.

version: 2.1
orbs:
  android: circleci/android@1.0.3
jobs:
  build-android:
    executor: android/android
    steps:
      - checkout
      - android/restore-build-cache
      - android/restore-gradle-cache
      - run: ./gradlew androidApp:assembleDebug
      - android/save-gradle-cache
      - android/save-build-cache
      - store_artifacts:
          path: androidApp/build/outputs/apk/debug
  build-ios:
    macos:
      xcode: 12.4.0
    steps:
      - checkout
      - run:
          name: Allow proper XCode dependency resolution
          command: |
            sudo defaults write com.apple.dt.Xcode IDEPackageSupportUseBuiltinSCM YES
            rm ~/.ssh/id_rsa || true
            for ip in $(dig @8.8.8.8 bitbucket.org +short); do ssh-keyscan bitbucket.org,$ip; ssh-keyscan $ip; done 2>/dev/null >> ~/.ssh/known_hosts || true
            for ip in $(dig @8.8.8.8 github.com +short); do ssh-keyscan github.com,$ip; ssh-keyscan $ip; done 2>/dev/null >> ~/.ssh/known_hosts || true
      - run:
          name: Install Gem dependencies
          command: |
            cd iosApp
            bundle install
      - run:
          name: Fastlane Tests
          command: |
            cd iosApp
            fastlane scan
      - store_artifacts:
          path: iosApp/fastlane/test_output
      - store_test_results:
          path: iosApp/fastlane/test_output/report.junit
workflows:
  build-all:
    jobs:
      - build-android
      - build-ios

Building the Android app

The best way to start building Android apps is by using the Android orb. This gives you a few built-in jobs and commands that make it easier to build. For this exercize, we are installing Gradle dependencies, and running a Gradle command to run the unit tests.

The main difference Kotlin Multiplatform brings is that the app is not the top level project. Instead, it is located in the androidApp module. This module has a dependency on shared, so we need to invoke Gradle commands with the androidApp: prefix.

orbs:
  android: circleci/android@1.0.3
jobs:
  build-android:
    executor: android/android
    steps:
      - checkout
      - android/restore-build-cache
      - android/restore-gradle-cache
      - run: ./gradlew androidApp:assembleDebug
      - android/save-gradle-cache
      - android/save-build-cache
      - store_artifacts:
          path: androidApp/build/outputs/apk/debug

Building the iOS app

As we mentioned above, building iOS apps requires Mac hardware with XCode installed. For the build-ios job, we will use macos executor where we pass the xcode version as parameter. In our case it is: xcode: 12.4.0.

The rest of the steps are common, regardless of whether you use CocoaPods or Swift Package Manager. You will usually install some dependencies, and then you can build the app and run the tests using Fastlane. For this tutorial, we will just run some tests.

The shared Kotlin code is automatically built as a framework when we initiate a build with Fastlane, so there is nothing left to do here.

jobs:
  ...
 build-ios:
    macos:
      xcode: 12.4.0
    steps:
      - checkout
      - run:
          name: Allow proper XCode dependency resolution
          command: |
            sudo defaults write com.apple.dt.Xcode IDEPackageSupportUseBuiltinSCM YES
            rm ~/.ssh/id_rsa || true
            for ip in $(dig @8.8.8.8 bitbucket.org +short); do ssh-keyscan bitbucket.org,$ip; ssh-keyscan $ip; done 2>/dev/null >> ~/.ssh/known_hosts || true
            for ip in $(dig @8.8.8.8 github.com +short); do ssh-keyscan github.com,$ip; ssh-keyscan $ip; done 2>/dev/null >> ~/.ssh/known_hosts || true
      - run:
          name: Install Gem dependencies
          command: |
            cd iosApp
            bundle install
      - run:
          name: Fastlane Tests
          command: |
            cd iosApp
            fastlane scan
      - store_artifacts:
          path: iosApp/fastlane/test_output
      - store_test_results:
          path: iosApp/fastlane/test_output/report.junit

Creating an advanced pipeline with dynamic config

This sample pipeline works well to build and test both target platforms of your KMM app every time someone commits to the repository. But we can do better. The reality of mobile development is that we have specialists focusing their work on their respective platforms. There are the folks who like and use Android, love crafting great UX and experiences for Android, and know the platform inside and out. On the other hand, there are developers who focus on iOS in the same way. There is always some cross over working on the common codebase. There will still be individuals who specialize on either one or the other platform on most teams.

Development progress usually takes shape in one of these configurations:

Android frontend codebase only
iOS frontend codebase only
Shared KMM codebase that is consumed by both frontend codebases

For the first two scenarios, building only for the platform being worked on makes much more sense, and saves time, credits, and efficiency.

To optimize this flow we can use CircleCI’s dynamic config feature, which helps us orchestrate projects like this much more efficiently. Dynamic config lets you build only the parts of the app that you have changed.

Dynamic config works through a setup workflow, which in the first step evaluates the codebase, detects what has changed (more on this a bit later), and only then runs the real workflow, the one that is building the portion of the app that is relevant.

The setup workflow still uses the config.yml that CircleCI users are probably familiar with. The workflow that gets executed uses a path filtering orb, which detects the changes compared to a specified branch. Here is the full config.

version: 2.1
setup: true
orbs:
  # the path-filtering orb is required to continue a pipeline based on
  # the path of an updated fileset
  path-filtering: circleci/path-filtering@0.0.2
workflows:
  select-for-build:
    jobs:
      - path-filtering/filter:
          base-revision: master
          config-path: .circleci/continue-config.yml
          mapping: |
            shared/.*|^(?!shared/.*|iosApp/.*|androidApp/.*).*  build-all     true
            androidApp/.*                                       build-android true
            iosApp/.*                                           build-ios     true

Here is a break down of this config:

setup:true indicates to CircleCI that we are using dynamic configuration, and that this is the first part of the pipeline to run. The setup workflow is this configuration’s only workflow. It has a single job, path-filtering/filter, which is from the path-filtering orb. The job takes a few parameters:

base-revision indicates the branch to compare against
mapping determines which paths to compare (more about this later)
The config-path points to continue-config.yml, which is the next config to evaluate after this setup workflow. In our case we are pointing to a static file, but we could also construct this new configuration file programmatically beforehand.

To use mapping, you define the paths of the project to compare against the codebase in the base revision. There is one path per line. Then, set a pipeline parameter to pass to the subsequent pipeline, and its value. All these are separated by whitespace.

Lines 2 and 3 are fairly straightforward:

androidApp/.* build-android  true

In this case, the path we are interested in is all files and subdirectories in androidApp/ directory. That is where the codebase for the Android-specific frontend is. The pipeline parameter, build-android, is set to true.

The first matching line however is more complex because of the much longer regex matcher:

shared/.*|^(?!shared/.*|iosApp/.*|androidApp/.*).*  build-all  true

It matches paths in both shared/ directory, which is where our Kotlin multiplatform code is located, as well as anything that is not in shared/, iosApp/, or androidApp/ directories, which includes any other top level files. We have set the pipeline parameter build-all to true, which will trigger builds of the apps for both platform.

As specified in continue-config.yml, after the mapping has evaluated all changes and set all relevant pipeline parameters, CircleCI stops this job, and kicks off the second part of this dynamic workflow. Here is the complete file:

version: 2.1
orbs:
  android: circleci/android@1.0.3
parameters:
  build-all:
    type: boolean
    default: false
  build-android:
    type: boolean
    default: false
  build-ios:
    type: boolean
    default: false
jobs:
  build-android:
    executor: android/android
    steps:
      - checkout
      - android/restore-build-cache
      - android/restore-gradle-cache
      - run:
          name: Build Android app
          command: ./gradlew androidApp:assembleDebug
      - android/save-gradle-cache
      - android/save-build-cache
      - store_artifacts:
          path: androidApp/build/outputs/apk/debug
  build-ios:
    macos:
      xcode: 12.4.0
    steps:
      - checkout
      - run:
          name: Allow proper XCode dependency resolution
          command: |
            sudo defaults write com.apple.dt.Xcode IDEPackageSupportUseBuiltinSCM YES
            rm ~/.ssh/id_rsa || true
            for ip in $(dig @8.8.8.8 bitbucket.org +short); do ssh-keyscan bitbucket.org,$ip; ssh-keyscan $ip; done 2>/dev/null >> ~/.ssh/known_hosts || true
            for ip in $(dig @8.8.8.8 github.com +short); do ssh-keyscan github.com,$ip; ssh-keyscan $ip; done 2>/dev/null >> ~/.ssh/known_hosts || true
      - run:
          name: Install Gem dependencies
          command: |
            cd iosApp
            bundle install
      - run:
          name: Fastlane Tests
          command: |
            cd iosApp
            fastlane scan
      - store_artifacts:
          path: iosApp/fastlane/test_output
      - store_test_results:
          path: iosApp/fastlane/test_output/report.junit
workflows:
  run-android:
    when:
      or:
        - << pipeline.parameters.build-android >>
        - << pipeline.parameters.build-all >>
    jobs:
      - build-android
  run-ios:
    when:
      or:
        - << pipeline.parameters.build-ios >>
        - << pipeline.parameters.build-all >>
    jobs:
      - build-ios

The setup:true line is not included here, which indicates that this is a ‘standard’ CircleCI configuration file. It does contain the parameters section with the pipeline parameters we defined in the previous stage.

parameters:
  build-all:
    type: boolean
    default: false
  build-android:
    type: boolean
    default: false
  build-ios:
    type: boolean
    default: false

To use pipeline parameters we first need to define them in the pipeline. We used them earlier. Now you specify their types and default values: all boolean and set to false.

workflows:
  run-android:
    when:
      or:
        - << pipeline.parameters.build-android >>
        - << pipeline.parameters.build-all >>
    jobs:
      - build-android
  run-ios:
    when:
      or:
        - << pipeline.parameters.build-ios >>
        - << pipeline.parameters.build-all >>
    jobs:
      - build-ios

Once the parameters are defined, we can use them when filtering workflows. Unlike the much simpler config shown the previous section, we need to split the jobs across 2 workflows:

The job for Android is named run-android
The job for iOS is named run-ios

We can use the when stanza combined with logical operators to specify filters for when to run a particular workflow. For Android, it is either a build-all or build-android pipeline parameter. For iOS, it is a build-ios or build-all.

This workflow runs selectively based on the changed files that run the jobs relevant to the workflow. This set up lets your team operate faster and more efficiently, while still leveraging all the multiplatform capabilities of KMM.

Conclusion

In this tutorial, you have learned how to set up a pipeline that builds Kotlin Multiplatform Mobile projects on CircleCI. It is similar to building any other project, except for knowing which platform to build for first. Mobile platforms are often developed separately, so we have used the CircleCI dynamic configuration feature to build just the portion of the app developers are currently working on.

If you have any questions or suggestions about this article, or ideas for future articles and guides, reach out to me on Twitter - @zmarkan or email me.

What I learned in 6 months at an AI company

Zan Markan — Thu, 21 May 2020 17:04:39 +0000

Back in January I joined a company called DataRobot as a Developer Advocate. My job is to inspire, enable, and represent developers using our machine learning (ML) products.
If you're not familiar with DataRobot, we provide artificial intelligence (AI) products and solutions to some of the world's largest and well known enterprises and governmental organizations.

Before joining, I had no meaningful experience in ML, data science, or AI. The farthest I got to ML was building an AI powered chatbot for a product I used to work with. I just knew that ML was hard 😅.
Since then I have built a few AI powered apps of my own, so I feel like I can talk a bit more about it. This article describes some of the things I learned in my first half year in the AI and ML industry - going from complete newbie to someone who can leverage ML in apps.

AI is everywhere! 🗺

I used to dismiss ML as just another hype, in the same bucket as blockchain or virtual reality. I thought it was mostly confined to Big Tech, academia, and hobbyists, definitely not something you'd interact with on a daily basis.

This was mostly true... about a decade ago. Boy, was I wrong.

The reality is, many organizations you interact with on a daily basis use ML in some way in their business, without you even realising. Your bank might use ML for fraud detection, your supermarket to forecast how much flour to stock, or a factory might use it to detect defects in the products they produce before they get to you.

Where ML especially shines is with economies of scale. When you have huge amounts of data to run through - imagine the number of transactions a bank or a large retailer might process, and even savings in the low single digit percentages compound dramatically.
These are only a few examples, but ML is often good fit in many areas where you want to automate and augment tasks that would otherwise be hard, or tedious for human workers... Now where have I heard this one before? 🤔

There's a reason why we call the current rise and rise of AI the Fourth Industrial Revolution.

Machine Learning is super hard... 😱

As an industry, ML is evolving rapidly. It's a moving target on all fronts - algorithms, frameworks, hardware, to use cases ranging from academic projects like AlphaGo, to self driving cars, to medical and health research. There's lots of hard science at work pushing the boundaries of what's possible. All of this progress is incredibly difficult to keep track of, and ensure you're really following the best practices.

Beyond all the technical difficulties of the bleeding edge, you have some great operational challenges as well. For example, once you've trained a model, how do you ensure that your model remains accurate over time. This is crucial, especially when data you're using starts to change in ways you haven't foreseen, and your models aren't reflective of the real world.

Last but not least, there are also challenges with governance and AI bias. Most AIs are seen as black boxes where you can't know why a certain answer was given. If a laboratory wants to use AI to help produce a diagnosis, they can't risk relying on a black box when human lives are at stake.

...but using ML doesn't have to be a struggle 🦸‍♀️

Luckily, today's bleeding edge is more accessible than ever. We've solved many hard things.
You don't need a degree in electronics to use a computer, or a networking degree to be a web developer, you don't need a data science degree anymore to build and deploy AI. The whole AI industry is moving towards democratization, and enabling everyone to work with, and benefit from AI.
There are awesome tools out there that let you build and leverage leverage models, and you don't even need to touch the frameworks like Tensorflow or Pytorch yourself.

Take Automated Machine Learning (AutoML) for example. AutoML is DataRobot's product and an associated technique that lets you train dozens of ML models using a variety of algorithms on a single dataset, pitting them against each other in order to determine which model is the best for your data. What used to take days can now be accomplished in mere minutes. New and refined algorithms are constantly being added to AutoML tools, making the models they produce ever better and better. All you need to do is to press a button.

We have also made some serious progress towards overcoming operational and governance challenges I mentioned earlier. We have tools available that let you monitor your ML models in the wild and that let you know if a model is at risk of becoming inaccurate due to changes in underlying data (we call that data drift).
Our models are also not black boxes anymore, and we can explain with increasing confidence why a given model has returned a certain prediction. Unsurprisingly, this field is called Explainable AI.

Difficulties with Data Dealing 🐘

For all the problems we've solved already, one thing remains clear - we need data for machine learning, and obtaining that data for ML is difficult.

Datasets need to be representative of your problem domain. In very few cases are you able to apply a dataset from the web that works well for your own use-case. Face and object detection is a good example of where that works - that's why many companies are in the business of providing AI-powered image and speech recognition software and services.

Often you will also need to combine multiple data sources from various sections of the business into a single learning dataset (unless you're lucky and have a data warehouse at hand already).
For this, SQL rules the world. Sometimes you need to revisit your SQL knowledge because you have obviously forgotten everything apart from the most basic commands in the last 8-ish years. Speaking for a friend, of course. 😇

Finally, and this might be pretty shocking for developers, but sometimes constructing a good training dataset also requires lots of human effort at some point in the process. Whether you're manually classifying a dataset you're preparing, or validating the output of a script - it's eyes before AI. 👀

The Pervasive Popularity of Python 🐍

Python is one of the most popular programming languages. I knew that, yet it still surprised me. This shouldn't be strange as it's the one language that caters to both data scientists as well as developers (as opposed to say Java, R, or Matlab). Python lets you build both apps, and manipulate large sets of data with the same syntax.

One of Python's strongest suits is it's readability. Python reads like English - it's probably the most English-like out of all programming languages. This makes it incredibly accessible for learning. It also has a mature set of libraries for a wide range of use-cases, from web development to scientific computing, giving it a broad appeal.
Yet, not everything about Python is perfect. The language is almost 30 years old, some 5 years senior to Java, JavaScript, and PHP, and this age shows. One of my pet peeves is the situation with dependency management. Sure, it comes with a package manager with PyPI, but the only way to ensure you're isolating your environment properly is by using an archaic combination of console commands to create and initialise Virtualenv. I find Virtualenv so confusing that I have a gist I can refer to each time I deal with a new project 🙈.

Finally, the ways developer and data scientists might use Python are vastly different. Sure, the syntax is the same, but the ecosystems vary wildly. From the libraries and frameworks you might use for either development or data science, all the way to tools used and best practices followed.

For example, data scientists love their Jupyter notebooks 📓. They are a sort of dynamic, executable documentation, where you write part markdown and part code, and then execute it alongside the documentation - rendering spreadsheets and graphs side by side! They are the preferred way to interact with Kaggle competitions and datasets. This concept is great for experimentation and therefore ideal for data scientists, yet not at all suited for building and maintaining production software - something I cherish more as a developer. What's more - all that fancy dynamic approach generates some extremely hard to review JSON code, if you ever decide to host that on GitHub 😬.

What about the Rise of the Robots? 🤖

You've probably encountered film, books, or games talking about the rise and revolt of intelligent machines - the Matrix, Terminator, 2001: A Space Odyssey, and many others. It's easy for media to paint a bleak picture.

Yet, from what I've seen so far, we have little to fear from the AI that we currently possess. Indeed, AI is developing at a rapid pace - some of the brightest people I know are working hard on pushing the boundaries.

At the same time, it's become more and more accessible for you and me to build - and benefit from - AI, and we have some amazing tools available that make it easy to approach.

So, if you'd like to see what the bleeding edge looks like and what it can do, we recently opened a trial for DataRobot for developers. If you'd like to try out the latest and greatest of ML tools, please reach out to me for access to DataRobot trial - at zan@datarobot.com, quoting dev.to in the subject.

That's just a few thoughts on my perspective after 6 months in the AI industry. There has been a lot of learning, and a lot of surprises, and I'm more enthusiastic about the future than ever, and how dramatically our lives will change because of it.
What about you? What are your thoughts and perspectives on AI and the tools that are available? Let me know in the comments!

Why I believe in the brave new world of serverless

Zan Markan — Thu, 30 May 2019 15:02:49 +0000

A few days ago I took part in a lunch & learn session at work, where we discussed the paper Serverless Computing - One step forward, two steps back. It was an excellent discussion, and if there weren't any time constraints I'm sure we could have talked about it for the rest of the week, not just our lunch hour.

Note: In my eyes, serverless is not just AWS Lambda functions. Or Azure, Google, or IBM/Apache Openwhisk. It's all the other technologies that work well alongside it - databases, queues, event-driven computation, and technologies that make it super easy to configure and deploy new services in code.

What made it such a great discussion is the fact that while most technologies are polarizing, only a few are as deeply polarizing as serverless.

The Serverless Hater's greatest hits include:

serverless has servers
I can't use it for machine learning or [insert obscure academic research topic here]
It's more expensive than renting your own EC2 instances and running your own servers!
Vendor lock-in!
It's slow!
I can't run it on custom hardware!

All of these… are true in their sense. Not all use cases are well supported by serverless technologies (or at all).
My argument is just that… it doesn't really matter. There are two main reasons for it.

Reason 1: The new liberal computing 🗽

Most profoundly, serverless is just a marketing name (yet a pretty good one - it certainly has that Marmite quality…) for a new computing paradigm - an idea.

It's the idea of writing some code and letting the vendor deploy it anywhere without worrying too much about the where, how, or how much maintaining it is going to cost you, as you only pay per invocation. Serverless is about making it simple and fun to experiment. It's what Heroku was, but more.

The Serverless framework (with a capital S) makes most of your configurations portable between vendors, and opinionated frameworks like Architect make it great for quick productivity.

Tools like Glitch, in addition to being beautifully weird, lets you tinker with deployed code from users of the whole platform, directly in your web browser, like GitHub without any of the git.

To paraphrase Paweł Ledwoń, our CTO at Pusher - "Serverless might be this generation's PHP".

That's awesome. PHP made a whole generation build things.

// Detect dark theme var iframe = document.getElementById('tweet-1132826772976701440-819'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1132826772976701440&theme=dark" }

Just like PHP did back in the day, serverless is opening doors to programming to the new generation of app developers (also known as "kids these days").
Returning to my earlier point, as of why the arguments against serverless (mostly) don't matter? As long as serverless (or any technology, or computing paradigm) is able to lower the barrier to entry to, the playing field has expanded enough, so that it has free rein to take up a bunch of new market share. As the market grows, the earlier arguments merely become fringe or niche concerns.

And the kids? They are the ones who are going to build the future - The kids are alright.

Reason 2: Birth of an ecosystem (of ecosystems), and mash it like it's 2005 🕺

The idea of "just" liberalizing computing was mostly true for the first generation of serverless, way back when Lambda was new. In 2019, serverless is also liberalizing how easy it is for services to interact with one another. It's increasingly being used to extend the functionality of existing services with whatever you really wish to create - flows, extended logic, you name it.

Webhooks are everywhere, and let you integrate and glue services together, with ease.

Zeit Now lets you make any app serverless, and deploy it in seconds, with one command. They have also launched an integration marketplace, that lets developers connect other services, and manage them from Zeit.

The good people at Zeit are also running a global hackathon with some cool prizes between 1 and 2 June 2019, and Pusher is one of the sponsors. If you're building something with Pusher APIs, please come talk to me.

We also we have Netlify Functions -they let you add dynamic components to otherwise static sites that deploy in seconds.

Or Github Actions, to create complex flows based on what's going on in GitHub, taking ops to the next level.

Or Auth0 has their rules that extend the log in functionality of their services.

Cloudflare workers deploy and execute anywhere in the world - and execute nearest to your request.

We're seeing an ecosystem of ecosystems emerge. Want to handle a webhook? Build a chatbot, even? Serverless makes that really, really easy. Just add JavaScript ✨.

What the new generation of serverless reminds me of is actually fulfilling the promise of API mash-ups.

For the younger generation, API mash-ups were the idea from circa 2005 (or when Twitter wasn't a thing yet) - of making dynamic and fully-featured websites by calling on different APIs and connecting their results on your own site.

API mash-ups never really took off in the mid noughties, because the tech just wasn't there at the time. The ecosystem wasn't there. But now, I believe it is here, and serverless is the glue. The JAMstack (JavaScript, APIs, Markdown) is a prime example of where we currently are, and it's amazing.

To summarize, I am a huge fan of Serverless. I believe serverless is a paradigm that greatly lowers the barriers for getting into software programming, and will see serious adoption and development in the next few years.
It's also showing itself as a great "glue" technology, connecting various service ecosystems, and making them work well together. These two benefits greatly outweigh any naysayer arguments.
More, and together - that's progress, and I'm sure that amazing things will happen. 🚀