DEV Community: Kyle TechSquidTV

Introducing the CircleCI Config SDK

Kyle TechSquidTV — Mon, 19 Sep 2022 18:00:00 +0000

We are excited to announce the new CircleCI Config SDK is now available as an open-source TypeScript library. Developers can now write and manage their CircleCI config.yml files using TypeScript and JavaScript.

For developers used to the ecosystem and flexibility of a full-fledged programming language, sometimes YAML can feel limiting or intimidating. With the Config SDK you can define and generate your YAML config from type-safe and annotated JavaScript. You can even take advantage of package management to modularize any portion of your config code for reuse.

When paired with CircleCI’s dynamic configuration, the Config SDK makes it possible to dynamically build your CI configuration at run-time, allowing you to choose what you want to execute based on any number of factors, such as the status of your Git repo, an external API, or just the day of the week.

Getting started

For our example, let’s imagine we manage several Node.js projects that are all built using the same framework and generally require the same CI configuration. So we decided that we want to build a config “template” that all of our projects will use, and which can be centrally managed and updated. We’ll create and publish an NPM package that will generate the perfect config file for all of our Node projects.

Let’s build the config template package, and then build the pipelines that will use it.

Setup

We’re going to start by creating a standard NPM package. You can use TypeScript or JavaScript, but we’ll use JavaScript in this example for the sake of speed. The example shown here is based onthis page from the repo’s wiki.

Begin by initializing a JavaScript project in a new directory.

mkdir <your-package-name>

cd <your-package-name>

npm init -y

npm i --save @circleci/circleci-config-sdk

The @circleci/circleci-config-sdk package will allow us to define a CircleCI config file with JavaScript. While we could simply define a config and export it, we can also take advantage of dynamic config and export a function instead. In our example, we’ll keep it simple and create a config generation function that will take a tag parameter for our deployments, and a path parameter to choose where the config file will be exported to.

Create the app

Create an index.js file and import the CircleCI Config SDK package and Node’s fs package so we can write the config to a file.

const CircleCI = require("@circleci/circleci-config-sdk");
const fs = require('fs');

Next we’ll start building up the components of our config file using the Config SDK. You’ll notice because we are working with a TypeScript-based library, we are able to receive code hints, type definitions, documentation and auto-completion.

Create executor

Given we are building a config for our Node.js projects, we’ll begin by defining the Docker executor our jobs will use. You can pass in the Docker image, resource class, and any other parameters you may want to configure.

// Node executor
const dockerNode = new CircleCI.executors.DockerExecutor(
  "cimg/node:lts"
);

Create jobs

We are building up to a workflow that will test our application on every commit, and deploy it when we provide a certain tag. Like our executor, we’ll define these two jobs, both using the executor we just defined and each with a unique set of steps for their respective purposes.

// Test Job
const testJob = new CircleCI.Job("test", dockerNode);
testJob.addStep(new CircleCI.commands.Checkout());
testJob.addStep(new CircleCI.commands.Run({ command: "npm install && npm run test" }));

//Deploy Job
const deployJob = new CircleCI.Job("deploy", dockerNode);
deployJob.addStep(new CircleCI.commands.Checkout());
deployJob.addStep(new CircleCI.commands.Run({ command: "npm run deploy" }));

Jobs can be instantiated with steps or dynamically added to an existing job like shown above. In this overly simplified example, we lack a caching step, but you can see how we build up elements of our configuration.

Create a workflow

With our jobs defined, it’s time to implement them in a workflow and define how they should run. We mentioned earlier that we want the test job to run on all commits, and the deploy job to only run on the given tag.

Now that we are working with top-level components of our config file, let’s finally define a new CircleCI config object, and name the workflow that we will add our jobs to.

//Instantiate Config and Workflow
const nodeConfig = new CircleCI.Config();
const nodeWorkflow = new CircleCI.Workflow("node-test-deploy");
nodeConfig.addWorkflow(nodeWorkflow);

We are not adding any parameters to our testing job because we want to run it on all commits, so we can directly add it to our config object.

nodeWorkflow.addJob(testJob);

For the deploy job, we need to first define the workflow job so that we can add filters to it. We are going to add one filter now which tells CircleCI to ignore this job for all branches, so it doesn’t execute on every commit. We’ll deal with enabling it for a tag in a moment.

const wfDeployJob = new CircleCI.workflow.WorkflowJob(deployJob, {requires: ["test"], filters: {branches: {ignore: ".*"}}});
nodeWorkflow.jobs.push(wfDeployJob);

Export the config generator function

Now that we have everything defined we are ready to create and export the final piece. We are going to create a function which takes in the tag and path parameters we mentioned earlier and will write what we have defined to a new file.

/**
* Exports a CircleCI config for a node project
*/
export default function writeNodeConfig(deployTag, configPath) {
 // next step
}

In the newly created writeNodeConfig function, we’ll add the tag filter being passed in here to the deploy job in our workflow and finally write the config to the file supplied by the path parameter using the generate function on the config object.

/**
* Exports a CircleCI config for a node project
*/
export default function writeNodeConfig(deployTag, configPath) {
  wfDeployJob.parameters.filters.tags = {only: deployTag}
  fs.writeFile(configPath, nodeConfig.generate(), (err) => {
    if (err) {
      console.error(err);
      return
    }
  })
}

Here is the full source code, which you can also find in the wiki:

const CircleCI = require("@circleci/circleci-config-sdk");
const fs = require('fs');

// Node executor
const dockerNode = new CircleCI.executors.DockerExecutor(
  "cimg/node:lts"
);

// Test Job
const testJob = new CircleCI.Job("test", dockerNode);
testJob.addStep(new CircleCI.commands.Checkout());
testJob.addStep(new CircleCI.commands.Run({ command: "npm install && npm run test" }));

//Deploy Job
const deployJob = new CircleCI.Job("deploy", dockerNode);
deployJob.addStep(new CircleCI.commands.Checkout());
deployJob.addStep(new CircleCI.commands.Run({ command: "npm run deploy" }));

//Instantiate Config and Workflow
const nodeConfig = new CircleCI.Config();
const nodeWorkflow = new CircleCI.Workflow("node-test-deploy");
nodeConfig.addWorkflow(nodeWorkflow);

//Add Jobs. Add filters to deploy job
nodeWorkflow.addJob(testJob);
const wfDeployJob = new CircleCI.workflow.WorkflowJob(deployJob, {requires: ["test"], filters: {branches: {ignore: ".*"}}});
nodeWorkflow.jobs.push(wfDeployJob);

/**
* Exports a CircleCI config for a node project
*/
export default function writeNodeConfig(deployTag, configPath) {
  wfDeployJob.parameters.filters.tags = {only: deployTag};
  fs.writeFile(configPath, nodeConfig.generate(), (err) => {
    if (err) {
      console.error(err);
      return
    }
  });
}

Publish the package

With your index.js file completed with the writeNodeConfig function exported, it’s time to publish the package to your package repository of choice, such as NPM or GitHub.

When complete, you should be able to import your package in other projects, just like we imported @circleci/circleci-config-sdk earlier.

Create a CI pipeline

You now have a published NPM package that can generate a CircleCI config file. We can use CircleCI’s dynamic configuration to pull in this package at run-time and dynamically run our generated config file. We can replicate this basic template across our many similar NodeJS projects, and when we want to update or change our config, we will be able to simply update the package we created.

Create config.yml

As usual, our CircleCI project will require a .circleci directory and a config.yml file inside. The config file in this case will be a basic template used in all of our projects which simply tells CircleCI to enable the dynamic configuration feature for the current pipeline, generate the new config file, and run it. We will also create a dynamic directory that we will use later.

└── .circleci/
    ├── dynamic/
    └── config.yml` \

Use the following example configuration file:

version: 2.1
orbs:
  continuation: circleci/continuation@0.3.1
  node: circleci/node@5.0.2
setup: true
jobs:
  generate-config:
    executor: node/default
    steps:
      - checkout
      - node/install-packages:
          app-dir: .circleci/dynamic
      - run:
          name: Generate config
          command: node .circleci/dynamic/index.js
      - continuation/continue:
          configuration_path: ./dynamicConfig.yml
workflows:
  dynamic-workflow:
    jobs:
      - generate-config

You can find this config file and other examples in the Wiki on GitHub.

This config is a boilerplate responsible for executing our package which contains the “real” logic we want to execute.

Notice the setup key is set to true, enabling dynamic config for the pipeline. Using the Node orb, we install a Node app located in .circleci/dynamic (we’ll come back to this), and run the .circleci/dynamic/index.js in Node. This is what will use the package we wrote earlier and create a new config file at ./dynamicConfig.yml, which will finally be executed by the continuation orb.

We will use a config like this one in all of our Node projects, and it is unlikely to need to be updated or changed often. Rather than modifying this config, we update the package we created earlier.

Create the config app

The last thing to do is build our “Config application”. This is the app responsible for implementing our package and acts as the source of our “real” config we plan on executing. Because in this example we have outsourced the majority of the logic of our config to an external package, in this example, our config app is mostly boilerplate as well.

Change directory into .circleci/dynamic where we will set up our application. Initialize a new repository and install the package you published previously.

npm init -y

npm i <your-package-name>

After running these two commands, you should have a package.json file with a dependency showing your package.

"dependencies": {
  "my-circleci-node-config": "^1.0.0",
}

You can modify the semantic version string to dictate what version of your package should be pulled at run time. This means, you could update your my-circleci-node-config package and, if you choose, have all of your CircleCI projects that utilize this package immediately pick up these changes the next time your CI pipeline is triggered.

Not Recommended: Say you always wanted to pull the latest version of the custom dependency you have created, you could use:

"dependencies": {
  "my-circleci-node-config": "x",
},

To be safer, pull in only minor and patch updates, not major releases.

"dependencies": {
  "my-circleci-node-config": "1.x",
},

Finally, utilize your custom package in the .circleci/dynamic.index.js.

Our package exports a function named writeNodeConfig which takes in the value of the tag we want for triggering deployments, and the path we want to export the config to. We know the path from earlier in our config.yml, we set to be ./dynamicConfig.yml, because we are in the dynamic directory, we will prepend ... For the tag, we’ll use a generic regex string v.*

import writeNodeConfig from '<your/package>';

writeNodeConfig("v.*", "../dynamicConfig.yml")

That is our entire application. We simply need to pull whichever version of our package we desire and invoke it to generate our config file from the template we created in the package.

Running the pipeline

To recap, we have a boilerplate config.yml file which instructs CircleCI to enable dynamic configuration, and uses Node.js to build a new config file at run-time. The Node app responsible for building our new config uses a package dependency which contains the “template” of our desired config. We can now use this package in many different projects and update it centrally. Our projects can either pull in the latest version of this package by specifying an x in the package.json, or we can use tools like dependabot to open a pull request automatically to all of our projects that use this package when it is updated.

With dynamic configuration and the Config SDK together, the possibilities are endless. The tutorial above was based on this page from the wiki on GitHub. Check out the rest of our wiki and documentation for even more examples of interesting things you can do with the Config SDK.

We’d love to hear from you and see how you utilize the Config SDK in your own pipelines. Connect with us on Twitter, say hello in our discussion forum, or chat with us on Discord.

Trigger your CircleCI pipelines from a GitHub Actions workflow

Kyle TechSquidTV — Fri, 17 Dec 2021 17:30:00 +0000

If you are already a GitHub user, you may know that GitHub Actions provides you with powerful tools to increase efficiencies in your software delivery life cycle. Actions can be impactful for team collaborations and process simplification. For example, you can automate things like building a container, welcoming new users to your open source projects, managing branches, or triaging issues.

It’s important to use the right tool for the right job. Using GitHub Actions along with CircleCI allows you to automate aspects of your version control system while also benefiting from features that are only available in your CircleCI pipelines. Imagine extending your VCS workflow to include advanced features like SSH debugging, test splitting, robust CPU and RAM options and GPU and Arm support.

For example, you can use Actions for team collaboration and process simplification while accelerating your build, test, and deploy stages with CircleCI.

To make it even easier to kick off your CircleCI pipelines from any event on GitHub, our team built the Trigger CircleCI Pipeline action, which is now available in the GitHub Marketplace.

In this article, we will describe a few ways you can incorporate the Trigger CircleCI Pipeline action into your workflow to make the most of your GitHub automation credits while benefiting from CircleCI’s world-class continuous integration capabilities.

Trigger CircleCI Pipeline: A GitHub Action for triggering workflows in CircleCI

With the Trigger CircleCI Pipeline action, you can kick off a CircleCI pipeline from any event on a given branch or tag. The action passes along a set of pipeline parameters containing metadata that can be used to control the flow of your CircleCI pipelines and workflows.

One potential use case for the Trigger CircleCI Pipeline action is triggering a CircleCI pipeline whenever a pull request is opened by a developer. Triggering on a pull request allows you to test changes in a PR even if the last commit was made prior to the PR being opened (or to trigger a build and test cycle any time changes are pushed to the branch a PR was opened from).

Another use case for the Trigger CircleCI Pipeline action is triggering a build and deployment from CircleCI whenever a release is created, edited, or published in GitHub. Triggering on a release allows you to build and deploy an application—such as deploying a Dockerized microservice to a Kubernetes cluster or publishing a binary to an app store—when the repo owner specifies a new release version of the application.

You can view complete implementations of these two use cases, including templates for both your CircleCI and GitHub Actions configuration files, in the Examples directory of the Trigger CircleCI Pipeline repository. Next we’ll explore an example of using the Trigger CircleCI Pipeline Action to initiate a CircleCI workflow when a new release is published.

How to use the Trigger CircleCI Pipeline Action to trigger a pipeline from a new release

The first step in setting up GitHub Actions to work with CircleCI is to create a GitHub Actions workflow for the desired CircleCI pipeline. You can do this by adding a workflow YAML file (we’ll use main.yml) to ./.github/workflows.

In the following example, you use the on and types syntax to specify that the trigger-circleci action triggers when a new release is published. Select a custom name and id for the step to provide additional contextual metadata in your CircleCI pipeline:

on:
  release:
    types: [published]
jobs:
  trigger-circleci:
    runs-on: ubuntu-latest
    steps:
      - name: <customize name>
        id: <customize id>
        uses: circleci/trigger_circleci_pipeline@v1.0
        env:
          CCI_TOKEN: $

Next, create an encrypted secret named CCI_TOKEN containing the personal API token that will be used to trigger the pipelines. We recommend making this a machine user.

Now that you have your GitHub workflow set up, you can modify your CircleCI config file to run when your GitHub Action kicks off. Start by adding the pipeline parameter definitions to your CircleCI config. This data will be entered by the GitHub Action when triggered.

Add the following to the top of your .circleci/config.yml file. Make sure you specify version 2.1:

version: 2.1
parameters:
  GHA_Event:
    type: string
    default: ""
  GHA_Actor:
    type: string
    default: ""
  GHA_Action:
    type: string
    default: ""
  GHA_Meta:
    type: string
    default: ""

The action generates several pipeline parameters:

GHA_Actor indicates which user triggered the pipeline.
GHA_Action is the id assigned in the workflow configuration.
GHA_Event is the type of GitHub event that triggered the pipeline.
GHA_Meta is an optional additional metadata parameter that allows you to specify additional metadata using input parameters.

You can use this pipeline parameter data to run workflows conditionally. In this example, the GHA_Event parameter will be populated with the value release, and you can use a when clause in your CircleCI config to specify workflows that should run only when triggered by the GitHub workflow:

jobs:
  release:
    docker:
      - image: cimg/node:lts
    steps:
      - run: npm install
      - run: npm build
      - run: npm publish

workflows:
  release:
    when:
      equal: ["release", << pipeline.parameters.GHA_Event >>]
    jobs:
      - release

This config snippet specifies a release job that publishes an npm package and a release workflow that invokes the release job. In the when clause, you specify that the release workflow will run only when the GitHub workflow runs and populates the GHA_Event parameter with the value release.

Although this example demonstrates how to set up a release trigger, you can use the same techniques to kick off a CircleCI pipeline from any of the available GitHub events for triggering workflows. To view the complete setup for triggering CircleCI from events on a pull request, visit the example GitHub repo.

How to prevent double execution of jobs in GitHub and CircleCI

It is important to emphasize that GitHub Actions runs alongside native CircleCI integration. By default, when a repository is connected to CircleCI, if a workflow within that project’s configuration does not specify any conditionals or filters that would otherwise prevent execution, then the workflow will execute on every push event by default. This means that it is possible to accidentally run a job twice, once on the push event from CircleCI and again on other events triggered by the GitHub Action.

If you are relying on GitHub Actions to provide all of your API triggers, ensure that each of your CircleCI configuration’s workflows contains a conditional limiting its execution to only the GitHub Action trigger, as in the following example:

workflows:
  # This workflow is set to be triggered conditionally, only when
  # the GitHub Action is triggered.
  # With no other workflows, normal push events will be ignored.
  when: << pipeline.parameters.GHA_Action >>
  test:
    jobs:
      - test

Using the when clause in your workflow declaration allows you to specify that CircleCI should run the workflow only when the specified GHA_Action is triggered and not after normal push events.

Conclusion

GitHub Actions is a useful tool for automating a variety of version control processes, from checking out a Git repository at a particular version to automatically creating or merging pull requests to updating README files based on changes to your code. With the help of the Trigger CircleCI Pipeline action, you can combine the time-saving benefits of version control automation with the powerful workflow acceleration that only the fastest, most secure, and feature-rich continuous integration platform can provide.

To add the Trigger CircleCI Pipeline action to your repository, visit the GitHub Marketplace and copy the workflow syntax to your workflow.yml file. If your team is not already using CircleCI to validate and ship changes to your users quickly and confidently, sign up for a free account today.

Combine Objects In TypeScript

Kyle TechSquidTV — Sun, 11 Apr 2021 20:41:32 +0000

// Detect dark theme var iframe = document.getElementById('tweet-1381343242130690055-554'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1381343242130690055&theme=dark" }

If you Google how to combine multiple objects into a single object with shared properties, you will get some information back on using the spread operator. Let's look at a basic example of the spread operator with TypeScript.

In this example, we will be creating Super Heros by combining our regular Heros with SuperPowers to result in a single Super Hero.

Begin by creating an Interface for each of the two objects. One object should ideally contain all of the required properties, and our secondary object should contain only optional properties.

Our Hero will be the base object which we will later combine with Super Powers.

interface Hero {
  class: "human" | "elf" | "orc",
  baseHealth: number
}

Our secondary object Interface, SuperPowers will contain only optional properties describing our possible superpowers.

interface SuperPowers {
  flying?: boolean,
  superSpeed?: boolean,
  strengthMultiplier?: number
}

Now, if our human Hero were to fall into a vat of nuclear waste we can only assume they would potentially gain Super Powers. Let's describe an object with the properties of both.

interface SuperHero extends Hero, SuperPowers {}

Our SuperHero interface simply extends both of our prior definitions, effectively adding them together into this new Interface.

Now we have our definitions, let's create our new human Hero and an object containing the SuperPowers we plan to add to our human Hero later.

const human: Hero = {class: "human", baseHealth: 100}
const powers: SuperPowers = {flying: true}

Time to combine. Our Hero has fallen into a vat of nuclear waste, what emerges will no longer be a Hero, but a SuperHero.

const superMan: SuperHero = {...human, ...powers}

superMan will have the following types:

baseHealth
class
flying?
strengthMultiplier?
superSpeed?

Full example

interface Hero {
  class: "human" | "elf" | "orc",
  baseHealth: number
}

interface SuperPowers {
  flying?: boolean,
  superSpeed?: boolean,
  strengthMultiplier?: number
}

interface SuperHero extends Hero, SuperPowers {}

const human: Hero = {class: "human", baseHealth: 100}
const powers: SuperPowers = {flying: true}

const superMan: SuperHero = {...human, ...powers}

TypeScript Playground: TS Playground
Spread Syntax: MDN Spread Syntax

Kubernetes overview in 10 minutes

Kyle TechSquidTV — Sun, 06 Sep 2020 20:15:29 +0000

So you’ve used Docker, deployed a service or two, maybe even ran some production application using Docker-Compose. Now your audience has grown and you need to support an increasing demand on your server.

Or like me, you just work in #devops. One of the most critical tools of DevOps today is the Kubernetes platform, a way to manage and scale our containerized deployments to “the cloud”.

What is “the cloud”?

“The cloud” is an abstract colloquialism to describe a range of different services, platforms, and ideologies with the goals of storing and serving data on the internet. That could mean your cloud backup provider who is storing your data with Amazon’s AWS S3 or Google’s GCP Cloud storage, or how Netflix serves and delivers its vast content library.

A cloud is a representation of some number of machines (servers, virtual machines, ect) which are communicating together over the internet in cooperation of some task. A slightly more technical term we’ll use from now on is “cluster”. A cluster, in particular, refers to some group of machines.

What does Kubernetes do?

Well, a lot.

But, one of the main things Kubernetes does is maintain clusters for us. Kubernetes is a platform for managing the lifecycle of a group of machines, virtual or not. When you compare that capability with a cloud provider that can give you “unlimited” compute at any time, that gives us a way to scale up our apps or services when they are in high demand, but then turn off and remove those extra servers when they are no longer needed, saving a lot of headaches and of course, money 💰.

There’s also a bunch of other great features such as rolling deployments, and rolling back deployments.

How does Kubernetes work?

Kubernetes Cluster

Kubernetes is a platform made of many micro-services that work together to create and manage the cluster of machines that will soon run our application.

Master Node and the API service

The main service of the platform is the Kubernetes API, with which each node in the cluster will communicate with.

The node(s) in the cluster running the API service are typically referred to as “master” nodes, while any other node is referred to as a “worker” node.

This API is also how we as administrators will interact with the cluster and issue command and deployments, etc. However, we don’t usually deal with the API directly (but we could), we usually use a CLI tool called “kubectl”.

If you are familiar with Docker, this should be familiar. Docker also operates in a Client-Server relationship using APIs.

Worker Nodes

The other nodes in the cluster, “worker” nodes are responsible for reacting to events from the API service and reporting back their status, and ultimately will run our containerized applications.

Typically, you do not self-manage the hardware for a Kubernetes cluster, instead you work with a cloud provider. Working with a cloud provider who can supply a large number of machines means we can request and pay for only as many of these worker nodes as we need, and remove them from the cluster when demand goes down.

On each node in a cluster, you will find an agent installed called “kublet”. The “kublet” agent will self-register with the API server and become a part of the cluster as soon as it is online. This two-way communication between the Node and the API service allows the API to maintain state, so if a node were to go offline it can be detected and replaced.

While the API service manages and maintains the worker nodes, the worker nodes are maintaining their own components called pods and if any pod were to fail, the worker node would “self-heal” by replacing the pod.

Pods

We mentioned above that worker nodes ultimately run our containerized applications, they do so by creating and maintaining “pods”. Pods are the smallest deployable unit in Kubernetes and their lifecycle is handled by the node on which it is running.

In Kubernetes we don’t interact with Docker directly, everything we deal with is purely Kubernetes API and the Docker stuff happens behind the scenes. Pods also manage their own networking and any attached volumes individually.

If the pod, or container within, crashes or fails, the node should automatically attempt to rebuild the pod.

Deploy a sample app to the cloud with Kubernetes

In the video above we go a little deeper in-depth on Kubernetes and write our own configuration file which we use to deploy a sample application to our Kubernetes cluster.

Sample deploy.yaml config file:

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: echoserver-deploy
spec:
  selector:
    matchLabels:
      app: echoserver
  replicas: 3 # tells deployment to run 3 pods matching the template
  template:
    metadata:
      labels:
        app: echoserver
    spec:
      containers:
      - name: echoserver
        image: gcr.io/google-samples/node-hello:1.0
        ports:
        - containerPort: 80

The instructions for configuring your cluster will vary slightly depending on the cloud provider you choose.

Once we have a cluster configured and are authenticated with kubectl, we apply our configuration file.

kubectl apply -f ./deploy.yaml

The Kubernetes API service will now work to mirror the state of this config file by issuing commands to worker nodes to stand up 3 pods containing the container specified.

Run “get deployments” to watch the status of the deployment. Once 3/3 have completed (in this case), we are nearly done.

kubectl get deployments

To expose our internally deployed service to the outside internet, we must forward the port from our containerized service to the outside port on the node.

kubectl expose deployment/echoserver-deploy --type="NodePort" --port 80

If at this point you run the get services command, you should see both the Kubernetes API service and the service we have just deployed.

kubectl get services

Now, if you visit the IP address of any of your nodes at the port specified you should see your live service.

🎉

With your service now live on each node, it’s time to place these services behind a load balancer. But, that will be a tutorial for part2!