Multibranch CI/CD for Serverless Functions

Serverless is one of those concepts that in hindsight seems obvious and makes you wonder why no one came up with it earlier.

At its core, it involves breaking up the application into smaller functions, called lambdas, and distributing them in the cloud. Lambdas, by their very nature, scale; this is because they are only run when needed, with resources allocated on-demand. Also, since there are no wasted resources, they tend to be cheaper to run than a full-fledged server or even a container—unless you run them 24x7.

In this article, we will use two fantastic platforms to test and deploy an application: ZEIT Now to build and host it and Semaphore to drive our Continuous Integration and Delivery.

Meet the Application

During this post, we will work with a demo app. Step by step, we’ll see how to get it online and how to make part of a continuous delivery process.

The project has a single API Endpoint that simply replies Hello World! to all requests. APIs are the way applications communicate with each other over the web, so mastering them is crucial.

To get started, fork the demo and install the Semaphore CLI. Then, add the project to Semaphore:

$ cd semaphore-demo-zeit-now
$ sem init

The main file is located at api/hello.js. It has a function that replies with the hello message. Strictly speaking, this is the whole application. You could deploy this file and call it a day. We’ll see that ZEIT Now does a lot of work for us. However, the developer’s life calls for two more things: a development environment and some kind of testing.

Lambdas Made Easy With ZEIT Now

With ZEIT Now, we can make a global serverless deployment with just a few keystrokes. The magic resides in their builders which take your existing application code and transforms it into lambda functions. Do you have a lot of static files? No problem. With minification and a smart built-in global CDN, they have you covered.

Internally, Now Lambdas work on top of AWS Lambdas; ZEIT Now also supports other cloud providers (only for caching and routing). As we’ll see next, we won’t have to configure or set up any service in AWS to run our application.

In ZEIT Now, there are no servers to manage and no containers to build. It integrates nicely with any workflow and plays well with CI/CD platforms. With instant, automatic deployments, ZEIT Now is a perfect fit for our microservice app.

Apps in ZEIT Now are immutable, meaning they cannot be changed once deployed. Each newly published version gets a unique deployment URL. As it turns out, this clever approach allows us to roll back to any previous version at any time.

ZEIT Now addresses follow this format:

• https://APP_NAME.USERNAME.now.sh: Public-facing URL that points to the latest version.
• https://APP_NAME-UUID.now.sh: Deployment URL. UUID is a unique, automatically generated string for each release.

What’s All the Buzz About Continuous Integration

Wouldn’t it be great if we could automate every step of testing and deployment? This is precisely the problem that continuous integration and continuous delivery solve. CI/CD takes care of everthing for us. And it does it as a reproducible, battle-hardened process.

Older continuous integration and delivery platforms, like the stand-alone version of Jenkins, encumber developers an infrastructure to manage. In Semaphore there is no back-end to maintain, no servers to install, or any Java versions to troubleshoot—another point for serverless platforms.

In Semaphore, we define pipelines in a clean, easy-to-read format and do as many Git pushes as needed. Semaphore will silently provision everything to drive the pipeline, at any scale.

In this section, we’ll review how the Continuous Integration pipeline works. The CI pipeline builds and tests the code in the Semaphore environment.

Open the pipeline file located at .semaphore/semaphore.yml.. We’ll review its elements.

Name, Version and Agent

Let’s start with the basics: the name, version, and agent. In the agent we choose which of the available machine types will run our jobs.

version: v1.0
name: Build and test
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804

Blocks and Jobs

These define what to do at each step of the pipeline. Each block can have many jobs. Jobs contain the list of commands to execute. Within a block, jobs run concurrently; each one runs in a fully isolated virtual machine. Once all jobs in a block are done, the next block begins.

The first block downloads and installs all the required packages:

blocks:
  - name: Install dependencies
    task:
      jobs:
        - name: npm install and cache
          commands:
            - checkout
            - nvm use
            - cache restore
            - npm install
            - cache store

• Checkout clones the code from GitHub.
• Cache is used to share node modules between jobs. It is smart enough to determine which files need to be stored and retrieved.

Since each job lives in an isolated environment, the second block must repeat some of the commands and get the node_modules from the cache. Then we can run the test scripts.

  - name: Run tests
    task:
      jobs:
        - name: npm test
          commands:
            - checkout
            - nvm use
            - cache restore
            - npm test

Promotions

Promotions chain pipelines together to build complex workflows. Since they can be triggered by user-defined conditions, they can be used to gracefully manage failures or to make a release.

promotions:
  - name: Deploy to staging
    pipeline_file: deploy-staging.yml

branch:
  - name: Deploy to production
    pipeline_file: deploy-production.yml
    auto_promote_on:
      - result: passed
        branch:
          - master

We have two branching promotions:

• Deploy to production: automatically started once all tests are green for the master branch.
• Deploy to staging: can be manually initiated from a Semaphore workflow on any branch.

Continuous Deployment Pipeline

Now that we have all the pieces of the puzzle in place, you will see for yourself the power and convenience of CI/CD.

To connect Semaphore and ZEIT Now we’ll need to get a token from the latter.

• In ZEIT Now go to: Account > Settings > Tokens > Create

The token, being private information, does not belong in the repository. Semaphore has a secure mechanism to handle secrets:

$ sem create secret now -e ZEIT_TOKEN=YOUR_ZEIT_TOKEN

We have two almost identical continuous delivery pipelines: staging and production. They only differ in the deployment name. The difference allows us to have separate URLs for each environment.

blocks:
  - name: Deploy to production 
    task:
      secrets:
        - name: now
      jobs:
      - name: Deploy to Zeit Now 
        commands:
          - checkout 
          - nvm use 
          - npm install now -g 
          - now --token $ZEIT_TOKEN -n semaphore-demo-zeit-now

The deployment itself couldn’t be easier:

1. The secret token is decrypted and imported as $ZEIT_TOKEN.
2. The now cli tool is installed.
3. Now does the deployment.

Start the Pipeline

This is where all our hard work pays off:

$ touch any_file
$ git add any_file
$ git commit -m "start semaphore pipeline"
$ git push origin master

In a few seconds, you should have your lambda function deployed to production.

Click on the Promote button under “Deploy to staging” to create the staging site.

Once all the blocks are done, you should have both sites online. Try the APIs:

$ curl -w "\n" https://semaphore-demo-zeit-now-staging.YOUR_NAME.now.sh/api/hello
Hello World!

$ curl -w "\n" https://semaphore-demo-zeit-now.YOUR_NAME.now.sh/api/hello 
Hello World!

You Did It!

Nice work! Once you taste the power of serverless architecture and CI/CD, you will want to use it in all your projects. I hope that what you learned today helps you to build better and smarter.

If this article got you interested in serverless, you should check these links:

• Continuous Delivery Patterns with Serverless Applications
• A CI/CD Pipeline for Serverless Cloudflare Workers - Netlify Continuous Deployment

Did you find the post useful? Hit those ❤️ and 🦄, follow me or leave a comment below!

Thanks for reading!

Comments

[Robin L.]

Great intro to serverless functions. FaaS has a lot of potential in satisfying hungry key functions in projects with micro service architectures.
We are currently converting a few projects from pure msa into mixed msa and a function oriented architecure. Hopes of improved scalability are high and for now seem legit. I recommend people starting new projects to have FaaS in mind because extracting specific functions out of microservices can be quite hard.

[Tomas Fernandez]

Thank you, I'm happy you enjoyed it. I think people are just now realizing what a neat thing serverless is. It's not yet as mature as I would hope but it's getting there.

Best of luck with your projects!

[Robin L.]

Thank you. I think eg. the knative stack feels pretty mature already. But I agree one should only slowly integrate specific FaaS cases into production and see how they perform. Our experience is good so far and generally speaking there is alot of development happening upstream atm so stay tuned :)