Integrating AWS CodeBuild with Cloudsmith Repositories

AWS Codebuild and Cloudsmith

At Cloudsmith, we pride ourselves in being the central source of truth for your packages and build artifacts. We provide one global platform to centralize your assets and give you the control and visibility that you need as part of a modern DevOps process.

As part of that effort, we’re delighted to announce that if you’re using AWS CodeBuild as part of your CI/CD workflow and pipelines, we can offer you a simple way to get the output of your build processes into your Cloudsmith repository

What is AWS CodeBuild?

AWS CodeBuild is a fully managed continuous integration service that allows you to automate your source compilation, test running and packaging of your software. There is nothing to install or maintain, and you can be up and running in minutes with your own build pipeline.

Why would I want to integrate Cloudsmith with CodeBuild?
Much like CodeBuild, Cloudsmith is a fully managed repository service. And just like CodeBuild, there is nothing to install or maintain, we take care of all of that for you.

When you automate your build process with CodeBuild, you’ll need somewhere to store the resulting build artifact(s). Of course, you could store these somewhere basic, like an S3 bucket or maybe even an on-premise storage system of some kind. But doing that means you’d miss out on some of the distinct advantages Cloudsmith provides, namely:

• Performance / Reliability. Cloudsmith will deliver your artifacts and packages to customers or distributed teams at the lowest latency possible. Configurable global storage regions, configurable edge caching rules and a fast Global CDN mean users never have to wait to consume or use the output of your builds - no matter where they are.
• Access Controls / Security. We provide fine-grained access controls, enabling you to grant teams and end users the precise level of access that they need, and no more - including SAML/SSO integration so that you can plug into your existing enterprise auth systems. We also allow you to create read-only access tokens with additional configurable restrictions - if you are vendoring / selling software to end users, for example.
• Universality. We support a wide variety of package formats, including raw files, binaries or disk images. So whatever the output of your CodeBuild process is, we can support it - including integration with the native package management tools for most formats. The goal with Cloudsmith is simple: that it just works. This list is far from exhaustive - there are many, many reasons you’d want to use Cloudsmith as a platform to host and store your build artifact(s). We are happy to talk about this at great length, far longer than we can fit in here 😊 Just contact us if you’d like to chat more about it!

The bottom line: in much the same way you probably wouldn’t entertain the overheads involved in standing up your own server farm to run your CI/CD process, you don’t need to do that either for your package repositories.

Using AWS CodeBuild with Cloudsmith - A worked example.
OK, so how do we go about integrating our Cloudsmith repository with our CodeBuild projects? Let's work through an example.

This example will be pretty straightforward - we will build a Debian package from source stored in a GitHub repository and then push the resulting build to our private Cloudsmith repository.

Step 1. Set up our CodeBuild Project

You use the AWS Console to set up a new CodeBuild Project. Just select “Create Build Project” to get started:

Next, you give your project a name, an optional description and any additional optional tags:

You then select a source for your project, in this case it’s a GitHub repository, but other source repository types are supported:

As shown above, I have configured a GitHub personal access token to allow CodeBuild to access my private GitHub repository.

Then we configure the webhook that we want to use to kick start this build. In this example, we will just start the build on any push to the specified GitHub repository:

Now we select the container environment that we need to run this build, and in this case we will use the latest available Ubuntu image, and we will create a new AWS IAM service role at the same time:

And finally, we will specify that this project is going to use a buildspec file:

A buildspec file is where we define the phases of the build and what commands / actions we wish to perform as part of the build process. We will go into this in more detail in the next step!

You then just click “Create Build Project” to finish.

Step 2 - The buildspec file

Our buildspec file contains all the information required to define our build process. We store this buildspec file in our GitHub repository along with the source code that we are building. A buildspec file can be as complex or as simple as your individual project requires, but for this example it’s pretty simple.

You’ll need to set up any environment variables that your build process requires. In our example, we only need a single environment variable; our Cloudsmith API Key. We need this to use the Cloudsmith CLI to push our build artifact to our Cloudsmith repository, and we don’t want to store our API-Key as plain text in the buildspec file as it would then be revealed in any build logs - not good!

To achieve this, we use AWS Secrets Manager to securely store our API-Key, and then we can reference it in our buildspec file as follows:

env:
  secrets-manager:
    CLOUDSMITH_API_KEY:CodeBuild/CloudsmithAPI:CLOUDSMITH_API_KEY

At a minimum you then, you’ll need 3 phases in your build:

• An install phase
• A build phase
• A post-build phase

The Install phase:

The install phase is where we install any prerequisites that our build needs, and this of course will vary depending on what you are building and what your individual requirements are. In our case that means the required tooling to build our Debian package, and of course the correct Python runtime for the Cloudsmith CLI and the Cloudsmith CLI itself:

install:
  runtime-versions:
    python: 3.x
  commands:
    - apt update
    - apt-get install ruby ruby-dev rubygems build-essential -y
    - gem install --no-document fpm
    - pip install cloudsmith-cli

The Build phase:

The build phase is where we execute the commands required to actually build our package. For our Debian package, it’s pretty straightforward:

build:
  commands:
    - make
    - fpm -f -s dir -t deb -v 1.0.1 -n cloudsmith-codebuild-test .

The Post-build phase

This is where the magic happens! In the post build phase, we only require a single command - cloudsmith push to upload the result of the build to our Cloudsmith repository

post_build:
  commands:
    - cloudsmith push deb demo/codebuild-demo/debian/buster cloudsmith-codebuild-test_1.0.1_amd64.deb

That’s it, that’s all we need to build this package, and store it in our Cloudsmith repository. It really is that simple. Now of course, your build process itself may be a lot more complex, but to add Cloudsmith to your build process really is just two commands:

1. pip install cloudsmith-cli
2. cloudsmith push FORMAT OWNER/REPOSITORY PACKAGE_FILENAME

We don’t get in the way, you don’t need to totally reconfigure your entire build and we try to make it as easy as possible for you to get up and running!

Step 3 - Running the Build

As we configured this CodeBuild project to start on a push to our GitHub repository, If we just make a change to our source and then commit the change the build will start:

The job has been Submitted, Queued and CloudBuild is now provisioning the container to run the build. Once provisioned, we can tail the logs to watch the progress of the build:

Installing the Cloudsmith CLI:

Pushing build artifact to Cloudsmith:

When the build finishes, our package has been built successfully and uploaded to our Cloudsmith Repository. If we now login to Cloudsmith and view the repository, we can see our package:

To sum up

We hope that this demonstrates how simple it is to integrate Cloudsmith with your existing (or new) AWS CodeBuild projects, and gives you an idea of what you can achieve with your own CI/CD pipelines.

If you’d like to try it yourself, you can sign up for a fully-featured free trial at cloudsmith.com, and we are here to help you every step of the way.

We wish you very happy and successful builds!