Continuously Integrate Go Applications

At the end of this tutorial you will learn,

• How to build Go application container image without using a Dockerfile
• What are Secrets and how to add them to your Project
• What are Connectors and how to add a Docker Registry Connector to your Project

Pre-requisites

Before you get started with the tutorial make sure you have the following accounts, credentials and tools,

• A GitHub account, where you may need to fork the tutorial sources.
• A Docker Registry account e.g DockerHub, Quay.io
• A Harness CI free tier account.

The following tools are required to try building the sources locally for test and verification,

• Drone CLI to build the application locally.
• Docker Desktop

Overview

As part of this tutorial we will be building a simple Go REST API called fruits-api. The application uses a RDBMS(PostgreSQL or MySQL) or NOSQL(MongoDB) to store the fruits data.

Tutorial Source

The complete demo source is available here https://github.com/harness-apps/go-fruits-api, fork the repository on to your GitHub account. For rest of the tutorial we will refer to this repository as $TUTORIAL_GIT_REPO.

Building Application Locally

Languages and package formats have build specific tools. One of the core problems that a developer might face is to install the right version of those tools on their local machines. This approach has potential pit falls and leads to Works only on my machine scenarios.

Docker containers solved this problem and helped us to have clean environment that had right set of tools, encouraging the DevOps best practices right from the start. This approach also helps to identify the potential issues with the application at development stage.

Drone by Harness is an open source CI platform that can help building and testing on your local machines without the need of installing the tools as required by the programming languages.

But before we start to build the application, we need place to store the artifacts of the build i.e. container images. In container/Cloud Native world this is called a Container Registry e.g Docker Hub, Quay.io, Harbor etc.,

Configure Container Registry

Like any file you want to share with the world, storing them in an external spot makes them more accessible. A big benefit of using containers as a packaging format is the ecosystem of container registries out there. Your firm might have a registry provider such as Docker Hub, Quay.io, Harbor, Google Container Registry(GCR), Elastic Container Registry(ECR) etc.,

For this tutorial we will be using Docker Hub. If you do not have a registry available to you, you can create a Docker Hub account and then create a repository fruits-api, where we will push our fruits-api application container image.

With us having created the fruits-api repository, lets test our repository by building and pushing the image to the registry,

Login to your Docker Hub Account,

echo -n "$DOCKER_HUB_PASSWORD" |\
  docker login -u `$DOCKER_HUB_USERNAME` --password-stdin

Info

• $DOCKER_HUB_USERNAME - Docker Hub username, the one you used while registering the for the Docker Hub account or the one you wish to use if you already have an account with Docker Hub.

• $DOCKER_HUB_PASSWORD - Docker Hub user password

Let us clone the tutorial application from https://github.com/harness-apps/go-fruits-api,

#  clone go-fruits-api repository
git clone https://github.com/harness-apps/go-fruits-api.git \
  && cd "$(basename "$_" .git)"
# navigate to the clone repository folder
export TUTORIAL_HOME="$PWD"

Tip

GitHub Cli is very handy tool to work with the GitHub repositories from the command line.

Create your fork of the tutorial repository,

gh repo fork

Info

You can also create your fork from the tutorial repository https://github.com/harness-apps/go-fruits-api directly from GitHub.

To make things simple let use Drone by Harness to build and push the image from your laptops to the Docker Hub repository fruits-api,

Copy $TUTORIAL_HOME/.env.example to $TUTORIAL_HOME/.env,

cp $TUTORIAL_HOME/.env.example $TUTORIAL_HOME/.env

Edit the $TUTORIAL_HOME/.env and update it with following,

PLUGIN_REGISTRY=docker.io
PLUGIN_USERNAME=$DOCKER_HUB_USERNAME
PLUGIN_PASSWORD=$DOCKER_HUB_PASSWORD
PLUGIN_REPO=$DOCKER_HUB_USERNAME/fruits-api
PLUGIN_TAG=0.0.1

Info
Replace the $DOCKER_HUB_USERNAME, DOCKER_HUB_PASSWORD with your docker hub username and password values.

drone exec --env-file=.env

Info
It will take few mins for the build and push to complete as Drone will try to pull the container images if not exists.

If all went well your command line output(trimmed for brevity) should like,

...
[push:350] The push refers to repository [docker.io/$DOCKER_HUB_USERNAME/fruits-api:0.0.1]
[push:351] 639e874c7280: Preparing
[push:352] 96e320b34b54: Preparing
[push:353] c306578afebb: Preparing
[push:354] 96e320b34b54: Layer already exists
[push:355] c306578afebb: Pushed
[push:356] 639e874c7280: Pushed
...

You can check the pushed image at https://hub.docker.com/repository/docker/$DOCKER_HUB_USERNAME/fruits-api.

Tip
You can use tools like crane, that allows you check the image and its tags from cli
e.g.

 crane ls docker.io/$DOCKER_HUB_USERNAME/fruits-api

Simple enough locally to get your local build and packaging in. Our process to build and push the go application looks like,

These sequence of steps is referred to as a Pipeline in Continuous Integration(CI) world.

The drone pipeline build and push step uses ko-build which can build go container images without the need for Dockerfile. It also allows you to build the multi arch/platform images with much ease.

The drone exec that we did earlier is OK as long you are playing/learning a technology in other words laptop use cases, when you are working on a team to deliver some enterprise application then it becomes super critical that this process be centralized and automated. Harness Platform helps you do exactly that and much more.

The next sections this tutorial helps you get started on the building your CI Pipeline using Harness platform.

Your First Continuous Integration Pipeline

If you took a closer look at what your machine was doing during those local builds, the machine was bogged down for a few moments. For yourself, that is fine, but imagine having to support 10’s or 100’s or even 1000’s of engineers, this process can be taxing on systems. Luckily, modern Continuous Integration Platforms are designed to scale with distributed nodes. Harness Continuous Integration is designed to scale and simplify getting your local steps externalized; this is the Continuous Integration Pipeline. Let’s enable Harness Continuous Integration to mimic your local steps and create your first CI Pipeline. Once you are done, you will have a repeatable, consistent, and distributed build process.

There are a few Harness resources to create along the way, which this guide will walk through step-by-step.There are two paths to take. One path is to have Harness host all of the needed infrastructure for a distributed build. The second is to bring your own infrastructure for the distributed build.

Hosted Infrastructure:

Bring Your Own Infrastructure:

For this tutorial we will be using the Hosted Infrastructure as thats the only infrastructure available for Free Tier.

Starting off with Harness

Harness is a Platform which has lot of modules, but for this tutorial we will focus on the Continuous Integration(CI) module.

First, sign up for a Harness account to get started.

GitHub Personal Access Token(PAT)

Assuming you are leveraging GitHub, Harness will need access to the repository. It is recommended to use GitHub Personal Access Token(PAT) as a mode of providing Github credentials.

If you have not created a PAT before, on your GitHub account navigate to Settings -> Developer Settings -> Personal Access Tokens.

Important

• Make sure to jot down the token as the token will only be displayed once. For rest of the tutorial we will refer to this token value as $GITHUB_PAT.

If you plan to bring in your PAT then make sure it has the scopes admin:repo_hook and user.

Create Project

Harness Platform organizes the resources like pipelines, secrets, connectors at various scopes such as Account, Organization and Project. For this tutorial we will create all our resources at Project scope.

Login to your Harness Account that you created earlier and create a new project,

On the new project page, click Create Project to create a new project named Fruits API.

Leave other options to defaults and click Save and Continue. On the modules select Continuous Integration,

Now you are ready to wire in the pieces to Harness Continuous Integration.

Create Your First Pipeline

In the Build Module Harness Continuous Integration, walking through the wizard is the fastest path to get your build running. Click Get Started. This will create a basic Pipeline for you.

Click Get Started, select GitHub as the repository to use, and enter your GitHub Access Token $GITHUB_PAT and finally click Test Connection to verify your credentials work,

Click Continue, click Select Repository to select the Git Hub Repository that you want to build [the sample is called go-fruits-api].

Info
Please ensure the repository you select here is your fork of https://github.com/harness-apps/go-fruits-api.

Can leverage one of the Starter Configs or create a Starter Pipeline. In this case if leveraging the example app which is Go based, leveraging the Go Starter Configuration works fine.

Click Create Pipeline to start adding the pipeline steps.

There are two ways to add your pipeline steps, visual or YAML. For rest of the tutorial we will use the visual editor.

The scaffolding would have added a single step called Build Go App. In the upcoming sections we will add the other steps like lint, test and push.

Before we get to adding other steps, we need some resources that the steps require namely secrets and connectors.

Create Docker Hub Password Secret

Navigate to Project Setup --> Secrets,

Click + New Secret and select Text,

Fill your Docker Hub password on the Add new Encrypted Text window,

Create Docker Hub Registry Connector

Next let we need to add Connector that allows us to connect and later push the image to our Docker Hub repository.

Navigate to Project Setup --> Connectors,

Click + New Connector and select Docker registry,

On the new connector wizard Overview screen, enter the name of the connector as docker hub,

Click Continue to configure the credentials,

Info

• Update the Username with your $DOCKER_HUB_USERNAME
• For the Password field click Create or Select a Secret to select the secret docker hub password.

Click Continue and use the Harness Platform as the connectivity mode option,

Click Save and Continue to perform the connectivity test,

Click Finish to complete the creation of Connector resource.

Now you are all set to add other steps to the Build Go pipeline.

Update Pipeline

Navigate to the Projects --> Pipelines,

Click Build Go pipeline,

Delete the existing Build Go App step by clicking the x that appears when you hover over the step.

Click Save to save the pipeline.

Click Add Step to add a new step called lint, from the Step Library choose step type as Run and configure the step with details:

Name:

lint

Description:

Lint the go application

Select the Shell to be Bash.

Command:

golangci-lint run

Click Apply Changes to save the step and click Save to save the pipeline.

As did earlier click Add Step to add a new step called test, from the Step Library choose step type as Run and configure the step with details:

Name:

test

Description:

Test the go application

Select the Shell to be Bash.

Command:

go test -timeout 30s -v ./... 

While building the application locally we used SQLite as our database. The go application can also run with PostgreSQL or MySQL or Mongodb. For this tutorial we will be using MySQL.

For the test step to connect to the mysql service add the following environment variables to the step configuration.

FRUITS_DB_TYPE: mysql
MYSQL_HOST: "mysql"
MYSQL_PORT: 3306
MYSQL_ROOT_PASSWORD: superS3cret!
MYSQL_PASSWORD: pa55Word!
MYSQL_USER: demo
MYSQL_DATABASE: demodb

The environment variables could be added by clicking + Add under Environment Variables section of the step configuration,

Click Apply Changes to save the step.

Tip
You can awake step configuration screen by clicking the step on the visual editor.

Click Save to save the pipeline.

How can the test step connect to MySQL database ?

Harness Pipelines support a concept called as Service Dependency, it is a detached service that's accessible to all Steps in a Stage. Service dependencies support workflows such as

• Integration testing: You can set up a service and then run tests against this service.

• Running Docker-in-Docker: You can set up a dind service to process Docker commands in Run Steps.

In our tutorial we will use the Integration testing workflow to make the test step to connect to MySQL and run the integration test cases against it.

Add the MySQL Service Dependency

On the Pipeline editor click Add Service Dependency,

Configure the MySQL Dependency Service with details:

Name:

mysql

Description:

the mysql or mariadb server that will be used for testing.

Select the Container Registry to be docker hub.

Image:

mariadb

The service dependency need to be configured with the same environment variables that we added to test step.

MYSQL_PORT: 3306
MYSQL_ROOT_PASSWORD: superS3cret!
MYSQL_PASSWORD: pa55Word!
MYSQL_USER: demo
MYSQL_DATABASE: demodb

Click Apply Changes to save the step and then click Save to save the pipeline.

Lint and Test the Application

Let us verify if were able to lint and test our go application.

Click Run from the pipeline editor page,

Leaving everything to defaults namely Git Branch and Branch Name to be main, click Run Pipeline to start the pipeline run. If all ran well you should see a successful pipeline run as shown,

Tip
You can click on each step to view the logs of the respective step

Having tasted the success with our pipeline run, let us add the other step of building and pushing the go application to the container registry.

Build and Push Image to Container Registry

As did earlier navigate to the Projects --> Pipelines,

And click Build Go pipeline to open the pipeline editor,

Click Add Step to add a new step called build and push, from the Step Library choose step type as Run and configure the step with details,

Name:

build and push

Description:

Build go application

Choose Bash to be the Shell

Command:

echo -n "$DOCKER_HUB_PASSWORD" | ko auth login docker.io -u "$DOCKER_HUB_USERNAME" --password-stdin
ko build --bare --platform linux/amd64 --platform linux/arm64 .

We also need to configure few environment variables that are required by ko to build and push the image to fruits-api container repository.

Update the Environment Variables section with following values,

DOCKER_HUB_USERNAME: $DOCKER_HUB_USERNAME
DOCKER_HUB_PASSWORD: <+secrets.getValue("docker_hub_password")>
KO_DOCKER_REPO: docker.io/$DOCKER_HUB_USERNAME/fruits-api

Info

• As marked ensure the DOCKER_HUB_PASSWORD is of type Expression
• secrets.getValue is an expression that allows to get the value from the secret docker_hub_password, that was created earlier in the tutorial. Check the docs for more info
• All $DOCKER_HUB_USERNAME references should your Docker Hub Username

Click Apply Changes to save the step and click Save to save the pipeline.

With those changes saved, you are ready to lint, test, build and push your go application to container registry(DockerHub).

Run CI Pipeline

As did earlier click Run from the pipeline editor window,

Leaving everything to defaults namely Git Branch and Branch Name to be main, click Run Pipeline to start the pipeline run.

Now you are ready to execute. Click "Run Pipeline".

Once a successful run, head back to Docker Hub, and tag latest is there!

This is just the start of your Continuous Integration journey. It might seem like multiple steps to get your local build in the platform, but it unlocks the world of possibilities.

Exercise

The https://github.com/harness-apps/go-fruits-api has another branch mongodb. Adapt your pipeline so that it build and test the code from mongodb branch.

Continuing on Your Continuous Integration Journey

You can now execute your builds whenever you want in a consistent fashion. Can modify the trigger to watch for SCM events so upon commit, for example, the Pipeline gets kicked off automatically. All of the objects you create are available for you to re-use. Lastly, you can even save your backing work / have it as part of your source code. Everything that you do in Harness is represented by YAML; feel free to store it as part of your project.

After you have built your artifact, the next step is to deploy your artifact. This is where Continuous Delivery steps in and make sure to check out some other CD Tutorials.