Learn Continuous Integration: Self-hosting Drone CI

First things first: Why continuous integration (CI) anyways?

There is a lot of competition outside. Every company you see around has one thing in common — making their customers happy. You deliver new features fast with quality, you win the game. The whole world is geared towards quality software getting delivered every day.

You see that vegetable delivery company, it is actually a software company. Yes, you can go and check to see how they get their orders on their website, how they process the orders and set a delivery date and time and eventually deliver vegetables to customers. Who does all these things step by step so fast and in an organized way? Software, written by developers.

That’s why = every company is becoming a software company these days.

Continuous integration is the first step towards automating your software delivery. As the name suggests, it involves the integration of work by different developers, where one developer makes changes to his or her code very often and merges it into the main branch, triggering an automated build whenever a code commit happens and verifies the code works for the tests specified. This helps developers find bugs and errors early in the development process and prevent them from entering the production environment.

For those of you who don’t know Drone, here is a brief introduction before we start. Drone is a modern open-source CI platform that automatically builds, tests and releases workflows. Drone was created specifically for Docker containers, and it is written in Go.

The Goal

The goal of using Drone CI is to test your code to make sure everything is working fine, and then build your container whenever a code commit happens.

Scenario

Developer commits the code in GitHub and this triggers the Drone CI. Drone CI integrates the code and tests against the tests specified and builds a container image and pushes it to your Docket Hub account.

Pre-requisites

• GitHub account
• A virtual machine from any cloud provider

Let’s start setting up Drone CI

• We need to set up a public URL to expose our local web server, and it will help us create a webhook for Github. To do this, you can use [ngrok]https://ngrok.com/) — sign up at ngrok.com and follow these instructions to set it up. We can skip this if you have access to a virtual machine. I’ll use a virtual machine to set up and access Drone CI in this article.

Choose your preferred cloud vendor and get a virtual machine up and running. In this example, we are using Azure cloud services to create a virtual machine. Note down the VM’s public IP address and make sure it is accessible publicly over HTTP.

• Now, login to your GitHub account and create a new OAuth app. For this, you need to have your VM’s IP address ready. Along with this, you should also save your client id and client secrets (you need to generate this just with a click) you generate during this step. You create the OAuth app to login authenticate GitHub through the IP address you specify in the homepage URL.

• Drone works on a server runner model: Server orchestrates the runners by integrating with our SCM platform. Server instructs runners on what to do as per the Drone configuration file (Drone configuration file: .drone.yaml)

Now we are going to configure the Drone server and runners. This is done through Docker compose. We will create a file named .env and specify the environment variables as shown below.

# Server configuration
DRONE_SERVER_HOST=<ngrok hostname>
DRONE_SERVER_PROTO=http
DRONE_GITHUB_CLIENT_ID=<github client ID>
DRONE_GITHUB_CLIENT_SECRET=<github secret>
DRONE_RPC_SECRET=<Drone secret>
DRONE_TLS_AUTOCERT=false
DRONE_USER_CREATE=username:<github_username>,admin:true# Runners configuration
DRONE_RPC_HOST=<ngrok hostname>
DRONE_RPC_PROTO=http
DRONE_RUNNER_NAME="Drone.io_runner" # It CANNOT contain spaces!

Since we are using a vm instead of ngrok, we will share our vm’s ip address in the place of DRONE_SERVER_HOST.

DRONE_RPC_SECRET can be generated by running the below command on your terminal

openssl rand -hex 16

DRONE_RPC_HOST is again your vm’s ip address

• Start your virtual machine and connect via ssh

• Now, we need to create a file named docker-compose.yml to launch both the drone server and runners with the following content
  

version: '3.8'services:
  drone:
    image: 'drone/drone:2'
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "./volumes/drone:/data"
      - '/etc/localtime:/etc/localtime:ro'
    restart: always
    ports:
      - 80:80
    env_file:
    - .env  drone-agent:
    image: drone/drone-runner-docker:1
    command: agent
    restart: always
    depends_on:
      - drone
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - '/etc/localtime:/etc/localtime:ro'
    env_file:
      - .env

• The next step is, you need to store/save both these files in your virtual machine.

[You can do that by creating and saving both the files using the command,
vim .env press enter
You get a window to enter the file contents and copy paste the .env file contents
Save the file with the command esc+shift and semicolon+w+q+! and hit enter
Do the same for docker-compose.yml file]

• Get both docker and docker-compose installed on your virtual machine.

Here’s a sample script that would help you to install and update docker and compose on a Ubuntu VM:

curl https://gist.githubusercontent.com/hrittikhere/3117497bfaf1736a144b0a3d9463c807/raw/07f6dc8c7f3c4aad093d4075e5ebcca106304c72/pavan.sh | bash

• Then do sudo docker-compose up to start the Drone. Copy the host public IP and paste it on the browser, you should see Drone starting page like this below.

TADAAAA!!!

• Continue and add required details like name, email id, etc
• In the next step, drone will ask you to authenticate with your SCM tool

• You can see your repositories listed in the Drone dashboard.

• Now, create a new repository on GitHub, add the .drone.yml file in the root folder to give instructions to Drone. Also, create a Dockerfile to give instructions to the container on what to do. BTW, you can also do all these things on your local and push your code to GitHub.

.drone.yml can contain instructions to test your commit. But in this tutorial we would be going with a simpler workflow.
The .drone.yml file looks like this,

kind: pipeline
type: docker
name: hello-world

steps:
- name: docker  
  image: plugins/docker
  settings:
    # registry: docker.io
    repo: pavansa/bangaloretour
    username:
      from_secret: docker_username
    password:
      from_secret: docker_password
    tags: 
      - latest

Note: [In the dockerhub_username, specify your Dockerhub username]

Dockefile contains instructions to build your container. But in this tutorial, we would be going with a simpler Dockefile as shown below,

FROM alpine
CMD ["echo", "Hello World"]

• To test our Drone CI, create a similar repository on Docker Hub to store our image.

• Go to your Drone dashboard and sync the new repository you just created on GitHub. Once the syncing is complete, go to the specific repository and set up the secrets. You can do that by going to the settings tab and then secrets.

Add docker_username as the first secret and specify the value
Add docker_password as the second secret and specify the value

• Once you add your secrets and save them, you can go to + NEW BUILD tab on the upper right corner and just click on it and a new tab will pop up as shown below and click Create.

• Once you click on Create in the previous step, you should see Drone fetching the changes from your repository and listening to the drone.yml file

• It will say that the build has started successfully and the dashboard can be shown as below

• If everything goes well, you can click on the executions thread and see that the build becoming successful.

This is how we are creating an image and pushing it to the Docker hub.

You can verify this by going to your Docker hub if it is up to date with the latest changes.

Well, we just installed Drone CI locally on our machine:)

BTW, there is also a graph view that will show you the graphical representation of your pipeline. You can click on any of the stages and it will show you the logs too. It makes it easy to debug your issues and fix them as fast as possible.

BTW, Harness Continuous Delivery is now source available! You can try it here.

For more info on Drone CI, visit their official website