Git-Driven Deployments on Origin Kubernetes Distribution

This is the second post in a series explaining how uniqkey.eu does git-driven infrastructure-as-code on the OKD distribution of Kubernetes. We made our tools open source as OCD. They can be used as rocket powering rainbow dust to get your features shipped 🌈✨🚀 This post will give an overview of a short OCD tutorial that deploys and then upgrade a ReactJS app when you push changes into a git repo. Here is a video of what the tutorial has you do:

We call this git-driven infrastructure as code. As a bonus step, the tutorial has you rollback the upgrade with a single helm rollback command. Why do we think this is such an awesome idea?

The last post has a video of putting a git URL into OpenShift and it building the code and deploying a realworld.io ReactJS app. Did you spot what would ultimately disempower the team in that video? I used the web console with a 🐁 or 🐾 pad. That is bad as it gives short term gain but long term pain. Why?

Clicking on a web console to set up multiple webapps and APIs in identical staging and live environments will be repetitive and boring. When the pressure is on inconsistencies will appear. Over time there will be drift both within and across environments. Only one person does the clicking which isn't recorded or reusable by other team members. Ultimately setting up environments manually on the web console will become kryptonite to both accuracy and team empowerment. Cloud-native environments can be driven by APIs and built from templates. So why not automate everything?

If we accept that we must automate everything what should we add to our wishlist? How about:

• Automatic deployments when we push configuration changes into a git repo. We believe that a rainbow appears somewhere in the world whenever this happens ✨🌈🙌
• Using templates to hide the boilerplate so you can focus on what is unique about each webapp or API you deploy onto Kubernetes
• Using a declarative style of configuration with idempotent updates. Don't worry I will explain this later
• Putting everything into git including encrypting secrets so that we can manage infrastructure just like code with pull requests
• Automatically create a release build image from a git release webhook event and apply the same tag to the image
• Use consistent runtime versions across applications and making it easy to routinely apply security patches to the base image layers

Is that too much to ask? Of course not! You can test drive OCD using Minishift which lets you run Kubernetes on your laptop. Rather than try to cover all those points in a single post lets look at the short tutorial from the OCD wiki that covers the declarative deployment of prebuilt images. That demo covers the first four points. It has you set up the things shown in the video above on your laptop 💻✨🌈. Here is the sequence diagram:

On the left is a git push of configuration into a git rep. We use GitHub. You can use any git server you like. The demo uses Gitea as the git server running inside Minishift so that everything can run on your laptop. The webhook triggers an application called ocd-environment-webhook. This is an instance of the awesome adnanh/webhook tool configured to run scripts to pull the configuration from git and install it into Kubernetes using Helmfile and Tiller. Tiller will install our rainbows into Kubernetes. We will introduce Helmfile and Tiller in later posts. Here is the full configuration that the demo installs:

repositories:
  - name: ocd-meta 
    url: https://ocd-scm.github.io/ocd-meta/charts
releases:
  - name: {{ requiredEnv "ENV_PREFIX" }}-realworld
    labels: 
      deployer: {{ requiredEnv "ENV_PREFIX" }}-realworld
    chart: ocd-meta/ocd-deployer
    version: "1.0.0"
    values:
      - name: react-redux-realworld
      - replicas: 2
      - imageStreamTag: "react-redux-realworld:v0.0.1"
      - deploy_env: 
        - name: API_ROOT
          value: https://conduit.productionready.io/api

That configuration is Helmile yaml. The main body is a single release with a chart type ocd-deployer. A chart is a set of yaml templates packaged in a zip and downloaded from a website. The header names the OCD chart repo on GitHub as the location to download the chart. The template values applied to the chart are very simple. It specifies the container image to use as react-redux-realworld:v0.0.1 and that two replica pods are to be maintained. This is a prebuilt image of the realworld.io ReactJS demo app. It also sets an environment variable API_ROOT to be the public API that the app will use.

To get this deployed you need some prerequisites:

1. Minishift with Helm Tiller installed detailed here
2. Gitea installed on Minishift detailed here

Once you have Gitea running you can register a user and create an empty repo ocd-demo-env-short and push the demo configuration into it:

# clone the code
git clone https://github.com/ocd-scm/ocd-demo-env-short.git
cd ocd-demo-env-short
# make sure we can load details about the gitea url
oc project gitea
# this should print the gitea url. If it doesn't set it manually
GITEA_URL=$(oc get routes | awk '$1~/gitea/{print $2}')
echo $GITEA_URL
# see instructions to setup your own person access token
ACCESS_TOKEN=f64960a3a63f5b6ac17916c9be2dad8dc76c7131
# set this to your username in gitea needed to get the url to your repo below 
USER_NAME=you_not_me
# add the gitea repo as a remote
git remote add minishift http://$ACCESS_TOKEN@$GITEA_URL/$USER_NAME/ocd-demo-env-short.git
# push the code into Gitea
git push minishift master

Why are we doing that? Because​ we cannot set up a GitHub webhook on my repo to fire an event into your deployment pipeline running in Minishift on your laptop. We loaded the code into a Gitea repo so you can configure your own webhook.

Next we need to deploy our ocd-enviroment-webhook handler that will catch webhook events and deploy our configuration. We can set that up in its own project using a script:

oc logout ; oc login -u developer -p password
echo Use this git repo url http://$ACCESS_TOKEN@$GITEA_URL/$USER_NAME/ocd-demo-env-short.git
# this must match where tiller is installed
export TILLER_NAMESPACE=tiller-namespace
# create a new project
export PROJECT=ocd-short-demo
oc new-project $PROJECT
oc project $PROJECT
# upgrade to admin
oc logout ; oc login -u admin -p admin
oc project $PROJECT
pushd /tmp && oc project $PROJECT && curl -L https://github.com/ocd-scm/ocd-environment-webhook/archive/v1.0.1.tar.gz | tar zxf - \
&& cd ocd-environment-webhook-1.0.1 \
&& ./wizard.sh && popd

Note that echos out the Git repo URL that you need to feed into the wizard.sh script. Here is the transcript of my run where I mostly hit enter to accept the defaults:

The git repo url? http://[e-mail]/simbo1905/ocd-demo-env-short.git
The project where the images are built and promoted from? ocd-short-demo
Repo name? (default: simbo1905/ocd-demo-env-short): 
Branch ref? (default: refs/heads/master): 
Chart instance prefix? (default: ocd-short-demo): 
Use --insecure-no-tls-verify? (default: false): 

It set up the webhook in Gitea you need the webhook URL and the webhook secret. This outputs the URL:

$ oc get route ocd-environment | awk 'NR>1{print "http://" $2 "/hooks/ocd-environment-webhook"}'
http://ocd-environment-ocd-short-demo.192.168.99.100.nip.io/hooks/ocd-environment-webhook

And this the secret:

$ oc describe dc ocd-environment-webhook | awk '$1~/WEBHOOK_SECRET:/{print $2}'
M7MuW6aZnn

You then use those to set up a Gitea webhook of type application/json:

You can now fire the webhook from git on the commandline or by editing files using the Gitea web console. To do it on the commandline try:

echo "testing" >> README.md
git commit -am 'test commit'
git push minishift master

You should then see the webhook fire and our app deploy 💨✨🌈

After that you can follow the steps in the video above to edit the container image version number to see the application perform a rolling upgrade. For bonus marks you can try the steps at the end of the tutorial to rollback to the first release.

In that tutorial we covered four out of out six items on the wishlist above. In the next post we will automate a release build from a git webhook release event and apply the same tag to the image. That way we can track exactly what code is being promoted between environments. At the same time we will making it easy to use consistent runtime versions across applications. That will make it easy to security patch the runtime image.