Deploying Jenkins on Google Cloud

Jenkins is one of them most popular tools for doing CI/CD. It provide plugins to build, deploy and automate deployment for any project. While jenkins is amazing, deploying and managing is not straightforward. Adding more workers often involves creating new VMs and adding them to the cluster.

In this post I'll demonstrate how can easily deploy jenkins on GCP using opta. I will also walk you through how you can tune various parameters to using opta itself to for scaling your jenkins deployment. With opta you can follow the same tutorial and deploy the entire thing on AWS with minor tweaks - we will discuss more about this in the end of this post.

Why Opta and not helm directly?

While going through the tutorial you might think why don't we just use helm on a kubernetes cluster? Well you've thought it right but if you look closely opta doesn't just deploy helm chart on a kubernetes cluster - it also creates a kubernetes cluster with all the important tooling up and running.

Opta ensures the following:

• It setups an Ingress to route traffic to the pods
• A secure network configuration with secure defaults
• Kubernetes provisioning and configuration with best practices

And we need to make sure this is all done securely so we don’t introduce unnecessary vulnerabilities! That’s why Opta is a good fit here — it sets up all these resources for us with a very robust architecture.

Jenkins Deployment Architecture

Pre-Requisites

Opta has some pre-requisites like terraform, kubectl, aws-cli/gcp-sdk. Ensure you have those configured. Refer this link for more information https://docs.opta.dev/installation/#prerequisites

To proceed further, make sure you have opta cli installed. For most users the script below should work. For detailed instructions you can follow this link:

/bin/bash -c "$(curl -fsSL https://docs.opta.dev/install.sh)"

Opta configuration for Deploying Jenkins

Basics

To get jenkins up and running on GCP using opta these are two important things

1. Deploy kubernetes cluster
2. Deploy jenkins service on top of it

Code

Let's look at the code structure. All our code lies inside folder opta-gcp

opta-gcp
├── opta.yml
└── staging
└── opta.yml

File contents: opta-gcp/staging/opta.yml

name: gcp-k8
org_name: opta
providers:
 google:
   region: asia-south2
   project: project-name
modules:
 - type: base
 - type: dns
   domain: gcp-example.runx.dev
   delegated: false
 - type: k8s-cluster
   max_nodes: 3
   node_instance_type: n2-standard-2
 - type: k8s-base
   linkerd_enabled: false

opta-gcp/opta.yml

name: hello-world
environments:
 - name: gcp-k8
   path: "staging/opta.yml"
modules:
 - type: helm-chart
   repository: https://charts.jenkins.io
   version: 3.5.2
   chart: jenkins
   values:
     controller:
       ingress:
         enabled: true
         apiVersion: "extensions/v1beta1"
         hostName: jenkins.example.com
     agent:
       idleMinutes: 20

Create Infra

Since we are trying to deploy our project on gcp make sure you have configured the gcp sdk correctly on your system before moving further (https://cloud.google.com/sdk/docs/install). Also ensure that in your opta-gcp/staging/opta.yml file you have provided correct gcp region and project name.

# Create infra, setup networking, deploy kubernetes
cd opta-gcp/staging
opta apply

As you run the above command you will see a lot of google cloud apis getting activated, opta automatically activates all the APIs which you need to create a full blown kubernetes cluster on gcp

Google service container.googleapis.com activated
Google service iam.googleapis.com activated
....
....

Further in your output you will see some messages related to terraform initialisation - opta converts the config file that you provide in yaml to good old terraform which we all love. Opta under the hood uses terraform to manage the infra and that's why you can rely on it. You will also see the detailed plan that terraform has generated. You can go through it. Opta creates a production ready terraform code to deploy kubernetes 3with all the best practices baked in. You can review the plan and accept to proceed further.

Now: Hold tight - because it can take 3-7 mins for things to finish. Once all goes well you can see that the kubernetes cluster has been provisioned for you

> gcloud container clusters list
NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS
gcp-k8 asia-south2 1.19.9-gke.1900 34.131.191.178 n2-standard-2 1.19.9-gke.1900 3 RUNNING

Deploy Jenkins

In previous step we bootstraped our infra and deployed kubernetes on it. In this step we are going to deploy jenkins on top of the infra which we created

cd opta-gcp/
opta apply

In this step opta generates terraform code to deploys helm chart. You can review the plan and accept it. It will take around 2-3 minutes for the plan to get applied. For specifying values to the helm chart you can modify the values inside opta-gcp/opta.yml . Refer to jenkins helm chart for more details https://github.com/helm/charts/tree/master/stable/jenkins. After opta apply finishes you will have jenkins up and running on your infra.

Configure Kubectl

You will have to configure kubectl to query things inside the cluster which you just created.

# Modify this command as per the parameters you passed on opta.yml
gcloud container clusters get-credentials opta-gcp-k8 --region asia-south2

# OR use this. It abstracts out all the complexities for you
opta configure-kubectl

Login to Jenkins

While opta sets up your kubernetes cluster it also installs basic components in your cluster like ingress-controller, cert-manager, etc to make your life simpler. For more details refer to: https://docs.opta.dev/modules-reference/environment-modules/gcp/#k8s-base . Since opta configures the ingress for us - let's go ahead and get the IP address of our load balancer which will be the entry point for traffic entering our cluster.

> kubectl get svc ingress-nginx-controller -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.0.64.13 34.131.159.112 80:31634/TCP,443:32007/TCP 47h

For this tutorial we are manually adding an entry into /etc/hosts file for jenkins.example.com however opta also provides you automated dns configuration which you can setup by following the instructions https://docs.opta.dev/miscellaneous/ingress/. For this tutorial we will add a manual entry into our etc/hosts file (On windows you can refer this for modifying /etc/hosts https://www.howtogeek.com/howto/27350/beginner-geek-how-to-edit-your-hosts-file/))

echo "<EXTERNAL-IP> jenkins.example.com" | sudo tee -a /etc/hosts

Now browse to jenkins.example.com

Now obtain the login credentials for jenkins

# Get username
> kubectl get secret hello-world-helmchart-jenkins -o jsonpath="{.data.jenkins-admin-user}" | base64 --decode
admin

# Get password
> kubectl get secret hello-world-helmchart-jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode
tSDwbahxAStXkj2gBlPky

You should be able to login to jenkins now.

To put this behind a domain name and use SSL, please follow the dns delegation steps here.

Debugging

To debug things or explore the underlying containers, opta enables you to use kubectl.

Just run opta configure-kubectl and it will set up the appropriate config for kubectl. Now you can run kubectl get pods -n airflow to look at all the pods we are running and kubectl exec -it -n airflow pod/<pod-name> to ssh into a pod and explore the environment.

Running on AWS

Running this same example on AWS is pretty straightforward — as Opta is (mostly) cloud agnostic! We just need to update the env file to point to our AWS project and run apply.

The “service” yml file doesn’t need to be changed as it’s completely cloud agnostic!

What’s next

This was a quick overview of how we can use Opta to deploy Airflow to AWS or GCP. We were able to get a robust environment set up with minimal work!

All this code can be found on our github. Make sure to check out the Airflow docs and the Opta docs — for further configuration.

https://github.com/jenkinsci/helm-charts/blob/main/charts/jenkins/values.yaml

If you run into any problems or have suggestions for what else you’d like to use Opta for, please let us know in the comments or in our slack :))