DEV Community: Tommy Heng-Chuan Lin

Pipeline example using Terraform, Jenkins, Ansible, and Docker on Google Cloud-Part 2

Tommy Heng-Chuan Lin — Sun, 05 Mar 2023 15:29:29 +0000

Today I will keep working on part 2

Clone or Fork project folder

https://github.com/aks60808/AiGames

make sure you have this project in your repository!

Jenkins Plugin Install

Get into your Jenkins page http://YOUR_CICD_SERVER_IP:8080/

The followings are plugins we needed for this walkthrough

Ansible
Docker
Docker Pipeline
NodeJS
ThinBackup
Google Cloud Storage

GO TO Dashboard ⇒ manage plugins

Search and Tick all the required plugin to install (Click Download & install with restart)

Configure Global tool

once you had the required plugins installed, GO TO Dashboard ⇒ Global Tool Configuration

We are going to configure some useful tool while building pipeline

Name the NodeJS plugins ⇒ MyNodeJS (this should match your naming in jenkinsfile)

Name the Docker plugins ⇒ MyDocker (this should match your naming in jenkinsfile)

Configure Depolyment Server

to get into deployment server with ansible, we need to configure hosts/inventory again.

Generate ssh keys for cicd server

first open terminal and ssh into cicd server

ssh YOUR_CICD_SERVER_IP

in your home directory, run

ssh-keygen -t rsa -f ~/.ssh/id_rsa

read the public key file

cat ~/.ssh/id_rsa.pub

copy the content on the terminal then paste to GOOGLE COMPUTE ENGINE page ⇒ metadata ⇒ ssh keys

then saved. (like what we did in part 1)

Put ssh private key into Jenkins

During Jenkins building process, I found that it is hard to get in user’s home directory unless you specify the absolute path. Hence, the more secure and well-managed way is to put credentials in Jenkins-managed Credentials.

Click Manage Credentials →SYSTEM → Global credentials → Add Credentials

CICD ssh private key

we need to put the private key we’ve created previously (~/.ssh/id_rsa).

select SSH username with private and specify your credential ID, ssh login username and private key content.

Please be noted that the ID should match the Credential ID setup in the Jenkinsfile pipeline script.

cat .ssh/id_rsa

copy the content of the private key put into key section then saved.

Configure hosts on CICD server

We will use ansible to configure Deployment server throughout the CICD pipeline.

Go to ansible folder

cd /etc/ansible/

Edit hosts file with sudo

sudo vim hosts

Put following content into the hosts file

[depolyment]
x.x.x.x 
# replace above with depolyment server IP address
[depolyment:vars]
ansible_user=tommylin   
# replace above with your login username
ansible_ssh_common_args='-o StrictHostKeyChecking=no'

Don’t forget to put deployment server ip and your ssh login username in the file!

Configure Deployment server

in your local project directory, use text editor to open ansible_playbook/deployment_server_init_config.yml

---
- name: Install Docker
  hosts: depolyment
  become: yes
  vars:
    username: "tommylin"
    # replace the above username with yours.

Replace with your ssh login user name for purpose of adding $USER to docker group.

After that, in project directory run the following:

ansible-playbook -i inventory ansible_playbook/deployment_server_init_config.yml

The playbook will configure deployment server with docker installation.

Create a pipeline job

Enable Discard Old builds

We will create docker image release throughout building action, we don’t need to keep every build as it could be tons of large build increase the burden of backup.

Tick GitHub hook trigger

This is where our github webhook magic enabled on Jenkins’ side!

GET the pipeline Script from Github

Put your project repo URL into Repositories URL blank. If your repo is private then Credentials is needed(Can configure it using Jenkins managed credentials). Other box I just leave them as default settings.

GITHUB WEBOOK setup

In your GitHub Repo page, click Settings icon ⇒ Webhooks ⇒ Add webhook

PUT IN http://CICD_SERVER_IP:8080/github-webhook into the Payload URL blank.

DON’T FORGET to select JSON format as your Content type

Click Add webhook button then we are almost there!!

Configure the place where you store releases

go to Google Container Registry console to configure visibility to public.

Build Action triggered by GitHub push!

you can make dummy commit then push to the master testing the automation. You can click the build number to get detailed information as Console output.

Check the release on Google Container Registry after build success.

Well done! CHECK THE DEPLOYMENT SERVER!

you can go visit http://YOUR_DEPLOYMENT_SERVER_IP:3000/AiGames/game/sudoku to have some fun with these games

Backup

Setup ThinBackup

In this walkthrough, Google Cloud Storage are used to backup jenkins.

GO TO JENKINS Dashboard and SCROLL DOWN, you will see the ThinBackup plugin we just installed.

Click Settings

Configure the following

Set Backup directory:

${HOME}/.jenkins-backup

Backup schedule ( I backup @7:00 AM everyday)

I only backed up build results, userContent folder and Backup plugins archives since I don’t need any build artifact being backup from Jenkins. I only backup my worksapce configuration.

after click SAVE button, thinbackup will backup jenkins on your VM.

Bring Backup to Google Cloud Storage

PUT Service account key into Jenkins-managed credentials

Specify you GCP project ID and service account key .json file then click create.

Create a pipeline job

This job will start 9:00 AM everyday, right after thinbackup!

In Pipeline Script section, Please copy and paste this snippet.

pipeline {
    environment {
     JENKINS_BACKUP_FILE_NAME = ''
         SA_KEY_CRED_ID = 'YOUR PROJECT ID(JENKINS MANAGED CRED)'
            GCS_BUCKET_URL='YOUR GCS BUCKET URL'
    }
    agent any
    stages {
        stage('Backing up Jenkins') {
            steps {
                script {
                    JENKINS_BACKUP_FILE_NAME = sh (
                        script: '''#!/bin/bash
                            # find latest backup
                            thinbackup_path="$JENKINS_HOME/.jenkins-backup"
                            latest_backup_folder=$(ls -t $thinbackup_path | head -n1)
                            full_path="$thinbackup_path/$latest_backup_folder"

                            # pack it up
                            backup_name="$latest_backup_folder.tar.gz"
                            tar cvzf $backup_name $full_path
                            echo ${backup_name}
                            ''',
                        returnStdout: true
                    ).trim().tokenize().last()
                }
                step([$class: 'ClassicUploadStep', credentialsId: "${SA_KEY_CRED_ID}",  
                bucket: "${GCS_BUCKET_URL}",
                      pattern: JENKINS_BACKUP_FILE_NAME])
                sh '''
                 rm '''+JENKINS_BACKUP_FILE_NAME+'''
                '''
            }
        }
    }
}

Go to Google Cloud Storage

you can directly click build now to run the upload mission. you should see these kind of backup files in your cloud storage.

Please be noted that in cicd_server.tf I’ve stated the lifecycle_rule age = 2 days for cloud storage. Dpn’t get confused if it only keep 2 copies all the time.

Cleanup

In the end, don’t forget to cleanup using:

terraform destroy

DONT FORGET ENTER YES !

Summary:

In part 2, I use Jenkins, Anisble, and docker to host a web on Google Compute Engine. I understand that there are tons of act can be improved to obtain a more resilient and reliable pipeline. This is the very first pipeline I had ever built. If you are a beginner as me, hope this walkthrough can get you some feeling about how a pipeline being constructed. Thank you for your time reading this :)

Pipeline example using Terraform, Jenkins, Ansible, and Docker on Google Cloud-Part 1

Tommy Heng-Chuan Lin — Sat, 25 Feb 2023 13:59:14 +0000

Project brief intro:

This project I made is for the demonstration of pipelines. I utilised Terraform Cloud provisioning infras on GCP, using Ansible to configure remote VM instances for initiation and deployment, Docker image as artifact for deployment and finally, a Jenkins server integrating webhook from GitHub and further CICD process.

This project was split into two articles for the sack of large context, I tried to template the source code so that it should be easy to follow stepwise. The first part would be the prerequisite walkthrough, provisioning infra and initialising Jenkins on CICD server.

Workflow

Prerequisite

OS: Linux/MacOS
Local installation of Terraform and Ansible
Google Cloud Platform Account
Terraform Cloud Account
GCP CLI

Enable GCP APIs

Compute Engine API
Google Container Registry API
Google Cloud Storage API

Create Organization and Workspace on Terraform Cloud

Terraform | HashiCorp Cloud Platform

Create a new GCP project

go to GCP console then create a new project and have your gcloud cli configured to that new project

Google Console Page

Clone Pipeline Project repo

Pipeline Project Links



git clone https://github.com/aks60808/Devops-Aigames

Configure GCP

To make terraform cloud provisioning infra from GCP, the following are required to provide:

Google Service Account
Grant permission for the role
Create a Service Account Key for further authentication

Login to GCP first



gcloud auth login

Create Service Account

replacing SA_NAME with your preferred Service Account name.



gcloud iam service-accounts create SA_NAME \
    --description="DESCRIPTION" \
    --display-name="DISPLAY_NAME"

Grant IAM as an editor role

replacing PROJECT_ID with your PROJECT_ID on gcp console



gcloud projects add-iam-policy-binding PROJECT_ID \
    --member="serviceAccount:SA_NAME@PROJECT_ID.iam.gserviceaccount.com" \
    --role="roles/editor"

Create a service account key

It’s up to you to determine the location of the service account key, which typically would be stored in ~/.ssh/ directory.



gcloud iam service-accounts keys create YOUR_SA_KEY_PATH \
    --iam-account=SA_NAME@PROJECT_ID.iam.gserviceaccount.com

Configure Terraform to Terraform Cloud

in these .tf files, they will provision several resources including

Google Compute Instance *2 (e2.medium) for service hosting
Google VPC and corresponding firewall rules allowing http and ssh
Google Container registry for docker image push in Part 2.
Google Cloud Storage for backing up Jenkins in Part 2.

Frist, Let's get into the project directory:



cd /Devops-Aigames

Modify the following main.tf



# store terraform states on terraform cloud
terraform {
  cloud {
    # replace with your organization
    organization = "REPLACE WITH YOUR ORGANIZATION NAME"

    workspaces {
    # replace with your workspace name
      name = "REPLACE WITH YOUR WORKSPACE NAME"
    }
  }
}

create a file named terraform.tfvars in project directory



touch terraform.tfvars

add the following content to terraform.tfvars
Regions & their corresponding zones can be referred to Google compute Regions-Zones List



project_id = "REPLACE WITH　YOUR　GCP　PROJECT　ID"
region = "REPLACE WITH　YOUR　PREFERRED　REGION"
zone = "REPLACE WITH　YOUR　PREFERRED　ZONE"

# these are variables that will be used in the terraform configuration

Add your GCP service account key to Terraform Cloud

Go to the Terraform Cloud service, select your right organization/workpace, click Variables on the left Menu Bar

Click Add variable button
Select Terraform variable
In KEY section, you MUST type in GOOGLE_CREDENTIALS
COPY and PASTE all content of the Service Account key(JSON file) into the Value section.
Don’t forget to tick Sensitive and click Save

For the reasoning of the above configuration, please refer to Terraform documentation

Terraform Registry

Login to Terraform Cloud

run terraform login and have the token value pasted into terminal



terraform login

Initial your local terraform to terraform cloud

run the following command to get terraform connected to terraform cloud



terraform init

Start Provisioning your infra on GCP using terraform



terraform plan
terraform apply
terraform output # to see your VM ip

Ansible Configure management

Now we will utilise ansible to configure the remote CICD server

Generate ssh key if needed



# USERNAME should match your current username
ssh-keygen -t rsa -f ~/.ssh/KEY_FILENAME -C USERNAME -b 2048

Visit GCE instance page click metadata>> ssh keys>>add ssh key

copy your generated .pub key then saved.

Create an inventory file in project directory



touch inventory

put the following content into inventory file



[cicd]
x.x.x.x 
# replace above with CICD server IP address
[cicd:vars]
ansible_ssh_private_key_file= /path/to/your/private/ssh/key
[depolyment]
x.x.x.x 
# replace above with depolyment server IP address
[depolyment:vars]
ansible_ssh_private_key_file= /path/to/your/private/ssh/key

Configure ansible.cfg file

check the location of ansible.cfg by running the following:



ansible-config --version

if you saw the config file = None, then in current directory, run



ansible-config init --disabled > ansible.cfg

modify the ansible.cfg by setting:



host_key_checking=False

then saved.

Try reaching out CICD and Deployment server via ansible

run the following command to test the ssh connectivity of two hosts.



# -i flag specify your inventory path
ansible all -m ping -i ./inventory

you should see the following output:

Configure CICD server using Ansible

before start configuring, go to ansible_playbook/ then open

ansible_deployment.yml to modify the project_id, replace it with yours.



- name: deploy app to server
  hosts: all
  become: yes
  vars:
    project_id: YOUR_GCP_PROJECT_ID

cicd_server_init_config.yml to modify the service account key and ansible_deployment.yml playbook location(ansible_deployment.yml is in project dir/ansible_playbooks)

the Path should be in absolute path format, key_dest and playbook_dest are set to jenkins home directory, so you don't need to modify them.



- name: Activate Service account
  hosts: cicd
  become: yes
  vars:
    key_src: YOUR/LOCAL/ABSOLUTE/PATH/TO/sa_key.json 
    key_dest: /var/lib/jenkins/key.json

......

- name: Upload playbook to CICD server
  hosts: cicd
  become: yes
  vars:
    playbook_src: YOUR/LOCAL/ABSOLUTE/PATH/TO/ansible_deployment.yml
    playbook_dest: /var/lib/jenkins/ansible_deployment.yml

the ansible playbook will

pass your service account key to CICD server
install Docker, Jenkins, and Ansible
Configure what’s necessary for running the operations including adding jenkins to docker group and cgroup mounting
pass deployment playbook to CICD server

in the project directory,



ansible-playbook -i inventory \
                    ansible_playbook/cicd_server_init_config.yml

After completion, you should open your browser via

PS. Initial ADMIN PWD will can be found either in the file location or the stdout on your terminal

Cleanup

DO NOT CLEAN UP UNLESS YOU WANT TO SKIP PART II!

If you wish to terminate the whole setup, in the project directory run the following command:



terraform destroy

DON'T FORGET TO ENTER YES TO PROCEED!

To be continued

In the next article, I will continue building Jenkins CICD pipeline to build, test, package and finally deploy for hosting a website offering algorithm-based fun games written by my good friend, Noah(big shoutout to you:)).

Part 2 article link