DEV Community: Atharva Shinde

Chapter: Jenkins

Atharva Shinde — Tue, 08 Feb 2022 11:59:05 +0000

Want to get notifications instantly when your code breaks? Need some tool to integrate automation for your application or software? Need a wide range of plugins to support your build, test and deployment process? Easy to configure and Free? And have an amazing community around itself? Solution: Jenkins.

Why is Jenkins needed?

Above diagram depicts a raw process on how the development cycle worked traditionally. Consider your team working on a new functionality for an application and you want to have it deployed to the upcoming release version. Initially you push your code to a Version Control system like Github, BitBucket or your company's private VC system. All the manual work ends here.

Now starts the integration and delivery process. The problem here is that the build process takes a considerable amount of time to build your software. Therefore the development teams had a custom nightly build convention where all the build processes were scheduled at night, but "nightly" build is relative to different time-zones. Let's say, a company decides for night build at 12:30AM EST, but that build would be 10:30AM IST and 4:00PM AEST and hence all IST and AEST developers would have to commit their code for build during their work time zones which affected their productivity. Also the test cases were limited and did not have wide coverage. So you would have to wait till your code was built and pushed to the test server to check if it passed the predefined tests, and if the tests and build failed the source-code was pushed back to your team.

In the whole process if the code broke and the tests failed there were no ascertained means to pick-point which team or particularly who wrote that piece of code.

As we observe here, this whole process is definitely not an optimal way to deliver software and make it production ready.

Jenkins

Jenkins is an open-source versatile automation server completely built in Java, for integrating the workflow of building, testing and deploying a software. Jenkins transitioned the traditional software delivery cycle to a fast, extensive and customisable CI/CD process. With the help of Jenkins its possible to-

Get instant build notifications every-time your code is pushed from the repository to build.
Parallel Build i.e run more than one build process in parallel to reduce your build time.
Test the built code on a wide range of test combinations for achieving maximum test-coverage and statistical data on how the code performed for each test-input.
Get instant result notifications after every test-session.
Create roles for each responsibility and manage what to hide for each role on what they could read and/or write to.
Build your own set of integration tools using wide range of plugins available at https://plugins.jenkins.io

Fun Fact: We can integrate Jenkins with some popular cloud platforms including AWS, VMWare, Google Cloud Platform, Azure, IBM Cloud, Digital Ocean etc.

This is how Jenkins helped in rejuvenating to a frictionless CI/CD process.

Installing Jenkins

To install Jenkins on your local setup read Jenkins official download guide

Installing Jenkins as a container inside Docker.

To install Jenkins as a container from Jenkins Docker image, execute-
docker pull jenkins/jenkins:lts-jdk11

Architecture of Jenkins Model

Here's a high level view of Jenkins application

Components of Jenkins

Plugins
Jenkins Pipeline
JenkinsFile

Plugins

A Plugin is an extension tool to help streamline the process of building, testing or deploying a software. There are 1800+ plugins available built for integrating purposes such as analysing or testing the codebase, sending custom notifications of test or build results to developers, integrating Git with Jenkins and many more. We can install more than one plugin on top of our Jenkins local environment to implement these different functionalities that increase the efficiency of the CI/CD process. These plugins are built by awesome developers of Jenkins Open-source community and you can browse all these plugins and their information on Jenkins Plugins Website.

Few examples of plugins are-
GitHub Groovy Library, GitHub Branch Source Plugin, SSH Build Agents, LDAP, Credentials Binding Plugin, OWASP Markup Formatter, PAM Authentication.

Plugins can be installed using 2 ways-

** "Plugin Manager" in the web User Interface. **
Jenkins Command Line Interface.

java -jar jenkins-cli.jar -s http://localhost:8080/ install-plugin SOURCE ... [-deploy] [-name VAL] [-restart]

This command installs a plugin either from a file, an URL, or from update centre.

SOURCE: If this points to a local file, that file will be installed. If this is an URL, Jenkins downloads the URL and installs that as a plugin.
-deploy: Deploy plugins right away without postponing them until the reboot.
-name VAL : If specified, the plugin will be installed as this short name (whereas normally the name is inferred from the source name automatically).
-restart : Restart Jenkins upon successful installation.

Jenkins Pipeline

Jenkins Pipeline is a structured set of plugins, required for integration of your software, declared inside a file called JenkinsFile.

JenkinsFile

To define a Jenkins pipeline we need to write a set of steps inside a text file called JenkinsFile.Thus JenkinsFile implements "Pipeline as a Code". Every software using Jenkins is required to define JenkinsFile inside its project repository.

We can build your Pipeline's JenkinsFile through either-

Jenkins User Interface

BlueOcean User Interface

Using Source Code Management (SCM)

And to code the JenkinsFile we can use either of the two syntaxes:

Scripted Pipeline(should be written in Groovy , is flexible but have steep learning curve)
Declarative Pipeline(recently added feature, easy to learn but not so powerful/flexible)

Scripted Pipeline: These pipelines are initiated with the directive node. Here's an example structure of Scripted Pipeline-

node{
    stage('Build'){
        try{
        }
        catch(e){
        }
    }
    stage('Test'){
        if(condition){
        }
        else{
        }
    }
    stage('Deploy'){
    }
}

Declarative Pipeline: These pipelines are initiated with syntax pipeline Here's an example structure of a combination of: parallel and non-parallel declarative pipeline.

pipeline{
    agent any
    stages{
        stage('Build'){
            steps{
            }
        }
        stage('TestAndDeploy'){
            parallel{
                stage('Test'){
                    steps{
                    }
                }
                stage('Deploy'){
                    steps{
                    }
                }
            }
        }
    }
}

in the above pipeline stage('Build') is executed first and stages: stage('Test') & stage('Deploy') are run in parallel because of the parallel directive.

Just like the parallel keyword there are many others available each having their discrete functionalities, these keywords are called Directives; a few to mention are:

when - It allows the Pipeline to decide whether the stage should be executed depending on the given condition. Example: when{ branch main}
stage - This directive can comprise of one or more directives collectively used to define-build a particular process of development cycle. Example: stage('Test'){...} the stage 'Test' would consist a set of instructions that would trigger every time any code is being sent for testing.
tools - It is used to define tools to auto-install for Jenkins server.

Wrap Up

Alternative to Jenkins

Here are few other CI/CD tool options to Jenkins with each having its own advantages and disadvantages:

CircleCI
GitLab
Travis CI

Reference: Jenkins Documentation

What next?

Now that you are aware of Jenkins as a next step you might need to containerise your application. Containerising your application helps in shipping your application to different systems without worrying about dependencies, scaling and even deploying your application faster. I have made a detailed "Guide to Docker" so check that out!

Thank you for taking time to read my article :)

Do connect with me on-
Twitter,
GitHub, and
LinkedIn

Chapter: A guide to Docker

Atharva Shinde — Wed, 01 Dec 2021 19:49:53 +0000

What is Docker?

Docker is a platform that allows developers to containerise, build, test, deploy, ship applications much faster within multiple workspaces and deliver production ready applications. Docker wraps your application inside an abstraction called container which makes your development workflow quick and easier.

Why Docker?

If you need streamlined, fast, lightweight, efficient and orchestrated way to run, scale, deploy or test your application/software on one or more virtualised/physical systems, Docker is your solution.
Docker being lightweight and portable makes it a cost-effective alternative to hypervisor virtual machines.

With its simple CLI commands we can easily build, delete, deploy and manage containerised images on our machines. Moreover these containers are highly portable which enables them to be able to run on local systems, virtual machines, cloud providers or even in hybrid workspaces without worrying about setting up libraries and all other configuration files.

Docker Architecture

Docker implements Client-Server architecture. So what is Client-Server architecture? Consider online shopping. You (the client) orders a product from app-store and the service-provider (server) places the order, takes necessary actions and ships it to your address. Similarly a client-server architecture is an architecture where the client requests a service (in this case through Docker CLI) and the service provider (in this case Docker daemon) works on the request and provides desired result.
Referring the above diagram, if user wants to create a container from one of the installed images and therefore enters docker run command, docker daemon responds to the request and gives user a configured virtualised container along with its metadata.

What is Docker Daemon?

Daemon is a program that runs consistently in the background which responds to particular request or action, and is not under any direct control of user, that means user cannot change the nature of the program. Usually daemon files/processes names are suffixed by 'd', for example:- sshd, mysqld, dockerd.

Docker Daemon listens to Docker Client through API requests and manages docker objects like images, containers, volumes, network.

Docker Client

Docker client is the primary way to interact with Docker. When we run a Docker command through terminal, the client sends the request to Docker Daemon which is responsible for managing commands.

Docker Image

Docker Image is built of read-only stacked layers generated from instructions inside the image's dockerfile. Each layer is a representation of an instruction from the dockerfile. Images that are pre-built by developers and are available inside Container-Registries, a place to store and download images. One such popular public registry is DockerHub. Few companies also have private registries to store their images.

# to install an image in your system
docker pull <image-name:version>
# example docker pull ubuntu:20.04

# to see if the image is installed
docker image ls
# or
docker images

# to remove an image 
docker image rm <image-name or image-id>
# if you get an error message similar to daemon: conflict: unable to delete, then you must delete the container first
read more about containers below 

# to check layers of an image
docker image inspect <image-name or image-id>

Dockerfile

To build your customised docker image create a Dockerfile with the extension .dockerfile. A new image layer is stacked for an instruction you define inside Dockerfile, but not every instruction is responsible for creating a layer.

Following is an example of dockerfile

# this is a comment
FROM ubuntu:18.04
LABEL org.opencontainers.image.authors="org@example.com"
COPY . /app
RUN make /app
RUN rm -r $HOME/.cache
CMD python /app/app.py

.dockignore

If you have worked with GitHub before you may have used .gitignore file to exclude .env variables, dependency-directories, temporary files etc. Similarly for Docker we have .dockignore file to exclude files that are not relevant to the build.
Before a request from docker clients CLI reaches docker daemon it goes through .dockignore file and checks whether there is a context that needs to be excluded before passing to the daemon and thus preventing any large or sensitive information from reaching daemon.

Docker Containers

Containers are isolated processes which run on a single host machine. Containers consists of packages and dependencies required by your application to run on your system. We can create, start, stop, delete, move and modify containers which is performed inside a thin writable layer know as 'Container Layer' built on top of immutable, read-only image layers. Containers therefore are running instances of an image. Each container has its own binaries, dependencies, and container-layer therefore each container being an isolated process make them fast, light-weight and more efficient to work with.
Below is an example of Image layers structure-

P.S- If user downloads two or more version of the same image, docker only builds the layers which are new from previous versions and all the layers which are mutual won't be installed again.

# to build a container from installed image
docker run <image-name:version>

# to get list of all running containers metadata
docker ps 

# to get list of all containers metadata
docker ps -a

# to stop a container
docker stop <container-name or container-id>

# to restart a container
docker start <container-name or container-id>

# to remove a container 
docker container rm <container-name or container-id>

# to rename a container
docker stop <container-name or container-id>
docker run --name <your desired container-name> <image-name>

# to navigate inside a container's terminal
docker exec -it <container id or container name> /bin/bash

Containers helps in setting up environment on any OS without worrying about configurations and dependencies, and actually focusing on building applications.

So what's the catch?
Containers are not persistent. Container when deleted or restarted, loses all its data and starts again from its image definition. So when we delete/restart a container the data inside 'container-layer' is lost and it starts from a fresh state. Now if we try to install/restart it again, the container layer is created from scratch not having any history of operations we did inside previous one. How do we solve this?

Persistent data

Persisting database is the solution to the above conflict. The idea is to store the data inside the hosts filesystem apart from dockers virtual filesystem.
Basically there are 3 ways to persist the data-

Volume
Bind mount
tmpfs mount (Linux)

However Docker recommends the use of Volume as your primary choice for persisting database because of its fair advantages over the other, which includes easy and secured migrating and backup between containers and systems, flexibility on working on linux and windows, user can easily communicate to volumes using Docker Client CLI, etc.

# to create a volume 
docker volume <volume-name>

# to list all volumes
docker volume ls

# to delete a volume
docker volume rm <volume-name>

What next?

Let's consider you have more than 4-5 containers on your machine and you want to scale them up, configure their networks, test, deploy and manage them. Some organisations even have thousands of containers to manage. Managing containers becomes difficult as they scale up. So we need a tool that take care of our containers.
That's where Container Orchestration comes in picture. Container Orchestration deploys, scales, removes, checks containers health, load balances the traffic and manages all the containers. And the most widely used Container Orchestration tool is Kubernetes.
So now you know what to explore next!

Thank you for taking time to read my article :)

** Do connect with me on-
Twitter,
GitHub, and
LinkedIn
**