DEV Community: Akshat Gautam

Kafka Demystified: A Developer's Guide to Efficient Data Streaming

Akshat Gautam — Sun, 01 Sep 2024 06:00:00 +0000

When diving into the world of distributed systems and real-time data processing, Apache Kafka stands out as a powerful tool. Often mistaken for just another messaging system, Kafka is much more than that—it's a distributed streaming platform capable of handling high-throughput, low-latency data streams.

In this article, we'll explore Kafka's architecture, shed light on the relationships between producers and consumers, and walk you through setting up Kafka with a simple producer-consumer model. By the end, you'll have a solid understanding of Kafka's core concepts and a bit of practical knowledge to start using Kafka in your projects.

What is Apache Kafka?

At its core, Apache Kafka is an open-source distributed streaming platform designed to process streams of data in real-time. Unlike traditional messaging systems, Kafka’s distributed architecture allows it to handle high-performance data pipelines, streaming analytics, data integration, log aggregation and mission-critical applications with resilience, scalability, and fault tolerance.

What does it means by "distributed" streaming platform ?
Kafka clusters consist of multiple brokers, each of which handles a portion of the data. The brokers work together to ensure that even in the event of a failure, your data is safe and can be processed without interruption. This makes Kafka ideal for scenarios where real-time data processing is critical.

Key Concepts in Kafka

Streams: Continuous flows of data that Kafka processes in real-time.
Brokers: Servers that form a Kafka cluster, each responsible for storing a portion of the data.
Topics: Categories or feeds to which records are published. Topics are partitioned for parallel processing.
Producers: Applications that send records to Kafka topics.
Consumers: Applications that read records from Kafka topics.

Kafka's Architecture

Kafka's architecture is what makes it a game-changer in the world of distributed systems. Let’s break down the key components:

Topics and Partitions

In Kafka, data is organized into topics. Each topic is divided into partitions, which allow Kafka to parallelize processing and distribute data across multiple brokers. This partitioning is crucial for Kafka's scalability and fault tolerance.

Partitions: Each partition in a topic is an ordered, immutable sequence of records that are continually appended. Partitions are the fundamental unit of parallelism in Kafka.
Offsets: Each record in a partition has an offset, which is a unique identifier within the partition.

Brokers and Replication

Kafka's brokers are the backbone of its distributed architecture. A Kafka cluster is composed of multiple brokers, each identified by an ID.

Replication: Kafka replicates partitions across multiple brokers to ensure reliability and fault tolerance. Each partition has a leader and replicas. The leader handles all read and write requests, while replicas sync data from the leader. If the leader fails, a replica takes over automatically.

Producers and Consumers

Producers and consumers are the primary actors in Kafka's ecosystem.

Producers: They publish data to topics in a round-robin manner across partitions. Producers can also assign a key to messages, ensuring that messages with the same key go to the same partition.
Consumers: They subscribe to topics and read data from partitions. Kafka consumers can belong to a consumer group, which allows for load balancing. Each partition in a topic is consumed by exactly one consumer within a group.

Consumer Groups

Consumer groups are vital for scaling your Kafka consumer applications. When a consumer group is used, Kafka ensures that each partition is consumed by only one consumer in the group. This enables horizontal scaling, where multiple consumers can process data from the same topic in parallel.

Setting Up Kafka: A Practical Tutorial

Now, let’s move on to setting up Kafka and writing simple producers and consumers.

Being a Docker fan, Let's spin up Kafka using Docker

Create a Docker Compose File

First, create a docker-compose.yml file that defines the services for both Kafka and Zookeeper (Kafka's dependency).

version: '3.8'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - "2181:2181"

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    ports:
      - "9092:9092"

Start Kafka and Zookeeper

Run the command to start the services.

docker-compose up -d

Verify Kafka is Running

You can check if Kafka is running properly by listing the running Docker containers

docker ps

You should see both Kafka and Zookeeper containers up and running.

Access Kafka Command Line Interface (CLI)

To interact with Kafka, you can use the CLI by executing a bash shell inside the Kafka container:

docker exec -it <kafka_container_id> /bin/bash

Create a Topic

Once inside the Kafka container, you can create a topic:

kafka-topics --create --topic test-topic --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1

This command creates a topic named test-topic.

Write a Kafka Producer

We'll write a simple producer in Python to send messages to our test-topic. Install the kafka-python library first:

pip install kafka-python

Then, create a producer script:

from kafka import KafkaProducer

producer = KafkaProducer(bootstrap_servers='localhost:9092')

for i in range(10):
    message = f'Message {i}'
    producer.send('test-topic', value=message.encode('utf-8'))
    print(f'Sent: {message}')

producer.flush()

This script sends 10 messages to the test-topic_.

Write a Kafka Consumer

Now, let’s write a consumer to read these messages:

from kafka import KafkaConsumer

consumer = KafkaConsumer('test-topic', bootstrap_servers='localhost:9092')

for message in consumer:
    print(f'Received: {message.value.decode("utf-8")}')

This consumer subscribes to the test-topic and prints any messages it receives.

Run the Producer and Consumer

First, run the consumer script. Then, in another terminal, run the producer script. You should see the messages being produced by the producer and consumed by the consumer.

Real-World Applications of Kafka

Kafka's versatility has made it a cornerstone in many industries. Here are some of its most common applications:

Real-Time Analytics: Companies use Kafka to process and analyze streams of data in real-time, providing instant insights into customer behavior, system performance, and more.
Log Aggregation: Kafka aggregates logs from multiple services, making it easier to analyze and monitor system health.
Event Sourcing: Kafka is often used to implement event-driven architectures, where state changes in an application are captured as events.
Stream Processing: With Kafka Streams, you can build robust stream processing applications that filter, aggregate, and join data in real-time.
Data Integration: Kafka serves as the backbone for connecting various data sources and sinks, enabling seamless data integration across systems.
Mission Critical Use Cases: Support mission-critical use cases with guaranteed ordering, zero message loss, and efficient exactly-once processing.

Companies using Kafka

Uber: Has one of the largest Kafka deployments in the world, using it to exchange data between drivers and users.
LinkedIn: Uses Kafka for message exchange, activity tracking, and logging metrics, processing over 7 trillion messages daily.
Netflix: Uses Kafka to track activity for over 230 million subscribers, including watch history, movie likes and dislikes, and what they watch.
Spotify: Uses Kafka as part of its log delivery system.
Pinterest: Uses Kafka as part of its log collection pipeline.
Financial Institutions: Use Kafka to ingest transaction data from multiple channels and detect suspicious activities.

Security Tips for Kafka

As with any distributed system, security is the most important when deploying Kafka. Here are some key security practices to follow:

Enable SSL Encryption: Protect your data in transit by configuring SSL for Kafka brokers, producers, and consumers.
Use Authentication and Authorization: Implement SASL (Simple Authentication and Security Layer) to authenticate users and ACLs (Access Control Lists) to authorize access to Kafka resources.
Encrypt Data at Rest: Use encryption tools like Apache Kafka Connect to encrypt data stored in Kafka topics.
Monitor and Audit: Regularly monitor Kafka logs and set up auditing to detect and respond to unauthorized access attempts.

Conclusion

Kafka's distributed, scalable, and fault-tolerant architecture makes it an essential tool in modern data-driven applications. By understanding its architecture and learning how to set up producers and consumers, you can harness the power of Kafka in your projects.

There's also an amazing eBook that I found: Apache Kafka: A Visual Introduction

With this article, I tried to dive deep into Kafka, you’re now well-equipped to implement Kafka in a robust and secure manner.

Drop a like if you found the article helpful.
Follow me for more such content.

Happy Learning !

Exclusively authored by,

👨‍💻 Akshat Gautam

Google Certified Associate Cloud Engineer | Full-Stack Developer

Feel free to connect with me on LinkedIn.

Optimizing Docker Images for Size and Security: A Comprehensive Guide

Akshat Gautam — Thu, 22 Aug 2024 09:00:00 +0000

Docker is a powerful tool that enables developers to containerize their applications and ensure consistency across various environments.

However, without careful consideration, Docker images can become bloated, slow, and vulnerable to security risks. In this guide, I’ll walk you through the strategies to optimize Docker images for both size and security, ensuring efficient and safe deployments.

Optimizing Docker Images for Size

The size of your Docker image directly affects how quickly it can be pulled and deployed, which will significantly reduce the pipeline run-time and artifact storage costs, so reducing the image size is crucial for performance and resource efficiency.

At the end of this section, I will show you my portfolio website's image size being reduced by almost 96%!

Here’s how you can minimize your image size:

1) Use Official Minimal Base Images

When building Docker images, always start with an official base image. Instead of using a full-sized OS image like ubuntu, opt for lightweight versions like alpine or debian-slim. These minimal images contain only the essentials, significantly reducing the image size.

Taking an example for node image, Here are the image sizes for node:latest vs node:alpine:

That's almost 7 times bigger !

By using minimal base images, you avoid unnecessary packages, leading to faster builds and smaller images.

2) Minimize Layers

Each instruction in your Dockerfile (RUN, COPY, etc.) creates a new layer in the final image. Combining related commands into a single layer reduces the number of layers and therefore the image size.

Instead of doing this



RUN apt-get update
RUN apt-get install -y curl
RUN rm -rf /var/lib/apt/lists/*

Do this



RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*

3) Exclude Unnecessary Files with '.dockerignore'

When building Docker images, Docker copies the entire context (everything in your project directory) into the image unless you specify otherwise. To prevent unnecessary files from being included, create a .dockerignore file.

Example .dockerignore



node_modules
.git
logs
tmp

This file works similarly to .gitignore

4) Use Static Binaries and the 'scratch' Base Image

If your application can be compiled into a static binary, you can use the scratch base image, which is essentially an empty image. This leads to extremely small final images.

Example



FROM scratch
COPY myapp /
CMD ["/myapp"]

Works well for applications that don’t need operating system-level dependencies.

5) Multi Stage Builds (Most Effective)

Multi-stage builds allow you to separate the build process from the runtime environment. This is especially useful when your application requires tools for compiling but doesn’t need them in the final image.

Example



# Stage 1: Build
FROM golang:1.16-alpine AS builder
WORKDIR /app
COPY . .
RUN go build -o myapp .

# Stage 2: Runtime
FROM alpine:latest
WORKDIR /app
COPY --from=builder /app/myapp .
CMD ["./myapp"]

Quantitative Comparison

My Portfolio Website which was built using React was previously built using node:14-alpine image which was still a smaller image than the node:latest image.

The Dockerfile went like:



# Use an official Node runtime as a parent image
FROM node:14-alpine

# Set the working directory
WORKDIR /app

# Copy package.json and package-lock.json to the working directory
COPY package*.json ./

# Install dependencies
RUN npm install

# Copy the rest of the application code to the working directory
COPY . .

# Build the React app
RUN npm run build

# Install a lightweight HTTP server to serve the app
RUN npm install -g serve

# Set the default command to serve the build folder
CMD ["serve", "-s", "build"]

# Expose the port the app will run on
EXPOSE 3000

The image built was of size:

Much later after this I learnt about Multi-Stage Builds and redesigned my Dockerfile.

The new Dockerfile looked like:



# Build environment (Stage - I)
FROM node:14-alpine as build
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build

# Production environment (Stage - II)
FROM nginx:alpine
COPY --from=build /app/build /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

Astonishingly, The new image size was ...

The application worked exactly as before and was much faster to spin up this version !

The difference created was of ~1079 MBs which is a decrease of almost 96% !

This is an illustration of the effect of Multi Stage Builds

Optimizing Docker Images for Security

1) Use Trusted and Official Base Images

Always use official base images from trusted sources like Docker Hub or your organization’s trusted registries. These images are regularly updated and are more secure compared to custom or unofficial images. Keep your base images up-to-date to mitigate any vulnerabilities.

2) Run Containers as Non-Root Users

Running containers as root can expose your host system to security risks. Create a non-root user inside the Dockerfile and configure your container to run under that user.

Example:



RUN adduser --disabled-password myuser
USER myuser

Such simple change reduces the attack surface and improves security by limiting access to system resources.

3) Scan Images for Vulnerabilities

Regularly scan your Docker images for known vulnerabilities using tools like:

Trivy: An open-source vulnerability scanner.
Docker Scan: Built into the Docker CLI.
Clair: A static analysis tool for discovering vulnerabilities.

These tools scan your images for outdated or insecure packages and alert you to potential threats.

4) Limit Network Exposure

Limit the network exposure of your container by restricting the ports and IP addresses it listens on. By default, Docker exposes ports to all interfaces. Bind them to localhost if external access is unnecessary.

Example:



docker run -p 127.0.0.1:8080:8080 myimage

This restricts access to the container’s services to the local machine only, preventing external access.

4) Secrets Management

Avoid hardcoding sensitive information like API keys or passwords directly into your Dockerfile or environment variables. Instead, use Docker secrets or external secrets management tools like AWS Secrets Manager or HashiCorp Vault.

Example Using Docker Secrets:



docker secret create my_secret secret.txt

Docker secrets ensure that sensitive data is only available to services that need it, without leaving traces in the container filesystem.

Conclusion

By following these strategies, you can build Docker images that are both lightweight and secure. Optimizing for size helps reduce deployment times, save resources, reduce costs and improve performance, while security best practices protect your application and infrastructure from vulnerabilities.

Remember, containerization offers many advantages, but it also introduces new challenges. With thoughtful image optimization, you can leverage Docker to its full potential while maintaining a robust security posture.

Drop a like if you found the article helpful.
Follow me for more such content.

Happy Learning !

Exclusively authored by,

👨‍💻 Akshat Gautam

Google Certified Associate Cloud Engineer | Full-Stack Developer

Feel free to connect with me on LinkedIn.

KhataLook - Face Recognition Meets Retail Debt Tracking in React

Akshat Gautam — Sun, 18 Aug 2024 12:17:45 +0000

Welcome to my weekend project, KhataLook!

For those unfamiliar with Hindi, "KhataLook" combines two words: Khata, meaning ledger, and Look, meaning search. So, KhataLook translates to "looking through the ledger" — but with a modern twist.

Borrowed money is a significant challenge for retail shop owners, and keeping track of it can be a cumbersome task. This inspired me to create KhataLook, a face recognition system designed to simplify this process. With KhataLook, shop owners can register faces of borrowers in the system, and when a recognized face enters the store, the system automatically announces the amount pending 💀

So, without wasting much time procrastinating, I built a prototype for this idea, and here it is!

GitHub Repository: KhataLook

Made the Logo using Canva

Project Overview

KhataLook allows shop owners to get a relief from remembering the people with borrows.
For the prototype, I decided to build a web application.
Tech Stack:

Frontend: React
Face Recognition: face-api.js
Database: Cloud Firestore
Text-to-Speech conversion: Google Cloud TTS API

Project Setup and Dependencies

As soon as I initiated the React project, First step was to install dependencies.

Material UI (I just love it)
Axios
face-api.js
Firebase
React-Router

Database Setup

I chose to move on with NoSQL databases. As I am a Google Cloud Enthusiast, I went ahead with Cloud Firestore

Initiated a Firebase Project

Went to Run > Cloud Firestore and created a database

My database was ready within minutes

Went to Project Settings > Add app and copied my Firebase Configuration

// Sample Configuration
import { initializeApp } from "firebase/app";
import { getFirestore } from "firebase/firestore";

const firebaseConfig = {
  apiKey: "YOUR_API_KEY",
  authDomain: "YOUR_PROJECT_ID.firebaseapp.com",
  projectId: "YOUR_PROJECT_ID",
  storageBucket: "YOUR_PROJECT_ID.appspot.com",
  messagingSenderId: "YOUR_MESSAGING_SENDER_ID",
  appId: "YOUR_APP_ID",
};

const app = initializeApp(firebaseConfig);
const db = getFirestore(app);

export { db };

Face Registration

Initiated with downloading the models and putting them in the public/models folder

Loading the models

// Load face-api.js models
export const loadModels = async () => {
  await faceapi.nets.ssdMobilenetv1.loadFromUri('/models');
  await faceapi.nets.faceLandmark68Net.loadFromUri('/models');
  await faceapi.nets.faceRecognitionNet.loadFromUri('/models');
};

Fired up the video stream

const startVideo = () => {
    navigator.mediaDevices
        .getUserMedia({ video: true })
        .then((stream) => {
            videoRef.current.srcObject = stream;
            videoRef.current.play();
            setCapturing(true);
        })
        .catch((err) => {
            console.error("Error accessing the webcam: ", err);
        });
};

Helper function to push data

export const registerFace = async (name, mobileNumber, amount_pending, descriptor) => {
  try {
    await addDoc(collection(db, 'users'), {
      name,
      mobileNumber,
      amount_pending,
      faceDescriptor: Array.from(descriptor),  // Convert Float32Array to array
    });
    alert('Face registered successfully!');
  } catch (e) {
    console.error('Error adding document: ', e);
  }
};

And TaDa

Face Recognition

Fired up the video stream from camera, Pulled all the registered faces from the database and searched through them

const recognizeFace = async () => {
    const context = canvasRef.current.getContext('2d');
    context.drawImage(videoRef.current, 0, 0, canvasRef.current.width, canvasRef.current.height);

    const descriptor = await detectFace(canvasRef.current);
    if (!descriptor) return;

    // Compare the detected face with registered faces
    const faceMatcher = new faceapi.FaceMatcher(registeredFacesRef.current.map(face => new faceapi.LabeledFaceDescriptors(
        face.name, [face.faceDescriptor])), 0.6); // Set a threshold for similarity

    const bestMatch = faceMatcher.findBestMatch(descriptor);
    if (bestMatch.label !== 'unknown') {
        const recognizedFace = registeredFacesRef.current.find(face => face.name === bestMatch.label);
        setRecognizedName(recognizedFace.name);
        setAmountPending(recognizedFace.amount_pending);
        playAudioMessage(recognizedFace.name, recognizedFace.amount_pending);
    } else {
        setRecognizedName('Face not recognized');
        setAmountPending(0);
    }
};

And here the result is

Audio Announcement

Went straight to Google TTS AI, Created an API Key

Pasted the API Key in this helper function

const getSpeechAudio = async (text) => {
    try {
        const response = await axios.post(
            `https://texttospeech.googleapis.com/v1/text:synthesize?key=YOUR_API_KEY_HERE`,
            {
                input: { text: text },
                voice: { languageCode: 'hi-IN', ssmlGender: 'FEMALE' }, // Language and gender of the voice
                audioConfig: { audioEncoding: 'MP3' },
            }
        );
        const binaryString = window.atob(response.data.audioContent);
        const binaryLen = binaryString.length;
        const bytes = new Uint8Array(binaryLen);
        for (let i = 0; i < binaryLen; i++) {
            bytes[i] = binaryString.charCodeAt(i);
        }
        return bytes.buffer;
    } catch (error) {
        console.error('Error generating speech:', error);
        return null;
    }
};

Finally played the audio

const playAudioMessage = async (name, amount) => {
    let message = '';

    // This text is in Hindi, Modify it as per your preference
    message = `${name} Ji, aapke ${amount} rupye udhaar hai.`; // Default message

    const audioContent = await getSpeechAudio(message);

    if (audioContent && audioRef.current) { // Check if audioRef is defined
        const audioBlob = new Blob([audioContent], { type: 'audio/mp3' });
        const audioUrl = URL.createObjectURL(audioBlob);
        audioRef.current.src = audioUrl;
        audioRef.current.play();
    } else {
        console.error('Audio element is not available or audio content is invalid.');
    }
};

Clone the project, Set it up on your local system and listen to the audio and use the application by yourself.

Conclusion

While it was just a very simple prototype of a random spontaneous idea.

If anyone is interested in taking this project further, feel free to contribute by submitting a pull request.

I’d love to hear your thoughts in the comments! Let me know if you think this idea is feasible.

Could just be another fun project or maybe a popular product !

Building an Enterprise CI/CD Pipeline with Jenkins, Docker, Trivy, and GKE

Akshat Gautam — Fri, 16 Aug 2024 18:45:18 +0000

Jenkins, the most popular open-source tool for build automation, plays a vital role in efficiently building, testing, and rolling out new applications across various environments.

One of those environments is Kubernetes, which is widely available as a managed service from all the leading cloud providers. In this article, we will explore how to build an automated, enterprise-ready CI/CD pipeline using Jenkins, Docker, and GitHub, and deploy our application to a Kubernetes Cluster on Google Kubernetes Engine (GKE).

Prerequisites

We need a running Jenkins server with SSH access to it. For this project, I have used Compute Engine to spin up an Ubuntu instance for me.
We need an active GCP Project and required access to it.
We have a Kubernetes cluster running on Google Kubernetes Engine (GKE). If you are new to GCP, You can use free-tier credits offered by the GCP for this demo.
We have a GitHub account for our code repository.

Pipeline Architecture Overview

Here’s a high-level look at the CI/CD pipeline that we'll be building:

Source Code Management: GitHub will serve as the version control system, where changes in the codebase trigger the pipeline execution.
Continuous Integration: Jenkins will be used to automate the entire build and deployment workflow. We'll also incorporate Trivy for security scans.
Continuous Deployment: The application will be built as a Docker image, pushed to Google Artifact Registry, and then deployed to GKE.
Monitoring and Notifications: Prometheus and Grafana will be set up to continuously monitor the health and performance of the deployed application. Email notifications will be configured to alert the team about build statuses and vulnerability scans.

Let's Dive in !

Setting Up Jenkins Server

Fire up an Ubuntu instance with a recommended disk space of 15GB and above.

Once the instance is up and running, SSH into it.

Run these commands on the machine to install Jenkins, Docker, Trivy and Kubectl



wget https://raw.githubusercontent.com/akshat2503/Jenkins-CI-CD/main/jenkins-server-setup.sh
chmod +x jenkins-server-setup.sh
./jenkins-server-setup.sh

I am doing this tutorial in an Ubuntu Instance. For other Linux distributions, installation steps may differ. Please follow the official documentation Jenkins Installation

After installation completes, The output will be something like this:

Access the Jenkins server on the given link & authenticate using the first time password.

Install all suggested plugins

and finally create the admin user.

Prepare Jenkins

Install Required Plugins from Dashboard > Manage Jenkins > Plugins

Create credentials in Dashboard > Manage Jenkins > Credentials

1) Service Account from GCP

Create a service account, assign necessary permissions (like Artifact Registry Administrator, Kubernetes Engine Admin, Service Account User, etc.)

Create a key for it, and it will be downloaded on your local machine.

Save the credential in Jenkins. Choose kind as "Secret File" & upload the downloaded key.

2) Google App Password

Go to Manage Google Account > Security > 2-Step verification > App passwords

Create a new app password and copy it somewhere

Set username as the email address of the account from which the app password was generated & the password as the copied app password.

Explaining Pipeline Steps

1) Authenticate with Google Cloud

This step will use gcloud SDK to activate the service account & authorize Docker using the key you provided in Jenkins credentials.



withCredentials([file(credentialsId: 'service-acc-cred', variable: 'GC_KEY')]) {

    sh '''

        gcloud auth activate-service-account --key-file="$GC_KEY"

        gcloud config set project $PROJECT_ID

        gcloud auth configure-docker gcr.io -q

    '''

}

2) Clean Workspace

Cleans the workspace by deleting all the files from previous builds.



deleteDir()

3) Checkout

Clones the GitHub repository to the Jenkins server for further processing.



sh 'git clone https://github.com/akshat2503/akshatgautam'

4) File System Scan

Scans the downloaded GitHub repository for any possible vulnerability using Trivy and generates a report for it.



sh "trivy fs --format table -o trivy-fs-report.html ."

5) Build Docker Image

Uses the Dockerfile present in the repository to build a Docker image.

You can also tag the image with some other parameter such as BUILD_NUMBER, I used latest just for simplicity.



sh 'cd akshatgautam && docker build -t gcr.io/$PROJECT_ID/portfolio-website:latest .'

6) Docker Image Scan

Scans the generated Docker image for any possible vulnerability using Trivy and generates a report for it.



sh "trivy image --format table -o trivy-report.html gcr.io/$PROJECT_ID/portfolio-website:latest"

7) Push Docker Image

The Docker image is then pushed to Google Artifact Repository.



sh 'docker push gcr.io/$PROJECT_ID/portfolio-website:latest'

8) Deploy to GKE

This step fetches credentials for connection to your GKE cluster and then applies a deployment.yaml file.

The YAML file creates a deployment and a service of LoadBalancer type that will serve the application.



withCredentials([file(credentialsId: 'service-acc-cred', variable: 'GOOGLE_APPLICATION_CREDENTIALS')]) {

    sh """

        export PATH=$PATH:$HOME/bin

        gcloud config set project "$PROJECT_ID"

        gcloud container clusters get-credentials "$CLUSTER_NAME" --zone "$CLUSTER_ZONE"

        kubectl apply -f akshatgautam/k8s/deployment.yaml

    """

}

9) Verify Deployment

Verifies the deployment using kubectl and logs the output to console.



sh 'kubectl get services'

9) Email the final status

The post script sends an email with the status of the pipeline execution and also attaches both of the Trivy reports with the email.

Pipeline Creation

Let's get started with setting up your Jenkins pipeline:

Go to Dashboard > New Item

Name: jenkins-gcp-integration-pipeline
Project Type: Pipeline

Scroll down and add the pipeline script, Get the script from here

Our pipeline is set up, but not yet ready to build. Mail configuration to receive mail updates is still pending !

Go to Dashboard > Manage Jenkins > System, scroll down to find Extended E-mail Notification

SMTP Server: smtp.gmail.com
SMTP Port: 465
Credentials: Choose the one we previously created
Check "Use SSL"

Now just below this, You'll find E-mail Notification

SMTP Server: smtp.gmail.com
Check "Use SMTP Authentication"
Username: Your email
Password: App Password that we copied earlier
Check "Use SSL"
SMTP Port: 465

Click "Save"

Voila! Your Jenkins pipeline is now ready

Pipeline Testing

Go to Dashboard > jenkins-gcp-integration-pipeline and click Build Now

A build will trigger within a second. All the pipeline steps will be followed sequentially.

Here are the results:

Verifying Deployment

Go to GKE > Workloads

Go to GKE > Gateways, Services & Ingress. Your application should be listed there.

Conclusion

At this point, we have successfully created a simple CI/CD pipeline between Jenkins, GitHub, Artifact Registry and one Kubernetes cluster running on Google Kubernetes Engine.

You can further enhance this pipeline by adding some features like:

GitSCM Polling/Webhook to trigger build on code commit
Including integration, performance tests on the code
Multi-cluster deployment
Error handling and retry mechanisms
and what not ...

Monitoring through Prometheus & Grafana will be documented in another article to keep this one concise.

Drop your questions in comments and I will be very happy to help !