DEV Community: ProDevOpsGuy Tech Community

Loop Engineering: The Karpathy Method - and the workflow that just made it 5x better

H A R S H H A A — Mon, 06 Jul 2026 06:08:58 +0000

Most people use AI the same way they used Google in 2005

Type something, read what comes back, type again

The AI sits there doing nothing until you push it - you are the engine - The AI is a wrench you pick up and put down between every turn.

That worked fine for two years. It does not work anymore.

The people getting 10x more out of AI right now are not writing better prompts and not using secret models.

They are building loops. And the person who made this idea impossible to ignore is Andrej Karpathy

This article will explain:

1 - from scratch what a loop is

2 - show you how to use them the way Andrej Karpathy does

3 - then describe a method to improve this approach by a factor of 5

4 - and provide tips for getting started.

Part 1 · The Basics

1.1 - What a loop actually is ?

A prompt is one instruction. You ask, you get an answer, you decide what to do next.

A loop is a goal the AI keeps working toward until it gets there - without you sitting in the chair prompting every step

The AI discovers what needs doing, plans how to do it, does the work, checks the result, and if it is not there yet - feeds the result back in and goes again. You define the purpose once.

The loop handles the rest.

Three parts make or break it:

A verifier is what turns repetition into progress. Without a real check on the result, you do not have a loop - you have the agent agreeing with itself on repeat. The check can be a test that passes or fails, a metric that goes up or down, a build that compiles or crashes. No gate means the agent grades its own homework.
State is what makes the loop learn. Each pass, the AI has to know what it already tried. Without that, it repeats the same mistake every cycle. A small file on the side records what is done, what failed, what is next. Tomorrow's run resumes instead of starting from zero.

A stop condition keeps it sane. A loop with no exit runs until it succeeds, breaks, or drains your account. Every working loop has two ways to stop: the goal is met, or a hard limit says "after N tries, stop and report"

2.2. Do you actually need one? Run this test first.

Most articles sell you the loop before they tell you when it is a mistake.

A loop earns its cost only when all four of these are true. Miss one and it costs more than it returns.

The task repeats - at least weekly. Less than that and the setup cost never pays itself back. A one-time job is still better served by one good prompt.
Verification is automated - a test suite, a type checker, a linter, a build. Something that can fail the work without you in the room. No automated check means you are back reading every diff - the exact job the loop was supposed to remove.
Your token budget can absorb the waste - loops re-read context, retry, explore. That burns tokens whether the run ships anything or not. This scales with budget, which is why it reads as obvious to people with free tokens and reckless to people on a $20 plan.
The agent has real tools - logs, a reproduction environment, the ability to run the code it writes and see what breaks. Without that, the loop iterates blind.

The honest take: loop engineering is real, and most people do not need the heavy version yet. If you are on a consumer plan with limited tokens, a heavy loop will hit your rate limit or your wallet before the productivity gain arrives.

Part 2 · The Karpathy Loop

How Karpathy stopped prompting and started looping

In March 2026, Andrej Karpathy released a GitHub repo called AutoResearch.

Three files. About 630 lines of code.

Within a month it had 66,000+ stars and Fortune magazine gave it a name: The Karpathy Loop.

karpathy autoresearch

The setup is almost absurdly simple:

train.py - the training script. The only file the agent is allowed to touch.
prepare.py - the evaluator that scores the model. The agent cannot touch it. If it could, it would just make the test easier instead of making the model better.
program.md - the instructions that tell the agent what to explore and what constraints to respect.

The agent runs in a loop:

Read the code → propose a change → train for five minutes → check if the result improved → commit if it did, roll back if it did not → repeat.

You go to sleep. You wake up to a log of experiments and hopefully a better model.

The human never touches train.py. You write program.md. - the agent handles execution.

The results:

Karpathy pointed it at a model he had already carefully tuned by hand over two decades of experience. He let it run for two days.

The agent ran 700 experiments
Found 20 improvements he missed
Things like a missing scalar multiplier in the attention mechanism that made attention too diffuse across heads
Not a bug a fuzzer would catch - a subtle optimization that a careful human could have found but did not

Because humans get tired after experiment number twelve. The agent does not get tired at all.

Shopify CEO Tobi Lutke tried it overnight on an internal model:

He woke up to a 19% quality improvement
The optimized model was half the size of his previous one
A smaller model beating a bigger one - because the agent optimized for the hardware instead of defaulting to "bigger is better"

The Karpathy Loop Fortune

Karpathy's core insight: if you have an objective metric, you should not be the one running the experiments. You are the bottleneck. Remove yourself from the loop and let it run.

Part 3 · The Building Blocks

Five pieces that make a loop work

Every working loop - whether you build it in Claude Code, Codex, or bash scripts - is assembled from five pieces.
Both Claude Code and Codex ship all five now.

Automation - the heartbeat. Something that fires the loop on a schedule, on an event, or on a trigger.

In Claude Code: /loop for cadence, /goal for running until a condition holds. In Codex: the Automations tab. Without the heartbeat, you ran a script once and forgot about it. That is not a loop.

A skill - stores project knowledge so the agent stops guessing every session. Your conventions, your build steps, the thing you do not do because of that one incident three months ago.

Written once in a markdown file, read by every run.

Without skills, the loop re-derives your entire project context from zero every cycle. With skills, intent compounds.

Sub-agents - split the maker from the checker.

The model that wrote the code is too generous grading its own homework.

A second agent with different instructions catches the things the first one talked itself into. Your writer can be fast and cheap, your reviewer slow and strict. That separation is most of the quality.

Connectors - let the loop act inside your real environment. Read your issue tracker, open a PR, ping Slack, update a Linear ticket.

The difference between an agent that says "here is the fix" and a loop that ships the fix and tells you about it in the morning.

A verifier - the gate. The test, type check, or build that automatically rejects bad work.

Everything else is plumbing.

This is the part that makes the loop real. Without it you are paying for an agent to agree with itself all night

Part 4 · What Comes After Karpathy

The Bilevel - a Loop on top of the Loop

this is where it gets interesting, hehe xD
In March 2026, two researchers published a paper on arxiv called "Bilevel Autoresearch: Meta-Autoresearching Itself."

They took Karpathy's loop and asked a simple question:

If autoresearch is itself a form of research - can you autoresearch autoresearch?

Bilevel Autoresearch

They built a second loop on top of the first one:

Inner loop - does what Karpathy's original does: propose a change, train, evaluate, keep or discard
Outer loop - watches the inner loop work, reads its code and traces, identifies where the search process itself is getting stuck, and generates new Python code that changes how the inner loop searches. Then it injects that code and lets the inner loop run again.

The result:

On Karpathy's GPT pretraining benchmark:

5x improvement over the standard single loop (-0.045 vs -0.009 val_bpb)
Not 5% better. Five times better.
Both loops used the same LLM - you do not need a smarter model for the meta level
The improvement comes from the architecture, not from raw intelligence

What the outer loop actually found: the inner loop kept falling into the same search patterns. The LLM has priors about what optimizations to try, and it keeps going back to those priors even when they stopped working. The outer loop broke those patterns by forcing exploration in directions the model's instincts avoided.

The paper ends with a line worth sitting with: "If autoresearch can meta-autoresearch itself, it can in principle meta-autoresearch anything with a measurable objective."

Part 5 · Try It Yourself

Run a loop right now - no tools needed

You do not need Claude Code or Codex to feel how this works. Paste this into any LLM and watch what happens:

You will work in a loop until the task meets the bar.

TASK:
[describe exactly what you want produced]

SUCCESS CRITERIA (be strict):
- [criterion 1]
- [criterion 2]
- [criterion 3]

LOOP PROTOCOL, repeat every turn:
1. PLAN   - state the single next step.
2. DO     - produce or improve the work.
3. VERIFY - score the result 1-10 on each criterion.
            Be brutally honest. List exactly what is still weak.
4. DECIDE - if every criterion is 8+, print FINAL and stop.
            Otherwise print ITERATING and go again, fixing
            the weakest point first.

RULES:
- Never call it done until every criterion is 8 or higher.
- Each pass must fix the weakest score from the last VERIFY.
- Do not ask me questions. Make a sensible assumption
  and keep going.

Begin.

The model drafts, grades its own work against your criteria, finds the weak spot, rewrites, and repeats until it clears the bar.

That is a loop. You just built one with a paragraph.

It is limited - you are still the trigger, there is no schedule, no persistent state, close the tab and it is gone. But it shows the core mechanic. The jump from this to a full autonomous loop is adding the automation, the state file, and the gate.

Part 6 · The Honest Part

What the loop does not fix

Loops change the work. They do not delete you from it.

And two problems get sharper as the loop gets better - not easier:

Comprehension debt. The faster the loop ships code you did not write, the larger the gap between what exists in your repo and what you actually understand. A smooth running loop charges compound interest on that gap. The day you have to debug a system nobody on the team has read will cost more than the tokens ever did.
Cognitive surrender. When the loop runs itself, it is tempting to stop forming an opinion and just accept whatever comes back. Designing the loop is the cure when you do it with judgment - and the accelerant when you do it to avoid thinking. Same action, opposite result.

Two people can build the exact same loop and get completely opposite outcomes. One uses it to move faster on work they understand deeply. The other uses it to avoid understanding the work at all.

The loop does not know the difference. You do!

Karpathy stopped writing code. Cherny stopped prompting. Neither of them stopped thinking. If you take one thing from this, take that.

start use loop like engineer.

Credits: codila

Top 10 Python Libraries Every DevOps Engineer Should Master (With Use Cases & Code)

H A R S H H A A — Sat, 02 Aug 2025 10:38:22 +0000

In the world of DevOps, automation isn't just a luxury — it's a necessity. From provisioning infrastructure to managing deployments and monitoring services, DevOps engineers rely heavily on scripting to streamline operations.

Python is one of the most widely used programming languages in DevOps due to its simplicity, readability, and massive ecosystem of libraries. Whether you're writing infrastructure-as-code (IaC), automating CI/CD pipelines, or interacting with cloud APIs, Python makes it easier and more efficient.

In this article, we’ll explore 10 essential Python libraries every DevOps engineer should know, along with real-world use cases and practical code examples.

1. 🔹 boto3 – AWS Automation and Resource Management

➤ What is boto3?

boto3 is the Amazon Web Services (AWS) SDK for Python, enabling direct interaction with AWS services like EC2, S3, Lambda, IAM, CloudFormation, and more.

➤ Why It's Useful

DevOps engineers frequently interact with cloud resources. boto3 lets you create, update, and delete resources programmatically — great for dynamic infrastructure, auto-scaling scripts, or custom provisioning tools.

➤ Example: List all S3 buckets

import boto3

s3 = boto3.client('s3')
response = s3.list_buckets()

for bucket in response['Buckets']:
    print(f"Bucket: {bucket['Name']} (Created on {bucket['CreationDate']})")

✅ Real-World Use Cases

Automate provisioning of EC2 instances during deployments.
Backup and restore S3 objects via scripts.
Monitor resource usage and send alerts (via CloudWatch + boto3).

2. 🔹 paramiko – SSH into Remote Servers

➤ What is paramiko?

paramiko is a low-level SSH client library for executing commands and managing files on remote Linux/Unix servers over SSH.

➤ Why It's Useful

Instead of manually SSHing into servers, you can write Python scripts to automate maintenance, patching, or deployments.

➤ Example: Restart a remote web server

import paramiko

client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect('192.168.1.10', username='ec2-user', key_filename='/path/to/key.pem')

stdin, stdout, stderr = client.exec_command('sudo systemctl restart nginx')
print(stdout.read().decode())
client.close()

✅ Real-World Use Cases

Restart services or perform log cleanups across multiple hosts.
Automate health-check scripts.
Perform patch updates remotely during CI/CD.

3. 🔹 docker (Python SDK) – Manage Docker Containers

➤ What is docker SDK for Python?

This SDK allows you to interact with the Docker daemon — build images, manage containers, monitor events — all from Python.

➤ Why It's Useful

Many DevOps workflows use containers. This SDK allows integration of Docker container management into custom tools or CI scripts.

➤ Example: Start a new container with a mounted volume

import docker

client = docker.from_env()
container = client.containers.run(
    "nginx:latest",
    detach=True,
    ports={'80/tcp': 8080},
    volumes={"/host/data": {"bind": "/data", "mode": "rw"}}
)
print(f"Started container {container.short_id}")

✅ Real-World Use Cases

Automatically spin up containers for test environments.
Remove dangling images or stopped containers via cron jobs.
Monitor and auto-restart unhealthy containers.

4. 🔹 kubernetes – Interact with Kubernetes API

➤ What is the kubernetes Python client?

It’s the official Kubernetes client for Python, offering complete access to the Kubernetes API for managing clusters, pods, deployments, services, and more.

➤ Why It's Useful

Modern infrastructure often runs in Kubernetes. You can build tools or scripts that deploy apps, fetch logs, or auto-scale workloads.

➤ Example: Delete a Kubernetes pod

from kubernetes import client, config

config.load_kube_config()
v1 = client.CoreV1Api()

pod_name = "example-pod"
namespace = "default"

v1.delete_namespaced_pod(name=pod_name, namespace=namespace)
print(f"Deleted pod {pod_name}")

✅ Real-World Use Cases

Monitor and restart failed pods.
Automate rolling updates for deployments.
Generate custom dashboards or alerts.

5. 🔹 pyyaml – Parse and Write YAML Files

➤ What is pyyaml?

A YAML parser/emitter for Python. YAML is the de facto standard for configuration in DevOps — Kubernetes, Ansible, GitHub Actions, etc.

➤ Why It's Useful

You often need to generate or modify YAML files dynamically during CI/CD or environment-specific deployments.

➤ Example: Update Kubernetes deployment replicas

import yaml

with open('deployment.yaml', 'r') as f:
    data = yaml.safe_load(f)

data['spec']['replicas'] = 5

with open('deployment.yaml', 'w') as f:
    yaml.dump(data, f)

✅ Real-World Use Cases

Modify Helm chart values before deployment.
Generate dynamic GitHub Actions workflows.
Validate or lint YAML files before applying.

6. 🔹 requests – Make HTTP Requests Easily

➤ What is requests?

A simple and intuitive HTTP client library for sending GET, POST, PUT, and DELETE requests — perfect for REST APIs.

➤ Why It's Useful

Almost every DevOps tool (Jenkins, GitHub, GitLab, Prometheus, Slack, etc.) exposes APIs. Use requests to automate integration or monitoring.

➤ Example: Send a Slack notification after deployment

import requests

url = "https://hooks.slack.com/services/..."
payload = {"text": "🚀 Deployment completed successfully!"}
requests.post(url, json=payload)

✅ Real-World Use Cases

Trigger webhooks on successful builds.
Monitor API endpoints for health checks.
Integrate with external services like PagerDuty or Datadog.

7. 🔹 fabric – Remote Execution at a Higher Level

➤ What is fabric?

A high-level Python library built on paramiko that simplifies SSH operations by abstracting them into tasks and commands.

➤ Why It's Useful

While paramiko is flexible, fabric makes remote command execution cleaner and better structured — ideal for deployment scripts.

➤ Example: Automate deployment steps

from fabric import Connection

c = Connection(host='myserver.com', user='ubuntu')
c.run('cd /var/www/myapp && git pull origin main')
c.run('docker-compose down && docker-compose up -d')

✅ Real-World Use Cases

Execute release processes across multiple servers.
Maintain consistent deployment workflows.
Combine with CI/CD triggers for remote execution.

8. 🔹 pytest – Write and Run Tests

➤ What is pytest?

A powerful testing framework for writing unit, integration, and functional tests — essential for test automation in DevOps.

➤ Why It's Useful

CI/CD pipelines should include automated testing to validate changes before pushing to production.

➤ Example: Test a web service endpoint

import requests

def test_homepage():
    response = requests.get("http://localhost:8000")
    assert response.status_code == 200

✅ Real-World Use Cases

Smoke test after deployment.
Validate configuration scripts or playbooks.
Integrate with GitHub Actions, Jenkins, or GitLab CI for test automation.

9. 🔹 ansible-runner – Programmatically Run Ansible

➤ What is ansible-runner?

An API and CLI tool to interface with Ansible playbooks using Python. Useful when integrating Ansible with custom Python automation.

➤ Why It's Useful

While Ansible is CLI-driven, ansible-runner lets you trigger playbooks as part of custom apps or Python workflows.

➤ Example: Run a playbook and get output

import ansible_runner

r = ansible_runner.run(private_data_dir='/ansible/project', playbook='deploy.yml')
print(f"Status: {r.status}")
print(f"Final RC: {r.rc}")

✅ Real-World Use Cases

Integrate Ansible with Flask/Django dashboards.
Trigger tasks in response to webhooks or events.
Run playbooks across multiple environments with different inventories.

10. 🔹 sys – System-Specific Parameters

➤ What is sys?

A built-in Python module that provides access to system-level information, such as arguments, exit codes, and environment info.

➤ Why It's Useful

Most CLI tools in DevOps require parsing input arguments, handling exits, or checking platform details — sys is essential for that.

➤ Example: Handle arguments in a script

import sys

if len(sys.argv) != 2:
    print("Usage: python deploy.py <env>")
    sys.exit(1)

env = sys.argv[1]
print(f"Deploying to {env} environment...")

✅ Real-World Use Cases

Write custom deployment scripts.
Exit gracefully with meaningful messages in pipelines.
Combine with argparse for better CLI tools.

🚀 Wrapping Up

DevOps isn’t just about tools — it’s about choosing the right tools for the job and automating effectively. These 10 Python libraries form a solid foundation for building reliable, maintainable, and scalable DevOps workflows.

Library	Key Use Case
boto3	AWS automation
paramiko	Remote SSH commands
docker	Manage Docker containers
kubernetes	Kubernetes cluster management
pyyaml	Read/write YAML configs
requests	REST API integration
fabric	High-level remote execution
pytest	Test automation
ansible-runner	Programmatic Ansible execution
sys	Command-line scripts & system info

✨ Bonus Tip:

Start small — write a script using boto3 to list AWS resources or use requests to hit an API endpoint. Gradually layer in other libraries as your workflows evolve.

🛠️ Author & Community

This project is crafted by Harshhaa 💡.

I’d love to hear your feedback! Feel free to share your thoughts.

📧 Connect with me:

📢 Stay Connected

Full-Stack Application Deployment Guide Using Docker, Kubernetes, Jenkins, and Prometheus Monitoring

H A R S H H A A — Sat, 19 Jul 2025 17:59:50 +0000

This detailed guide focuses on deploying the MERN E-Commerce Store source code (from https://github.com/HuXn-WebDev/MERN-E-Commerce-Store.git) with a modern DevOps pipeline involving Docker for containerization, Kubernetes for orchestration, Jenkins for CI/CD, and Prometheus for monitoring—purpose-built for DevOps engineers to demonstrate deployment and operational practices.

Overview & Prerequisites
Application Preparation
Dockerization of MERN App
Jenkins CI/CD Setup
Kubernetes Deployment
Prometheus Monitoring Setup
Best Practices & Tips
Troubleshooting and Future Enhancements

1. Overview & Prerequisites

Objective

Deploy a production-grade MERN (MongoDB, Express, React, Node.js) e-commerce app using DevOps best practices.

Prerequisites

OS: Linux preferred (Windows WSL acceptable).
Installed Tools:
- Docker & Docker Compose
- Kubernetes (Minikube or managed, e.g., EKS, GKE)
- kubectl CLI
- Jenkins (standalone or Dockerized)
- Prometheus and Grafana
- Git
Knowledge Requirements:
- Familiarity with containerization, Kubernetes constructs, CI/CD principles, and service monitoring.

2. Application Preparation

Clone the Repository

git clone https://github.com/HuXn-WebDev/MERN-E-Commerce-Store.git
cd MERN-E-Commerce-Store

Review Source Structure

MERN-E-Commerce-Store/
  backend/
  frontend/
  .env
  package.json
  README.md

Environment Setup

Set up a .env file in the backend directory with MongoDB URI, JWT secrets, and port.
Example snippet:

  MONGO_URI=mongodb://mongo:27017/mernstore
  JWT_SECRET=your_jwt_secret
  PORT=5000

3. Dockerization of MERN App

Backend Dockerfile (`backend/Dockerfile`)

FROM node:18-alpine

WORKDIR /app

COPY package*.json ./
RUN npm install

COPY . .

EXPOSE 5000
CMD ["npm", "start"]

Frontend Dockerfile (`frontend/Dockerfile`)

FROM node:18-alpine as build

WORKDIR /app

COPY package*.json ./
RUN npm install

COPY . .
RUN npm run build

FROM nginx:alpine
COPY --from=build /app/build /usr/share/nginx/html

EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

Docker Compose for Local Testing (`docker-compose.yml`)

version: '3.8'

services:
  mongo:
    image: mongo
    restart: always
    ports:
      - "27017:27017"
    volumes:
      - mongo-data:/data/db

  backend:
    build: ./backend
    ports:
      - "5000:5000"
    environment:
      - MONGO_URI=mongodb://mongo:27017/mernstore
      - JWT_SECRET=your_jwt_secret
    depends_on:
      - mongo

  frontend:
    build: ./frontend
    ports:
      - "3000:80"
    depends_on:
      - backend

volumes:
  mongo-data:

Test Docker Setup

docker-compose up --build

Browse to http://localhost:3000 to verify the frontend and backend are connected.

4. Jenkins CI/CD Setup

Jenkins Installation Sample

docker run -p 8080:8080 -p 50000:50000 -v jenkins_home:/var/jenkins_home jenkins/jenkins:lts

Access Jenkins at http://localhost:8080.

Jenkins Pipeline Example (`Jenkinsfile`)

pipeline {
  agent any
  environment {
    DOCKERHUB_USER = 'yourdockerhubusername'
    REPO = 'mern-ecommerce-store'
    TAG = 'latest'
  }
  stages {
    stage('Checkout') {
      steps {
        git 'https://github.com/HuXn-WebDev/MERN-E-Commerce-Store.git'
      }
    }
    stage('Build Backend Image') {
      steps {
        sh 'docker build -t $DOCKERHUB_USER/$REPO-backend:$TAG ./backend'
      }
    }
    stage('Build Frontend Image') {
      steps {
        sh 'docker build -t $DOCKERHUB_USER/$REPO-frontend:$TAG ./frontend'
      }
    }
    stage('Push Images') {
      steps {
        withCredentials([string(credentialsId: 'dockerhub-pass', variable: 'PASS')]) {
          sh 'echo $PASS | docker login -u $DOCKERHUB_USER --password-stdin'
          sh 'docker push $DOCKERHUB_USER/$REPO-backend:$TAG'
          sh 'docker push $DOCKERHUB_USER/$REPO-frontend:$TAG'
        }
      }
    }
    stage('Deploy to Kubernetes') {
      steps {
        sh 'kubectl apply -f k8s/'
      }
    }
  }
}

Store Docker registry credentials in Jenkins.
Integrate GitHub webhooks for automation.

5. Kubernetes Deployment

Example Kubernetes Manifests (`k8s/`)

MongoDB Deployment and Service (`k8s/mongo-deployment.yaml`)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongo
  template:
    metadata:
      labels:
        app: mongo
    spec:
      containers:
      - name: mongo
        image: mongo
        ports:
        - containerPort: 27017
        volumeMounts:
        - name: mongo-data
          mountPath: /data/db
      volumes:
      - name: mongo-data
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: mongo
spec:
  ports:
  - port: 27017
  selector:
    app: mongo

Backend Deployment and Service (`k8s/backend-deployment.yaml`)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
spec:
  replicas: 2
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
      - name: backend
        image: yourdockerhubusername/mern-ecommerce-store-backend:latest
        ports:
        - containerPort: 5000
        env:
          - name: MONGO_URI
            value: "mongodb://mongo:27017/mernstore"
          - name: JWT_SECRET
            value: "your_jwt_secret"
---
apiVersion: v1
kind: Service
metadata:
  name: backend
spec:
  ports:
  - port: 5000
    targetPort: 5000
  selector:
    app: backend

Frontend Deployment and Service (`k8s/frontend-deployment.yaml`)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  replicas: 2
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: frontend
        image: yourdockerhubusername/mern-ecommerce-store-frontend:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: frontend
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: frontend

Ingress Example (`k8s/ingress.yaml`)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mernecommerce-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: your-domain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: frontend
            port:
              number: 80

Deploy to Kubernetes

kubectl apply -f k8s/
kubectl get pods,svc,ingress

Access the app using Minikube IP or configured ingress.

6. Prometheus Monitoring Setup

Install Prometheus Stack

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack

Enable Backend Metrics

Add Prometheus annotations to backend pod spec:

metadata:
  annotations:
    prometheus.io/scrape: 'true'
    prometheus.io/port: '5000'
    prometheus.io/path: '/metrics'

Expose /metrics in your Node.js backend using a library like prom-client.

Access Grafana Dashboard

kubectl port-forward svc/prometheus-grafana 3000:80

Dashboard is accessible at http://localhost:3000 (login: admin/admin by default).

7. Best Practices & Tips

Construct multi-stage Docker builds to optimize image layers.
Place sensitive configuration in Kubernetes Secrets.
Specify CPU/memory limits for each deployment.
Ensure CI/CD runs all tests before deployments.
Keep dependencies up-to-date and avoid unnecessary image layers.
Use resource requests for optimal cluster utilization.
Automate monitoring alerts using Prometheus Alertmanager.

8. Troubleshooting and Future Enhancements

Inspect pod/container logs for failure diagnosis: kubectl logs
Ensure Jenkins has access to your kubeconfig.
Confirm Prometheus scrape targets are valid and reachable.
Consider canary or blue/green deployment patterns.
Integrate alert notifications to Slack or email via Alertmanager.
Use custom Grafana dashboards for e-commerce KPIs (orders, sales, latency).

🛠️ Author & Community

This project is crafted by Harshhaa 💡.

I’d love to hear your feedback! Feel free to share your thoughts.

📧 Connect with me:

📢 Stay Connected

Unlock 1000+ DevOps & Cloud Resources! 📚 - Gateway to a powerful library

H A R S H H A A — Sat, 07 Jun 2025 09:44:37 +0000

🚀 Introducing the Ultimate Premium Plan – Your One-Stop DevOps & Cloud Resource Hub

Finding quality DevOps and Cloud documentation online can be frustrating — outdated links, scattered sources, and endless searching.

That’s why we created the Premium Plan — a powerful, all-in-one portal where you can access 1000+ curated DevOps & Cloud resources instantly, with new content added daily.

💡 What Is This Portal?

The Premium Portal is a one-click download platform designed for DevOps learners, professionals, and job seekers. Instead of wasting time searching across the internet, we bring everything to one clean, ad-free interface — just Go → Search → Download.

📦 What You’ll Get with the Premium Plan

✅ 1000+ DevOps & Cloud Docs
✅ Career Roadmaps
✅ Real-time Project Docs
✅ Interview Preparation Kits
✅ Cheatsheets & Troubleshooting Guides
✅ Exam Dumps & Practice Materials
✅ DevOps Scripts & Tool Notes
✅ Handwritten Notes
✅ Premium-Only Content
✅ High-quality, regularly updated PDFs
✅ 30–40+ new docs added every day!

🏆 Premium Member Benefits

🌟 Unlimited File Downloads
🧠 Early Access to New Content
📈 Weekly Updates with Fresh Docs
💬 Priority Support
💎 Premium handy picked Resources
♾️ Lifetime Access
❌ No Recurring Fees, Ever
🧹 Ad-Free Experience
🖥️ Clean, Simple, Fast Interface

This is more than just a resource site. It’s your permanent DevOps library.

💰 Pricing

🔹 India: ₹499/- (One-time)
🔹 Outside India: $15 USD (One-time)

💡 No subscriptions. No hidden costs. One-time payment for lifetime value.

🎯 Why Join?

If you're tired of Googling for hours, jumping between tabs, or relying on incomplete resources — this is built for you.

We’ve done the hard work of collecting high-value, hard-to-find documentation so you don’t have to.

🔗 Get Started Now

👉 Create your account and unlock lifetime access:
https://docs-admin.prodevopsguytech.com/create-account

📖 Read the full announcement:
https://www.devops-engineering.site/blog/premium-plan-announcement

🌐 Visit the Portal:
https://docs.prodevopsguytech.com/

Still Have Questions?

Feel free to Comment or reach out directly — we're happy to help you get started with your DevOps journey.

Let your learning be focused, fast, and limitless. 🔥

🚀Introducing the DevOps & Cloud Docs Portal – Your Ultimate Learning Hub! 📚

H A R S H H A A — Fri, 04 Apr 2025 05:50:55 +0000

🌟 What is the DevOps & Cloud Docs Portal?

Learning DevOps and Cloud computing can be overwhelming. With so many tools, concepts, and platforms, it's often difficult to find reliable, high-quality learning materials in one place.

That’s why we created the DevOps & Cloud Docs Portal – a centralized and easy-to-use platform where you can access a vast collection of 900+ DevOps and Cloud documents, instantly available for download. No complicated sign-ups, no searching across multiple sites—just one click to access everything you need!

Whether you're a beginner looking to get started or an experienced engineer seeking in-depth knowledge, this portal has something for everyone.

🔗 How to Access the Portal?

Getting started is super easy!

🌎 Visit: 👉 docs.prodevopsguytech.com

📂 Search & Download: Browse through 900+ documents and download instantly.

⚡ Start Learning & Upskilling: Apply your knowledge to real-world projects.

📂 What’s Inside the Portal?

The DevOps & Cloud Docs Portal is designed to cover all major areas of DevOps, Cloud Computing, and Infrastructure Automation.

Here’s what you’ll find inside:

🔥 Comprehensive DevOps Resources

DevOps Roadmaps & Learning Paths – Step-by-step guides to becoming a DevOps engineer.
Best Practices & Cheat Sheets – Quickly reference important DevOps concepts.
Security & Compliance – Learn how to implement security best practices in your DevOps pipeline.

☁️ Cloud Computing Documentation

AWS, Azure, and Google Cloud – Official documentation, study materials, and architecture guides.
Multi-cloud & Hybrid Cloud – Learn how to work with multiple cloud providers efficiently.
Serverless Computing – Resources on AWS Lambda, Azure Functions, and Google Cloud Functions.

🐳 Containers & Kubernetes

Docker & Containerization – Learn how to build, manage, and deploy containers.
Kubernetes (K8s) Fundamentals – From basic concepts to advanced cluster management.
Helm, Istio, and Service Mesh – Manage Kubernetes workloads effectively.

🏗️ Infrastructure as Code (IaC)

Terraform & CloudFormation – Automate infrastructure deployment across multiple cloud providers.
Ansible, Puppet, Chef – Configuration management and automation tools.
Infrastructure Best Practices – Learn how to manage infrastructure efficiently and securely.

🔄 CI/CD Pipelines & Automation

Jenkins, GitHub Actions, GitLab CI/CD, ArgoCD – Learn how to automate deployment pipelines.
DevSecOps – Integrate security into CI/CD workflows.
Infrastructure Automation – Learn to deploy applications automatically with pipelines.

📊 Monitoring & Logging

Prometheus & Grafana – Monitor infrastructure and application performance.
ELK Stack (Elasticsearch, Logstash, Kibana) – Centralized logging for troubleshooting and analysis.
Datadog, New Relic, Splunk – Cloud-native monitoring solutions.

🔍 More Exclusive Resources

DevOps & Cloud Interview Questions – Prepare for your next job interview.
Real-World Case Studies – Learn how DevOps is implemented in industry-leading companies.
E-books, Whitepapers, and Research Papers – Deep dive into advanced topics.

💡 Why Should You Use the DevOps & Cloud Docs Portal?

There are plenty of DevOps and Cloud resources available online, but here’s why this portal stands out:

✅ Saves You Time – No more wasting hours searching for guides across different websites. Everything is in one place.

✅ Instant Downloads – No login, no sign-up, no waiting. Just one click and download any document.

✅ High-Quality Content – Curated learning materials from trusted sources and real-world experts.

✅ Updated Frequently – New documents added regularly to keep you up-to-date with the latest DevOps trends.

✅ Community-Driven – Get recommendations, share insights, and grow with a vibrant DevOps community.

👥 Join the DevOps & Cloud Community!

💬 Want to connect with like-minded DevOps professionals? Join our Telegram group for exclusive resources, discussions, and updates!

👉 Join Now: @devopsdocs

This project is crafted by Harshhaa

🚀 Whether you’re a beginner looking to get started or an expert searching for advanced materials, the DevOps & Cloud Docs Portal is the perfect place to boost your skills, stay updated, and grow in your career!

📢 Help Us Grow – Share With Your Network!

If you find this portal useful, share it with your colleagues, friends, and anyone interested in DevOps & Cloud!

📢 Let’s build and grow the DevOps community together! 💙🚀

🔁 Spread the word!

📧 Connect with me:

📢 Stay Connected

CI/CD DevOps Pipeline Project: Deployment of Java Application on Kubernetes

H A R S H H A A — Sat, 22 Mar 2025 16:36:32 +0000

Introduction

In today's fast-paced software development environment, implementing DevOps practices is crucial for organizations striving for agility, efficiency, and high-quality software delivery. This project demonstrates the implementation of a robust Continuous Integration/Continuous Deployment (CI/CD) pipeline using Jenkins, automating the entire software lifecycle from development to deployment.

Architecture

Objectives

The primary goal of this project is to automate software delivery, improving speed, reliability, and efficiency while minimizing manual intervention. The key objectives include:

Automated CI/CD Pipeline: Implementing a seamless Jenkins-based CI/CD pipeline to streamline build, test, and deployment processes.
Integration of DevOps Tools: Utilizing Maven, SonarQube, Trivy, Nexus Repository, Docker, Kubernetes, Prometheus, and Grafana for end-to-end automation.
Code Quality & Security: Enhancing software quality through static code analysis (SonarQube) and security vulnerability scanning (Trivy).
Scalable Deployments: Ensuring reliable, scalable, and load-balanced deployments on a Kubernetes cluster.
Automated Notifications: Enabling email alerts for build and deployment status.
Robust Monitoring & Alerting: Implementing Prometheus and Grafana for real-time system health monitoring and performance analysis.

Tools & Technologies Used

Tool	Purpose
Jenkins	CI/CD pipeline orchestration
Maven	Build automation and dependency management
SonarQube	Static code analysis for quality assurance
Trivy	Docker image vulnerability scanning
Nexus Repository	Artifact management and version control
Docker	Containerization for consistency and portability
Kubernetes	Container orchestration for scalable deployments
Gmail Integration	Email notifications for build and deployment updates
Prometheus & Grafana	Monitoring and visualization of system metrics
AWS EC2	Infrastructure provisioning and hosting

Infrastructure Setup: Virtual Machines on AWS

To support this pipeline, AWS EC2 instances were provisioned for various DevOps tools:

🔹 Kubernetes Master Node: Manages cluster state, scheduling, and coordination.

🔹 Kubernetes Worker Nodes (2): Host and execute containerized applications.

🔹 SonarQube Server: Performs static code analysis to ensure code quality.

🔹 Nexus Repository Manager: Manages build artifacts, dependencies, and Docker images.

🔹 Jenkins Server: Orchestrates CI/CD processes and integrates DevOps tools.

🔹 Monitoring Server: Runs Prometheus and Grafana for real-time system monitoring.

Each instance is configured with the necessary CPU, memory, and storage resources. Security best practices such as access controls and network configurations were applied to safeguard infrastructure.

EC2 Instances :

Security Group:

Setting Up Kubernetes Cluster with Kubeadm

Prerequisites

Ubuntu OS (Xenial or later) with sudo privileges
t2.medium instance type or higher
Internet access
Ensure all AWS instances are in the same Security Group
Open port 6443 to allow worker nodes to join the cluster

1️⃣ Prepare Nodes for Kubernetes Installation

Run the following commands on both Master and Worker nodes:

# Disable swap
sudo swapoff -a

Load required kernel modules:

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

Set sysctl parameters:

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

sudo sysctl --system

2️⃣ Install CRI-O Runtime

sudo apt-get update -y
sudo apt-get install -y software-properties-common curl apt-transport-https ca-certificates gpg

sudo curl -fsSL https://pkgs.k8s.io/addons:/crio:/prerelease:/main/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/cri-o-apt-keyring.gpg

echo "deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://pkgs.k8s.io/addons:/cri-o:/prerelease:/main/deb/ /" | sudo tee /etc/apt/sources.list.d/cri-o.list

sudo apt-get update -y
sudo apt-get install -y cri-o
sudo systemctl enable crio --now

3️⃣ Install Kubernetes Packages

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update -y
sudo apt-get install -y kubelet=1.29.0-* kubectl=1.29.0-* kubeadm=1.29.0-*

4️⃣ Initialize Kubernetes Cluster on Master Node

sudo kubeadm config images pull
sudo kubeadm init

mkdir -p "$HOME"/.kube
sudo cp -i /etc/kubernetes/admin.conf "$HOME"/.kube/config
sudo chown "$(id -u)":"$(id -g)" "$HOME"/.kube/config

Install Calico as the Network Plugin:

kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.0/manifests/calico.yaml

Retrieve the join command for worker nodes:

kubeadm token create --print-join-command

5️⃣ Join Worker Nodes

Run the join command obtained from the master node:

sudo kubeadm join <master-ip>:6443 --token <your-token> --discovery-token-ca-cert-hash sha256:<hash> --v=5

Verify the cluster status on the master node:

kubectl get nodes

Setting Up DevOps Tools

Jenkins Installation

sudo apt install openjdk-17-jre-headless -y
wget -O /usr/share/keyrings/jenkins-keyring.asc https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key
echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian-stable binary/" | sudo tee /etc/apt/sources.list.d/jenkins.list
sudo apt-get update
sudo apt-get install jenkins -y

Docker Installation

sudo apt-get update
sudo apt-get install -y ca-certificates curl

sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io

Nexus Repository Setup

docker run -d --name nexus -p 8081:8081 sonatype/nexus3:latest

Retrieve the Nexus admin password:

docker exec -it <container_id> cat /nexus-data/admin.password

Segment 2: Setting Up a Private Git Repository

Follow these steps to create a private Git repository, generate a personal access token, connect to the repository, and push your code securely.

1. Create a Private Git Repository

Log in to your preferred Git hosting platform (e.g., GitHub, GitLab, Bitbucket).
Navigate to the "Repositories" section.
Click on "New Repository" and set its visibility to Private.
Copy the repository URL for later use.

2. Generate a Personal Access Token (PAT)

Go to your account Settings.
Find the Developer Settings (GitHub) or Access Tokens section (GitLab, Bitbucket).
Generate a new Personal Access Token (PAT) with the required permissions, such as:
- repo (for full repository access)
- read:packages (if accessing packages)
Copy and save the token securely, as it will not be displayed again.

3. Clone the Repository Locally

Open a terminal or Git Bash.
Navigate to the directory where you want to store the repository.
Run the following command, replacing <repository_URL> with your actual repository link:

  git clone <repository_URL>

Navigate into the cloned repository folder:

  cd <repository_name>

4. Add Your Code to the Repository

Move or create your project files inside the repository directory.

5. Stage and Commit Changes

Stage all files for the commit:

  git add .

Commit the staged changes with a meaningful message:

  git commit -m "Initial commit: Added project files"

6. Push Changes to the Private Repository

Push your changes to the remote repository:

  git push

If it's your first time pushing, set the upstream branch and push:

  git push -u origin main

(Replace main with master or any other branch name, if applicable.)

7. Authenticate Using a Personal Access Token

When prompted for credentials:
- Username: Enter your GitHub/GitLab/Bitbucket username.
- Password: Use the Personal Access Token (PAT) instead of your password.

8. Verify the Push

Go to your Git hosting platform and open the repository.
You should see the committed files successfully uploaded.

Segment 3: CI/CD Setup in Jenkins

Installing Required Jenkins Plugins

Before setting up the CI/CD pipeline, install the following Jenkins plugins:

Eclipse Temurin Installer
- Enables Jenkins to install and configure the Eclipse Temurin JDK (formerly AdoptOpenJDK).
- Go to Jenkins Dashboard → Manage Jenkins → Manage Plugins → Available tab.
- Search for Eclipse Temurin Installer, select it, and click Install without restart.
Pipeline Maven Integration
- Adds Maven support to Jenkins Pipelines, allowing Maven commands within scripts.
- Install it by searching for Pipeline Maven Integration in the plugin manager.
Config File Provider
- Allows centralized storage of configuration files (properties, XML, JSON) in Jenkins.
- Install via the plugin manager.
SonarQube Scanner
- Integrates SonarQube for code quality and security analysis during builds.
- Search for SonarQube Scanner in the plugin manager and install it.
Kubernetes CLI
- Enables Jenkins to interact with Kubernetes clusters using kubectl.
- Install it from the plugin manager.
Kubernetes Plugin
- Allows Jenkins agents to run as Kubernetes pods for dynamic scaling.
- Install via the plugin manager.
Docker Plugin
- Enables Jenkins to interact with Docker for building and managing containers.
- Install it from the plugin manager.
Docker Pipeline Plugin
- Extends Jenkins Pipeline with steps to build, publish, and manage Docker containers.
- Install it via the plugin manager.

After installation, configure these plugins in Manage Jenkins → Global Tool Configuration and set up credentials where required.

Creating a Jenkins CI/CD Pipeline

Once the plugins are installed, create a new Pipeline job in Jenkins and use the following script:

Jenkinsfile: CI/CD Pipeline for Java Application Deployment

pipeline {
    environment {
        SCANNER_HOME = tool 'sonar-scanner'
    }
    tools {
        jdk 'jdk17'
        maven 'maven3'
    }
    stages {
        stage('Git Checkout') {
            steps {
                git branch: 'main', credentialsId: 'git-cred', url: 'https://github.com/jaiswaladi246/Boardgame.git'
            }
        }
        stage('Compile') {
            steps {
                sh "mvn compile"
            }
        }
        stage('Run Tests') {
            steps {
                sh "mvn test"
            }
        }
        stage('Trivy File System Scan') {
            steps {
                sh "trivy fs --format table -o trivy-fs-report.html ."
            }
        }
        stage('SonarQube Code Analysis') {
            steps {
                withSonarQubeEnv('sonar') {
                    sh '''
                    $SCANNER_HOME/bin/sonar-scanner -Dsonar.projectName=BoardGame -Dsonar.projectKey=BoardGame -Dsonar.java.binaries=.
                    '''
                }
            }
        }
        stage('Quality Gate Check') {
            steps {
                script {
                    waitForQualityGate abortPipeline: false, credentialsId: 'sonar-token'
                }
            }
        }
        stage('Build Artifact') {
            steps {
                sh "mvn package"
            }
        }
        stage('Publish Artifacts to Nexus') {
            steps {
                withMaven(globalMavenSettingsConfig: 'global-settings', jdk: 'jdk17', maven: 'maven3', traceability: true) {
                    sh "mvn deploy"
                }
            }
        }
        stage('Build and Tag Docker Image') {
            steps {
                script {
                    withDockerRegistry(credentialsId: 'docker-cred', toolName: 'docker') {
                        sh "docker build -t jaiswaladi246/Boardgame:latest ."
                    }
                }
            }
        }
        stage('Docker Image Security Scan') {
            steps {
                script {
                    withDockerRegistry(credentialsId: 'docker-cred', toolName: 'docker') {
                        sh "trivy image --format table -o trivy-image-report.html jaiswaladi246/Boardgame:latest"
                    }
                }
            }
        }
        stage('Push Docker Image to Registry') {
            steps {
                script {
                    withDockerRegistry(credentialsId: 'docker-cred', toolName: 'docker') {
                        sh "docker push jaiswaladi246/Boardgame:latest"
                    }
                }
            }
        }
        stage('Deploy to Kubernetes') {
            steps {
                withKubeConfig(credentialsId: 'k8-cred', namespace: 'webapps', serverUrl: 'https://172.31.8.22:6443') {
                    sh "kubectl apply -f deployment-service.yaml"
                    sh "kubectl get pods -n webapps"
                }
            }
        }
    }
    post {
        always {
            script {
                def jobName = env.JOB_NAME
                def buildNumber = env.BUILD_NUMBER
                def pipelineStatus = currentBuild.result ?: 'UNKNOWN'
                def body = """
                ${jobName} - Build ${buildNumber}
                Pipeline Status: ${pipelineStatus.toUpperCase()}
                Check the console output.
                """
                emailext(
                    subject: "${jobName} - Build ${buildNumber} - ${pipelineStatus.toUpperCase()}",
                    body: body,
                    to: 'jaiswaladi246@gmail.com',
                    from: 'jenkins@example.com',
                    replyTo: 'jenkins@example.com',
                    mimeType: 'text/html',
                    attachmentsPattern: 'trivy-image-report.html'
                )
            }
        }
    }
}

Pipeline Breakdown

Git Checkout
- Clones the project from GitHub using credentials.
Build and Test
- Compiles the project and runs tests using Maven.
Security Scanning
- Scans the file system and Docker image for vulnerabilities using Trivy.
SonarQube Code Analysis
- Runs a SonarQube scan for code quality and security issues.
Quality Gate Check
- Ensures that the SonarQube quality gate is passed before proceeding.
Build and Publish Artifacts
- Packages the application and pushes it to Nexus.
Docker Image Build and Push
- Builds a Docker image and pushes it to a registry.
Deploy to Kubernetes
- Deploys the application to a Kubernetes cluster using kubectl.
Notification
- Sends an email notification with pipeline status and the Trivy security report.

Segment 4: Monitoring Setup

Prometheus Installation and Setup

Download Prometheus and Exporters

Download Prometheus, Node Exporter, and Blackbox Exporter from: 👉 Prometheus Downloads

Install and Run Prometheus

Extract the downloaded .tar.gz file.
Navigate to the extracted directory and run:

   ./prometheus &

By default, Prometheus runs on port 9090.
Access it in a browser:

   http://<instance_IP>:9090

Install and Run Blackbox Exporter

Extract the Blackbox Exporter .tar.gz file.
Navigate to the extracted directory and run:

   ./blackbox_exporter &

Grafana Installation and Setup

Download and Install Grafana

Download Grafana from: 👉 Grafana Downloads

Alternatively, Install Grafana on Ubuntu

Run the following commands on your Monitoring VM:

sudo apt-get install -y adduser libfontconfig1 musl
wget https://dl.grafana.com/enterprise/release/grafana-enterprise_10.4.2_amd64.deb
sudo dpkg -i grafana-enterprise_10.4.2_amd64.deb

Start Grafana Service

sudo /bin/systemctl start grafana-server

Grafana runs on port 3000 by default.
Access it in a browser:

  http://<instance_IP>:3000

Configuring Prometheus

Modify Prometheus Configuration (`prometheus.yaml`)

Edit the file to include the Blackbox Exporter:

scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]  # Checks for HTTP 200 response
    static_configs:
      - targets:
        - http://prometheus.io  # HTTP Target
        - https://prometheus.io # HTTPS Target
        - http://example.com:8080  # HTTP on port 8080
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: <instance_IP>:9115

Replace <instance_IP> with your server's IP address.

Restart Prometheus

Find the Prometheus process ID:

   pgrep prometheus

Kill the existing process and restart Prometheus.

Adding Prometheus as a Data Source in Grafana

Login to Grafana (http://<instance_IP>:3000).
Navigate to Settings → Data Sources → Add Data Source.
Select Prometheus as the source.
Enter the Prometheus server URL (http://<instance_IP>:9090).
Save and test the connection.
Import a Prometheus dashboard from Grafana's public dashboard library.

🚀 Your monitoring setup with Prometheus and Grafana is now ready!

Results:

JENKINS PIPELINE:

PROMETHEUS:

BLACKBOX:

GRAFANA:

APPLICATION:

Conclusion

The successful implementation of the DevOps CI/CD pipeline project marks a significant milestone in enhancing the efficiency, reliability, and quality of software delivery processes. By automating key aspects of the software development lifecycle, including compilation, testing, deployment, and monitoring, the project has enabled rapid and consistent delivery of software releases, contributing to improved time-to-market and customer satisfaction.

Acknowledgment of Contributions

I want to express my gratitude to DevOps Shack for their excellent project and implementation guide.

Final Thoughts

Looking ahead, the project's impact extends beyond its immediate benefits, paving the way for continuous improvement and innovation in software development practices. By embracing DevOps principles and leveraging cutting-edge tools and technologies, we have laid a solid foundation for future projects to build upon. The scalability, flexibility, and resilience of the CI/CD pipeline ensure its adaptability to evolving requirements and technological advancements, positioning our organization for long-term success in a competitive market landscape.

References

Jenkins Documentation: https://www.jenkins.io/doc/
Maven Documentation: https://maven.apache.org/guides/index.html
SonarQube Documentation: https://docs.sonarqube.org/latest/
Trivy Documentation: https://github.com/aquasecurity/trivy
Nexus Repository Manager Documentation: https://help.sonatype.com/repomanager3
Docker Documentation: https://docs.docker.com/
Kubernetes Documentation: https://kubernetes.io/docs/
Prometheus Documentation: https://prometheus.io/docs/
Grafana Documentation: https://grafana.com/docs/

These resources provided valuable insights, guidance, and support throughout the project lifecycle, enabling us to achieve our goals effectively.

🛠️ Author & Community

This project is crafted by Harshhaa 💡.

I’d love to hear your feedback! Feel free to share your thoughts.

📧 Connect with me:

📢 Stay Connected

DevOps Tools Cheatsheet Collection – Your Ultimate Guide🚀

H A R S H H A A — Wed, 19 Mar 2025 16:02:04 +0000

Introduction

In the fast-paced world of DevOps, efficiency is key. Whether you’re a DevOps engineer, a sysadmin, or a developer, having a quick reference guide for essential tools can significantly boost productivity. That’s where the DevOps Tools Cheatsheet Collection comes in – a comprehensive, structured, and regularly updated repository designed to help you master DevOps tools effortlessly.

This guide provides an in-depth overview of the cheatsheet collection available in your GitHub repository: DevOps Cheatsheet. Let’s dive into what makes this repository an indispensable resource!

🚀 Master DevOps with Our Ultimate Cheatsheet!

💡 Stay ahead in your DevOps journey with a comprehensive collection of commands, concepts, and best practices—all in one place!

🔗 Check out the full repository: DevOps Cheatsheet 🔥

🌟 Why Use This DevOps Cheatsheet Collection?

This repository is carefully curated to ensure you get the most relevant and up-to-date commands, best practices, and tips for various DevOps tools. Here’s why it stands out:

✅ Comprehensive Coverage – Covers CI/CD, Containerization, Cloud, Security, Monitoring, and more.

✅ Beginner-Friendly & Advanced – Useful for new learners and seasoned professionals alike.

✅ Well-Structured & Easy to Navigate – Quickly find what you need.

✅ Regularly Updated – Stay ahead with the latest tools and practices.

Whether you’re deploying applications, managing cloud infrastructure, automating workflows, or enhancing security, this cheatsheet collection is your go-to resource.

📂 Repository Structure

The repository is structured in a clear and organized manner to ensure quick access to relevant information. Below is a directory breakdown:

/devops-cheatsheet/
│
├── README.md
├── CONTRIBUTING.md
│
├── CI-CD/
│   ├── Jenkins.md
│   ├── GitHub-Actions.md
│   ├── GitLab-CI.md
│   └── CircleCI.md
│
├── Containerization/
│   ├── Docker.md
│   ├── Kubernetes.md
│   ├── CRI-O.md
│   ├── OpenShift.md
│   ├── Helm.md
│   └── Podman.md
│
├── Monitoring/
│   ├── Prometheus.md
│   ├── Grafana.md
│   ├── ELK-Stack.md
│   ├── CloudWatch.md
│   └── Nagios.md
│
├── Security/
│   ├── Trivy.md
│   ├── SonarQube.md
│   ├── AquaSec.md
│   └── HashiCorp-Vault.md
│
├── Version-Control/
│   ├── GitLab.md
│   ├── GitHub.md
│   └── Bitbucket.md
│
├── Cloud/
│   ├── AWS.md
│   ├── Azure.md
│   ├── Ansible.md
│   ├── GCP.md
│   ├── Kubernetes-on-AWS.md
│   └── Terraform.md
│
└── Networking/
    ├── Istio.md
    ├── Envoy.md
    ├── Consul.md
    └── Linkerd.md

Each category covers essential tools along with cheatsheets that contain quick commands, best practices, and troubleshooting tips.

📚 Cheatsheet Categories and Highlights

🔄 CI/CD Automation

Speed up deployments with Continuous Integration (CI) and Continuous Deployment (CD) tools:

🚀 Jenkins – Automate builds and deployments.
🚀 GitHub Actions – CI/CD workflows directly within GitHub.
🚀 GitLab CI – Built-in CI/CD for GitLab projects.
🚀 CircleCI – Fast and efficient CI/CD pipelines.

🐳 Containerization & Orchestration

Easily build, manage, and deploy containers:

🔹 Docker – Most popular containerization platform.
🔹 Kubernetes – Automate container management.
🔹 CRI-O – Lightweight runtime for Kubernetes.
🔹 OpenShift – Enterprise Kubernetes.
🔹 Helm – Manage Kubernetes applications.
🔹 Podman – Rootless container runtime.

📊 Monitoring & Observability

Keep track of system health and logs:

📈 Prometheus – Metrics and alerting.
📈 Grafana – Data visualization dashboard.
📈 ELK Stack – Log management and analytics.
📈 Nagios – Infrastructure monitoring.
📈 CloudWatch – AWS monitoring service.

🔐 Security & Compliance

Ensure security and vulnerability scanning:

🛡️ Trivy – Container vulnerability scanning.
🛡️ SonarQube – Code quality and security.
🛡️ AquaSec – Container security platform.
🛡️ HashiCorp Vault – Secrets management.

🔖 Version Control & GitOps

Streamline collaboration and repository management:

📂 GitLab
📂 GitHub
📂 Bitbucket

☁️ Cloud & Infrastructure as Code (IaC)

Manage cloud infrastructure efficiently:

☁️ AWS – Amazon Web Services commands.
☁️ Azure – Microsoft Azure resources.
☁️ Ansible – Configuration management.
☁️ Google Cloud Platform (GCP) – GCP-related operations.
☁️ Terraform – Infrastructure as Code automation.
☁️ Kubernetes on AWS – Running K8s on AWS.

🌐 Networking & Service Mesh

Enhance connectivity and service discovery:

🌍 Istio – Service mesh for Kubernetes.
🌍 Envoy – Proxy for microservices.
🌍 Consul – Service discovery.
🌍 Linkerd – Lightweight service mesh.

👥 Who Should Use This Cheatsheet?

✅ DevOps Engineers – Quick access to essential commands.

✅ Sysadmins – Simplify system management with structured Cheatsheets.

✅ Developers – Understand DevOps tools and workflows.

✅ Beginners – Learn step-by-step with curated resources.

🚀 Conclusion

The DevOps Tools Cheatsheet Collection is your one-stop resource for mastering essential DevOps tools. Whether you’re automating deployments, managing cloud infrastructure, monitoring applications, or securing environments, this collection provides quick, structured, and easy-to-follow guides to boost your productivity.

🔗 Explore the full repository here: DevOps Cheatsheet

💡 Found something useful? Star the repo ⭐ and contribute to make it even better! 🚀

🛠️ Author & Community

This project is crafted by Harshhaa 💡.

I’d love to hear your feedback! Feel free to share your thoughts.

📧 Connect with me:

📢 Stay Connected

🚀 The Ultimate DevOps Interview Questions & Answers Repository – 550+ Questions & Growing!

H A R S H H A A — Wed, 26 Feb 2025 14:26:14 +0000

Are you preparing for a DevOps interview? Whether you're a beginner stepping into the DevOps world or an experienced engineer aiming for FAANG-level jobs, interview preparation can be overwhelming.

To help you crack DevOps interviews with confidence, I’ve created an open-source GitHub repository with 550+ carefully curated DevOps interview questions covering real-world scenarios, troubleshooting challenges, and best practices.

🔗 Access the Repository

👉 GitHub Repo: DevOps Interview Questions & Answers

⭐ Star the repo to stay updated with new questions!

💡 Why This Repository?

🔥 Comprehensive & Up-to-Date

This repo isn’t just a random list of questions—it’s a structured and well-organized collection covering all major DevOps concepts and tools.

✅ Core DevOps Principles – CI/CD, Automation, Infrastructure as Code (IaC)

✅ Cloud Providers – AWS, Azure, GCP, OpenStack

✅ Containers & Orchestration – Docker, Kubernetes, Helm

✅ CI/CD & Automation – Jenkins, ArgoCD, GitHub Actions

✅ Monitoring & Logging – Prometheus, Grafana, ELK Stack

✅ Networking & Security – DNS, Load Balancing, SSL, Firewalls

✅ Scripting & Configuration Management – Ansible, Terraform, Bash, Python

✅ DevOps Interview Scenarios & Real-World Problems

🎯 Bonus:

📄 Downloadable PDFs & Docs for quick revision
🎭 Mock Interviews & Real-World Troubleshooting Scenarios

📂 How the Repository is Structured

The questions are categorized by topics and difficulty levels (Beginner, Intermediate, Advanced) to help you prepare effectively.

📌 Repository Structure:

📦 devops-interview-questions  
 ├── 📁 core-concepts/             # DevOps fundamentals  
 ├── 📁 cloud/                     # AWS, Azure, GCP, OpenStack  
 ├── 📁 ci-cd/                     # Jenkins, GitHub Actions, GitLab CI/CD  
 ├── 📁 containers/                # Docker, Kubernetes, Helm  
 ├── 📁 infrastructure-as-code/     # Terraform, Ansible, CloudFormation  
 ├── 📁 monitoring-logging/         # Prometheus, Grafana, ELK Stack  
 ├── 📁 networking-security/        # IAM, Firewalls, Load Balancers  
 ├── 📁 automation-scripting/       # Bash, Python, YAML, Groovy  
 ├── 📁 best-practices/             # Real-world scenarios & case studies  
 ├── 📁 mock-interviews/            # Mock interview questions and solutions  
 ├── 📁 cheat-sheets/               # Quick reference guides  
 ├── 📁 docs/                       # Downloadable PDFs & Interview Guides  
 ├── 📄 CONTRIBUTING.md             # Contribution guidelines  
 ├── 📄 LICENSE                     # License information  
 └── 📄 README.md                   # Project overview

💡 Want to get started? Head over to the GitHub repo and explore the questions.

🚀 How to Use This Repository for Interview Prep?

✅ Step 1: Follow a Structured Approach

If you're a beginner, start with DevOps Fundamentals and Cloud Basics.
For experienced candidates, focus on real-world troubleshooting scenarios.
Practice Kubernetes, Terraform, and CI/CD pipeline questions—they are must-know for any DevOps interview.

✅ Step 2: Download & Study the Docs

Check out the PDFs, cheat sheets, and best practices docs for quick learning.

✅ Step 3: Join Mock Interviews & Discussions

Try answering scenario-based questions before checking the answers.
Participate in discussions on GitHub Issues or Telegram.

🎯 Why Should You Use This Repo?

✔️ Covers Every Important DevOps Topic

✔️ Scenario-Based & Real-World Questions (Not just theory!)

✔️ Mock Interview Questions for Hands-on Practice

✔️ Regularly Updated with New Questions & Answers

✔️ Free & Open-Source!

📌 If this repo helps you, support me by starring ⭐ the repository and sharing it with others!

💰 Support This Project

If you find this repo useful and want to support future updates, you can:

☕ Buy Me a Coffee: BuyMeACoffee

💰 Donate via PayPal: PayPal

Your support helps keep this resource free and growing!

🚀 Join the Community

💬 Telegram: Join Here

📝 Blog: ProDevOpsGuy

🔗 GitHub: @NotHarshhaa

⭐ Ready to Ace Your DevOps Interview?

👉 Visit the GitHub Repo and Start Practicing Today!

🚀 Let’s crack DevOps interviews together! 🚀

Automating Infrastructure with AWS CloudFormation: A Beginner's Guide

H A R S H H A A — Thu, 26 Dec 2024 05:23:58 +0000

Automating infrastructure is a cornerstone of modern DevOps practices, and AWS CloudFormation is a robust tool designed to simplify this process. This guide is a step-by-step tutorial for beginners, offering detailed insights and hands-on instructions to help you confidently manage your infrastructure.

Introduction to AWS CloudFormation
Why Choose CloudFormation? Benefits Explained
Key Components of CloudFormation
Setting Up AWS CloudFormation
Deep Dive into CloudFormation Templates
Step-by-Step Guide: Deploying Your First CloudFormation Stack
Advanced Features: Nested Stacks and Change Sets
Best Practices for Using CloudFormation
Common Challenges and How to Overcome Them
Conclusion: Why Master CloudFormation?

1. Introduction to AWS CloudFormation

AWS CloudFormation is an Infrastructure as Code (IaC) service that automates the creation and management of AWS resources. Instead of manually provisioning resources, you define them in JSON or YAML templates and let CloudFormation do the rest.

Why CloudFormation Matters

Automation: Reduces manual configuration errors.
Scalability: Easily scale infrastructure as your application grows.
Consistency: Ensures uniform configurations across environments.

2. Why Choose CloudFormation? Benefits Explained

AWS CloudFormation is packed with benefits that simplify infrastructure management.

Benefit	Explanation
Automation	Saves time by automating resource creation and updates.
Cost Efficiency	Helps track costs using tags and prevents over-provisioning.
Version Control	Enables template tracking through systems like Git for collaborative workflows.
Error Reduction	Ensures consistent and tested configurations across environments.
Flexibility	Integrates with other AWS services for a seamless DevOps pipeline.

3. Key Components of CloudFormation

Before diving into deployment, let’s break down the essential parts of CloudFormation.

1. Templates

The core of CloudFormation, templates define the resources you need, such as EC2 instances, S3 buckets, or RDS databases.

2. Stacks

A stack is a collection of resources created from a single template. For example, a stack can deploy a complete application environment.

3. StackSets

Use StackSets for multi-account and multi-region deployments.

4. Change Sets

A preview of changes before updating a stack. This helps prevent unintended disruptions.

4. Setting Up AWS CloudFormation

Step 1: Prerequisites

AWS Account: Ensure you have an active account. Sign up at aws.amazon.com if you don’t already have one.
IAM Role Setup:
- Navigate to IAM Management Console.
- Create a new role with the AdministratorAccess policy.

Step 2: Access CloudFormation

Log in to the AWS Management Console.
Search for CloudFormation under the Services tab.

5. Deep Dive into CloudFormation Templates

CloudFormation templates are written in JSON or YAML and define your infrastructure as code. Below is a detailed breakdown of a template’s structure.

Basic Template Structure

AWSTemplateFormatVersion: "2010-09-09"
Description: Launch an EC2 instance
Resources:
  MyEC2Instance:
    Type: "AWS::EC2::Instance"
    Properties:
      InstanceType: "t2.micro"
      ImageId: "ami-0c02fb55956c7d316"

Section	Purpose
`AWSTemplateFormatVersion`	Specifies the template version (optional but recommended).
`Description`	Provides a brief explanation of the template.
`Resources`	Lists AWS resources to create (e.g., EC2, S3).

6. Step-by-Step Guide: Deploying Your First CloudFormation Stack

Scenario: Launching a Web Server on EC2

Step 1: Write the Template

Create a YAML file named webserver.yaml with the following content:

AWSTemplateFormatVersion: "2010-09-09"
Description: Deploy a web server
Resources:
  WebServerInstance:
    Type: "AWS::EC2::Instance"
    Properties:
      InstanceType: "t2.micro"
      ImageId: "ami-0c02fb55956c7d316"
      UserData:
        Fn::Base64: |
          #!/bin/bash
          yum update -y
          yum install -y httpd
          systemctl start httpd
          systemctl enable httpd

Step 2: Create the Stack

Go to the CloudFormation console.
Click Create Stack → With New Resources (Standard).
Choose Upload a Template File and select your webserver.yaml.

Step 3: Configure Stack Details

Provide a Stack Name: For example, WebServerStack.
Add optional tags for resource tracking.

Step 4: Review and Deploy

Verify the template and configuration.
Click Create Stack to deploy.

Step 5: Validate Deployment

Navigate to the EC2 dashboard.
Find the newly created instance and copy its public IP address.
Open a browser and access the web server using the IP.

7. Advanced Features: Nested Stacks and Change Sets

Nested Stacks

Nested stacks allow you to modularize templates for better reusability. For instance:

Main Template: References smaller templates for networking, security groups, and applications.

Change Sets

Change Sets let you preview updates to a stack before applying them.

Command:

  aws cloudformation create-change-set --stack-name WebServerStack --template-body file://updated.yaml

8. Best Practices for Using CloudFormation

Parameterize Your Templates

Use parameters to make your templates reusable across environments.
Organize Resources

Group related resources together for better readability and management.
Enable Rollbacks

Rollbacks automatically undo failed deployments.
Tag Resources

Use meaningful tags for cost tracking and management.
Validate Templates

Run the following command before deployment:

   aws cloudformation validate-template --template-body file://template.yaml

9. Common Challenges and How to Overcome Them

Challenge	Explanation	Solution
Stack Rollback	Resource creation failed.	Check the Events tab in CloudFormation for error details.
Template Errors	Syntax or configuration issues.	Validate templates before deploying.
IAM Permission Issues	Missing permissions for specific actions.	Attach the necessary IAM policies.

10. Conclusion: Why Master CloudFormation?

AWS CloudFormation offers a robust, automated way to manage infrastructure, reducing manual effort and minimizing errors. Whether you’re deploying a simple web server or orchestrating a complex multi-region setup, CloudFormation is an invaluable tool for DevOps engineers.

Key Takeaways

Learn to write reusable templates.
Experiment with advanced features like Nested Stacks and Change Sets.
Always follow best practices for a smooth deployment experience.

Start building and share your experiences—automation is the future, and CloudFormation is your gateway to mastering it! 🚀

👤 Author

Join Our Telegram Community || Follow me on GitHub for more DevOps content!

How to Crack Your First DevOps Interview: Tips and Sample Questions

H A R S H H A A — Tue, 10 Dec 2024 06:46:44 +0000

Landing your first DevOps role can be a transformative step in your career. As a DevOps engineer, you will bridge the gap between development and operations teams, leveraging automation, cloud platforms, and best practices to streamline software delivery. In this guide, we’ll cover everything you need to prepare for your first DevOps interview, including technical skills, cultural fit, and example questions.

What is DevOps?
Understanding DevOps Culture
Essential Skills for DevOps Engineers
Pre-Interview Preparation Tips
Common DevOps Interview Topics
Top DevOps Tools to Know
Sample DevOps Interview Questions with Answers
Soft Skills and Behavioral Questions
Post-Interview Best Practices
Conclusion

1. What is DevOps?

DevOps is not just a methodology but a philosophy that fosters collaboration between Development and Operations teams. It emphasizes shared responsibilities, continuous feedback, and automation to deliver software more efficiently.

Core Benefits of DevOps:

Benefit	Description
Faster Delivery	Shortened development and deployment cycles, leading to quicker software releases.
Improved Quality	Automated testing ensures code quality and reduces errors in production.
Better Collaboration	Breaking silos between development and operations fosters teamwork.
Scalability	Infrastructure as Code (IaC) allows rapid scaling of resources.

How DevOps Differs from Traditional IT Approaches:

Aspect	Traditional Approach	DevOps Approach
Development Speed	Slow due to manual processes.	Fast with CI/CD and automation.
Testing	Performed at the end of development.	Continuous, throughout the lifecycle.
Collaboration	Teams operate in silos.	Teams work collaboratively.
Deployment	Risky and infrequent.	Automated and frequent (even daily).

2. Understanding DevOps Culture

DevOps culture is a mindset that promotes collaboration, experimentation, and shared ownership. It requires breaking down traditional silos between development, operations, and QA teams to work as one cohesive unit.

Key Pillars of DevOps Culture:

Pillar	Description
Collaboration	Developers, operations, and QA teams share responsibility for delivering high-quality software.
Continuous Feedback	Feedback loops, from code reviews to production monitoring, ensure constant improvement.
Automation First	Prioritizing automation to eliminate repetitive tasks and reduce human errors.
Experimentation	Encourages innovation, with the understanding that failures are opportunities to learn.

Example: Adopting a DevOps Mindset

Old Culture: Developers write code and hand it off to operations, often leading to delays.
DevOps Culture: Developers write infrastructure code alongside application code, deploy it themselves, and monitor performance post-deployment.

3. Essential Skills for DevOps Engineers

As a DevOps engineer, you need a mix of technical expertise, process understanding, and soft skills.

In-Depth Breakdown of Skills:

Skill Category	Details	Example Tools/Technologies
Programming	Write automation scripts for builds, deployments, or monitoring.	Python, Bash, Go
Cloud Platforms	Deploy and manage scalable applications in the cloud.	AWS, Azure, GCP
CI/CD	Automate testing, builds, and deployments for continuous delivery.	Jenkins, GitLab CI/CD, GitHub Actions
Containers	Package and deploy applications in lightweight containers.	Docker
Orchestration	Manage and scale containerized applications.	Kubernetes, OpenShift
Monitoring	Monitor system health, set alerts, and analyze logs.	Prometheus, Grafana, ELK Stack
IaC (Infrastructure)	Automate infrastructure provisioning and management.	Terraform, Ansible, AWS CloudFormation

4. Pre-Interview Preparation Tips

Preparing for a DevOps interview involves more than revising concepts—you must demonstrate practical expertise.

Step-by-Step Prep Plan:

Understand the Job Requirements:
- Review the job description carefully.
- List the tools and skills mentioned, e.g., Kubernetes, Jenkins, or Terraform.
Review Key Concepts:
- Brush up on core topics like CI/CD pipelines, containerization, cloud architecture, and monitoring tools.
Practice Hands-On Tasks:
- CI/CD: Create a Jenkins pipeline to build and deploy a simple application.
- Containers: Package an app using Docker and deploy it to Kubernetes.
- IaC: Write a Terraform script to spin up AWS EC2 instances.
Prepare Real-World Scenarios:
- Be ready to discuss examples where you automated a process, reduced deployment time, or resolved a critical incident.
Prepare Your Portfolio:
- Showcase your skills via projects on GitHub. For example:
  - A CI/CD pipeline you created.
  - A Kubernetes cluster you deployed.
  - An IaC script for provisioning infrastructure.

5. Common DevOps Interview Topics

Here’s a detailed look at the core topics you’ll likely encounter during a DevOps interview:

Topic	What You Need to Know	Sample Question
Version Control	Concepts like Git, branching strategies, and resolving merge conflicts.	"Explain the difference between Git rebase and merge."
CI/CD	Designing pipelines, best practices, and troubleshooting failed builds.	"How do you implement a Jenkins pipeline for a Node.js application?"
Containerization	Dockerfile syntax, multi-stage builds, and container networking.	"What is the difference between Docker’s ENTRYPOINT and CMD?"
Orchestration	Kubernetes components (pods, deployments, services), scaling, and troubleshooting.	"How does Kubernetes handle load balancing?"
Cloud Infrastructure	Deploying resources on AWS, Azure, or GCP, and cost optimization strategies.	"Describe the differences between EC2 and Lambda."
Monitoring	Setting up alerts, visualizing metrics, and troubleshooting performance issues.	"How would you use Prometheus and Grafana to monitor an application?"
IaC (Infrastructure)	Writing scripts in Terraform or Ansible, and managing configurations.	"How do you ensure your IaC scripts are reusable and modular?"

Pro Tip:

Make your preparation interactive by setting up mini-projects for each topic:

CI/CD Pipeline: Automate the deployment of a simple Node.js or Python app using Jenkins.
Containerization: Package and deploy a web server (e.g., Nginx) in Docker.
IaC: Use Terraform to provision an AWS VPC with subnets and EC2 instances.

6. Top DevOps Tools to Know

DevOps is tool-driven, and knowing the right tools for various stages of the pipeline is crucial. Each tool serves a specific purpose, helping automate tasks, monitor systems, or streamline collaboration.

Essential DevOps Tools and Their Use Cases

Category	Purpose	Popular Tools
Version Control	Track code changes, collaborate with teams.	Git, Bitbucket, GitHub
CI/CD	Automate testing, integration, and deployment pipelines.	Jenkins, GitLab CI/CD, CircleCI
Containerization	Package applications into portable environments.	Docker, Podman
Orchestration	Manage containerized applications at scale.	Kubernetes, OpenShift
Infrastructure as Code	Automate infrastructure provisioning and management.	Terraform, CloudFormation, Ansible
Monitoring and Logging	Monitor application and system health, set alerts, analyze logs.	Prometheus, Grafana, ELK Stack, Datadog
Collaboration Tools	Facilitate communication and project management.	Slack, Microsoft Teams, Jira

How to Learn These Tools:

Hands-On Practice:
- Set up a Jenkins CI/CD pipeline for a small project.
- Deploy a Dockerized app to Kubernetes.
Documentation: Read official documentation for each tool.
Projects: Implement tools in small-scale DevOps projects.

7. Sample DevOps Interview Questions with Answers

Practicing DevOps interview questions helps you familiarize yourself with the kind of topics you may encounter.

1. CI/CD

Q: Explain the difference between Continuous Integration, Continuous Delivery, and Continuous Deployment.

A:

Continuous Integration (CI): Automating the integration of code changes into a shared repository, with automated builds and tests.
Continuous Delivery (CD): Ensuring the code is ready for deployment by automating testing and packaging.
Continuous Deployment: Automatically deploying every change that passes tests directly to production.

2. Containerization

Q: What is the difference between a virtual machine and a container?

A:

Virtual Machine: Runs an entire operating system, including a kernel, on a hypervisor.
Container: Shares the host OS kernel but isolates the application and its dependencies.

3. Infrastructure as Code (IaC)

Q: What are the advantages of Infrastructure as Code?

A:

Consistency in infrastructure provisioning.
Easy version control and rollback.
Faster deployment and scaling.

8. Soft Skills and Behavioral Questions

Soft skills are as important as technical expertise in a DevOps interview. They highlight your ability to work in cross-functional teams and handle challenges effectively.

Common Behavioral Questions and How to Answer

Question	Approach
"Describe a time you automated a task."	Use the STAR method (Situation, Task, Action, Result) to explain how you identified and automated a task.
"How do you handle disagreements in a team?"	Show your ability to listen, mediate, and find a solution that aligns with team goals.
"Tell me about a time you failed at something."	Discuss the failure, what you learned, and how you improved processes to avoid future mistakes.

Soft Skills for DevOps Engineers

Collaboration: Work closely with developers, testers, and operations.
Communication: Explain technical concepts to non-technical stakeholders.
Adaptability: Learn new tools and processes quickly in a fast-paced environment.

9. Post-Interview Best Practices

The post-interview phase is as crucial as the preparation. A thoughtful follow-up can leave a lasting impression.

Steps to Take After Your Interview

Send a Thank-You Email:
- Express gratitude for the opportunity.
- Reiterate your interest in the role and highlight one key point from the interview.

Example:
"Thank you for taking the time to interview me today. I’m especially excited about the opportunity to contribute to your CI/CD pipeline optimization project. Please let me know if there’s anything else you need from me."

Reflect on Your Performance:
- Identify questions you answered well and areas where you struggled.
- Research the topics you felt less confident about.
Stay Patient:
- Wait for the company’s response while continuing to prepare for other opportunities.

10. Conclusion

Cracking your first DevOps interview requires a blend of technical knowledge, practical experience, and soft skills. By following this guide, you can:

Master the essentials: Learn DevOps principles, tools, and methodologies.
Prepare thoroughly: Practice real-world scenarios and sample questions.
Showcase your skills: Highlight your projects and ability to solve problems.

Key Takeaways:

Focus on foundational tools and concepts (e.g., CI/CD, IaC, Kubernetes).
Practice hands-on tasks to demonstrate practical expertise.
Communicate effectively to show cultural fit and adaptability.

With determination and preparation, you’ll not only ace your first DevOps interview but also build a strong foundation for a thriving career in this exciting field. Best of luck on your journey! 🚀

👤 Author

Join Our Telegram Community || Follow me on GitHub for more DevOps content!

End-to-End AWS DevOps Project: Automating Build and Deployment of a Node.js Application to Amazon ECS using GitLab CI/CD

H A R S H H A A — Fri, 29 Nov 2024 17:58:16 +0000

Introduction
Project Overview
Technology Stack
Architecture Diagram
Step 1: Prerequisites
Step 2: Configuring GitLab as Version Control
Step 3: Preparing AWS Resources
Step 4: Building and Pushing the Docker Image
Step 5: Setting Up Amazon ECS with Fargate
Step 6: Creating the GitLab CI/CD Pipeline
Step 7: Adding Monitoring with AWS CloudWatch
Conclusion

Introduction

In this project, we will create an automated pipeline for building and deploying a Node.js application to Amazon ECS. The project showcases the use of GitLab as version control, Docker for containerization, and AWS services like ECS, ECR, and CodePipeline for orchestration and deployment.

By the end of this guide, you will have a complete understanding of the CI/CD workflow in AWS, which is critical for modern DevOps practices.

Project Overview

Objective

We will automate the following tasks:

Build a Node.js application.
Containerize the application using Docker.
Push the Docker image to Amazon ECR.
Deploy the container to Amazon ECS using Fargate.
Use GitLab CI/CD for continuous integration and deployment.
Add monitoring and notifications using AWS CloudWatch and SNS.

Technology Stack

AWS Services:
- Amazon ECS (Elastic Container Service)
- Amazon ECR (Elastic Container Registry)
- AWS CodePipeline
- AWS Security Hub
- Amazon EventBridge
- Amazon SNS
- AWS CloudWatch
Other Tools:
- GitLab: Source code management and CI/CD pipeline.
- Docker: Application containerization.
- Node.js: Sample web application framework.

Architecture Diagram

The high-level architecture for this project is as follows:

Developers push code to GitLab.
GitLab CI/CD pipeline builds and pushes a Docker image to Amazon ECR.
The image is deployed to Amazon ECS (Fargate).
Monitoring and logging are done using AWS CloudWatch.
Notifications are sent using Amazon SNS.

Step 1: Prerequisites

Ensure you have the following set up before starting:

AWS Account: With admin access to ECS, ECR, and CodePipeline.
GitLab Account: With a repository created for the Node.js application.
AWS CLI Installed: For interacting with AWS services from the command line.

   curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
   sudo installer -pkg AWSCLIV2.pkg -target /
   aws --version

Docker Installed: For building container images.

   sudo apt update
   sudo apt install docker.io
   docker --version

kubectl Installed: To interact with Amazon ECS clusters.

   curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
   chmod +x kubectl
   sudo mv kubectl /usr/local/bin/

Step 2: Configuring GitLab as Version Control

Since you are using the existing GitLab repository, follow these steps to clone and use it in your project:

2.1: Fork the Repository

Open the repository in GitLab: Nanuchi Node.js App.
Click Fork to create a copy under your GitLab account.

2.2: Clone the Repository

After forking, clone it to your local machine:

   git clone https://gitlab.com/<your-username>/node-app.git
   cd node-app

Verify that the repository contains the following:
- Application Code (Node.js):
  - server.js
  - package.json
- Dockerfile for containerization.
- .gitlab-ci.yml for CI/CD pipeline (we'll modify this later).

2.3: Push Updates (Optional)

If you want to make changes to the repository (e.g., updating code, adding more files), push the updates back:

git add .
git commit -m "Updated application for CI/CD project"
git push origin main

Step 3: Preparing AWS Resources

This step remains largely the same, but now it aligns with the Node.js application you're deploying.

3.1: Create an Amazon ECR Repository

Create a private ECR repository to store the Docker images for your application:

aws ecr create-repository --repository-name node-app

3.2: Authenticate Docker with ECR

Authenticate your local Docker client with the ECR registry:

aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <account_id>.dkr.ecr.<region>.amazonaws.com

3.3: Create an ECS Cluster

Create an ECS cluster for running the application containers:

aws ecs create-cluster --cluster-name node-app-cluster

3.4: IAM Role, VPC, and Security Groups

Follow the steps outlined earlier to:

Create the IAM task execution role.
Set up the VPC, subnets, and security groups.
Open port 3000 in the security group for application traffic.

Step 4: Building and Pushing the Docker Image

Using the cloned GitLab application, build and push the Docker image.

4.1: Build the Docker Image

Navigate to the repository's root directory and build the image:

docker build -t node-app .

4.2: Tag the Docker Image

Tag the image for your ECR repository:

docker tag node-app:latest <account_id>.dkr.ecr.<region>.amazonaws.com/node-app:latest

4.3: Push the Image to ECR

Push the image to your Amazon ECR repository:

docker push <account_id>.dkr.ecr.<region>.amazonaws.com/node-app:latest

4.4: Verify the Image

Confirm that the image has been successfully pushed:

aws ecr list-images --repository-name node-app

Step 5: Setting Up Amazon ECS with Fargate

Amazon ECS (Elastic Container Service) is a managed service that allows you to run containers. We are using Fargate, a serverless option that eliminates the need to manage EC2 instances manually. Here's a detailed walkthrough of setting up ECS for our project:

5.1: Create a Cluster

A cluster is a logical grouping of resources needed to run your tasks or services.

Run the following command to create a cluster:

   aws ecs create-cluster --cluster-name node-app-cluster

This command creates a new cluster named node-app-cluster.

Verify the Cluster:

   aws ecs list-clusters

Ensure the node-app-cluster is listed as one of the clusters.

5.2: Define a Task Definition

A task definition specifies the container settings (e.g., memory, CPU, ports) for running your application. Think of it as a blueprint for your containerized application.

Create a task-def.json file:

   {
     "family": "node-app-task",
     "executionRoleArn": "arn:aws:iam::account_id:role/ecsTaskExecutionRole",
     "networkMode": "awsvpc",
     "containerDefinitions": [
       {
         "name": "node-app-container",
         "image": "<account_id>.dkr.ecr.<region>.amazonaws.com/node-app:latest",
         "memory": 512,
         "cpu": 256,
         "essential": true,
         "portMappings": [
           {
             "containerPort": 3000,
             "hostPort": 3000,
             "protocol": "tcp"
           }
         ],
         "logConfiguration": {
           "logDriver": "awslogs",
           "options": {
             "awslogs-group": "/ecs/node-app",
             "awslogs-region": "<region>",
             "awslogs-stream-prefix": "ecs"
           }
         }
       }
     ],
     "requiresCompatibilities": ["FARGATE"],
     "cpu": "256",
     "memory": "512"
   }

Replace <account_id> and <region> with your AWS account ID and region.
Ensure executionRoleArn points to a valid ECS task execution role.

   aws ecs register-task-definition --cli-input-json file://task-def.json

This registers the blueprint with ECS.

5.3: Create a Service to Manage the Task

An ECS service ensures that the required number of tasks are running and enables load balancing for the tasks.

Create a service:

   aws ecs create-service \
     --cluster node-app-cluster \
     --service-name node-app-service \
     --task-definition node-app-task \
     --desired-count 1 \
     --launch-type FARGATE \
     --network-configuration "awsvpcConfiguration={subnets=[subnet-xxx],securityGroups=[sg-xxx],assignPublicIp=ENABLED}" \
     --region <region>

Replace subnet-xxx and sg-xxx with the IDs of your VPC's public subnet and security group.
desired-count is the number of tasks to run.

Verify the Service:

   aws ecs describe-services --cluster node-app-cluster --services node-app-service

Ensure the service is active and running.

5.4: Test the Application

Find the public IP of your task:

   aws ecs list-tasks --cluster node-app-cluster

Use the task ID to describe the task and find the public IP address:

   aws ecs describe-tasks --cluster node-app-cluster --tasks <task_id>

Access your application in the browser using the public IP:

   http://<public_ip>:3000

Step 6: Creating the GitLab CI/CD Pipeline

GitLab CI/CD automates the build and deployment process, ensuring the application is always up to date. Follow these steps to set up the pipeline:

6.1: Add `.gitlab-ci.yml`

This file defines the stages, jobs, and commands for the pipeline.

Add the following .gitlab-ci.yml file to the root of your project:

   stages:
     - build
     - deploy

   build:
     image: docker:latest
     services:
       - docker:dind
     script:
       - docker build -t node-app .
       - docker tag node-app <account_id>.dkr.ecr.<region>.amazonaws.com/node-app:latest
       - aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <account_id>.dkr.ecr.<region>.amazonaws.com
       - docker push <account_id>.dkr.ecr.<region>.amazonaws.com/node-app:latest

   deploy:
     image: amazon/aws-cli:latest
     script:
       - aws ecs update-service --cluster node-app-cluster --service node-app-service --force-new-deployment --region <region>

Key Steps Explained:
- Build Stage:
  - Builds a Docker image from your Dockerfile.
  - Tags the image with the ECR repository URL.
  - Pushes the image to Amazon ECR.
- Deploy Stage:
  - Updates the ECS service to use the latest image in Amazon ECR.

6.2: Configure Variables in GitLab

Go to Settings → CI/CD → Variables in your GitLab repository and add the following environment variables:

AWS_ACCESS_KEY_ID: Your AWS access key.
AWS_SECRET_ACCESS_KEY: Your AWS secret key.
AWS_REGION: Your AWS region.

Step 7: Adding Monitoring with AWS CloudWatch

CloudWatch enables monitoring and logging for your application and infrastructure.

7.1: Set Up CloudWatch Logs

Create a Log Group:

   aws logs create-log-group --log-group-name /ecs/node-app

Create a Log Stream:

   aws logs create-log-stream --log-group-name /ecs/node-app --log-stream-name app-logs

Integrate Logs with ECS Task Definition: In the task definition (task-def.json), ensure the logConfiguration section is as follows:

   "logConfiguration": {
     "logDriver": "awslogs",
     "options": {
       "awslogs-group": "/ecs/node-app",
       "awslogs-region": "<region>",
       "awslogs-stream-prefix": "ecs"
     }
   }

7.2: Set Up Alarms for Monitoring

You can set up alarms in CloudWatch to monitor metrics such as CPU usage, memory, and application errors.

Create an Alarm:

   aws cloudwatch put-metric-alarm \
     --alarm-name HighCPUUsage \
     --metric-name CPUUtilization \
     --namespace AWS/ECS \
     --statistic Average \
     --period 300 \
     --threshold 80 \
     --comparison-operator GreaterThanThreshold \
     --evaluation-periods 1 \
     --alarm-actions <sns_topic_arn>

Receive Notifications: Create an SNS topic to send notifications:

   aws sns create-topic --name ecs-alerts
   aws sns subscribe --topic-arn <sns_topic_arn> --protocol email --notification-endpoint <your_email>

Now, you will receive email notifications for high CPU usage or other alerts.

👤 Author

Join Our Telegram Community || Follow me on GitHub for more DevOps content!

Writing Kubernetes Manifests: From Beginner to Advanced

H A R S H H A A — Thu, 21 Nov 2024 16:45:31 +0000

Kubernetes manifests are essential for defining, deploying, and managing workloads in Kubernetes clusters. As a DevOps engineer, mastering the creation of Kubernetes manifests helps you efficiently manage containerized applications, ensure scalability, and streamline deployments.

This article takes you from the fundamentals of writing Kubernetes manifests to advanced techniques and best practices. Whether you’re new to Kubernetes or looking to enhance your expertise, this guide will equip you with the knowledge to write effective and optimized manifests.

What Are Kubernetes Manifests?
Why Learn Kubernetes Manifests?
Understanding Manifest Structure
Basic Manifest Examples
Intermediate Manifest Concepts
Advanced Manifest Techniques
Best Practices for Writing Manifests
Common Mistakes and How to Avoid Them
Conclusion

1. What Are Kubernetes Manifests?

A Kubernetes manifest is a configuration file, usually written in YAML or JSON, used to describe the desired state of resources in a Kubernetes cluster. It provides a declarative way to tell Kubernetes what you want your application and infrastructure to look like. Kubernetes then reconciles the actual state of the cluster with the desired state defined in these manifests.

Key Features:

Declarative Configuration:

You describe the desired end-state (e.g., “I want 3 replicas of my application running”), and Kubernetes takes care of ensuring this state is achieved.
Portability:

Kubernetes manifests can be reused across different environments (e.g., development, staging, and production). For example, you can deploy the same manifest in any Kubernetes cluster, ensuring consistent deployments.
Automation:

Manifests integrate seamlessly with CI/CD pipelines, enabling automated application builds, tests, and deployments.

Example Use Case:

Imagine you have a web application that runs on multiple replicas, behind a load balancer.

Without Kubernetes: You would manually configure servers, set up load balancing, and monitor each instance.
With Kubernetes Manifests: All this complexity is abstracted into a few lines of YAML. Kubernetes ensures the replicas are running, load balancing is in place, and failed instances are automatically replaced.

2. Why Learn Kubernetes Manifests?

For DevOps engineers, Kubernetes manifests are a core skill. Understanding them unlocks powerful capabilities in managing and deploying containerized applications.

Reasons to Learn:

Streamlined Deployments:

Automate complex setups (e.g., multi-container applications) with ease by defining resources declaratively.
Scalability:

Modify manifests to scale your application vertically (more resources per container) or horizontally (more containers).
Consistency Across Environments:

Use the same manifests for local testing, staging, and production to ensure consistency and reduce deployment errors.
Customization:

Tailor manifests to specific workloads, such as deploying applications in specific namespaces or attaching storage volumes for stateful applications.
Integration with DevOps Workflows:

Kubernetes manifests work seamlessly with CI/CD tools like Jenkins, GitHub Actions, and GitLab CI for automated deployments.

Real-World Benefits:

Faster deployment times.
Reduced manual intervention.
Easier debugging of infrastructure and application issues.

3. Understanding Manifest Structure

Before writing manifests, it’s crucial to understand their structure. Each Kubernetes manifest follows a standard format that specifies what resource to create and its desired configuration.

Core Components of a Manifest:

1. apiVersion

Specifies the Kubernetes API version for the resource type. Each resource type (e.g., Pod, Service) belongs to a specific API group and version.

Example:
- v1 for Pod and Service.
- apps/v1 for Deployment.
Why It Matters: Using the correct API version ensures compatibility with the Kubernetes cluster version.

2. kind

Defines the type of resource being created. Common resource types include:

Pod
Deployment
Service
ConfigMap
Secret

Each resource serves a specific purpose. For instance, a Pod represents a single container or a group of tightly coupled containers, while a Deployment manages replica sets and rolling updates.

3. metadata

Contains identifying information about the resource, such as:

name: A unique name for the resource.
labels: Key-value pairs used to organize and query resources.
namespace: Isolates the resource within a specific environment (e.g., dev, prod).

Example:

metadata:
  name: my-app
  namespace: development
  labels:
    app: my-app
    environment: dev

4. spec

The heart of the manifest, defining the desired state of the resource. This section is resource-specific. For example:

In a Pod, spec defines the containers, images, and ports.
In a Service, spec defines how to expose applications (e.g., via ClusterIP, NodePort, or LoadBalancer).

YAML vs. JSON

Although Kubernetes supports both YAML and JSON, YAML is more commonly used because:

It’s human-readable and easier to write.
It supports comments, which are helpful for documentation.

Example Comparison:

YAML:

kind: Pod
apiVersion: v1
metadata:
  name: my-pod
spec:
  containers:
  - name: nginx
    image: nginx:latest

JSON:

{
  "kind": "Pod",
  "apiVersion": "v1",
  "metadata": {
    "name": "my-pod"
  },
  "spec": {
    "containers": [
      {
        "name": "nginx",
        "image": "nginx:latest"
      }
    ]
  }
}

4. Basic Manifest Examples

Let’s look at a few simple manifests for common Kubernetes resources.

4.1 Pod Manifest

A Pod is the smallest deployable unit in Kubernetes.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
  labels:
    app: web
spec:
  containers:
  - name: nginx
    image: nginx:1.23
    ports:
    - containerPort: 80

Explanation:

Creates a single Pod named my-pod running an nginx container.
Labels (app: web) help group and query resources.
The container exposes port 80 internally.

4.2 Deployment Manifest

Deployments provide declarative updates for Pods and ReplicaSets.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.23
        ports:
        - containerPort: 80

Explanation:

Deploys three replicas of the nginx Pod.
The selector ensures the Pods are linked to the Deployment.
The template defines the Pod specification for the replicas.

4.3 Service Manifest

Services expose Pods to the network.

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  type: ClusterIP

Explanation:

Exposes Pods labeled app: nginx.
Maps port 80 of the Service to port 80 of the Pods.
ClusterIP is the default service type, exposing the Pods within the cluster.

4.4 ConfigMap Manifest

ConfigMaps store configuration data in key-value pairs.

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  log_level: debug
  environment: production

Explanation:

Stores non-sensitive application configurations, such as environment variables or logging levels.
Accessible to Pods as environment variables or mounted volumes.

5. Intermediate Manifest Concepts

As you grow in Kubernetes expertise, understanding intermediate concepts like multi-container pods, resource management, and secrets becomes essential. These enhance your ability to handle more complex workloads efficiently.

5.1 Multi-Container Pods

A single Pod can host multiple containers that share the same lifecycle and storage. This is useful when containers need to work together closely, such as a main application container and a sidecar container for logging or monitoring.

Example:

apiVersion: v1
kind: Pod
metadata:
  name: multi-container-pod
spec:
  containers:
  - name: app
    image: my-app:latest
  - name: sidecar-logger
    image: fluentd:latest
    volumeMounts:
    - name: shared-logs
      mountPath: /var/log
  volumes:
  - name: shared-logs
    emptyDir: {}

Explanation:

The app container runs the main application, while the sidecar-logger collects logs.
Both containers share a temporary volume (emptyDir).

5.2 Resource Requests and Limits

Properly managing CPU and memory ensures that no single Pod consumes excessive cluster resources.

Example:

resources:
  requests:
    memory: "128Mi"
    cpu: "250m"
  limits:
    memory: "256Mi"
    cpu: "500m"

Explanation:

Requests: The guaranteed minimum resources allocated to a container.
Limits: The maximum resources a container can use.

By specifying these values, you prevent resource starvation and overloading.

5.3 Secrets

Secrets store sensitive data like passwords, tokens, and SSH keys securely.

Example:

apiVersion: v1
kind: Secret
metadata:
  name: db-secret
type: Opaque
data:
  username: dXNlcm5hbWU=  # Base64-encoded 'username'
  password: cGFzc3dvcmQ=  # Base64-encoded 'password'

Explanation:

Secrets are Base64-encoded and stored securely.
They can be injected into Pods as environment variables or mounted as files.

5.4 Persistent Storage

For stateful applications, persistent storage ensures data is not lost when a Pod is terminated.

Example (Persistent Volume Claim):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Explanation:

The PersistentVolumeClaim requests 1GB of storage.
This is used by Pods requiring persistent data.

6. Advanced Manifest Techniques

As your Kubernetes usage matures, advanced techniques enable you to optimize workloads, enhance security, and streamline operations.

6.1 Multi-Stage Manifests

Break your manifests into multiple stages for modularity and reusability. Use Helm charts or Kustomize for templating and environment-specific configurations.

Example with Kustomize:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- base/deployment.yaml
- base/service.yaml
patchesStrategicMerge:
- overlays/dev/replicas.yaml
- overlays/prod/limits.yaml

Explanation:

Kustomize simplifies creating environment-specific configurations without duplicating manifests.

6.2 Rolling Updates

Deploy updates with zero downtime using rolling updates.

Example (Deployment):

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 1
    maxSurge: 1

Explanation:

maxUnavailable: The number of Pods that can be unavailable during an update.
maxSurge: The number of extra Pods allowed during an update.

6.3 Horizontal Pod Autoscaling

Automatically scale Pods based on CPU, memory, or custom metrics.

Example:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        averageUtilization: 70

Explanation:

Scales the application between 2 and 10 replicas based on CPU usage.

7. Best Practices for Writing Manifests

Organize Resources with Namespaces:

Use namespaces for logical separation of environments or applications (e.g., dev, staging, prod).
Use Labels and Annotations:

Attach metadata to resources to enable efficient management and querying.
Avoid Hardcoding Values:

Use ConfigMaps and Secrets to separate configuration from code.
Version Control Your Manifests:

Store manifests in Git repositories for tracking and collaboration.
Validate Manifests Before Applying:

Use kubectl apply --dry-run=client to check for syntax errors.
Use Templating Tools:

Helm and Kustomize simplify creating reusable, environment-specific configurations.

8. Common Mistakes and How to Avoid Them

Incorrect Indentation in YAML:

YAML is sensitive to spaces. Use proper indentation to avoid syntax errors.
Skipping Resource Requests and Limits:

Failing to define these can lead to resource starvation or cluster overload.
Not Using Probes:

Without liveness and readiness probes, Kubernetes cannot detect or restart failing containers.
Hardcoding Sensitive Data:

Always use Secrets to store sensitive information like passwords or API keys.
Forgetting Labels and Selectors:

Misaligned labels and selectors can break connections between resources like Deployments and Services.

9. Conclusion

Kubernetes manifests are the cornerstone of managing workloads in Kubernetes. By mastering the basics, progressing to intermediate concepts, and leveraging advanced techniques, you can efficiently deploy and manage containerized applications at scale.

As a DevOps engineer, understanding and applying these concepts will make your Kubernetes deployments robust, scalable, and maintainable. Keep experimenting, validate your manifests, and incorporate best practices to stay ahead in the rapidly evolving DevOps ecosystem.

👤 Author

Join Our Telegram Community || Follow me on GitHub for more DevOps content!

DEV Community: ProDevOpsGuy Tech Community

Loop Engineering: The Karpathy Method - and the workflow that just made it 5x better

Part 1 · The Basics

Part 2 · The Karpathy Loop

Part 3 · The Building Blocks

Part 4 · What Comes After Karpathy

Part 5 · Try It Yourself

Part 6 · The Honest Part

Credits: codila

Top 10 Python Libraries Every DevOps Engineer Should Master (With Use Cases & Code)

1. 🔹 boto3 – AWS Automation and Resource Management

➤ What is boto3?

➤ Why It's Useful

➤ Example: List all S3 buckets

✅ Real-World Use Cases

2. 🔹 paramiko – SSH into Remote Servers

➤ What is paramiko?

➤ Why It's Useful

➤ Example: Restart a remote web server

✅ Real-World Use Cases

3. 🔹 docker (Python SDK) – Manage Docker Containers

➤ What is docker SDK for Python?

➤ Why It's Useful

➤ Example: Start a new container with a mounted volume

✅ Real-World Use Cases

4. 🔹 kubernetes – Interact with Kubernetes API

➤ What is the kubernetes Python client?

➤ Why It's Useful

➤ Example: Delete a Kubernetes pod

✅ Real-World Use Cases

5. 🔹 pyyaml – Parse and Write YAML Files

➤ What is pyyaml?

➤ Why It's Useful

➤ Example: Update Kubernetes deployment replicas

✅ Real-World Use Cases

6. 🔹 requests – Make HTTP Requests Easily

➤ What is requests?

➤ Why It's Useful

➤ Example: Send a Slack notification after deployment

✅ Real-World Use Cases

7. 🔹 fabric – Remote Execution at a Higher Level

➤ What is fabric?

➤ Why It's Useful

➤ Example: Automate deployment steps

✅ Real-World Use Cases

8. 🔹 pytest – Write and Run Tests

➤ What is pytest?

➤ Why It's Useful

➤ Example: Test a web service endpoint

✅ Real-World Use Cases

9. 🔹 ansible-runner – Programmatically Run Ansible

➤ What is ansible-runner?

➤ Why It's Useful

➤ Example: Run a playbook and get output

✅ Real-World Use Cases

10. 🔹 sys – System-Specific Parameters

➤ What is sys?

➤ Why It's Useful

➤ Example: Handle arguments in a script

✅ Real-World Use Cases

🚀 Wrapping Up

✨ Bonus Tip:

🛠️ Author & Community

📧 Connect with me:

📢 Stay Connected

Full-Stack Application Deployment Guide Using Docker, Kubernetes, Jenkins, and Prometheus Monitoring

Table of Contents

1. Overview & Prerequisites

2. Application Preparation

Clone the Repository

Review Source Structure

Environment Setup

3. Dockerization of MERN App

Backend Dockerfile (backend/Dockerfile)

Frontend Dockerfile (frontend/Dockerfile)

Docker Compose for Local Testing (docker-compose.yml)

Test Docker Setup

4. Jenkins CI/CD Setup

Jenkins Installation Sample

Jenkins Pipeline Example (Jenkinsfile)

Backend Dockerfile (`backend/Dockerfile`)

Frontend Dockerfile (`frontend/Dockerfile`)

Docker Compose for Local Testing (`docker-compose.yml`)

Jenkins Pipeline Example (`Jenkinsfile`)

Example Kubernetes Manifests (`k8s/`)

MongoDB Deployment and Service (`k8s/mongo-deployment.yaml`)

Backend Deployment and Service (`k8s/backend-deployment.yaml`)

Frontend Deployment and Service (`k8s/frontend-deployment.yaml`)

Ingress Example (`k8s/ingress.yaml`)