DEV Community: Sourav kumar

How Your Microservice Actually Gets Deployed on Kubernetes (Model 1 vs Model 2)

Sourav kumar — Mon, 16 Mar 2026 11:13:06 +0000

If you’re building or operating microservices on Kubernetes (especially EKS), one of the biggest architectural decisions is:

Where do application code, Helm charts, and deployment configurations live — and how do they flow to production?

In real production environments, two models dominate:

1. Model 1 — Same Repo (Service-Owned Deployments)
2. Model 2 — Hybrid GitOps (Platform-Driven)

This guide explains both end-to-end: repositories, ECR usage, CI/CD, versioning, folder structures, deployment flow, and when to use each.

Model 1 — App + Helm in the SAME Repository

Each microservice owns everything needed to run it.
One repo per service contains:

Application source code
Dockerfile
Helm chart
Environment values
CI/CD pipeline

Deployment is usually CI-driven (not pure GitOps)

Typical Repository Structure

payment-service/
├── src/                          # Application source code
│   ├── main.py / index.js / App.java
│   ├── config/
│   └── modules/
│
├── tests/                        # Unit & integration tests
│   ├── unit/
│   └── integration/
│
├── Dockerfile                    # Container build instructions
├── .dockerignore                 # Files excluded from image build
│
├── helm/                         # Helm chart for this service
│   ├── Chart.yaml                # Chart metadata
│   ├── values.yaml               # Default configuration
│   ├── values-dev.yaml           # Dev environment overrides
│   ├── values-staging.yaml       # Staging overrides
│   ├── values-prod.yaml          # Production overrides
│   ├── .helmignore               # Files Helm should ignore
│   │
│   ├── templates/                # Kubernetes manifest templates
│   │   ├── _helpers.tpl          # Template helper functions ⭐
│   │   ├── deployment.yaml       # Deployment resource
│   │   ├── service.yaml          # Service resource
│   │   ├── ingress.yaml          # Ingress resource
│   │   ├── serviceaccount.yaml   # ServiceAccount (if needed)
│   │   ├── hpa.yaml              # Horizontal Pod Autoscaler
│   │   ├── configmap.yaml        # ConfigMap
│   │   ├── secret.yaml           # Secret (optional)
│   │   ├── poddisruptionbudget.yaml
│   │   ├── networkpolicy.yaml    # Network security rules
│   │   ├── NOTES.txt             # Post-install info
│   │
│   └── charts/                   # Subcharts (dependencies)
│       └── (empty or dependencies)
│
├── scripts/                      # Optional helper scripts
│   └── deploy.sh
│
├── README.md                     # Service documentation
├── .gitignore
│
└── .github/
    └── workflows/
        └── ci-cd.yaml            # GitHub Actions pipeline

Deep Explanation of Important Helm Files

Chart.yaml — Chart Metadata : Defines the Helm chart identity.

apiVersion: v2
name: payment-service
description: Helm chart for Payment Service
type: application
version: 1.2.0          # Chart version
appVersion: "1.2.0"     # Application version

👉 version = chart version
👉 appVersion = container app version

values.yaml — Default Configuration : Base config used unless overridden.

replicaCount: 2

image:
  repository: <ecr-url>/payment-service
  tag: "latest"
  pullPolicy: IfNotPresent

service:
  type: ClusterIP
  port: 80

Environment-Specific Values Files: Override defaults for each environment.

values-dev.yaml

replicaCount: 1

image:
  tag: "dev"

resources:
  requests:
    cpu: 100m
    memory: 128Mi

values-prod.yaml

replicaCount: 4

image:
  tag: "1.2.0"

resources:
  requests:
    cpu: 500m
    memory: 512Mi

templates/_helpers.tpl (VERY IMPORTANT)

Reusable template functions.

Used for:

Naming conventions
Labels
Chart metadata
DRY templates

Example:

{{- define "payment-service.fullname" -}}
{{ .Release.Name }}-payment-service
{{- end }}

Container Image Storage (ECR)

Each service has its own ECR repository:

123456789.dkr.ecr.ap-south-1.amazonaws.com/payment-service

ECR stores Docker images only — not Helm charts.

Complete Deployment Flow (Model-1, CI-Driven Helm on EKS)

1️⃣ Developer Pushes Code Change

A change is pushed to the service repository:
Examples:

Business logic update
New environment variable
Resource changes
Bug fix
Dependency update

Git event triggers CI.

Developer → Git push → main branch

2️⃣ CI Pipeline Starts

Triggered by:

GitHub Actions
Pipeline checks out code.

3️⃣ Build Container Image

Docker image is created from the Dockerfile.

docker build -t payment-service:1.5.0 .

Tagging usually includes:

Semantic version
Git SHA
Build number
Timestamp

Example:

payment-service:1.5.0
payment-service:sha-7f3a9c

4️⃣ Authenticate to AWS ECR

Pipeline logs into ECR:

aws ecr get-login-password \
 | docker login --username AWS --password-stdin <ecr-url>

5️⃣ Push Image to ECR

docker push <ecr-url>/payment-service:1.5.0

Now ECR stores the new image.
👉 This is the artifact Kubernetes will run.

6️⃣ Pipeline Prepares Deployment

Helm chart references the new image tag.
This may happen in two ways:

🔹 Method A — Dynamic Override (most common)

helm upgrade payment-service ./helm \
  --set image.tag=1.5.0 \
  -f values-prod.yaml
🔹 Method B — Update values file

Pipeline edits values file with new tag.

7️⃣ Helm Upgrade / Install Executes

Pipeline runs Helm:

helm upgrade --install payment-service ./helm \
  -f values-prod.yaml

Helm does NOT run containers.
👉 It generates Kubernetes manifests and sends them to the cluster.

8️⃣ Kubernetes API Server Receives New Spec

Deployment resource updated:

spec:
  template:
    spec:
      containers:
        - image: <ecr-url>/payment-service:1.5.0

Changing image tag updates the Pod template hash.

9️⃣ Deployment Controller Triggers Rolling Update

Kubernetes detects a change in Pod template.

It creates a new ReplicaSet.

Old ReplicaSet → payment-service-abc123
New ReplicaSet → payment-service-def456

🔟 Scheduler Assigns Pods to Nodes

New Pods created but initially Pending.

Scheduler selects nodes based on:

Available CPU/memory
Node selectors
Taints/tolerations
Affinity rules

1️⃣1️⃣ Kubelet Starts Pod on Selected Node

Kubelet (agent running on node) receives instruction:

👉 “Run this container image”

⭐ THIS IS THE CRITICAL PART
🔥 How Cluster Pulls Image from ECR
1️⃣2️⃣ Node Needs Image Locally

Container runtime (containerd/Docker) checks:

👉 Is this image already cached on the node?

✔️ If YES → use cached image
❌ If NO → pull from registry
1️⃣3️⃣ Authentication to ECR

In EKS, nodes are allowed to pull from ECR via IAM.

How authentication happens:

👉 Node IAM Role (Instance Profile) has permission:

ecr:GetAuthorizationToken
ecr:BatchCheckLayerAvailability
ecr:GetDownloadUrlForLayer
ecr:BatchGetImage

1️⃣4️⃣ Kubelet Requests Auth Token

AWS provides temporary registry credentials.

1️⃣5️⃣ Container Runtime Pulls Image Layers

Image downloaded layer by layer from ECR.

ECR → Internet/VPC Endpoint → Node

Large images take longer.

🔐 Important: Private Cluster Scenario

If nodes have no internet:

👉 Use VPC Interface Endpoint for ECR.

1️⃣6️⃣ Image Stored Locally on Node

After download:

/var/lib/containerd/

Future Pods may reuse it.

1️⃣7️⃣ Container Created & Started

Runtime launches container process.

Now Pod enters:

ContainerCreating → Running
1️⃣8️⃣ Health Checks Begin

Kubernetes executes:

Readiness probe

Liveness probe

Startup probe

If probes fail → Pod restarted.

1️⃣9️⃣ Service Starts Routing Traffic

Once Pod is Ready:

Added to Endpoints list

Load balancer can send traffic

2️⃣0️⃣ Rolling Update Completes

Old Pods gradually terminated.

Zero-downtime deployment achieved.

🏆 Complete Visual Flow

Code change pushed
        ↓
CI pipeline triggered
        ↓
Build Docker image
        ↓
Push image to ECR
        ↓
Helm upgrade executed
        ↓
Kubernetes Deployment updated
        ↓
New ReplicaSet created
        ↓
Pods scheduled to nodes
        ↓
Node authenticates to ECR
        ↓
Image pulled to node
        ↓
Container started
        ↓
Health checks pass
        ↓
Service routes traffic

Team Collaboration

Developers can modify:

Environment variables
Resource limits
Feature configs

DevOps/Platform team enforces:

PR reviews
Branch protection
Deployment policies

✅ Advantages of Model 1

✔ Simple to implement
✔ Fast delivery
✔ Single source of truth
✔ Easy debugging
✔ Ideal for small–mid teams

❌ Limitations

⚠ Harder governance at scale
⚠ Risk of inconsistent configs
⚠ Limited multi-cluster control
⚠ Not ideal for hundreds of services

Model 2 — Hybrid GitOps (Enterprise Standard)

Separates application ownership from runtime control.

Developers own code
Platform team owns deployments

Deployments are GitOps-driven (Argo CD / Flux).

Architecture Components
Application Repository (Service Team)

Contains:

Source code
Dockerfile
Sometimes Helm chart templates

🏗️ Platform / GitOps Repository

Contains:

Environment configs
Helm values per environment
ArgoCD applications
Cluster definitions

App Repository Structure

payment-service/
├── src/
├── Dockerfile
├── tests/
└── helm/ (optional)

Deployment (GitOps) Repository Structure

k8s-platform-config/
├── dev/
│   └── payment-service/
│       └── values.yaml
├── staging/
│   └── payment-service/
│       └── values.yaml
├── prod/
│   └── payment-service/
│       └── values.yaml
└── argocd-apps/
    └── payment-service.yaml

ECR Usage
Same as Model 1:

👉 ECR stores container images only.

Some enterprises also store Helm charts as OCI artifacts, but not required.

Deployment Flow — End-to-End

1️⃣ Developer Pushes Code

To app repository.

2️⃣ CI Builds and Pushes Image to ECR

docker build -t payment-service:2.1.0 .
docker push <ecr-url>/payment-service:2.1.0

3️⃣ Deployment Repo Updated

Image tag updated in environment config:

image:
  repository: <ecr-url>/payment-service
  tag: 2.1.0

This update may be:

Manual PR
Automated bot
Promotion pipeline

4️⃣ GitOps Tool Detects Change

Argo CD / Flux watches the repo.

5️⃣ Automatic Deployment

GitOps controller syncs cluster state.

👉 No direct kubectl or helm from CI.

🔄 Versioning in Model 2

Application Version: Container image tag.
Deployment Version: Tracked by Git commits in deployment repo.

Environment Promotion

Controlled separately:
Dev → Staging → Prod
Often via PR approvals.

Governance Benefits

Platform team controls:

Resource policies
Security standards
Network rules
Secrets integration
Multi-cluster routing

Developers cannot accidentally break production infrastructure.

Advantages of Model 2

✔ Scales to hundreds of services
✔ Strong governance
✔ Full audit trail
✔ Safer production changes
✔ Multi-environment consistency
✔ True GitOps workflows

❌ Trade-offs

⚠ More complex setup
⚠ Requires platform team
⚠ Slower initial development
⚠ Additional repositories to manage

Model 1 vs Model 3 — Side-by-Side

When to Use Each Model

Model 1 — Best For

Startups
Small–mid teams
Rapid development
Few services
CI-based deployments

Model 3 — Best For

Large organizations
Platform engineering culture
Many microservices
Multi-cluster setups
Compliance requirements
Production at scale

Final Takeaway

There is no universally “best” model.

👉 Organizations typically evolve:

Startup → Model 1
Growing company → Hybrid approaches
Enterprise → Model 3 (GitOps)

Understanding both gives you the ability to design systems for any scale.

The Ultimate Guide to Kubernetes Architecture — A Complete CKA-Level Deep Dive

Sourav kumar — Mon, 13 Oct 2025 11:21:39 +0000

If you’re preparing for the Certified Kubernetes Administrator (CKA) exam or aiming to truly master Kubernetes, understanding the architecture is non-negotiable.
Kubernetes isn’t just a collection of containers; it’s a distributed control system — one of the most elegant and resilient designs in modern cloud computing.

This blog is your complete study reference — explaining every component, process, call flow, and real-world scenario — so you’ll never have to read another Kubernetes architecture guide again.

The Big Picture: Kubernetes Cluster Overview

A Kubernetes Cluster is made up of two planes:

Control Plane (Master Node): Responsible for making global decisions — scheduling, monitoring, scaling, and maintaining the desired state.
Worker Nodes: Where the actual applications (Pods) run.

Core Concept

Kubernetes operates on a Declarative Model — you tell it what you want, and it constantly works to make the current state match that desired state.

CONTROL PLANE COMPONENTS
Kube-API Server — The Front Door of Kubernetes

Everything in Kubernetes goes through the Kube-API Server — it’s the central communication hub for both internal and external components.

What It Does

Exposes the Kubernetes API (via RESTful interface).
Validates and processes incoming requests.
Persists the cluster state in etcd.
Acts as a message bus for all controllers and nodes.

Internal Flow: When You Run kubectl apply -f pod.yaml

Let’s trace what really happens:

1️⃣ Request Sent:

kubectl sends an HTTPS REST call to the Kube-API Server (default port 6443).

2️⃣ Authentication:

API Server checks who you are.
Common methods:
- Certificates (client certs signed by cluster CA)
- Bearer tokens (ServiceAccount tokens, OIDC, etc.)
- Static tokens in /etc/kubernetes/pki/
Example file for basic authentication (not recommended in production):

/etc/kubernetes/pki/users.csv

3️⃣ Authorization:

Once authenticated, the request goes through RBAC (Role-Based Access Control) or ABAC.
Example check: “Does this user have permission to create Pods in this namespace?”
RBAC data is stored in Roles and ClusterRoles:

kubectl get clusterrolebinding

4️⃣ Admission Control:

Final checkpoint before persistence.
Admission Controllers can mutate (modify) or validate requests.
Examples:
- NamespaceLifecycle: Prevents use of deleted namespaces.
- LimitRanger: Ensures Pod requests don’t exceed limits.
- DefaultStorageClass: Assigns default storage class to PVCs.

Check which admission plugins are active:

cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep admission

5️⃣ Validation and Persistence:

API Server validates your Pod spec, then stores it in etcd.

6️⃣ Notification to Controllers/Scheduler:

The API Server uses watchers to notify other control plane components (like Scheduler and Controller Manager) that a new Pod exists.

CKA Scenario:

If the API Server fails, the entire cluster becomes read-only — existing Pods will run, but you can’t create or delete new resources.
Check its health:

kubectl get componentstatuses

etcd — The Source of Truth

etcd is a distributed key-value store that stores all cluster data — think of it as the “brain” of Kubernetes.

What It Stores

Everything!
Cluster state
Secrets, ConfigMaps
Node and Pod objects
RBAC policies
CRDs (Custom Resources)

It uses the Raft consensus algorithm to ensure strong consistency across all master nodes.

Common Commands
Backup etcd before upgrades or maintenance:

ETCDCTL_API=3 etcdctl snapshot save /tmp/etcd_backup.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Restore from snapshot:

ETCDCTL_API=3 etcdctl snapshot restore /tmp/etcd_backup.db \
  --data-dir /var/lib/etcd-from-backup

CKA Scenario:

If your cluster becomes unhealthy or components fail to start after an upgrade, restoring etcd from a snapshot is often the safest recovery path.

Kube-Controller-Manager — The Enforcer of Desired State

This component runs multiple controllers that continuously monitor and reconcile the cluster’s actual state against the desired state (stored in etcd).

Key Controllers

Node Controller: Detects node failures.
Replication Controller / ReplicaSet Controller: Ensures Pod count.
Endpoint Controller: Connects Services to Pods.
Job Controller: Manages batch job completion.
Namespace Controller: Cleans up resources on namespace deletion .

Reconciliation Loop

1️⃣ Controller queries the API Server for the desired object state.
2️⃣ It compares it with the current state.
3️⃣ If there’s a mismatch, it acts — e.g., spawns or deletes Pods.

CKA Tip:

Understanding this loop helps in troubleshooting “CrashLoopBackOff” or “Pending” issues — it’s always a reconciliation failure somewhere.

Kube-Scheduler — The Intelligent Matchmaker

Assigns Pods to Nodes.

Process:

1️⃣ Watches the API Server for new Pods (with no Node assigned).
2️⃣ Filters nodes based on feasibility (CPU, memory, taints, affinity).
3️⃣ Scores feasible nodes based on priorities (spread, image locality, etc.).
4️⃣ Assigns the highest scoring Node to the Pod.

CKA Tip:

Check why a Pod didn’t schedule:

kubectl describe pod <pod-name>

Look for messages like:

“0/3 nodes are available: 1 node(s) had insufficient memory, 2 node(s) had taints.”

*Cloud-Controller-Manager — The Cloud Integrator
*
Handles tasks that interact with cloud providers:

Creates load balancers for Services of type LoadBalancer.
Attaches persistent volumes (like AWS EBS, GCP PD).
Manages node lifecycle in cloud environments.

WORKER NODE COMPONENTS

Kubelet — The Node’s Caretaker

The Kubelet runs on every node and ensures containers are up and healthy.

Responsibilities:

Watches API Server for Pods assigned to its Node.
Works with the Container Runtime (e.g., containerd or CRI-O).
Monitors containers using liveness and readiness probes.
Reports status to the API Server.

Pod Lifecycle Example:

1️⃣ Scheduler assigns a Pod → Kubelet receives instruction.
2️⃣ Kubelet pulls images → starts containers.
3️⃣ Runs probes defined in the Pod spec.
4️⃣ Reports health & status.

Kubelet Logs Useful for troubleshooting:

journalctl -u kubelet -f

Kube-Proxy — The Network Router

Manages network rules (via iptables or IPVS) for service-to-pod and pod-to-pod communication.

Functionality:

Maintains rules to route traffic from Services to backend Pods.

Supports TCP, UDP, and SCTP.

Works closely with the Service abstraction.

CKA Scenario:
If a Service isn’t reachable, check:

kubectl get pods -o wide
kubectl get svc
sudo iptables -t nat -L -n | grep <service-name>

Container Runtime — The Executor

Responsible for running containers.
Supported runtimes include:
- containerd
- CRI-O
- Docker (deprecated)

The Kubelet uses CRI (Container Runtime Interface) to talk to the runtime.

🔄 How a Request Flows Through the Cluster

Let’s visualize a full lifecycle:

1️⃣ You run:

kubectl apply -f pod.yam

2️⃣ API Server authenticates → authorizes → validates → stores Pod spec in etcd.
3️⃣ Scheduler assigns Pod → updates spec.nodeName.
4️⃣ Kubelet on that Node sees new assignment → pulls image → runs container.
5️⃣ Kube-Proxy updates routing for network access.
6️⃣ Controller Manager ensures Pod stays running and replicas match.

CKA-Focused Scenarios and Commands

Final Words for CKA Candidates

To truly master Kubernetes:

Understand how components communicate.
Know where configurations live (/etc/kubernetes/manifests/).
Be comfortable with etcdctl, kubectl describe, and journalctl.
Always trace the flow of a Pod — from kubectl apply to container runtime.

If you grasp this architecture deeply, every CKA troubleshooting or architecture question becomes intuitive.

Day-09/100 - Understanding the Difference Between du and df in Linux

Sourav kumar — Sun, 02 Mar 2025 14:02:57 +0000

When working with Linux, managing disk space is crucial. Two commonly used commands for checking disk usage are du (disk usage) and df (disk free). While they seem similar, they serve different purposes. Let's explore their differences with examples and real-world use cases.

1. What is du?

The du (disk usage) command shows how much space files and directories are consuming on the disk. It helps find large files and directories.

sudo du -sh /*

Breaking Down the Command

du (Disk Usage) → Reports the amount of space used by files and directories.
-s (Summarize) → Displays only the total size of each directory, instead of listing sizes of all subdirectories and files.
-h (Human-readable) → Converts sizes into readable formats like KB, MB, or GB instead of raw bytes.
/* (All Top-Level Directories) → It calculates the size of each top-level directory under /.

Output

4.0K    /bin
16K     /boot
0       /dev
12M     /etc
120G    /home
4.0G    /lib
0       /proc
50G     /usr
20G     /var

Explanation:

Each line represents the disk usage of a top-level directory.
The size is in human-readable format (-h flag).
/home, /usr, and /var typically consume the most space.
/dev and /proc show 0 because they contain virtual files that don’t take up disk space.

More Example:

du -sh /home/user/Documents

Output:

2.5G    /home/user/Documents

This means the /home/user/Documents directory is using 2.5 GB of disk space.Basically we can find the disk usage of a specific folder also by giving its path.

Useful Options available with du :

-h (human-readable): Shows sizes in KB, MB, GB.
-s (summary): Shows only the total size of a directory.
-a (all files): Displays sizes of both files and directories.

Use Cases of du:

Find the largest directories on your system:

du -ah / | sort -rh | head -10

This command sorts directories by size and lists the top 10 largest ones.

Analyze space usage of a specific user:

du -sh /home/username

This helps system administrators track user disk usage.

Monitor growing log files:

du -sh /var/log

Helps find large log files that may need rotation or deletion.

2. What is df?

The df (disk free) command shows the available and used space on file systems, helping monitor overall disk usage.

Example-1 : when we run on any linux machine

df -h

output

Filesystem      Size  Used  Avail Use% Mounted on
/dev/sda1       100G   75G   25G  75% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G  1.1M  3.9G   1% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda2       200G  150G   50G  75% /home
/dev/sdb1        50G   10G   40G  20% /mnt/external_drive
tmpfs           796M   16K  796M   1% /run/user/1000

Example-2 : when we run on aws ec2 linux machine

df -h

output

Filesystem      Size  Used  Avail Use% Mounted on
/dev/root       8.0G  3.2G  4.8G  40% /
devtmpfs        481M     0  481M   0% /dev
tmpfs           495M     0  495M   0% /dev/shm
tmpfs           495M  436K  495M   1% /run
tmpfs           495M     0  495M   0% /sys/fs/cgroup
/dev/loop0       69M   69M     0 100% /snap/core18/2785
/dev/loop1       45M   45M     0 100% /snap/snapd/19457
tmpfs            99M     0   99M   0% /run/user/1000

Difference :
When you run df -h on an AWS EC2 instance, you often see an entry like /dev/root instead of /dev/sda1. This happens because AWS uses virtualized storage, and the root filesystem is typically mounted from an Elastic Block Store (EBS) volume.

Useful Options available with df:

-h (human-readable): Displays sizes in KB, MB, GB.
-T (filesystem type): Shows filesystem types like ext4, xfs, etc.
-i (inodes): Displays inode usage instead of disk usage.

Use Cases of df:

Check available space on all mounted filesystems:

df -hT

This helps in monitoring different storage partitions.

Find which partition is nearly full:

df -h | grep -E 'Filesystem|%'

Useful for proactively cleaning up space before a system runs out of storage.

Check inode usage (for small file-heavy servers):

df -i

If inodes are exhausted, no new files can be created even if space is available.

3. Key Differences Between du and df

4. Why Do du and df Show Different Values?

Sometimes, du and df may show different results because:

Deleted but open files: If a process is using a deleted file, df still counts its space.
Different mount points: df reports disk-wide usage, while du only shows a selected directory.
Files stored in different places: Disk compression, symlinks, and snapshots can cause variations.

Example of Difference:

echo "Hello, world!" > testfile.txt
rm testfile.txt

du -sh .
df -h .

Even though the file is deleted, df might still show the space as used until the process holding it is closed.

5. When to Use du and df?

Use du to find large files and analyze specific directory usage.
Use df to check overall disk space and monitor filesystem health.

Conclusion

Understanding du and df helps efficiently manage disk space. Use du for detailed file usage and df for a high-level disk overview. Next time you see "No space left on device," use these commands to diagnose the issue!

I hope this blog makes it easier to understand du and df. Let me know if you want more examples or explanations!

Day- 08 /100 - Deploying Multi-Environment Infrastructure with Terraform

Sourav kumar — Wed, 26 Feb 2025 14:05:35 +0000

Introduction

In real-world DevOps practices, managing multiple environments (development, staging, and production) efficiently is crucial. Terraform, an Infrastructure as Code (IaC) tool, allows us to automate cloud infrastructure provisioning. In this blog, we will build a multi-environment setup (Dev, Stage, and Prod) using Terraform modules, while also implementing best practices like state management with S3 and DynamoDB.

Project Overview

We will create three environments (Dev, Stage, and Prod) with different configurations for:

EC2 Instances (varying counts per environment)
S3 Buckets (varying counts per environment)
VPC with Public & Private Subnets
Security Groups
RDS Database
DynamoDB for state locking
S3 for Terraform state storage

📁 Folder Structure

terraform-project/
│── modules/
│   ├── ec2/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   ├── s3/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   ├── vpc/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   ├── security-groups/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   ├── rds/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   ├── dynamodb/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│── environments/
│   ├── dev/
│   │   ├── main.tf
│   │   ├── backend.tf
│   │   ├── variables.tf
│   │   ├── terraform.tfvars
│   ├── stage/
│   │   ├── main.tf
│   │   ├── backend.tf
│   │   ├── variables.tf
│   │   ├── terraform.tfvars
│   ├── prod/
│   │   ├── main.tf
│   │   ├── backend.tf
│   │   ├── variables.tf
│   │   ├── terraform.tfvars
│── global/
│   ├── s3-backend/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│── provider.tf
│── variables.tf
│── terraform.tfvars
│── outputs.tf
│── README.md

1️⃣ Configure Terraform Backend (State Management)

File: global/s3-backend/main.tf

resource "aws_s3_bucket" "terraform_state" {
  bucket = "mycompany-terraform-state"
  acl    = "private"
  versioning {
    enabled = true
  }
  lifecycle {
    prevent_destroy = true
  }
}

resource "aws_dynamodb_table" "terraform_lock" {
  name         = "terraform-lock"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

2️⃣ Backend Configuration for Each Environment

File: environments/dev/backend.tf

terraform {
  backend "s3" {
    bucket         = "mycompany-terraform-state"
    key            = "dev/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-lock"
    encrypt        = true
  }
}

3️⃣ VPC Module

File: modules/vpc/main.tf

resource "aws_vpc" "main" {
  cidr_block = var.vpc_cidr
}

resource "aws_subnet" "public" {
  count = length(var.public_subnets)

  vpc_id                  = aws_vpc.main.id
  cidr_block              = var.public_subnets[count.index]
  map_public_ip_on_launch = true
}

resource "aws_subnet" "private" {
  count = length(var.private_subnets)

  vpc_id     = aws_vpc.main.id
  cidr_block = var.private_subnets[count.index]
}

4️⃣ EC2 Module

File: modules/ec2/main.tf

resource "aws_instance" "app" {
  count = var.instance_count

  ami           = var.ami_id
  instance_type = var.instance_type
  subnet_id     = var.subnet_id

  tags = {
    Name = "EC2-${var.environment}"
  }
}

5️⃣ RDS Module

File: modules/rds/main.tf

resource "aws_db_instance" "database" {
  allocated_storage    = var.db_storage
  engine              = "mysql"
  instance_class      = var.db_instance_class
  db_name             = var.db_name
  username           = var.db_username
  password           = var.db_password
  publicly_accessible = false
  skip_final_snapshot = true
}

6️⃣ Dev Environment Implementation

Each environment (dev, stage, prod) will have its own terraform.tfvars file inside its respective folder.
File: environments/dev/main.tf

environments/dev/terraform.tfvars

# VPC
vpc_cidr        = "10.0.0.0/16"
public_subnets  = ["10.0.1.0/24"]
private_subnets = ["10.0.2.0/24"]

# EC2
instance_count  = 2
ami_id          = "ami-12345678"
instance_type   = "t2.micro"

# RDS
db_storage         = 20
db_instance_class  = "db.t3.micro"
db_name            = "devdb"
db_username        = "admin"
db_password        = "password123"

environments/stage/terraform.tfvars

# VPC
vpc_cidr        = "10.1.0.0/16"
public_subnets  = ["10.1.1.0/24"]
private_subnets = ["10.1.2.0/24"]

# EC2
instance_count  = 3
ami_id          = "ami-87654321"
instance_type   = "t3.medium"

# RDS
db_storage         = 50
db_instance_class  = "db.t3.medium"
db_name            = "stagedb"
db_username        = "admin"
db_password        = "securepassword456"

environments/prod/terraform.tfvars

# VPC
vpc_cidr        = "10.2.0.0/16"
public_subnets  = ["10.2.1.0/24"]
private_subnets = ["10.2.2.0/24"]

# EC2
instance_count  = 5
ami_id          = "ami-11223344"
instance_type   = "t3.large"

# RDS
db_storage         = 100
db_instance_class  = "db.t3.large"
db_name            = "proddb"
db_username        = "admin"
db_password        = "supersecurepassword789"

Now, environments/dev/main.tf (or stage/main.tf and prod/main.tf) will only reference variables, making it more reusable.

module "vpc" {
  source          = "../../modules/vpc"
  vpc_cidr        = var.vpc_cidr
  public_subnets  = var.public_subnets
  private_subnets = var.private_subnets
}

module "ec2" {
  source         = "../../modules/ec2"
  instance_count = var.instance_count
  ami_id         = var.ami_id
  instance_type  = var.instance_type
  subnet_id      = module.vpc.public_subnets[0]
  environment    = "dev"
}

module "rds" {
  source           = "../../modules/rds"
  db_storage       = var.db_storage
  db_instance_class = var.db_instance_class
  db_name         = var.db_name
  db_username     = var.db_username
  db_password     = var.db_password
}

8️⃣ Apply Terraform
Step 1: Initialize Backend

terraform init

Step 2: Plan

terraform plan -var-file="terraform.tfvars"

Step 3: Apply

terraform apply -var-file="terraform.tfvars" -auto-approve

Steps to Run the Terraform Project in All 3 Environments (Dev, Stage, Prod)

Here we will understand how to set up, initialize, and deploy the Terraform project for each environment: Dev, Stage, and Prod.

1️⃣ Prerequisites
Before running Terraform, ensure you have the following installed:

Terraform CLI (≥ v1.0) → Install from Terraform Download
AWS CLI (≥ v2.0) → Install from AWS CLI
AWS Credentials Configured (~/.aws/credentials)

aws configure

S3 Bucket for Remote Backend (Created in global/s3-backend/)

2️⃣ Initialize Backend (S3 & DynamoDB)
Since we are using remote backend for Terraform state, we first create the S3 bucket and DynamoDB table.

🌍 Navigate to global/s3-backend/ and Apply

cd global/s3-backend/
terraform init
terraform apply -auto-approve

This creates an S3 bucket for storing Terraform state and a DynamoDB table for state locking.

3️⃣ Deploy a Specific Environment (Dev, Stage, Prod)
Each environment (dev, stage, prod) has its own directory. You must navigate to the specific environment before running Terraform commands.

▶️ Deploy Dev Environment

cd environments/dev/
terraform init
terraform plan -var-file="terraform.tfvars"
terraform apply -var-file="terraform.tfvars" -auto-approve

✅ This provisions the Dev environment.

▶️ Deploy Stage Environment

cd environments/stage/
terraform init
terraform plan -var-file="terraform.tfvars"
terraform apply -var-file="terraform.tfvars" -auto-approve

✅ This provisions the Stage environment.

▶️ Deploy Prod Environment

cd environments/prod/
terraform init
terraform plan -var-file="terraform.tfvars"
terraform apply -var-file="terraform.tfvars" -auto-approve

✅ This provisions the Prod environment.

4️⃣ Verify Deployments
After running Terraform, verify that resources have been created or not either through console or through aws cli commands.

5️⃣ Destroy Resources (When Needed)
To destroy an environment, navigate to its directory and run:

terraform destroy -var-file="terraform.tfvars" -auto-approve

Example:
cd environments/dev/
terraform destroy -var-file="terraform.tfvars" -auto-approve
🔴 ⚠️ WARNING: This permanently deletes all resources in the environment!

Summary

Day- 07/100 - User Management in Linux

Sourav kumar — Mon, 24 Feb 2025 18:02:28 +0000

User management is a crucial aspect of Linux administration, allowing system owners to control access, permissions, and security settings for different users. In this guide, we’ll cover user management fundamentals, starting with understanding ‘sudo’, followed by essential system commands, and finally diving into user management commands.

Understanding ‘sudo’

sudo — short for Superuser Do is a command in Linux that allows a permitted user to execute a command as the superuser (root) or another specified user. It is commonly used to run administrative tasks without switching to the root user entirely.

How sudo Works
When you use sudo, the system temporarily grants elevated privileges for that specific command. The user must be in the sudoers file (/etc/sudoers) to execute commands with sudo. By default, sudo asks for the user’s password before executing the command.

Example command:-

sudo apt update

Key Features of sudo
Security & Control – Users don’t need to log in as root, reducing security risks.
Logging & Auditing – Commands run with sudo are logged in /var/log/auth.log.
Time-Limited Authentication – Once authenticated, sudo allows repeated use for a short period (default: 5 minutes).

Add a user to the sudoers list:

sudo usermod -aG sudo username

usermod → Modifies the user account.
-aG → A combination of two options:
-a (Append): Adds the user to a group without removing existing group memberships.
-G (Groups): Specifies the groups to which the user should be added, here it is added to the sudo group.

We will learn about ‘groups’ later in this blog.

Running shutdown using normal user won’t work, it works as root user or using sudo:

shutdown

If the user is a sudoer, use the following command:

sudo shutdown

To restart the system:

sudo reboot

Essential System Commands for User Information

Before managing users, it’s helpful to gather system information using these commands:

who — shows a list of logged-in users, their terminals, and login times.
whoami — Displays the current logged-in user's username.

There is only one user right now, so who is showing only 1 user otherwise it gives a list of logged-in users.

id — it tells the info about user id, group id, for current user.

check for another user — id username
show only UID — id -u
show only GID — id -g
show only groups — id -G

User Management Commands

useradd

add new user – sudo useradd -m user1

-m → Creates a home directory (/home/username) for the user.

What Happens without -m?

A new user is created.
No home directory is created (unlike with -m).
The user will not have a default working directory under /home/username.
The user may not have a personal environment setup (e.g., .bashrc, .profile)

To see all the users you can check the ‘/etc/passwd’ file:

cat /etc/passwd

The newly created users - user1, user2 are visible at the end of the file.

passwd – set password for user:

sudo passwd user1

su – switch user

It will ask password and switch user:

su user1

Notice the username changed from ‘ubuntu’ to ‘user1’.
Use exit to go to primary user.

There are two ways to switch user:
su username vs su - username

su john

What Happens?

Switches to john, but keeps the current shell environment (variables, paths, etc.).
Does not load john's profile settings (~/.bashrc, ~/.profile).
Current directory remains unchanged.

su - john

What Happens?

Completely switches to john's environment, just like a fresh login.
Loads john's shell profile (~/.bashrc, ~/.profile).
Current directory changes to john's home (/home/john).
Sets PATH, HOME, and other variables specific to john.

userdel — delete user

Delete ‘user1’:

sudo userdel user1

Delete ‘user1’ and its home directory:

sudo userdel -r user1

Force delete ‘user1’ even if the user is logged in:

sudo userdel -f user1

If you have deleted the user using the first command, and home directory is not removed, use the following to delete it manually:

sudo rm -rf /home/user1

rm → The remove (delete) command in Linux.
-r → Stands for recursive, meaning it deletes directories and all their contents.
-f → Stands for force, meaning it bypasses confirmation prompts and deletes files without asking.

groupadd – command is used to create a new group in Linux

Create a group named devops:

sudo groupadd devops

Create a group with a specific GID (Group ID):

sudo groupadd -g 5001 testers

Run this command to see all the groups:

cat etc/group

Displayed output is cropped.

There is a group of each user also, when we create a user, a group with same name gets created automatically

usermod → Recommended for Adding user to Multiple Groups

Adding 'user1' to 'devops' group:

sudo usermod -aG devops user1

Adding to 'john' multiple groups at the same time:

sudo usermod -aG developers,testers,QA john

Add a user to sudo group:

sudo usermod -aG sudo username

Change the default/primary group using -g:

sudo usermod -g QA john

Assign primary and secondary groups in one line:

sudo usermod -g developers -aG testers,QA john

Options in usermod command:

-d /new/home/directory → Changes the user's home directory to a new location.
-m → Moves all existing files from the old home directory to the new one.
Example: Change john's home directory to /home/devuser and move files
sudo usermod -d /home/devuser -m john

-p is not recommended to set password for a user, use passwd instead
sudo passwd user1 → it will give option to set password

gpasswd → Recommended for Single Group Changes

Add user1 to devops group:

sudo gpasswd -a user1 devops

-a → appends the user1 to devops group without removing other memberships

Add multiple users to testers group:

sudo gpasswd -M user1,user2 testers

see the users in devops and testers

To check the groups in which user1 is present:

groups user1

groupdel – delete a group:

sudo groupdel testers – delete testers group

Testers group no more showing up

It just deletes the group not the users inside that group, you can see the user1, user2 are still there.

Conclusion
Proper user and group management in Linux is vital for maintaining security and control over system access. By understanding and utilizing these commands, administrators can efficiently manage users and permissions, ensuring smooth and secure operations. In the next blog, we will cover user permissions and file access management to further enhance security and control.

Day-06/100 - Automating Backups with Rotation using ZIP method | Comparision with rsync method

Sourav kumar — Thu, 20 Feb 2025 17:55:17 +0000

Backups are essential for protecting data from accidental deletion, corruption, or hardware failures. However, managing backups efficiently requires automating both the backup creation process and the rotation (deletion of old backups). In this blog, we'll explore a simple Bash script that performs these tasks seamlessly.

Overview of the Backup Script

This Bash script automates the process of creating compressed backups of a specified directory and retains only the latest five backups. Older backups are automatically deleted to save storage space.

Features:

Creates a zip backup of a specified directory.
Uses a timestamp to uniquely name each backup.
Retains only the latest five backups and removes older ones.
Runs efficiently with minimal user intervention.

Script Breakdown

Here's the script:

#! /bin/bash

<< readme
This is a script for backup with 5-day rotation

Usage:
./backup_and_rotation.sh <path to your source> <path to backup folder>
readme

function display_usage {
    echo "Usage: ./backup_and_rotation.sh <path to your source> <path to backup folder>"
}

if [ $# -eq 0 ]; then
    display_usage
    exit 1
fi

source_dir=$1
timestamp=$(date '+%Y-%m-%d-%H-%M-%S')
backup_dir=$2

function create_backup() {
    zip -r "${backup_dir}/backup_${timestamp}.zip" "${source_dir}" > /dev/null

    if [ $? -eq 0 ]; then
        echo "Backup generated successfully for ${timestamp}"
    else
        echo "Backup failed!" >&2
    fi
}

function perform_rotation() {
    # Using glob instead of `ls` to avoid errors if no backups exist
    backups=("${backup_dir}"/backup_*.zip)
    backups=($(ls -t "${backups[@]}" 2>/dev/null))  # Sort backups by timestamp

    echo "Existing backups: ${backups[@]}"

    if [ "${#backups[@]}" -gt 5 ]; then
        echo "Performing rotation for 5 days"

        backups_to_remove=("${backups[@]:5}")  # Keep latest 5 backups
        echo "Removing backups: ${backups_to_remove[@]}"

        for backup in "${backups_to_remove[@]}"; do
            rm -f "${backup}"
        done
    fi
}

create_backup
perform_rotation

Understanding the Script

1. Usage Instructions

The script includes a usage guide in a commented block to inform users how to execute it.

Usage: ./backup_and_rotation.sh <path to your source> <path to backup folder>

This ensures that users provide the required arguments: the source directory and the backup folder.

2. Checking for User Input

Before proceeding, the script verifies that the required arguments are provided. If not, it displays a usage message and exits.

if [ $# -eq 0 ]; then
    display_usage
    exit 1
fi

3. Backup Creation

The create_backup function compresses the specified directory into a .zip file, using a timestamp in the filename for uniqueness.

zip -r "${backup_dir}/backup_${timestamp}.zip" "${source_dir}" > /dev/null

This ensures backups do not overwrite each other and makes it easier to track backup dates.

4. Backup Rotation

The script retains only the latest five backups. It sorts the existing backups by timestamp and removes older ones.

if [ "${#backups[@]}" -gt 5 ]; then
    backups_to_remove=("${backups[@]:5}")
    for backup in "${backups_to_remove[@]}"; do
        rm -f "${backup}"
    done
fi

This ensures that old backups do not consume unnecessary storage space.

Running the Script

To execute the script, use:

chmod +x backup_and_rotation.sh
./backup_and_rotation.sh /path/to/source /path/to/backup

For example:


./backup_and_rotation.sh /home/user/Documents /home/user/Backups

Comparision

When to Use Zip-based Backup
Best for: Situations where you need complete, standalone backups that can be easily transferred, archived, or restored as a whole.

✅ Use Cases:

Periodic Full Backups – If you need a full copy of the data every time (e.g., weekly full backups).

Cloud Storage or Remote Archival– When backing up data to cloud storage (Google Drive, S3, etc.) where a single compressed file is easier to manage.

Disaster Recovery– If you need to store long-term snapshots of your system that can be restored independently.

File Versioning and Portability – When you need to download or send backups via email, FTP, or portable storage.

Backup Before System Updates – Creating a complete archive before running major software updates or upgrades.

⚠ Downside: Requires more storage space and can be slower due to compression overhead.

When to Use Rsync-based Backup
Best for: Situations where efficiency, speed, and incremental backups are needed.

✅ Use Cases:

Daily or Frequent Backups – Since rsync only copies changed files, it's perfect for daily or hourly backups.

Server or Web App Backups – For Linux servers, web applications, and databases, where only the latest changes matter.

Large File Systems – If you're backing up large directories, rsync saves time and disk space.

Network or Remote Backups – When syncing files between machines (e.g., from a local machine to a remote server).

Local System Synchronization – If you need to keep two directories synchronized without unnecessary duplication.

⚠ Downside: Does not create a single compressed file; you need to manually zip it later if needed.

Day 05/100 - Bash special commands (part-1)

Sourav kumar — Thu, 20 Feb 2025 05:41:36 +0000

Let's create a Bash script that demonstrates these special features step by step. The script will explain and showcase:

✅ /dev/null (The "black hole" of Linux)
✅ File descriptors: 0 (stdin), 1 (stdout), 2 (stderr)
✅ Redirections (>, >>, 2>, 2>&1, &> and |)
✅ commmand -v

Script: bash_special_features.sh

#!/bin/bash

<< Information ===================================================

# Bash Special Features Demo
# Author: Sourav Kumar
# Date: $(date +"%Y-%m-%d")
# Description: Demonstrates advanced Bash concepts

Information=======================================================

# Set strict mode for better error handling
set -euo pipefail  # Prevents script from continuing on errors

LOG_FILE="script.log"  # Log file for debugging

# -------------------------------
# Logging Function
# -------------------------------
log() {
    local timestamp
    timestamp=$(date +"%Y-%m-%d %H:%M:%S")
    echo "[$timestamp] $*" | tee -a "$LOG_FILE"
}

# -------------------------------
# Function to Check Command Existence
# -------------------------------
check_command() {
    if command -v "$1" > /dev/null 2>&1; then
        log "✅ $1 is installed"
    else
        log "❌ $1 is NOT installed. Please install it."
        return 1  # Exit with error status
    fi
}

# -------------------------------
# Start of Script Execution
# -------------------------------
log "🚀 Starting Bash Special Features Demo..."
log "🔹 File Descriptors, Redirections, and /dev/null"
echo "---------------------------------------------------------------"

# 1️⃣ Understanding File Descriptors
log "   - 0 = Standard Input (stdin)"
log "   - 1 = Standard Output (stdout)"
log "   - 2 = Standard Error (stderr)"

# 2️⃣ Demonstrating stdout (1) and stderr (2)
log "Running: ls /home > stdout.txt 2> stderr.txt"
ls /home > stdout.txt 2> stderr.txt
log "   - Standard Output saved in stdout.txt"
log "   - Standard Error saved in stderr.txt"

# 3️⃣ Redirecting both stdout and stderr to the same file
log "Running: ls /nonexistent_path > output.txt 2>&1"
ls /nonexistent_path > output.txt 2>&1 || log "⚠️ Error logged in output.txt"

# 4️⃣ Redirecting both stdout and stderr using &>
log "Running: ls /root &> combined_output.txt"
ls /root &> combined_output.txt || log "⚠️ Error logged in combined_output.txt"

# 5️⃣ Suppressing output using /dev/null
log "Running: ls / > /dev/null"
ls / > /dev/null
log "   - Standard Output is discarded"

log "Running: ls /nonexistent_path 2> /dev/null"
ls /nonexistent_path 2> /dev/null || log "⚠️ Error suppressed"

# 6️⃣ Using /dev/null to test command existence
check_command rsync

# 7️⃣ Using Pipe (|) to send stdout to another command
log "Running: ls / | grep etc"
ls / | grep etc || log "⚠️ 'etc' not found in /"

# -------------------------------
log "✅ Script Execution Completed ✅"
log "Check the output files: stdout.txt, stderr.txt, output.txt, combined_output.txt, and $LOG_FILE"

What does /dev/null do?
/dev/null is a special file in Linux that acts as a "black hole."
Anything sent to /dev/null is discarded (i.e., it disappears and does not show up in the terminal).

Example 1: Redirecting Output to /dev/null

ls > /dev/null

The ls command normally lists files in the current directory.
"> /dev/null" sends the output (file list) into the "black hole," so nothing appears on the screen.

Example 2: Suppressing Command Output

echo "Hello World" > /dev/null

Normally, echo "Hello World" prints "Hello World" to the terminal.
With "> /dev/null", the output is discarded, so you see nothing.

Understanding File Descriptors and Redirection in Bash
In Bash, file descriptors and redirection help control where input and output go.

File Descriptors (FD)

FD Name Description
0 stdin Standard Input (Keyboard/Input File)
1 stdout Standard Output (Terminal/Log File)
2 stderr Standard Error (Error Messages)

Redirection Operators (>, >>)

Operator Description
">" Redirects output to a file (overwrites existing content)
">>" Appends output to a file (does not overwrite)

ls /home > stdout.txt 2> stderr.txt

Redirect stdout and stderr separately
">" stdout.txt saves normal output.
"2>" stderr.txt saves errors separately.

Summary Table

Understanding command -v in Bash
The command -v command is used to check if a command exists and determine its path. It is useful in scripting to ensure that required tools are available before executing commands.

1. Basic Usage of command -v

command -v ls

output

/bin/ls

Explanation:

This checks whether ls exists in the system and returns its full path.
If the command is not found, it returns nothing.

2. Checking If a Command Exists in a Script

if command -v git > /dev/null 2>&1; then
    echo "Git is installed!"
else
    echo "Git is not installed. Please install it."
fi

Explanation:

command -v git checks if git is installed.
> /dev/null 2>&1 suppresses output.
If found, it prints "Git is installed!", otherwise, it prints an error message.

3. Using command -v to Get a Command Path

echo "Python is located at: $(command -v python3)"

Output (if installed):

Python is located at: /usr/bin/python3

Explanation:

This prints the full path of python3.

4. Assigning the Path to a Variable

PYTHON_PATH=$(command -v python3)
echo "Python is installed at: $PYTHON_PATH"

Explanation:

Stores the path of python3 in PYTHON_PATH.
Prints the location of Python.

5. Using command -v with Multiple Commands

for cmd in docker kubectl terraform; do
    if command -v "$cmd" > /dev/null 2>&1; then
        echo "$cmd is installed"
    else
        echo "$cmd is not installed"
    fi
done

Explanation:

This checks if docker, kubectl, and terraform are installed.

Difference Between command -v, which, and type

example

command -v echo  # Output: /bin/echo
which echo       # Output: /bin/echo
type echo        # Output: echo is a shell builtin

Day-04/100 - Automating Backups with Rsync: A Function-Based Bash Script with Rotation

Sourav kumar — Thu, 20 Feb 2025 05:11:44 +0000

Backups are crucial for data security and disaster recovery. Whether you're safeguarding personal files or managing production environments, automating backups reduces risk and ensures data integrity.

In this blog, we’ll explore an industry-standard Bash script for automated, incremental backups using rsync, following best practices such as:
✔ Function-based scripting for readability and reusability
✔ Efficient incremental backups using rsync
✔ Automated backup rotation (retaining only the last 3 backups)
✔ Error handling and logging for reliability

Why Use Rsync for Backups?
Unlike full backups (e.g., ZIP-based archives), rsync efficiently synchronizes only changed files, making it:

Faster – Transfers only modified data
Storage-efficient – Keeps incremental backups, saving disk space
Versatile – Supports remote backups over SSH

The Rsync Backup Script
Below is a function-based industry-standard Bash script that automates incremental backups with a 3-day retention policy.

🔹 Script: backup_and_rotation_rsync.sh

#! /bin/bash

<< readme
This is an industry-standard script for incremental backups using rsync with a 3-day rotation.

Usage:
./backup_and_rotation_rsync.sh <path to your source> <path to backup folder>
readme

# Function to display usage information
function display_usage {
    echo "Usage: ./backup_and_rotation_rsync.sh <path to your source> <path to backup folder>"
}

# Check if the correct number of arguments is provided
if [ $# -ne 2 ]; then
    display_usage
    exit 1
fi

# Assign arguments to variables
source_dir=$1
backup_dir=$2
timestamp=$(date '+%Y-%m-%d-%H-%M-%S')
backup_target="${backup_dir}/backup_${timestamp}"
log_file="${backup_dir}/backup_${timestamp}.log"

# Function to check if rsync is installed
function check_rsync_installed {
    if ! command -v rsync > /dev/null 2>&1; then
        echo "Error: rsync is not installed. Please install it and try again." >&2
        exit 2
    fi
}

# Function to create a backup using rsync
function create_backup {
    mkdir -p "$backup_target"  # Ensure backup directory exists

    rsync -av --delete "$source_dir/" "$backup_target/" > "$log_file" 2>&1

    if [ $? -eq 0 ]; then
        echo "Backup completed successfully at $backup_target"
    else
        echo "Backup failed! Check log: $log_file" >&2
        exit 3
    fi
}

# Function to perform backup rotation (retain only the last 3 backups)
function perform_rotation {
    backups=("${backup_dir}"/backup_*)  # Find existing backups
    backups=($(ls -td "${backups[@]}" 2>/dev/null))  # Sort backups by timestamp (newest first)

    echo "Existing backups: ${backups[@]}"

    if [ "${#backups[@]}" -gt 3 ]; then
        echo "Performing rotation: Keeping latest 3 backups"

        backups_to_remove=("${backups[@]:3}")  # Keep latest 3 backups
        echo "Removing backups: ${backups_to_remove[@]}"

        for backup in "${backups_to_remove[@]}"; do
            rm -rf "${backup}"
        done
    fi
}

# Run the functions in order
check_rsync_installed
create_backup
perform_rotation

Step-by-Step Explanation
1️⃣ Argument Validation

if [ $# -ne 2 ]; then
    display_usage
    exit 1
fi

The script requires two arguments:

Source directory – Folder to back up
Backup directory – Where backups will be stored
If not provided, it displays the correct usage and exits.

2️⃣ Checking Rsync Installation

function check_rsync_installed {
    if ! command -v rsync > /dev/null 2>&1; then
        echo "Error: rsync is not installed. Please install it and try again." >&2
        exit 2
    fi
}

Since rsync is essential for this script, we check if it's installed. If not, an error message prompts the user to install it.

3️⃣ Creating the Backup

function create_backup {
    mkdir -p "$backup_target"  # Ensure backup directory exists

    rsync -av --delete "$source_dir/" "$backup_target/" > "$log_file" 2>&1
}

rsync -av preserves permissions, timestamps, and symbolic links.
--delete removes files in the backup that no longer exist in the source.
Logs are saved for later review.

4️⃣ Backup Rotation (Keeping Only Last 3 Backups)

function perform_rotation {
    backups=("${backup_dir}"/backup_*)  
    backups=($(ls -td "${backups[@]}" 2>/dev/null))  # Sort backups by timestamp (newest first)

    if [ "${#backups[@]}" -gt 3 ]; then
        backups_to_remove=("${backups[@]:3}")  # Keep latest 3 backups
        for backup in "${backups_to_remove[@]}"; do
            rm -rf "${backup}"
        done
    fi
}

This ensures only the latest 3 backups are retained by:
✅ Sorting backups by timestamp
✅ Keeping the latest 3
✅ Deleting older backups automatically

⏳ How to Use This Script

1️⃣ Make the script executable

chmod +x backup_and_rotation_rsync.sh

2️⃣ Run the script with the source and backup folder paths

./backup_and_rotation_rsync.sh /home/user/documents /backups

3️⃣ Check backup logs

cat /backups/backup_YYYY-MM-DD-HH-MM-SS.log

Day -03 /100 - Configuring mail server using Postfix, Dovecot and Mutt

Sourav kumar — Tue, 18 Feb 2025 18:56:00 +0000

We will set up three things in ubuntu machine:-

Postfix (It will provide SMTP Implementation, which will work as an email transfer agent - MTA)
Dovecot (It will provide POP3/IMAP service, which will act as an email delivery agent.)
Mutt (It will work as an email client and will provide a nice interface to write/send/receive emails.)

We will set up ubuntu server as an email server, hence we need to assign an adequate hostname and a fully qualified domain name.

To change the hostname, run the following command:

sudo hostnamectl set-hostname "mailserver-name"

Open another terminal and you will see the hostname has changed.

Next, we will change the fully qualified domain name:

sudo nano /etc/hosts

Go to the terminal and find your IP address by running - ifconfig (run command shown in image if "ifconfig" not found)

Copy the IP address and paste it in the "/etc/hosts" file

The hostname and fully qualified domain name are changed successfully.

Step 1 - Postfix Installation

sudo apt install postfix -y

Change the system mail name from "mail.sourav.com" to "sourav.com" and press Enter.

Verify the installation of Postfix using the following command:

sudo systemctl status postfix

Let's change the location in the configuration file where the emails will be saved.

We will create a folder "Maildir" and will give its path

sudo postconf "home_mailbox = Maildir/"

Step 2 - "Dovecot" Installation

Run:

sudo apt install dovecot-imapd dovecot-pop3d dovecot-core

Verify the installation using

sudo systemctl status dovecot

To reload any service use this:

sudo systemctl reload dovecot
sudo systemctl reload postfix

Let's change the location in the configuration file for dovecot also.

sudo nano /etc/dovecot/conf.d/10-mail.conf

Comment the old mail location and uncomment the new one as shown in the image:

Check the changed location using:

doveconf -n

Whenever a new user signs up on the mail server, we want the "Maildir" to be created automatically in the user's home directory. This "Maildir" will work as the Inbox for the user.

To achieve this, we will go to /etc/skel/

Note- Whatever we create in this directory whether a file or folder, gets copied to home when a new user signs up.

cd /etc/skel
sudo mkdir -p Maildir/cur Maildir/new Maildir/tmp

All 3 folders are created.

Step 3 - Installing mutt

Run

sudo apt install mutt

Now we want that if anybody signs up then he/she should have access to mutt so that they can get a nice interface to send/receive/write emails.

So we will make all configurations of mutt in the same location - /etc/skel

By doing this every user will get access to mutt without installing it.

sudo mkdir /etc/skel/.mutt

We have created a hidden directory mutt.

Now go inside .mutt and create a file "muttrc"

cd .mutt/
sudo nano muttrc

and paste this content

set imap_user = ""
set imap_pass = ""
set folder = imaps://mail
set spoolfile = +INBOX
set realname = ''
set from = "$imap_user"
set use_from = yes
set sort=reverse-date
mailboxes = INBOX
set timeout=1
set sidebar_visible = yes
source ~/.mutt/mutt_colors

Now create another file mutt_colors using command

sudo nano mutt_colors

and copy paste the content


# Colours for items in the index
color index brightcyan black ~N
# Hmm, don't like this.
# color index brightgreen black "~N (~x byers.world)|(~x byers.x)|(~x langly.levallois123.axialys.net)|(~x the.earth.li)"
color index brightyellow black ~F
color index black green ~T
color index brightred black ~D
mono index bold ~N
mono index bold ~F
mono index bold ~T
mono index bold ~D

# Highlights inside the body of a message.

# URLs
color body brightgreen black "(http|ftp|news|telnet|finger)://[^ \"\t\r\n]*"
color body brightgreen black "mailto:[-a-z_0-9.]+@[-a-z_0-9.]+"
mono body bold "(http|ftp|news|telnet|finger)://[^ \"\t\r\n]*"
mono body bold "mailto:[-a-z_0-9.]+@[-a-z_0-9.]+"

# email addresses
color body brightgreen black "[-a-z_0-9.%$]+@[-a-z_0-9.]+\\.[-a-z][-a-z]+"
mono body bold "[-a-z_0-9.%$]+@[-a-z_0-9.]+\\.[-a-z][-a-z]+"

# header
color header green black "^from:"
color header green black "^to:"
color header green black "^cc:"
color header green black "^date:"
color header yellow black "^newsgroups:"
color header yellow black "^reply-to:"
# color header brightcyan black "^subject:"
color header yellow black "^subject:"
color header red black "^x-spam-rule:"
color header green black "^x-mailer:"
color header yellow black "^message-id:"
color header yellow black "^Organization:"
color header yellow black "^Organisation:"
color header yellow black "^User-Agent:"
color header yellow black "^message-id: .*pine"
color header yellow black "^X-Fnord:"
color header yellow black "^X-WebTV-Stationery:"
color header yellow black "^X-Message-Flag:"
color header yellow black "^X-Spam-Status:"
color header yellow black "^X-SpamProbe:"
color header red black "^X-SpamProbe: SPAM"

# Coloring quoted text - coloring the first 7 levels:
color quoted cyan black
color quoted1 yellow black
color quoted2 red black
color quoted3 green black
color quoted4 cyan black
color quoted5 yellow black
color quoted6 red black
color quoted7 green black


# Default color definitions
#color hdrdefault white green
color signature brightmagenta black
color indicator black cyan
color attachment black green
color error red black
color message white black
color search brightwhite magenta
# color status brightyellow blue
color status blue black
color tree brightblue black
color normal white black
color tilde green black
color bold brightyellow black
color underline magenta black
color markers brightcyan black

# Colour definitions when on a mono screen
mono bold bold
mono underline underline
mono indicator reverse

Now set the permission for the directory so that the content in this directory can be copied automatically.

sudo chmod 700 -R /etc/skel/

Step 4 - Creating users

Create two users with names user1 and user2 and set passwords for them.

sudo adduser --gecos "" user1
sudo adduser --gecos "" user2

Now we will switch to user1 and will view the files in the home directory.

Now we will go to this file and we will update the username, password, and real name (signing up) so that the user doesn't have to give the password every time he is sending or receiving mail.

nano .mutt/muttrc

Now open two terminals - one for user1 and one for user2.

Run

mutt

it will open the certificate window. Then press "a" to accept the certificate. Do this for user2 also.

Inboxes for both users are open now.

To send email from user1 to user2:

Go to user1 terminal where inbox is open
Press "m" to create a new email.
Write the email id of user2 "user2@sourav.com" in the To section and press Enter
Give any subject, press Enter.
Write the email in the editor, press Ctrl + X, then "y" then Enter.
Press "y" to send the email.

Go to user2 terminal where inbox is open.
Press Enter on the new email received, this will open the email sent by user1.

And, we have setup the mail server successfully!!

Day-02 / 100 - Archive Older Files

Sourav kumar — Mon, 17 Feb 2025 18:28:46 +0000

Project Requirement

In the given directory, if you find files more than a given size ex: 20MB or files older than given days ex: 10 days

Compress those files and move in a ‘archive’ folder.

1. Why are we making this script?

We are creating this Bash script to automate the process of managing large and old files in a given directory.
Over time, directories accumulate large and outdated files, consuming unnecessary disk space. Manually identifying and archiving these files is time-consuming and inefficient.
This script helps automate the process by finding files based on size and age, compressing them, and moving them to an archive folder for better storage management.

2. Purpose of the script

The purpose of this script is to improve disk space management and system performance by:

Identifying large files exceeding a specific size (e.g., 20MB)
Finding old files that have not been modified for a specified number of days (e.g., 10 days)
Compressing files to reduce storage consumption
Moving archived files to a separate directory for better organization
Automating cleanup to avoid manual intervention

This script is useful in DevOps workflows for log file management, backup automation, and system maintenance. 🚀

Steps of script:

Defining Variables
Checking if the Base Directory Exists
Create ‘archive’ folder if not already present
Find all the files with size more than 20MB or being stored for more than 10 days
Compress each file and Move the compressed files in ‘archive’ folder

1. Defining Variables
At the beginning of the script, we define a few key variables:

BASE=/home/sourav/tutorials/find_command  # The base directory to search files in
DAYS=10  # (Unused in this script but could be used for age-based filtering)
DEPTH=1  # The depth level for the find command
RUN=0    # A control flag (currently set to always archive)

BASE : Defines the directory where we want to search for large files.
DAYS: Unused in this script, but it might be intended for filtering files based on modification time.
DEPTH: Restricts how deep the find command should search in subdirectories.
RUN: This flag is currently set to 0, meaning the script will always run the archiving logic.

2. Checking if the Base Directory Exists
Before performing any operations, the script verifies if the target directory exists:

if [ ! -d $BASE ]
then
    echo "directory does not exist: $BASE"
    exit 1
fi

if [ ! -d $BASE ]: Checks if $BASE is not a directory.
If the directory does not exist, the script prints an error message and exits.

3. Create ‘archive’ folder if not already present
If an archive folder does not exist inside $BASE, the script creates one:

if [ ! -d $BASE/archive ]
then
    mkdir $BASE/archive
fi

mkdir $BASE/archive: Creates the archive folder to store compressed files.
This ensures that files have a designated place for storage.

4. Find all the files with size more than 20MB or being store for more than 10 days

The core functionality of this script is to locate files larger than 20MB or Files older than 10 days and compress them:

for i in $(find "$BASE" -maxdepth "$DEPTH" -type f \( -size +20M -o -mtime +"$DAYS" \))

find $BASE → Starts searching from the base directory.
-maxdepth $DEPTH → Limits search depth to avoid scanning deep subdirectories.
-type f → Ensures only files (not directories) are selected.
(...) → Groups conditions to apply the OR (-o) logic properly.
-size +20M → Finds files larger than 20MB.
-o (OR operator) → Ensures files matching either condition are selected.
-mtime +"$DAYS" → Finds files older than $DAYS days.

5. Compressing and Moving the Files

Inside the loop, the script checks the RUN variable before proceeding:

if [ $RUN -eq 0 ]

Since RUN is set to 0, it executes the archiving process:

echo "[ $(date "+%Y-%m-%d %H:%M:%S") ] archiving $i ==> $BASE/archive"
gzip $i || exit 1
mv $i.gz $BASE/archive || exit 1

date "+%Y-%m-%d %H:%M:%S": Prints the timestamp for logging purposes.
gzip $i || exit 1: Compresses the file using gzip. If compression fails, the script exits.
mv $i.gz $BASE/archive || exit 1: Moves the compressed file (.gz format) to the archive directory.

whole script

BASE=/home/sourav/tutorials/find_command
DAYS=10
DEPTH=1
RUN=0

# Check if the directory is present or not
if [ ! -d "$BASE" ]; then
    echo "directory does not exist: $BASE"
    exit 1
fi

# Create 'archive' folder if not present
if [ ! -d "$BASE/archive" ]; then
    mkdir "$BASE/archive"
fi

# Find files that are either larger than 20MB OR older than 10 days
for i in $(find "$BASE" -maxdepth "$DEPTH" -type f \( -size +20M -o -mtime +"$DAYS" \)); 
do
    if [ "$RUN" -eq 0 ]; then
        echo "[ $(date "+%Y-%m-%d %H:%M:%S") ] archiving $i ==> $BASE/archive"
        gzip "$i" || exit 1
        mv "$i.gz" "$BASE/archive" || exit 1
    fi
done

The logic handles these condtions

A 5MB file that is 12 days old → ✅ Archived (because it's old).
A 25MB file that is 3 days old → ✅ Archived (because it's large).
A 5MB file that is 3 days old → ❌ Not Archived (doesn't match either condition).

Save this as archive_script.sh, then give it execute permissions and run it:

chmod +x archive_script.sh
./archive_script.sh

📌 Before Running the Script

$ ls -lh /home/sourav3366/tutorials/find_command
-rw-r--r-- 1 sourav3366 users  25M Feb 12  test1.log
-rw-r--r-- 1 sourav3366 users  5M  Feb 1   test2.log
-rw-r--r-- 1 sourav3366 users  30M Feb 15  test3.log
-rw-r--r-- 1 sourav3366 users  10M Feb 10  test4.log

Output of the Script Execution

[ 2025-02-17 14:30:45 ] Archiving /home/sourav3366/tutorials/find_command/test1.log ==> /home/sourav3366/tutorials/find_command/archive
[ 2025-02-17 14:30:46 ] Archiving /home/sourav3366/tutorials/find_command/test2.log ==> /home/sourav3366/tutorials/find_command/archive
[ 2025-02-17 14:30:47 ] Archiving /home/sourav3366/tutorials/find_command/test3.log ==> /home/sourav3366/tutorials/find_command/archive

📂 After Running the Script

$ ls -lh /home/sourav3366/tutorials/find_command
-rw-r--r-- 1 sourav3366 users  10M Feb 10  test4.log
drwxr-xr-x 2 sourav3366 users  4K  Feb 17  archive  # Archive folder created

$ ls -lh /home/sourav3366/tutorials/find_command/archive
-rw-r--r-- 1 sourav3366 users  2M  Feb 17  test1.log.gz
-rw-r--r-- 1 sourav3366 users  500K Feb 17  test2.log.gz
-rw-r--r-- 1 sourav3366 users  2.5M Feb 17  test3.log.gz

✅ Final Outcome

Day- 01/100 - Monitoring Free RAM Space with a Bash Script

Sourav kumar — Sun, 16 Feb 2025 16:05:40 +0000

Introduction

Monitoring free RAM is crucial for maintaining system performance, especially in production environments where resource constraints can impact applications. In this blog, we'll create a simple Bash script to monitor available RAM and send alerts if it falls below a specified threshold.

Prerequisites

Before proceeding, ensure you have:
A Linux-based system (Ubuntu, CentOS, etc.)
Basic knowledge of Bash scripting
Access to free, awk, and mail commands

Understanding Free RAM Command

To check available RAM on a Linux system, we use the free command:

If we want a total column of memory and swap then we can use "free -mt"

But this is not human readable format . To make it human readable we can use " -h " command.

Now we can take out only the total row also using "grep" ( global regular expression print)

Here after grep we used expression "Total" means it will all rows which will have "Total" string in it.

We can also use awk after grep to take out fixed column also like, if i want to takeout only "free" column then i can use "awk '{print $4}' as free column number is 4.

We will focus on the free column to determine how much memory is completely unused by the system.

Writing the Bash Script

Let's create a Bash script that:

Retrieves free RAM.
Compares it with a threshold value.
Sends an alert if RAM is below the threshold.

Step 1: Create the script file

touch monitor_free_ram.sh
chmod +x monitor_free_ram.sh
vi monitor_free_ram.sh

Step 2: Add the script content

#!/bin/bash

# Define RAM threshold (in MB)
THRESHOLD=500  # Adjust as needed

# Get free RAM
FREE_RAM=$(free -mt | grep "Total" | awk '{print $4}')

# Log current status
echo "Free RAM: $FREE_RAM MB"

# Check if free RAM is below threshold
if [[ "$FREE_RAM" -lt "$THRESHOLD" ]]; then
    MESSAGE="Warning: Low Free RAM! Only $FREE_RAM MB free."
    echo "$MESSAGE"

    # Send email alert (Requires configured mail service)
    echo "$MESSAGE" | mail -s "Low Free RAM Alert" 
    souravbit3366@gmail.com
fi

How the Script Works

We set a threshold value (e.g., 500MB).
The script extracts the free RAM using free -m and awk.
If the free RAM falls below the threshold, a warning message is printed and emailed.

Automating the Script with Cron Job

To run this script automatically, schedule a cron job:

crontab -e

Add this line to check every 5 minutes:

*/5 * * * * /path/to/monitor_free_ram.sh >> /var/log/ram_monitor.log 2>&1

Conclusion

This simple script helps monitor system memory and prevent performance issues. We can extend it by logging alerts, integrating it with monitoring tools like Prometheus, or sending notifications via Slack or Telegram.