DEV Community: Tanvir Rahman

Automating Harbor Registry Cleanup with Kubernetes

Tanvir Rahman — Mon, 26 May 2025 10:35:00 +0000

Container registries like Harbor can quickly accumulate unused images, consuming valuable storage space. Manual cleanup is tedious, so this script will safely automates the process while respecting images currently in use by Kubernetes.

The Challenge

After running Harbor in production for several months, we noticed:

Storage usage growing exponentially
Hundreds of outdated image tags
No easy way to identify which images were actually in use
Fear of breaking production by deleting important images

Script

#!/usr/bin/env python3
"""
Harbor Image Cleanup Script

This script cleans up old/unused images from Harbor registry while ensuring images
used by Kubernetes deployments are preserved.

Features:
- Connects to Harbor API to list and delete images
- Checks Kubernetes deployments to identify images in use
- Applies configurable retention policies (by age or count)
- Provides detailed logging
- Can be run manually or as a cron job

Requirements:
- Python 3.6+
- Required packages: requests, kubernetes, pyyaml, argparse, logging
"""

import os
import sys
import time
import re
import json
import base64
import logging
import argparse
import datetime
import yaml
import requests
from requests.auth import HTTPBasicAuth
from kubernetes import client, config
from kubernetes.client.rest import ApiException
from typing import Dict, List, Set, Tuple, Optional, Any

# Configure logging
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    handlers=[
        logging.StreamHandler(sys.stdout),
        logging.FileHandler('harbor-cleanup.log')
    ]
)
logger = logging.getLogger('harbor-cleanup')

class HarborClient:
    """Client for interacting with Harbor API"""

    def __init__(self, harbor_url: str, username: str, password: str, verify_ssl: bool = True):
        """
        Initialize Harbor client

        Args:
            harbor_url: URL of Harbor instance (e.g., https://harbor.example.com)
            username: Harbor username
            password: Harbor password
            verify_ssl: Whether to verify SSL certificates
        """
        self.harbor_url = harbor_url.rstrip('/')
        self.api_url = f"{self.harbor_url}/api"
        self.username = username
        self.password = password
        self.verify_ssl = verify_ssl
        self.session = requests.Session()
        self.session.auth = HTTPBasicAuth(username, password)
        self.session.verify = verify_ssl
        # Add X-Xsrftoken header to all requests
        self.session.headers.update({'X-Xsrftoken': 'token'})

        # Test connection
        try:
            self.ping()
            logger.info(f"Successfully connected to Harbor at {harbor_url}")
        except Exception as e:
            logger.error(f"Failed to connect to Harbor: {str(e)}")
            raise

    def ping(self) -> bool:
        """Test connection to Harbor API"""
        response = self.session.get(f"{self.harbor_url}/api/ping")
        response.raise_for_status()
        return True

    def get_projects(self) -> List[Dict]:
        """Get list of projects from Harbor"""
        response = self.session.get(f"{self.harbor_url}/api/projects")
        response.raise_for_status()
        return response.json()

    def get_project_id(self, project_name: str) -> int:
        """
        Get project ID from project name

        Args:
            project_name: Name of the Harbor project

        Returns:
            Project ID as integer

        Raises:
            ValueError: If project not found
        """
        projects = self.get_projects()
        for project in projects:
            if project.get('name') == project_name:
                return project.get('project_id')
        raise ValueError(f"Project {project_name} not found")

    def get_repositories(self, project_name: str) -> List[Dict]:
        """
        Get list of repositories in a project

        Args:
            project_name: Name of the Harbor project

        Returns:
            List of repository objects
        """
        project_id = self.get_project_id(project_name)
        response = self.session.get(f"{self.harbor_url}/api/repositories?project_id={project_id}")
        response.raise_for_status()
        return response.json()

    def get_tags(self, repository_name: str) -> List[Dict]:
        """
        Get list of tags in a repository

        Args:
            repository_name: Full name of repository (project/repo)

        Returns:
            List of tag objects
        """
        # repository_name should be in format "project/repository"
        response = self.session.get(f"{self.harbor_url}/api/repositories/{repository_name}/tags")
        response.raise_for_status()
        return response.json()

    def delete_tag(self, repository_name: str, tag: str) -> bool:
        """
        Delete a tag from Harbor

        Args:
            repository_name: Full name of repository (project/repo)
            tag: Tag to delete

        Returns:
            True if deletion was successful
        """
        response = self.session.delete(
            f"{self.harbor_url}/api/repositories/{repository_name}/tags/{tag}"
        )
        response.raise_for_status()
        logger.info(f"Deleted tag {repository_name}:{tag}")
        return True


class KubernetesClient:
    """Client for interacting with Kubernetes API"""

    def __init__(self, kubeconfig: Optional[str] = None, context: Optional[str] = None):
        """
        Initialize Kubernetes client

        Args:
            kubeconfig: Path to kubeconfig file (defaults to ~/.kube/config)
            context: Kubernetes context to use
        """
        try:
            if kubeconfig:
                config.load_kube_config(kubeconfig, context=context)
            else:
                config.load_kube_config(context=context)
            self.apps_v1 = client.AppsV1Api()
            self.core_v1 = client.CoreV1Api()
            logger.info("Successfully connected to Kubernetes")
        except Exception as e:
            logger.error(f"Failed to connect to Kubernetes: {str(e)}")
            raise

    def get_images_in_use(self) -> Set[str]:
        """
        Get set of all images currently in use in the cluster

        Returns:
            Set of image references (including tags/digests)
        """
        images = set()

        # Check deployments
        try:
            deployments = self.apps_v1.list_deployment_for_all_namespaces()
            for deployment in deployments.items:
                for container in deployment.spec.template.spec.containers:
                    images.add(container.image)
                if deployment.spec.template.spec.init_containers:
                    for container in deployment.spec.template.spec.init_containers:
                        images.add(container.image)
        except ApiException as e:
            logger.error(f"Error getting deployments: {str(e)}")

        # Check statefulsets
        try:
            statefulsets = self.apps_v1.list_stateful_set_for_all_namespaces()
            for statefulset in statefulsets.items:
                for container in statefulset.spec.template.spec.containers:
                    images.add(container.image)
                if statefulset.spec.template.spec.init_containers:
                    for container in statefulset.spec.template.spec.init_containers:
                        images.add(container.image)
        except ApiException as e:
            logger.error(f"Error getting statefulsets: {str(e)}")

        # Check daemonsets
        try:
            daemonsets = self.apps_v1.list_daemon_set_for_all_namespaces()
            for daemonset in daemonsets.items:
                for container in daemonset.spec.template.spec.containers:
                    images.add(container.image)
                if daemonset.spec.template.spec.init_containers:
                    for container in daemonset.spec.template.spec.init_containers:
                        images.add(container.image)
        except ApiException as e:
            logger.error(f"Error getting daemonsets: {str(e)}")

        # Check pods (for CronJobs and Jobs)
        try:
            pods = self.core_v1.list_pod_for_all_namespaces()
            for pod in pods.items:
                for container in pod.spec.containers:
                    images.add(container.image)
                if pod.spec.init_containers:
                    for container in pod.spec.init_containers:
                        images.add(container.image)
        except ApiException as e:
            logger.error(f"Error getting pods: {str(e)}")

        logger.info(f"Found {len(images)} unique images in use in Kubernetes")
        return images


class HarborCleanup:
    """Main class for Harbor cleanup operations"""

    def __init__(
        self,
        harbor_url: str,
        harbor_username: str,
        harbor_password: str,
        kubeconfig: Optional[str] = None,
        kube_context: Optional[str] = None,
        dry_run: bool = False,
        projects: Optional[List[str]] = None,
        exclude_projects: Optional[List[str]] = None,
        keep_days: Optional[int] = None,
        keep_tags: Optional[int] = None,
        skip_in_use: bool = True,
        always_keep_tags: Optional[List[str]] = None,
        verify_ssl: bool = True,
    ):
        """
        Initialize Harbor cleanup

        Args:
            harbor_url: URL of Harbor instance
            harbor_username: Harbor username
            harbor_password: Harbor password
            kubeconfig: Path to kubeconfig file
            kube_context: Kubernetes context to use
            dry_run: If True, don't actually delete anything
            projects: List of projects to clean up (if None, clean all)
            exclude_projects: List of projects to exclude from cleanup
            keep_days: Keep images newer than this many days
            keep_tags: Keep this many most recent tags per repository
            skip_in_use: Skip images in use by Kubernetes
            always_keep_tags: List of tag patterns to always keep (e.g., ['latest', 'stable', 'prod-*'])
            verify_ssl: Whether to verify SSL certificates
        """
        self.harbor_client = HarborClient(harbor_url, harbor_username, harbor_password, verify_ssl)
        self.kube_client = KubernetesClient(kubeconfig, kube_context) if skip_in_use else None
        self.dry_run = dry_run
        self.projects_filter = projects
        self.exclude_projects = exclude_projects or []
        self.keep_days = keep_days
        self.keep_tags = keep_tags
        self.skip_in_use = skip_in_use
        self.always_keep_tags = always_keep_tags or ["latest", "stable", "master"]

        if not self.keep_days and not self.keep_tags:
            logger.warning("No retention policy specified, defaulting to keeping 30 days")
            self.keep_days = 30

        if self.dry_run:
            logger.info("DRY RUN MODE: No images will be deleted")

    def should_keep_tag(self, tag_name: str) -> bool:
        """
        Check if a tag should be kept based on tag patterns

        Args:
            tag_name: Name of the tag

        Returns:
            True if the tag should be kept
        """
        for pattern in self.always_keep_tags:
            if '*' in pattern:
                # Convert glob pattern to regex
                regex_pattern = pattern.replace('.', '\\.').replace('*', '.*')
                if re.match(f"^{regex_pattern}$", tag_name):
                    return True
            elif pattern == tag_name:
                return True
        return False

    def run(self):
        """Run the cleanup process"""
        # Get all images in use by Kubernetes
        k8s_images = self.kube_client.get_images_in_use() if self.skip_in_use else set()

        # Normalize Kubernetes image references to match Harbor format
        normalized_k8s_images = self._normalize_k8s_images(k8s_images)

        # Get projects to clean
        all_projects = self.harbor_client.get_projects()
        projects_to_clean = []

        for project in all_projects:
            project_name = project['name']

            if self.projects_filter and project_name not in self.projects_filter:
                logger.debug(f"Skipping project {project_name} (not in filter)")
                continue

            if project_name in self.exclude_projects:
                logger.debug(f"Skipping project {project_name} (in exclude list)")
                continue

            projects_to_clean.append(project_name)

        logger.info(f"Found {len(projects_to_clean)} projects to clean")

        # Process each project
        for project_name in projects_to_clean:
            self._clean_project(project_name, normalized_k8s_images)

    def _normalize_k8s_images(self, k8s_images: Set[str]) -> Dict[str, Set[str]]:
        """
        Normalize Kubernetes image references to match Harbor format

        Args:
            k8s_images: Set of image references from Kubernetes

        Returns:
            Dict mapping repository names to sets of tags/digests
        """
        normalized = {}
        harbor_domain = self.harbor_client.harbor_url.replace('https://', '').replace('http://', '')

        for image in k8s_images:
            # Skip images not from our Harbor
            if not image.startswith(harbor_domain):
                continue

            # Extract repository and tag/digest
            if '@sha256:' in image:
                repo, digest = image.split('@sha256:', 1)
                digest = f"sha256:{digest}"
                repo = repo.replace(f"{harbor_domain}/", '')

                if repo not in normalized:
                    normalized[repo] = set()
                normalized[repo].add(digest)
            else:
                if ':' in image[image.find('/')+1:]:  # Make sure we're not splitting on the port
                    repo, tag = image.rsplit(':', 1)
                    repo = repo.replace(f"{harbor_domain}/", '')

                    if repo not in normalized:
                        normalized[repo] = set()
                    normalized[repo].add(tag)
                else:
                    # Image without tag (implicitly 'latest')
                    repo = image.replace(f"{harbor_domain}/", '')

                    if repo not in normalized:
                        normalized[repo] = set()
                    normalized[repo].add('latest')

        return normalized

    def _clean_project(self, project_name: str, k8s_images: Dict[str, Set[str]]):
        """
        Clean a single Harbor project

        Args:
            project_name: Name of the Harbor project
            k8s_images: Dict mapping repository names to sets of tags/digests
        """
        logger.info(f"Cleaning project: {project_name}")

        # Get repositories in the project
        try:
            repositories = self.harbor_client.get_repositories(project_name)
        except Exception as e:
            logger.error(f"Error getting repositories for project {project_name}: {str(e)}")
            return

        logger.info(f"Found {len(repositories)} repositories in project {project_name}")

        for repo in repositories:
            repo_name = repo['name']  # Should be in format "project/repo"
            self._clean_repository(repo_name, k8s_images)

    def _clean_repository(self, repo_name: str, k8s_images: Dict[str, Set[str]]):
        """
        Clean a single Harbor repository

        Args:
            repo_name: Full name of the repository (project/repo)
            k8s_images: Dict mapping repository names to sets of tags/digests
        """
        logger.info(f"Cleaning repository: {repo_name}")

        # Get tags in the repository
        try:
            tags = self.harbor_client.get_tags(repo_name)
        except Exception as e:
            logger.error(f"Error getting tags for repository {repo_name}: {str(e)}")
            return

        # Sort tags by creation time (newest first)
        tags.sort(key=lambda x: x.get('created', ''), reverse=True)

        # Keep track of how many tags we've kept
        kept_count = 0

        for i, tag in enumerate(tags):
            tag_name = tag['name']
            created_time = tag.get('created')

            # Check if tag is in use by Kubernetes
            in_use = False
            if self.skip_in_use and repo_name in k8s_images:
                if tag_name in k8s_images[repo_name]:
                    in_use = True
                    logger.info(f"Keeping {repo_name}:{tag_name} (in use by Kubernetes)")

            # Check if we should keep this tag based on retention policies
            should_keep = False

            # Check if tag matches the always-keep patterns
            if self.should_keep_tag(tag_name):
                should_keep = True
                logger.info(f"Keeping {repo_name}:{tag_name} (matches always-keep pattern)")

            # Check age-based retention
            if not should_keep and self.keep_days is not None and created_time:
                created_date = datetime.datetime.fromisoformat(created_time.replace('Z', '+00:00'))
                age_days = (datetime.datetime.now(datetime.timezone.utc) - created_date).days
                if age_days < self.keep_days:
                    should_keep = True
                    logger.info(f"Keeping {repo_name}:{tag_name} (age: {age_days} days, keep_days: {self.keep_days})")

            # Check count-based retention
            if not should_keep and self.keep_tags is not None:
                if kept_count < self.keep_tags:
                    should_keep = True
                    logger.info(f"Keeping {repo_name}:{tag_name} (keep_tags: {self.keep_tags})")

            # Delete or keep tag
            if in_use or should_keep:
                kept_count += 1
            else:
                logger.info(f"Deleting {repo_name}:{tag_name}")

                if not self.dry_run:
                    try:
                        self.harbor_client.delete_tag(repo_name, tag_name)
                    except Exception as e:
                        logger.error(f"Error deleting tag {repo_name}:{tag_name}: {str(e)}")


def get_harbor_credentials_from_kube() -> Tuple[Optional[str], Optional[str], Optional[str]]:
    """
    Attempt to extract Harbor credentials from Kubernetes context

    Returns:
        Tuple of (harbor_url, username, password)
    """
    # Try to load current context
    try:
        config.load_kube_config()
        v1 = client.CoreV1Api()

        # First, try to find harbor-auth secret in harbor namespace
        try:
            secret = v1.read_namespaced_secret("harbor-auth", "harbor")
            harbor_url = base64.b64decode(secret.data.get("url", "")).decode("utf-8")
            username = base64.b64decode(secret.data.get("username", "")).decode("utf-8")
            password = base64.b64decode(secret.data.get("password", "")).decode("utf-8")
            return harbor_url, username, password
        except:
            pass

        # Try to find any Harbor credentials in any namespace
        namespaces = v1.list_namespace()
        for ns in namespaces.items:
            ns_name = ns.metadata.name
            try:
                secrets = v1.list_namespaced_secret(ns_name)
                for secret in secrets.items:
                    if "harbor" in secret.metadata.name.lower():
                        data = secret.data
                        if data:
                            # Try to find url, username, password
                            harbor_url = None
                            username = None
                            password = None

                            for key in data:
                                value = base64.b64decode(data[key]).decode("utf-8")
                                key_lower = key.lower()

                                if "url" in key_lower or "host" in key_lower:
                                    harbor_url = value
                                elif "user" in key_lower:
                                    username = value
                                elif "pass" in key_lower:
                                    password = value

                            if harbor_url and username and password:
                                return harbor_url, username, password
            except:
                continue
    except:
        pass

    return None, None, None


def parse_args():
    """Parse command line arguments"""
    parser = argparse.ArgumentParser(
        description="Clean up old/unused images from Harbor registry"
    )

    # Harbor connection options
    harbor_group = parser.add_argument_group("Harbor Connection Options")
    harbor_group.add_argument(
        "--harbor-url", help="Harbor URL (e.g., https://harbor.example.com)"
    )
    harbor_group.add_argument("--harbor-username", help="Harbor username")
    harbor_group.add_argument("--harbor-password", help="Harbor password")
    harbor_group.add_argument(
        "--no-verify-ssl", action="store_true", help="Skip SSL certificate verification"
    )

    # Kubernetes options
    k8s_group = parser.add_argument_group("Kubernetes Options")
    k8s_group.add_argument("--kubeconfig", help="Path to kubeconfig file")
    k8s_group.add_argument("--kube-context", help="Kubernetes context to use")
    k8s_group.add_argument(
        "--no-skip-in-use", action="store_true", 
        help="Don't skip images in use by Kubernetes"
    )

    # Cleanup options
    cleanup_group = parser.add_argument_group("Cleanup Options")
    cleanup_group.add_argument(
        "--dry-run", action="store_true", help="Don't actually delete anything"
    )
    cleanup_group.add_argument(
        "--projects", nargs="*", help="Projects to clean up (if not specified, clean all)"
    )
    cleanup_group.add_argument(
        "--exclude-projects", nargs="*", help="Projects to exclude from cleanup"
    )
    cleanup_group.add_argument(
        "--keep-days", type=int, help="Keep images newer than this many days"
    )
    cleanup_group.add_argument(
        "--keep-tags", type=int, help="Keep this many most recent tags per repository"
    )
    cleanup_group.add_argument(
        "--always-keep-tags", nargs="*", 
        default=["latest", "stable", "master"],
        help="Tag patterns to always keep (supports wildcards, e.g., 'prod-*')"
    )

    # Other options
    parser.add_argument(
        "--config", help="Path to YAML config file (overrides command line options)"
    )
    parser.add_argument(
        "--use-kube-auth", action="store_true",
        help="Extract Harbor credentials from Kubernetes secrets"
    )
    parser.add_argument(
        "--verbose", "-v", action="count", default=0, help="Increase verbosity"
    )

    args = parser.parse_args()

    # Handle verbosity
    if args.verbose == 1:
        logger.setLevel(logging.INFO)
    elif args.verbose >= 2:
        logger.setLevel(logging.DEBUG)

    # Load config file if specified
    if args.config:
        try:
            with open(args.config, 'r') as f:
                config_data = yaml.safe_load(f)

            # Update args with config file values
            for key, value in config_data.items():
                if value is not None:
                    setattr(args, key.replace('-', '_'), value)
        except Exception as e:
            logger.error(f"Error loading config file: {str(e)}")

    # Try to extract Harbor credentials from Kubernetes if requested
    if args.use_kube_auth and not (args.harbor_url and args.harbor_username and args.harbor_password):
        harbor_url, username, password = get_harbor_credentials_from_kube()
        if harbor_url and username and password:
            logger.info("Successfully extracted Harbor credentials from Kubernetes")
            args.harbor_url = args.harbor_url or harbor_url
            args.harbor_username = args.harbor_username or username
            args.harbor_password = args.harbor_password or password
        else:
            logger.warning("Failed to extract Harbor credentials from Kubernetes")

    # Validate required options
    if not args.harbor_url:
        parser.error("Harbor URL is required")
    if not args.harbor_username:
        parser.error("Harbor username is required")
    if not args.harbor_password:
        parser.error("Harbor password is required")

    return args


def main():
    """Main entry point"""
    args = parse_args()

    cleanup = HarborCleanup(
        harbor_url=args.harbor_url,
        harbor_username=args.harbor_username,
        harbor_password=args.harbor_password,
        kubeconfig=args.kubeconfig,
        kube_context=args.kube_context,
        dry_run=args.dry_run,
        projects=args.projects,
        exclude_projects=args.exclude_projects,
        keep_days=args.keep_days,
        keep_tags=args.keep_tags,
        skip_in_use=not args.no_skip_in_use,
        always_keep_tags=args.always_keep_tags,
        verify_ssl=not args.no_verify_ssl,
    )

    try:
        cleanup.run()
        logger.info("Cleanup completed successfully")
    except KeyboardInterrupt:
        logger.info("Cleanup interrupted by user")
        sys.exit(1)
    except Exception as e:
        logger.error(f"Cleanup failed: {str(e)}")
        sys.exit(1)


if __name__ == "__main__":
    main()

The Solution

A Python script that:

Integrates with Harbor's API
Checks Kubernetes for active deployments
Applies configurable retention policies
Provides safety mechanisms like dry-run mode

Key Features

# Sample configuration showing main features
cleanup = HarborCleanup(
    harbor_url="https://harbor.example.com",
    harbor_username="admin",
    harbor_password="secret",
    keep_days=30,          # Keep images newer than 30 days
    keep_tags=5,           # Keep 5 most recent tags per repo
    skip_in_use=True,      # Don't delete images used by Kubernetes
    always_keep_tags=[     # Protected tag patterns
        "latest",
        "stable",
        "prod-*"
    ],
    dry_run=True           # Safety first!
)

Safety Mechanisms

Dry Run Mode - Preview deletions without actually removing anything

./harbor-cleanup.py --dry-run --keep-days 30

Kubernetes Integration - Automatically detects in-use images

# Gets images from:
# - Deployments
# - StatefulSets
# - DaemonSets
# - Pods (for Jobs/CronJobs)
k8s_images = kube_client.get_images_in_use()

Protected Tags - Never delete important tags like latest or prod-*
Dual Retention Policies - Combine age-based and count-based rules

Getting Started

Install dependencies:

pip3 install pyyaml kubernetes requests

Run a test dry-run:

./harbor-cleanup.py \
  --use-kube-auth \
  --dry-run \
  --keep-days 30 \
  --keep-tags 5 \
  --always-keep-tags latest stable master "prod-*"

For production run (after verifying dry-run):

./harbor-cleanup.py \
  --harbor-url https://harbor.example.com \
  --harbor-username $HARBOR_USER \
  --harbor-password $HARBOR_PASS \
  --keep-days 30 \
  --keep-tags 5

Advanced Usage

Configuration File

Instead of command-line args, use a YAML config:

# config.yaml
harbor-url: https://harbor.example.com
harbor-username: admin
keep-days: 30
keep-tags: 5
exclude-projects:
  - infrastructure
  - legacy

Then run with:

./harbor-cleanup.py --config config.yaml

The script has helped us reduce Harbor storage usage by 70% while maintaining all production-critical images. The Kubernetes integration gives us confidence we won't break running applications.

Would love to hear about your registry cleanup experiences in the comments!

Secure Your Linux Server with Fail2Ban (Step-by-Step Guide)

Tanvir Rahman — Sun, 25 May 2025 11:09:00 +0000

Brute force attacks on SSH, mail and web services are a constant threat to any internet-facing server. Fail2Ban is a powerful, lightweight tool that helps mitigate these attacks by automatically banning IPs that exhibit malicious behavior.

In this guide, we’ll walk through the complete installation and configuration of Fail2Ban on a Linux server to secure SSH and Postfix (mail server). We’ll also set up email alerts and configure persistent bans.

🧰 Prerequisites

A Linux server (Debian/Ubuntu or CentOS/RHEL)
Root or sudo access
SSH access to the server
(Optional) A mail server (Postfix) if you want to monitor email service

🛠️ 1. Update System Packages

Before installing anything, make sure your package manager is up-to-date.

sudo apt update && sudo apt upgrade -y  # For Debian/Ubuntu
# or
sudo yum update -y                      # For CentOS/RHEL

📦 2. Install Fail2Ban and Dependencies

Install Fail2Ban using your system’s package manager:

sudo apt install -y fail2ban            # Debian/Ubuntu
# or
sudo yum install -y fail2ban            # CentOS/RHEL

📁 3. Create Custom Jail Configuration

Fail2Ban uses jails to define which services to monitor. Let’s back up the default config and create a custom one.

Backup the Default Config

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Modify Basic Settings

Set a 24-hour ban time, 10-minute find time, and allow a maximum of 5 retries:

sudo sed -i 's/bantime  = 10m/bantime  = 24h/;s/findtime  = 10m/findtime  = 600/;s/maxretry = 5/maxretry = 5/' /etc/fail2ban/jail.local

📄 4. Configure Jails for SSH and Postfix

Now, let’s create a separate jail configuration file:

sudo tee /etc/fail2ban/jail.d/custom.conf << 'EOF'
[sshd]
enabled = true
port = 22
filter = sshd
logpath = /var/log/auth.log
maxretry = 5

[postfix]
enabled = true
port = 25
filter = postfix
logpath = /var/log/mail.log
maxretry = 5
EOF

📧 5. Configure Email Notifications

Ensure mailutils is installed and the mail command is available:

which mail
# Should return: /usr/bin/mail

Then, add your email settings in jail.local:

sudo tee -a /etc/fail2ban/jail.local << 'EOF'

[DEFAULT]
banaction = iptables-multiport
banaction_allports = iptables-allports
loglevel = DEBUG
ignoreself = false
action = %(action_mwl)s
sender = fail2ban@yourdomain.com
destemail = admin@yourdomain.com
mta = sendmail

EOF

Replace yourdomain.com and the email addresses with your actual values.

📊 6. Enable and Start Fail2Ban

Enable the service to start on boot and start it now:

sudo systemctl enable fail2ban
sudo systemctl start fail2ban
sudo systemctl status fail2ban

Restart Fail2Ban to Apply All Changes

sudo systemctl restart fail2ban

🧪 7. Testing & Monitoring

Simulate a Ban

Try multiple failed SSH login attempts from another IP.

Check Logs

sudo tail -f /var/log/fail2ban.log

View Real-Time Jail Status

sudo fail2ban-client status sshd

Unban an IP

sudo fail2ban-client set sshd unbanip <IP_ADDRESS>

🔧 Optional: Adjust UFW if Installed

If you use ufw, remove conflicting rules and allow SSH manually:

sudo ufw delete limit 22/tcp
sudo ufw delete limit 22/tcp "v6"
sudo ufw allow 22/tcp

✅ Summary

Here’s what we’ve accomplished:

✅ Updated system packages
✅ Installed and configured Fail2Ban
✅ Created custom jails for SSH and Postfix
✅ Set ban time (24h), find time (10m), max retry (5)
✅ Configured email alerts
✅ Enabled persistent bans and logging
✅ Tested the setup

Your server is now better protected against brute force attacks. 🎉

📌 Next Steps

Add additional jails (e.g., Nginx, Apache) if you run web servers
Whitelist internal or VPN IPs (/etc/fail2ban/jail.d/whitelist.conf)
Monitor logs regularly for suspicious activity
Fine-tune thresholds if you experience false positives

Stay safe and secure! 🔐

📬 Setting Up Postfix with SendGrid Relay on Ubuntu: A Step-by-Step Guide

Tanvir Rahman — Mon, 19 May 2025 10:14:20 +0000

If you’ve ever tried to send email directly from a Linux server, you know it can be a hassle—especially when dealing with deliverability and authentication. Fortunately, with Postfix and a reliable SMTP provider like SendGrid, you can offload the heavy lifting and ensure your emails land safely in inboxes.

In this guide, I’ll walk you through how to configure Postfix to relay mail through SendGrid, complete with authentication, sender rewriting, and testing.

🛠️ Prerequisites

A Linux server (tested on Ubuntu)
A SendGrid account and SMTP credentials
Root or sudo access
Basic comfort with the command line

✅ Step 1: Install Postfix
Let’s start by installing Postfix non-interactively to avoid configuration prompts:

DEBIAN_FRONTEND=noninteractive sudo apt-get install -y postfix

🔐 Step 2: Set Up SASL Authentication for SendGrid
To authenticate with SendGrid’s SMTP server, we’ll create a file containing your credentials:

echo "[smtp.sendgrid.net]:587 username:password" | sudo tee /etc/postfix/sasl_passwd
sudo chmod 600 /etc/postfix/sasl_passwd
sudo postmap /etc/postfix/sasl_passwd

Replace username:password with your actual SendGrid credentials (your username is usually "apikey", and the password is your generated API key).

📨 Step 3: Configure Sender Canonical Mapping
Sometimes you want all outgoing emails to appear as if they came from a specific address. This is where sender_canonical comes in.

echo "/.+@mail\.example\.com/ fromAddress@example.com
/^root@.*/ fromAddress@example.com
/@.*/ fromAddress@example.com" | sudo tee /etc/postfix/sender_canonical
sudo postmap /etc/postfix/sender_canonical

These regex patterns help normalize the sender addresses. For instance, even if an internal process sends an email as root@mail.example.com, it will appear to come from fromAddress@example.com.

✉️ Step 4: (Optional) Create a Generic Mapping File
If you’re rewriting sender addresses more broadly, this is useful. Otherwise, you can skip it.

echo "username@example.com relayuser@sendgrid.net" | sudo tee /etc/postfix/generic
sudo postmap /etc/postfix/generic

Make sure this matches your intended email addresses.

⚙️ Step 5: Configure Postfix (main.cf)
Now for the core config. We’ll overwrite the main.cf file to define everything, including SendGrid relay, network settings, and sender rewriting.

sudo tee /etc/postfix/main.cf > /dev/null << 'EOL'
# Basic Settings
compatibility_level = 2
queue_directory = /var/spool/postfix
command_directory = /usr/sbin
daemon_directory = /usr/lib/postfix/sbin
data_directory = /var/lib/postfix
mail_owner = postfix

# Network Settings
myhostname = mail.example.com
mydomain = example.com
myorigin = example.com
inet_interfaces = all
inet_protocols = ipv4
mydestination = $myhostname, localhost.$mydomain, localhost

# SendGrid Relay Configuration
relayhost = [smtp.sendgrid.net]:587
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_tls_security_level = none

# Additional Settings
smtpd_banner = $myhostname ESMTP
biff = no
append_dot_mydomain = no
readme_directory = no

# Sender Settings
smtp_generic_maps = hash:/etc/postfix/generic

# SMTP Client Restrictions

# Message Size Limits
message_size_limit = 10240000
mailbox_size_limit = 0

# Local Recipient Settings
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
local_recipient_maps = proxy:unix:passwd.byname $alias_maps
local_transport = local:$myhostname

# Network Security
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
sender_canonical_maps = regexp:/etc/postfix/sender_canonical
EOL

Adjust myhostname, mydomain, and fromAddress@example.com to match your real domain and sending address.

👥 Step 6: Update Aliases and Restart Postfix
Make sure aliases are recognized and apply the config changes:

sudo newaliases && sudo systemctl restart postfix && sudo systemctl status postfix

You should see Active: active (running) in the output.

🧪 Step 7: Send a Test Email
Let’s test if your relay setup is working:

echo "This is a test email from Postfix with SendGrid relay" | mail -s "Test Email from Postfix" -r fromAddress@example.com toAddress@example.com

Make sure mailutils is installed if this fails:

sudo apt-get install -y mailutils

📄 Step 8: Monitor Logs
To debug or confirm delivery, keep an eye on the logs:

tail -f /var/log/mail.log

This log is your go-to source for tracking down any delivery issues.

📂 Bonus: View Postfix Config and Files
Here’s a snapshot of how your /etc/postfix/ directory should look after setup:

ls -la /etc/postfix/ && echo -e "\nMain config file contents:" && cat /etc/postfix/main.cf

You should see key files like:

main.cf – your main configuration
sasl_passwd & sasl_passwd.db – credentials for SendGrid
sender_canonical & sender_canonical.db – for rewriting sender addresses
generic & generic.db – for optional address rewriting

✅ Summary
You’ve now got a reliable, authenticated Postfix setup using SendGrid’s SMTP relay. Here's what you've accomplished:

Installed and configured Postfix
Authenticated with SendGrid securely
Rewritten sender addresses as needed
Sent and verified test emails
Monitored logs and verified configuration

This setup is great for sending system alerts, contact form emails, or even automating reports via cron.

🚨 Auto-Reboot Your Server on High CPU / Memory Load (With Safety Checks)

Tanvir Rahman — Thu, 15 May 2025 06:39:33 +0000

Sometimes a Linux server can get overwhelmed by sustained high CPU / Memory load — due to runaway processes, DDoS attacks or rogue scripts. Manually catching and fixing it in real-time isn't always possible. This guide will show you how to automatically reboot your server when CPU load is critically high for multiple minutes — but safely.

We’ll create a lightweight bash script, track load over time, and schedule it with cron to run every minute.

📜 Step 1: Create the Auto-Reboot Script
Let’s start by writing the script that checks system load and initiates a reboot after 3 consecutive high-load readings.

✏️ Create or Edit the Script

sudo nano /usr/local/bin/reboot-on-high-load.sh

Paste this logic into the file:

#!/bin/bash

# Configuration
CPU_THRESHOLD=0.8                # 80% load per CPU core
MEM_THRESHOLD=90                 # 90% memory usage threshold
MAX_RETRIES=3                    # Reboot if high load/memory persists N times
CHECK_FILE="/tmp/highload.counter"
LOG_FILE="/var/log/reboot-load.log"
MAX_LOG_SIZE=$((10 * 1024 * 1024))  # 10MB

# Auto-truncate log if too large
if [ -f "$LOG_FILE" ] && [ "$(stat -c%s "$LOG_FILE")" -ge "$MAX_LOG_SIZE" ]; then
    echo "[!] Log file exceeded 10MB. Truncating..." >> "$LOG_FILE"
    truncate -s 0 "$LOG_FILE"
fi

# Detect CPU cores and load threshold
CPU_CORES=$(nproc)
LOAD_THRESHOLD=$(echo "$CPU_CORES * $CPU_THRESHOLD" | bc)

# CPU Load Check
LOAD_AVG=$(awk '{print $1}' /proc/loadavg)
LOAD_OK=$(echo "$LOAD_AVG < $LOAD_THRESHOLD" | bc)

# Memory Check
MEM_USED_PERCENT=$(free | awk '/Mem:/ { printf("%.0f", $3/$2 * 100) }')
MEM_OK=$( [ "$MEM_USED_PERCENT" -lt "$MEM_THRESHOLD" ] && echo 1 || echo 0 )

# Initialize counter file if needed
if [ ! -f "$CHECK_FILE" ]; then
    echo 0 > "$CHECK_FILE"
fi

COUNTER=$(cat "$CHECK_FILE")

# Evaluate system status
if [ "$LOAD_OK" -eq 0 ] || [ "$MEM_OK" -eq 0 ]; then
    echo "[!] High resource usage detected at $(date):" >> "$LOG_FILE"
    [ "$LOAD_OK" -eq 0 ] && echo "    - CPU Load: $LOAD_AVG / Threshold: $LOAD_THRESHOLD" >> "$LOG_FILE"
    [ "$MEM_OK" -eq 0 ] && echo "    - Memory Usage: $MEM_USED_PERCENT% / Threshold: $MEM_THRESHOLD%" >> "$LOG_FILE"
    COUNTER=$((COUNTER + 1))
    echo "$COUNTER" > "$CHECK_FILE"
else
    if [ "$COUNTER" -ne 0 ]; then
        echo "[✓] Resources back to normal at $(date): Load = $LOAD_AVG, Mem = $MEM_USED_PERCENT%" >> "$LOG_FILE"
    fi
    echo 0 > "$CHECK_FILE"
fi

# Reboot if over threshold for too long
if [ "$COUNTER" -ge "$MAX_RETRIES" ]; then
    echo "[!!!] High resource usage sustained for $MAX_RETRIES checks. Rebooting at $(date)..." >> "$LOG_FILE"
    rm -f "$CHECK_FILE"
    /sbin/shutdown -r now
fi

🔓 Step 2: Make the Script Executable

sudo chmod +x /usr/local/bin/reboot-on-high-load.sh

⏰ Step 3: Schedule It to Run Every Minute
Open root crontab:

sudo crontab -e

Add this line to the bottom:

* * * * * /usr/local/bin/reboot-on-high-load.sh >> /var/log/reboot-load.log 2>&1

This schedules the script to run every minute and logs its output to /var/log/reboot-load.log.

🛡️ Why This Is Safe
✅ No reboots on single spikes — It only reboots if load stays high for 3 checks.

✅ Self-resetting — If load normalizes, the counter resets.

✅ Persistent state tracking — Uses /tmp/highload.counter.

✅ Simple logging — Outputs to /var/log/reboot-load.log.

✅ Cron-scheduled — Lightweight, runs every 60 seconds.

🧪 Optional: Test the Script by Simulating High Load
🛠️ 1. Install a CPU Stress Tool
On Ubuntu/Debian:

sudo apt update && sudo apt install -y stress

On Alpine Linux:

apk add stress

If stress is not available, use yes as a simple CPU loader (see below).

🚀 2. Simulate High Load for Over 3 Minutes
Option A: With stress (preferred)

stress --cpu $(nproc) --timeout 200

--cpu $(nproc) starts 1 thread per core.

--timeout 200 runs for 200 seconds (~3.3 minutes).

Option B: With yes Command

for i in $(seq 1 $(nproc)); do yes > /dev/null & done

Let it run for at least 3 minutes, then stop it:

killall yes

📝 3. Monitor the Logs
In a second terminal, run:

tail -f /var/log/reboot-load.log

You’ll see output like:

yaml
Copy
Edit
[!] Load is high: 7.23 / Threshold: 6.40
[!] Load is high: 7.45 / Threshold: 6.40
[!!!] High load sustained for 3 checks. Rebooting...
Then the system will automatically reboot.

✅ Recap
By following this guide, you’ve set up a safe and efficient way to auto-reboot your Linux server during persistent high-load events:

✅ Script with thresholds and state tracking

✅ Cronjob for regular checks

✅ No false positives from single spikes

✅ Full log output for auditing

Comprehensive Guide to Optimizing Nginx Configuration

Tanvir Rahman — Wed, 21 Feb 2024 17:35:31 +0000

Nginx stands out as a high-performance web server and reverse proxy known for its scalability and extensive configuration options. In this comprehensive guide, we'll thoroughly explore each directive within a sample Nginx configuration file, dissecting its functionality and discussing best practices for optimization, all while preserving the integrity of the original configuration.

Understanding the Nginx Configuration File

Let's commence our exploration by carefully examining the provided Nginx configuration file:

# Setting the number of worker processes to auto to dynamically adjust based on available system resources
worker_processes auto;

# Configuring event handling, including setting maximum worker connections
events {
    # Checking the system limit for file descriptors using 'ulimit -n'
    worker_connections 1024;
}

# HTTP block containing global settings
http {
    # Including MIME types configuration file for proper content type detection
    include mime.types;

    # Configuring buffer sizes for handling client requests
    client_body_buffer_size 10K;
    client_max_body_size 8m;
    client_header_buffer_size 1k;

    # Configuring timeouts for handling client requests
    client_body_timeout 12;
    client_header_timeout 12;
    keepalive_timeout 15;
    send_timeout 10;

    # Enabling sendfile for efficient file transmission
    sendfile on;

    # Optimizing sendfile packets for better performance
    tcp_nopush on;

    # Server block defining HTTP server settings
    server {
        listen 80;
        server_name 172.16.133.129;
        root /home/vagrant/sites/blog;
        try_files $uri /not-found;

        # Location block for handling root path
        location / {
            index index.html;
        }

        # Location block for handling /welcome path
        location /welcome {
            return 200 'Hello from welcome page';
        }


        # Exact match - will only match greet
        # location = /welcome {
        #     return 200 'Hello from welcome page';
        # }

        # Regex match
        # location ~ /welcome[0-9] {
        #     return 200 'Hello from welcome page';
        # }

        # Preferred
        # location ^~ /welcome\d {
        #     return 200 'Hello from welcome page';
        # }

        # Regex match and case insensitive
        # location ~* /welcome[0-9] {
        #     return 200 'Hello from welcome page';
        # }

        # Location block for handling /arguments path
        location /arguments {
            return 200 "$arg_name";
        }

        # Location block for handling /get-weekend path
        location /get-weekend {
            return 200 '$weekend $date_local';
        }

        # Capture part of the request and rewrite it
        location ~ ^/week/(\w+)$ {
            rewrite ^/week/(\w+)$ /weekend/$1 last;
        }

        # Match the rewritten request and return the captured value
        location ~ ^/weekend/(?<day>\w+)$ {
            return 200 "$day";
        }

        # Redirect
        location /logo {
            return 307 /assets/brand/bootstrap-logo.svg;
        }

        # Location block for handling /secret path
        location /secret {
            access_log /var/log/nginx/secret.access.log;
            access_log /var/log/nginx/access.log;
            return 200 "Welcome to secret area";
        }

        # Location block for handling /most-secret path
        location /most-secret {
            access_log off;
            return 200 "Welcome to most secret area";
        }

        # Location block for handling non-existent resources
        location /not-found {
            return 404 'Page Not Found';
        }

        # Additional location blocks...
    }
}

Detailed Explanation of Directives

Worker Processes and Events

The worker_processes directive sets the number of worker processes Nginx should use to handle incoming connections. When set to auto, Nginx dynamically adjusts the number of worker processes based on available system resources. Inside the events block, worker_connections determines the maximum number of simultaneous connections each worker process can handle. It's crucial to adjust this value according to your server's capacity and expected traffic levels.

HTTP Settings

The http block encapsulates global settings related to HTTP functionality. The include mime.types directive is essential for proper content type detection, as it includes a file (mime.types) mapping file extensions to MIME types. This ensures accurate interpretation of file types when serving content.

Buffer Settings

Nginx employs buffers to efficiently manage client requests and responses. The client_body_buffer_size and client_header_buffer_size directives set the buffer sizes for reading client request bodies and headers, respectively. Proper adjustment of these values is crucial, especially for handling large requests. Additionally, client_max_body_size specifies the maximum size of client request bodies, helping prevent denial-of-service attacks and ensuring server stability.

Timeout Settings

Timeouts play a vital role in managing client connections and preventing resource exhaustion. The client_body_timeout and client_header_timeout directives define the maximum time allowed for reading client request bodies and headers, respectively. Fine-tuning these values is essential for optimizing server performance and mitigating potential security risks. Similarly, keepalive_timeout determines how long Nginx should keep a connection open for subsequent requests from the same client, reducing latency and improving user experience. The send_timeout directive sets the maximum time for the client to accept or receive a response, preventing stalled connections.

File Handling and Optimization

Nginx offers efficient mechanisms for handling static files. Enabling sendfile allows Nginx to use the operating system's sendfile system call to transmit files directly, significantly improving file transmission speed and reducing server load. The tcp_nopush directive optimizes the transmission of file packets over TCP connections by sending file packets as soon as possible without waiting for the entire buffer to fill up, thus reducing latency and enhancing network performance.

Server Block

The server block defines settings specific to the HTTP server. Here, we configure Nginx to listen on port 80 for incoming requests and specify the server's IP address or domain name. Additionally, we set the root directory for serving files and define a fallback mechanism (try_files) for handling requests to non-existent resources.

Location Block Matching Options

In Nginx, location blocks are used to define how the server should respond to different URI patterns. There are several matching options available, each serving a specific purpose. Let's explore some of these options and their implications:

Exact Match

The location = /welcome block, when uncommented, specifies an exact match for the URI /welcome. This means that only requests to /welcome will be handled by this location block. Any other requests, such as /welcome/ or /welcome.html, will not match this block.

Regex Match

The location ~ /welcome[0-9] block, when uncommented, uses a regular expression to match URIs that start with /welcome followed by a numeric digit. For example, it would match URIs like /welcome1, /welcome2, etc.

Preferred Match

The location ^~ /welcome\d block, when uncommented, is a preferred match for URIs starting with /welcome followed by a numeric digit. The ^~ modifier ensures that if a URI matches this pattern, Nginx will use this location block instead of others with the same prefix.

Case-Insensitive Regex Match

The location ~* /welcome[0-9] block, when uncommented, performs a case-insensitive regex match for URIs starting with /welcome followed by a numeric digit. This means it will match URIs like /welcome1, /Welcome2, /WELCOME3, etc., regardless of case.

Rewrite and Redirect

Within location blocks, the rewrite directive modifies the URI of incoming requests. For example, the rewrite directive captures part of the request and rewrites it to another URI pattern. Additionally, the return directive performs HTTP redirects. For instance, the /logo location block issues a 307 redirect to /assets/brand/bootstrap-logo.svg.

Handling Arguments

The /arguments location block illustrates how Nginx handles query string arguments. The $arg_name variable retrieves the value of the name parameter from the request's query string and returns it as the response.

Conclusion

Mastering Nginx configuration is essential for optimizing server performance, enhancing security, and delivering a seamless user experience. By understanding each directive in the configuration file and applying best practices for optimization, you can create a highly efficient and secure Nginx setup tailored to your application's requirements. Experiment with different configurations, monitor server metrics, and stay updated on Nginx developments to continually refine and improve your web server infrastructure.

Part 1: Installing Nginx

Tanvir Rahman — Mon, 05 Feb 2024 17:02:18 +0000

Downloading and Installing Nginx from Source

Nginx is a powerful and efficient web server thats widely used for hosting websites and applications. In this blog, well walk through the steps to download and install Nginx from source on a Linux system. This method is particularly useful when you need a specific version of Nginx or want to customize the build.

Prerequisites

Before beginning, ensure you have a Linux system with internet access and root privileges. Then update the package list and install net-tools

sudo apt-get update
sudo apt-get install net-tools

Step 1: Downloading the Nginx Package

First, download the desired version of Nginx. As of this writing, lets use version 1.22.1. You can download it using the wget command:

wget https://nginx.org/download/nginx-1.22.1.tar.gz

This command fetches the tar.gz file containing the Nginx source code.

Step 2: Extracting the File

Once the download is complete, extract the file and navigate to the extracted directory:

tar -zxvf nginx-1.22.1.tar.gz
cd nginx-1.22.1

Step 3: Installing Build Essentials

Before compiling Nginx, you need to install build-essential, a package that contains references to numerous tools and utilities required for building software:

apt-get install build-essential

Then, initiate the configuration process:

./configure

Step 4: Installing Dependencies

Nginx has several dependencies that need to be installed for it to function correctly. These include libraries for regular expression processing, encryption, and compression. Install these using the following command:

apt-get install libpcre3 libpcre3-dev zlib1g zlib1g-dev libssl-dev

Step 5: Final Configuration and Installation

After installing the dependencies, run the configure script again. This time, it should complete without any errors:

./configure

Step 6: Additional Configuration for Nginx

This command configures Nginx with specific paths and options:

./configure --sbin-path=/usr/bin/nginx \
            --conf-path=/etc/nginx/nginx.conf \
            --error-log-path=/var/log/nginx/error.log \
            --http-log-path=/var/log/nginx/access.log \
            --with-pcre \
            --pid-path=/var/run/nginx.pid \
            --with-http_ssl_module

Explanation of the configuration options:
--sbin-path: Specifies the path where the nginx executable will be located.
--conf-path: Sets the path for the nginx configuration file.
--error-log-path: Designates the location for the nginx error log.
--http-log-path: Specifies the path for the nginx access log.
--with-pcre: Enables usage of Perl Compatible Regular Expressions for location matching.
--pid-path: Sets the path of the nginx PID file.
--with-http_ssl_module: This option enables the SSL module, which is essential for handling HTTPS traffic.

This configuration provides a structured approach to organizing the Nginx installation.
It helps in managing logs, configuration, and executable files in a standard and accessible manner.

Step 7: Compiling and Installing Nginx

make install

This sequence compiles the Nginx source code and then installs it on your system.

Step 8: Verifying the Installation

Listing the contents of the Nginx configuration directory

ls -l /etc/nginx

This command will show the files in the '/etc/nginx' directory, ensuring that Nginx is installed correctly.

Step 9: Checking the Nginx Version and Configuration

nginx -V

Step 10: Running Nginx

To stop Nginx run

sudo /usr/bin/nginx -s stop

Configuring and Launching an EC2 Instance: A Comprehensive Guide with Detailed Commands

Tanvir Rahman — Sat, 25 Nov 2023 10:53:43 +0000

Launching an Amazon EC2 (Elastic Compute Cloud) instance is a foundational skill for leveraging the AWS cloud platform. This guide will walk you through the steps required to set up, connect to, and manage an EC2 instance, including specific commands for accessing instance metadata.

Initial Setup: Naming, Tagging, and Image Selection

Name and Tags Section: Assign a name to your EC2 instance for easy identification. Tags, defined as Key/Value pairs, aren't mandatory but highly recommended for efficient organization in production environments.

Application and OS Images: Select 'Amazon Linux' under Quick Start options. Amazon provides a range of AMIs, featuring various versions of Linux and Windows, each pre-configured with essential software packages.

Selecting Instance Type and Key Pair Configuration

Instance Type Selection: Review the list of available instance types, focusing on their hardware resources like CPU and memory.

Key Pair Creation: Generate a new key pair for secure SSH access. Download the .pem file containing your private key after maintaining the default settings.

Network and Storage Settings

Network Settings: Ensure SSH traffic is allowed from 'Anywhere' in the Security Groups section.

Configure Storage: Stick with the default 8 GiB gp3 root volume unless your application demands otherwise.

Launching and Monitoring the Instance

Launching the Instance: Launch your instance after reviewing your configurations. Monitor its deployment on the EC2 console's Instances screen.

Establishing an SSH Connection

Set the correct permissions for your key file: chmod 400 /path/to/your/keypair.pem.
Connect to your instance: ssh -i /path/to/your/keypair.pem ec2-user@server-ip, where server-ip is your instance's Public IP.

Accessing EC2 Instance Metadata

Instance metadata provides valuable information about your running instance.

Creating a Token: Generate a token for secure metadata access: TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 3600").

Listing Metadata: Retrieve all instance metadata with the command: curl -w "\n" -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/.
Extracting Specific Metadata: Use the following commands to get detailed information:
- Security groups: curl -w "\n" -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/security-groups
- AMI ID: curl -w "\n" -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/ami-id
- Hostname: curl -w "\n" -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/hostname
- Instance ID: curl -w "\n" -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/instance-id
- Instance type: curl -w "\n" -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/instance-type

Instance Termination Process

Learn to terminate your instance via the AWS console, an important step for managing costs and resources.

In the AWS console, navigate to 'Instances', select your instance, and choose 'Terminate instance' from the 'Instance State' dropdown.

This guide provides a thorough understanding of launching and managing an EC2 instance, including detailed commands for accessing instance metadata, ensuring you are well-equipped to utilize this powerful AWS service.

DevContainer and Makefile: A Duo for Simplified GCP Workflows

Tanvir Rahman — Sun, 27 Aug 2023 16:12:45 +0000

Setting up a development environment can sometimes feel like orchestrating a symphony with disparate instruments, given how many moving parts are involved. However, DevContainers within Visual Studio Code (VSCode) are here to change that tune. Below, let's explore how to streamline this process and create a seamless, shareable, and consistent development environment.

Step 1: Installing the DevContainer Extension in VSCode

Before diving into the code, ensure that the DevContainer extension is installed within your VSCode setup. This extension is your gateway to crafting isolated and reproducible development environments.

Step 2: Setting Up the Directory Structure

Create a new folder which will act as the root of your project.
Inside this folder, create another directory named .devcontainer.
Within .devcontainer, create two essential files: devcontainer.json and Dockerfile.

Step 3: Crafting the Dockerfile

The Dockerfile is like a recipe, dictating the ingredients and steps required to create your development environment. Below is an example using Ubuntu as the base image and installing Google Cloud SDK among other packages.

# Use an official Ubuntu as a parent image
FROM ubuntu:latest

# Set environment variables to non-interactive (this prevents some prompts)
ENV DEBIAN_FRONTEND=non-interactive

# Run package updates, install packages, and clean up
RUN apt-get update -y \
    # Install common packages and openssh-client
    && apt-get install -y \
        ca-certificates \
        curl \
        gnupg \
        openssh-client \
        python3 \
        make \
    # Add Google Cloud SDK repo and its key
    && echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list \
    && curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg add - \
    && apt-get update \
    && apt-get install -y google-cloud-sdk \
    # Remove unnecessary files
    && rm -rf /var/lib/apt/lists/* \
    # Generate SSH Key
    && ssh-keygen -t rsa -b 4096 -f /root/.ssh/id_rsa -N ""

WORKDIR /automation
COPY . /automation/

Step 4: Configuring devcontainer.json

This JSON file is the control center for the DevContainer. It specifies how the container should behave when we open VSCode inside it.

// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
{
    "name": "Code space configuration",
    "build": {
        "dockerfile": "Dockerfile",
        // "args": { 
        //  // Update 'VARIANT' to pick a .NET Core version: 2.1, 3.1, 5.0
        //  "VARIANT": "5.0",
        //  // Options
        //  "INSTALL_NODE": "true",
        //  "NODE_VERSION": "lts/*",
        //  "INSTALL_AZURE_CLI": "true",
        //  "INSTALL_TERRAFORM": "true",
        //  "TERRAFORM_FILE":"terraform_1.0.4_linux_amd64.zip",
        //  "TERRAFORM_VERSION":"https://releases.hashicorp.com/terraform/1.0.4/terraform_1.0.4_linux_amd64.zip"
        // }
    },

    // Set *default* container specific settings.json values on container create.
    "settings": {
    "terminal.integrated.shell.linux": "/bin/zsh"
  },

    // Add the IDs of extensions you want installed when the container is created.
    "extensions": [
        // "ms-dotnettools.csharp",
        // "hashicorp.terraform",
        // "ms-vscode.azure-account",
        // "ms-azuretools.vscode-azurefunctions",
        // "ms-azuretools.vscode-azureresourcegroups",
        // "github.copilot",
        // "ms-mssql.mssql",
        // "hookyqr.beautify",
        // "ms-azuretools.vscode-docker"
    ],

    // Use 'forwardPorts' to make a list of ports inside the container available locally.
    // "forwardPorts": [5000, 5001],

    // Use 'postCreateCommand' to run commands after the container is created.
    "postCreateCommand": "uname -a",

    // Comment out connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
    "remoteUser": "root"
}

Step 5: Initiating the DevContainer

Run the Dev Containers: Reopen in Container command from VSCode. If everything is set up correctly, you'll land inside the container, where you can check the Google Cloud SDK version, among other things.

[If you encounter issues, you can debug by running

docker run -it ubuntu:latest /bin/bash

and manually inputting the commands from your Dockerfile for analysis.]

Step 6: Automating Tasks with Makefile

You can further enhance your DevContainer experience by utilizing a Makefile. This file serves as a task runner that can execute a series of commands for you. Here's a sample that demonstrates printing a simple "hello":

greet := hello
echo:
    @ echo ${greet}

Step 7: Integrating Google Cloud Platform

To start working with Google Cloud Platform (GCP), obtain service credentials and place them in a auth.json file. Then, update your Makefile to interact with GCP resources:

# Makefile for Google Cloud Platform (GCP)

# Variables
KEY_FILE := auth.json
PROJECT_ID := playground-s-11-b2eb9fb3
VPC_NAME := vpc-tr
SUBNET_NAME := subnet-tr
REGION := us-central1
COMPUTE_ZONE := us-central1-a
CIDR_BLOCK := 10.10.0.0/16
VM_NAME := vm-tr
VM_TYPE := n1-standard-1
VM_IMAGE_FAMILY := debian-11
VM_IMAGE_PROJECT := debian-cloud
INTERNAL_IP_RANGE := 31.43.23.23

SSH_KEY_PATH := /root/.ssh/id_rsa.pub
# Read the SSH public key into a variable
SSH_PUBLIC_KEY := root $(shell cat ${SSH_KEY_PATH})

# Default rule
default: gcloud_login gcloud_set_project gcloud_list_networks

# Rule for logging into GCP
gcloud_login:
    @ gcloud auth activate-service-account --key-file=${KEY_FILE}

# Rule for setting GCP project
gcloud_set_project:
    @ gcloud config set project ${PROJECT_ID}

# Rule for setting GCP compute zone
gcloud_set_zone:
    @ gcloud config set compute/zone ${COMPUTE_ZONE}

# Rule for listing GCP networks
gcloud_list_networks:
    @ gcloud compute networks list

# Rule for creating a custom VPC
create_vpc:
    @ gcloud compute networks create ${VPC_NAME} --subnet-mode=custom

# Rule for creating a custom VPC with subnets
create_vpc_with_subnets: create_vpc
    @ gcloud compute networks subnets create ${SUBNET_NAME} --network=${VPC_NAME} --region=${REGION} --range=${CIDR_BLOCK}

# Rule for deleting a VPC
delete_vpc:
    @ gcloud compute networks delete ${VPC_NAME} --quiet

# Rule for creating firewall rule for internal traffic
create_firewall_internal:
    @ gcloud compute firewall-rules create allow-internal --network ${VPC_NAME} --allow tcp,udp,icmp --source-ranges ${INTERNAL_IP_RANGE}

# Rule for creating firewall rule to allow all IP ranges
create_firewall_allow_all:
    @ gcloud compute firewall-rules create allow-all-${VPC_NAME} --network ${VPC_NAME} --allow tcp,udp,icmp --source-ranges 0.0.0.0/0

# Rule for creating firewall rule for SSH, RDP, and ICMP
create_firewall_ssh_rdp_icmp:
    @ echo "Creating firewall rules for VPC: ${VPC_NAME}"
    @ gcloud compute firewall-rules create allow-ssh-rdp-icmp-${VPC_NAME} --network ${VPC_NAME} --allow tcp:22,tcp:3389,icmp

# Rule for describing a specific GCP network
describe_network:
    @ gcloud compute networks describe ${VPC_NAME}

# Rule for listing subnets in the VPC
gcloud_list_subnets:
    @ gcloud compute networks subnets list --filter="network:${VPC_NAME}"

# Rule for creating a VM instance
gcloud_create_vm:
    @ gcloud compute instances create ${VM_NAME} \
        --zone=${COMPUTE_ZONE} \
        --machine-type=${VM_TYPE} \
        --image-family=${VM_IMAGE_FAMILY} \
        --image-project=${VM_IMAGE_PROJECT} \
        --boot-disk-size=10GB \
        --metadata=ssh-keys="$(SSH_PUBLIC_KEY)"

# Rule for deleting a VM instance
gcloud_delete_vm:
    @ gcloud compute instances delete ${VM_NAME} --zone=${COMPUTE_ZONE} --quiet

# Rule for deploying a GCP function
deploy_function:
    @ gcloud functions deploy FUNCTION_NAME --runtime RUNTIME --trigger-http --allow-unauthenticated

# Phony targets
.PHONY: gcloud_login gcloud_set_project gcloud_set_zone gcloud_list_networks create_vpc delete_vpc create_firewall_internal create_firewall_allow_all create_firewall_ssh_rdp_icmp describe_network create_vpc_with_subnets gcloud_list_subnets gcloud_create_vm gcloud_delete_vm deploy_function

Running Any File

You're not limited to specific languages or files. The DevContainer allows you to run any script, be it Shell, Python, or even a simple HTTP server.

//hello.sh
echo "hello"

// Dockerfile
FROM nginx

//makefile
image_name:=tanvir/web
tag:=v1.1
container_name:=web
build:
    @ docker build -t ${image_name}:${tag} .
run:
    @ docker run --name ${container_name} -d -p 8080:80 ${image_name}:${tag}
b:
    @ chmod +x hello.sh 
    @ ./hello.sh

files:=hello.sh hello1.sh hello2.sh
chmod:
    @ chmod +x ${files}
run1:
    @ ./hello.sh
    @ ./hello1.sh

With that, you have a modular, version-controlled, and fully functional development environment. The best part is that all of this is contained within a DevContainer, ensuring that each team member can replicate the exact environment, thereby eliminating the "it works on my machine" syndrome. Happy coding!

Supercharge Your Container Networking: Seamless Host Communication with VxLAN and Docker

Tanvir Rahman — Sat, 15 Jul 2023 05:19:42 +0000

In this hands-on demo, we will explore how to set up communication between two hosts using Virtual Extensible LAN (VxLAN) and Docker. The goal is to create a VxLAN overlay network tunnel between the hosts' containers, allowing them to communicate with each other. Let's dive into the steps involved in this process.

What are we going to cover in this hands-on demo?

Creating two VMs and installing Docker: We will use two virtual machines (VMs) and install Docker to run the containers.
Creating separate subnets and assigning static IP addresses: To simplify the setup, we'll create separate subnets for each VM and assign static IP addresses.
Creating a VxLAN bridge: We'll utilize the Linux "ip link vxlan" feature to create a VxLAN bridge.
Binding the VxLAN to the Docker bridge: We'll bind the VxLAN to the Docker bridge to establish the tunnel.
Verifying communication between containers: Finally, we'll test the communication between containers on different hosts.

Let's get started!
To facilitate effective communication between hosts, we need to deploy two VMs using any hypervisor or virtualization technology. It is crucial to ensure that both VMs are connected to the same network.

Step 1: Creating the VMs

We begin by creating two Lubuntu VMs using UTM / Multipass. These VMs will serve as the hosts for our containers.

Step 2: Installing necessary tools and Docker

In this step, we will install several necessary tools and Docker on our Ubuntu VMs. Here's a brief overview of the tools we are going to install:

net-tools: This package includes important network management tools such as ifconfig, netstat, route, etc.
iputils-ping: This package provides the ping command, which is used to test the reachability of a network host.
bridge-utils: This package provides utilities for configuring Ethernet bridging on Linux.

To install these tools and Docker, execute the following commands:

apt-get update
apt-get install net-tools
apt-get install iputils-ping
apt-get install bridge-utils
apt-get install -y docker.io

Additionally, if the IP address of one of the VMs is the same as the other, you'll need to modify it to avoid conflicts. For example, you can change the IP address of one of the VMs to 192.168.64.7/24 using the following commands:

ifconfig eth0 192.168.64.7/24
route add default gw 192.168.64.1 eth0

To configure DNS resolution, open the resolv.conf file:

nano /etc/resolv.conf

Delete the existing content and add the following lines:

nameserver 8.8.8.8
nameserver 8.8.4.4

Now, let's proceed to set up the containers and their communication.

Step 3: Running Docker Containers on the VxLAN Network
Now for host1(amicable-hyena),

# create a separate docker bridge network 
docker network create --subnet 172.18.0.0/16 vxlan-net

# list all networks in docker
docker network ls
# The output should include the newly created vxlan-net network.
NETWORK ID     NAME        DRIVER    SCOPE
53be5ce8e682   bridge      bridge    local
757eb3a14b73   host        host      local
c1c0e4f01fa6   none        null      local
08bdd2dc3a82   vxlan-net   bridge    local

# Check interfaces
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 36:db:7c:3f:11:58 brd ff:ff:ff:ff:ff:ff
    inet 192.168.64.9/24 brd 192.168.64.255 scope global dynamic enp0s2
       valid_lft 85395sec preferred_lft 85395sec
    inet6 fde4:cfbc:2cca:cd78:34db:7cff:fe3f:1158/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 2591923sec preferred_lft 604723sec
    inet6 fe80::34db:7cff:fe3f:1158/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:aa:59:e4:e3 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
4: br-08bdd2dc3a82: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:4b:42:67:6b brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-08bdd2dc3a82
       valid_lft forever preferred_lft forever

For Host 2(affluent-toad),

docker network create --subnet 172.18.0.0/16 vxlan-net

docker network ls
cdc5ceed0de2   bridge      bridge    local
11ed966e63df   host        host      local
5ea8856ad30c   none        null      local
55316818ea9f   vxlan-net   bridge    local

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether f2:5f:8d:d3:9f:c9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.64.10/24 brd 192.168.64.255 scope global dynamic enp0s2
       valid_lft 85276sec preferred_lft 85276sec
    inet6 fde4:cfbc:2cca:cd78:f05f:8dff:fed3:9fc9/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 2591908sec preferred_lft 604708sec
    inet6 fe80::f05f:8dff:fed3:9fc9/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:b3:cb:9a:b9 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
4: br-55316818ea9f: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:2c:83:cd:df brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-55316818ea9f
       valid_lft forever preferred_lft forever

Step 4: Run docker container
Let's run docker container on top of newly created docker bridge network and try to ping docker bridge

For Host 1,

# running alpine container with "sleep 3000" and a static ip
docker run -d --net vxlan-net --ip 172.18.0.11 alpine sleep 3000

# check the container running or not
docker ps
CONTAINER ID   IMAGE     COMMAND        CREATED              STATUS              PORTS     NAMES
d2b96eddea7a   alpine    "sleep 3000"   About a minute ago   Up About a minute             funny_jepsen

# check the IPAddress to make sure that the ip assigned properly
docker inspect d2 | grep IPAddress
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "172.18.0.11",

# ping the docker bridge ip to see whether the traffic can pass
ping 172.18.0.1 -c 2
PING 172.18.0.1 (172.18.0.1) 56(84) bytes of data.
64 bytes from 172.18.0.1: icmp_seq=1 ttl=64 time=0.166 ms
64 bytes from 172.18.0.1: icmp_seq=2 ttl=64 time=0.050 ms

--- 172.18.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1018ms
rtt min/avg/max/mdev = 0.050/0.108/0.166/0.058 ms

For Host 2,

docker run -d --net vxlan-net --ip 172.18.0.12 alpine sleep 3000

docker ps
CONTAINER ID   IMAGE     COMMAND        CREATED         STATUS         PORTS     NAMES
9d2e17598b8b   alpine    "sleep 3000"   3 seconds ago   Up 2 seconds             eloquent_black

docker inspect 9d | grep IPAddress
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "172.18.0.12",

ping 172.18.0.1 -c 2

PING 172.18.0.1 (172.18.0.1) 56(84) bytes of data.
64 bytes from 172.18.0.1: icmp_seq=1 ttl=64 time=0.193 ms
64 bytes from 172.18.0.1: icmp_seq=2 ttl=64 time=0.067 ms

--- 172.18.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1017ms
rtt min/avg/max/mdev = 0.067/0.130/0.193/0.063 ms

Step 5: Testing Communication
Now, let's access one of the running containers and test communication between hosts. Note that containers within the same host should communicate, but container-to-container communication between hosts will fail at this stage because there is no tunnel or anything to carry the traffic.

docker exec -it d2 sh

apk update
apk add net-tools
apk add iputils-ping

ping 172.18.0.12 -c 2
ping: -c: Try again

ping -c 2 192.168.64.10
PING 192.168.64.10 (192.168.64.10) 56(84) bytes of data.
64 bytes from 192.168.64.10: icmp_seq=1 ttl=63 time=3.93 ms
64 bytes from 192.168.64.10: icmp_seq=2 ttl=63 time=0.967 ms

--- 192.168.64.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.967/2.447/3.928/1.480 ms

Step 6: Creating the VxLAN Tunnel
To establish communication between the containers, we need to create a VxLAN tunnel and attach it to the Docker bridge. Make sure the VNI (Virtual Network Identifier) ID is the same for both hosts.

For Host 1,

# check the bridges list on the hosts
brctl show

bridge name bridge id       STP enabled interfaces
br-08bdd2dc3a82     8000.02424b42676b   no
docker0     8000.0242aa59e4e3   no

# create a vxlan
# 'vxlan-demo' is the name of the interface, type should be vxlan
# VNI ID is 100
# dstport should be 4789 which a udp standard port for vxlan communication
# 192.168.64.10  is the ip of another host
ip link add vxlan-demo type vxlan id 100 remote 192.168.64.10 dstport 4789 dev enp0s2

# check interface list if the vxlan interface created
ip a | grep vxla

7: vxlan-demo: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000

# make the interface up
ip link set vxlan-demo up

# now attach the newly created vxlan interface to the docker bridge we created
brctl addif br-08bdd2dc3a82 vxlan-demo

# check the route to ensure everything is okay. here '172.18.0.0' part is our concern part.
route -n

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.64.1    0.0.0.0         UG    100    0        0 enp0s2
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-08bdd2dc3a82
192.168.64.0    0.0.0.0         255.255.255.0   U     0      0        0 enp0s2
192.168.64.1    0.0.0.0         255.255.255.255 UH    100    0        0 enp0s2

For Host 2,

brctl show

bridge name bridge id       STP enabled interfaces
br-55316818ea9f     8000.02422c83cddf   no
docker0     8000.0242b3cb9ab9

ip link add vxlan-demo type vxlan id 100 remote 192.168.64.9 dstport 4789 dev enp0s2

ip a | grep vxlan

7: vxlan-demo: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000

ip link set vxlan-demo up
brctl addif br-55316818ea9f vxlan-demo

route -n

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.64.1    0.0.0.0         UG    100    0        0 enp0s2
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-55316818ea9f
192.168.64.0    0.0.0.0         255.255.255.0   U     0      0        0 enp0s2
192.168.64.1    0.0.0.0         255.255.255.255 UH    100    0        0 enp0s2

Step 7: Testing Communication between Containers
Now that the VxLAN overlay network tunnel has been created, let's test the communication between the containers on different hosts.

docker exec -it a9 bash
ping 192.168.64.10 -c 2

You can find result of ping from below images. [Note: Re ran docker container in both host as previous container already ran for 3000 seconds]

You should see successful ping responses, indicating that communication between the containers on different hosts is now established.

Congratulations! You have successfully set up communication between hosts using VxLAN and Docker.

Reference:
The demonstration code and steps mentioned in this blog post were adapted from the following GitHub repository: vxlan-docker-hands-on

Feel free to explore the repository for more in-depth details and examples.

Thank you for reading this blog post, and I hope you found it informative and useful.

Simplifying Underlay - Overlay Networks, VxLAN and Packet Walk: A Journey of Networks

Tanvir Rahman — Thu, 13 Jul 2023 20:09:39 +0000

In the intricate world of networking, there are two crucial concepts: Underlay Networks and Overlay Networks. These concepts play a vital role in establishing efficient and secure data transmission. Let's unravel these concepts and explore their significance in our interconnected world.

What is an Underlay Network?

An Underlay Network forms the physical infrastructure on which overlay networks are built. It acts as the foundational layer responsible for delivering data packets across networks. Underlay networks operate at different layers of the OSI model, such as Layer 2 (Data Link Layer) or Layer 3 (Network Layer). For example, Ethernet-based Layer 2 underlay networks use Virtual Local Area Networks (VLANs) for segmentation. The Internet itself serves as a familiar example of a Layer 3 underlay network, providing the foundation for various overlay networks to operate.

Delving into Overlay Networks

Overlay Networks are virtual networks constructed on top of the underlying network infrastructure, i.e., the underlay network. They abstract the physical network, creating a virtualized environment where overlay nodes (e.g., routers) communicate over the physical network. Overlay networks implement network virtualization concepts and employ Layer 2 and Layer 3 tunneling encapsulation protocols like VXLAN, GRE, and IPSec. These protocols enable the transportation of data packets across the overlay network, bridging the gap between different network segments or sites that may not natively support direct communication.

Making the Connection Between Underlay and Overlay Networks:

To better grasp the relationship between underlay and overlay networks, let's use a relatable example. Imagine embarking on a road trip from New York to Los Angeles. The physical highways, streets, bridges, and tunnels that you traverse represent the underlay network—the tangible pathways guiding your journey.

Now, consider relying on a GPS navigation system throughout your trip. The GPS system utilizes the physical road infrastructure (the underlay network) but creates a virtual pathway (the overlay network) specifically guiding you from your starting point to your destination. The GPS system is not concerned with the physical attributes of the roads; its focus is on virtually connecting your origin and endpoint.

In the context of computer networks:

The Underlay Network resembles the physical highways and roads comprising physical routers, switches, and cables. Data packets use this infrastructure to travel across the network.
The Overlay Network acts like the GPS system. It is a virtual network built on top of the underlay network, providing a specific, optimized path for data packets to travel from source to destination.

To summarize, the Underlay Network represents the physical infrastructure facilitating data packet movement, while the Overlay Network creates virtual pathways within the underlying physical network to optimize data transmission.

VxLAN: Extending Layer 2 Networks Beyond Limits

VxLAN, short for Virtual Extensible LAN, revolutionizes network virtualization by extending Layer 2 networks across Layer 3 infrastructures. This technology proves invaluable in dynamic data centers with multi-tenant Virtual Machines (VMs). VxLAN segments, or overlays, establish virtual communication paths, enabling VMs within the same segment to communicate seamlessly, even across different physical networks. Each segment is identified by a unique VNI (VxLAN Network Identifier), allowing for up to 16 million segments within the same administrative domain.

VNI: Unleashing Network Virtualization Potential

The VxLAN Network Identifier (VNI) unlocks network virtualization capabilities by providing a vast address space. Unlike VLANs, VxLAN's 24-bit ID space allows for approximately 16 million unique VNIs. Each VNI acts as a unique identifier, similar to a postcode, ensuring data reaches the intended destination accurately. This extensive range of VNIs guarantees uniqueness across the entire network, enhancing flexibility and scalability.

VTEP: Orchestrating Traffic Encapsulation

VxLAN traffic encapsulation and direction are handled by VTEP (VxLAN Tunnel End Point), acting as the orchestrator. VTEPs create stateless tunnels across the network, encapsulating traffic from the source switch and delivering it to the destination switch. Equipped with an IP address in the underlay network and associated with one or more VNIs, VTEPs perform the intricate task of adding and removing headers to ensure seamless frame delivery across the network.

Making Sense of VxLAN, VNI, and VTEP with an Engaging Analogy

Imagine you're sending letters to friends residing in different cities. Your goal is to make it seem like all these letters originate from your home city. In this scenario, VxLAN acts as a clever post office system that enables precisely that!

VxLAN: The Clever Post Office

VxLAN, the virtual post office, empowers your computer (representing your home city) to send data to different parts of a network (different cities) as if they were part of your local network. This technology proves especially useful in large data centers where numerous computers (or VMs) need to communicate as if they were in the same location, regardless of their physical separation.

VNI: The Unique Postcodes

Each letter (or data piece) requires a unique postcode to reach the correct friend (or VM). The VNI or VxLAN Network Identifier, fulfills this role. Operating as a unique identifier, similar to a postcode, the VNI ensures data reaches its intended destination accurately. With VxLAN, you have the capability to employ up to 16 million unique postcodes, far surpassing the limitations of traditional VLAN systems.

VTEP: The Reliable Postman

The diligent postman, VTEP (VxLAN Tunnel End Point), encapsulates your letter (data) into an envelope (encapsulation) adorned with the correct address (IP and UDP headers). At the recipient's end,

the envelope is opened (decapsulation), and the letter is safely delivered to the intended friend (the corresponding VM).

In essence, VxLAN serves as our clever post office system, facilitating the seamless delivery of data to different friends (VMs) residing in various cities (networks) while maintaining the illusion that all letters originate from your home city (your local network). Each letter receives a unique postcode (VNI), and VTEP acts as the reliable postman, ensuring accurate delivery.

Understanding Packet Walk in a VxLAN Network: A Letter's Journey

Let's simplify it using an analogy: the journey of a letter from one city to another.

Step 1: Starting the Journey

The journey commences when a letter (data) arrives at a switch, acting as our post office. The letter originates from a host (home) and arrives through an untagged access port (regular pathway). The post office assigns an area code (VLAN) to the letter.

Step 2: Deciding the Destination

The post office determines the letter's destination is a remote post office (switch) located in another city (location). This remote post office is connected to the local post office through an array of roads (an IP network).

Step 3: Preparing for the Journey

The letter's area code (VLAN) is associated with a specific mailbox (VNI). To prepare the letter for its journey, it is placed in a dedicated envelope (VxLAN header applied). The local post office (VTEP) wraps the letter in additional envelopes (encapsulation) comprising UDP and IP headers. The letter sets off on its journey along the roads (IP network).

Step 4: Arriving at the Destination

Upon reaching the remote city (remote switch), the local post office receives the letter and removes the additional envelopes (decapsulation). The original letter, complete with its area code (a regular layer-2 frame with a VLAN ID), remains.

Step 5: Delivery to the Recipient

The remote post office selects an egress port (destination) based on the recipient's address (normal MAC lookups). The letter proceeds on its final leg of the journey, reaching the intended recipient just as expected.

In summary, VxLAN technology enables our "letter" to travel seamlessly from one "home" to another, across cities, while maintaining the illusion of a neighborhood environment.

Thank you for joining us on this exploration of Underlay and Overlay Networks, VxLAN, and the Packet Walk. These concepts are essential in building efficient and secure networks. By understanding the roles of underlay and overlay networks, the power of VxLAN technology, and the journey of data packets, we gain valuable insights into the world of networking. Keep exploring and embracing the possibilities that networking offers in our interconnected world. Stay connected and informed!

References:

Unveiling the Mysteries of Network Management in Linux

Tanvir Rahman — Thu, 22 Jun 2023 19:21:21 +0000

To successfully navigate the labyrinth of network configuration in a Linux environment, we must first ensure that our system is appropriately primed. This preparatory stage involves keeping our software up-to-date and confirming the presence of essential networking utilities. Here's how we do it:

sudo apt update: This command refreshes our local index of software packages, paving the way for us to access the most recent versions.
sudo apt upgrade -y: An execution of this command results in an upgrade of all the installed software on our system. The '-y' flag automatically approves any prompts that might arise during the process.
sudo apt install iproute2: This command sets about installing 'iproute2', a bundle of essential utilities for handling TCP/IP networking and traffic control in Linux.
sudo apt install net-tools: By running this command, we are introducing 'net-tools' into our system. This package provides commands like 'ifconfig' that are instrumental in configuring network interfaces.

With these preparatory steps complete, we've established a sturdy foundation that will support our subsequent foray into network exploration and manipulation.

Mastering Network Configuration

Illustration

Building Our Network Bridge

Our first order of business is to erect a new network bridge, which we'll christen 'v-bridge'. The subsequent commands breathe life into 'v-bridge', sets it in an active (UP) state, and assigns it an IP address:

ip link add dev v-bridge type bridge
ip link set v-bridge up
ip addr add 192.168.0.1/24 dev v-bridge
ip addr show dev v-bridge

Carving Out Network Namespaces

Next, we shift our attention towards creating three distinct network namespaces, appropriately dubbed "red," "green," and "blue":

ip netns add red
ip netns add green
ip netns add blue

We can verify the successful creation of our namespaces by executing ip netns list.

Crafting Virtual Ethernet Interfaces

Our next stride involves the creation of virtual Ethernet (veth) pairs. These pairs function as a conduit, allowing seamless network communication between two endpoints:

ip link add veth-red-ns type veth peer name veth-red-br
ip link add veth-green-ns type veth peer name veth-green-br
ip link add veth-blue-ns type veth peer name veth-blue-br

Connecting Virtual Ethernet Interfaces

Our freshly minted veth interfaces are then linked to their corresponding network namespaces and our primary 'v-bridge':

ip link set dev veth-red-ns netns red
ip link set dev veth-green-ns netns green
ip link set dev veth-blue-ns netns blue

and

ip link set dev veth-red-br master v-bridge
ip link set dev veth-green-br master v-bridge
ip link set dev veth-blue-br master v-bridge

To activate these interfaces and prepare them for network communication, we execute:

ip link set dev veth-red-br up
ip link set dev veth-green-br up
ip link set dev veth-blue-br up

and

ip netns exec red ip link set dev veth-red-ns up


ip netns exec green ip link set dev veth-green-ns up
ip netns exec blue ip link set dev veth-blue-ns up

Assigning IP Addresses and Default Routes in Namespaces

Now we dive into the network namespaces to configure IP addresses and default routes for our veth interfaces:

ip netns exec red ip address add 192.168.0.2/24 dev veth-red-ns
ip netns exec red ip route add default via 192.168.0.1

and similar commands for the "green" and "blue" namespaces.

We've now configured our network namespaces, laying the groundwork for network communication and establishing a common gateway (192.168.0.1/24) for outbound traffic.

We can confirm inter-namespace communication by pinging one namespace from another, as shown below:

ip netns exec red ping -c 2 192.168.0.4

Enabling Internet Connectivity

Activating IP Forwarding

We start this stage by activating IP forwarding on our system, accomplished by setting the 'net.ipv4.ip_forward' sysctl parameter to 1: sysctl -w net.ipv4.ip_forward=1.

Configuring NAT and Firewall Rules

Next, we employ iptables to configure NAT. This allows our network namespaces to access the internet via the "enp0s2" interface: iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -o enp0s2 -j MASQUERADE.

The snapshot below provides a glimpse into the NAT configuration process:

We can verify internet connectivity by initiating a ping from any network namespace to an external IP address. An example follows: ip netns exec red ping -c 2 8.8.8.8.

The successful ping to an external IP is displayed below:

In Conclusion

In this detailed guide, we've navigated the complexities of managing network configuration in a Linux environment. We've delved into creating network namespaces and virtual Ethernet pairs, connecting them via a network bridge, assigning IP addresses and default routes within namespaces, and establishing communication between namespaces. Further, we've covered the enabling of IP forwarding, NAT configuration, and firewall rule setup to allow internet access to our network namespaces.

Should you want to dive deeper into these subjects or seek professional networking, I'm always open to stimulating discussions. Feel free to connect with me on LinkedIn.

Exploring Linux: Kernel Space, User Space, Namespaces and Network Chaining Unveiled

Tanvir Rahman — Thu, 22 Jun 2023 04:25:43 +0000

In the sprawling world of Linux, terms like kernel space, user space and namespaces are often bandied about. These concepts serve as the backbone of how a Linux system operates, making them indispensable knowledge for any system administrator or developer. Today, we're going to delve into what these terms mean and their importance in the Linux ecosystem

The Realm of Kernel Space and User Space

A Linux operating system's memory space is divided into two main parts – kernel space and user space. This segregation is instrumental in maintaining system performance, security, and efficiency.

Kernel Space: Kernel space is the privileged realm where the heart of the operating system, the kernel, resides. Responsible for interacting directly with the hardware, controlling memory, and managing system services, the kernel is a critical part of the OS that requires a secure, isolated space - hence, the kernel space.
User Space: User space is the unprivileged territory where user mode applications get to play. Each application gets its private memory space, ensuring that it cannot interfere with others or the kernel. This separation safeguards system integrity and facilitates efficient task management.

Decoding Linux Namespaces

Linux namespaces are a powerful feature of the Linux kernel that essentially isolates different system resources for different sets of processes. Each namespace gives processes an illusion of having their unique system resources.

For instance, the PID namespace provides an isolated environment where process IDs are unique within the namespace. Similarly, the network namespace provides independent network devices, IP routing tables, and port numbers. The mount namespace isolates the set of filesystem mount points, giving processes their unique view of the filesystem hierarchy.

This resource isolation makes processes feel like they're running on their own private machine, paving the way for lightweight virtualization solutions such as Docker.

The Intricate Relations

The division between kernel space and user space is a fundamental design aspect of Linux, ensuring system stability and security. User applications running in user space interact with hardware indirectly through the kernel, which operates in kernel space and has direct hardware access.

Linux namespaces, a feature of the Linux kernel, operate within kernel space. The kernel manages these namespaces, providing resource isolation and allocation. However, applications running in user space can reside within namespaces, offering them an isolated system view and mimicking the feeling of running on a private machine.

Navigating Through Prerouting, Postrouting, Forwarding, Input, and Output Chains

These terms are often used to describe various stages of processing within the Netfilter framework, which is part of the Linux kernel. Netfilter provides a means for packet filtering, network address [and port] translation (NAT/NAPT) and other packet mangling. It is the infrastructure that facilitates building network-facing services in the kernel and is used by the iptables tool.

Here is an explanation of each term, along with an example:

Prerouting: This is the very first chain that an incoming network packet will hit. At this point, any decisions about where to route the packet (i.e., to a local process or to another network interface for forwarding on) are yet to be made. This is also the stage where Destination NAT (DNAT) happens.
Example: Imagine you have a Linux machine set up as a router, and it receives a packet destined for an IP that the router has been configured to handle (via DNAT). The Prerouting chain would be used to translate the destination address to the appropriate internal IP.
Input: If the packet is intended for a local process (the destination IP is one of the server's own IP), it will go to the Input chain next. This is where you would typically place rules to handle packets destined for the server itself.
Example: If a packet arrives at the server that is destined for port 22 (SSH), you could have a rule in the Input chain to accept such packets.
Forward: If the packet is intended to be forwarded to another device (the destination IP is not any of the server's IP), it will be sent to the Forward chain next. This is where decisions are made about forwarding the packets to other networks.
Example: If a packet arrives at the router destined for a different subnet, the Forward chain is used to decide if the packet should be forwarded, and to which interface it should be sent.
Output: This is the first chain that packets coming from local processes will hit. Any packet that your server is sending out will go through this chain.
Example: If a local process, such as a web server, is sending out a packet in response to a request, it would go through the Output chain.
Postrouting: This is the last chain that a packet will hit before it leaves the server. This is typically where you would do Source NAT (SNAT), where you change the source address of outgoing packets so they appear to come from the router itself.
Example: Imagine you have a private network and a router that connects your network to the internet. When a packet is leaving your network, the Postrouting chain would be used to translate the source IP to the public IP of the router.

Well, that wraps up our deep-dive into the intricate world of Linux's Kernel Space, User Space, Namespaces and the all-important network chains. As we've seen, the kernel forms the heart of Linux, with namespaces and network chains acting as its vital arteries, ensuring everything runs smoothly and efficiently.

The beauty of Linux is that it gives us, the users, an unprecedented level of control over our systems. Understanding these fundamental concepts is the first step towards leveraging that control to create powerful, efficient and secure systems.

Remember, there's always more to learn in the fascinating world of Linux. So, keep exploring, keep tinkering and keep building. And of course, if you found this blog useful, do share it with your peers who might also benefit from it.

Until next time, happy coding!