DEV Community: John Doe

Automating Full-Stack Cloud Deployments Using Terraform and Ansible

John Doe — Tue, 03 Dec 2024 22:15:17 +0000

Infrastructure as Code (IaC) is the cornerstone of automation. With the right tools, it’s possible to manage cloud resources, deploy applications, and configure servers without needing to manually perform these tasks each time.

In this article, we will walk you through an end-to-end project involving Terraform and Ansible to provision cloud infrastructure, deploy application and monitoring stacks, and configure everything automatically. This project will provide a deep dive into Docker, Terraform, and Ansible, as well as how they work together to automate your infrastructure setup and deployment pipeline.

Overview of the Project

This project involves:

Building Docker images for a frontend and backend application.
Designing and provisioning cloud infrastructure using Terraform.
Using Terraform to trigger Ansible playbooks for deploying the application stack and setting up the monitoring stack.
Creating a reverse proxy using Traefik or Nginx for routing between services.
Configuring monitoring using Prometheus, Grafana, cAdvisor, and Loki.

The ultimate goal is to run a single Terraform command that provisions the infrastructure, deploys all services, and configures routing automatically.

Step 1: Building and Pushing Docker Images

Before we can deploy anything, we need Docker images for our frontend and backend applications. This step ensures that we have prebuilt images ready for deployment on any cloud instance.

Frontend Application

For the frontend, we use React to build a single-page application. The Dockerfile for the frontend might look like this:

#-----------------
# Building the app
#-----------------
FROM node:17-alpine AS build

WORKDIR /app/frontend

COPY package*.json ./

RUN npm install

COPY . .

ARG VITE_API_URL

ENV VITE_API_URL=$VITE_API_URL

RUN npm run build

#----------------
# Serving the app
#----------------
FROM nginx:alpine

COPY --from=build /app/frontend/dist /usr/share/nginx/html

COPY nginx.conf /etc/nginx/conf.d/default.conf

EXPOSE 80

CMD [ "nginx", "-g", "daemon off;" ]

Backend Application

The backend is built using FastAPI, which is a modern, fast web framework for building APIs with Python. The Dockerfile for the backend would look something like this:

FROM python:3.11 AS build

WORKDIR /app/backend

COPY pyproject.toml poetry.lock ./

RUN pip install poetry

RUN poetry config virtualenvs.create false && \
    poetry install --no-root

COPY . .

RUN pip install sqlalchemy sqlmodel alembic

ENV PYTHONPATH=/app/backend

RUN chmod +x /app/backend/prestart.sh

EXPOSE 8000

CMD ["sh", "-c", "./prestart.sh && uvicorn app.main:app --host 0.0.0.0 --port 8000"]

Once the Dockerfiles are in place, you build and push the images to Docker Hub (or any other container registry of your choice):

# Build the frontend image
docker build -t <your-username>/frontend:latest .

# Build the backend image
docker build -t <your-username>/backend:latest .

# Push the images to Docker Hub
docker push <your-username>/frontend:latest
docker push <your-username>/backend:latest

This will make the Docker images available for deployment later on when using Terraform and Ansible.

Step 2: Designing the Infrastructure

Before provisioning resources, it’s essential to define the architecture. In this project, the architecture involves:

A virtual machine (VM) that hosts the application stack (frontend, backend, and reverse proxy).
Networking to ensure the components can communicate with each other.
A reverse proxy (either Traefik or Nginx) to route traffic between services.
Prometheus, Grafana, cAdvisor, Loki, and Promtail to monitor the applications and collect logs.

Architecture Diagram

Here’s a simple architecture diagram to help visualize the components:

Step 3: Writing Terraform Configuration

Terraform is used to provision the infrastructure, such as the cloud instance (VM), and to trigger the execution of the Ansible playbook. The main.tf file will define the infrastructure, and it will include:

The creation of an AWS EC2 instance.
An Elastic IP (EIP) to give the instance a static public IP.
A security group that allows SSH access.
Outputs to show important information such as the public IP of the instance.

Here is a simple Terraform configuration for provisioning an AWS EC2 instance:

provider "aws" {
  region = "us-east-1"
}

resource "aws_instance" "app" {
  ami           = "ami-12345678"  # Replace with an appropriate AMI ID
  instance_type = "t2.micro"
  key_name      = "your-key-pair"

  tags = {
    Name = "App Instance"
  }

  associate_public_ip_address = true
}

resource "aws_eip" "dojo-eip" {
  instance = aws_instance.app.id
}

output "public_ip" {
  value = aws_instance.app.public_ip
}

Terraform and Ansible Integration

Terraform will use Ansible to configure the application after the EC2 instance is created. The null_resource and remote-exec provisioner are used to execute commands on the instance via SSH.

Here is the relevant Terraform configuration that integrates Ansible with Terraform:

resource "null_resource" "ansible" {
  provisioner "remote-exec" {
    connection {
      host        = aws_instance.app.public_ip
      user        = "ubuntu"
      private_key = file("/path/to/your/private-key.pem")
    }

    inline = ["echo 'connected!'"]
  }

  provisioner "local-exec" {
    command = "ansible-playbook -vvv -T 180 -i /path/to/inventory /path/to/ansible/playbook.yml --extra-vars '@/path/to/ansible/vars/docker_hub.yml'"
  }

  depends_on = [aws_instance.app]
}

Step 4: Writing Ansible Playbooks

Ansible is used for configuring the application and monitoring stacks. The playbook.yml file will define the steps to set up the system.

Server Setup: Install necessary dependencies and configure the environment.
Application Setup: Pull Docker images from Docker Hub and run the frontend and backend services.
Monitoring Setup: Install Prometheus, Grafana, and other monitoring tools.
Reverse Proxy Setup: Configure Traefik or Nginx to route traffic.

Here’s an example of the playbook:

---
- name: Configure app server
  hosts: all
  become: yes
  tasks:
    - name: Install Docker
      apt:
        name: docker.io
        state: present

    - name: Create a shared Docker network
      command: docker network create app-network

    - name: Pull frontend Docker image
      docker_image:
        name: "{{ docker_hub.frontend_image }}"
        source: pull

    - name: Pull backend Docker image
      docker_image:
        name: "{{ docker_hub.backend_image }}"
        source: pull

    - name: Run frontend container
      docker_container:
        name: frontend
        image: "{{ docker_hub.frontend_image }}"
        state: started
        restart_policy: always
        networks:
          - name: app-network

    - name: Run backend container
      docker_container:
        name: backend
        image: "{{ docker_hub.backend_image }}"
        state: started
        restart_policy: always
        networks:
          - name: app-network

    - name: Install Traefik for routing
      docker_container:
        name: traefik
        image: traefik:v2.5
        state: started
        restart_policy: always
        ports:
          - "80:80"
          - "443:443"
        command: "--api.insecure=true --providers.docker"

    - name: Install Prometheus
      docker_container:
        name: prometheus
        image: prom/prometheus
        state:

 started
        restart_policy: always
        ports:
          - "9090:9090"
        volumes:
          - /prometheus.yml:/etc/prometheus/prometheus.yml

Step 5: Finalizing the Deployment

Once the Terraform configuration and Ansible playbooks are set up:

Run terraform apply to provision the infrastructure.
Terraform will automatically trigger Ansible, which will configure the server and deploy the application.
After execution, you can access the application via the public IP provided by Terraform, and the monitoring stack (Grafana, Prometheus, etc.) will be up and running.

Testing the Setup

You can test the entire setup by visiting the public IP address in your browser. Ensure that:

The frontend and backend applications are accessible.
The monitoring dashboard in Grafana is functioning.
Logs are being collected by Loki.

Conclusion

This project demonstrated how to automate cloud infrastructure provisioning and application deployment using Terraform and Ansible. By combining IaC and CM, you can ensure repeatable, consistent, and scalable deployments. In this case, we also integrated monitoring into the workflow, which is an essential part of maintaining cloud-based systems.

Through this guide, you learned how to:

Build and push Docker images for your applications.
Write Terraform configurations to provision cloud resources.
Use Ansible to configure servers and deploy applications.
Set up monitoring using Prometheus, Grafana, and other tools.

This approach can be adapted to suit more complex applications and environments, making it an invaluable skill for any DevOps engineer or cloud automation specialist. Happy automating!

Connecting Jenkins to Slack: A Beginner's Guide

John Doe — Fri, 29 Nov 2024 13:04:08 +0000

Integrating Jenkins with Slack allows your development team to receive real-time notifications about build statuses, deployments, and other important events directly in Slack channels. This seamless communication enhances collaboration, accelerates issue resolution, and keeps everyone informed throughout the development lifecycle.

This guide provides a step-by-step walkthrough on how to connect Jenkins to Slack, ensuring that your team stays updated on your CI/CD pipeline activities

Benefits of Integrating Jenkins with Slack

Immediate Feedback: Slack notifications provide instant updates on build processes, enabling faster troubleshooting.
Improved Team Collaboration: Real-time alerts keep all team members on the same page, enhancing communication and efficiency.

Prerequisites

Before you begin, ensure you have:

Jenkins installed and running on your local Windows machine. (Refer to previous documentation on installing Jenkins using Docker if needed.)
Administrative access to your Jenkins instance.
A Slack workspace where you can install apps and integrations.
Working understanding of Jenkins and Slack.

Installing the Slack Notification Plugin in Jenkins
Creating a Slack App and Obtaining Credentials
Configuring Jenkins Global Settings for Slack
Setting Up a Jenkins Job to Send Slack Notifications
Testing the Integration
Conclusion

1. Installing the Slack Notification Plugin in Jenkins

Why This Step?

To enable Jenkins to communicate with Slack, you need to install the Slack Notification Plugin, which provides the necessary functionality for sending messages to Slack channels.

Step-by-Step Guide

Access Jenkins Dashboard:
- Open your web browser and go to your Jenkins instance (e.g., http://localhost:8080 or whatever port you have configured your Jenkins server to listen on).
Navigate to Plugin Manager:
- Click on "Manage Jenkins" on the left sidebar.
- Select "Manage Plugins".

Install the Slack Notification Plugin:
- Go to the "Available" tab.
- In the "Filter" search box, type "Slack Notification".

Check the box next to "Slack Notification Plugin".
Click "Install without restart".

Wait for Installation:
- The plugin will download and install. Wait until you see a confirmation message.

2. Creating a Slack App and Obtaining Credentials

Why This Step?

Creating a Slack App and obtaining an OAuth token allows Jenkins to authenticate with Slack and send messages to your channels.

Step-by-Step Guide

Go to Your Slack Workspace:
- Click on the dropdown arrow besides your Slack workspace name and select "Tool & Settings" and then "Manage Apps" next.

Add Jenkins App:
- On the "Instaled Apps" window that follows, enter Jenkins CI in the "Search Slack Marketplace" search bar.
- In the window that follows, click on the pill shaped green button "Add to Slack".

Set Up App Details:
- Follow the setup instructions in the edit configuration window.
- Go to "Manage Jenkins" again in the left navigation in your Jenkins server, then click on "System".

3. Configuring Jenkins Global Settings for Slack

Why This Step?

Configuring Jenkins with your Slack credentials allows it to send notifications to your Slack channels.

Step-by-Step Guide

Access Jenkins Configuration:
- From the Jenkins dashboard, click "Manage Jenkins".
- Select "Configure System".
Find Slack Plugin Settings:
- Scroll down until the "Slack" settings section (typically at the end of the page).

Fill in Slack Configuration:
- Workspace: Enter your Slack workspace domain (e.g., myworkspace from myworkspace.slack.com).
- Credential:
  - Click on the "+Add" button, just beneath the credential field
  - Choose "Global" for scope.
  - Kind: Select "Secret text".
  - Secret: Paste the Integration token credential ID.
  - ID: Name it (e.g., slack-token).
  - Click "Add".

Default Channel: Enter the Slack channel (e.g., #cicd).
Test Connection:
- Click "Test Connection".
- Look for a success message.

Save Configuration:
- Click "Save" at the bottom.

4. Setting Up a Jenkins Job to Send Slack Notifications

Placeholder for Job Configuration Screenshot

Why This Step?

Configuring individual jobs to send notifications ensures that you receive updates for specific builds or projects.

Step-by-Step Guide

Open a Jenkins Job:
- Go to the Jenkins dashboard.
- Click on an existing job or create a new one.
Configure the Job:
- Click "Configure" on the left.
Enable Slack Notifications:
- Scroll to "Post-build Actions".
- Click "Add post-build action".
- Select "Slack Notifications".
Placeholder for Adding Slack Notifications Screenshot
Set Notification Preferences:
- Check the boxes for when you want notifications (e.g., "Notify Success", "Notify Failure").
- Optionally, click "Advanced..." to specify a different channel.
Placeholder for Slack Notification Settings Screenshot
Save the Job Configuration:
- Click "Save".

5. Testing the Integration

Placeholder for Build Now Screenshot

Why This Step?

Testing ensures that your setup works correctly and that notifications are received in Slack.

Step-by-Step Guide

Run the Jenkins Job:
- Go to the job page.
- Click "Build Now".
Check Slack Channel:
- Open Slack and navigate to your channel.
- Look for notifications about the build.
Placeholder for Slack Notification Screenshot
Verify the Notification:
- Ensure the notification shows the correct build status and details.

Conclusion

By connecting Jenkins to Slack, you've enhanced your development workflow. Installing Jenkins locally allows you to experiment and learn, while Slack integration keeps your team informed in real-time.

Key Benefits

Immediate Awareness: Instant notifications about builds help quickly address issues.
Enhanced Collaboration: Teams stay aligned and can discuss problems directly in Slack.
Improved Efficiency: Automated updates reduce the need for manual checks.

Happy Building and Collaborating!

Deploying a Fullstack Application with Monitoring: A Comprehensive Guide

John Doe — Thu, 28 Nov 2024 15:49:53 +0000

In this article, we will walk through deploying a fullstack application (frontend, backend, and database) and setting up a monitoring stack using Prometheus, Grafana, and cAdvisor. The goal is to simplify the process so even beginners can follow along and understand the purpose of each step.

Overview

Fullstack Application Deployment: Deploy a frontend, backend, and database with Docker and Docker Compose.
Monitoring Stack Setup: Integrate monitoring tools to track application and server performance.
NGINX Proxy Manager: Use it for routing traffic, enabling SSL, and managing domain redirections.

1. Prerequisites

Before starting, ensure you have the following installed on your server or local machine:

Docker: To containerize and run applications.
Docker Compose: To manage multi-container setups.
A Domain Name: For accessing your application and monitoring stack (e.g., example.com).
Basic Linux Skills: Ability to use the terminal for commands.

2. Setting Up the File Structure

Here’s how your project folder should look:

project/
├── backend/          # Contains the backend application
├── frontend/         # Contains the frontend application
├── monitoring/       # Contains configurations for Prometheus, Promtail, and Loki
├── docker-compose.yml
└── README.md

Backend: API service logic (e.g., Node.js, Python).
Frontend: UI served via NGINX.
Monitoring: Monitoring tools (Prometheus, Grafana, etc.).
docker-compose.yml: Defines all services for deployment.

3. Writing the `docker-compose.yml`

The docker-compose.yml file combines all the services into one deployable stack.

version: '3.8'

services:
  backend:
    build:
      context: ./backend
      dockerfile: Dockerfile
    container_name: backend
    ports:
      - "8000:8000"
    networks:
      - app-network
    depends_on:
      - db

  frontend:
    build:
      context: ./frontend
      dockerfile: Dockerfile
    container_name: frontend
    ports:
      - "5173:80"
    networks:
      - app-network

  db:
    image: postgres:14
    container_name: db
    volumes:
      - db-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: appuser
      POSTGRES_PASSWORD: password123
      POSTGRES_DB: appdb
    networks:
      - app-network

  prometheus:
    image: prom/prometheus
    container_name: prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.external-url=http://localhost/prometheus'
    ports:
      - "9090:9090"
    volumes:
      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml
    networks:
      - app-network

  grafana:
    image: grafana/grafana
    container_name: grafana
    environment:
      - GF_SERVER_ROOT_URL=http://localhost/grafana
      - GF_SERVER_SERVE_FROM_SUB_PATH=true
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
    networks:
      - app-network

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    ports:
      - "8083:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    networks:
      - app-network

volumes:
  db-data:
  grafana-data:

networks:
  app-network:
    driver: bridge

4. Configuring the Monitoring Tools

4.1 Prometheus Configuration

Prometheus scrapes metrics from services like cAdvisor. Add a prometheus.yml file in the monitoring folder.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "cadvisor"
    static_configs:
      - targets: ["cadvisor:8080"]

4.2 Grafana Configuration

Grafana uses Prometheus as a data source. Configure dashboards by:

Starting Grafana (http://localhost:3000).
Logging in (default: admin/admin).
Adding Prometheus as a data source (http://prometheus:9090).

5. Deploying the Fullstack Application

Build and Start Services: Run the following command to start all services:

   docker-compose up -d

Access the Services:
- Backend: http://localhost:8000
- Frontend: http://localhost:5173
- Prometheus: http://localhost:9090
- Grafana: http://localhost:3000
- cAdvisor: http://localhost:8083

6. Setting Up NGINX Proxy Manager

NGINX Proxy Manager provides routing, HTTPS, and domain management.

6.1 Installation

Install NGINX Proxy Manager as a Docker container:

services:
  proxy:
    image: 'jc21/nginx-proxy-manager:latest'
    ports:
      - "80:80"
      - "443:443"
      - "81:81"
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt

Run docker-compose up -d to start the proxy manager.

6.2 Configuring Proxy Hosts

Log in to NGINX Proxy Manager (http://<your-server-ip>:81).
Add a Proxy Host for each service:
- Domain: example.com
- Forward Hostname: Container name (e.g., frontend, backend).
- Forward Port: Port of the service (e.g., 8000 for the backend).
Enable Force SSL and request a Let's Encrypt certificate.

7. Testing and Debugging

Logs: Check logs for errors:

   docker logs <container-name>

Network:
Ensure containers can communicate via docker network inspect app-network.
DNS:
Verify domain resolves to the server IP using:

   nslookup example.com

8. Enhancements

Add Alerting: Integrate Alertmanager with Prometheus for alert notifications.
Expand Monitoring: Use Node Exporter to monitor the host system.
Automate Deployment: Use CI/CD pipelines to deploy updates automatically.

Conclusion

This guide covered deploying a fullstack application with a monitoring stack and routing traffic using NGINX Proxy Manager. With Docker Compose, the process is streamlined, and tools like Prometheus and Grafana ensure your application is monitored effectively. Let me know if you need further assistance! 😊

How to Install Jenkins on a Local Windows Machine

John Doe — Thu, 28 Nov 2024 15:46:01 +0000

Jenkins is an open-source automation server that enables developers to build, test, and deploy their software reliably. It plays a pivotal role in the DevOps lifecycle by automating tasks, facilitating continuous integration and continuous delivery (CI/CD), and improving collaboration among development teams.

Installing Jenkins on your local Windows machine allows you to:

Experiment with Jenkins features without affecting a production environment.
Develop and test CI/CD pipelines locally.
Gain hands-on experience with DevOps practices.

This guide will walk you through two methods to install Jenkins on Windows:

Using Docker (Recommended)
Using Windows Subsystem for Linux (WSL)

We'll compare both methods and explain why using Docker is the better choice.

Method 1: Installing Jenkins Using Docker

Placeholder for Docker and Jenkins Logos

Prerequisites

Windows 10 or later
Docker Desktop installed on your machine

Steps to Install Jenkins with Docker

Step 1: Install Docker Desktop

If you haven't installed Docker Desktop:

Download Docker Desktop from the official website.

Check the architecture of your Windows machine (x86_64 or Arm). Here's a medium article that helps.
After a successful download, run the installer and follow the on-screen instructions.
After installation, launch Docker Desktop and ensure it's running.

Step 2: Pull the Jenkins Docker Image

Open Command Prompt or PowerShell and run:

docker pull jenkins/jenkins:lts

This command downloads the latest stable (LTS) Jenkins image.

Step 3: Run Jenkins Container

docker run \
  -p 8080:8080 \
  -p 50000:50000 \
  -d \
  -v jenkins_home:/var/jenkins_home \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v $(which docker):/usr/bin/docker \
  jenkins/jenkins:lts

docker run: This is the Docker command used to create and run a new container from a specified image.
-p 8080:8080:
- Purpose: Maps port 8080 of the host machine to port 8080 of the container.
- Explanation: Jenkins uses port 8080 as its default web interface port. This mapping allows you to access the Jenkins web UI via http://localhost:8080.
-p 50000:50000:
- Purpose: Maps port 50000 of the host to port 50000 of the container.
- Explanation: Port 50000 is used by Jenkins for communication with build agents (JNLP agents). This mapping enables distributed builds and agent connections.
-d :
- Purpose: Runs the container in detached mode.
- Explanation: The container runs in the background, allowing you to continue using the terminal for other commands.
-v jenkins_home:/var/jenkins_home :
- Purpose: Mounts the Docker volume jenkins_home to the container's /var/jenkins_home directory.
- Explanation:
  - Data Persistence: Jenkins stores its configuration, plugins, and job data in /var/jenkins_home. Mounting a volume ensures that this data persists across container restarts or recreations.
  - Named Volume: Using jenkins_home as a named volume allows Docker to manage the volume, and it can be reused or backed up easily.
-v /var/run/docker.sock:/var/run/docker.sock :
- Purpose: Mounts the Docker daemon socket from the host into the container.
- Explanation:
  - Docker Control: By mounting the Docker socket, Jenkins running inside the container can communicate with the Docker daemon on the host.
  - Use Case: This is essential if you want Jenkins to build Docker images or manage Docker containers as part of your CI/CD pipelines.
-v $(which docker):/usr/bin/docker :
- Purpose: Mounts the Docker executable from the host into the container at /usr/bin/docker.
- Explanation:
  - Docker CLI Access: This makes the Docker command-line tool (docker) available inside the Jenkins container.
  - Version Consistency: Ensures that the Docker CLI inside the container matches the version installed on the host, avoiding compatibility issues.

Note: $(which docker) is a command substitution that returns the path of the Docker executable on the host system.

jenkins/jenkins:lts :
- Purpose: Specifies the Docker image to use, in this case, the Jenkins Long-Term Support (LTS) version.
- Explanation:
  - Stable Version: Using the LTS version ensures you are running a stable and well-supported release of Jenkins.
  - Image Source: The image jenkins/jenkins:lts is maintained by the Jenkins project on Docker Hub.

Summary of the Command's Purpose

This Docker command sets up a Jenkins server in a container with the capability to:

Persist data through the jenkins_home volume.
Be accessible via the web interface on localhost:8080.
Communicate with build agents through port 50000.
Interact with Docker on the host machine to build images and manage containers from within Jenkins jobs.

Why Mount Docker Socket and Executable?

By mounting both the Docker socket and executable:

Docker Socket (/var/run/docker.sock):
- Grants the container access to the Docker daemon running on the host.
- Allows Jenkins to execute Docker commands that affect the host's Docker environment.
Docker Executable (/usr/bin/docker):
- Provides the docker CLI inside the container.
- Ensures Jenkins can run Docker commands as if it were on the host.

This setup is commonly used when you need Jenkins to perform Docker-related tasks, such as:

Building Docker images.
Running Docker containers.
Managing Docker registries.

Security Considerations

Privileges: Mounting the Docker socket and executable gives the Jenkins container elevated permissions over the host's Docker daemon.
Risk: If the Jenkins server or any of its jobs are compromised, malicious commands could be executed on the host.
Mitigation:
- Limit access to the Jenkins server.
- Use proper authentication and authorization within Jenkins.
- Consider using Docker-in-Docker or Docker agents with limited permissions for additional isolation.

Alternative Approaches

Docker-in-Docker (dind):
- Run a separate Docker daemon inside the container.
- Adds complexity and potential performance overhead.
- Provides better isolation from the host's Docker daemon.
Docker Agents:
- Use separate Docker containers as build agents.
- Avoids giving the main Jenkins container direct access to the host's Docker daemon.

Example Usage in a Jenkins Pipeline

With this setup, you can create Jenkins pipelines that include steps like:

pipeline {
    agent any
    stages {
        stage('Build Docker Image') {
            steps {
                sh 'sudo docker build -t my-image .'
            }
        }
        stage('Run Docker Container') {
            steps {
                sh 'sudo docker run -d --name my-container my-image'
            }
        }
    }
}

This pipeline builds a Docker image and runs a container using the Docker commands available inside the Jenkins container.

Step 4: Access Jenkins Web Interface

Open a web browser and navigate to http://localhost:8080.

You'll see the "Unlock Jenkins" screen. Retrieve the initial admin password by checking the Docker container logs or reading the initialAdminPassword file:

   docker exec -it <container_id> cat /var/jenkins_home/secrets/initialAdminPassword

Placeholder for Screenshot of Command to Retrieve Password

Copy the password, paste it into the web interface, and proceed with the setup.
Install suggested plugins or customize as needed.
Create your first admin user, and you're ready to use Jenkins!

Advantages of Using Docker

Portability: Docker containers can run on any machine with Docker installed.
Isolation: Each container is isolated, preventing conflicts with other applications.
Ease of Setup: Simplifies the installation process with minimal configuration.
Consistency: Ensures the environment is consistent across different setups.
Easy Cleanup: Containers can be stopped and removed without leaving residual files.

To allow the Jenkins user inside the Docker container to run Docker commands without needing to prefix them with sudo. This involves adding the Jenkins user to the Docker group inside the container.

Here's how you can do it:

Allowing Jenkins to Run Docker Commands Without `sudo`

By default, Docker commands require root privileges. To enable the Jenkins user inside the container to execute Docker commands without sudo, you need to add the Jenkins user to the Docker group. Here's how to do it:

Step 1: Access the Running Jenkins Container as Root

First, you'll need to access the running Jenkins container with root privileges. You can do this using the docker exec command:

docker exec -u 0 -it <container-name> bash

docker exec: Runs a command in a running container.
-u 0: Specifies the user ID to execute the command as. 0 is the user ID for root.
-it: Allocates a pseudo-TTY and keeps STDIN open, allowing you to interact with the container's shell.
<container-name>: Replace this with the actual name or ID of your Jenkins container.
bash: The command to run, in this case, starting a Bash shell.

Example:

docker exec -u 0 -it jenkins-container bash

Note: Replace jenkins-container with your container's actual name or ID.

Step 2: Add Jenkins User to the Docker Group

Once inside the container as the root user, execute the following commands:

Check if the Docker Group Exists

First, verify whether the docker group exists inside the container:

grep -i docker /etc/group

If it doesn't exist, create it:

  groupadd docker

Add Jenkins User to the Docker Group

Now, add the jenkins user to the docker group:

usermod -aG docker jenkins

usermod: A command to modify a user's settings.
-aG docker: Appends the user to the docker group.

Verify the User's Group Membership

You can verify that the jenkins user is now part of the docker group:

groups jenkins

You should see docker listed among the groups.

Step 3: Exit and Restart the Container

For the group membership changes to take effect, you need to restart the Jenkins container:

Exit the container shell:

   exit

Restart the container:

   docker restart <container-name>

Example:

   docker restart jenkins-container

Step 4: Test Docker Commands in Jenkins

Now, inside your Jenkins jobs or pipelines, you should be able to run Docker commands without needing sudo.

Example Pipeline Step:

pipeline {
    agent any
    stages {
        stage('Test Docker Command') {
            steps {
                sh 'docker ps'
            }
        }
    }
}

This pipeline stage will list running Docker containers, and it should execute without permission errors.

Why These Steps Are Necessary

Understanding User Permissions

Docker Daemon Access: The Docker daemon (dockerd) listens on a Unix socket (/var/run/docker.sock) by default. Access to this socket is controlled by Unix file permissions.
Docker Group: Users who are members of the docker group can interact with the Docker daemon without sudo, as the socket is owned by the docker group.

Security Considerations

Root Privileges: Granting access to the Docker daemon is equivalent to giving root access, as Docker commands can affect the host system.
Caution Advised: Ensure that only trusted users and processes have this access.

Important Notes

1. Security Implications

Risk of Privilege Escalation: Users with access to the Docker daemon can gain root-level access to the host.
Best Practices:
- Limit access to the Jenkins interface.
- Secure your Jenkins jobs to prevent arbitrary code execution.
- Regularly update Jenkins and its plugins to patch security vulnerabilities.

2. Alternatives to Adding Jenkins User to Docker Group

If security is a major concern, consider the following alternatives:

Docker-in-Docker (DinD):
- Run a separate Docker daemon inside the Jenkins container.
- Isolate Docker operations from the host.
- Cons: More complex setup, potential performance overhead.
Use Docker Agents:
- Instead of running Docker commands on the Jenkins master, use agents (nodes) that have Docker installed and configured.
- Assign jobs that require Docker to these agents.
- Pros: Better isolation, easier to manage permissions.
Use Pipeline Plugins:
- Utilize plugins like the Docker Pipeline plugin, which provides DSL for building and running containers.
- Pros: Abstracts Docker operations, can improve security.

3. Ensure Consistent Docker Versions

Since you're mounting the Docker executable and socket from the host, make sure that the versions are compatible with the Jenkins environment inside the container.
Incompatibilities can lead to unexpected behavior or errors.

Method 2: Installing Jenkins on Windows Subsystem for Linux (WSL)

Placeholder for WSL and Jenkins Logos

Prerequisites

Windows 10 Version 2004 or higher
Windows Subsystem for Linux (WSL) enabled
A Linux distribution installed (e.g., Ubuntu from Microsoft Store)

Steps to Install Jenkins on WSL

Step 1: Enable WSL

Open PowerShell as Administrator.
Run:

   wsl --install

Placeholder for Screenshot of WSL Installation Command

Restart your computer if prompted.

Step 2: Install a Linux Distribution

Open the Microsoft Store.
Search for Ubuntu and install it.

Placeholder for Screenshot of Ubuntu in Microsoft Store

Launch Ubuntu from the Start menu.

Step 3: Update Linux Packages

In the Ubuntu terminal, run:

sudo apt update
sudo apt upgrade -y

Placeholder for Screenshot of Package Update Commands

Step 4: Install Java

Jenkins requires Java. Install it using:

sudo apt install openjdk-21-jre -y

Placeholder for Screenshot of Java Installation Command

Step 5: Add Jenkins Repository and Key

Add the Jenkins Debian repository key:

   wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -

Add the repository to the sources list:

   sudo sh -c 'echo deb https://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'

Placeholder for Screenshot of Adding Jenkins Repository

Step 6: Install Jenkins

Update packages and install Jenkins:

sudo apt update
sudo apt install jenkins -y

Placeholder for Screenshot of Jenkins Installation Command

Step 7: Start Jenkins Service

sudo service jenkins start

Verify Jenkins is running:

sudo service jenkins status

Placeholder for Screenshot of Jenkins Service Status

Step 8: Access Jenkins Web Interface

Open a web browser and navigate to http://localhost:8080.

Placeholder for Screenshot of Jenkins Unlock Screen via WSL

Retrieve the initial admin password:

cat /var/lib/jenkins/secrets/initialAdminPassword

Continue with the setup as described in the Docker method.

Advantages of Using WSL

Linux Environment: Provides a genuine Linux environment on Windows.
Direct Access: Ability to use Linux commands and tools directly.
Integration: Seamless integration between Windows and Linux file systems.

Why Docker is the Better Method

While both methods allow you to run Jenkins locally, using Docker has several advantages over WSL:

Simplified Setup: Docker requires fewer steps and configurations.
Resource Efficiency: Docker containers are lightweight compared to running a full Linux distribution in WSL.
Isolation: Docker provides better isolation, reducing potential conflicts.
Consistency Across Environments: Docker ensures that the Jenkins instance behaves the same across different systems.
Easier Maintenance: Updating or removing Jenkins is straightforward with Docker commands.
Portability: Docker images can be easily shared and deployed on other machines or servers.

In contrast, WSL may present challenges such as networking configurations, potential performance overhead, and a more complex setup process.

Conclusion

Installing Jenkins locally on your Windows machine empowers you to develop and test CI/CD pipelines in a controlled environment. While both Docker and WSL methods are viable, Docker offers a more streamlined, efficient, and beginner-friendly approach.

By leveraging Jenkins in your DevOps cycle, you can:

Automate repetitive tasks.
Enhance collaboration between development and operations teams.
Accelerate software delivery.
Improve code quality through continuous integration and testing.

Happy Building with Jenkins!

How to Automate User and Group Creation and Management Using a Simple Bash Script

John Doe — Wed, 03 Jul 2024 19:23:17 +0000

One of the telling signs someone's a DevOps engineer is their unrivaled urge to automate process. It isn't just for automation sake, but to:

Enhance productivity
Reduce human error

One such task is onboarding new developers to the team. This includes creating user accounts, assigning them to appropriate groups, and setting secure passwords.

This article will walk you how you can streamline this process using a bash script. You don't have to be a pro at this, it shows you why each step is important.

What does this script do?
The script automates the following tasks:

Prompts the user for the path to an employee configuration file.
Validates the provided file path.
Creates necessary directories and sets appropriate permissions.
Checks for and installs the makepasswd tool if not already present.
Generates secure passwords for new users.
Creates user accounts and assigns them to specified groups.
Logs all actions for auditing purposes.

Let's break down each section of the script.

Color-Coded Output Functions

bash
#!/bin/bash

#--------------------------
# Function to echo in green
#--------------------------
green_echo() {
    echo -e "\e[32m$1\e[0m"
}

#------------------------
# Function to echo in red
#------------------------
red_echo() {
    echo -e "\e[31m$1\e[0m"
}

These functions, green_echo and red_echo, are used to print messages in green and red. The idea is to help you differentiate between successful and error messages, making your script's output more readable. Typically, output messages from running scripts are in white (against the set background of your terminal) making it difficult to see errors among the sea of output messages. Red and green-codded outputs make it easier to scan your script output.

Checking if a File Argument Was Provided

#---------------------------------------------------------
# Check if a file path argument is provided and validate it
#---------------------------------------------------------
if [ -z "$1" ]; then
    red_echo "Error: No file path provided. Please provide the employee config file path as the first argument."
    exit 1
fi

EMPLOYEE_CONFIG_FILE="$1"

if [ ! -f "$EMPLOYEE_CONFIG_FILE" ]; then
    red_echo "Error: The file '$EMPLOYEE_CONFIG_FILE' does not exist or is not a regular file."
    exit 1
fi

green_echo "File path is valid."

This portion of the script confirms the argument the user attaches to the script is a valid file and exists. If the file path is invalid, it will prompt the user to try again, ensuring an argument is provided and it exists before proceeding.

Defining File Paths

#-----------------------------------------
# Variables for the password and log files
#-----------------------------------------
PASSWORD_FILE="/var/secure/user_passwords.txt"
LOG_FILE="/var/log/user_management.log"

These variables define the paths for storing generated passwords and logs. It is standard DevOps practice to store passwords in a file. And for even better practice, ensures these files are secure and only readable by appropriate personnel.

Another standard practice is logging actions. It helps you in auditing and troubleshooting should you be stuck. Defining the variables for the paths to these files early on makes it easier to maintain the script.

Creating and Securing Directories

#----------------------------------------------------------
# Create necessary directories with appropriate permissions
#----------------------------------------------------------
sudo mkdir -p /var/secure
sudo mkdir -p /var/log
sudo chmod 600 /var/secure

This script block creates the directories to the files for storing passwords and logs if they do not already exist. It also sets strict permissions (chmod 600) on the /var/secure directory so that only the user can read and write to it, enhancing security.

Ensuring `makepasswd` is Installed

#----------------------------------------------
# Checking and ensuring makepasswd is installed
#----------------------------------------------
if ! command -v makepasswd &>/dev/null; then
    sudo apt-get update && sudo apt-get install -y makepasswd
fi

makepasswd is a tool used to generate random passwords. It doesn't come preinstalled in most Linux/Unix systems. So, you'd have to check if it is installed before moving on to other parts. This block checks if makepasswd is installed and installs it if necessary. Ensuring this dependency is met is crucial for the script to function correctly.

Password Generation Function

#--------------------------------------------
# Generate a random password of 16 characters
#--------------------------------------------
generate_password() {
    makepasswd --chars 16
}

This function uses makepasswd to generate a random password of 16 characters. Strong passwords are essential for security, and automating their generation helps maintain consistency and complexity.

Clearing Previous Logs and Passwords

#--------------------------------------
# Clear previous log and password files
#--------------------------------------
sudo truncate -s 0 "$LOG_FILE"
sudo truncate -s 0 "$PASSWORD_FILE"

Before starting the onboarding process, the script clears previous log and password files. This ensures that the new run starts with clean files, preventing confusion with old data.

Note that if you use echo "" | sudo tee "$LOGFILE" > /dev/null or echo "" | sudo tee "$PASSWORD_FILE" > /dev/null instead, what you get instead will be a file with an empty line. Any lines appended to the file will begin from the next line, leaving an empty first line. Using truncate sets the file to whatever size you specify. And in this case, it's zero bytes which effectively clears the file"

Reading and Processing the Configuration File

while IFS=';' read -r username groups; do

    #----------------------------------------
    # Remove leading and trailing whitespaces
    #----------------------------------------
    username=$(echo "$username" | xargs)
    groups=$(echo "$groups" | xargs)

    #-----------------
    # Skip empty lines
    #-----------------
    [ -z "$username" ] && continue

    #---------------------------------
    # Split the groups field by commas
    #---------------------------------
    IFS=',' read -ra group_array <<<"$groups"

The script reads the configuration file line by line, splitting each line into a username and groups. It goes further to remove any leading or trailing whitespace to ensure clean data processing. There's the added contingency to skip empty lines to prevent errors.

Creating Users and Setting Passwords

    #---------------------------------
    # Check if the user already exists
    #---------------------------------
    if id "$username" &>/dev/null; then
        red_echo "The user $username already exists." | sudo tee -a "$LOG_FILE"
    else
        sudo useradd -m -s /bin/bash "$username" &&
            green_echo "The user $username has been created." | sudo tee -a "$LOG_FILE"

        #---------------------------
        # Generate a random password
        #---------------------------
        password=$(generate_password)

        #------------------------
        # Set the user's password
        #------------------------
        echo "$username:$password" | sudo chpasswd
        echo "$username:$password" | sudo tee -a "$PASSWORD_FILE"
    fi

This block checks if the user already exists. If the user does not exist, it creates the user with a home directory and Bash shell. A random password is generated and set for the user. The username and password are logged for record-keeping.

Creating Primary Groups

    #--------------------------------------------------------
    # Create a primary group for the user if it doesn't exist
    #--------------------------------------------------------
    if ! getent group "$username" >/dev/null; then
        sudo groupadd "$username" &&
            green_echo "Primary group $username created." | sudo tee -a "$LOG_FILE"
    fi

For each new user, the script checks if a primary group with the same name exists. If not, it creates the group. This ensures that each user has a corresponding group, which is a common practice in Unix-like systems.

Assigning Users to Additional Groups

    for group in "${group_array[@]}"; do
        if ! getent group "$group" >/dev/null; then
            sudo groupadd "$group" &&
                green_echo "Group $group created." | sudo tee -a "$LOG_FILE"
        fi
        sudo usermod -aG "$group" "$username" &&
            green_echo "User $username added to group $group." | sudo tee -a "$LOG_FILE"
    done

The script processes each group specified for the user. It checks if the group exists and creates it if necessary. It then adds the user to each group, ensuring they have the required permissions and access.

Setting Home Directory Permissions

    #-------------------------------
    # Set home directory permissions
    #-------------------------------
    sudo chown -R "$username":"$username" "/home/$username"
    sudo chmod 700 "/home/$username"

done <"$EMPLOYEE_CONFIG_FILE"

green_echo "User onboarding script completed. See $LOG_FILE for details."

Finally, the script sets the ownership and permissions for the user's home directory. This ensures that the user has the necessary access to their files while also securing the directory from unauthorized access.

To run the script, make it executable.

chmod +x create_users.sh

Then execute it:

./create_users.sh new_developers.txt

So the script runs as intended, ensure that the file the script reads is written and formatted this way (username to the left of the semi-colon and user groups to the right):

light; sudo,dev,www-data
idimma; sudo
mayowa; dev,www-data

The assumption behind this example is that a user can have multiple groups, each group delimited by a comma ",".

Here's the link to the GitHub repo if you want to take a look at it closely.

Final Words...

When you automate the user onboarding process with a Bash script ensures consistency, accuracy, and security. By following the steps outlined in this article, you can streamline the creation of user accounts, assignment of groups, and setting of secure passwords, all while maintaining detailed logs for auditing purposes.

To improve the security of the script, you can:

Hash the passwords after generation
Configure the script to send each employee their login details to their emails

Tweak and improve as desired.

Looking for projects like this to help you better understand DevOps concepts, enroll for HNG Internship. If you are already a professional looking for a community of professionals to collaborate with and have recruiters review your resumes, then opt for the HNG Premium package.

DEV Community: John Doe

Automating Full-Stack Cloud Deployments Using Terraform and Ansible

Overview of the Project

Step 1: Building and Pushing Docker Images

Frontend Application

Backend Application

Step 2: Designing the Infrastructure

Architecture Diagram

Step 3: Writing Terraform Configuration

Terraform and Ansible Integration

Step 4: Writing Ansible Playbooks

Step 5: Finalizing the Deployment

Testing the Setup

Conclusion

Connecting Jenkins to Slack: A Beginner's Guide

Benefits of Integrating Jenkins with Slack

Prerequisites

Table of Contents

1. Installing the Slack Notification Plugin in Jenkins

Why This Step?

Step-by-Step Guide

2. Creating a Slack App and Obtaining Credentials

Why This Step?

Step-by-Step Guide

3. Configuring Jenkins Global Settings for Slack

Why This Step?

Step-by-Step Guide

4. Setting Up a Jenkins Job to Send Slack Notifications

Why This Step?

Step-by-Step Guide

5. Testing the Integration

Why This Step?

Step-by-Step Guide

Conclusion

Key Benefits

Deploying a Fullstack Application with Monitoring: A Comprehensive Guide

Overview

1. Prerequisites

2. Setting Up the File Structure

3. Writing the docker-compose.yml

4. Configuring the Monitoring Tools

4.1 Prometheus Configuration

4.2 Grafana Configuration

5. Deploying the Fullstack Application

6. Setting Up NGINX Proxy Manager

6.1 Installation

6.2 Configuring Proxy Hosts

7. Testing and Debugging

8. Enhancements

Conclusion

How to Install Jenkins on a Local Windows Machine

Method 1: Installing Jenkins Using Docker

Prerequisites

Steps to Install Jenkins with Docker

Step 1: Install Docker Desktop

Step 2: Pull the Jenkins Docker Image

Step 3: Run Jenkins Container

Summary of the Command's Purpose

Why Mount Docker Socket and Executable?

Security Considerations

Alternative Approaches

Example Usage in a Jenkins Pipeline

Step 4: Access Jenkins Web Interface

Advantages of Using Docker

Allowing Jenkins to Run Docker Commands Without sudo

Step 1: Access the Running Jenkins Container as Root

Step 2: Add Jenkins User to the Docker Group

Check if the Docker Group Exists

Add Jenkins User to the Docker Group

Verify the User's Group Membership

Step 3: Exit and Restart the Container

Step 4: Test Docker Commands in Jenkins

Why These Steps Are Necessary

Understanding User Permissions

Security Considerations

Important Notes

1. Security Implications

2. Alternatives to Adding Jenkins User to Docker Group

3. Ensure Consistent Docker Versions

Method 2: Installing Jenkins on Windows Subsystem for Linux (WSL)

3. Writing the `docker-compose.yml`

Allowing Jenkins to Run Docker Commands Without `sudo`

Ensuring `makepasswd` is Installed