DEV Community: Afeez Oluwashina Adeboye

Customizing and Deploying a Static Website with NGINX on AWS EC2

Afeez Oluwashina Adeboye — Wed, 29 Jan 2025 13:58:12 +0000

Introduction

As part of the DevOps Stage 0 task, I was required to install and configure NGINX on an Ubuntu server, setting up a custom HTML page as the default page. To demonstrate automation skills, I chose to deploy this setup on an AWS EC2 instance using user-data scripting for automatic configuration.

Approach

To complete this task, I followed these steps:

1. Launching the EC2 Instance

Logged into AWS and navigated to EC2 Dashboard.
Created a new Ubuntu 22.04 LTS instance.
Chose t2.micro (free-tier eligible).
Configured security group rules to allow SSH (port 22) but forgot to open HTTP (port 80) initially.
Attached a key pair for secure SSH access.

2. Using User-Data for Automated Setup

To automate the NGINX installation and configuration, I used the following user-data script:

#!/bin/bash
sudo apt update && sudo apt upgrade -y
sudo apt install -y nginx
sudo systemctl enable nginx
sudo systemctl start nginx
echo "<html><body><h1>Welcome to DevOps Stage 0 - Afeez Adeboye/A.A</h1></body></html>" | sudo tee /var/www/html/index.html
sudo systemctl restart nginx

3. Encountered Issues and Fixes

Issue 1: Port 80 Not Open

Since I relied on user-data for configuration, I initially forgot to open port 80 in the security group. As a result, I thought NGINX was not working. Fix: I manually updated the security group to allow HTTP (port 80).

Issue 2: Browser Redirecting to HTTPS

When I tried accessing the instance via a web browser, it kept redirecting to HTTPS, but the project only required HTTP. Fix: I had to explicitly type http:// instead of relying on automatic redirection.

Final Verification

Once the fixes were applied:

Visiting http://<EC2-PUBLIC-IP> displayed:

Learning & Takeaways

This task reinforced key DevOps concepts, including:
✅ Automating server setup using user-data.
✅ Managing AWS security groups effectively.
✅ Troubleshooting web server issues related to ports and protocols.
✅ Understanding how NGINX serves static files.

Hiring DevOps and Cloud Engineers

If you're looking to hire skilled professionals in DevOps or Cloud Engineering, check out these resources:

Implementing a Complete GitOps Pipeline: Infrastructure Provisioning and Application Deployment Automation

Afeez Oluwashina Adeboye — Sun, 15 Dec 2024 22:17:42 +0000

Introduction

In this article, I'll walk you through implementing a complete GitOps pipeline that handles both infrastructure provisioning and application deployment. We'll use a combination of Terraform, Ansible, and Docker to create a fully automated CI/CD pipeline that follows best practices and maintains clear separation of concerns.

🔗 Project Repository: DevOps Pipeline Project

You can find all the code and configurations discussed in this article in the repository above. Feel free to star ⭐ it if you find it helpful!

In modern DevOps practices, automation is key to maintaining reliable and efficient software delivery. This project demonstrates the implementation of a comprehensive GitOps pipeline that handles both infrastructure provisioning and application deployment through automated workflows. By leveraging tools such as Terraform, Ansible, and Docker, along with GitHub Actions, we've created a system that ensures consistent infrastructure deployment, cost-aware planning, and automated application updates.

The solution addresses common challenges in DevOps practices:

How to maintain infrastructure as code with proper validation and cost control
How to automate monitoring system deployment alongside infrastructure
How to manage application deployments with proper versioning and rollout
How to keep infrastructure and application deployments separate yet coordinated

Architecture Overview

Project Overview

Core Components and Workflow Structure

The project is organized into two main pipelines:

Infrastructure Pipeline (infra_features → infra_main)
- Handles all infrastructure provisioning through Terraform
- Automatically deploys monitoring stack using Ansible
- Includes cost estimation for infrastructure changes
- Located in main branch for centralized workflow management and also in each branch for proper triggers
Application Pipeline (integration → deployment)
- Manages application container builds and deployments
- Handles Docker image versioning and updates
- Automates deployment to provisioned infrastructure

Workflow Triggers and Placement

Infrastructure Workflows:
- terraform-validate.yml: Triggers on push to infra_features
- terraform-plan.yml: Triggers on PR to infra_main
- terraform-apply.yml: Triggers on PR merge to infra_main
- ansible-monitoring.yml: Triggers after successful terraform apply
Application Workflows:
- ci-application.yml: Triggers on push to integration
- cd-application.yml: Triggers on PR merge to deployment

Expected Outcomes

Infrastructure Pipeline:
- Automated validation of Terraform configurations
- Cost estimation in PR comments
- Provisioned AWS infrastructure
- Deployed monitoring stack (Prometheus, Grafana, etc.)
Application Pipeline:
- Built and versioned Docker images
- Updated docker-compose configurations
- Deployed application stack to infrastructure

The separation of concerns is maintained through:

Different branches for infrastructure and application code
Separate workflows for different stages of deployment
Clear triggers that prevent unintended deployments

This structure ensures that:

Infrastructure changes are validated and cost-estimated before deployment
Application deployments only occur on properly provisioned infrastructure
Monitoring is always in place before application deployment
Changes can be tracked and reversed if needed.

Infrastructure Configuration Pipeline Deep Dive

terraform-validate.yml

This workflow is our first quality gate for infrastructure changes. While it exists in the main branch for centralization, it must also exist in the infra_features branch to properly trigger on push events.

name: Terraform Validate
run-name: ${{ github.actor }} triggered Terraform validation

on:
  workflow_dispatch: # Allows manual trigger
  push:
    branches:
      - infra_features # Trigger on pushes to infra_features branch
    paths:
      - 'terraform/**' # Only trigger if files in the terraform directory change

env:
  # AWS Credentials
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  TF_VAR_aws_region: ${{ vars.AWS_REGION }}

jobs:
  validate:
    name: Terraform Validate
    runs-on: ubuntu-latest

    steps:
      - name: Checkout Specific Branch
        uses: actions/checkout@v3
        with:
          ref: ${{ github.ref }}

      - name: Debug Branch Information
        run: |
          echo "Triggered by branch: ${{ github.ref }}"
          echo "Current branch: $(git branch --show-current)"

      - name: Set up Terraform
        uses: hashicorp/setup-terraform@v2

      - name: Validate Terraform Formatting
        id: fmt-check
        run: terraform fmt -check
        working-directory: terraform
        continue-on-error: true  # Continue even if there are formatting issues

      - name: Fix Terraform Formatting Issues
        if: failure()  # Run only if the previous step fails
        run: terraform fmt
        working-directory: terraform

      - name: Terraform Init
        id: init
        run: terraform init
        working-directory: terraform 

      - name: Terraform Validate
        id: validate
        run: terraform validate
        working-directory: terraform

Key Components:

Triggers:
- Push events to infra_features branch
- Manual trigger via workflow_dispatch
- Path filters for terraform directory changes
Environment Setup:
- AWS credentials configuration
- Region specification from variables
Validation Steps:
- Checkout code
- Setup Terraform
- Check and fix Terraform formatting
- Initialize and validate Terraform configurations

The workflow ensures:

Early detection of configuration errors
Consistent code formatting
Basic syntax and configuration checks
Automatic format fixing when issues are found
Immediate feedback to developers

terraform-plan.yml

This workflow handles infrastructure planning and cost estimation. It exists in both main and infra_main branches to provide cost insights before any infrastructure changes are approved.

name: Terraform Plan and Cost Estimation
on:
  workflow_dispatch:
  pull_request:
    branches:
      - infra_main
    types:
      - opened
      - synchronize
      - reopened
permissions:
  pull-requests: write

env:
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  TF_VAR_aws_region: ${{ vars.AWS_REGION }}


  TF_VAR_ami_id: ${{ vars.AMI_ID }}
  TF_VAR_instance_type: ${{ vars.INSTANCE_TYPE }}
  TF_VAR_key_pair_name: ${{ vars.KEY_PAIR_NAME }}
  TF_VAR_instance_name: ${{ vars.INSTANCE_NAME }}
  TF_VAR_domain_name: ${{ vars.DOMAIN_NAME }}
  TF_VAR_frontend_domain: ${{ vars.FRONTEND_DOMAIN }}
  TF_VAR_db_domain: ${{ vars.DB_DOMAIN }}
  TF_VAR_traefik_domain: ${{ vars.TRAEFIK_DOMAIN }}
  TF_VAR_grafana_domain: ${{ vars.GRAFANA_DOMAIN }}
  TF_VAR_prometheus_domain: ${{ vars.PROMETHEUS_DOMAIN }}
  TF_VAR_cert_email: ${{ vars.CERT_EMAIL }}
  TF_VAR_private_key_path: ${{ vars.PRIVATE_KEY_PATH }}
  TF_VAR_app_dir: ${{ vars.APP_DIR }}
  TF_VAR_repo: ${{ vars.REPO }}

jobs:
  terraform-plan:
    name: Terraform Plan and Cost Estimation
    runs-on: ubuntu-latest

    steps:
      - name: Checkout PR Branch
        uses: actions/checkout@v3
        with:
          ref: ${{ github.head_ref }}

      - name: Checkout Base Branch
        uses: actions/checkout@v3
        with:
          ref: ${{ github.base_ref }}
          path: base-branch

      - name: Debug Branch Information
        run: |
          echo "Base branch: ${{ github.base_ref }}"
          echo "Head branch: ${{ github.head_ref }}"

      - name: Prepare Terraform
        uses: hashicorp/setup-terraform@v2

      - name: Initialize Terraform Configuration
        run: terraform init
        working-directory: terraform

      - name: Generate Terraform Plan
        id: tf_plan
        run: |
          terraform plan -out=tfplan -lock=false
          terraform show -no-color tfplan > /tmp/plan_output.txt
        working-directory: terraform

      - name: Install Cost Analysis Tool
        uses: infracost/actions/setup@v2
        with:
          api-key: ${{ secrets.INFRACOST_API_KEY }}

      - name: Perform Base Branch Cost Breakdown
        run: |
          cd base-branch/terraform
          infracost breakdown --path=. --format=json > /tmp/base_cost_analysis.json
        continue-on-error: true

      - name: Perform Current Branch Cost Breakdown
        run: |
          cd terraform
          infracost breakdown --path=. --format=json > /tmp/current_cost_analysis.json
          infracost breakdown --path=. --format=table > /tmp/cost_summary.txt

      - name: Generate Cost Difference
        run: |
          infracost diff \
            --path=terraform \
            --compare-to=/tmp/base_cost_analysis.json \
            --format=json > /tmp/cost_difference.json || true
        continue-on-error: true

      - name: Prepare Workflow Report
        uses: actions/github-script@v6
        if: github.event_name == 'pull_request'
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          script: |
            const fs = require('fs');

            const planContent = fs.readFileSync('/tmp/plan_output.txt', 'utf8');
            const costSummary = fs.readFileSync('/tmp/cost_summary.txt', 'utf8');

            const commentBody = `### 🔍 Infrastructure Validation Report

            <details>
            <summary>📋 Terraform Plan Insights 📋</summary>

            \`\`\`terraform
            ${planContent}
            \`\`\`
            </details>

            <details>
            <summary>💰 Cost Estimation Overview 💰</summary>

            \`\`\`
            ${costSummary}
            \`\`\`
            </details>

            *Analysis triggered by @${{ github.actor }}*`;

            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: commentBody
            });

Key Components:

Triggers:
- Pull requests to infra_main branch
- PR events: opened, synchronized, reopened
- Manual workflow dispatch for testing
Environment Setup:
- AWS credentials from secrets
- Comprehensive Terraform variables including:
  - Infrastructure settings
  - Domain configurations
  - Application paths
  - Service endpoints
Planning Process:
- Format validation
- Base branch comparison
- Plan generation
- Cost analysis via Infracost
PR Integration:
- Automated PR comments with:
  - Infrastructure changes
  - Cost estimations
  - Resource modifications
  - Base vs proposed comparisons

The workflow ensures:

Detailed cost breakdowns
Change cost implications
Resource-specific pricing
Cost comparison with current state
Clear plan presentation in PR comments

Important: The workflow requires an Infracost API key stored in GitHub secrets for cost estimation features to work properly.

terraform-apply.yml

This workflow is responsible for actual infrastructure deployment. It exists in both main and infra_main branches to ensure proper triggering on PR merges and to allow manual infrastructure management.

name: Terraform Infrastructure Apply
on:
  pull_request:
    branches:
      - 'infra_main'  
    types:
      - closed  

  workflow_dispatch:
    inputs:
      action:
        type: choice
        description: 'Select the action to perform'
        required: true
        default: 'destroy'
        options:
          - 'destroy'
          - 'apply'

env:
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  TF_VAR_aws_region: ${{ vars.AWS_REGION }}

# Terraform Variables to be passed as environment variables
  TF_VAR_ami_id: ${{ vars.AMI_ID }}
  TF_VAR_instance_type: ${{ vars.INSTANCE_TYPE }}
  TF_VAR_key_pair_name: ${{ vars.KEY_PAIR_NAME }}
  TF_VAR_instance_name: ${{ vars.INSTANCE_NAME }}
  TF_VAR_domain_name: ${{ vars.DOMAIN_NAME }}
  TF_VAR_frontend_domain: ${{ vars.FRONTEND_DOMAIN }}
  TF_VAR_db_domain: ${{ vars.DB_DOMAIN }}
  TF_VAR_traefik_domain: ${{ vars.TRAEFIK_DOMAIN }}
  TF_VAR_grafana_domain: ${{ vars.GRAFANA_DOMAIN }}
  TF_VAR_prometheus_domain: ${{ vars.PROMETHEUS_DOMAIN }}
  TF_VAR_cert_email: ${{ vars.CERT_EMAIL }}
  TF_VAR_private_key_path: ${{ vars.PRIVATE_KEY_PATH }}
  TF_VAR_app_dir: ${{ vars.APP_DIR }}
  TF_VAR_repo: ${{ vars.REPO }}

jobs:
  terraform-apply:
    name: Terraform Infrastructure Apply
    runs-on: ubuntu-latest

    # Trigger on merged PR to infra_main or manual destroy
    if: >
      (github.event_name == 'pull_request' && 
       github.event.pull_request.merged == true && 
       github.base_ref == 'infra_main') ||
      (github.event_name == 'workflow_dispatch')

    steps:
      - name: Checkout PR Branch
        uses: actions/checkout@v3
        with:
          ref: ${{ github.head_ref }}

      - name: Install Terraform
        uses: hashicorp/setup-terraform@v2

      - name: Initialize Terraform Configuration
        run: terraform init
        working-directory: terraform

      - name: Terraform Apply
        if: |
          github.event_name == 'pull_request' && 
          github.event.pull_request.merged == true || 
          (github.event_name == 'workflow_dispatch' && github.event.inputs.action == 'apply')
        run: terraform apply --auto-approve -lock=false
        working-directory: terraform

      - name: Terraform Destroy
        if: github.event_name == 'workflow_dispatch' && github.event.inputs.action == 'destroy'
        run: terraform destroy --auto-approve -lock=false
        working-directory: terraform

      - name: Create Action Indicator
        run: echo ${{ github.event.inputs.action }} > ./ansible/action_indicator.txt

      # Upload inventory, trigger  and key as artifacts
      - name: Upload Inventory and Key
        if: github.event_name != 'workflow_dispatch' || github.event.inputs.action != 'destroy'
        uses: actions/upload-artifact@v4
        with:
          name: infrastructure-artifacts
          path: |
            ansible/inventory.ini
            ansible/action_indicator.txt
            terraform/${{ vars.KEY_PAIR_NAME }}
          retention-days: 1

      - name: Save Infrastructure Details as Secrets
        if: github.event_name != 'workflow_dispatch' || github.event.inputs.action != 'destroy'
        run: |
          # First check if files exist
          echo "Checking infrastructure files..."
          if [ ! -f "terraform/${{ vars.KEY_PAIR_NAME }}" ] || [ ! -f "ansible/inventory.ini" ]; then
            echo "Required files not found!"
            exit 1
          fi

          echo "Reading infrastructure files..."
          # Read files and encode in base64 to handle multiline content safely
          SSH_KEY=$(cat "terraform/${{ vars.KEY_PAIR_NAME }}" | base64 -w 0)
          INVENTORY=$(cat "ansible/inventory.ini" | base64 -w 0)

          echo "Saving to GitHub Secrets..."
          # Save SSH key
          gh secret set EC2_SSH_KEY --body "$SSH_KEY"
          # Save inventory
          gh secret set EC2_INVENTORY --body "$INVENTORY"

          echo "Infrastructure details saved as secrets successfully!"
        env:
          GITHUB_TOKEN: ${{ secrets.PAT_TOKEN }}
          GH_TOKEN: ${{ secrets.PAT_TOKEN }}

Key Components:

Triggers:
- PR merge to infra_main branch
- Manual workflow dispatch with options:
  - Apply: For manual infrastructure deployment
  - Destroy: For infrastructure teardown
Environment Configuration:
- AWS credentials from secrets
- Terraform variables for:
  - AWS region
  - AMI configuration
  - Domain settings
  - Application directories etc
Critical Operations:
- Infrastructure deployment via Terraform apply
- Infrastructure destruction (manual trigger)
- Creation of artifacts for downstream workflows
- Important: Saves infrastructure details as GitHub secrets:
  - EC2_SSH_KEY: Base64 encoded SSH key
  - EC2_INVENTORY: Base64 encoded inventory file
  - These secrets are crucial for CD-application workflow

Note: The saving of infrastructure details as secrets is crucial for the CD-application workflow which we'll cover later. These secrets enable secure access to the deployed infrastructure.

ansible-monitoring.yml and Associated Playbook

This workflow automates our monitoring stack deployment, being triggered automatically after successful infrastructure provisioning. The workflow exists in both main and infra_main branches for proper functionality.

name: Monitoring Stack Deployment

on:
  workflow_run:
    workflows: ["Terraform Infrastructure Apply"]
    types:
      - completed

jobs:
  ansible-monitoring:
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    name: Monitoring Stack Deployment
    runs-on: ubuntu-latest

    steps:
      - name: Checkout infra_main Branch
        uses: actions/checkout@v4
        with:
          ref: infra_main  # Explicitly checkout infra_main branch
          fetch-depth: 0   # Get full history

      - name: Download artifacts
        uses: dawidd6/action-download-artifact@v3
        with:
          workflow: terraform-apply.yml
          workflow_conclusion: success
          name: infrastructure-artifacts
          path: ./artifacts

      - name: Check Action Indicator
        id: check-action
        run: |
          ACTION=$(cat ./artifacts/ansible/action_indicator.txt)
          echo "Action: $ACTION"
          if [ "$ACTION" != "apply" ]; then
            echo "Monitoring stack deployment skipped due to action: $ACTION"
            exit 0
          fi

      # Copy the key to the terraform directory
      - name: Setup SSH Key
        run: |
            cp ./artifacts/terraform/${{ vars.KEY_PAIR_NAME }} ./terraform/
            chmod 600 ./terraform/${{ vars.KEY_PAIR_NAME }}

      - name: Update Inventory File
        run: |
          sed -i "s|ansible_ssh_private_key_file=\./${{ vars.KEY_PAIR_NAME }}|ansible_ssh_private_key_file=./terraform/${{ vars.KEY_PAIR_NAME }}|" ./artifacts/ansible/inventory.ini
          cat ./artifacts/ansible/inventory.ini  # Debug print

      - name: Set up Ansible
        uses: alex-oleshkevich/setup-ansible@v1.0.1
        with:
          version: "9.3.0"


      - name: Run Ansible Playbook
        run: |
          ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i ./artifacts/ansible/inventory.ini ./ansible/playbook.yml \
              --extra-vars "domain_name=${{ vars.DOMAIN_NAME }} \
                traefik_domain=${{ vars.TRAEFIK_DOMAIN }} \
                cert_email=${{ vars.CERT_EMAIL }}\
                repo=${{ vars.REPO }}\
                app_dir=${{ vars.APP_DIR }}\
                branch=infra_main"

Key Components:

Workflow Triggers:
- Automatic trigger after terraform-apply completion
- Checks action indicator to ensure proper execution should terraform apply workflow was only triggered for destroy
- Only proceeds on 'apply' action
Infrastructure Access:
- Downloads terraform-generated artifacts
- Configures SSH access using infrastructure keys
- Updates inventory file paths and permissions
Monitoring Setup:
- Deploys complete monitoring stack via Ansible
- Configures domains and certificates
- Sets up repository and application directory

Associated Ansible Playbook Overview

main playbook
- name: Setting up application and monitoring servers
  hosts: all
  become: yes
  roles:
    - docker
    - file_setup
    - monitoring_setup
--------
file setup task
- name: Clone the repository
  git:
    repo: "{{ repo }}"
    dest: "{{ app_dir }}"
    version: "{{ branch }}"


- name: Ensure the Traefik directory exists
  file:
    path: "{{ app_dir }}/traefik"
    state: directory
    mode: '0755'

- name: Create acme.json with proper permissions
  file:
    path: "{{ app_dir }}/traefik/acme.json"
    state: touch
    mode: '0600'

- name: Check if the data folder exists
  stat:
    path: "{{ app_dir }}/data"
  register: data_folder

- name: Ensure Loki data folder has the correct ownership and permissions
  block:
    - name: Change ownership of data folder for Loki
      command: chown -R 10001:10001 ./data
      args:
        chdir: "{{ app_dir }}"

    - name: Set permissions for the data folder
      command: chmod -R 755 ./data
      args:
        chdir: "{{ app_dir }}"
  when: data_folder.stat.exists

- name: Configure monitoring compose file using template
  template:
    src: "monitoring_stack_template.j2"
    dest: "{{ app_dir }}/monitoring-stack.yml"
    mode: '0644'
-----
the monitoring task
- name: Bring up the monitoring stack
  command: docker compose -f monitoring-stack.yml up -d
  args:
    chdir: "{{ app_dir }}"

The playbook orchestrates the entire monitoring setup through three distinct roles:

Docker Role:
- Ensures Docker is installed and configured
- Sets up required Docker services
- Configures networking prerequisites
File Setup Role:
- Clones configuration repository
- Creates necessary directory structure
- Sets up Traefik and SSL configurations
- Manages permissions and ownership
- Prepares data persistence directories
Monitoring Setup Role:
- Deploys the complete monitoring stack
- Configures Prometheus, Grafana, and Loki
- Sets up reverse proxy with Traefik
- Ensures all services are running

The workflow ensures:

Setup Integrity:
- Proper directory structure
- Correct file permissions
- Required configurations
Security Measures:
- Secure credential handling
- Protected certificates
- Proper file permissions
Monitoring Components:
- Prometheus for metrics
- Grafana for visualization
- Loki for logs
- Traefik for routing

Note: This workflow is dependent on the artifacts generated by terraform-apply.yml, showcasing the workflow dependencies in our infrastructure pipeline and it is located both in the main branch and the infra_main branch.

ci-application.yml

This workflow manages the continuous integration process for our application, building and pushing Docker images while updating deployment configurations(compose file).

name: Application CI Pipeline

on:

  push:
    branches:
      - 'integration'
    paths:
      - 'backend/**'
      - 'frontend/**'

env:
  DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
  DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
  IMAGE_TAG: 1.0.${{ github.run_number }}
  FRONTEND_IMAGE: frontend-dojodevops
  BACKEND_IMAGE: backend-dojodevops

jobs:
  build-and-push:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITHUB_TOKEN }} 
      - name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build and push backend
        uses: docker/build-push-action@v5
        with:
          context: ./backend
          push: true
          tags: |
            ${{ secrets.DOCKERHUB_USERNAME }}/${{ env.BACKEND_IMAGE }}:${{ env.IMAGE_TAG }}
            ${{ secrets.DOCKERHUB_USERNAME }}/${{ env.BACKEND_IMAGE }}:latest

      - name: Build and push frontend
        uses: docker/build-push-action@v5
        with:
          context: ./frontend
          push: true
          tags: |
            ${{ secrets.DOCKERHUB_USERNAME }}/${{ env.FRONTEND_IMAGE }}:${{ env.IMAGE_TAG }}
            ${{ secrets.DOCKERHUB_USERNAME }}/${{ env.FRONTEND_IMAGE }}:latest


      - name: Update docker-compose.yml
        run: |
          echo "=== Current docker-compose template content ==="
          cat ./ansible/roles/app_deploy/templates/docker-compose.yml.j2

          echo -e "\n=== Attempting to update image tags ==="
          # Update backend image
          sed -i "s|image: ${{ secrets.DOCKERHUB_USERNAME }}/${{ env.BACKEND_IMAGE }}:.*|image: ${{ secrets.DOCKERHUB_USERNAME }}/${{ env.BACKEND_IMAGE }}:${{ env.IMAGE_TAG }}|" ./ansible/roles/app_deploy/templates/docker-compose.yml.j2
          # Update frontend image
          sed -i "s|image: ${{ secrets.DOCKERHUB_USERNAME }}/${{ env.FRONTEND_IMAGE }}:.*|image: ${{ secrets.DOCKERHUB_USERNAME }}/${{ env.FRONTEND_IMAGE }}:${{ env.IMAGE_TAG }}|" ./ansible/roles/app_deploy/templates/docker-compose.yml.j2

          echo -e "\n=== Updated docker-compose template content ==="
          cat ./ansible/roles/app_deploy/templates/docker-compose.yml.j2

          # Check if any changes were made
          if git diff --quiet ./ansible/roles/app_deploy/templates/docker-compose.yml.j2; then
            echo "No changes were made to docker-compose template"
            echo "Current content of docker-compose template:"
            cat ./ansible/roles/app_deploy/templates/docker-compose.yml.j2
            exit 1
          else
            echo "Changes detected in docker-compose template:"
            git diff ./ansible/roles/app_deploy/templates/docker-compose.yml.j2
          fi

      # Using GitHub's default token for authentication
      - name: Commit and push updated docker-compose
        run: |
          git config --global user.name "${{ github.actor }}"
          git config --global user.email "${{ github.actor }}@users.noreply.github.com"
          git add ./ansible/roles/app_deploy/templates/docker-compose.yml.j2
          git commit -m "Update image tags to ${{ env.IMAGE_TAG }}"
          git push origin integration

Key Components:

Triggers:
- Push events to integration branch
- Path filters for backend and frontend directories
Environment Setup:
- Docker Hub credentials
- Image versioning using GitHub run number
- Separate images for frontend and backend
Build Process:
- Docker Buildx setup for multi-platform builds
- Parallel builds for frontend and backend
- Tag management with latest and versioned tags
- Docker compose template updates

The workflow ensures:

Automated image builds
Version control of images
Configuration updates
Proper image tagging
Automated commit of updated configurations

cd-application.yml

This workflow handles the continuous deployment of our application, utilizing infrastructure secrets from previous workflows.

name: Application CD Pipeline

on:
  workflow_dispatch:
  pull_request:
    branches:
      - 'deployment'
    types:
      - closed

jobs:
  application-deploy:
    if: github.event.pull_request.merged == true
    runs-on: ubuntu-latest
    steps:
      - name: Checkout deployment branch
        uses: actions/checkout@v4
        with:
          ref: deployment

      - name: Setup Infrastructure Files
        run: |
          # Create directories
          mkdir -p ./tmp/ ./tmp/ansible

          # Decode and save SSH key
          echo "${{ secrets.EC2_SSH_KEY }}" | base64 -d > ./tmp/${{ vars.KEY_PAIR_NAME }}
          chmod 600 ./tmp/${{ vars.KEY_PAIR_NAME }}

          # Decode and save inventory
          echo "${{ secrets.EC2_INVENTORY }}" | base64 -d > ./tmp/ansible/inventory.ini

      - name: Update Inventory File
        run: |
          sed -i "s|ansible_ssh_private_key_file=\./${{ vars.KEY_PAIR_NAME }}|ansible_ssh_private_key_file=./tmp/${{ vars.KEY_PAIR_NAME }}|" ./tmp/ansible/inventory.ini

      - name: Set up Ansible
        uses: alex-oleshkevich/setup-ansible@v1.0.1
        with:
          version: "9.3.0"

      - name: Run Application Deployment Playbook
        run: |
          ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i ./tmp/ansible/inventory.ini ./ansible/playbook.yml \
            --extra-vars '{
              "repo": "${{ vars.REPO }}",
              "app_dir": "${{ vars.APP_DIR }}",
              "backend_env": ${{ toJSON(secrets.BACKEND_ENV) }},
              "frontend_env": ${{ toJSON(secrets.FRONTEND_ENV) }},
              "frontend_domain": "${{ vars.FRONTEND_DOMAIN }}",
              "DOCKERHUB_USERNAME": "${{ secrets.DOCKERHUB_USERNAME }}",
              "FRONTEND_IMAGE": "frontend-dojodevops",
              "BACKEND_IMAGE": "backend-dojodevops",
              "db_domain": "${{ vars.DB_DOMAIN }}",
              "branch": "deployment"
            }'

Key Components:

Triggers:
- Pull request merge to deployment branch
- Manual workflow dispatch
Infrastructure Setup:
- Utilizes secrets from terraform-apply:
  - EC2_SSH_KEY for server access
  - EC2_INVENTORY for server details
Deployment Process:
- Ansible configuration
- Environment setup
- Application deployment

The workflow ensures:

Secure deployment process
Environment configuration
Infrastructure access
Application setup by triggering the ansible playbook

Important Environment Variables and Secrets Guide:

Infrastructure Secrets (Generated by terraform-apply):

EC2_SSH_KEY: Base64 encoded SSH key for EC2 access

EC2_INVENTORY: Base64 encoded Ansible inventory file

Docker Hub Credentials:

DOCKERHUB_USERNAME: Docker Hub account username

DOCKERHUB_TOKEN: Docker Hub access token

Application Environment Variables:

BACKEND_ENV: JSON object containing backend environment variables

FRONTEND_ENV: JSON object containing frontend environment variables

Domain Configuration:

FRONTEND_DOMAIN: Domain for frontend service

DB_DOMAIN: Domain for database service

Repository Settings:

REPO: Repository URL

APP_DIR: Application directory path

KEY_PAIR_NAME: Name of the SSH key pair

Note: The CD workflow showcases the importance of proper secret management, using infrastructure secrets generated during provisioning for secure deployment access.

Clean Up

This can be done by manually triggering terraform destroy.

Complete Guide to Containerizing and Deploying a Full-Stack Application with Docker Compose, Nginx Manager, and Monitoring Tools

Afeez Oluwashina Adeboye — Sat, 23 Nov 2024 13:30:52 +0000

Containerizing a Full-Stack Application with Advanced Monitoring

A. Introduction
B. Prerequisites
C. Project Setup
D. Backend Configuration
E. Frontend Configuration
F. Docker Setup
G. Docker Compose Configuration
H. Reverse Proxy & SSL Setup
I. Database Management
F. Setting up the monitoring stack

Introduction

In this project, we will containerize and deploy a multi-tier web application using Docker, Docker Compose, and reverse proxy configurations. The application consists of a React frontend that interacts with a FastAPI backend, which is backed by a PostgreSQL database. To ensure smooth monitoring and management of the application's performance, we will integrate a Monitoring Stack that includes Prometheus for metrics collection, Grafana for data visualization, Loki for log aggregation, and cAdvisor for container performance monitoring.

The overall architecture will use Docker Compose for orchestration, ensuring that all services are interconnected and can communicate seamlessly. The services will be routed through a reverse proxy, either Traefik or Nginx Proxy Manager, to ensure proper URL routing, security, and access control for both the application stack and the monitoring stack.

Prerequisites

Linux-based operating system (Ubuntu, CentOS, etc.)
Docker and Docker Compose installed
Cloud provider account (AWS, Google Cloud, or Azure) for deployment
Domain name
Basic knowledge of:
- Docker & Docker Compose
- React & FastAPI
- Prometheus, Grafana, Loki, and cAdvisor

Project Setup

A. Clone the repository:

git clone https://github.com/The-DevOps-Dojo/cv-challenge01.git
cd cv-challenge01

Backend Configuration

A. Navigate to the backend directory:

cd backend

B. Install Poetry (dependency manager):

For macOS/Linux:

 curl -sSL https://install.python-poetry.org | python3 -

For Windows: Follow Poetry's documentation

C. Verify Poetry installation:

poetry --version

D. Install project dependencies:

poetry install

PostgreSQL Setup

A. Install PostgreSQL:

sudo apt update
sudo apt install postgresql postgresql-contrib

B. Switch to PostgreSQL user:

sudo -i -u postgres
psql

C. Create database and user:

CREATE USER app WITH PASSWORD 'changethis123';
CREATE DATABASE app;
\c app
GRANT ALL PRIVILEGES ON DATABASE app TO app;
GRANT ALL PRIVILEGES ON SCHEMA public TO app;
\q
exit

D. Configure environment variables:
Create/edit .env file in the backend directory:

POSTGRES_SERVER=localhost
POSTGRES_PORT=5432
POSTGRES_DB=app
POSTGRES_USER=app
POSTGRES_PASSWORD=changethis123

E. Initialize database:

poetry run bash ./prestart.sh

F. Start backend service:

poetry run uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

Frontend Configuration

A. In a new terminal, navigate to frontend:

cd frontend

B. Install Node.js and dependencies:

sudo apt update
sudo apt install nodejs npm
npm install

C. Start frontend server:

npm run dev -- --host

When attempting to log in, you might encounter a network error:

Frontend Configuration Updates

A. Update API URL in frontend .env:

VITE_API_URL=http://<your_server_IP>:8000

After updating the frontend configuration, you may encounter a CORS error:

B. Update CORS settings in backend .env:

BACKEND_CORS_ORIGINS="http://localhost,http://localhost:5173,https://localhost,https://localhost:5173,http://<your_server_IP>:5173"

C. Use these default login credentials:

Username: chanllenge@devopsdojo.com
Password: devopsdojo57

After successful configuration, you should see:

Accessing Swagger Documentation

Access the Swagger API documentation at http://<your_server_IP>:8000/docs:

Docker Setup

Backend Dockerfile

Create a Dockerfile in the backend directory:

FROM tiangolo/uvicorn-gunicorn-fastapi:python3.10
WORKDIR /app/
RUN curl -sSL https://install.python-poetry.org | POETRY_HOME=/opt/poetry python && \
    cd /usr/local/bin && \
    ln -s /opt/poetry/bin/poetry && \
    poetry config virtualenvs.create false
COPY ./pyproject.toml ./poetry.lock* /app/
RUN poetry install --no-root --only main
ENV PYTHONPATH=/app
COPY ./alembic.ini /app/
COPY ./prestart.sh /app/
COPY ./app /app/app

# Ensure prestart.sh is executable
RUN chmod +x /app/prestart.sh

# Override the CMD to run the prestart.sh script and then start Uvicorn
CMD ["/bin/bash", "-c", "/app/prestart.sh && uvicorn app.main:app --host 0.0.0.0 --port 8000"]

Frontend Dockerfile

Create a Dockerfile in the frontend directory:

# Use the latest official Node.js image as a base
FROM node:latest

# Set the working directory
WORKDIR /app

# Copy the application files
COPY . .

# Install dependencies
RUN npm install

# Expose the port the development server runs on
EXPOSE 5173

# Run the development server
CMD ["npm", "run", "dev", "--", "--host"]

Docker Compose Configuration

A. Create a docker-compose.yml file in the project root:

services:
  backend:
    build:
      context: ./backend
    container_name: backend_fastapi_app
    ports:
      - "8000:8000"
    depends_on:
      - db
    env_file:
      - ./backend/.env

  frontend:
    build:
      context: ./frontend
    container_name: frontend_nodejs_app
    ports:
      - "5173:5173"
    env_file:
      - ./frontend/.env

  db:
    image: postgres:latest
    container_name: postgres_db
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
    env_file:
      - ./backend/.env

  adminer:
    image: adminer
    container_name: adminer-service
    ports:
      - "8080:8080"

  proxy:
    image: jc21/nginx-proxy-manager:latest
    container_name: nginx_proxy_manager
    ports:
      - "80:80"
      - "443:443"
      - "81:81"
    environment:
      DB_SQLITE_FILE: "/data/database.sqlite"
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
    depends_on:
      - db
      - backend
      - frontend
      - adminer

volumes:
  postgres_data:
  data:
  letsencrypt:

B. Start the services:

# Build and start all services
docker compose up --build -d

# Check service status
docker compose ps

# View logs
docker compose logs -f

Reverse Proxy and SSL Setup

A. Access Nginx Proxy Manager

URL: http://<your-server-ip>:81
Default credentials:
- Email: admin@example.com
- Password: changeme

B. Configure Proxy Hosts for Services

Add Frontend Proxy Host
- Domain: your_domain.com
- Forward Hostname: frontend
- Forward Port: 5173
- Options:
  - Enable Block Common Exploits

SSL:
- Request a Let's Encrypt certificate.
- Enable Force SSL.

C. Configure Advanced Proxy Settings

Go to the Advanced tab for the backend service and add the following routing rules:

   location /api {
       proxy_pass http://backend:8000;
       proxy_set_header Host $host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
   }

   location /docs {
       proxy_pass http://backend:8000;
       proxy_set_header Host $host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
   }

Add Nginx Manager Proxy Host (API)
- Domain: proxy.your_domain.com
- Forward Hostname: proxy
- Forward Port: 81
- SSL: Follow the same steps as above.
Add Adminer Proxy Host
- Domain: db.your_domain.com
- Forward Hostname: db
- Forward Port: 8080
- SSL: Follow the same steps as above.
Redirect www to Non-www:
- Navigate to Proxy Hosts in Nginx Proxy Manager.
- Add a new Proxy Host for www.your_domain.com.
- Forward Hostname: your_domain.com
- Forward Port: 443
- Enable Force SSL.

D. Verify Service Accessibility

After completing the above configurations:

Frontend: https://your_domain.com
Backend API: https://api.your_domain.com
Adminer: https://db.your_domain.com

Database Management

A. Access Adminer

Navigate to https://db.your_domain.com.
Login with the credentials from the backend's .env file:
- System: PostgreSQL
- Server: db

B. Manage Database

Use Adminer to perform tasks like creating tables, managing records, or running SQL queries.
Example Usage:
- List All Tables: Access the left navigation menu in Adminer after logging in.
- Run Queries: Use the SQL tab to execute custom queries.

By following these steps, you have a structured reverse proxy and database management setup. The dashboard ensures www requests are redirected seamlessly, and all services are accessible via specific subdomains.

Setting Up a Complete Monitoring Stack for Your Containerized Application

NOTE: Please update the application compose file with a shared network. This is to make all services communicate internally with each other as shown below

services:
  backend:
    build:
      context: ./backend
    container_name: backend_fastapi_app
    ports:
      - "8000:8000"
    depends_on:
      - db
    env_file:
      - ./backend/.env
    networks:
      - shared-network

  frontend:
    build:
      context: ./frontend
    container_name: frontend_nodejs_app
    ports:
      - "5173:5173"
    env_file:
      - ./frontend/.env
    networks:
      - shared-network

  db:
    image: postgres:latest
    container_name: postgres_db
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
    env_file:
      - ./backend/.env
    networks:
      - shared-network

  adminer:
    image: adminer
    container_name: adminer-service
    ports:
      - "8080:8080"
    networks:
      - shared-network

  proxy:
    image: jc21/nginx-proxy-manager:latest
    container_name: nginx_proxy_manager
    ports:
      - "80:80"
      - "443:443"
      - "81:81"
    environment:
      DB_SQLITE_FILE: "/data/database.sqlite"
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
    depends_on:
      - db
      - backend
      - frontend
      - adminer
    networks:
      - shared-network

volumes:
  postgres_data:
  data:
  letsencrypt:

networks:
  shared-network:
    name: shared-network

Now that we have our application running, let's add comprehensive monitoring capabilities. We'll set up Prometheus for metrics collection, Grafana for visualization, cAdvisor for container metrics, and Loki with Promtail for log aggregation.

Getting Started

First, create a new file called monitoring-compose.yml:

services:
  grafana:
    image: grafana/grafana
    container_name: grafana
    restart: unless-stopped
    environment:
      - GF_SERVER_DOMAIN=your.domain
      - GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s/grafana
      - GF_SERVER_SERVE_FROM_SUB_PATH=true
      - GF_AUTH_PROXY_ENABLED=false
      - GF_SECURITY_ALLOW_EMBEDDING=true
      - GF_SECURITY_COOKIE_SAMESITE=none
      - GF_SECURITY_COOKIE_SECURE=true
      - GF_SESSION_PROVIDER=memory
      - GF_SESSION_PROVIDER_CONFIG=sessions
    ports:
      - "3000:3000"
    volumes:
      - grafana-storage:/var/lib/grafana
    networks:
      - shared-network

  prometheus:
    image: prom/prometheus
    container_name: prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.external-url=/prometheus'
      - '--web.route-prefix=/prometheus'
    ports:
      - "9090:9090"
    restart: unless-stopped
    volumes:
      - ./prometheus:/etc/prometheus 
      - prom_data:/prometheus
    depends_on:
      - cadvisor
    networks:
      - shared-network

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    ports:
      - "8083:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    networks:
      - shared-network

  loki:
    image: grafana/loki:3.0.0
    container_name: loki
    user: "10001"
    volumes:
      - ./loki-config.yaml:/mnt/config/loki-config.yaml
      - ./data:/tmp/loki
    ports:
      - "3100:3100"
    command: -config.file=/mnt/config/loki-config.yaml
    networks:
      - shared-network

  promtail:
    image: grafana/promtail:3.0.0
    container_name: promtail
    depends_on:
      - loki
    volumes:
      - ./promtail-config.yaml:/mnt/config/promtail-config.yaml
      - /var/log:/var/log
    command: -config.file=/mnt/config/promtail-config.yaml
    networks:
      - shared-network

volumes:
  grafana-storage:
  prom_data:

networks:
  shared-network:
    external: true

Important Configuration Notes

Network Configuration

Remember the shared-network we created earlier? Our monitoring stack will use the same network to communicate with our application. If you haven't created it yet:

docker network create shared-network

This network allows our monitoring services to communicate with both the application services and each other.

Grafana Configuration

Looking at the Grafana service, you'll notice several environment variables:

environment:
  - GF_SERVER_DOMAIN=your.domain
  - GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s/grafana
  - GF_SERVER_SERVE_FROM_SUB_PATH=true

Replace your.domain with your actual domain name. These settings ensure Grafana works correctly under a subpath (/grafana) with your domain.

Service Configuration Files

Before starting the stack, you'll need configuration files for Prometheus, Loki, and Promtail:

Create a prometheus.yml file in a new prometheus directory:

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
      - targets: []
      scheme: http
      timeout: 10s
      api_version: v2

scrape_configs:
  - job_name: prometheus
    honor_timestamps: true
    scrape_interval: 15s
    scrape_timeout: 10s
    metrics_path: /prometheus/metrics
    scheme: http
    static_configs:
      - targets: ['prometheus:9090']

  - job_name: cadvisor
    scrape_interval: 5s
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets: ['cadvisor:8080']

Download Loki and Promtail configurations:

# Get Loki config
wget https://raw.githubusercontent.com/grafana/loki/v3.0.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml

# Get Promtail config
wget https://raw.githubusercontent.com/grafana/loki/v3.0.0/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml

Nginx Proxy Manager Configuration

Add these locations to your existing proxy host configuration:

location /prometheus {
    proxy_pass http://prometheus:9090;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

location /grafana {
    proxy_pass http://grafana:3000;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

Starting the Monitoring Stack

Create required directories and set permissions:

# Create directories
mkdir -p prometheus data

# Set Loki permissions (important!)
sudo chown -R 10001:10001 ./data
sudo chmod -R 755 ./data

Start the stack:

docker compose -f monitoring-compose.yml up -d

Verify services are running:

docker compose -f monitoring-compose.yml ps

Accessing Your Monitoring Stack

Once everything is running, you can access:

Grafana: https://your.domain/grafana
Prometheus: https://your.domain/prometheus

Troubleshooting Tips

If you encounter issues:

Check service logs:

docker compose -f monitoring-compose.yml logs grafana
docker compose -f monitoring-compose.yml logs prometheus

Verify network connectivity:

docker network inspect shared-network

Setting Up Your Monitoring Dashboard

Now that our monitoring stack is running, let's configure our dashboards and data sources for effective monitoring.

Adding Data Sources to Grafana

A. Access Grafana at https://your.domain/grafana

Default credentials: admin/admin
You'll be prompted to change the password on first login

B. Configure Prometheus Data Source:

Go to Settings (⚙️) > Data Sources
Click "Add data source"
Select "Prometheus"
Configure:
- Name: Prometheus
- URL: http://prometheus:9090/prometheus #because of the sub-path requirements in our projects
- Access: Server (default)
Click "Save & Test"

Setting Up Loki for Log Aggregation

A. Add Loki Data Source:

Go to Settings (⚙️) > Data Sources
Click "Add data source"
Select "Loki"
Configure:
- Name: Loki
- URL: http://loki:3100
- Access: Server (default)
Click "Save & Test"

References

"My HNG Journey: Stage Two Containerization and Deployment of a Three-Tier Application Using Docker and Nginx Proxy Manager"

A detailed guide on deploying a three-tier application with Docker and Nginx Proxy Manager, which influenced the reverse proxy configuration steps.

Read more on dev.to
Techdox Documentation

Comprehensive documentation for setting up each monitoring services.

Visit Docs Here
Youtube videos

Insightful YouTube videos with insight on deployment strategies and troubleshooting shared, particularly for Loki and Nginx.

Automating User and Group Management with a Bash Script

Afeez Oluwashina Adeboye — Tue, 02 Jul 2024 07:55:42 +0000

Overview

For system administrators, maintaining user accounts and groups can be a tedious chore. Automating this process can save time and the possibility of human error. Our script, create_users.sh, makes this procedure simpler, which reads user and group data from a file and runs the required system commands.

Script Breakdown

Let's break down the script to understand how it works.

Script Initialization

#!/bin/bash

The shebang #!/bin/bash tells the system to execute the script using the Bash shell.

Defining Color Codes

# Color codes
RED="\e[31m"
BLUE="\e[34m"
YELLOW="\e[33m"
YELLOW_ITALIC="\e[3;33m"
RESET="\e[0m"

We define color codes using ANSI escape sequences to make our script's output more readable. These colors will highlight different types of messages, such as errors, successes, and prompts.

Logging Function

# Function to log actions with timestamps and color coding
log() {
    local COLOR="$2"
    local TEXT="$(date +"%Y-%m-%d %T") - $1"

    echo -e "${COLOR}${TEXT}${RESET}" | tee -a $LOG_FILE
}

Timestamped messages are formatted and logged by the log function. It appends these log entries to a log file and shows them on the terminal, using the designated color to distinguish between different log entry types.

Password Generation Function

# Function to generate a random password
generate_password() {
    tr -dc A-Za-z0-9 </dev/urandom | head -c 12
}

This function generates a random 12-character password using the /dev/urandom pseudo-random number generator.

Root User Check

# Check if the script is run as root
if [[ $EUID -ne 0 ]]; then
    echo -e "${RED}This script must be run as root or with sudo${RESET}"
    exit 1
fi

This block checks if the script is being run as the root user or with the sudo command.

Setting Up Log and Password Files

# Default log and password files
LOG_FILE="/var/log/user_management.log"
PASSWORD_FILE="/var/secure/user_passwords.txt"

# Ensure the log and password files exist with secure permissions
mkdir -p /var/secure
touch $LOG_FILE
touch $PASSWORD_FILE
chmod 600 $PASSWORD_FILE

We define the paths for our log file and password file. The mkdir -p /var/secure command creates the secure directory if it doesn't exist.

Input File Check

# Check if an input file is provided, otherwise prompt the user
if [[ "$#" -ne 1 ]]; then
    echo -e "${YELLOW}Enter the filename containing the user information: ${RESET}"
    read INPUT_FILE
else
    INPUT_FILE=$1
fi

# Validate the input file
if [[ ! -f $INPUT_FILE ]]; then
    log "Input file does not exist: $INPUT_FILE" "${RED}"  # Red color for errors
    exit 1
fi

This part checks if the script was given an input file as an argument. If not, it prompts the user to enter the filename. The script then checks if the file exists. If not, it logs an error and exits.

Processing Each Line of the Input File

while IFS=';' read -r username groups; do
    # Remove leading and trailing whitespace
    username=$(echo $username | xargs)
    groups=$(echo $groups | xargs)

    # Check if the username is empty
    if [[ -z "$username" ]]; then
        log "Empty username. Skipping..." "${YELLOW_ITALIC}"  # Yellow color for skipped (italic)
        continue
    fi

    # Check if the user already exists
    if id "$username" &>/dev/null; then
        log "User $username already exists. Skipping..." "${YELLOW_ITALIC}"  # Yellow color for skipped (italic)
        continue
    fi

    # Create the user with a home directory
    useradd -m -s /bin/bash "$username"
    if [[ $? -ne 0 ]]; then
        log "Failed to create user $username. Skipping..." "${RED}"  # Red color for errors
        continue
    fi
    log "Created user $username with home directory /home/$username" "${BLUE}"  # Blue color for success

    # Set home directory permissions
    chown "$username:$username" "/home/$username"
    chmod 700 "/home/$username"
    log "Set permissions for /home/$username" "${BLUE}"  # Blue color for success

    # Create and add the user to additional groups
    IFS=',' read -ra group_array <<< "$groups"
    for group in "${group_array[@]}"; do
        group=$(echo $group | xargs)  # Remove whitespace
        if [[ ! $(getent group $group) ]]; then
            groupadd $group
            if [[ $? -eq 0 ]]; then
                log "Created group $group" "${BLUE}"  # Blue color for success
            else
                log "Failed to create group $group. Skipping group assignment for $username." "${RED}"  # Red color for errors
                continue
            fi
        fi
        usermod -aG "$group" "$username"
        log "Added user $username to group $group" "${BLUE}"  # Blue color for success
    done

    # Generate and set a random password for the user
    password=$(generate_password)
    echo "$username,$password" >> $PASSWORD_FILE
    echo "$username:$password" | chpasswd
    log "Set password for user $username" "${BLUE}"  # Blue color for success

done < "$INPUT_FILE"

This is the core part of the script. It processes each line of the input file, which is expected to have the format username;group1,group2,....

Whitespace Removal: We remove leading and trailing whitespaces from usernames and groups.
Empty Username Check: If a username is empty, it logs a message and skips to the next line.
User Existence Check: If the user already exists, it logs a message and skips to the next user.
User Creation: If the user doesn't exist, it creates the user with a home directory and logs the action.
Set Permissions: It sets appropriate permissions for the user's home directory.
Group Management: The script ensures each group exists and adds the user to the specified groups.
Password Management: It generates a random password, sets it for the user, and securely logs it.

Final Log and Script Exit

log "User creation process completed." "${BLUE}"  # Blue color for success

exit 0

The script logs that the user creation process is complete and exits with a status code of 0, indicating success.

How to Run the Script

To execute the script, follow these steps:

Ensure the script has executable permissions:

   bash
   chmod +x create_users.sh

Prepare the input file as shown below;

Lagos;sudo,dev,www-data
Abuja;sudo
Lokoja;dev,www-data

Run the script with the input file as an argument:
```
sudo bash create_users.sh file.txt 
```

Find the complete code Here.

CONCLUSION

Administrative operations can be greatly streamlined by using Bash scripts to automate user and group management. This script shows how to create users, manage groups, handle passwords safely, and read user information from a file.

You are welcome to alter this script to meet your own requirements; just keep in mind that scripts should always be tested in a secure setting before being used in production.

For more insights and opportunities to grow as a developer, check out the HNG Internship and explore how to hire talented developers through the HNG platform.

DEV Community: Afeez Oluwashina Adeboye

Customizing and Deploying a Static Website with NGINX on AWS EC2

Introduction

Approach

1. Launching the EC2 Instance

2. Using User-Data for Automated Setup

3. Encountered Issues and Fixes

Issue 1: Port 80 Not Open

Issue 2: Browser Redirecting to HTTPS

Final Verification

Learning & Takeaways

Hiring DevOps and Cloud Engineers

Implementing a Complete GitOps Pipeline: Infrastructure Provisioning and Application Deployment Automation

Introduction

Architecture Overview

Project Overview

Core Components and Workflow Structure

Workflow Triggers and Placement

Expected Outcomes

Infrastructure Configuration Pipeline Deep Dive

terraform-validate.yml

Key Components:

The workflow ensures:

terraform-plan.yml

Key Components:

The workflow ensures:

terraform-apply.yml

Key Components:

ansible-monitoring.yml and Associated Playbook

Key Components:

Associated Ansible Playbook Overview

The workflow ensures:

ci-application.yml

Key Components:

The workflow ensures:

cd-application.yml

Key Components:

The workflow ensures:

Clean Up

Complete Guide to Containerizing and Deploying a Full-Stack Application with Docker Compose, Nginx Manager, and Monitoring Tools

Containerizing a Full-Stack Application with Advanced Monitoring

Table of Contents

Introduction

Prerequisites

Project Setup

Backend Configuration

PostgreSQL Setup

Frontend Configuration

Frontend Configuration Updates

Accessing Swagger Documentation

Docker Setup

Backend Dockerfile

Frontend Dockerfile

Docker Compose Configuration

Reverse Proxy and SSL Setup

A. Access Nginx Proxy Manager

B. Configure Proxy Hosts for Services

C. Configure Advanced Proxy Settings

D. Verify Service Accessibility

Database Management

A. Access Adminer

B. Manage Database

Setting Up a Complete Monitoring Stack for Your Containerized Application

Getting Started

Important Configuration Notes

Network Configuration

Grafana Configuration

Service Configuration Files

Nginx Proxy Manager Configuration

Starting the Monitoring Stack

Accessing Your Monitoring Stack

Troubleshooting Tips

Setting Up Your Monitoring Dashboard

Adding Data Sources to Grafana

Setting Up Loki for Log Aggregation

References

Automating User and Group Management with a Bash Script

Overview

Script Breakdown

Script Initialization