DEV Community: Tushar Gayakwad

SecureOpsFlow - End-to-End Deployment with Notifications and Secure.🚀

Tushar Gayakwad — Wed, 25 Dec 2024 17:33:47 +0000

✨Introduction

This architecture represents an efficient and secure CI/CD pipeline to improve the development and release process while upholding good code quality, security, and automated approvals. Tools will include GitHub Actions, SonarQube, Docker, and AWS services, all incorporating mechanisms like approval and vulnerability scans using Docker images.

🏗️SecureOpsFlow Architecture

📦SecureOpsFlow Project Setup Prerequisites

Git with GitHub: Version control system and code repository hosting
GitHub Actions: Automation tool for CI/CD flow triggered by code changes
SonarQube: A code quality issue with possible bugs, anti-patterns & security hotspots.
Docker: A tool for packaging applications into the best possible portable images,
Docker Compose: A tool used to define and manage multi-container Docker applications
AWS EC2: Virtual servers in the cloud for hosting and running your Docker containers.
AWS ECR: A host of Docker registries that provides storage, encryption, and cross-region connectivity.
AWS CloudFront: A fast, secure, and inexpensive CDN Service that delivers highly secured content quickly.
GoDaddy: GoDaddy is a site where you can find your website's unique public-facing domain name and register it.

📋Step-by-Step Project Setup and Explanation

🛠️1. Create EC2 Machine for web service deployment

1. Log in to AWS

Move to the AWS Control Console.
Open the EC2 Dashboard.

2. Launch Instance

Continue to Launch Instances for instance type from the "Instances" tab.

3. Instance Configuration.

Name: Input the name of your instance (e.g.: SecureOpsFlow_Web).
Instance Type: Choose t2. medium.
Key pair Create or Select a Key Pair for SSH Access.
Network Settings: Chose security group.
Storage: Customize storage (ie 15 GB, or more if needed).

4. Launch the Instance

Verify your settings, and then choose Launch Instance.

🧩2. Setup a SonarQube on SecureOpsFlow_Web

1. SSH into the EC2 Instance

Use your terminal to connect to the EC2 instance:

ssh -i "your-key.pem" ubuntu@your-ec2-public-ip

2. Install a Docker and Docker-compose

Using these commands:

sudo yum update -y
sudo yum install docker -y
sudo service docker start
sudo usermod -a -G docker $USER
sudo curl -L "https://github.com/docker/compose/releases/download/2.25.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

3. Clone the Repository

Clone your GitHub repository:

git clone https://github.com/TusharGayakwad/SecureOpsFlow.git

Go into the repository directory:

cd SecureOpsFlow

4. Run Docker Compose

Now run the command for running Docker Compose:

docker-compose -f SonarQube_Setup_compose.yml up --build -d

Note: After running this command, open a 9000 port in the instance security group to access a SOnarQube container.

6. Verify the Setup

Check if the containers are up and running:

docker ps

Open a browser and use the EC2 public IP and the configured port for accessing the SonarQube,(e.g., http://:9000 for SonarQube).

7. Configure a Project in SonarQube

Log in to SonarQube : Open your SonarQube instance (e.g., http://instance_public_ip:9000) and log in using the default password (default: admin/admin, change the password after logging in).
Create a project : Go to Projects > Create Project.
(Write a project name and a key that has not yet been used, then click Set Up.)
Generate a Token : Click on Generate Token, name it (e.g., SecureOpsFlow_Token), and store it in a safe location.
Select a Scanner : Choose your build tool (e.g. SonarScanner CLI).
Follow the instructions to download and set up the scanner.
Add Configuration :
Update the configuration in the main.yml file(.github/workflows/main.yml) place of the project key, host URL, and log in.
add a github action secret :
Go on the github> settings>sacrets and variables>add secrets

8. Configure a Project in SonarQube

Navigate to Quality Gates : From the upper menu choose Quality Gates Press Create and give the name.
Add Conditions to Quality Gate : Add a condition by pressing Add Condition. This defines the rules for your Quality Gate. For example:
- Coverage: Coverage is below 80% → FAIL.
- Bugs: The number of Bugs exceeds 0 → FAIL.
- Vulnerabilities: The number of Vulnerabilities is greater than 0 → FAIL.
- Code Smells: Maintainability Rating is below A → FAIL.
Save the Quality Gate - After adding all the desired conditions, click Save.

9. Apply Quality Gate to your project.

Go to your project : Click on Projects and select your project.
Assign the Quality Gate : Go to Administration > Quality Gate. Select the newly created Quality Gate from the list.

10. Update to the developer about the SonarQube quality checks way mail:

If the SonarQube quality check becomes a "FAILED" or "SUCCESS" the pipeline sends a mail to the developer:

Note: Update the username and secrets as follows:
- Username: Use your office email.
- Password: Add a secret named EMAIL_PASSWORD in GitHub Secrets.
- To: Add a secret named DEVELOPER_EMAIL in GitHub Secrets with the developer's email.
- From: Add a secret named OFFICIAL_EMAIL in GitHub Secrets with the office email.

3. Build a Docker Image, Scan with Trivy Scanner Tool, and Push to AWS ECR:

Build a docker image way a docker file:
Scan a docker image way a Trivy:
Trivy is a lightweight and open-source vulnerability scanner built to find security issues in container images, file systems, and source code repositories. Simple and fast to use, it makes an excellent choice for vulnerability or misconfiguration scanning. Trivy works perfectly with DevOps tools and keeps the checks and your projects secure.

Send a vulnerability report to the DevOps team:

Note: Go to the GitHub Secrets and add a secret named DEVOPS_GUY_EMAIL with the DevOps team email.
Configure AWS, Push Images to ECR, and send a final mail to the Manager:

Stage 1: Configuring AWS Credentials

This stage is concerned with configuring the AWS Credentials necessary for performing actions with the ECR Service.
We accomplish this through the aws-actions/configure-aws-credentials action, which accepts inputs due to the following:

Access Key ID
Secret Access Key
Region (ap-south-1)

Note: Log in to AWS Console > Click on User > Go to Security credentials tab > Create Access Key > Download the .csv file.

Stage 2: Logging into ECR and Pushing Docker Image

Log in to Amazon ECR using the amazon-ecr-login action.Tag the Docker image (in this case, it is named secureopsflow) and push it to the ECR repository.
The following commands would achieve this:

docker tag secureopsflow:l atest $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG

Note: Log in to AWS Console > Search ECR > Go to Repositories > Click Create Repository > Enter Repository Name (e.g., secureopsflow) > Click Create → Copy the Repository URI

Stage 3: Sending Notification Email to Manager

The final step in the build stage involves sending an email to the manager once the Docker image is successfully pushed to ECR.

🚀4. Deployment to EC2:

Stage 1:Set up SSH Connection to EC2

Now we need to create a secure SSH connection to the EC2 instance. This is done using the private SSH key stored in a GitHub secret for security purposes.

- name: Set up SSH
uses: webfactory/ssh-agent@v0.5.3
with:
ssh-private-key: ${{ secrets.EC2_SSH_KEY }}

Note: create a secret for the SSH(EC2_SSH_KEY)

Stage 2: Deploy the Application to EC2

Once connected via SSH, you run the deployment commands in the EC2 instance using Docker Compose. The workflow uses the ssh command to log into the EC2 instance and execute deployment commands.

- name: Deploy Application to EC2
  run: |
    ssh -o StrictHostKeyChecking=no ubuntu@${{ secrets.EC2_HOST }} << 'EOF'
    cd /home/ubuntu/SecureOpsFlow
    docker-compose -f compose.yml up -d
    EOF

The stepwise functioning is as follows:

SSH into the EC2 instance with username ubuntu and the EC2 Host IP stored as secret.
Change directory to the application folder (/home/ubuntu/SecureOpsFlow).
Issue the docker-compose up command in the detached mode to start the application. Note: create a secret for the Server IP address(EC2_HOST).

Stage 3: Send Deployment Confirmation Email to the Manager:

The final step of the deployment process sends a confirmation email to the manager upon the deployment's successful completion.

Connect a CloudFront to EC2🌐:

Stage 1: crate Distribution:

In the AWS console, go to CloudFront and create a new distribution:

Origin Domain: Use your EC2 instance’s public IP or DNS as the origin.
Cache Behavior: Set up the caching and protocol policies (HTTP or HTTPS) according to your needs.
Create Distribution: Click Create to set up the distribution.

Stage 2: Update EC2 Security Group:

Allow Traffic: Ensure your EC2 security group allows traffic from CloudFront’s IP ranges.

Configure GoDaddy (If using a custom domain)📝

DNS Settings: In GoDaddy, add a CNAME record to point your domain to the CloudFront distribution URL(https://d1pm7k06iepqhp.cloudfront.net).

Thank you for taking the time to read my blog! I hope you found it insightful and engaging. 💡 Your support means a lot to me, and I look forward to sharing more content with you. Stay tuned! 🚀📚

Docker Networking – Basics, Network Types & Examples

Tushar Gayakwad — Sun, 20 Oct 2024 09:21:17 +0000

Basic Understanding of Docker

Docker is a platform that helps developers package applications and their dependencies into units called containers. Containers are lightweight, standalone environments that can run consistently on any machine with Docker installed, be it a local computer or a remote cloud server. This makes it easy to develop, test, and deploy applications in different environments without worrying about compatibility issues.

Key Concepts in Docker

Docker Image: A Docker image is like a blueprint for containers. It contains everything the application needs to run—such as the code, system tools, libraries, and settings. Images are reusable, so you can use the same image on different machines.

Docker Container: A container is a running instance of a Docker image. It’s isolated from other containers and the host system, ensuring that the application behaves the same everywhere. Containers are lightweight because they share the host machine’s operating system kernel, unlike traditional virtual machines that need a full OS.

Dockerfile: A Dockerfile is a simple text file with instructions for building a Docker image. It defines the base image (e.g., Ubuntu or Alpine), the application code, and the steps required to set up the environment. When you build an image from a Dockerfile, Docker follows the instructions and packages everything into a container-ready image.

for deep knowledge refer to a Docker Doc

Docker Network Types

Docker has a built-in networking system that manages communication between containers, the Docker host, and external networks. It supports different types of networks to handle a range of use cases, ensuring secure and flexible communication within containerized environments.

Docker Networks

bridge
host
overlay
IPvLAN
macvlan
None

1. bridge

Bridge network is a default network, Bridge networks in Docker create a virtual connection between the host system and the containers. Containers in this network can communicate with each other but are isolated from external networks unless specifically configured.

Each container gets its IP address within the bridge network. Through the bridge, containers can access the local network (LAN) and the internet, but they won’t appear as separate physical devices on the LAN. This type of network is commonly used for container communication on a single host.

2. host

In host network mode, Docker containers directly share the host’s network, meaning there’s no network isolation between the container and the host. The container uses the same IP address as the host, and any ports exposed by the container are bound directly to the host’s network. For instance, if a container is configured to listen on port 80, it will bind to the host's IP and port 80 (:80). This setup can improve performance since there’s no virtual network layer, but it sacrifices the isolation and security offered by other Docker network modes. It's often used for specific cases requiring direct access to the host network.

3. overlay

Overlay networks are used in Docker to connect containers running on different hosts, allowing them to communicate as if they were on the same network. This type of network spans multiple Docker hosts and doesn't require OS-level routing, making it useful for scaling applications across distributed systems.

Overlay networks are essential for Docker Swarm clusters, but they can also be used independently to connect containers across separate Docker Engine instances. This enables you to build distributed environments where containers from different hosts can communicate seamlessly, simulating a Swarm-like setup without using it.

4. IPvLAN

IPvlan is a networking option that allows you to manage how IP addresses are assigned to your Docker containers. This means you can better organize your network traffic by tagging it with VLANs, which helps separate different data types. This setup is instrumental if you want to connect your containerized applications directly to a physical network, like your office network while improving performance compared to standard bridge networks. Overall, IPvlan helps create a more efficient and organized network environment.

5. macvlan

Macvlan allows each container to act like a physical device on the network. By assigning a unique MAC address to each container, it makes them recognizable as separate devices. To use macvlan, you need to dedicate one of your host's network interfaces to this purpose. This means the external network must be capable of handling multiple MAC addresses. Macvlan is great for scenarios where containers need to be treated like independent devices, such as for network monitoring or when specific network permissions are required. This way, you get the benefits of containerization while still having the flexibility of physical network devices.

6. None

A "none" Docker network is a special network type where the container doesn't have any network interface, except a loopback interface (lo). Essentially, it isolates the container entirely from any external network connectivity. This is useful when you want to ensure that the container doesn't communicate with the outside world or other containers unless explicitly connected to another network.

In short, a container on the "none" network has no external network access by default.

How Docker Networking Works

Docker uses your host's network to enable containers to communicate. It does this by setting up special rules using IPtables, a tool in Linux that controls how traffic moves through the network. These rules automatically send the right traffic to your containers, so you don't need to set them up manually.

Each Docker container has its virtual network environment, called a network namespace, which keeps it isolated. Docker also creates virtual network interfaces on the host, allowing the containers to communicate with the outside world using the host’s network.

Although the process behind Docker networking is complex, Docker handles everything for you, making it easy to use. You can find more details in Docker's documentation.

Basic commands for Docker Network

Here are the basic commands for creating and managing Docker networks:

1. Create a Docker Network

docker network create my-network

This command creates a network named my-network.

2. List Docker Networks

docker network ls

Shows all the existing Docker networks on your system.

3. Inspect a Docker Network

docker network inspect my-network

Provides detailed information about the my-network, such as connected containers, subnet, and configuration.

4. Remove a Docker Network

docker network rm my-network

Deletes the network my-network (only if no containers are connected to it).

5. Remove all networks with a single command

docker network prune

Creating a Docker Network

To create a new network, use the docker network create command. You can specify the driver (like bridge or host) with the -d flag. If you don't include this flag, Docker will create a bridge network by default.

Run the following command in your first terminal:

$ docker network create demo-network -d bridge
50ed05634f6a3312e56700ef683ca39df44bfc826e2e4da9179c2593c79910f9 ---output

Connecting Containers to Networks

You can connect new containers to a network using the --network flag in the docker run command. In your second terminal , run:

$ docker run -it --rm --name container1 --network demo-network busybox:latest

Next, open your third terminal window and start another container without the --network flag:

$ docker run -it --rm --name container2 busybox:latest

Now, try to communicate between the two containers using their names:

# in container1
/ # ping container2
ping: bad address 'container2'

Since the containers are not on the same network, they can't communicate yet.

To connect container2 to the network, use the first terminal window:

$ docker network connect demo-network container2

Now, both containers share the same network, allowing them to

communicate:
#in container1
/ # ping container2
PING container2 (172.22.0.3): 56 data bytes
64 bytes from 172.22.0.3: seq=0 ttl=64 time=4.205 ms

Using Host Networking

While bridge networks are commonly used to connect containers, you can also use host networking, which connects containers directly to the host's network interfaces. To enable host networking, run:

$ docker run -d --name nginx --network host nginx:latest

With this setup, NGINX listens on port 80 by default, and you can access it via localhost:80 on your host:

$ curl localhost:80

You should see the NGINX welcome page.

Disabling Networking

To completely disable a container's network connectivity, you can attach it to the none network:

$ docker run -it --rm --network none busybox:latest
/ # ping google.com
ping: bad address 'google.com'

This isolates the container, allowing you to sandbox unknown services.

Removing Containers from Networks

Docker allows you to manage network connections without needing to restart your containers. You can remove a container from a network like this:

$ docker network disconnect demo-network container2

Any changes you make will apply immediately.

Using Networks with Docker Compose

You can also use networks with Docker Compose services. When using Compose, services in your stack are automatically added to a shared bridge network, reducing the need for manual configuration. Here's an example Compose file:

version: "3"
services:
  app:
    image: php:7.2-apache
  mysql:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: changeme

Deploy the stack with:

$ docker-compose up -d

You will see the output indicating that the network was created for your stack:

$ docker network ls

Adding Extra Networks

You can define additional networks in your Compose file. Specify the network in the top-level networks field and connect your services by referencing it in the networks field for each service:

version: "3"
services:
  app:
    image: php:7.2-apache
    networks:
      - db
  helper:
    image: custom-image:latest
  mysql:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: changeme
    networks:
      - db
networks:
  db:

In this example, only the app service can communicate with the mysql service, while the helper service cannot reach the database because it’s not on the same network.