DEV Community: taqiyeddinedj

SQUID PROXY SERVER

taqiyeddinedj — Wed, 30 Aug 2023 14:54:36 +0000

HMM, Now it is time to create a caching/proxy server using squid proxy open source project

sudo apt install squid
sudo systemctl enable --now squid

touk@ubuntu:/etc/squid$ tree
.
├── conf.d
│   ├── debian.conf
│   └── local.conf
├── errorpage.css
└── squid.conf

1 directory, 4 files

For simplicity, we’ll create our customize config file in conf.d, and as you can see i have local.conf file under the conf.d directory

touk@ubuntu:/etc/squid/conf.d$ cat local.conf
http_port 8080
cache_dir ufs /var/spool/squid 100 16 256

Here, I have setup squid to use port 8080, and use it as my caching server
this will work only for my local host because i need to create some acl and decide which clients will use it

sudo systemctl restart squid

squid will use the default port 3128 by default ( other ports are in UNCONN state)

touk@ubuntu:/etc/squid/conf.d$ sudo ss -tunap | less -s | grep 8080
tcp    LISTEN  0       256                                             *:8080                                              *:*                                   users:(("squid",pid=117073,fd=17))
touk@ubuntu:/etc/squid/conf.d$ sudo ss -tunap | less -s | grep squid
udp    UNCONN  0       0                                         0.0.0.0:53267                                       0.0.0.0:*                                   users:(("squid",pid=117073,fd=9))
udp    UNCONN  0       0                                               *:38167                                             *:*                                   users:(("squid",pid=117073,fd=5))
udp    ESTAB   0       0                                           [::1]:48837                                         [::1]:47244                               users:(("squid",pid=117073,fd=20))
tcp    LISTEN  0       256                                             *:8080                                              *:*                                   users:(("squid",pid=117073,fd=17))
tcp    LISTEN  0       256

And sure enough, from the access logs i can see my caching is working perfect:

root@ubuntu:/etc/squid/conf.d# tail -f /var/log/squid/access.log
1693403379.468 293103 127.0.0.1 TCP_TUNNEL/200 2459 CONNECT incoming.telemetry.mozilla.org:443 - HIER_DIRECT/34.120.208.123 -
1693403379.468 293171 127.0.0.1 TCP_TUNNEL/200 5155 CONNECT incoming.telemetry.mozilla.org:443 - HIER_DIRECT/34.120.208.123 -
1693403381.469 171233 127.0.0.1 TCP_TUNNEL/200 5919 CONNECT adservice.google.dz:443 - HIER_DIRECT/142.251.143.98 -
1693403381.469 289049 127.0.0.1 TCP_TUNNEL/200 6976 CONNECT googleads.g.doubleclick.net:443 - HIER_DIRECT/142.251.143.162 -
1693403381.469 171521 127.0.0.1 TCP_TUNNEL/200 8278 CONNECT adservice.google.com:443 - HIER_DIRECT/142.251.143.162 -
1693403390.429 175780 127.0.0.1 TCP_TUNNEL/200 18484 CONNECT encrypted-tbn0.gstatic.com:443 - HIER_DIRECT/142.251.143.206 -
1693403391.429 171131 127.0.0.1 TCP_TUNNEL/200 8341 CONNECT id.google.com:443 - HIER_DIRECT/142.251.143.131 -
1693403391.429 286197 127.0.0.1 TCP_TUNNEL/200 18109 CONNECT www.gstatic.com:443 - HIER_DIRECT/142.251.143.99 -
1693403392.429 294374 127.0.0.1 TCP_TUNNEL/200 1654131 CONNECT www.google.com:443 - HIER_DIRECT/142.251.143.100 -
1693403478.431 171006 127.0.0.1 TCP_TUNNEL/200 5449 CONNECT contile.services.mozilla.com:443 - HIER_DIRECT/34.117.237.239 -

RESTRICTING SERVER ACCESS

root@ubuntu:/etc/squid/conf.d# cat local.conf
http_port 8080
cache_dir ufs /var/spool/squid 800 16 256
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/htpasswd
auth_param basic realm proxy

acl internal src 192.168.1.0/255.255.255.0
acl authenticated proxy_auth REQUIRED


acl blocked_websites dstdomain facebook.com fb.com linux.com
http_access deny blocked_websites
http_access allow internal authenticated

Let me explain this:

I am using the proxy via port 8080
I am using the cache directory in /var/spool/squid and 800 is the size by default with a megabyte
Now i have created the ACL, i am creating the hosts which are defined by 'internal' and then the action that will be applied on them

acl internal src 192.168.1.0/255.255.255.0
http_access allow internal

With Squid you can restrict access by domain name or regular expression

If it is a long list of domain you can use a file populated by these domains, maybe you have a script that will do this for you
In this example, i'll block video streaming from domains inserted into a file

acl video_streaming dstdomain "/etc/squi/streaming.list"
http_access deny video_streaming

You can view a full list of ACL TYPES on this page

My First CI/CD Pipeline with JENKINS!

taqiyeddinedj — Fri, 18 Aug 2023 09:44:11 +0000

Project Scope and Tools Used:

This project centers around a comprehensive CI/CD pipeline, showcasing the integration of popular tools that are widely utilized in the field.

GitHub Repository Overview:

When you take a look at my GitHub repository, you'll notice several key files that play a crucial role in this pipeline's functionality.

project-root/
├── templates/
│   └── index.html
├── Dockerfile
├── app.py
├── Jenkinsfile
├── deployment-service.yml
├── script.groovy
├── .gitignore
└── requirements.txt

The repository contains files such as 'app.py,' which is a Flask web application."

Docker Image Creation:

To build the application image, I leverage a Dockerfile.

FROM python:3.7-alpine
COPY . /app
WORKDIR /app 
RUN pip install flask
CMD ["python", "app.py"]

Check my docker hub repo

In the deployment stage of our CI/CD pipeline, I've crafted a Kubernetes Deployment configuration file, deployment.yml

which defines the specifications for our application's deployment

The accompanying Service configuration file, service.yml, defines how our application can be accessed

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deployment
  namespace: default 
spec:
  selector:
    matchLabels:
      app: myapp
  replicas: 2
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: taqiyeddinedj/my-repo:webapp-2.0
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m" 
        ports:
        - containerPort: 8080

---

apiVersion: v1
kind: Service
metadata:
  name: myapp-service
  namespace: default
spec:
  type: NodePort 
  selector:
    app: myapp
  ports:
    - port: 8080
      targetPort: 8080
      nodePort: 30000

Jenkins Setup and Building Triggers:

For continuous integration, I've set up Jenkins on a separate local machine using a Docker container.

Inside the Jenkins container, I've included the Docker runtime, enabling it to build Docker images directly.

Using a webhook, any push to the repository automatically triggers the build process."

Pipeline Stages**:

The pipeline consists of several distinct stages, each serving a specific purpose.

These stages include initialization, testing (which identifies the active branch), building and pushing to Docker Hub, and the final deployment to a Kubernetes cluster.

Jenkins File and Groovy Syntax:

The pipeline is orchestrated using a Jenkinsfile, written in Groovy syntax."

def buildDockerImage() {
    echo "Building the docker image...."
    withCredentials([usernamePassword(credentialsId:'dockr-hub-repo', passwordVariable: 'PASS', usernameVariable: 'USER')]){
        sh "docker build -t taqiyeddinedj/my-repo:webapp-2.0 ."
        sh " echo $PASS | docker login -u $USER --password-stdin"
        sh "docker push taqiyeddinedj/my-repo:webapp-2.0"
    }
}

def deploytok8s() {
    echo "Deploying now the apllication on the kubernetes cluster"
    kubernetesDeploy (configs: 'deployment-service.yml', kubeconfigId: 'kubernetes')
}

return this

Setting up Your Own Kubernetes Cluster:

Establishing my personal Kubernetes cluster quite challenging, but i made it work."
Troubleshooting was a significant aspect of getting the cluster operational.

Integration of Kubernetes with Jenkins:

Connecting Kubernetes with Jenkins was a critical step. I discovered a helpful plugin on Stack Overflow, conveniently provided by the Jenkins community.

An obstacle I encountered was a certificate signing issue, likely related to port forwarding. This led me to seek a cluster accessible from the public network.

Transition to Azure and Deploying a Cluster**:

To address these challenges, I migrated to Microsoft Azure and successfully deployed a Kubernetes cluster.

There's a specific command that needs to be included in the Jenkinsfile for this Azure-based cluster setup.

#!/usr/bin/env groovy
def gv

pipeline {
    agent any
    stages {
        stage('Init') {
            steps {
                script {
                    gv = load "script.groovy"
                }
            }
        }
        stage('test') {
            steps {
                script {
                    echo "Testing the application"
                    echo "Executing pipeline for branch $BRANCH_NAME"
                }
            }
        }
        stage ('Build & pushing'){
            steps {
                script {
                    gv.buildDockerImage()

                }
            }
        }
        stage ('Deploy to K8S'){
            steps {
                script {
                    gv.deploytok8s()

                }
            }
        }
    }
}

To ensure seamless connectivity, the kubeconfig file is stored in a hidden directory (.kube) within your home directory, and its contents are uploaded to Jenkins as special credentials."

Azure Cluster Status:

Currently, the Azure-based Kubernetes cluster is up and running, serving as the backbone of our robust CI/CD pipeline.

Conclusion and Summary:

In summary, i have taken you through the intricate process of establishing a comprehensive CI/CD pipeline.
From the initial setup of Jenkins and Docker, to overcoming Kubernetes integration challenges, and finally transitioning to a reliable Azure-based cluster, we've covered a wealth of insights and practical steps.

Automating Infrastructure with Ansible

taqiyeddinedj — Sat, 29 Jul 2023 12:45:22 +0000

Introduction:

In this article, we will explore how to set up a load balancing infrastructure using Ansible on three Azure instances. The manager instance will serve as the load balancer, while the other two instances will act as web servers. To streamline the process, we will organize our playbooks into a single file and leverage Ansible's import feature for modularity and better management.

Setting Up Azure Instances:

To begin, we provision three Azure instances: one as the manager and the other two as web servers. The manager instance will be responsible for handling the load balancing, while the web server instances will serve the application.

Installing Ansible on the Manager:

Once the instances are up and running, we install Ansible on the manager instance. Ansible provides a simple and efficient way to automate the configuration and management of our infrastructure.

Since Ansible is available in the Extra Packages for Enterprise Linux (EPEL)

sudo yum install epel-release

sudo yum install ansible

Configuring the Ansible Inventory:

Next, we configure the Ansible inventory to include the two web server instances. The inventory allows Ansible to know which hosts it should target for various tasks.

Organizing Playbooks:

To keep our playbooks organized and modular, we create a single playbook file named "all-playbooks.yml." This file will include multiple playbooks using the import feature in Ansible.

#all-playbooks.yml
---
  - import-playbook: update.yml
  - import-playbook: install-services.yml
  - import-playbook: setup-app.yml
  - import-playbook: setup-lb.yml

Package Management:

In our "all-playbooks.yml" file, the first imported playbook is "update.yml." This playbook is responsible for updating the system packages on all three instances. It ensures that our instances have the latest updates before proceeding with the configuration.

---
- name: updating nodes
  hosts: all
  become: true
  tasks:
  - name: updating
    yum:
      name: '*'
      state: latest

Installing Apache and PHP:
In the same "all-playbooks.yml" file, we import the "install-services.yml" playbook. This playbook installs Apache and PHP, but only on the manager instance. As the load balancer, the manager does not need the same application as the web servers.

---
- hosts: all
  become: true
  tasks:
  - name: install apache
    yum:
      name: 
        - httpd
        - php
      state: present

  - name: Ensure appache starts
    service:
      name: httpd
      state: started
      enabled: yes

Uploading Application Files:

We upload the application file "index.php" to the web server instances. To ensure that Apache restarts whenever changes happen, we use handlers, which are triggered when specific events occur.

---
- hosts: nodes
  become: true
  tasks:
  - name: upload application file
    copy: 
      src: ../index.php
      dest: /var/www/html
      mode: 0755
    notify: restart apache

  handler:
    - name: restart apache
      service: name=httpd state=restarted

Load Balancer Configuration:

To set up the manager as a load balancer, we leverage the power of Jinja templates within Ansible. In the "setup-lb.yml" playbook, we insert a Jinja snippet inside an Apache directive called "proxybalancer." This configuration enables the manager to distribute incoming requests across the two web server instances, effectively balancing the load.

See the jinja code :

ProxyRquests off
<Proxy balancer://webcluster >
    {% for hosts in groups ['nodes'] %}
        BalancerMember http://{{hostvars[hosts]['anible_host']}}
    {% endfor %}
    ProxySet lbmethod=byrequests
</Proxy>

# Optional
<Location /balancer-manager>
  SetHandler balancer-manager
</Location>
ProxyPass /balancer-manager !
ProxyPass / balancer://webcluster/

Now, let's take a look at the result:

Conclusion:

By following the steps outlined in this article, we have successfully configured a load balancing infrastructure using Ansible and three Azure instances. The manager instance acts as the load balancer, while the other two instances serve as web servers. Organizing playbooks into a single file using Ansible's import feature makes the management and maintenance of the infrastructure much more efficient. With this setup, we have a scalable and robust system that can handle increased traffic and ensure high availability for our application.

DOCKER SWARM CLUSTER & NFS

taqiyeddinedj — Mon, 24 Jul 2023 16:39:11 +0000

Docker Swarm emerges as an excellent solution, offering a simple and scalable cluster management system. To complement this, utilizing NFS (Network File System) as a shared volume storage brings significant advantages, providing seamless data sharing and robustness across the Swarm cluster. Moreover, it is essential to emphasize the importance of referring to official documentation for both Docker Swarm and NFS, as it ensures a well-informed and successful setup.

The web app code :

My Cluster:

On the NFS Server (NFS Server Host):
The necessary steps and commands to set up an NFS server and mount the NFS share on the client node (Docker Swarm node):
Install NFS Server:
sudo yum install nfs-utils
Create the Shared Directory:
sudo mkdir -p /shared_dir
Export the Shared Directory at /etc/exports:
/shared_dir *(rw,sync,no_root_squash)
Apply the NFS Export Changes:
sudo exportfs –a
Start NFS Server:
sudo systemctl enable nfs-server
On the NFS Client (Docker Swarm Node):
sudo yum install nfs-utils
Create the Target Mount Directory:
sudo mkdir -p /mnt/shared_dir
Mount the NFS Share:
sudo mount -t nfs nfs_server:/shared_dir /mnt/shared_dir

NOTE: fro NFS3 and NFS4 you should explicitly allow Port 2049

Now we create the NFS docker volume on the worker node :
docker volume create --driver local --name website_volume --opt type=nfs4 --opt device=:/shared_dir --opt o=addr=20.111.58.70,rw,nolock
Now Create Docker service with NFS volume :
docker service create --replicas=2 --name hostname_service --restart-condition on-failure -p 80:80 --mount type=volume,source=website_volume,target=/app taqiyeddinedj/hostname_web_app

AWS HA WEB APP

taqiyeddinedj — Fri, 14 Jul 2023 22:21:23 +0000

Experience seamless performance and uninterrupted availability as you interact with the app's cutting-edge features
The servers are not mine tho lol
SO
Starting with the creation of RDS because it takes some time to deploy,

then going to S3 and create two buckets:

The deployment of the word press site can take 15 minutes, I strongly recommend to follow official steps !
https://aws.amazon.com/getting-started/hands-on/deploy-wordpress-with-amazon-rds/

Some Useful TIPS:

launch EC2 from a template, you will need it for the Auto scaling group
The ALB, is created to span 3 subnets for the ASG instances
It's very helpful to use health check of ALB

And finally this is my Wordpress HIGHLY AVAILABLE !
And as you can see (check link URI), I am using Cloudfront which means it gets the data it is sharing from the S3 bucket so images can load faster!!

Final Result: