DEV Community: san dev

Deploy AWS VPC with ALB, NAT Gateway, and Send Apache Logs to CloudWatch Using CloudWatch Agent

san dev — Sun, 18 May 2025 20:34:55 +0000

In this blog, we'll walk through the process of setting up a scalable and secure AWS infrastructure. The setup includes:

A Virtual Private Cloud (VPC)
Public and private subnets
An Internet Gateway
A NAT Gateway
An Application Load Balancer (ALB)
EC2 instances in private subnets
Apache HTTP server installed on EC2
CloudWatch Agent to send Apache logs to CloudWatch

This guide is ideal for beginners and intermediate users aiming to understand foundational AWS networking and logging.

Step 1: VPC and Subnet Design

Create a VPC using the "VPC and more" option for visual configuration.
Select two Availability Zones (e.g., ap-south-1a and ap-south-1b).
Create four subnets:

Two public subnets (one per AZ)
Two private subnets (one per AZ)
Use a CIDR block like 11.0.0.0/16 for your VPC.

Step 2: Configure Routing and Internet Access

Create Route Tables:
- One for public subnets (routes to Internet Gateway)
- One or two for private subnets (routes to NAT Gateway)
Create an Internet Gateway and attach it to the VPC.
Create a NAT Gateway in one of the public subnets.
Update routing tables:
- Public subnet: 0.0.0.0/0 to Internet Gateway
- Private subnet: 0.0.0.0/0 to NAT Gateway

Step 3: Launch EC2 Instances

Launch two Ubuntu Linux EC2 instances in private subnets.
Choose instance type t2.micro.
Disable public IP assignment.
In user data, provide a script to install Apache and a simple HTML response with the hostname:
Script is is in below Screen shot

Step 4: Configure the Application Load Balancer (ALB)

Create an Application Load Balancer:

Internet-facing
IPv4
Attach to the public subnets

Create a Security Group for ALB:

Allow inbound HTTP traffic from the internet

we select previously created VPC and 2 AZ inside 2 AZ select previously created 2 public subnets

Then we need to create a target group

Target type will be instance , port 80 , same vpc like before , other settings are unchanged

Select both the instances for target group and click include as pending below , When you add EC2 instances to an Application Load Balancer (ALB) target group, they are listed as “Registered targets – pending below” or “pending” until the health check passes.
This is the summery you can check before the create ALB

Modify the EC2 Security Group:

In above picture we can see in the common SG of 2 private instance we delete the previous inbound rule and add SG of ALB and add here

Above diagram is explaining user when access the private ec2 instance the request only go through ALB

Step 5: Install and Configure CloudWatch Agent
So far, after completing my tasks, the ALB isn't working. To fix this, I need to debug the issue, which requires sending Apache logs from the Ubuntu EC2 (in the private subnet) to CloudWatch.

For that 1st we create a jump server or Bastian host
this is public instance in same vpc with a public subnet , in SG we created SG which will accept traffic from anywhere on port 22 SSH port
now this private instance again create inbound rule which will take inbound traffic from the SG which created for public subnet

In above picture we can see its taking ssh inbound traffic from SG created for jump server

Now we will install the CloudWatch Agent on Ubuntu ) private instances ) `# Go to a temporary directory cd /tmp

Download the CloudWatch agent .deb package from AWS

wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb

Install the downloaded .deb package

sudo dpkg -i -E ./amazon-cloudwatch-agent.deb
**Verify Installation **ls /opt/aws/amazon-cloudwatch-agent/bin//opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent --version ( showing the version )`

Create CloudWatch Agent Configuration File
Create the file at sudo vi /opt/aws/amazon-cloudwatch-agent/bin/config.json

{ "logs": { "logs_collected": { "files": { "collect_list": [ { "file_path": "/var/log/apache2/access.log", "log_group_name": "ApacheAccessLogs", "log_stream_name": "{instance_id}/access_log", "timestamp_format": "%d/%b/%Y:%H:%M:%S" }, { "file_path": "/var/log/apache2/error.log", "log_group_name": "ApacheErrorLogs", "log_stream_name": "{instance_id}/error_log", "timestamp_format": "%d/%b/%Y:%H:%M:%S" } ] } } } }
Start CloudWatch Agent
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \ -a fetch-config \ -m ec2 \ -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json \ -s
How to Verify It’s Now Running
sudo systemctl status amazon-cloudwatch-agent

below is the expected output

Now head to the AWS Console:

Go to CloudWatch → Log Groups
Look for:
- ApacheAccessLogs
- ApacheErrorLogs Repeat for both private instances after some checks and balance Now we can see the ssh msg from both the instance

this is the message return from both the instances with there private IP address , every time I refresh IP is changing it shows that ALB is working

and finally in CloudWatch access_log we can see this msg

11.0.17.14 - - [17/May/2025:17:11:12 +0000] "GET / HTTP/1.1" 200 289 "-" "ELB-HealthChecker/2.0" it means load balancer working
it means it can access the logs

Automating Static Website Deployment to AWS S3 with Terraform and GitHub Actions

san dev — Wed, 07 May 2025 14:43:12 +0000

Overview
This guide demonstrates how to automate the deployment of a static website to Amazon S3 using Terraform for infrastructure provisioning and GitHub Actions for continuous integration and deployment (CI/CD). By the end, you'll have a streamlined process that updates your website upon each code push.

Setting Up AWS Credentials in Terraform Cloud
To enable Terraform Cloud to provision resources in your AWS account, you need to securely store your AWS access keys within the platform.
So we 1st create a organization first
go to
then create a new organization

Crete a new workspace

Now come to variable and create 2 variable for AWS ACCESS KEY and SECRET ACCESS KEY

now Navigate to Terraform Cloud > Organization Settings > API Tokens > create an API token > give name generate token > save to notepad

Now come to Github
you can fork this repo code-pipeline-game

Now come to git-repo -> settings -> secrets & variables -> action
here you can save the API token you have generated from terraform cloud

This token we will mention here in Github action workflow YAML file

`name: Deploy website to AWS S3
on:
push:
branches:
- main
jobs:
Terraform:
name: 'Terraform'
runs-on: ubuntu-latest

steps:
- name: Checkout
  uses: actions/checkout@v4      

- name: Setup Terraform
  uses: hashicorp/setup-terraform@v1
  with:
    cli_config_credentials_token: ${{ secrets.tfc_team_token }}

- name: Terraform Init
  run: terraform init

- name: Terraform Validate
  run: terraform validate 

- name: Terraform Plan
  run: terraform plan

- name: Terraform Apply
  run: terraform apply -auto-approve

- name: Terraform destroy
  run: terraform destroy -auto-approve`

Now You can find all the other terraform files in github repo
so we can run the the Github action file and check all the stage

We can see in the image above that our job is completed. Its time to verify from our AWS console.

We can confirm now that S3 bucket has been created and the website files uploaded in it.

Its time to connect to the website.

We will copy the website url from outputs in the workflow

Its time to visit the website. Paste the website url into your browser

Using NFS in Kubernetes – A Simple Guide to Shared Storage

san dev — Tue, 15 Apr 2025 13:03:03 +0000

Kubernetes is powerful when it comes to managing containerized workloads, but things get tricky when your applications need shared storage.
That’s where NFS (Network File System) comes in. It's a tried-and-tested way to share files across multiple pods and services in a cluster.
In this blog, we’ll walk through why and when to use NFS in Kubernetes, how to set it up using Persistent Volumes (PV) and Persistent Volume Claims (PVC), and key things to watch out for.

we will set up a simple NFS server on Ubuntu and make use of it in a nginx container running on a Kubernetes cluster

NFS is pretty straightforward to set up, you install the server, export a share and away you go. However it seems that there's a lot of struggles around permissions and I have had some issues my self so I thought I'd create a short write up, mostly for my own reference.

Step 1: Launch an EC2 Instance for Your Lab
To begin, we need a Linux machine that will act as our NFS server and possibly host a lightweight Kubernetes cluster (like Minikube or K3s).

For this setup, I'm using an Amazon EC2 instance with the following specs:

AMI: Ubuntu 22.04 LTS
Instance Type: t2.medium (2 vCPUs, 4 GB RAM)
Storage: 20 GB (or more, depending on your needs)
Security Group: Allow SSH (port 22) and NFS (ports 2049, 111)

Step 2: we need to check if the existing file system is mounted
for that use

fdisk -l

here you can see root partition is xvda1
now we will check if its mounted or not for this we will use

mount | grep xvda1

step 3 :Install NFS server

apt-get install nfs-kernel-server

Step 4 : Configure exports
We'll add our directory to the exports of the server in the /etc/exports file.

Now we need to actually tell the server to export the directory

exportfs -ar

We can verify that the directory has been shared with the -v parameter

exportfs -v

step 4: Now we will verify which directories are shared (exported) by an NFS server.

showmount -e

Configure firewall
If the firewall is active we need to tell it to allow NFS traffic

ufw allow from 34.229.82.109/24 to any port nfs

step 5:Test mount from container
To test the mounting of this directory from a container let's first create a file in the directory.

here I created a file call myfile.txt

step 6:Now let's try to mount the nfs share from a container/pod.

we will install a Kubernetes cluster using kubeadm , we can use the below code just run the script


#!/bin/bash
swapoff -a 
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# sysctl params required by setup, params persist across reboot

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# Apply sysctl params without reboot

sudo sysctl --system
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl

sudo mkdir /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet=1.28.1-1.1 kubeadm=1.28.1-1.1 kubectl=1.28.1-1.1 docker.io
sudo apt-mark hold kubelet kubeadm kubectl docker.io

sudo mkdir /etc/containerd
sudo containerd config default > /etc/containerd/config.toml
sudo sed -i 's/            SystemdCgroup = false/            SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd
sudo systemctl restart kubelet

kubeadm config images pull
kubeadm init --pod-network-cidr=192.168.0.0/16

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/tigera-operator.yaml
curl https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/custom-resources.yaml -O
kubectl create -f custom-resources.yaml 
kubectl taint nodes --all node-role.kubernetes.io/control-plane-

UUID=85538820-9e86-47ea-9d4a-f18019b855c3

step 7: Now we will deploy a nginx pod

root@ip-172-31-16-213:/nfs# cat nginx.yaml 
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: nginx
  name: nginx
spec:
  containers:
  - image: nginx
    name: nginx
    volumeMounts:
    - name: nfs-vol
      mountPath: /var/nfs # The mountpoint inside the container
  volumes:
  - name: nfs-vol
    nfs:
      server: 34.224.65.228 # IP to our NFS server
      path: /nfs # The exported directory

step 8: Now we will check the mount directory which was set into yaml file as /var/nfs

we can see that my.txt and other files which are inside /nfs directory is mounted to inside container /var/nfs directory so its a success
A big note with NFS, and what's creating issues for users are the permission and user mappings. Normally an NFS share will not be shared with root access (root_squash) and the user that needs access to the share will need to exist on the NFS server.
Refer to this NFS how-to for more information

Use NFS for the static content

Now that we know that we can mount a NFS share in the container let's see if we can use it to host our static html files
By default nginx serves files from the /usr/share/nginx/html directory which we can verify from our running container

With this knowledge, let's create a new directory for NFS to export, add a static html file to the directory and mount this directory to the directory inside of the container

step 9: Recreate pod with mountpoint to correct directory

As we can see our container sees the html file, now let's try to see if we can get it to work through http as well.

First we'll expose our pod so that it we can access it from outside the cluster, and then let's open it up in a browser

we will create a node port service and expose to web

what ever changes we have done inside /nfs/index.html file it will affect inside pod