DEV Community: Muhammad Awais Zahid

Stock Market Real-Time Data Analytics Pipeline on AWS

Muhammad Awais Zahid — Sat, 28 Mar 2026 11:34:14 +0000

🚀 Building a Real-Time Stock Market Analytics Pipeline on AWS

Author: Muhammad Awais | DevSecOps Engineer | AWS | Kubernetes | Terraform

Overview ☁️

In this project, I built a fully serverless, event-driven Real-Time Stock Market Data Analytics Pipeline on AWS. The pipeline ingests live stock data, processes it, stores it, analyzes trends, and sends smart alerts — all with minimal cost and zero servers to manage.

Here is what the pipeline does end to end:

Streams real-time stock data from yfinance into Amazon Kinesis Data Streams
Triggers AWS Lambda to process, clean, and store incoming data
Stores raw data in Amazon S3 for historical analysis
Stores processed data in Amazon DynamoDB for low-latency querying
Queries historical data using Amazon Athena via AWS Glue Catalog
Detects stock trends using moving averages (SMA-5 and SMA-20)
Sends real-time buy/sell alerts via Amazon SNS (Email/SMS)

Architecture at a Glance 🏗️

Python Script (yfinance)
        ↓
Amazon Kinesis Data Streams
        ↓
AWS Lambda (Process & Clean Data)
        ↓                    ↓
Amazon DynamoDB          Amazon S3
        ↓                    ↓
AWS Lambda            AWS Glue Catalog
(Trend Analysis)             ↓
        ↓             Amazon Athena
Amazon SNS                   ↓
(Email/SMS Alerts)       Amazon S3
                       (Query Results)

Steps to Build This Pipeline 👩‍💻

Step 1 — Setting Up Data Streaming with Amazon Kinesis

The first step is creating a Kinesis Data Stream to act as the real-time data pipeline.

Create the Kinesis Stream:

Go to AWS Console → Kinesis → Data Streams
Click Create data stream
Name it stock-market-stream
Choose On-demand capacity mode (cost-efficient for variable workloads)
Click Create

Python Producer Script:

This script continuously fetches stock data from yfinance and pushes it to Kinesis every 30 seconds.

import boto3
import json
import time
import yfinance as yf

kinesis_client = boto3.client('kinesis', region_name='us-east-1')
STREAM_NAME = "stock-market-stream"
STOCK_SYMBOL = "AAPL"
DELAY_TIME = 30

def get_stock_data(symbol):
    stock = yf.Ticker(symbol)
    data = stock.history(period="2d")

    return {
        "symbol": symbol,
        "open": float(round(data.iloc[-1]["Open"], 2)),
        "high": float(round(data.iloc[-1]["High"], 2)),
        "low": float(round(data.iloc[-1]["Low"], 2)),
        "price": float(round(data.iloc[-1]["Close"], 2)),
        "previous_close": float(round(data.iloc[-2]["Close"], 2)),
        "change": float(round(data.iloc[-1]["Close"] - data.iloc[-2]["Close"], 2)),
        "change_percent": float(round(
            ((data.iloc[-1]["Close"] - data.iloc[-2]["Close"]) / data.iloc[-2]["Close"]) * 100, 2
        )),
        "volume": int(data.iloc[-1]["Volume"]),
        "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
    }

def send_to_kinesis():
    while True:
        stock_data = get_stock_data(STOCK_SYMBOL)
        print(f"Sending: {stock_data}")

        kinesis_client.put_record(
            StreamName=STREAM_NAME,
            Data=json.dumps(stock_data),
            PartitionKey=STOCK_SYMBOL
        )
        time.sleep(DELAY_TIME)

send_to_kinesis()

💡 Tip: Always convert np.float64 values to plain float before sending to Kinesis to avoid serialization issues downstream.

Step 2 — Processing Data with AWS Lambda

Two Lambda functions handle the data processing:

Lambda 1 — Kinesis Consumer (Store to S3 + DynamoDB)

This function is triggered automatically by Kinesis, decodes the data, computes metrics, and stores it in both S3 and DynamoDB.

Setup:

Go to AWS Lambda → Create Function
Name it stock-market-processor
Runtime: Python 3.12
Add trigger: Kinesis → select stock-market-stream
Attach IAM role with permissions for: DynamoDB, S3, Kinesis

import json
import boto3
import base64
from decimal import Decimal

dynamodb = boto3.resource("dynamodb")
s3 = boto3.client("s3")

DYNAMO_TABLE = "stock-market-data"
S3_BUCKET = "your-s3-bucket-name"
table = dynamodb.Table(DYNAMO_TABLE)

def lambda_handler(event, context):
    for record in event['Records']:
        raw_data = base64.b64decode(record["kinesis"]["data"]).decode("utf-8")
        payload = json.loads(raw_data)

        # Save raw data to S3
        s3_key = f"raw-data/{payload['symbol']}/{payload['timestamp'].replace(':', '-')}.json"
        s3.put_object(Bucket=S3_BUCKET, Key=s3_key, Body=json.dumps(payload), ContentType='application/json')

        # Compute metrics
        price_change = round(payload["price"] - payload["previous_close"], 2)
        price_change_percent = round((price_change / payload["previous_close"]) * 100, 2)
        is_anomaly = "Yes" if abs(price_change_percent) > 5 else "No"
        moving_average = round(
            (payload["open"] + payload["high"] + payload["low"] + payload["price"]) / 4, 2
        )

        # Store in DynamoDB
        table.put_item(Item={
            "symbol": payload["symbol"],
            "timestamp": payload["timestamp"],
            "price": Decimal(str(payload["price"])),
            "previous_close": Decimal(str(payload["previous_close"])),
            "change": Decimal(str(price_change)),
            "change_percent": Decimal(str(price_change_percent)),
            "volume": int(payload["volume"]),
            "moving_average": Decimal(str(moving_average)),
            "anomaly": is_anomaly
        })

    return {"statusCode": 200, "body": "Processing Complete"}

Lambda 2 — Trend Analyzer (SMA Crossover + SNS Alert)

This function runs on a schedule (EventBridge every 5 minutes), queries DynamoDB, computes SMA-5 and SMA-20, detects crossover signals, and sends alerts via SNS.

import boto3
import json
import decimal
from datetime import datetime, timedelta

dynamodb = boto3.resource("dynamodb")
sns = boto3.client("sns")

TABLE_NAME = "stock-market-data"
SNS_TOPIC_ARN = "arn:aws:sns:us-east-1:YOUR_ACCOUNT_ID:Stock_Trend_Alerts"

def get_recent_stock_data(symbol, minutes=5):
    table = dynamodb.Table(TABLE_NAME)
    past_time = datetime.utcnow() - timedelta(minutes=minutes)

    response = table.query(
        KeyConditionExpression="symbol = :symbol AND #ts >= :time",
        ExpressionAttributeNames={"#ts": "timestamp"},
        ExpressionAttributeValues={
            ":symbol": symbol,
            ":time": past_time.strftime("%Y-%m-%d %H:%M:%S"),
        },
        ScanIndexForward=True
    )
    return sorted(response.get("Items", []), key=lambda x: x["timestamp"])

def calculate_moving_average(data, period):
    if len(data) < period:
        return decimal.Decimal("0")
    return sum(decimal.Decimal(str(d["price"])) for d in data[-period:]) / period

def lambda_handler(event, context):
    for symbol in ["AAPL"]:
        stock_data = get_recent_stock_data(symbol)

        if len(stock_data) < 20:
            continue

        sma_5 = calculate_moving_average(stock_data, 5)
        sma_20 = calculate_moving_average(stock_data, 20)
        sma_5_prev = calculate_moving_average(stock_data[:-1], 5)
        sma_20_prev = calculate_moving_average(stock_data[:-1], 20)

        message = None
        if sma_5_prev < sma_20_prev and sma_5 > sma_20:
            message = f"{symbol} is in an Uptrend! Consider a buy opportunity."
        elif sma_5_prev > sma_20_prev and sma_5 < sma_20:
            message = f"{symbol} is in a Downtrend! Consider selling."

        if message:
            sns.publish(TopicArn=SNS_TOPIC_ARN, Message=message, Subject=f"Stock Alert: {symbol}")
            print(f"Alert sent: {message}")

    return {"statusCode": 200, "body": json.dumps("Trend analysis complete")}

Step 3 — Query Historical Stock Data using Amazon Athena

Raw JSON files land in S3. AWS Glue Catalog creates a structured schema on top of them so Athena can query them with standard SQL.

Setup Glue Crawler:

Go to AWS Glue → Crawlers → Create Crawler
Point it to your S3 bucket path: s3://your-bucket/raw-data/
Create a new database: stock_market_db
Run the crawler — it auto-detects schema and creates a table

Query with Athena:

-- Average daily price per symbol
SELECT symbol,
       DATE(timestamp) AS date,
       ROUND(AVG(price), 2) AS avg_price,
       MAX(high) AS day_high,
       MIN(low) AS day_low
FROM stock_market_db.raw_data
GROUP BY symbol, DATE(timestamp)
ORDER BY date DESC;

-- Detect anomalies
SELECT symbol, timestamp, price, change_percent
FROM stock_market_db.raw_data
WHERE ABS(change_percent) > 5
ORDER BY timestamp DESC;

💡 Tip: Set your Athena query results S3 bucket in Settings before running queries.

Step 4 — Stock Trend Alerts using SNS

Setup SNS Topic:

Go to AWS SNS → Topics → Create Topic
Type: Standard
Name: Stock_Trend_Alerts
Click Create

Add Subscription:

Click your topic → Create Subscription
Protocol: Email or SMS
Endpoint: your email or phone number
Confirm the subscription from your inbox

Once the trend analyzer Lambda detects a crossover, SNS fires an alert like:

Subject: Stock Alert: AAPL
Message: AAPL is in an Uptrend! Consider a buy opportunity.

Key Learnings 🧠

Kinesis is perfect for real-time event streaming — On-demand mode saves cost for unpredictable workloads
Lambda + Kinesis trigger gives you a fully serverless processing layer with automatic scaling
Always convert numpy types to native Python before serializing to JSON or sending to AWS services
DynamoDB works great for low-latency lookups but design your partition key carefully — symbol + timestamp as composite key works well here
Glue + Athena is a powerful combo for ad-hoc SQL queries on raw S3 data without spinning up any infrastructure
SMA crossover is a simple but effective signal for trend detection in real-time pipelines

Resources 🔗

Written by **Muhammad Awais* — DevSecOps Engineer | CKA | CKS*
Connect on LinkedIn | GitHub

Application Security - SAST, SCA & DAST

Muhammad Awais Zahid — Fri, 13 Feb 2026 10:08:06 +0000

Focus

This guide helps absolute beginners understand:

What SAST, SCA, and DAST are
Why we need all three
How security tools actually find vulnerabilities
How attackers exploit insecure applications
How developers should fix issues

You will build, scan, attack a real application.

Core Concepts (Before Touching Tools)

What is SAST?

Static Application Security Testing

Scans source code
Does not run the application
Finds insecure coding patterns
Best used early (IDE / Pull Requests)

Examples:

Hardcoded secrets
SQL injection patterns
Command execution risks
Insecure Deserialization (pickle)
Arbitrary Code Execution (eval/exec)
Unsafe YAML Loading (yaml.load)
Path Traversal (tarfile.extractall)
Insecure SSL/TLS (verify=False)
Weak Cryptography (MD5/SHA1 usage)
Insecure Temp Files (tempfile.mktemp)

Detailed Information

Hardcoded secrets: Storing sensitive data like passwords or API keys directly in your code where anyone who sees the file can steal them.
SQL injection patterns: Building database queries by gluing strings together, which lets hackers "trick" your database into deleting data or leaking secrets.
Command execution risks: Passing user input directly to your operating system, allowing a hacker to run any command (like format C:) on your server.
Insecure Deserialization (pickle): Using the pickle tool on data from the internet, which can automatically run hidden malicious code the moment the file is opened.
Arbitrary Code Execution (eval/exec): Using functions that turn text into live code, effectively giving a stranger the keyboard to your application.
Unsafe YAML Loading (yaml.load): Opening configuration files in a way that allows the file itself to trigger Python commands during the reading process.
Path Traversal (tarfile.extractall): Unzipping files without checking their names, which can let a malicious file overwrite important system files outside your project folder.
Insecure SSL/TLS (verify=False): Turning off "security checks" for internet connections, making it easy for hackers to spy on your encrypted data.
Weak Cryptography (MD5/SHA1 usage): Using "broken" mathematical formulas to hide data that modern computers can crack in seconds.
Insecure Temp Files (tempfile.mktemp): Creating a temporary file name without instantly "locking" it, creating a tiny window of time for a hacker to swap it with a malicious file.

What is SCA?

Software Composition Analysis

Scans dependencies
Matches versions against known CVEs
Answers: “Are we using vulnerable packages?”

Even perfect code can be insecure because of vulnerable libraries.

What is DAST?

Dynamic Application Security Testing

Attacks a running application
No access to source code
Simulates real attackers
Finds exploitable vulnerabilities

Why We Need All Three

Tool	Code	Dependencies	Running App
SAST	✅	❌	❌
SCA	❌	✅	❌
DAST	❌	❌	✅

👉 No single tool is enough.

Lab Overview

We will:

Build a vulnerable Python app
Run SAST using SonarQube
Run SCA using pip-audit
Run DAST using OWASP ZAP
Compare findings
Understand how to fix them

🏗️ Project Structure

vulnerable-app/
├── app.py
├── requirements.txt
├── users.db
└── sonar-project.properties

Step 1: Vulnerable Application

app.py

from flask import Flask, request, jsonify
import sqlite3
import random

app = Flask(__name__)

DATABASE = "users.db"


def get_db():
    return sqlite3.connect(DATABASE)


@app.route("/")
def home():
    return "User Management Service"


@app.route("/user")
def get_user():
    user_id = request.args.get("id")

    conn = get_db()
    cursor = conn.cursor()

    query = f"SELECT id, username FROM users WHERE id = '{user_id}'"
    cursor.execute(query)

    user = cursor.fetchone()
    conn.close()

    if not user:
        return jsonify({"error": "User not found"}), 404

    return jsonify({"id": user[0], "username": user[1]})


@app.route("/token")
def generate_token():
    token = str(random.random())
    return jsonify({"token": token})


@app.route("/admin/calc")
def admin_calculate():
    expr = request.args.get("expr")
    result = eval(expr)
    return jsonify({"result": result})


if __name__ == "__main__":
    app.run(debug=True)

📦 Step 2: Vulnerable Dependencies

requirements.txt

flask==1.0
requests==2.19.1

🔍 Step 3: SAST with SonarQube

Start SonarQube

docker run -d -p 9000:9000 --name sonarqube sonarqube:9.9-community

Open http://IP:9000

Login: admin/admin

sonar-project.properties

sonar.projectKey=vulnerable-python-app
sonar.projectName=Vulnerable Python App
sonar.sources=.
sonar.language=py
sonar.python.version=3

Install Sonar Scanner

Follow the Instructions provided here:

https://docs.sonarsource.com/sonarqube-server/10.8/analyzing-source-code/scanners/sonarscanner

Install Sonar CLI

wget "https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-8.0.1.6346-linux-x64.zip"
sudo apt update
sudo apt install unzip -y
unzip sonar-scanner-cli-8.0.1.6346-linux-x64.zip
sudo mv sonar-scanner-8.0.1.6346-linux-x64 /opt/sonar-scanner
nano ~/.bashrc
export SONAR_SCANNER_HOME=/opt/sonar-scanner
export PATH="$SONAR_SCANNER_HOME/bin:$PATH"
source ~/.bashrc

Run Scan

sonar-scanner -Dsonar.host.url=http://IP:9000 -Dsonar.login=<TOKEN>

Check severity
Block merge if Critical/Blocker
Assign issue
Fix or justify
Re-run scan
Track metrics
Improve the process of recurring

Expected Findings

Hardcoded secrets
SQL injection risks
Command injection risks
Debug mode enabled

Step 4: SCA with pip-audit

sudo apt update
sudo apt install python3-venv python3-full -y
python3 -m venv venv
source venv/bin/activate
pip install pip-audit
pip-audit -r requirements.txt

Now, we will check that CVE on the internet

Expected Findings

Vulnerable Flask version
Vulnerable Requests version

Application runs at http://IP:5000

🕷️ Step 6: DAST with OWASP ZAP

Run the Juice Shop Application

docker run -d \
  --name juice-shop \
  -p 3000:3000 \
  bkimminich/juice-shop

Open in browser:

IP:3000

Scan target by running ZAP from cli

docker run --rm \
  -v "$(pwd):/zap/wrk" \
  -t ghcr.io/zaproxy/zaproxy:stable \
  zap-baseline.py \
  -t http://IP:3000 \
  -r zap-report.html

Scan target by running ZAP from UI

docker run -it \                                                                 
  -p 8080:8080 \
  ghcr.io/zaproxy/zaproxy:stable \
  zap-webswing.sh

Open Zap UI:

http://localhost:8080/zap

Compare Results

Issue	SAST	SCA	DAST
Hardcoded Secret	✅	❌	❌
Vulnerable Library	❌	✅	❌
SQL Injection	✅	❌	✅
Command Injection	✅	❌	✅

🛠️ What to Fix Next

Use parameterised SQL queries
Remove shell=True
Move secrets to environment variables
Upgrade dependencies
Disable debug mode

Re-run scans and observe improvements 🚀

Deployed a Fully Microservices Stack on Kubernetes, AWS EKS

Muhammad Awais Zahid — Tue, 03 Feb 2026 13:45:28 +0000

Create an ec2 machine

First of all, create a ec2 machine of Ubuntu and t2.large and 30gb storage

SSH into the machine

run following commands, get ssh of your ec2 machine and install some packages

ssh -i "pem-key.pem" ubuntu@ec2-3-88-54-141.compute-1.amazonaws.com

sudo apt update -y

sudo apt install unzip

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws --version

curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.19.6/2021-01-05/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin
kubectl version --short --client

curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin
eksctl version

Create EKS-Cluster

Run the following commands to create eks-cluster

eksctl create cluster --name eksclicluster --region us-east-1 --zones us-east-1a,us-east-1b --without-nodegroup

eksctl utils associate-iam-oidc-provider --region us-east-1 --cluster eksclicluster --approve

eksctl create nodegroup --cluster=my-cluster --name=mynodegroup --region=us-east-1 --node-type=t3.medium --nodes=2 --nodes-min=2 --nodes-max=4 --node-volume-size=20 --ssh-access --ssh-public-key=pem-key --managed

Configure Loadbalancer Controller

curl -o iam-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.17.0/docs/install/iam_policy.json

aws iam create-policy \
    --policy-name AWSLoadBalancerControllerIAMPolicy \
    --policy-document file://iam-policy.json

eksctl create iamserviceaccount \
--cluster=<cluster-name> \
--namespace=kube-system \
--name=aws-load-balancer-controller \
--attach-policy-arn=arn:aws:iam::<AWS_ACCOUNT_ID>:policy/AWSLoadBalancerControllerIAMPolicy \
--override-existing-serviceaccounts \
--region <region-code> \
--approve

helm repo add eks https://aws.github.io/eks-charts
helm repo update eks

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \            
  -n kube-system \
  --set clusterName=<your-cluster-name> \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set region=<region> \
  --set vpcId=<your-vpc-id>

helm repo add eks https://aws.github.io/eks-charts

EBS CSI Plugin configuration

The Amazon EBS CSI plugin requires IAM permissions to make calls to AWS APIs on your behalf.

Create an IAM role and attach a policy. AWS maintains an AWS-managed policy, or you can create your own custom policy. You can create an IAM role and attach the AWS managed policy with the following command. Replace my-cluster with the name of your cluster. The command deploys an AWS CloudFormation stack that creates an IAM role and attaches the IAM policy to it

eksctl create iamserviceaccount \
    --name ebs-csi-controller-sa \
    --namespace kube-system \
    --cluster <YOUR-CLUSTER-NAME> \
    --role-name AmazonEKS_EBS_CSI_DriverRole \
    --role-only \
    --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
--approve

eksctl create addon --name aws-ebs-csi-driver --cluster <YOUR-CLUSTER-NAME> --service-account-role-arn arn:aws:iam::<AWS-ACCOUNT-ID>:role/AmazonEKS_EBS_CSI_DriverRole --force

Install Application Project

Either create with HELM or use following method

# from the chart directory (where Chart.yaml is)
helm template three-tier . > all.yaml

# apply (and re-apply) like normal kubectl
kubectl apply -f all.yaml

kubectl apply -f ingress.yaml

kubectl get ingress -n default -w

When you see an ADDRESS: k8s-...elb.amazonaws.com → your ALB is created.

User/Browser
    |
    | HTTPS
    v
AWS ALB  (created/managed by AWS Load Balancer Controller via Ingress)
    |
    v
Kubernetes Ingress (EKS)
    |
    v
Web Frontend Service
    |
    +--> Cart Service ------> Redis (StatefulSet)
    |
    +--> Catalogue Service -> MySQL ----\
    |                                    \
    +--> User Service ------> MongoDB -----> (PVCs -> EBS CSI Driver -> Amazon EBS)
    |
    +--> Ratings Service ---> MongoDB
    |
    +--> Shipping Service --> RabbitMQ <--- Dispatch Service
    |
    +--> Payment Service

Short Secure Terraform Project

Muhammad Awais Zahid — Fri, 02 Jan 2026 16:17:25 +0000

1. Create a Provider.tf file

In this file i specify the region and AWS user credentials using one of the following secure ways

aws configure
Environment variables
IAM roles (recommended for EC2, CloudShell, CI/CD)

provider "aws" {
  region                  = "us-east-1"
}

2. Create Main.tf file

In this file i created an S3 bucket for the remote backend to store terraform state file and make it private with versioning enabled

resource "aws_s3_bucket" "example" {
  bucket = "my-unique-bucket-name-12345"  # change this to a unique name
  acl    = "private"

  versioning {
    enabled = true
  }

  # Block public access
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

3. Create backend.tf file

In this file i created a remote backend S3 to store terraform state file and make it private with versioning enabled

terraform {
  backend "s3" {
    bucket = "my-terraform-state-johndoe123"  # Replace with your unique bucket name
    key    = "terraform-state-file"
    region = "us-east-1"
  }
}

4. Create an AWS Secret Manager

aws secretsmanager create-secret --name my-database-password-johndoe --secret-string "YourSecurePassword"

5. Update Main.tf file

updated my main.tf and create RDS resource

resource "aws_s3_bucket" "example" {
  bucket = "my-unique-bucket-name-12345"  # change this to a unique name
  acl    = "private"

  versioning {
    enabled = true
  }

  # Block public access
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

data "aws_secretsmanager_secret_version" "database_password" {
  secret_id = "my-database-password-johndoe"
}

resource "aws_db_instance" "my_secret_db" {
  identifier        = "rds-db-instance"
  allocated_storage = 20
  storage_type      = "gp2"
  engine            = "mysql"
  engine_version    = "8.0.43"
  instance_class    = "db.t3.micro"
  username          = "admin"
  password          = data.aws_secretsmanager_secret_version.database_password.secret_string  # Using the retrieved secret value
}

A Practical Guide to Troubleshooting Git Push Errors in Terraform Projects

Muhammad Awais Zahid — Wed, 24 Dec 2025 17:46:53 +0000

While working on a Terraform project, I ran into several Git push errors that initially felt confusing and frustrating. However, each error turned out to be a valuable learning moment. This article documents those issues step by step, explains why they happen, and shows how to fix them correctly.

If you’re learning Terraform, DevOps, or Infrastructure as Code, chances are you’ll encounter these same problems.

1️⃣GitHub Rejects Large Files (>100 MB)

Error:

File .terraform/...terraform-provider-aws is larger than 100 MB

Why this happens
The .terraform/ directory was committed. This directory contains Terraform provider binaries, which can be hundreds of megabytes in size and should never be version-controlled.

Correct Fix
Add the following to .gitignore

.terraform/
*.tfstate
*.tfstate.backup

If the file already exists in Git history, the cleanest approach for new projects is to reinitialise the repository:

rm -rf .git
git init
git add .
git commit -m <commit-id>

2️⃣GitHub Push Protection Blocks Secrets

Error:

Push cannot contain secrets (AWS Access Key detected)

Why this happens
AWS credentials were hardcoded inside provider.tf. GitHub automatically scans commits for secrets and blocks pushes to prevent credential leaks.

What Not to Do

provider "aws" {
  access_key = "AKIA..."
  secret_key = "xxxx"
}

Correct Approach

provider "aws" {
  region = "us-east-1"
}
(or add credentials in another file and add that file in .gitignore)

Provide credentials securely using:

aws configure
Environment variables
IAM roles (recommended for EC2, CloudShell, CI/CD)

⚠️ If credentials were committed, they should be rotated immediately, even if the push was blocked.

Blue/Green Deployment on AWS Using CodeDeploy and CodePipeline

Muhammad Awais Zahid — Wed, 10 Dec 2025 10:16:08 +0000

When you're learning AWS in depth, nothing teaches you better than building a real-world pipeline. In this project, I created a highly available and automated deployment architecture using EC2, CodeDeploy, Load Balancer, Auto Scaling Group, Launch Templates, GitHub, and CodePipeline.

This blog walks you through the exact process — every IAM role, installation, configuration, and deployment — so you can rebuild it on your own or use it as a learning reference.

This setup ensures:

High availability
Automated deployments
Auto-healing servers
CI/CD with zero manual intervention

1️⃣ Creating IAM Roles

"IAM Role for EC2"

This EC2 role needs permissions for:

Administrator Access
AmazonS3FullAccess
AWSCodeDeployFullAccess
AWSCodePipelineFullAccess
AutoScaling Full Access
ELB Full Access

Create IAM Role → AWS Service → EC2 → Attach Policies above → Create Role
Attach this role later when launching the EC2 instance.

"IAM Role for CodeDeploy"

CodeDeploy uses this role to interact with your EC2 instances.

AdministratorAccess
AmazonS3FullAccess
Service: CodeDeploy

2️⃣ Launching an EC2 Instance (Amazon Linux)

AMI: Amazon Linux 2
Instance Type: t2.micro (Free tier)
IAM Role: attach the EC2 role you created
Security Group: open HTTP (80) + SSH (22)

SSH into the instance:

sudo yum install httpd -y
sudo systemctl start httpd
sudo systemctl enable httpd

3️⃣ Installing the CodeDeploy Agent

sudo yum install ruby -y
sudo yum install wget -y
cd /home/ec2-user
wget https://aws-codedeploy-us-east-1.s3.us-east-1.amazonaws.com/latest/install
chmod +x ./install
sudo ./install auto
sudo systemctl start codedeploy-agent
sudo systemctl enable codedeploy-agent

Verify:

sudo systemctl status codedeploy-agent

4️⃣ Creating an AMI from the EC2 Instance

Now that the EC2 instance has a proper role attached:

Apache
CodeDeploy agent

→ Create an AMI:

Actions → Image and Templates → Create Image (AMI)
Include:

Root volume
OS

This AMI becomes your base image for Auto Scaling.

5️⃣ Creating a Launch Template

Go to EC2 → Launch Templates → Create Template
Use the AMI you just created
Attach the same IAM role
Set Auto Scaling Guidance = Enabled
Save template

After this → Terminate the original EC2 instance
(Your ASG will take over from here.)

6️⃣ Create a Load Balancer

Use Application Load Balancer (ALB):
Scheme: Internet-facing
Listeners: HTTP (80)
Target Group: Register later by ASG
Health Check Path: /

7️⃣ Creating the Auto Scaling Group (ASG)

Go to EC2 → Auto Scaling Groups → Create
Select the Launch Template

Select 2 AZs
Attach the Load Balancer Target Group
Desired Capacity: 2
Min: 2
Max: 3

Your ASG will launch 2 EC2 instances from the AMI → both have CodeDeploy agent → both are healthy behind ALB.

8️⃣ Setting Up CodeDeploy

Create Application → Type: EC2/On-premises
Create Deployment Group:
Select Service Role (CodeDeploy IAM Role)
Deployment Type: In-place
Environment: Amazon EC2 Auto Scaling Group
Choose your ASG
Install during deployment: Enable
Load Balancer: Select Target Group

9️⃣ Creating Your App Files on GitHub

Repo structure:

/index.html
/appspec.yml

index.html

<h1>Welcome to my AWS CI/CD Project!</h1>
<p>This deployment was automated using AWS CodeDeploy & CodePipeline.</p>

appspec.yml

version: 0.0
os: linux

files:
  - source: index.html
    destination: /var/www/html

🔟 Creating CodePipeline

Now connect the whole workflow:

Source Stage → GitHub
Connect to your repo
Detect changes automatically
Deploy Stage → CodeDeploy
Select Application
Select Deployment Group

Whenever you push changes to the main branch:

Pipeline is triggered
CodeDeploy pushes the latest files
ASG instances receive an update
ALB routes traffic to healthy updated instances

🎉 Full CI/CD and Auto Scaling setup is complete!

Getting Started with Amazon ECS: A Practical Guide for Real-World Deployments

Muhammad Awais Zahid — Mon, 08 Dec 2025 10:30:36 +0000

Over the past few years, containers have become the backbone of modern application delivery. But running containers at scale? That’s where the real challenge begins.
This is exactly where Amazon Elastic Container Service (ECS) shines.

In this blog, I’ll walk you through ECS in a simple, practical way — the way I wish someone had explained it to me when I started. We’ll talk about what ECS is, how it works, and how to deploy a containerised app without getting lost in the AWS jungle.

What Exactly Is Amazon ECS?

Amazon ECS is AWS’s fully managed container orchestration service.
Think of it as the engine that runs, scales, and monitors your containers — without requiring you to install, upgrade, or maintain control planes.

ECS gives you the flexibility to run containers on:

1. EC2 Launch Type
You manage the servers (EC2 instances), and ECS handles the containers.

2. Fargate Launch Type
Serverless containers — no servers to maintain, patch, or scale. AWS handles all the underlying compute.

Core Concepts

1. Cluster
A logical group where your tasks and services live. Think of it as a home for your containers.

2. Task Definition
This is the “recipe” for your application.
It includes:

Image location
CPU & memory
Ports
Environment variables
IAM roles

If you’re coming from Docker, the task definition is like a more structured Docker run command.

3. Task
A running instance of your task definition. You can run 1 or 1000 — ECS handles scaling.

4. Service
Responsible for keeping a certain number of tasks running at all times. If one task dies, ECS replaces it instantly.

Why Choose ECS Over Kubernetes?

Fully managed (no control plane to manage)
✔️ Deep AWS integration
✔️ Lower overhead for small teams
✔️ Great Fargate support
✔️ Simpler learning curve
✔️ Predictable scaling

Understanding the AWS Serverless Model: A Practical Guide

Muhammad Awais Zahid — Fri, 28 Nov 2025 10:55:59 +0000

Over the last few years, the shift from traditional servers to serverless architecture has changed the way developers build and scale applications. Instead of worrying about servers, patching, or capacity, serverless lets you focus entirely on writing code and delivering value. AWS has played a major role in this movement with its strong ecosystem of serverless services.

In this article, I’ll break down what “serverless” actually means, highlight the core AWS services involved, and share some best practices around security and cost optimisation—two areas that matter a lot when deploying real-world applications.

What Does “Serverless” Really Mean?

Despite the name, “serverless” doesn’t mean that servers disappear. It means that you don’t manage the servers.

With serverless:

You don’t provision instances
You don’t maintain operating systems
You don’t worry about scaling or capacity
You only pay for what you use

AWS automatically handles all the infrastructure behind the scenes. This model allows developers to build applications faster while reducing operational overhead.

Core AWS Serverless Services

AWS provides a wide range of fully managed, event-driven services that work together to build complete serverless applications:

1️⃣ AWS Lambda
The compute backbone of serverless. You upload your code, choose a runtime (Python, Node.js, etc.), and Lambda runs it only when triggered.

2️⃣ Amazon API Gateway
A fully managed service for creating REST or HTTP APIs. It acts as the secure entry point to your backend and integrates smoothly with Lambda.

3️⃣ Amazon DynamoDB
A scalable NoSQL database designed for high-performance serverless applications. It has built-in backups, auto scaling, and near-instant read/write latency.

4️⃣ Amazon EventBridge & Amazon SNS
Event-driven messaging services that help decouple architectures and build reliable workflows.

Security Best Practices in a Serverless Architecture

Security is one of the most important parts of any architecture. With serverless, AWS manages the infrastructure, but you remain responsible for your application logic, permissions, and data protection. Here are some practical best practices:

✔ Apply Least-Privilege IAM Policies
Every Lambda function should have only the permissions it absolutely needs. Avoid using broad policies like DynamoDBFullAccess in production.

✔ Use Environment Variables + KMS Encryption
Store sensitive data (API keys, database names, secrets) in encrypted Lambda environment variables or AWS Secrets Manager.

✔ Enable API Gateway Throttling & Request Validation
This protects your backend from abuse, DDoS-style traffic, and malformed requests.

✔ Keep Lambda Dependencies Small
Smaller packages mean faster cold starts and fewer vulnerabilities.

Serverless Student Management System (AWS)

Muhammad Awais Zahid — Thu, 27 Nov 2025 05:20:10 +0000

Summary

Create DynamoDB Table
Create IAM Role for Lambda
Lambda Function 1 – Get Students
Lambda Function 2 – Insert Student Data
Create API Using API Gateway
Build Frontend on S3
Create CloudFront Distribution

Create DynamoDB Table

Create a Table named "studentData" and Partition Key "studentid" (String). Used to store all student records securely with high availability.

Create IAM Role for Lambda

Created a custom IAM role with AmazonDynamoDBFullAccess permission. This role is attached to both Lambda functions to allow database read/write operations.

Lambda Function 1 – Get Students

Use Lambda Runtime "Python 3.12" and Execution role "Existing IAM role with DynamoDB full access". Added code from getstudents.py and deployed. Function retrieves student data from DynamoDB.

import json
import boto3

def lambda_handler(event, context):
    # Initialize a DynamoDB resource object for the specified region
    dynamodb = boto3.resource('dynamodb', region_name='us-east-1')

    # Select the DynamoDB table named 'studentData'
    table = dynamodb.Table('studentData')

    # Scan the table to retrieve all items
    response = table.scan()
    data = response['Items']

    # If there are more items to scan, continue scanning until all items are retrieved
    while 'LastEvaluatedKey' in response:
        response = table.scan(ExclusiveStartKey=response['LastEvaluatedKey'])
        data.extend(response['Items'])

    # Return the retrieved data
    return data

Lambda Function 2 – Insert Student Data

Use Lambda Runtime "Python 3.12" and the Execution role "same IAM role". Added code from insertstudentdata.py and deployed. Function inserts new student records into DynamoDB.

import json
import boto3

# Create a DynamoDB object using the AWS SDK
dynamodb = boto3.resource('dynamodb')
# Use the DynamoDB object to select our table
table = dynamodb.Table('studentData')

# Define the handler function that the Lambda service will use as an entry point
def lambda_handler(event, context):
    # Extract values from the event object we got from the Lambda service and store in variables
    student_id = event['studentid']
    name = event['name']
    student_class = event['class']
    age = event['age']

    # Write student data to the DynamoDB table and save the response in a variable
    response = table.put_item(
        Item={
            'studentid': student_id,
            'name': name,
            'class': student_class,
            'age': age
        }
    )

    # Return a properly formatted JSON object
    return {
        'statusCode': 200,
        'body': json.dumps('Student data saved successfully!')
    }

Create API Using API Gateway

Use API Type "REST API" (Edge Optimized)
Created two methods:
GET → Linked to Lambda Function 1
POST → Linked to Lambda Function 2
Enabled CORS to allow browser requests.
Deployed the API and copied the invoke URL for frontend integration.

Build Frontend on S3

Created S3 bucket and uploaded index.html + script.js.
Public access blocked (ACL disabled).
Added bucket policy allowing GetObject for CloudFront.
Enabled Static Website Hosting.

[
  {
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["GET", "POST", "PUT", "DELETE", "HEAD"],
    "AllowedOrigins": ["*"],
    "ExposeHeaders": ["ETag"],
    "MaxAgeSeconds": 3000
  }
]

Create CloudFront Distribution

Set index.html as the default root object.
Made the S3 bucket private for better security.
Updated bucket policy to allow CloudFront access.
CloudFront provides global caching and faster content delivery.

OUTPUT

CI/CD pipelines best practices

Muhammad Awais Zahid — Thu, 27 Nov 2025 04:59:17 +0000

💡 Building Efficient CI/CD Pipelines with Jenkins

Recently, I created a Jenkins pipeline to automate the entire build → test → push → deploy workflow. This project helped me understand how continuous integration and continuous delivery (CI/CD) can make software delivery faster, more reliable, and consistent.

Here’s what I learned and the best practices I followed while building Jenkins pipelines 👇

⚙️ CI/CD Best Practices for Jenkins Pipelines

Use Declarative Pipelines:

Keep your pipeline code in a Jenkinsfile inside your source repo for better version control and traceability.

Follow the "Build Once, Deploy Anywhere" Principle

Build your artifact once and promote it through stages (dev → staging → prod) without rebuilding.

Use Stages and Parallel Steps Wisely

Clearly define stages (build, test, push, deploy) and run independent tasks in parallel to save time.

Add Notifications & Error Handling

Include post blocks for success, failure, or always to send Slack/email notifications or perform cleanup.

Keep Secrets Secure

Store credentials and tokens using Jenkins Credentials Manager — never hardcode sensitive data.

Integrate Automated Testing

Run unit and integration tests automatically before deployment to maintain code quality.

Use Environment Isolation

Utilize containers or agents to ensure consistent builds across environments.

Visualize and Monitor Pipelines

The Jenkins Blue Ocean view (as seen in my screenshot) gives an intuitive visualization of each stage’s performance.

Fail Fast, Recover Gracefully

Detect issues early (e.g., failed push or deploy stage) and ensure logs are clear for quick debugging.

Keep It Modular and Reusable

Use shared libraries for common steps so you can reuse pipeline code across multiple projects.

About me... !

Muhammad Awais Zahid — Wed, 17 Sep 2025 16:58:04 +0000

Building and Running a Node.js App with Multi-Stage Docker Builds

Muhammad Awais Zahid — Sun, 07 Sep 2025 16:49:44 +0000

Create an EC2 machine on AWS

SSH using EC2 instance connect option, or MobaXterm or Putty

Install Docker

 sudo yum update -y
 sudo yum install -y docker
 sudo service docker start
 sudo usermod -a -G docker ec2-user

Install git

sudo yum install git -y

Write Dockerfile

# BUILD STAGE
FROM node:18-alpine AS build

WORKDIR /app

COPY package*.json ./
RUN npm ci

COPY . .
RUN npm run build

# DEPLOY STAGE
FROM node:18-alpine

WORKDIR /app

# ✅ Corrected line
COPY --from=build /app/dist ./dist

RUN npm install -g http-server

# EXPOSE PORT
EXPOSE 8080

CMD ["http-server", "dist", "-p", "8080", "-c-1"]

Clone Application Source Code

git clone https://github.com/awais684/Dockerize-node.js-app.git
cp Dockerfile <repditory name>
cd <repository name>

Build Docker image & Run container

docker build -t <image name> .
docker images
docker run -d -p 80:8080 --name cont1 <image name>