DEV Community: Randika Madhushan Perera

Our Synthetic Monitors Were Timing Out. The Scripts Were Fine.

Randika Madhushan Perera — Sun, 18 Oct 2026 07:00:51 +0000

How two compounding infrastructure problems masked a test-quality problem, and the dashboard we built so we'd never debug it blind again.

The complaint

"Most of our synthetic monitors are timing out. It's not the scripts"

That's how it started. Around 140 scripted-browser monitors, a smoke test suite covering two environments running against a New Relic Synthetics private location backed by t2.large 8 EC2s.
Monitors were failing in bulk with JobTimeoutError, Element never became visible, Timed-out waiting for element.

The QA team said scripts worked when run by hand. When SRE team looked at the servers and they seems fine, CPU wasn't pinned, memory was fine, nothing was crash-looping.

Both were right, which is why it took a while.

Background: How a private location actually works

Three properties matter:

A 'lane' is one concurrent heavyweight check. The job manager reserves one CPU for it self, so a 4-vCPU host gets 3 lanes. A browser check needs a full core, Chrome is no a background task.
Job managers pull. You cannot predict or control which host runs which check.
Telemetry arrives on completion. While a check is queued or running, nothing is reported. This is why "it looks fine" and "it's broken" can be both true.

The measurement that settled the argument

Everyone was staring at CPU graphs. The decisive query took thirty seconds and nobody had run it:

SELECT average(executionDuration),
       average(`nr.internalQueueDuration`),
       average(`nr.externalQueueDuration`)
FROM SyntheticCheck WHERE locationLabel = '<our-location>'
SINCE 1 day ago

average execution time          77 s
average internal queue wait      2.5 s
average EXTERNAL queue wait    553 s   ← 9.2 MINUTES

Queue wait was 7x the execution time. Worst observed 1,733 seconds, a check that sat for 29 minutes before it started.

That single ratio ends the "is the scripts or the servers?" argument permanently. The scripts weren't slow. The checks weren't running.

Lesson 1: measure queue wait separately from execution time. It is the one diagnostic that distinguishes a capacity problem from a code problem, and most dashboards don't show it.

Root cause #1: Burstable instances and CPU-bound work

The fleet was 8 t2.large. From AWS.

t2.large property : value

vCPUs: 2 → lanes = 2 − 1 = 1 per host
CPU credits earned/hour: 36
Baseline utilization: 30% per vCPU → 0.6 vCPU sustained, total
Credit accrual cap: 864 (≈24 hours' worth)
Default credit mode: Standard **(T2 — not Unlimited)
Credits on stop/start: **Lost

two separate problems in one line of that table:

Only 8 lanes across the whole fleet. 8 hosts x 1 lane. For 140 monitors.

Lesson 2: never put CPU-bond sustained work on burstable instances. And check CPUCreditBalance in CloudWatch before believing a CPU graph.

Root cause #2: queuing in non-linear

Demand was measurable:

demand                      = 6.35 jobs/min
load-weighted avg runtime   = 77.3 s
lanes needed at 100% busy   = 6.35 × (77.3 / 60) = 8.18
lanes available             = 8
                            ────────────────────
utilization                 = 102%

Not 300% over. Not catastrophically mis-sized. 2% over.

And that's the trap. Queue wait doesn't rise linearly with utilization, it rises asymptotically:

Utilisation 20% 40% 50% 60% 70% 80% 85% 90% 95% 100%
Queue wait ~0 s 1 s 2 s 5 s 14 s 38 s 80 s 170 s 330 s 470 s

Our two measured points sit at either end of that curve: 48% utilization -> 0.4 s, and 102% -> 438 s.

Lesson 3: size for 40-55% average utilization, not 90%. The spare capacity isn't waste; It's what keeps latency bounded under random arrivals.

The counter-intuitive fix: Fever servers

We replaced 8 × t2.large with 4 × m6i.xlarge.

Half the instance count. Here's why that's more capacity, not less:

	8 × t2.large	4 × m6i.xlarge
vCPU per host	2	4
Lanes per host	1	3
Total lanes	8	12
Sustained vCPU per host	0.6 (throttled)	4.0 (full)
Fleet sustained vCPU	4.8	16.0
Monthly cost	~$574	~$577

Same spend. 2.5× the lanes. 3.3× the sustained CPU. The old fleet was paying a premium for capacity it was never allowed to use.

The result

Metric	Before	After
Average queue wait	438 s	0.4 s
Average execution time	78.8 s	54.2 s
Job timeouts	4.7% of checks	3.1%
Lane utilisation	~102%	~48%
Monthly cost	~$574	~$577

A 99.9% reduction in queue wait at flat cost.

The surprise that changed the sizing

Execution time dropped 31% from 78.8 s to 54.2 s for the same scripts. That was the CPU throttling leaving. And it created a feedback loop in the arithmetic:

lanes needed = demand × avg_runtime

Sized with THROTTLED runtime (77 s):  6.35 × 1.29 = 8.2 lanes → ~18 lanes for headroom → 6 hosts
Sized with REAL runtime     (54 s):  6.35 × 0.90 = 5.7 lanes → ~12 lanes for headroom → 4 hosts

The strategy question: add capacity, or fix the schedule?

Monitors on harmonic periods (10/15/30 min) can collide. If all phases align, every monitor is due at the top of the hour at once:

Average demand 5.8/min. Peak demand 116/min. A 20x ratio, and no amount of average capacity fixes a peak like that.

So which problem did we have?

The Dashboard

The diagnosis above took days of ad-hoc NRQL. None of it was hard; all of it was findable only if you knew which five queries to run. We didn't want to rediscover that next quarter, so we built a dashboard.

The key architectural decision: don't build a database

The obvious design is a collector writing to Postgres, an API over it, and a UI. We nearly built that. Then:

New Relic is already the database. NRDB stores all the history and NRQL queries it well. Duplicating it into RDS buys an invoice and a sync bug.

The only reason a backend exists at all is that a browser cannot hold a New Relic User key, that key can read and modify the entire account. So the backends' entire job is to hold one credential and expose a narrow, read-only surface.

That reduced v1 to: static files, one Lambda, no database.

AWS services, and why each one

Service	Role	Why this one
S3	static UI	Account-level Block Public Access made website hosting impossible, which forced OAC, the better pattern anyway
CloudFront	HTTPS, auth, single origin	Serving the API as a second origin on the same distribution means no CORS, and the auth function covers the API too
CloudFront Functions	HTTP Basic at the edge	Cheaper and faster than Lambda@Edge for a header check
API Gateway HTTP API	routing, throttling, authorizer	~70% cheaper than REST API; everything here is a simple proxy
Lambda (arm64)	NerdGraph proxy	Holds the credential. arm64 is ~20% cheaper per ms
Secrets Manager + KMS	the New Relic key	Customer-managed key so usage is auditable and revocable
SSM Parameter Store	non-secret config + generated origin secret	Changeable without a redeploy; SecureString for the machine-only secret
CloudWatch	alarms + ops dashboard	Alarms watch the plumbing; the fleet's own alarms belong in New Relic
CloudFormation	five separate stacks	Separate, not nested, they change at very different rates

Strategies that mattered

Allow-listed panels, never raw NRQL. The browser sends ?panel=fleet,queue. If it could send NRQL, the dashboard would be an open NRDB query endpoint for anyone who got past the password. Adding a panel is a code change and a code review, deliberately.

One batched NerdGraph document per refresh. All panels in a single request using GraphQL aliases, cached 20 s server-side. Ten viewers cost the same New Relic quota as one.

Alias every NRQL aggregate. This one bit us hard enough to earn its own section below.

Defence in depth on the API. CloudFront injects a shared secret header; an API Gateway authorizer validates it; the proxy Lambda validates it again. If the authorizer is ever accidentally detached, the API doesn't silently open.

Reject before compute. Because the authorizer declares IdentitySource: $request.header.x-origin-secret, API Gateway returns 401 itself when the header is absent — no Lambda invocation at all. We proved it: 15 unauthenticated requests produced zero invocations on either function. Combined with a 5 req/s throttle, worst-case abuse cost fell from ~$97/month to ~$13/month.

Keep the Lambda out of the VPC. It only calls public endpoints. In-VPC would add ENI cold starts and NAT data charges for nothing.

Poll; don't build SSE. Underlying data only changes once a minute. The UI polls every 20 s and animates between snapshots client-side — visually live, with no WebSocket machinery.

What the dashboard shows that New Relic doesn't

New Relic ships a perfectly good private-location dashboard. We import it. The custom app exists only for what doesn't exist elsewhere:

Queue wait as a first-class column next to execution time — the ratio that ends the capacity-versus-code argument
A live flow view: queue → lanes → results, one dot per real check, coloured by outcome
Per-monitor capacity consumption — which monitors eat your lanes, ranked
Missing-check detection — expected (from configured period) versus observed
A what-if simulator — change lane count and period, see predicted queue before buying anything

One honest limitation: SyntheticCheck.minion returns "unknown" in our account, so per-check host attribution isn't available from telemetry. The flow view shows lane occupancy accurately but fills slots left-to-right by count, not identity. True per-lane attribution needs an agent on the hosts, which we deliberately deferred, because reading the Docker socket is root-equivalent and that's a conversation with the security team, not a code change.

The Dashboard UI

Video: https://youtu.be/rN8jl0jQT7o

A Pet Re-homing platform that costs $29 a month to run

Randika Madhushan Perera — Sun, 11 Oct 2026 16:24:16 +0000

The Architecture of PawReHome - 24 lambdas, 68 routes, Aurora Serverless v2 with no connection pool anywhere and the single constraint that decided all of it.

Series: Building PawReHome Ep-01

PawReHome is a pet-rehoming platform: people post stray or surrendered animals, adopters apply, admins verify listings, rescue organizations manage their own rosters.

The interesting part is the constraint I built it under: it had to cost almost nothing while idle. No funding, no revenue, no guarantee anyone would use it.

That one requirement decided the database, the way lambda talks to it, the networking and the error handling. This post is the map. This links to are where the actual difficulty lives.

The Stack

Layer and Choice

Frontend - Next.js 14.2.35, App Router, Tailwind, 30+ pages
Hosting - AWS Amplify Hosting (CloudFront underneath)
API - API Gateway REST plus WebSocket API
Compute - 24 Lambda functions, 256 MB 29s timeout
Database - Aurora PostgreSQL Serverless v2 via RDS Data API
Auth - Cognito user pool, group based roles
Storage - S3 for listing images
Email - SES
IaC - AWS SAM/ CloudFormation

The Shape of it

One Request, end to end

What is costs

Line - Cost (2026 September)

Aurora Serverless v2 ACU-hours - $29.21
Aurora storage I/O - $0.33
Aurora storage - $0.01
RDS Data API requests - $0.00
Lambda, API Gateway, S3, Cognito, SES - $0.00 at this time

See you in the episode 02 :)

Building a Serverless Life Visualization App with Next.js, AWS Lambda, and DynamoDB Single-Table Design - #01

Randika Madhushan Perera — Mon, 01 Jun 2026 15:09:09 +0000

How we built yourdeath.date - a memento mori tool that renders your life as a grid of weeks - on a fully serverless AWS stack

Introduction

Most apps are built to keep you engaged as long as possible. We built one designed to remind you that your time is finite.

yourdeath.date visualizes your entire life from birth to projected death as a grid of small squares. Each square is one week. Filled squares are weeks already spent. The rest are blank.

The app also lets users log "good deeds" onto specific weeks, pin life milestones to the grid, share their grid publicly, and connect with a community of people thinking about how they spend their time.

This article covers the technical architecture behind it. The AWS services, the serverless design decisions, the DynamoDB single-table schema, the auth flow, and the frontend rendering approach.

What is this Application?

Frontend -> Next.js 14 (App Router), React 18, TypeScript, Tailwind CSS
Backend -> AWS Lambda (Node.js 22.x), API Gateway REST
Database -> DynamoDB (single-table design, 3 GSIs)
Auth -> Amazon Cognito (User Pool + Identity Pool)
Email -> Amazon SES + Secrets Manager
Storage Amazon -> S3
Scheduling -> Amazon EventBridge
IaC -> AWS SAM (Serverless Application Model)
Hosting -> AWS Amplify (frontend), API Gateway (backend)

High Level Architecture

Application FLow

User Onboarding Flow

User visits yourdeath.date
Clicks "See your life →"
Redirected to /auth (Cognito hosted UI or email/password)
OAuth callback → /auth/callback
Amplify exchanges code for tokens (access + id + refresh)
POST /auth/sync → Lambda syncs Cognito identity to DynamoDB
Redirect to /onboarding → user enters DOB + life expectancy
PUT /user/profile → stores profile in DynamoDB
Redirect to /dashboard → LifeGrid renders with real data

Deed Logging Flow

User opens community page
Frontend reads current week index: Math.floor(msAlive / msPerWeek)
User fills deed form (description, category, helpedWho, isPublic)
POST /deeds → Lambda validates, stores in DynamoDB
- PK: USER#{userId} SK: DEED#{weekIndex:6-padded}#{deedId}
- GSI1PK: COMMUNITY#PUBLIC (if public) or USER#{userId}#PRIVATE
Dashboard: GET /deeds/me returns {weekIndex → count} map
LifeGrid highlights that week column green

Weekly Email Flow

EventBridge cron(0 6 ? * MON *) → WeeklyEmailFunction
→ ScanCommand: FilterExpression = emailOptIn = true
→ For each user:
→ Fetch weeks lived, weeks remaining
→ Generate HMAC token: sha256(userId:email, secret)
→ Build HTML email with unsubscribe link
→ SES SendEmail (10-message batches, 100ms throttle)

AWS Services Connectivity Flow

Interesting Engineering Decisions

Single-Table DynamoDB vs Multi-Table

We choose single table design to minimize cold-start latency (one connection pool) and reduce operational overhead. The trade-off is query complexity, GSI3 exists purely to solve the deed-by-ID lookup problem that arises from putting deeds under USER#{id} as primary key.

HMAC Unsubscribe Tokens

Rather than storing one-time tokens in DynamoDB, the unsubscribe system uses a stateless HMAC approach:

token = base64url( HMAC-SHA256( "{userId}:{email}", secret ) )

Validation is a single HMAC re-computation + constant-time comparison. No database lookup required. The secret lives in Secrets Manager and is fetched once per Lambda cold start.

Conclusion

The stack — Next.js on Amplify, API Gateway, Lambda, DynamoDB, Cognito — is a proven serverless pattern that scales to thousands of concurrent users with near-zero operational overhead. The interesting parts were the DynamoDB single-table schema design (especially GSI3 for deed lookup), the theme-aware LifeGrid renderer, and keeping analytics collection genuinely privacy-respecting.

The full codebase lives at yourdeath.date. If you have questions about any of the architecture decisions, drop them in the comments.

Built with AWS SAM · Next.js 14 · DynamoDB · Cognito · TypeScript

I Found a Critical Security Bug on Foundit.sg — Here's What Happened

Randika Madhushan Perera — Mon, 18 May 2026 15:44:10 +0000

How It Started

It began as a completely ordinary moment. A user (let's call them User A) received a job suggestion email from Foundit.sg — a job platform widely used across Singapore and Southeast Asia. Nothing unusual there.

But here's where it got interesting.

When User A forwarded **that job suggestion email to a friend (User B), something unexpected happened. User B could click the link inside that forwarded email and — without entering any credentials — **be automatically logged into User A's account.

Not just to view a job listing. Into the full account. With access to view, edit, and modify all of User A's personal details, job preferences, resume, and profile information.

The Bug Explained

The root cause is a classic but serious security flaw: unprotected magic links / auto-login tokens.
Foundit's job suggestion emails contained a one-click authentication link that:

Was not single-use (it could be clicked multiple times)
Was not bound to the recipient's device or IP address
Had no secondary verification step

This means anyone who received the forwarded email — intentionally or accidentally — could gain full unauthorized access to another person's account.
Impact:

✅ Unauthorized account access
✅ Personal data exposure (name, contact details, resume, etc.)
✅ Profile editing without consent
✅ Potential misuse of saved and applied jobs

Severity: CRITICAL

What I Did — Responsible Disclosure

I did not exploit the bug. I did not share it publicly. I followed the responsible *disclosure * process:

Emailed Foundit at info@foundit.sg with a full, detailed bug report including steps to reproduce, impact analysis, and suggested fixes
Waited for a response — no reply came via Email
Posted a gentle nudge on LinkedIn asking if they received my report — After several days they replied to me
Submitted to Open Bug Bounty to create an official public record of responsible disclosure

The Bug Was Fixed — But Not Properly

Eventually, the bug was addressed. However, the "fix" implemented was not a proper security patch.
Instead of fixing the underlying authentication token issue, they added an error message that reads:

"Ooops! We couldn't find this job under your current country preference. It might be listed under a different location. Try switching your country in Settings to view the job."

This is a UI-level workaround, not a true security fix. The root cause — insecure auto-login tokens — may still exist in other parts of the platform.

Zero Acknowledgment

Throughout this entire process:

❌ No reply to my email
✅ Response to my LinkedIn post - After several Days, But no mentioning further details.
❌ No acknowledgment of the report
❌ No credit or mention anywhere

I'm not sharing this to shame Foundit. I'm sharing this because responsible disclosure deserves acknowledgment, and the security community deserves to know how platforms respond (or don't respond) to good-faith reports.

Lessons for Developers

If you're building a platform with email-based authentication links, please ensure:

Magic links are single-use — expire immediately after first click
Bind tokens to the recipient's identity — not just a URL
Add device/IP verification for sensitive actions
Have a clear security reporting channel — and actually respond to it
Acknowledge responsible disclosures — even a simple "thank you" email goes a long way

Relevant Authorities (Singapore)

If you encounter similar issues on Singapore-based platforms, you can report to:

Cyber Security Agency of Singapore (CSA) — csa.gov.sg
Personal Data Protection Commission (PDPC) — pdpc.gov.sg

Final Thoughts

Cybersecurity is a community effort. When someone takes the time to responsibly report a vulnerability instead of exploiting it, that deserves recognition — not silence.

I hope this story encourages platforms to take security reports seriously, and encourages more people to report vulnerabilities responsibly rather than ignore or exploit them.

Found this useful? Share it with your network. The more awareness, the safer our digital spaces become.

Building a Scalable Pet Rehoming Platform with AWS Serverless Architecture

Randika Madhushan Perera — Fri, 09 Jan 2026 09:49:08 +0000

Introduction

PawReHome is a modern, full-featured pet rehoming platform that connects loving families with pets in need. Build with a serverless-first approach, the platform leverages AWS managed services to provide a scalable, cost-effective and secure solution for animal rescue organizations and induvial users.

This article explores the AWS architecture behind the PawReHome, how several AWS services work together to create a robust, role based access control, real-time data management.

Project Overview

PawReHome is a application fully integrated with AWS backend services. The platform supports:

User authentication and authorization with role-based access control
Multi-tenant organization management for rescue groups
Pet listing management with approval workflows
Image upload for pet photos
Adoption applications and inquiry systems
Platform administration for content moderation
Featured pets curation for homepage visibility

AWS Architecture Overview

This utilizes a serverless architecture that eliminates server management overhead and provides automatic scaling.

Complete AWS services stack:

AWS Amplify: Frontend hosting, CI/CD and deployment
Amazon Cognito: User authentication and authorization
AWS Lambda: Serverless backend functions
Amazon API Gateway: RESTFUL API management and routing
DynamoDB: NoSQL database for all application data
S3: Object storage for pet images
IAM: Identity and access management

Architecture Flow

High-Level Architecture Diagram

Deep Dive: AWS Services Implementation

1. AWS Amplify - Frontend Hosting & CI/CD

Purpose: As you can see Amplify hosts the Next.js frontend application and provides continues deployment capabilities.

Implementation Details:

The amplify.yml configuration defines the build process
Automatic builds triggered on Git commits
Environment variables managed through Amplify Console
Global CDN ensures low latency worldwide
Built-in SSL certificate provisioning via AWS Certificate Manager

Benefits of the Amplify

Zero server management, you do not need to worry about.
Automatic scaling when traffic spikes.
Preview deployment.
Rollback capabilities when you have an issue.

2. Cognito - Authentication & Authorization

Cognito provides secure user access and authentication across your application. This has role-based access control (RBAC) thorough Cognito groups.

Let me explain the Authorization Flow:

User logs in via Cognito
Cognito issues JWT token with user claims
Token includes cognito:groups claim with user's groups
API Gateway validates token
Lambda functions check group membership for authorization
DynamoDB queries are filtered based on user permissions

3. Lambda - Serverless Backend Functions

AWS Lambda functions handle all the business logic, data processing and API responses without managing servers. How relief :)

Example Lambda function (pet listing):

const { DynamoDBClient } = require("@aws-sdk/client-dynamodb");
const { DynamoDBDocumentClient, PutCommand, ScanCommand } = require("@aws-sdk/lib-dynamodb");

const docClient = DynamoDBDocumentClient.from(new DynamoDBClient({}));
const TABLE_NAME = "PawReHome-Pets";

exports.handler = async (event) => {
  // Handle pet rehoming submission
  if (event.httpMethod === "POST") {
    const pet = {
      id: uuidv4(),
      ...JSON.parse(event.body),
      status: "pending",
      createdAt: new Date().toISOString()
    };

    await docClient.send(new PutCommand({
      TableName: TABLE_NAME,
      Item: pet
    }));

    return { statusCode: 201, body: JSON.stringify(pet) };
  }

  // Get approved pets
  if (event.httpMethod === "GET") {
    const result = await docClient.send(new ScanCommand({
      TableName: TABLE_NAME,
      FilterExpression: "#status = :status",
      ExpressionAttributeNames: { "#status": "status" },
      ExpressionAttributeValues: { ":status": "approved" }
    }));

    return { statusCode: 200, body: JSON.stringify(result.Items) };
  }
};

4. Amazon API Gateway - RESTful API Management

API gateway servers as the front door for all backend API requests, providing routing, authentication and rate limiting. (Do not forget about the rate limit, your pocket will empty 😉 and will have some concerns. 🛡️)

API Gateway Configuration:

REST API endpoint
CORS enabled for frontend cross-origin requests
Cognito Authorizer for automatic JWT validation
Lambda Proxy Integration for seamless function invocation
Stage Variables for environment management

5. DynamoDB - NoSQL Database

DynamoDB stores all application data with single-digit millisecond latency and automatic scaling.

Table Design as below:

PawReHome-Pets

Partition Key: id (String)
Attributes: name, species, breed, age, size, gender, description, 
           images[], userId, organizationId, status, featured, 
           createdAt, updatedAt
GSIs:
  - UserIdIndex (userId) - Query user's pets
  - OrgIdIndex (organizationId) - Query organization's pets

PawReHome-Organizations

Partition Key: id (String)
Attributes: name, slug, email, phone, description, logo, 
           status, verified, createdAt, reviewedBy, reviewedAt
GSIs:
  - SlugIndex (slug) - Query by URL slug

PawReHome-OrgMembers

Partition Key: organizationId (String)
Sort Key: userId (String)
Attributes: role (admin/member), joinedAt, invitedBy
GSIs:
  - UserIdIndex (userId) - Query user's organizations

PawReHome-Applications

Partition Key: id (String)
Attributes: petId, userId, status, applicantInfo, 
           submittedAt, reviewedAt
GSIs:
  - UserIdIndex (userId) - Query user's applications
  - PetIdIndex (petId) - Query pet's applications

PawReHome-Questions

Partition Key: id (String)
Attributes: petId, userId, status, applicantInfo, 
           submittedAt, reviewedAt
GSIs:
  - UserIdIndex (userId) - Query user's applications
  - PetIdIndex (petId) - Query pet's applications

PawReHome-Favorites

Partition Key: userId (String)
Sort Key: petId (String)
Attributes: addedAt

6. S3 - Image Storage

S3 uses to stores pet photos with public read access for displaying images on the website.

to be continued.....

Made with ❤️ for pets everywhere

Building a Full-Stack E-Commerce Platform with AWS

Randika Madhushan Perera — Sat, 27 Dec 2025 10:59:22 +0000

A Technical Deep-Dive into Modern Serverless Architecture

Introduction

I recently build an e-commerce website that handles custom gifts orders with image uploads, payment processing, order tracking, and admin dashboard and all running on a completely serverless architecture. This blog walks through the complete technical decisions, architecture patterns and lessons learned.

The Tech Stack that I used

Layer: Frontend Framework
Technology: Next.js 15
Why I choose it: Server components, built in API routs

Layer: Database
Technology: Amazon DynamoDB
Why I choose it: Serverless, pay-per-request, automatic scaling

Layer: File Storage
Technology: S3
Why I choose it: Reliable, cheap, presigned URLs for secure access

Layer: Authentication
Technology: Cognito
Why I choose it: Managed auth, MFA support, integrates with Amplify

Layer: Email Service
Technology: Amazon SES
Why I choose it: Transactional emails, high deliverability, low cost

Layer: Hosting
Technology: Amazon Amplify Gen 2
Why I choose it: Git based deployment, automates SSL, CDN included.

Architecture Overview

The Customer Journey

Step by Step:

Landing Page - Customer learns about the product and clicks "Order Now"
Customization - Customer writes their custom message and optionally uploads a photo
Address Entry - Customer enters recipient shipping address and their billing details
Payment - Redirected to Payment Gateway
Success - After payment, customer sees their order ID and tracking link
Email - Customer receives confirmation email with order details
Tracking - Customer can check order status anytime using their order ID

Payment Flow (The Tricky Part)

Image Upload Architecture

Why I Used Presigned URLs?

Instead of making the S3 bucket public, I use presigned URLs that:

Expire after 1 hour
Are generated on-demand
Keep the bucket private and secure
Work with CloudFront CDN

Email Notification System

Email Features

HTML templates with incline CSS for email client compatibility
Multiple admin recipients
Reply-to headers, so admin can respond directly to customers
Order details formatted for easy reading and fulfillment

Deployment Pipeline

The entire deployment is automated,

Push the code to GitHub
Amplify detects the change
Builds the Next.js application
Injects environment variables
Deploys to CloudFront CDN
Site is live in 305 minutes

Deep Dive: AWS Services Powering This Application

AWS Services Architecture

1. AWS Amplify Gen 2

What is it?

AWS Amplify Gen 2 is a complete development platform that handles hosting, CI/CD, and backend infrastructure for web applications. It's the second generation of Amplify.

Why I chose it?

Git based deployments: Push to GitHub --> Automatically deployed into AWS
Built in CI/CD: No Jenkins, No GitHub Actions
Free SSL certificates: HTTPS enabled automatically
Global CDN: CloudFront is included
Custom domains: Easy DNS configurations
Environment variables: Secure secrets management
Preview Deployment: Test PRs before merging

How It's Used in This Project

2. Amazon DynamoDB

What is it?

DynamoDB is a fully managed NoSQL database service that provides single-digit millisecond performance at any scale. It's serverless, no provisioning, no patching or management required.

Why I chose it?

Serverless, Auto-scaling
Pay-per request
Single digit ms latency
AWS-native integration

3. Amazon S3

What is it?

S3 is object storage built for storing and retrieving any amount of data. It's the backbone of AWS storage.

Why I chose it?

Unlimited storage
Presigned URLs
Cheap
Direct SDK access

S3 Features Used

Private Bucket: Images not publicly accessible
Presigned URLs: Temporary access links (1 hour)
Server-side Upload: API route handles upload, not browser
Content-Type: Proper MIME types for images

Security Configuration

Bucket Policy: Private by default
CORS: Enabled for upload endpoints
Presigned URLs: Time-limited access
IAM: Scoped permissions for put/get only

4. Amazon Cognito

What is it?

Amazon Cognito provides authentication, authorization and user management for web and mobile apps. Users can sign in directly or through third party identity providers.

Why I chose it?

AWS-native
MFA support

How It's Used

5. Amazon SES (Simple Email Service)

What is it?

Amazon SES is a cloud based email service for sending transactional, marketing and notification emails.

How It's Used

6. Amazon CloudFront (via Amplify)

What is it?

CloudFront is a content delivery network that delivers content with low latency from 400+ edge locations worldwide.

How It's Used

Terraform-CodeGen0: A Terraform Code Generator

Randika Madhushan Perera — Sun, 12 Jan 2025 20:58:48 +0000

This is a submission for the GitHub Copilot Challenge : New Beginnings

What I Built

When we are deploying an application, especially in a cloud environment, we often need to manage the various infrastructure components, such as VPC, Servers, Databases, CDN, etc. Doing this manually is somewhat time-consuming and challenging. To overcome this, we can use terraform.

Terraform allows the development of the architecture that deploys the application in a cloud environment.

But writing the terraform is also somewhat time-consuming we need to create those codes manually, plus we need to know the terraform.

Using this Terraform-CodeGen0 you can easily generate the terraform codes. All you need to input the VPC name, Subnets, Route Tabels, IGW, and NAT details into this application. This Terraform-CodeGen0 provides your terraform code.

Demo

Repo

RandiakM / Terraform-CodeGen0

GitHub Copilot 1-Day Build Challenge: This is a Terraform Code generation application for VPC

VPC Terraform Generator

Introduction

The VPC Terraform Generator is a tool designed to generate Terraform code for AWS infrastructure components such as VPCs, Subnets, Route Tables, Internet Gateways (IGWs), and NAT Gateways. This project uses a microservices architecture and is hosted on Netlify, with MongoDB as the database.

How to Use

The VPC Terraform Generator allows users to generate Terraform code for various AWS components through a web interface. Users can input the necessary details for each component, and the application will generate the corresponding Terraform code.

Installation Steps

Clone the repository:

git clone https://github.com/your-repo/vpc-terraform-generator.git
cd vpc-terraform-generator

Install backend dependencies:

cd api
npm install

Install frontend dependencies:

cd ../frontend
npm install

Set up environment variables: Create a .env file in the api directory with the following content:

MONGODB_URI=your_mongodb_connection_string

Start the backend server:

cd ../api
npm run dev

Start the frontend server:

cd ../frontend
npm start

Netlify Hosting Steps

Create a…

View on GitHub

Copilot Experience

So for the initial start, I have used the below prompt to get an idea of this project.

Build a Terraform code generator for AWS infrastructure components (VPCs, Subnets, Route Tables, IGWs, NAT, etc.) using a microservices architecture. The application will be hosted on Netlify, and MongoDB will be used as the database.

Starting project structure:

Terraform Code Generation:

Ask to implement the utility functions to generate Terraform code-based user inputs for each resource type.

Edits:

Auto Complete:

For this code auto-completion, I provide the description as a comment and start to create the function, then Copilot starts suggesting the code sections that are related to my task.

Documentation Creation:

Used the below prompt:

@workspace create a README with the below details:

1. Introduction 
2. How to use 
3. Installation Steps 
4. Netlify Hosting Steps 
5. User guide

Conclusion

Through this challenge and the journey, Copilot was an AI pair programmer who helped me build this application. From the start and even from the ending of this project Copilot was really helped me.

Copilot reviewed the codes and provide alternative solutions to overcome some issues when developing this application.

With the help of Copilot I was able to create this amazing application within just 24 hours, Wihtout copilot help this may take long.

Configuring AWS WAF, CloudFront, and S3 Bucket for Secure Access

Randika Madhushan Perera — Fri, 03 Jan 2025 08:49:10 +0000

How to Set Up AWS WAF, CloudFront and an S3 Bucket to serve content securely and Restrict content Access to certain originating sources only. This guide looks at some issues, for instance, 403 Forbidden, in order to troubleshoot the problem.

1. Objective

The purpose of this assignment is to deploy AWS WAF, CloudFront and S3 bucket:

Restrict Access to certain origins, for example, https://dev.test.com and https://int.test.com.
When the CloudFront distribution is accessed, the index.html page should be served by default.
To prevent unauthorized access and to correct common errors.

2. You will require.

And AWS S3 bucket called, sample-bucket, where the files for the static website are stored.

An AWS CloudFront distribution created with OAC linked to the S3 bucket.

3. Configuration Steps

Step 1: Make Sure S3 Bucket Policy has below

Open the S3 Console.
Select your bucket (e.g., sample-bucket).
Check the following bucket policy to allow access from CloudFront:

{
    "Version": "2008-10-17",
    "Id": "PolicyForCloudFrontPrivateContent",
    "Statement": [
        {
            "Sid": "AllowCloudFrontServicePrincipal",
            "Effect": "Allow",
            "Principal": {
                "Service": "cloudfront.amazonaws.com"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::sample-bucket/*",
            "Condition": {
                "StringEquals": {
                    "AWS:SourceArn": "arn:aws:cloudfront::AWS_ACC_ID:distribution/CLOUDFRONT_ID"
                }
            }
        }
    ]
}

Step 2: Configure CORS for the S3 Bucket

In the S3 Console, go to the Permissions tab.
Edit the CORS configuration and set the following:

[
    {
        "AllowedHeaders": ["*"],
        "AllowedMethods": ["GET", "HEAD", "PUT", "POST", "DELETE"],
        "AllowedOrigins": [
            "https://dev.test.com",
            "https://int.test.com"
        ]
    }
]

Replace AllowedOrigins with your allowed domains.
Restrict AllowedHeaders to necessary headers for production.

Step 3: Configure AWS WAF

Create WAF Rules:

Open the AWS WAF Console and select your Web ACL.
Add the following rules:

Rule 1: Allow-Traffic

Description: Allows requests from specific origins.
Rule Definition:

{
    "Name": "Allow-Traffic",
    "Priority": 1,
    "Statement": {
        "OrStatement": {
            "Statements": [
                {
                    "ByteMatchStatement": {
                        "SearchString": "https://dev.test.com",
                        "FieldToMatch": {
                            "SingleHeader": {
                                "Name": "origin"
                            }
                        },
                        "TextTransformations": [
                            {
                                "Priority": 0,
                                "Type": "NONE"
                            }
                        ],
                        "PositionalConstraint": "EXACTLY"
                    }
                },
                {
                    "ByteMatchStatement": {
                        "SearchString": "https://int.test.com",
                        "FieldToMatch": {
                            "SingleHeader": {
                                "Name": "origin"
                            }
                        },
                        "TextTransformations": [
                            {
                                "Priority": 0,
                                "Type": "NONE"
                            }
                        ],
                        "PositionalConstraint": "EXACTLY"
                    }
                }
            ]
        }
    },
    "Action": {
        "Allow": {}
    },
    "VisibilityConfig": {
        "SampledRequestsEnabled": true,
        "CloudWatchMetricsEnabled": true,
        "MetricName": "Allow-Traffic"
    }
}

Rule 2: Block-All-Other-Traffic

Description: Blocks all requests that don’t match the allowed origins.
Rule Definition:

{
    "Name": "Block-All-Other-Traffic",
    "Priority": 2,
    "Statement": {
        "NotStatement": {
            "Statement": {
                "OrStatement": {
                    "Statements": [
                        {
                            "ByteMatchStatement": {
                                "SearchString": "https://dev.test.com",
                                "FieldToMatch": {
                                    "SingleHeader": {
                                        "Name": "origin"
                                    }
                                },
                                "TextTransformations": [
                                    {
                                        "Priority": 0,
                                        "Type": "NONE"
                                    }
                                ],
                                "PositionalConstraint": "EXACTLY"
                            }
                        },
                        {
                            "ByteMatchStatement": {
                                "SearchString": "https://int.test.com",
                                "FieldToMatch": {
                                    "SingleHeader": {
                                        "Name": "origin"
                                    }
                                },
                                "TextTransformations": [
                                    {
                                        "Priority": 0,
                                        "Type": "NONE"
                                    }
                                ],
                                "PositionalConstraint": "EXACTLY"
                            }
                        }
                    ]
                }
            }
        }
    },
    "Action": {
        "Block": {}
    },
    "VisibilityConfig": {
        "SampledRequestsEnabled": true,
        "CloudWatchMetricsEnabled": true,
        "MetricName": "Block-All-Other-Traffic"
    }
}

4. Testing and Verification

Test Allowed Origins

Run the following command:

curl -X GET https://cloudfront_domain_name \
     -H "Origin: https://dev.test.com" \
     -v

Expected Result: HTTP 200 OK.

Test Disallowed Origins

Run the following command:

curl -X GET https://cloudfront_domain_name \
     -H "Origin: https://unauthorized.com" \
     -v

Expected Result: HTTP 403 Forbidden.

Check WAF Logs

Enable logging for the Web ACL in the AWS WAF Console.
Monitor logs in CloudWatch to verify rule matches.

Full Stack Application Hosting in AWS

Randika Madhushan Perera — Mon, 30 Dec 2024 16:09:30 +0000

Todo Application Documentation

Introduction
Architecture Overview
Backend Setup
Frontend Setup
AWS Configuration
Deployment
Troubleshooting
Maintenance and Updates

1. Introduction

This document provides a comprehensive guide for setting up, deploying, and maintaining a full-stack Todo application using Node.js, React, Express, and MongoDB. The application is deployed on AWS, utilizing services such as Lambda, API Gateway, S3, and CloudFront.

2. Architecture Overview

Backend: Node.js with Express, deployed as an AWS Lambda function
Frontend: React, built with Vite, hosted on S3 and served via CloudFront
Database: MongoDB Atlas
API: AWS API Gateway
Authentication: (To be implemented)

3. Backend Setup

3.1 Lambda Function

Create a new file named index.js:

const mongoose = require('mongoose');

let cachedDb = null;

async function connectToDatabase() {
  if (cachedDb) {
    return cachedDb;
  }

  const connection = await mongoose.connect(process.env.MONGODB_URI, {
    useNewUrlParser: true,
    useUnifiedTopology: true,
  });

  cachedDb = connection;
  return connection;
}

const todoSchema = new mongoose.Schema({
  title: { type: String, required: true },
  completed: { type: Boolean, default: false },
  createdAt: { type: Date, default: Date.now },
  updatedAt: { type: Date, default: Date.now },
});

const Todo = mongoose.model('Todo', todoSchema);

exports.handler = async (event) => {
  const corsHeaders = {
    'Access-Control-Allow-Origin': 'https://your-cloudfront-domain.cloudfront.net',
    'Access-Control-Allow-Headers': 'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token',
    'Access-Control-Allow-Methods': 'GET,POST,PUT,DELETE,OPTIONS',
    'Access-Control-Allow-Credentials': 'true'
  };

  if (event.httpMethod === 'OPTIONS') {
    return {
      statusCode: 200,
      headers: corsHeaders,
      body: JSON.stringify({ message: 'CORS preflight request successful' })
    };
  }

  try {
    await connectToDatabase();

    const { httpMethod, resource, pathParameters, body } = event;

    switch (`\${httpMethod} \${resource}`) {
      case 'GET /todos':
        const todos = await Todo.find().sort({ createdAt: -1 });
        return { 
          statusCode: 200, 
          headers: { ...corsHeaders, 'Content-Type': 'application/json' },
          body: JSON.stringify(todos) 
        };

      case 'POST /todos':
        const newTodo = new Todo(JSON.parse(body));
        const savedTodo = await newTodo.save();
        return { 
          statusCode: 201, 
          headers: { ...corsHeaders, 'Content-Type': 'application/json' },
          body: JSON.stringify(savedTodo) 
        };

      case 'PUT /todos/{id}':
        const updatedTodo = await Todo.findByIdAndUpdate(
          pathParameters.id,
          { ...JSON.parse(body), updatedAt: Date.now() },
          { new: true }
        );
        if (!updatedTodo) {
          return { 
            statusCode: 404, 
            headers: { ...corsHeaders, 'Content-Type': 'application/json' },
            body: JSON.stringify({ message: 'Todo not found' }) 
          };
        }
        return { 
          statusCode: 200, 
          headers: { ...corsHeaders, 'Content-Type': 'application/json' },
          body: JSON.stringify(updatedTodo) 
        };

      case 'DELETE /todos/{id}':
        const deletedTodo = await Todo.findByIdAndDelete(pathParameters.id);
        if (!deletedTodo) {
          return { 
            statusCode: 404, 
            headers: { ...corsHeaders, 'Content-Type': 'application/json' },
            body: JSON.stringify({ message: 'Todo not found' }) 
          };
        }
        return { 
          statusCode: 200, 
          headers: { ...corsHeaders, 'Content-Type': 'application/json' },
          body: JSON.stringify({ message: 'Todo deleted successfully' }) 
        };

      default:
        return { 
          statusCode: 400, 
          headers: { ...corsHeaders, 'Content-Type': 'application/json' },
          body: JSON.stringify({ message: 'Invalid request' }) 
        };
    }
  } catch (error) {
    console.error('Error:', error);
    return { 
      statusCode: 500, 
      headers: { ...corsHeaders, 'Content-Type': 'application/json' },
      body: JSON.stringify({ message: 'Internal server error' }) 
    };
  }
};

3.2 Package Configuration

Create a package.json file:

{
  "name": "todo-api-lambda",
  "version": "1.0.0",
  "description": "Todo API Lambda Function",
  "main": "index.js",
  "dependencies": {
    "mongoose": "^6.0.0"
  }
}

3.3 Deployment Package

Install dependencies: npm install
Create a ZIP file: zip -r function.zip index.js node_modules

4. Frontend Setup

4.1 Create React App

Create a new Vite project: npm create vite@latest client -- --template react
Navigate to the project directory: cd client
Install dependencies: npm install

4.2 API Service

Create a file named src/services/todoService.js:

import axios from 'axios';

const API_URL = 'https://your-api-gateway-url.execute-api.us-west-2.amazonaws.com/prod/todos';

const api = axios.create({
  baseURL: API_URL,
  withCredentials: true
});

export const getTodos = async () => {
  const response = await api.get('');
  return response.data;
};

export const createTodo = async (title) => {
  const response = await api.post('', { title });
  return response.data;
};

export const updateTodo = async (id, updates) => {
  const response = await api.put(`/\${id}`, updates);
  return response.data;
};

export const deleteTodo = async (id) => {
  await api.delete(`/\${id}`);
};

4.3 Main App Component

Update src/App.jsx:

import React, { useState, useEffect } from 'react';
import { getTodos, createTodo, updateTodo, deleteTodo } from './services/todoService';
import './App.css';

function App() {
  const [todos, setTodos] = useState([]);
  const [newTodo, setNewTodo] = useState('');

  useEffect(() => {
    fetchTodos();
  }, []);

  const fetchTodos = async () => {
    const fetchedTodos = await getTodos();
    setTodos(fetchedTodos);
  };

  const handleCreateTodo = async (e) => {
    e.preventDefault();
    if (newTodo.trim()) {
      const createdTodo = await createTodo(newTodo);
      setTodos([createdTodo, ...todos]);
      setNewTodo('');
    }
  };

  const handleUpdateTodo = async (id, updates) => {
    const updatedTodo = await updateTodo(id, updates);
    setTodos(todos.map(todo => todo._id === id ? updatedTodo : todo));
  };

  const handleDeleteTodo = async (id) => {
    await deleteTodo(id);
    setTodos(todos.filter(todo => todo._id !== id));
  };

  return (
    <div className="App">
      <h1>Todo App</h1>
      <form onSubmit={handleCreateTodo}>
        <input
          type="text"
          value={newTodo}
          onChange={(e) => setNewTodo(e.target.value)}
          placeholder="Add a new todo"
        />
        <button type="submit">Add Todo</button>
      </form>
      <ul>
        {todos.map(todo => (
          <li key={todo._id}>
            <input
              type="checkbox"
              checked={todo.completed}
              onChange={() => handleUpdateTodo(todo._id, { completed: !todo.completed })}
            />
            <span style={{ textDecoration: todo.completed ? 'line-through' : 'none' }}>
              {todo.title}
            </span>
            <button onClick={() => handleDeleteTodo(todo._id)}>Delete</button>
          </li>
        ))}
      </ul>
    </div>
  );
}

export default App;

5. AWS Configuration

5.1 Lambda Function

Create a new Lambda function in the AWS Console.
Upload the ZIP file created in step 3.3.
Set the handler to index.handler.
Add environment variable: MONGODB_URI with your MongoDB connection string.

5.2 API Gateway

Create a new API in API Gateway.
Create resources and methods for /todos and /todos/{id}.
Integrate each method with your Lambda function.
Enable CORS for each method.
Deploy the API to a new stage (e.g., "prod").

5.3 S3 and CloudFront

Create an S3 bucket for hosting the React app.
Create a CloudFront distribution with the S3 bucket as the origin.
Set up Origin Access Control (OAC) for secure access to the S3 bucket.

6. Deployment

6.1 Backend Deployment

Update the Lambda function code:

zip -r function.zip index.js node_modules
aws lambda update-function-code --function-name YourFunctionName --zip-file fileb://function.zip

6.2 Frontend Deployment

Build the React app: npm run build
Upload to S3:

   aws s3 sync build/ s3://your-bucket-name --delete

Invalidate CloudFront cache:

   aws cloudfront create-invalidation --distribution-id YourDistributionID --paths "/*"

7. Troubleshooting

7.1 CORS Issues

If encountering CORS errors:

Ensure CORS is enabled in API Gateway for all methods.
Verify CORS headers in the Lambda function response.
Check that the Access-Control-Allow-Origin header matches your CloudFront domain.

7.2 API Gateway 5XX Errors

Check Lambda function logs in CloudWatch.
Verify that the Lambda function has the correct permissions to access other AWS services.

7.3 MongoDB Connection Issues

Ensure the MONGODB_URI environment variable is set correctly in Lambda.
Verify that the Lambda function has network access to MongoDB Atlas (may require VPC configuration).

8. Maintenance and Updates

8.1 Updating the Backend

Make changes to the Lambda function code.
Redeploy using the steps in section 6.1.

8.2 Updating the Frontend

Make changes to the React application.
Rebuild and redeploy using the steps in section 6.2.

8.3 Monitoring

Use CloudWatch to monitor Lambda function performance and errors.
Set up CloudWatch Alarms for critical metrics.

8.4 Scaling

Adjust Lambda function memory and timeout settings as needed.
Consider implementing caching at the API Gateway level for frequently accessed data.

This documentation provides a comprehensive guide for setting up, deploying, and maintaining your Todo application. Remember to keep your dependencies updated and regularly review AWS best practices for potential improvements to your architecture.

Docker Hands-On Part 06

Randika Madhushan Perera — Wed, 14 Feb 2024 12:51:54 +0000

06. Docker Container Networking

Introduction

The process of containerizing applications represents a significant leap forward in how we deploy and manage software. The Flask framework, popular for its simplicity and flexibility for web application development, is an excellent candidate for Dockerization. This transformative journey from a centralized server setup to a containerized deployment offers numerous advantages, including streamlined deployment processes, improved scalability, and enhanced isolation for development and production environments.

The Journey to Dockerize a Flask Application

The task at hand involves moving a Flask application, aptly named "Notes App," into a Docker container. This application, designed for employees to store notes on current projects, will benefit from the portability and efficiency that containers provide.

Step-by-Step Guide to Containerization

01. Initial Configuration:

Begin by writing the necessary configuration code, including a Dockerfile and .dockerignore file, to build a clean and efficient Docker image for the Notes App.

Step 01: Create Dokcerignore file

$ vim .dockerignore

add below

# Exclude files
.dockeringnore
Dockerfile
.gitignore
Pipfile.lock
migrations/

Create Dockerfile and add below

# version 01
FROM python:3
ENV PYBASE /pybase
ENV PYTHONUSERBASE $PYBASE
ENV PATH $PYABASE/bin:$PATH
#install pipenv
RUN pip install pipenv

WORKDIR /tmp
COPY Pipfile .
RUN pipenv lock
RUN PIP_USER=1 PIP_IGNORE_INSTALLED=1 pipenv install -d --system --ignore-pipfile

COPY . /app/notes
WORKDIR /app/notes
EXPOSE 80
CMD ["flask", "run", "--port=80", "--host=0.0.0.0"]

Building the First Image:

With the Dockerfile in place, build the initial image version of the Notes App. This step includes setting up the database within a container derived from the first image, ensuring that the application's data layer is ready for use.

Step 02: Build the image and container

$ docker build -t notesapp:0.1 .

Step 03: Create and run a container for postgres:12.1-alpine

$ docker pull postgres:12.1-alpine

$ docker run -d --name notesdb postgres:12.1-alpine

Step 04: Create a network and attach it to the bridge network

$ docker network create -d bridge notes

Step 05: List the docker networks

$ docker network ls

NETWORK ID     NAME      DRIVER    SCOPE
df7f5f15c7ab   bridge    bridge    local
7405f25b96f6   host      host      local
ee4c71834f71   none      null      local
14219a8cf079   notes     bridge    local

Step 06: Run a container using the notesapp image and mount the migrations directory

$ docker run --rm -it --network notes -v /home/ec2-user/notes/migrations:/app/notes/migrations notesapp:0.1 bash

Step 07:

# flask db migrate

Running the Application:

Deploy the Notes App container to verify its functionality. During this phase, monitoring the logs for any issues, such as the application running in debug mode, is crucial. Debug mode is not suitable for production due to potential security and performance implications.
Enhancing for Production:

Address the debug mode issue by switching the Flask application to run on a more robust WSGI (Web Server Gateway Interface) server, such as Gunicorn. This change involves updating the environment variables, modifying the pip file to include Gunicorn, and adjusting the Dockerfile to execute Gunicorn instead of Flask directly.

Docker Hands-On Part 05

Randika Madhushan Perera — Wed, 14 Feb 2024 12:51:28 +0000

05. Building Container Images Using Dockerfiles

Introduction

The evolution of container technology seeks to address and simplify the deployment process, making it more efficient and less prone to errors. Manual image creation, while possible, contradicts this objective by introducing unrepeatable processes fraught with potential mistakes. Dockerfiles represent Docker's solution to this challenge, enabling the automation of image creation through a process that is both reproducible and version-controllable.

The Dockerfile Advantage

A Dockerfile is essentially a script composed of various commands, each instructing Docker how to build a specific image. This method allows developers to define their images as code, significantly reducing the manual labor involved in image creation and ensuring that the process can be easily repeated and integrated into continuous integration/continuous deployment (CI/CD) pipelines.

Building an Image for a Web Server

In this tutorial, we'll demonstrate how to utilize a Dockerfile to build a container image designed to serve a company's website. The process will be divided into three key versions, each improving upon the last:

Version 1: Updating the Base Image

Start by creating a Dockerfile that updates the base image using the package manager. While updating the base image is generally not required for every Dockerfile, it serves as an illustrative first step in this tutorial.

Version 2: Removing the Default Apache Welcome Page

The second version of the Dockerfile will modify the image to remove the default Apache welcome page. This step is crucial for preparing the image to host custom web content.

Version 3: Adding Website Data to the Image

The final version of the Dockerfile will incorporate the actual website data into the image. This ensures that the website is served directly from the container upon startup, providing a seamless deployment experience.

Deploying the Final Image

Once the Dockerfile for version 3 is complete, the next step involves building the image and running a container from it. This process will demonstrate the web server in action, serving the company's website from the containerized environment.

Getting Started

To begin, create a Dockerfile in a directory dedicated to this project. The Dockerfile will evolve through each version, culminating in a container image that is ready for deployment. This method not only streamlines the development and deployment process but also aligns with best practices for managing application infrastructure as code.

Step 01: Create a Docker file

$ cd content-widget-factory-inc

$ vim Dokcerfile

add the below codes to it.

#base image httpd
FROM httpd:latest   
# Run the app update
RUN apt update -y && apt upgrade -y && apt autoremove -y && apt clean && rm -rf /var/lib/apt/lists*

Step 02: Build the image

$ docker build -t widgetfactory:0.1 .

Now you can see

$ docker images

REPOSITORY              TAG       IMAGE ID       CREATED          SIZE
widgetfactory           0.1       b1f9112e68d5   22 seconds ago   180MB

Step 03: Set variables to examine the image's size and layers

$ export showLayers='{{ range .RootFS.Layers }}{{ println . }}{{end}}'
$ export showSize='{{ .Size }}'

Step 04: Show the widgetfactory image's size

$ docker inspect -f "$showSize" widgetfactory:0.1

Step 05: We are going to change Dockerfile

Add the below two lines

WORKDIR /usr/local/apache2/htdocs COPY ./web .

#base image httpd
FROM httpd:latest   
# Run the app update
RUN apt update -y && apt upgrade -y && apt autoremove -y && apt clean && rm -rf /var/lib/apt/lists*
WORKDIR /usr/local/apache2/htdocs
COPY ./web .

$ docker build -t widgetfactory:0.2 .

Step 06: Get the widgetfactory:0.2 container

$ docker run --rm -it widgetfactory:0.3 bash

$ ls -l

Step 07: Run the container.

$ docker run -d --name web1 -p 80:80 widgetfactory:0.3

Step 08: Let's test

$ wget localhost

--2024-02-14 10:31:37--  http://localhost/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3276 (3.2K) [text/html]
Saving to: ‘index.html’

index.html                                          100%[=================================================================================================================>]   3.20K  --.-KB/s    in 0s

2024-02-14 10:31:37 (376 MB/s) - ‘index.html’ saved [3276/3276]

Docker Hands-On Part 04

Randika Madhushan Perera — Fri, 09 Feb 2024 17:15:47 +0000

04. Storing Container Data In Docker Volumes

Introduction

In this lesson, you are going to learn how to store your container data in Docker Volumes. You can store config files, static shared files, or any stateful information that you need.

Here we are going to create 3 docker volumes.

Step-by-Step Guide Storing Container Data In Docker Volumes

Step 01: Run the docker volume command to see if having existing docker volumes.

$ docker volume ls

DRIVER    VOLUME NAME

Step 02: Now pull the image for Postgres image for setting up the docker volumes. Here I'm using Postgres 12.1. Postgres needs volumes in the background for the data.

$ docker pull postgres:12.1

$ docker run -d --name db1 postgres:12.1

$ docker run -d --name db2 postgres:12.1

Step 03: Now run the docker volume command to see the volumes

$ docker volume ls

DRIVER    VOLUME NAME
local     1ae4ee6124f0bee423bec887931d9d80aec330a50b5e4edbaf4f151a19c2f23a
local     3dd9a005bb83fc892e2098237fa2bb3ad26407b82bb28096b64a327f26f4eb89

Step 04: It's very difficult to identify the volumes that using above. So we can run the below command.

$ docker inspect db1 -f '{{ json .Mounts }}' | python -m json.tool

[
    {
        "Type": "volume",
        "Name": "3dd9a005bb83fc892e2098237fa2bb3ad26407b82bb28096b64a327f26f4eb89",
        "Source": "/var/lib/docker/volumes/3dd9a005bb83fc892e2098237fa2bb3ad26407b82bb28096b64a327f26f4eb89/_data",
        "Destination": "/var/lib/postgresql/data",
        "Driver": "local",
        "Mode": "",
        "RW": true,
        "Propagation": ""
    }
]

Create the Docker Volumes

Step 01: We can create docker volumes using the create command.

$ docker volume create website

DRIVER    VOLUME NAME
local     1ae4ee6124f0bee423bec887931d9d80aec330a50b5e4edbaf4f151a19c2f23a
local            e534041b91b8b520b79a9f42c4f97fa260cae5e25a11517ce2eed0f92faefad2
local     website

Step 02: Put data in the docker volumes. So we are going to put our web app codes under the website volume

$  sudo cp -r /home/ec2-user/content-widget-factory-inc/web/* /var/lib/docker/volumes/website/_data/

To see the files

$ sudo ls -l /var/lib/docker/volumes/website/_data
total 16
drwxr-xr-x. 2 root root   76 Feb  9 16:51 img
-rw-r--r--. 1 root root 3276 Feb  9 16:51 index.html
-rw-r--r--. 1 root root 2910 Feb  9 16:51 quote.html
-rw-r--r--. 1 root root 2611 Feb  9 16:51 support.html
-rw-r--r--. 1 root root 2645 Feb  9 16:51 widgets.html

Remove Unused Docker Volumes

Run the docker volume prune command.

$ docker volume prune

WARNING! This will remove anonymous local volumes not used by at least one container.
Are you sure you want to continue? [y/N] y
Deleted Volumes:
1ae4ee6124f0bee423bec887931d9d80aec330a50b5e4edbaf4f151a19c2f23a
b61fc8243670de085a3fb120fceec864893ec097838e8be67c05b2e47daaef99
107c5e2f43fb50271133c7df2868716f2f4080e65dcc5cf2367581be6425149e

Total reclaimed space: 0B

Backup and Restore data from Docker Volumes

To do this we need to switch to the root.

# docker ps -a
CONTAINER ID   IMAGE           COMMAND                  CREATED        STATUS        PORTS                                   NAMES
77fc2b4639f4   postgres:12.1   "docker-entrypoint.s…"   4 hours ago    Up 4 hours    5432/tcp                                db2
dfa30582e464   postgres:12.1   "docker-entrypoint.s…"   4 hours ago    Up 4 hours    5432/tcp                                db1
53aace921e70   httpd:latest    "httpd-foreground"       12 hours ago   Up 12 hours   0.0.0.0:8080->80/tcp, :::8080->80/tcp   httpd


# docker volume ls
DRIVER    VOLUME NAME
local     3dd9a005bb83fc892e2098237fa2bb3ad26407b82bb28096b64a327f26f4eb89
local     e534041b91b8b520b79a9f42c4f97fa260cae5e25a11517ce2eed0f92faefad2
local     website

We need to inspect the volume.

# docker volume inspect website
[
    {
        "CreatedAt": "2024-02-09T16:48:21Z",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/website/_data",
        "Name": "website",
        "Options": null,
        "Scope": "local"
    }
]

To get the backup we need to follow the below instructions. We are storing the backup under/tmp/ dir.

# tar czf /tmp/website_$(date +%Y-%m-%d-%H%M).tgz -C /var/lib/docker/volumes/website/_data .

# ls -l /tmp/website_2024-02-09-1708.tgz
-rw-r--r--. 1 root root 11385 Feb  9 17:08 /tmp/website_2024-02-09-1708.tgz

Now we see how we are doing this using non-privileged users.

$ docker run -it --rm -v website:/website -v /tmp:/backup bash tar czf /backup/website_$(date +%Y-%m-%d-%H%M).tgz -C /website .

$ ls -l /tmp/website_*
-rw-r--r--. 1 root root 11385 Feb  9 17:08 /tmp/website_2024-02-09-1708.tgz
-rw-r--r--. 1 root root 11375 Feb  9 17:13 /tmp/website_2024-02-09-1713.tgz

DEV Community: Randika Madhushan Perera

Our Synthetic Monitors Were Timing Out. The Scripts Were Fine.

The complaint

Background: How a private location actually works

The measurement that settled the argument

Root cause #1: Burstable instances and CPU-bound work

Root cause #2: queuing in non-linear

The counter-intuitive fix: Fever servers

The surprise that changed the sizing

The strategy question: add capacity, or fix the schedule?

The Dashboard

The Dashboard UI

A Pet Re-homing platform that costs $29 a month to run

Building a Serverless Life Visualization App with Next.js, AWS Lambda, and DynamoDB Single-Table Design - #01

Introduction

What is this Application?

High Level Architecture

Application FLow

User Onboarding Flow

Deed Logging Flow

Weekly Email Flow

AWS Services Connectivity Flow

Interesting Engineering Decisions

Conclusion

I Found a Critical Security Bug on Foundit.sg — Here's What Happened

Building a Scalable Pet Rehoming Platform with AWS Serverless Architecture

Introduction

Project Overview

AWS Architecture Overview

Architecture Flow

High-Level Architecture Diagram

Deep Dive: AWS Services Implementation

Building a Full-Stack E-Commerce Platform with AWS

A Technical Deep-Dive into Modern Serverless Architecture

Introduction

The Tech Stack that I used

Architecture Overview

The Customer Journey

Payment Flow (The Tricky Part)

Image Upload Architecture

Why I Used Presigned URLs?

Email Notification System

Deployment Pipeline

Deep Dive: AWS Services Powering This Application

AWS Services Architecture

Terraform-CodeGen0: A Terraform Code Generator

What I Built

Demo

Repo

RandiakM / Terraform-CodeGen0

GitHub Copilot 1-Day Build Challenge: This is a Terraform Code generation application for VPC

VPC Terraform Generator

Introduction

How to Use

Installation Steps

Netlify Hosting Steps

Copilot Experience

Conclusion

Configuring AWS WAF, CloudFront, and S3 Bucket for Secure Access

Full Stack Application Hosting in AWS

Todo Application Documentation

Table of Contents

1. Introduction

2. Architecture Overview

3. Backend Setup

3.1 Lambda Function

3.2 Package Configuration

3.3 Deployment Package

4. Frontend Setup

4.1 Create React App

4.2 API Service

4.3 Main App Component

5. AWS Configuration

5.1 Lambda Function

5.2 API Gateway

5.3 S3 and CloudFront

6. Deployment

6.1 Backend Deployment

6.2 Frontend Deployment

7. Troubleshooting