DEV Community: Atul Vishwakarma

Building a Production-Ready 2-Tier Architecture on AWS with Terraform

Atul Vishwakarma — Fri, 17 Apr 2026 09:16:42 +0000

From Theory to Real-World Infrastructure 🚀

As part of my 30 Days of AWS Terraform challenge, Day 22 was a major milestone where I moved beyond individual resources and built a production-style 2-tier architecture using Terraform.

This project was all about combining networking, security, compute, and database layers into a cohesive and secure system — just like real-world applications.

🏗️ The Goal: Build a Secure 2-Tier Architecture

The objective of this project was to design and deploy:

Web Tier (Public Layer) → EC2 instances running a Flask application
Database Tier (Private Layer) → MySQL RDS instance

These two layers communicate securely while maintaining strict isolation from public access.

🔐 Architecture Overview

Web Tier (Public Subnet)

EC2 instance running Flask app
Accessible via Internet Gateway
Uses User Data for automated setup

Database Tier (Private Subnet)

Amazon RDS (MySQL)
No public access
Only accessible from Web Tier via Security Groups

⚙️ Core Infrastructure Components

To support this architecture, I provisioned:

🌐 Networking

Custom VPC
Public & Private subnets
Internet Gateway (for web tier)
NAT Gateway (for private subnet outbound access)

🔒 Security

Security Groups with strict rules
Principle of Least Privilege enforced

🔑 Secrets Management

AWS Secrets Manager for DB credentials
No hardcoded sensitive data

🖥️ Compute

EC2 instance with Flask app
Automated setup via User Data

🗄️ Database

RDS MySQL instance in private subnet

🧩 Modular Terraform Architecture

One of the biggest highlights of Day 22 was applying Terraform Modules in a real project.

Instead of one large configuration file, I structured my code into reusable modules:

Module Structure:

VPC Module → Networking setup
Security Group Module → Access control
RDS Module → Database provisioning
Secrets Module → Credential management
EC2 Module → Web server deployment

🔄 Data Flow Between Modules

Terraform modules don’t communicate directly — so I used:

outputs.tf → to expose values
variables.tf → to pass inputs

Example:

VPC module outputs subnet IDs
Root module passes them to EC2 & RDS modules
Secrets module outputs credentials used by EC2

This pattern ensures:

✔️ Loose coupling
✔️ High reusability
✔️ Clean architecture

🧪 Testing the Application

After deployment, I validated the setup by hitting:

/health → Application health check
/db/info → Database connectivity

Seeing the Flask app successfully connect to the RDS instance was a huge win! 🎉

🔐 Security Best Practices Implemented

This project reinforced critical security principles:

✔️ Private Database

RDS placed in private subnet
No direct internet exposure

✔️ Controlled Access

Only EC2 security group can access DB

✔️ Secrets Management

Credentials stored in AWS Secrets Manager

✔️ Least Privilege

Minimal permissions across components

💡 Key Learnings from Day 22

Modules are essential for scalable Terraform projects
Networking design is critical for security
Secrets should never be hardcoded
Private subnets protect sensitive resources
Testing validates real-world readiness

🧠 Why This Project Matters

This was not just another Terraform lab — it was a real-world architecture simulation.

It taught me how to:

Design secure systems
Structure modular IaC
Connect application & database layers
Apply DevOps best practices

This is the kind of project that bridges the gap between learning and real engineering.

🚀 What’s Next?

With only 8 days remaining, I’m excited to explore:

CI/CD integration
Advanced security practices
Multi-environment deployments

🎯 Final Thoughts

Day 22 was one of the most practical and rewarding days of this challenge.

It reinforced that:

👉 Good infrastructure is not just functional — it’s secure, modular, and scalable.

If you're learning Terraform, I highly recommend building projects like this to truly understand how cloud systems work.

Mastering Cloud Policy & Governance with Terraform

Atul Vishwakarma — Fri, 17 Apr 2026 05:56:46 +0000

Building Secure & Compliant Cloud Infrastructure with IaC 🚀

As part of my 30 Days of AWS Terraform challenge, Day 21 marked a major shift in perspective — from simply provisioning infrastructure to governing and securing it at scale.

Today’s focus was on AWS Policy and Governance using Terraform, and it was one of the most practical and impactful lessons so far.

Because in real-world cloud environments, success isn’t just about deploying resources — it’s about ensuring they are:

Secure 🔐
Compliant 📋
Auditable 🔍
Consistent ⚙️

Why Policy & Governance Matter

When infrastructure grows across teams, regions, and environments, manual management becomes:

❌ Error-prone
❌ Inconsistent
❌ Difficult to audit
❌ A major security risk

This is where Infrastructure as Code (IaC) combined with governance tools becomes critical.

👉 Terraform allows us to codify guardrails, ensuring that every deployment automatically follows best practices.

Core Concepts I Explored

Today’s lab focused on three essential pillars of cloud governance:

1. Preventive Controls with IAM Policies 🔐

IAM acts as the first line of defense.

Instead of reacting to issues, we can prevent them entirely by defining strict policies.

What I Implemented:

Denied S3 bucket deletion without MFA
Enforced encrypted uploads (HTTPS only)
Restricted unsafe operations based on conditions

Why It Matters:

✔️ Stops misconfigurations before they happen
✔️ Enforces least privilege
✔️ Protects critical infrastructure

2. Continuous Monitoring with AWS Config 📊

IAM prevents bad actions — but what about changes over time?

That’s where AWS Config comes in.

What I Built:

Enabled AWS Config recorder
Configured managed rules
Monitored compliance continuously

Example Checks:

Unencrypted EBS volumes
Missing resource tags
Non-compliant S3 buckets

Why It Matters:

✔️ Detects drift in infrastructure
✔️ Ensures continuous compliance
✔️ Provides audit visibility

3. Secure Logging & Audit Trails 🪵

Governance is incomplete without proper logging.

What I Implemented:

Centralized S3 bucket for logs
Enabled versioning
Enforced encryption
Restricted public access

Why It Matters:

✔️ Enables audits & investigations
✔️ Preserves historical data
✔️ Strengthens security posture

Hands-On Implementation Highlights ⚙️

Today’s project involved building governance controls using Terraform:

✔️ AWS Config Setup

Config recorder automation
Managed rule definitions

✔️ Tagging Enforcement

Standardized tags across all resources
Improved cost tracking & ownership

✔️ IAM Guardrails

Attached policies to roles
Controlled access behavior

This made the entire infrastructure:

👉 Self-governing
👉 Consistent
👉 Production-ready

The Real Challenge: IAM Policy Evaluation 🧠

One of the most valuable learnings today was understanding how IAM policies are evaluated.

It’s not just about writing policies — it’s about understanding:

Explicit Deny vs Allow
Policy precedence
Conditional logic behavior

Key Insight:

👉 An explicit deny always overrides an allow.

This concept is critical when designing secure systems.

Why This Matters in Real Organizations 🏢

In enterprise environments, governance ensures:

✔️ Compliance with regulations
✔️ Security at scale
✔️ Standardized deployments
✔️ Reduced human error

Without governance, cloud infrastructure quickly becomes chaotic.

With Terraform + AWS Config + IAM → you get automated compliance.

Key Takeaways from Day 21 💡

Terraform can enforce governance, not just provisioning
IAM policies act as preventive controls
AWS Config enables continuous monitoring
Logging is critical for auditing
Understanding policy evaluation is essential

What’s Next? 🔥

As I move forward in this journey, I’m excited to explore:

Policy as Code (OPA, Sentinel)
Advanced compliance automation
Security frameworks integration

Final Thoughts

Day 21 was a turning point.

It changed my mindset from:

➡️ “How do I deploy infrastructure?”
➡️ To “How do I secure and govern infrastructure at scale?”

That’s the real difference between writing Terraform and engineering cloud systems.

If you’re learning Terraform, don’t skip governance — it’s what makes your infrastructure production-ready.

Mastering Custom Terraform Modules for Scalable AWS Infrastructure

Atul Vishwakarma — Thu, 16 Apr 2026 10:48:58 +0000

Building Production-Ready Infrastructure with Reusable Terraform Modules 🚀

As part of my 30 Days of AWS Terraform challenge, Day 20 was a major milestone in my Infrastructure as Code journey.

Today’s focus was on one of the most important Terraform concepts for real-world DevOps: Custom Terraform Modules.

Until now, most of my learning revolved around creating and managing AWS resources directly. But Day 20 introduced the concept that truly separates beginner Terraform projects from production-grade cloud systems:

👉 Modularity and reusability.

This project gave me hands-on experience building an Amazon EKS cluster using a modular Terraform architecture — and it completely changed how I think about writing infrastructure code.

Why Terraform Modules Matter

When starting with Terraform, it’s common to place everything in a single main.tf file:

VPC
Subnets
Security groups
IAM roles
EKS cluster

This works for simple labs.

But in real-world environments, this quickly becomes:

❌ Hard to manage
❌ Difficult to debug
❌ Impossible to scale
❌ Error-prone for teams

Terraform modules solve this problem.

Benefits of Modules:

✅ Code reusability
✅ Better readability
✅ Easier collaboration
✅ Standardized deployments
✅ Simplified maintenance

Modules help transform Terraform from a scripting tool into a proper infrastructure framework.

Types of Terraform Modules I Explored

Today, I explored the three main types of modules:

1. Public Modules

Modules published in the Terraform Registry by providers like HashiCorp.

Example:

AWS VPC module

2. Partner Modules

Modules maintained jointly by HashiCorp and cloud vendors / partners.

Useful for:

Enterprise integrations
Verified architectures

3. Custom Modules (Main Focus Today)

Custom modules are built by your own team to:

Match internal standards
Enforce security controls
Improve reusability

This was the core focus of Day 20.

Project Goal: Build an EKS Cluster with Custom Modules 🎯

For today’s hands-on project, I built an Amazon EKS (Elastic Kubernetes Service) cluster using modular Terraform code.

Instead of writing one large configuration file, I split the infrastructure into reusable child modules:

Module Breakdown:

VPC Module → Networking, subnets, routing
IAM Module → Roles, policies, permissions
EKS Module → Cluster and worker node setup

This approach felt much closer to how real production systems are designed.

Understanding Root Module vs Child Modules 🧠

One of the biggest learnings today was understanding how Terraform structures module relationships.

Root Module

The root module acts as the main entry point.

Responsibilities:

Calling child modules
Passing variables
Managing overall orchestration

Child Modules

Child modules are reusable building blocks.

Each child module:

Handles one responsibility
Has its own variables
Exposes outputs

Why This Matters

This separation improves:

Team ownership
Maintainability
Debugging

This was a huge mindset shift for me.

Passing Data Between Modules 🔄

Today’s most important concept was learning how to pass information between modules.

Key Challenge:

Terraform child modules cannot directly communicate with each other.

So if:

VPC module creates subnets
IAM module creates roles

The EKS module still needs:

Subnet IDs
IAM Role ARN

Solution:

Using:

variables.tf
outputs.tf

Example Flow:

VPC module outputs subnet IDs
Root module receives outputs
Root passes them into EKS module

Why This Matters

This teaches clean architecture principles:

👉 Modules should stay independent, but data should flow intentionally.

This was the biggest technical takeaway from Day 20.

Encapsulation = Cleaner Infrastructure 🧩

Another major lesson was the power of encapsulation.

Instead of exposing all resource complexity in the root module:

I wrapped:

IAM roles
Node groups
Networking logic

Inside modules.

Result:

My root configuration became:

✔️ Cleaner
✔️ Easier to understand
✔️ Faster to extend

This makes future projects much easier to maintain.

Why This Matters in Real Organizations 🏢

Custom modules are essential for enterprise DevOps because they help enforce standards.

Examples:

Secure default VPC rules
Required resource tags
Approved IAM policies
Logging standards

Benefits for Teams:

✔️ Consistency across environments
✔️ Better compliance
✔️ Faster deployments
✔️ Reduced human error

Today made me realize that writing reusable Terraform is as important as writing correct Terraform.

Key Learnings from Day 20 💡

Today’s biggest takeaways:

✔️ Terraform modules improve scalability
✔️ Root / child architecture is powerful
✔️ Variables & outputs are the backbone of modularity
✔️ Encapsulation makes code production-ready
✔️ Reusability saves time and reduces errors

This felt like a major step toward thinking like a professional cloud engineer.

Advice for Beginners

If you’re learning Terraform:

Start small.

Try creating modules for:

VPC
EC2 instance
Security groups

Before moving to advanced services like EKS.

The sooner you understand modules, the easier Terraform becomes.

What’s Next? 🔥

Looking ahead, I’m excited to explore:

Module versioning
Remote module sources
CI/CD integration
Multi-environment deployments

Still 10 days to go — excited to keep building.

Final Thoughts

Day 20 was one of the most impactful milestones in this Terraform challenge.

It taught me that great Infrastructure as Code is not just about provisioning resources — it’s about building systems that are:

Reusable
Scalable
Maintainable

Custom Terraform modules are a game-changer for anyone serious about cloud engineering.

If you’re on your own Terraform journey, I highly recommend spending time mastering modules — they unlock the real power of IaC.

How are you using Terraform modules in your projects? I’d love to hear your best practices.

Building a Serverless Image Processing Pipeline with Terraform

Atul Vishwakarma — Wed, 15 Apr 2026 10:02:43 +0000

Automating Image Workflows with AWS Lambda, S3, and Terraform 🚀

As part of my 30 Days of AWS Terraform challenge, Day 18 was one of the most exciting and practical projects so far.

Today, I moved beyond basic infrastructure provisioning and built a fully automated serverless image processing pipeline using:

AWS S3
AWS Lambda
IAM Roles & Policies
Terraform

This project was a major step toward understanding how real-world event-driven cloud systems are designed and automated.

The Project Goal 🎯

The goal was simple but powerful:

👉 Whenever an image is uploaded to a source S3 bucket, AWS should automatically process it and store multiple optimized versions in a destination bucket.

Output Variants Included:

Compressed versions
Different file formats
Thumbnail image

This type of architecture is highly relevant for:

Media platforms
E-commerce websites
User profile image optimization
Content delivery systems

Why Serverless Architecture Matters

Traditional image processing systems often require:

Dedicated servers
Manual scaling
Ongoing maintenance

That creates:

Higher cost
Operational complexity
Slower deployments

Serverless changes everything.

Benefits of Serverless:

✅ Event-driven execution
✅ No server management
✅ Auto scaling
✅ Pay only for usage
✅ Faster deployment cycles

This project showed me exactly why serverless is such a powerful cloud pattern.

Architecture Overview 🏗️

The architecture for today’s project looked like this:

Step-by-Step Flow:

User uploads image to Source S3 bucket
S3 event notification triggers Lambda
Lambda processes image using Pillow library
Lambda generates multiple variants
Processed images are uploaded to Destination bucket

This entire workflow runs automatically without any manual intervention.

Terraform Resources Used ⚙️

This project was a great demonstration of how Terraform can automate complex serverless stacks.

Key Resources Provisioned:

1. S3 Buckets 📦

Terraform created:

Source bucket (incoming uploads)
Destination bucket (processed images)

Bucket setup included:

Event notification configuration
Permissions for Lambda access

2. AWS Lambda Function 🧠

The core logic was implemented inside a Lambda function.

Responsibilities:

Read uploaded image
Process and compress variants
Generate thumbnail
Upload outputs

Why Lambda?

Lambda is ideal because:

No server maintenance
Automatic scaling
Fast event response

3. IAM Roles & Least Privilege Security 🔐

A key learning today was implementing secure IAM access.

Terraform provisioned:

Lambda execution role
Scoped IAM policies

Permissions included:

s3:GetObject from source bucket
s3:PutObject to destination bucket
CloudWatch logs access

Why This Matters

Security in automation is critical.

This reinforced:

👉 Always grant only the permissions required.

4. CloudWatch Logging 📊

Terraform also provisioned:

Log groups
Monitoring support

This helped with:

Debugging failures
Monitoring execution
Troubleshooting events

CloudWatch visibility was extremely useful during testing.

Biggest Challenge: Dependency Management 🐳

One of the most valuable lessons today came from troubleshooting Python dependencies.

The Problem:

Libraries like:

Pillow

Often fail when packaged locally because:

Local machine OS differs from AWS Lambda runtime
Binary dependencies break

The Solution:

I used Docker to build Lambda Layers.

Why Docker Helped:

Matched AWS Linux environment
Prevented runtime compatibility issues
Ensured consistent builds

Key Lesson:

👉 “It works on my machine” is not enough in cloud engineering.

Environment consistency matters.

This was one of the most important practical takeaways so far.

Key Learnings from Day 18 💡

Today’s project taught me:

✔️ How event-driven serverless systems work
✔️ How S3 triggers Lambda in real workflows
✔️ Why least privilege IAM matters
✔️ How Terraform simplifies complex deployments
✔️ Why dependency packaging is critical

This was one of the clearest examples yet of Terraform enabling repeatable, scalable cloud systems.

Why Terraform Made This Easier

Without Terraform, setting this up manually would involve:

Creating buckets
Uploading Lambda code
Configuring IAM roles
Setting triggers
Setting logs

This would be:

❌ Time-consuming
❌ Error-prone
❌ Hard to reproduce

Terraform made the entire setup:

✅ Repeatable
✅ Version-controlled
✅ Easy to destroy / recreate

This is exactly why IaC is such a game changer.

What’s Next? 🔥

Future improvements I’d like to explore:

Adding image watermarking
Using Step Functions for complex workflows
Integrating API Gateway for direct uploads
Setting lifecycle rules on processed images
Adding alerts for failed Lambda runs

Excited to keep building.

Final Thoughts

Day 18 was one of the most practical and rewarding milestones in this challenge so far.

This project showed me how Terraform can be used not just to provision resources, but to automate intelligent business workflows.

Serverless architectures are efficient, scalable, and increasingly relevant in modern cloud systems — and building one from scratch was a huge confidence boost.

If you’re learning Terraform, I highly recommend exploring serverless projects like this. They teach infrastructure, automation, debugging, and cloud design all at once.

Have you built event-driven serverless workflows with Terraform? I’d love to hear your experiences.

Mastering Blue-Green Deployments with Terraform & AWS Elastic Beanstalk

Atul Vishwakarma — Tue, 14 Apr 2026 10:33:51 +0000

Achieving Zero-Downtime Deployments with Infrastructure as Code 🚀

As part of my 30 Days of AWS Terraform challenge, Day 17 was one of the most exciting milestones so far: implementing a Blue-Green Deployment strategy using Terraform and AWS Elastic Beanstalk.

This project took my Terraform journey beyond provisioning infrastructure into the world of release engineering, deployment safety, and production reliability.

Until now, I had focused on building resources and automating cloud workflows. But today’s challenge taught me something even more valuable:

👉 How to deploy application updates without downtime and with instant rollback capability.

Why Blue-Green Deployment Matters

In production systems, every deployment introduces risk.

Common challenges with traditional deployments:

Temporary downtime during updates
Risk of failed releases affecting users
Difficult rollbacks
Limited testing in production-like conditions

This is where Blue-Green Deployment becomes a game-changer.

What is Blue-Green Deployment?

Blue-Green is a deployment strategy where you maintain two identical production environments:

🔵 Blue Environment

Current live production version
Serving real users

🟢 Green Environment

New version deployment
Used for testing and validation

Once Green is validated:

👉 Traffic is switched from Blue to Green instantly.

If something breaks:

👉 Rollback is as simple as switching traffic back.

Project Goal 🎯

The goal for today’s project was to:

Deploy two versions of an application (v1 and v2)
Host them in separate Elastic Beanstalk environments
Manage infrastructure using Terraform
Perform a zero-downtime DNS cutover

This project was a perfect mix of:

Terraform provisioning
AWS application hosting
Deployment strategy
Risk mitigation

Architecture Overview 🏗️

Key AWS Components Used

1. Amazon S3 for Application Packaging 📦

I first packaged:

Application version v1.0
Application version v2.0

As ZIP archives.

These were uploaded to an S3 bucket, which served as the source for Elastic Beanstalk deployments.

Why This Matters

S3 acts as:

Central artifact storage
Version-controlled release source
Secure deployment package repository

2. AWS Elastic Beanstalk Environments 🌱

Terraform was used to provision:

Elastic Beanstalk Application
Blue environment (live)
Green environment (candidate release)

Elastic Beanstalk simplified:

EC2 provisioning
Load balancing
Auto scaling
Health monitoring

This allowed me to focus more on deployment logic than server management.

3. IAM Roles & Instance Profiles 🔐

To make the environments work securely, Terraform also provisioned:

EC2 IAM roles
Instance profiles
Required permissions

This ensured:

S3 access for artifacts
Elastic Beanstalk environment operations

A good reminder that security is always part of automation.

Terraform in Action ⚙️

This project helped me apply Terraform in a real deployment workflow.

Key Terraform Concepts Used:

✔️ Resource provisioning for Elastic Beanstalk
✔️ S3 object uploads for artifacts
✔️ IAM role automation
✔️ Environment lifecycle management
✔️ Infrastructure consistency between Blue & Green

Biggest Benefit

Because both environments were managed as code:

Configuration stayed consistent
Drift was minimized
Deployment became repeatable

This is exactly why IaC matters.

The Best Part: Zero-Downtime DNS Swap 🔄

The highlight of today’s project was performing the actual traffic cutover.

Once Green was deployed and tested:

I validated the new version
Confirmed health checks
Swapped the CNAME / DNS routing

Result:

✅ Users experienced zero downtime
✅ Traffic shifted instantly
✅ No service interruption

This was one of the most satisfying hands-on moments in this challenge so far.

Key Learnings from Day 17 💡

1. Downtime Can Be Avoided

Production deployments don’t have to impact users.

Blue-Green strategies provide:

Safer releases
Better customer experience
Lower operational risk

2. Rollbacks Should Be Simple

One of the strongest lessons:

👉 Good systems are not just deployable — they are recoverable.

If Green fails:

Switch traffic back to Blue
Restore production quickly

This kind of safety net is essential in real systems.

3. Testing Matters

Green environments allow:

Smoke tests
Health checks
Validation before release

This reduces bad deployments significantly.

Real-World Extensions 🔥

To make this production-grade, future improvements could include:

Route 53 weighted routing
Automated DNS cutover via Terraform / CLI
CI/CD integration with GitHub Actions / Jenkins
Canary deployments
Monitoring with CloudWatch

These are areas I’m excited to explore next.

Final Thoughts

Day 17 was a major mindset shift.

This project showed me that DevOps is not just about creating infrastructure — it’s about designing systems that are:

Reliable
Recoverable
Safe to change

Blue-Green deployment is one of the clearest examples of how cloud engineering directly improves user experience.

If you’re learning Terraform or AWS, I highly recommend trying a project like this. It teaches both technical depth and deployment discipline.

Scaling IAM User Management with Terraform

Atul Vishwakarma — Tue, 14 Apr 2026 09:16:23 +0000

Automating User Onboarding and Access Control at Scale 🚀

As part of my 30 Days of AWS Terraform challenge, Day 16 shifted from infrastructure provisioning into something just as critical in real-world DevOps: identity and access management (IAM) at scale.

Today’s hands-on project focused on automating AWS IAM user creation, login setup, tagging, and group assignment using Terraform.

This was a powerful reminder that Infrastructure as Code is not only about deploying servers and networks — it’s also about standardizing how people securely interact with cloud systems.

The Real Problem: Manual IAM Doesn't Scale

In many organizations, onboarding users manually through the AWS Console leads to:

Human errors
Inconsistent naming
Delayed access provisioning
Poor auditability

As teams grow, this process becomes inefficient and risky.

Terraform solves this by making IAM onboarding:

✅ Repeatable
✅ Scalable
✅ Auditable
✅ Secure-by-design

Project Goal 🎯

The goal for today’s project was simple:

👉 Automatically provision multiple IAM users from a CSV file and manage access dynamically.

This included:

Bulk user creation
Naming standardization
Metadata tagging
Login profile setup
Group assignment based on role/department

Architecture & Workflow ⚙️

1. CSV Data Parsing with `csvdecode()` 📄

The first step was handling structured user input.

I created a CSV file containing:

First name
Last name
Department
Role

Using Terraform’s built-in csvdecode() function, I converted the CSV into a list of maps that Terraform could iterate over.

Why This Matters

This approach makes onboarding easy:

Just update the CSV
Terraform handles the rest

Perfect for HR / DevOps collaboration.

2. Bulk User Provisioning with `for_each` 🔁

Instead of manually creating IAM users one by one, I used:

for_each
Dynamic resource blocks

This allowed Terraform to create multiple users in a single apply.

Benefits:

✔️ No duplicate code
✔️ Faster onboarding
✔️ Easier scaling

This is exactly where Terraform shines.

3. Dynamic Naming & Standardized Tags 🏷️

To enforce consistency, I used Terraform functions like:

lower()
substr()

Example:

Michael Scott → mscott

I also added tags such as:

Department
Role
Owner

Why Tags Matter

Tags improve:

Cost visibility
Auditing
Access control

This was a great exercise in combining automation with governance.

4. Secure Login Profiles 🔐

To make the users immediately usable, I provisioned:

aws_iam_user_login_profile

With:

password_reset_required = true

This ensures users must reset passwords on first login.

Security Lesson

While outputs were used for learning/demo purposes, this reinforced an important point:

👉 Sensitive credentials should never be exposed carelessly.

In production, this should be paired with:

AWS Secrets Manager
HashiCorp Vault
Secure password delivery workflows

5. Dynamic Group Assignment Based on Role 🧠

One of the most exciting parts of today’s project was automating IAM group membership.

Instead of manually assigning users to groups:

I used:

for_each
Conditional expressions
Tag-based logic

Example:

Users tagged as manager → Manager group
Finance users → Finance access group

Why This Matters

This makes onboarding smarter by:

✔️ Reducing manual work
✔️ Enforcing policy automatically
✔️ Improving consistency

This felt like true Infrastructure as Code in action.

Key Takeaways 💡

Day 16 taught me that DevOps is not just about infrastructure resources — it’s also about people, permissions, and secure workflows.

Today’s biggest lessons:

✔️ IAM automation improves speed and consistency
✔️ Terraform can simplify complex onboarding workflows
✔️ Security must always be part of automation design
✔️ Dynamic logic makes systems scalable

What’s Next? 🔥

To make this production-ready, my next steps would include:

Applying least-privilege IAM policies
Enabling MFA for all users
Integrating with AWS SSO / IAM Identity Center
Adding secure secret distribution

Excited to keep building and improving.

Final Thoughts

Day 16 was one of the most practical projects so far because it connected Terraform directly to real-world operational workflows.

Automating IAM user management showed me how Infrastructure as Code can improve not just systems, but also team productivity and security posture.

If you’re learning Terraform, don’t stop at servers and networks — explore IAM automation too. It’s one of the most valuable skills in cloud engineering.

Mastering AWS VPC Peering with Terraform

Atul Vishwakarma — Tue, 14 Apr 2026 07:17:23 +0000

Connecting Networks Across Regions with Infrastructure as Code 🚀

As part of my 30 Days of AWS Terraform challenge, Day 15 marked a major milestone: reaching the halfway point and stepping into one of the most practical areas of cloud engineering — AWS networking with VPC Peering.

Until now, much of my Terraform journey focused on provisioning resources and learning Terraform concepts. But today’s project shifted from deploying isolated services to solving a real-world networking challenge: securely connecting two private AWS VPCs across different regions using Terraform.

This project was a strong reminder that Infrastructure as Code is not just about automation — it’s about understanding architecture, networking principles, and system design.

Why VPC Peering Matters

In real-world cloud environments, applications and services often live in different VPCs:

Separate environments (Dev / Staging / Prod)
Multi-account setups
Regional redundancy architectures
Shared services and private APIs

To allow these isolated networks to communicate securely over private IP addresses, AWS provides VPC Peering.

What VPC Peering Does

VPC Peering creates a private connection between two VPCs, enabling resources in each network to talk to each other as if they were on the same local network.

Benefits include:

✅ Private communication (no internet exposure)
✅ Low latency
✅ Better security
✅ Simplified architecture

Project Overview 🏗️

For this mini-project, I provisioned:

VPC 1 in US East (N. Virginia)
VPC 2 in US West (Oregon)
EC2 instances in both VPCs
VPC Peering connection between them
Route table updates for bidirectional traffic

The goal was simple:

👉 Allow private communication between instances in both regions.

And yes — successfully testing cross-region private connectivity felt amazing. 🎯

Key Learnings from Day 15

1. Multi-Region Terraform with Provider Aliases 🌍

One of the biggest learnings today was managing resources across multiple AWS regions in a single Terraform project.

Using provider aliases, I configured:

aws.primary → US East 1
aws.secondary → US West 2

This made it possible to:

Deploy resources cleanly in separate regions
Keep code structured and maintainable
Avoid duplicated configurations

Why This Matters

In enterprise cloud environments, multi-region deployments are common for:

Disaster recovery
High availability
Geo-distributed applications

Understanding provider aliases is essential for scalable Terraform projects.

2. CIDR Planning is Critical 🧠

One of the first rules of VPC Peering:

❌ CIDR ranges must NOT overlap

If both VPCs share the same IP range:

Route tables conflict
Traffic breaks
Connectivity fails

This reinforced an important lesson:

👉 Good networking starts with good IP planning.

Even with Terraform, automation cannot fix poor architecture.

3. VPC Peering is NOT Transitive ⚠️

A major cloud networking concept I learned:

If:

VPC A ↔ VPC B
VPC B ↔ VPC C

That does NOT mean:

VPC A can talk to VPC C

This is called non-transitive peering.

Why This Matters

For larger enterprise setups:

VPC Peering can become hard to manage
Mesh architectures get complex quickly

This is where services like:

AWS Transit Gateway
Hub-and-spoke networking

Become important.

Understanding this limitation was one of today’s biggest takeaways.

4. Route Tables Make the Magic Happen 🛣️

Creating the peering connection is only half the setup.

The real work is in routing.

For both VPCs, I had to:

Update route tables
Add peer VPC CIDR blocks
Point traffic to the peering connection ID

Without correct routes:

Terraform may succeed… but traffic will still fail.

This taught me a practical lesson:

👉 Cloud networking is as much about traffic flow as it is about resource creation.

Hands-On Testing & Validation 🔍

The most satisfying part of today’s project was testing connectivity.

After setup, I verified:

Ping between instances
Curl requests to private IPs
Cross-region communication over private network

Seeing an EC2 instance in Oregon reach a web server in Virginia privately was an exciting milestone.

This was one of the clearest demonstrations so far of how powerful Terraform + AWS can be together.

Key Takeaways 💡

Day 15 taught me more than just Terraform syntax.

It reinforced:

✔️ Cloud networking fundamentals matter
✔️ Architecture decisions impact scalability
✔️ Debugging builds confidence
✔️ Terraform is powerful when paired with strong design principles

What’s Next? 🔥

Reaching the halfway point feels great, but there’s still a lot ahead:

Security hardening
Modules & reusable architecture
Advanced state management
Production deployment workflows

Excited for the second half of this challenge.

Final Thoughts

Day 15 was one of the most practical and rewarding days of my Terraform journey so far.

If you’re learning Terraform, don’t stop at basic resource creation. Start exploring networking, debugging routes, and understanding how systems communicate.

That’s where real cloud engineering begins.

Hosting a Static Website on AWS with S3 and CloudFront using Terraform

Atul Vishwakarma — Sat, 11 Apr 2026 17:16:39 +0000

From Terraform Basics to Real-World Deployment 🚀

As part of my 30 Days of AWS Terraform challenge, Day 14 marked a major milestone in my learning journey: deploying a secure, scalable static website on AWS using Terraform.

This project brought together everything I’ve learned so far—Terraform resources, variables, loops, functions, data sources, and debugging—to build something practical and production-relevant.

Instead of just creating isolated AWS resources, I built a complete architecture for static website hosting using Amazon S3 + CloudFront, fully managed through Infrastructure as Code.

Why Not Just Host Directly on S3?

Hosting a static website directly from an S3 bucket is simple, but it has limitations:

Public bucket exposure increases security risks
No built-in global caching
Slower delivery for international users
Harder to scale professionally

To solve this, I implemented a better architecture:

✅ Private S3 bucket for secure storage
✅ CloudFront CDN for global delivery
✅ Origin Access Control (OAC) for secure access
✅ Terraform automation for repeatability

Architecture Overview 🏗️

Key AWS Components Used

1. Private S3 Bucket

I created an S3 bucket to store all website assets:

HTML files
CSS files
JavaScript files
Images

Important: Public access was fully blocked to ensure security.

2. Origin Access Control (OAC)

Instead of using the older OAI method, I configured:

Origin Access Control (OAC)

This securely allows CloudFront to fetch content from the private bucket.

3. Bucket Policy

Using Terraform, I wrote a bucket policy that:

Allows only CloudFront to access bucket objects
Prevents direct public access

This follows AWS security best practices.

4. CloudFront Distribution

CloudFront was used as the CDN layer to:

Cache files globally
Improve load times
Reduce latency
Add edge security

This made the project feel much closer to real-world production hosting.

Terraform Concepts Applied ⚙️

This project helped me apply several advanced Terraform concepts:

🔹 for_each + fileset

One of the coolest parts was automating file uploads.

Instead of uploading files manually, I used:

fileset() to scan all files in the project folder
for_each to upload them dynamically

This made the setup scalable and clean.

🔹 MIME Type Handling

Different file types need proper content types.

Using Terraform functions like lookup(), I dynamically mapped:

.html → text/html
.css → text/css
.js → application/javascript

This ensured browsers rendered the site correctly.

🔹 Modern AWS Provider Practices

While working on the project, I faced deprecation issues.

For example:

Older resource: aws_s3_bucket_object
Updated resource: aws_s3_object

Debugging these issues taught me an important lesson:

👉 Always refer to the latest Terraform provider documentation.

Key Challenges & Learnings 💡

This project was not just about deployment—it was about troubleshooting and problem-solving.

Some valuable lessons:

Writing correct S3 bucket policies is critical
Small syntax issues can break deployments
AWS provider updates matter a lot
Debugging Terraform errors improves confidence

One big takeaway:

Errors are not setbacks—they are part of the learning process.

Every issue I solved gave me a better understanding of AWS and Terraform internals.

What’s Next? 🔥

This setup is already solid, but to make it production-ready, the next logical steps are:

Future Enhancements

Add a custom domain using Route 53
Configure HTTPS with AWS ACM
Build CI/CD pipelines for automatic deployment
Add cache invalidation workflows

These are areas I’m excited to explore next.

Final Thoughts

Day 14 was one of the most rewarding milestones in this challenge so far.

This project showed me how Infrastructure as Code can be used not just to create resources, but to design secure, scalable, and professional cloud systems.

If you’re learning Terraform, I highly recommend trying a project like this. It ties together so many foundational concepts and gives you hands-on experience with real-world architecture.

I’d love to hear your thoughts:
Have you hosted static websites using Terraform? Any best practices or lessons you’ve learned?

Setting Up the ELK Stack on AWS EC2 for Log Monitoring

Atul Vishwakarma — Sat, 30 Aug 2025 12:26:46 +0000

Welcome to this comprehensive guide on setting up the ELK Stack (Elasticsearch, Logstash, Kibana) on AWS EC2 instances. If you're managing applications in the cloud, especially Java-based ones, efficient log monitoring is crucial for debugging, performance analysis, and security. The ELK Stack, combined with Filebeat, provides a powerful, open-source solution for collecting, processing, visualizing, and analyzing logs in real-time.

In this blog post, we'll walk through a step-by-step setup using Ubuntu-based EC2 instances. This tutorial is based on a practical example involving a Java application, but the principles apply broadly. We'll cover everything from infrastructure provisioning to creating dashboards in Kibana. By the end, you'll have a fully functional log monitoring system.

Prerequisites:

An AWS account with access to EC2.
Basic knowledge of SSH, Linux commands, and AWS networking (e.g., security groups).
Three EC2 instances (t3.micro or similar): One for the ELK server (Elasticsearch, Logstash, Kibana), one for the client machine (hosting the app and Filebeat), and an optional web server for testing.
Ensure ports like 9200 (Elasticsearch), 5044 (Logstash), and 5601 (Kibana) are open in your security groups.

1. Overview of the ELK Stack

The ELK Stack is a collection of tools from Elastic:

Elasticsearch: A search and analytics engine that stores and indexes your logs.
Logstash: A data processing pipeline that ingests, transforms, and sends logs to Elasticsearch.
Kibana: A web interface for visualizing and querying your logs.
Filebeat: A lightweight shipper that forwards logs from your application servers to Logstash.

This setup allows you to centralize logs from distributed systems like EC2, parse them into structured data, and gain insights through dashboards.

2. Infrastructure Setup

Launch three Ubuntu EC2 instances:

ELK Server: Hosts the core ELK components. Assign a public IP for Kibana access.
Client Machine: Runs your Java app and Filebeat. Use the ELK server's private IP for communication.
Web Server (Optional): For simulating additional log sources.

Connect via SSH to each instance as the ubuntu user. Update packages on all machines:

sudo apt update

3. Step-by-Step Installation

Step 1: Install and Configure Elasticsearch (on ELK Server)

Elasticsearch is the backbone for storing logs.

Install Java (required for Elasticsearch and Logstash):

sudo apt update && sudo apt install openjdk-17-jre-headless -y

Install Elasticsearch:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list

sudo apt update

sudo apt install elasticsearch -y

Configure Elasticsearch: Edit /etc/elasticsearch/elasticsearch.yml:

sudo vi /etc/elasticsearch/elasticsearch.yml

Add or modify:

network.host: 0.0.0.0
cluster.name: my-cluster
node.name: node-1
discovery.type: single-node

Start and enable the service:

sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch
sudo systemctl status elasticsearch

Verify it's running:
```
curl -X GET "http://localhost:9200/"
```

Step 2: Install and Configure Logstash (on ELK Server)

Logstash processes incoming logs.

Install Logstash:
```
sudo apt install logstash -y
```

Configure Logstash: Edit /etc/logstash/conf.d/logstash.conf:

sudo vi /etc/logstash/conf.d/logstash.conf

Add:

input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{LOGLEVEL:log_level} %{GREEDYDATA:log_message}" }
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "logs-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}

This config accepts logs from Beats (like Filebeat), parses them using Grok, and stores them in Elasticsearch.

Start and enable the service:

sudo systemctl start logstash
sudo systemctl enable logstash
sudo systemctl status logstash

Allow traffic on port 5044:
```
sudo ufw allow 5044/tcp
```

Step 3: Install and Configure Kibana (on ELK Server)

Kibana provides the UI for log analysis.

Install Kibana:
```
sudo apt install kibana -y
```

Configure Kibana: Edit /etc/kibana/kibana.yml:

sudo vi /etc/kibana/kibana.yml

Modify:

server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]

Start and enable the service:

sudo systemctl start kibana
sudo systemctl enable kibana
sudo systemctl status kibana

Allow traffic on port 5601:
```
sudo ufw allow 5601/tcp
```
Access the Kibana dashboard: Open a browser and navigate to http://<ELK_Server_Public_IP>:5601.

Step 4: Install and Configure Filebeat (on Client Machine)

Filebeat ships logs from your app to Logstash.

Install Filebeat:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list

sudo apt update 

sudo apt install filebeat -y

Configure Filebeat: Edit /etc/filebeat/filebeat.yml:

sudo vi /etc/filebeat/filebeat.yml

Modify:

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /home/ubuntu/JavaApp/target/app.log

output.logstash:
  hosts: ["<ELK_Server_Private_IP>:5044"]

Start and enable the service:

sudo systemctl start filebeat
sudo systemctl enable filebeat
sudo systemctl status filebeat

Verify Filebeat:
```
sudo filebeat test output
```

Step 5: Deploy a Java Application and Generate Logs (on Client Machine)

To test, we'll use a sample Java app.

Install Java if needed:

sudo apt install openjdk-17-jre-headless -y

Download and run a sample app:

wget https://repo1.maven.org/maven2/org/springframework/boot/spring-boot-sample-simple/1.4.2.RELEASE/spring-boot-sample-simple-1.4.2.RELEASE.jar -O app.jar
nohup java -jar app.jar > /home/ubuntu/JavaApp/target/app.log 2>&1 &

Generate test logs:

echo "Test log entry $(date)" >> /home/ubuntu/JavaApp/target/app.log

Step 6: View and Analyze Logs in Kibana

In Kibana, go to Discover.
Select the logs-* index pattern.
Search for logs from your app, e.g., log.file.path: "/home/ubuntu/Boardgame/target/app.log".
View parsed fields like log_timestamp, log_level, and log_message.

Create visualizations:

Pie Chart: For log level distribution.
Line Chart: For logs over time.
Data Table: For a structured log view.

Build a dashboard:

Go to Dashboard > Create Dashboard.
Add your visualizations.
Save as "Java Application Log Monitoring".

Conclusion

Congratulations! You've set up the ELK Stack on AWS EC2, integrated Filebeat for log shipping, and created a dashboard for real-time monitoring. This setup scales well—add more Filebeat instances for multi-app environments. For production, consider security enhancements like SSL, authentication, and backups.

If you encounter issues, check service logs with journalctl or Elastic's documentation. Happy logging!

☕ Support My Work

If you found this guide helpful, consider buying me a coffee to support my work!

Common Data Loss Scenarios & Solutions in Prisma Schema Changes

Atul Vishwakarma — Tue, 07 Jan 2025 16:18:35 +0000

Common Data Loss Scenarios & Solutions in Prisma Schema Changes

When evolving a database schema using Prisma, care must be taken to ensure data integrity and avoid loss. Below, we explore common data loss scenarios and provide step-by-step solutions to address them effectively.

1. Enum to String Conversion

Issue:

Converting an enum column to a string type can result in data inconsistencies.

Example:

Before: payedBy Payment_By @default(NONE)
After: payedBy String?

Solution:

Add a new column:

ALTER TABLE "Orders" ADD COLUMN "payedBy_new" TEXT;

Copy data:

UPDATE "Orders" SET "payedBy_new" = "payedBy"::text;

Drop old column:

ALTER TABLE "Orders" DROP COLUMN "payedBy";

Rename the new column:

ALTER TABLE "Orders" RENAME COLUMN "payedBy_new" TO "payedBy";

2. Changing Column Type (e.g., Int to Decimal)

Issue:

Directly altering a column type can cause data loss.

Example:

Before: amount Int
After: amount Decimal

Solution:

Add a new column:

ALTER TABLE "TableName" ADD COLUMN "amount_new" DECIMAL;

Copy data:

UPDATE "TableName" SET "amount_new" = "amount"::DECIMAL;

Drop old column:

ALTER TABLE "TableName" DROP COLUMN "amount";

Rename the new column:

ALTER TABLE "TableName" RENAME COLUMN "amount_new" TO "amount";

3. Making a Nullable Column Non-Nullable

Issue:

Enforcing a non-null constraint without handling existing NULL values can break queries.

Example:

Before: email String?
After: email String

Solution:

Populate NULL values with placeholders:

UPDATE "TableName" SET "email" = 'placeholder@email.com' WHERE "email" IS NULL;

Alter the column:

ALTER TABLE "TableName" ALTER COLUMN "email" SET NOT NULL;

4. Changing JSON Structure

Issue:

Modifying the structure of a JSON column can lead to data mismatches.

Example:

Before: {oldField: "value"}
After: {newField: "value"}

Solution:

Add a temporary column:

ALTER TABLE "TableName" ADD COLUMN "metadata_new" JSONB;

Transform the data:

UPDATE "TableName" SET "metadata_new" = jsonb_build_object(
   'newField',
   CASE WHEN metadata->>'oldField' IS NOT NULL THEN metadata->>'oldField' ELSE NULL END
) WHERE metadata IS NOT NULL;

Drop the old column:

ALTER TABLE "TableName" DROP COLUMN "metadata";

Rename the new column:

ALTER TABLE "TableName" RENAME COLUMN "metadata_new" TO "metadata";

5. Array Type Changes

Issue:

Converting an array type (e.g., String[] to Int[]) can cause errors if the data types don’t align.

Example:

Before: tags String[]
After: tags Int[]

Solution:

Add a new column:

ALTER TABLE "TableName" ADD COLUMN "tags_new" INTEGER[];

Convert data:

UPDATE "TableName" SET "tags_new" = ARRAY(
   SELECT NULLIF(value, '')::INTEGER
   FROM unnest("tags") AS value
   WHERE value ~ '^[0-9]+$'
);

Drop the old column:

ALTER TABLE "TableName" DROP COLUMN "tags";

Rename the new column:

ALTER TABLE "TableName" RENAME COLUMN "tags_new" TO "tags";

6. Adding/Removing Unique Constraints

Issue:

Adding a unique constraint without addressing duplicate values can cause migration failures.

Example:

Before: email String
After: email String @unique

Solution:

Identify duplicates:

SELECT email, COUNT(*) FROM "TableName" GROUP BY email HAVING COUNT(*) > 1;

Handle duplicates:

WITH duplicates AS (
   SELECT email, ROW_NUMBER() OVER (PARTITION BY email ORDER BY id) AS row_num
   FROM "TableName"
)
UPDATE "TableName" t
SET email = t.email || '_' || d.row_num
FROM duplicates d
WHERE t.email = d.email AND d.row_num > 1;

Add the constraint:

ALTER TABLE "TableName" ADD CONSTRAINT email_unique UNIQUE (email);

Best Practices for Safe Schema Changes

Backup First:

pg_dump -U username -d database_name > backup.sql

Test in Development:
- Create a development database.
- Restore the backup.
- Test migrations.
Use Transactions:
```
BEGIN;
-- Migration steps
COMMIT;
```
Implement Rollback Plans:
- Save original data in a backup table.
- Rollback if necessary.

Conclusion

Schema changes in Prisma require meticulous planning and execution. By following these structured solutions and best practices, you can ensure safe migrations while maintaining data integrity.

If you found this article helpful, consider buying me a coffee to support my work!

Creating and Connecting to a Virtual Machine in a Custom Virtual Network using Azure Portal

Atul Vishwakarma — Wed, 24 Apr 2024 13:54:44 +0000

Step 01: Sign in to the Azure Portal

Go to Microsoft Azure Portal.
Sign in to your Azure account.

Step 02: Create a Virtual Network

In the Azure Portal, click on "Create a resource" in the upper-left corner.
Search for "Virtual network" and select it from the results.
Click on "Create" to start creating a virtual network.
Fill in the required information such as name, address space, and subnet details.
Once configured, click on "Review + create" and then click on "Create" to create the virtual network.

Step 03: Create a Virtual Machine

In the Azure Portal, click on "Create a resource" again.
Search for "Virtual machine" and select it from the results.
Click on "Create" to start creating a virtual machine.
Fill in the required information like resource group, virtual machine name, region, image, size, etc.
Under the "Networking" section, select the virtual network you created in step 2 and configure other networking settings as needed.
Configure other settings like disks, management, tags, etc., according to your requirements.
Once configured, click on "Review + create" and then click on "Create" to create the virtual machine.

Step 04: Connect to the Virtual Machine using RDP

Once the virtual machine is created, navigate to the virtual machine resource in the Azure Portal.
In the virtual machine's overview page, click on "Connect" at the top.
Download the RDP file and open it.
Enter the username and password for the virtual machine (these are the credentials you specified during VM creation).
Click "Connect" to initiate the RDP session.

Step 05: Verify Connection

After connecting via RDP, you should be able to access the virtual machine's desktop and interact with it as if you were physically sitting in front of it.

Install Hadoop on Ubuntu

Atul Vishwakarma — Sat, 04 Nov 2023 18:58:55 +0000

Apache Hadoop is a collection of utilities that allows you to manage the processing of large datasets across clusters of computers.

Also, it is tolerant to cluster failures. If one of your clusters crashes, Hadoop can be used to recover data from other nodes.

And in this guide, I will walk you through the installation process.

How to install Hadoop on Ubuntu

To install Hadoop, you will have to go through various steps, which include:

Installing Java and configuring environment variables
Creating user and configuring SSH
Installation and configuration of Hadoop

So let's start with the first step:

Step 1: Installing Java on Ubuntu

To install java on Ubuntu, all you have to do is execute the following command:

sudo apt install default-jdk default-jre -y

To verify the installation, check the java version on your system:

java -version

Step 2: Create a user for Hadoop and configure SSH

First, create a new user named hadoop:

sudo adduser hadoop

To enable superuser privileges to the new user, add it to the sudo group:

sudo usermod -aG sudo hadoop

Once done, switch to the user hadoop:

sudo su - hadoop

Next, install the OpenSSH server and client:

sudo apt install openssh-server openssh-client -y

Now, use the following command to generate private and public keys:

ssh-keygen -t rsa

Here, it will ask you:

Where to save the key (hit enter to save it inside your home directory)
Create passphrase for keys (leave blank for no passphrase)

Now, add the public key to authorized_keys:

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Use the chmod command to change the file permissions of authorized_keys:

sudo chmod 640 ~/.ssh/authorized_keys

Finally, verify the SSH configuration:

ssh localhost

If you have not configured the password, all you have to do is type yes and hit enter if you added a passphrase for the keys, it will ask you to enter here:

Step 3: Download and install Apache Hadoop on Ubuntu

If you have created a user for Hadoop, first, log in as the hadoop user:

sudo su - hadoop

Now, visit the download page for Apache Hadoop and copy the link for the most recent stable release.

While writing, its 3.3.6 so I will be using the wget command to download this release:

wget https://downloads.apache.org/hadoop/common/stable/hadoop-3.3.6.tar.gz

Once you are done with the download, extract the file using the following command:

tar -xvzf hadoop-3.3.6.tar.gz

Next, move the extracted file to the /usr/local/hadoop using the following command:

sudo mv hadoop-3.3.6 /usr/local/hadoop

Now, create a directory using mkdir command to store logs:

sudo mkdir /usr/local/hadoop/logs

Finally, change the ownership of the /usr/local/hadoop to the user hadoop:

sudo chown -R hadoop:hadoop /usr/local/hadoop

Step 4: Configure Hadoop on Ubuntu

Here, I will walk you through the configuration of the Hadoop environment variable.

First, open the .bashrc file using the following command:

sudo nano ~/.bashrc

Jump to the end of the line in the nano text editor by pressing Alt + / and paste the following lines:

export HADOOP_HOME=/usr/local/hadoop

export HADOOP_INSTALL=$HADOOP_HOME

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export YARN_HOME=$HADOOP_HOME

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"

Save changes and exit from the nano text editor.

To enable the changes, source the .bashrc file:

source ~/.bashrc

Step 5: Configure java environment variables

To use Hadoop, you are required to enable its core functions which include YARN, HDFS, MapReduce, and Hadoop-related project settings.

To do that, you will have to define java environment variables in hadoop-env.sh file.

Edit the hadoop-env.sh file

First, open the hadoop-env.sh file:

sudo nano $HADOOP_HOME/etc/hadoop/hadoop-env.sh

Press Alt + / to jump to the end of the file and paste the following lines in the file to add the path of the Java:

export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64

export HADOOP_CLASSPATH+=" $HADOOP_HOME/lib/*.jar"

Save changes and exit from the text editor.

Next, change your current working directory to /usr/local/hadoop/lib:

cd /usr/local/hadoop/lib

Here, download the javax activation file:

sudo wget https://jcenter.bintray.com/javax/activation/javax.activation-api/1.2.0/javax.activation-api-1.2.0.jar

Once done, check the Hadoop version in Ubuntu:

hadoop version

Next, you will have to edit the core-site.xml file to specify the URL for the name node.

Edit the `core-site.xml` file

First, open the core-site.xml file using the following command:

sudo nano $HADOOP_HOME/etc/hadoop/core-site.xml

And add the following lines in between <configuration> </configuration>:

<property>

     <name>fs.default.name</name>

          <value>hdfs://0.0.0.0:9000</value>

          <description>The default file system URI</description>

</property>

Save the changes and exit from the text editor.

Next, create a directory to store node metadata using the following command:

sudo mkdir -p /home/hadoop/hdfs/{namenode,datanode}

And change the ownership of the created directory to the hadoop user:

sudo chown -R hadoop:hadoop /home/hadoop/hdfs

Edit the `hdfs-site.xml` configuration file

By configuring the hdfs-site.xml file, you will define the location for storing node metadata, fs-image file.

So first open the configuration file:

sudo nano $HADOOP_HOME/etc/hadoop/hdfs-site.xml

And paste the following line in between <configuration> ... </configuration>:

<property>

     <name>dfs.replication</name>

     <value>3</value>

</property>

<property>

     <name>dfs.name.dir</name>

     <value>file:///home/hadoop/hdfs/namenode</value>

</property>

<property>

     <name>dfs.data.dir</name>

     <value>file:///home/hadoop/hdfs/datanode</value>

</property>

Save changes and exit from the hdfs-site.xml file.

Edit the `mapred-site.xml` file

By editing the mapred-site.xml file, you can define the MapReduce values.

To do that, first, open the configuration file using the following command:

sudo nano $HADOOP_HOME/etc/hadoop/mapred-site.xml

And paste the following line in between <configuration> ... </configuration>:

<property>

      <name>mapreduce.framework.name</name>

      <value>yarn</value>

   </property>

<property>
        <name>yarn.app.mapreduce.am.env</name>
        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>
    <property>
        <name>mapreduce.map.env</name>
        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>
    <property>
        <name>mapreduce.reduce.env</name>
        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>

Save and exit from the nano text editor.

Edit the `yarn-site.xml` file

This is the last configuration file that needs to be edited to use the Hadoop service.

The purpose of editing this file is to define the YARN settings.

First, open the configuration file:

sudo nano $HADOOP_HOME/etc/hadoop/yarn-site.xml

Paste the following in between <configuration> ... </configuration>:

<property>

      <name>yarn.nodemanager.aux-services</name>

      <value>mapreduce_shuffle</value>

   </property>

Save changes and exit from the config file.

Finally, use the following command to validate the Hadoop configuration and to format the HDFS NameNode:

hdfs namenode -format

Step 6: Start the Hadoop cluster

To start the Hadoop cluster, you will have to start the previously configured nodes.

So let's start with starting the NameNode and DataNode:

start-dfs.sh

Next, start the node manager and resource manager:

start-yarn.sh

To verify whether the services are running as intended, use the following command:

jps

Step 7: Access the Hadoop web interface

To access the Hadoop web interface, you will have to know your IP and append the port no 9870 in your address bar:

http://server-IP:9870

My IP is 10.0.2.15 so I will be entering the following:

http://10.0.2.15:98705

And there you have it!

Wrapping Up

While the tutorial was lengthy for sure, the major part was just copying and pasting lines to your terminal.

I hope you will find this guide helpful.

But if you encounter any errors while executing shown guide, let me know in the comments and I will try my best to come up with the best solution possible.