DEV Community: piyushsachdeva

Deploy Your Own 24/7 AI Agent on AWS EC2 with Docker & Tailscale (The Secure Way)

piyushsachdeva — Wed, 11 Feb 2026 04:39:29 +0000

No, you don't need a Mac Mini or fancy hardware setup to run your own AI agent 24/7. All you need is a cloud virtual machine or VPS (Virtual Private Server).

I know what you're thinking: "What about security?" And you should be thinking about it. That's exactly why I'm going to walk you through the step-by-step process to securely set up an AI agent running as a Docker container.

Imagine having your own private AI assistant running 24/7 on a cloud server you control-accessible from anywhere, yet completely shielded from the public internet. That's exactly what we're building today.

In this guide, we'll deploy OpenClaw (also known as Moltbot/Clawdbot) on an AWS EC2 instance using Docker containers. But we're not just going to spin up a container and call it a day. We'll harden our server, lock down SSH access, and use Tailscale to create a secure private network that only you can access.

What You'll Have by the End

A hardened Ubuntu server with non-standard SSH configuration
Docker running OpenClaw in an isolated container
Secure private access via Tailscale (no public ports exposed)
A fully functional AI assistant accessible from your browser Let's get started.

If you prefer video tutorial, you can watch the below end-to-end video (else, keep reading the blog)

Prerequisites

Before we dive in, make sure you have the following ready:

Cloud Infrastructure

AWS Account with EC2 access
A fresh Ubuntu 24.04 LTS instance (Recommended specs: 2 vCPU, 4GB RAM, 15GB storage)
Key Pair: Your .pem file from AWS (e.g., moltbot.pem) saved on your local computer
Current Access: Ability to SSH into the instance as the default ubuntu user

API Keys

Anthropic API Key (for Claude Sonnet/Opus - recommended for best performance)
OpenAI API Key (Optional, as a backup)

Networking

Tailscale Account: Free tier is sufficient (this is how we'll securely access our bot)

Pro Tip: If you're new to AWS, providers like Hetzner, DigitalOcean, or Vultr also work perfectly with this guide.

Phase 1: VM Hardening (Do This FIRST)

Before we install anything fun, we need to secure our server. Think of this as locking the doors and windows before you move your valuables inside.

Step 1: Create a New User

First, SSH into your server as the default ubuntu user:

ssh -i "moltbot.pem" ubuntu@<YOUR-EC2-PUBLIC-IP>

Once connected, let's update the system and create a dedicated user for OpenClaw:

# Update system
sudo apt update && sudo apt upgrade -y

# Create a new user (replace 'openclaw' if you want a different name)
sudo adduser openclaw
# (Enter a strong password when prompted)

# Grant sudo rights
sudo usermod -aG sudo openclaw

Why a new user? Running applications as a dedicated user follows the principle of least privilege. If something goes wrong, the blast radius is contained.

Step 2: Set Up SSH Keys for the New User

Here's where many tutorials fail you. The AWS key (.pem file) only works for the ubuntu user by default. We need to copy the authorized keys to our new user, or we'll be locked out.

Run these commands while still logged in as ubuntu:

# 1. Create the SSH directory for the new user
sudo mkdir -p /home/openclaw/.ssh

# 2. Copy the authorized keys from 'ubuntu' to 'openclaw'
sudo cp /home/ubuntu/.ssh/authorized_keys /home/openclaw/.ssh/

# 3. Fix permissions (CRITICAL: If this is wrong, login will fail)
sudo chown -R openclaw:openclaw /home/openclaw/.ssh
sudo chmod 700 /home/openclaw/.ssh
sudo chmod 600 /home/openclaw/.ssh/authorized_keys

The Permission Trinity: Directory at 700, keys file at 600, owned by the user. Get any of these wrong, and SSH will silently reject your login.

Step 3: Configure AWS Security Group (External Firewall)

Time to switch to your web browser. We need to tell AWS to allow traffic on our new SSH port.

Go to EC2 Dashboard → Security Groups
Select your instance's security group → Edit inbound rules
Add Rule:
- Type: Custom TCP
- Port: 2222
- Source: 0.0.0.0/0 (or "My IP" for extra security)
DO NOT delete the rule for Port 22 yet - this is your safety net
Save rules

Step 4: Configure UFW (Internal Firewall)

Back in your terminal, let's set up the host-level firewall:

# Set default policies
sudo ufw default deny incoming
sudo ufw default allow outgoing

# Allow the CURRENT port (Safety Net)
sudo ufw allow 22/tcp

# Allow the FUTURE port
sudo ufw allow 2222/tcp

# Enable the firewall
sudo ufw enable

When prompted, type y to confirm. You now have two layers of firewall protection.

Step 5: The Golden Test

This is the moment of truth. Before we disable password authentication or change ports, we need to prove that our new user can log in with the SSH key.

Keep your current terminal window open (this is your lifeline).

Open a NEW terminal window on your local computer and run:

ssh -i "moltbot.pem" openclaw@<YOUR-IP-ADDRESS>

Result	Action
Success: You log in without being prompted for a password	Proceed to Step 6
Failure: "Permission denied"	Go back and re-check Step 2. Do not proceed.

Step 6: Lock Down SSH Configuration

Now that we've verified key-based login works, it's time to harden SSH. This is where Ubuntu 24.04 throws a curveball that catches many admins off guard.

Edit the SSH config file:

sudo vi /etc/ssh/sshd_config

Make the following changes:

Port 2222
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes

Save and close the file (:wq in vi).

Disable Systemd Socket Activation (CRITICAL)

Here's the gotcha: Ubuntu 24.04 uses "socket activation" which holds Port 22 open regardless of what you put in sshd_config. You must disable this:

# Stop the socket listener
sudo systemctl stop ssh.socket
sudo systemctl disable ssh.socket

# Restart the SSH service to apply your new config
sudo systemctl restart ssh

Verify the Port Change:

sudo ss -tulpn | grep ssh

Output	Status
`0.0.0.0:2222`	Success!
`0.0.0.0:22`	Something went wrong. Do NOT disconnect.

Step 7: Final Verification & Cleanup

Open a NEW terminal window and test the new port:

ssh -p 2222 -i "moltbot.pem" openclaw@<YOUR-IP-ADDRESS>

Only if that works, go back to your server terminal and lock down Port 22:

sudo ufw delete allow 22/tcp

Then go to AWS Security Groups and delete the inbound rule for Port 22.

Important: From this point forward, you can no longer use "EC2 Instance Connect" (the browser console). You must always use the SSH command above.

Phase 2: Environment Setup

With our server hardened, it's time to install the software stack. We'll set up Docker to run OpenClaw in isolation and Tailscale to create our private access tunnel.

1. Install Docker & Docker Compose

Docker ensures OpenClaw's dependencies don't interfere with your host OS. Here's the official installation method:

# Install Docker dependencies
sudo apt install apt-transport-https ca-certificates curl software-properties-common -y

# Add Docker GPG key & Repository
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin -y

# Allow 'openclaw' user to run Docker without sudo
sudo usermod -aG docker ${USER}

Now log out and log back in for the group change to take effect:

exit

Then reconnect via SSH:

ssh -p 2222 -i "moltbot.pem" openclaw@<YOUR-IP-ADDRESS>

Verify Docker works without sudo:

docker run hello-world

2. Install Tailscale (The Secure Access Layer)

Here's where the magic happens. Instead of exposing OpenClaw's web port (18789) to the public internet, we'll use Tailscale to create a private mesh network. Only devices on your Tailnet can access the bot.

On your EC2 server:

curl -fsSL https://tailscale.com/install.sh | sh

After installation, authenticate with your Tailscale account:

sudo tailscale up

Follow the link provided to authorize your server.

On your local machine:

Install Tailscale from tailscale.com/download and sign in with the same account.

Once both devices are on your Tailnet, they can communicate securely without any public ports.

Phase 3: OpenClaw Installation

The infrastructure is ready. Let's deploy the bot.

1. Clone the Repository

git clone https://github.com/openclaw/openclaw.git
cd openclaw

2. Clean Up Old Configurations

If you've experimented with OpenClaw before, nuke any old config folders to prevent conflicts:

sudo rm -rf ~/.openclaw

3. Pre-Create the Configuration Folder

This step prevents the dreaded "Permission Denied" errors. We create the folder manually and set permissions so both you AND Docker can write to it:

# Create the folder structure
mkdir -p /home/openclaw/.openclaw/workspace

# Give full read/write access (fixes the "Permission Denied" error)
sudo chmod -R 777 /home/openclaw/.openclaw
sudo chown -R openclaw:openclaw ~/openclaw
sudo chmod -R 775 ~/openclaw

4. Run the Setup Script

Now we're ready for the main event:

cd ~/openclaw
./docker-setup.sh

A series of prompts will appear. Select EXACTLY these options to avoid crashes:

Prompt	Selection
Onboarding mode	Manual
Setup	Local gateway (this machine)
Workspace directory	(Press Enter to accept default)
Model/auth Provider	Anthropic
Anthropic API Key	(Enter your key)
Gateway port	18789
Gateway bind	Tailnet
Gateway auth	Token
Tailscale exposure	Off
Gateway Token	(Create a secure token)
Configure chat channel	(Whatsapp/Telegram - your choice)
Configure Skills	No
Hooks	Skip for Now

Critical Warning: Do NOT select "Serve" for Tailscale exposure. This bypasses complex proxy logic that causes crashes. Select "Off" - Tailscale already handles secure access.

Wait for completion. The container will start, but DO NOT LOGIN YET.

5. The "Insecure Auth" Injection

Here's a quirk with OpenClaw: At this stage, the bot is running but will block your browser because you're connecting over HTTP (not HTTPS). Since Tailscale already encrypts our traffic end-to-end, we can safely enable HTTP authentication.

First, install jq inside the container:

docker compose exec -u root openclaw-gateway bash -c "apt update && apt install -y jq"

Then inject the setting:

docker compose exec -T openclaw-gateway bash -c '
jq ".gateway.controlUi.allowInsecureAuth = true" \
/home/node/.openclaw/openclaw.json > /home/node/.openclaw/tmp.json && \
mv /home/node/.openclaw/tmp.json /home/node/.openclaw/openclaw.json'

Restart the bot to apply changes:

docker compose restart

Why is this safe? We're not exposing any ports to the public internet. Traffic flows exclusively through your encrypted Tailscale tunnel. The "insecure" part only refers to HTTP vs HTTPS - and Tailscale already provides encryption.

Phase 4: Login & Verify

The moment of truth. Your bot is running, security is configured, and Tailscale is connecting your devices. Let's access the interface.

1. Check Container Status

Make sure the containers are healthy and staying up:

docker compose ps

Watch it for a few seconds. If the status stays "Up" for more than 10 seconds, you're good.

2. Get Your Tailscale IP

tailscale ip -4

This returns something like 100.x.x.x - your private Tailscale IP address.

3. Access the Web Interface

On your local machine (which must also be connected to Tailscale), open your browser and navigate to:

http://<YOUR_TAILSCALE_IP>:18789

For example: http://100.64.0.1:18789

Enter the Gateway Token you created during setup and click Connect.

Congratulations! You now have a private AI assistant running on your own infrastructure, accessible only through your secure Tailscale network.

Phase 5: Best Practices & Maintenance

Your bot is running, but a few housekeeping practices will keep it healthy and your wallet safe.

Set API Spend Limits

Agentic AI can sometimes get stuck in loops and burn through API credits rapidly. Protect yourself:

Go to your Anthropic Console
Navigate to Settings → Limits
Set a hard monthly budget (e.g., $20)

Do the same for OpenAI if you configured it as a backup.

Enable Sandbox Mode

In the OpenClaw configuration, ensure Sandbox Mode is enabled if available. This restricts the bot's ability to execute unrestricted shell commands on the host system.

Back Up Your Bot's Memory

OpenClaw stores its "long-term memory" in Markdown files. If you lose this directory, the bot forgets everything it knows about you.

Create regular backups:

# Simple backup command
tar -czvf memory_backup_$(date +%F).tar.gz ./openclaw/memory

Consider automating this with a cron job for daily backups.

Monitor Logs

When things go sideways (and they sometimes do), logs are your best friend:

docker compose logs -f

The -f flag follows the log output in real-time. Press Ctrl+C to exit.

Conclusion

You've successfully deployed OpenClaw on a hardened EC2 instance with:

Non-standard SSH port (2222) with key-only authentication
Dual-layer firewall protection (AWS Security Groups + UFW)
Docker containerization for clean isolation
Tailscale private networking for secure, zero-trust access
No public ports exposed for the application itself

Your AI assistant is now running on infrastructure you control, accessible only to devices on your private Tailnet. No VPN configurations, no exposed ports, no attack surface for the public internet to probe.

Happy building!

DevOps and Cloud Engineering Roadmap 2025 🚀

piyushsachdeva — Fri, 20 Dec 2024 15:46:03 +0000

🚀 DevOps and Cloud Engineering Roadmap 2025

📋 Table of Contents

About
Learning Path
Projects
Resources

🎯 About

This repository contains a structured learning path for beginners entering the DevOps and Cloud Engineering field. The curriculum is designed for students, recent graduates, and career switchers, with an estimated completion time of 6-12 months of dedicated learning.

📚 Learning Path

Phase 1: Foundation (2-3 months)

Linux and Shell Scripting

# Topics:
├── Linux Administration
|   ├── Linux Installation
│   ├── Command Line Basics
│   ├── File System Management
│   ├── Process Management
│   ├── User Administration
│   └── Package Management
└── Bash Scripting
    ├── Variables and Data Types
    ├── Control Structures
    ├── Functions
    └── Automation Scripts

Programming Foundation

Choose one:

Python (Recommended for beginners)
- Basic syntax and data structures
- Object Oriented Programming
- File handling and automation
- API integration
- Libraries: requests, pandas, numpy
Golang
- Systems programming
- Concurrent programming
- Cloud-native development

Version Control

# Git & GitHub
├── Git Setup
├── Basic Commands
├── Branching Strategies
├── Git Flow
├── Pull Requests
├── Code Reviews
└── Collaborative Development

Phase 2: Core IT Skills (1 month)

Network Fundamentals

OSI Model - In-depth
TCP/IP Protocol Suite
Internet Protocols ( HTTP, HTTPS, SSH, SMTP, ICMP, etc)
How to analyze network packets using tools such as Wireshark
DDOS attack and how to mitigate that
CIDR and Subnetting Video Link
DNS and DHCP Video Link
Vertical V/s Horizontal Scaling
SSL/TLS, symmetric vs asymmetric encryption and how it works : Video Link
Reverse Proxy V/s Forward Proxy V/s load balancer
Troubleshooting application slowness, latency, unavailability etc
RAID
NAS vs SAN storage
SQL vs NoSQL
DB Sharding, Caching etc

Phase 3: Cloud Computing (2-3 months)

Choose your cloud provider:

Which Cloud Provider to choose? Confused? Watch the video below and you should be good Video link

Cloud provider	Certification type	Certification	Tutorials
AWS	Foundation Associate	• AWS Certified Cloud Practitioner • AWS Solutions Architect Associate
Azure	Foundation Associate	• AZ-900 Azure Fundamentals • AZ-104 Azure Administrator	Playlist
GCP	Foundation Associate	• Google Cloud Digital Leader • Associate Cloud Engineer	Playlist

Note: Got more questions about the Cloud/DevOps Certifications? Watch the video below:

Video Link

Phase 4: DevOps Tools (3-4 months)

1. Containerization (Docker)

# Docker Fundamentals
├── Container Basics
├── Dockerfile Creation, Multi-Stage Builds
├── Image Management
├── Docker Compose
├── Container Networking
├── Docker Best Practices
└── Volume Management

Checkout the free YouTube playlist 👉 here

2. Container Orchestration (Kubernetes)

Checkout the free YouTube playlist 👉 here

3. CI/CD Pipeline

Jenkins ( Checkout the free Youtube video here)
Modern CI/CD Tools(at least one from below)
- GitHub Actions
- Azure DevOps (Checkout the free Youtube Playlist here)
- GitLab CI/CD (Checkout the free Youtube video here)

4. Infrastructure as Code

# Terraform
├── HCL Syntax
├── Resource Management
├── State Management
├── Modules
└── Cloud Provider Integration

Checkout the free Terraform playlist here

5. Monitoring & Logging

Prometheus & Grafana
ELK Stack
Fluentd

Phase 5: 🛠️ Projects

Check out the Repo here for #10weeksofcloudops projects

📚 Learning Resources

Official Documentation

YouTube Channels

Tech Tutorials With Piyush

Practice Platforms

Made with ❤️ by [Piyush Sachdeva]
Last updated: December 2024

Top 3 Cloud and DevOps Projects To Supercharge Your Resume

piyushsachdeva — Wed, 08 May 2024 05:24:54 +0000

Introduction

In this blog, we'll be discussing three amazing DevOps and Cloud projects that you should have in your resume, and when I say have in your resume, it doesn't mean you add in your resume. It means that you should Implement those projects by yourself, get the learning out of it, create some useful artifacts like blogs or GitHub repositories, and then you should add them to your resume so that the recruiters/hiring managers will know that these projects you have implemented by yourself.

If you prefer watching a 10-minute video with all the details, you can check the below link; else, you can continue with the blog:

What can you expect from this blog?

In addition to sharing project ideas and steps, I will provide a complete end-to-end solution through blog posts, GitHub repositories, and YouTube videos. Anyone interested in implementing the solution can learn how to do so using these resources as guidance.

Project 1: Static website hosting and CICD

Difficulty Level: Beginner
Focus Area: Cloud Storage, CDN and CICD

Host a static website on the cloud of your choice, either AWS, Azure, or GCP, and Implement cicd on that.

Technologies Covered 📚 ( Use either of the below options)

AWS: Amazon S3, CloudFront, and Route 53
Azure: Azure Storage, Azure CDN, and DNS Management
Google Cloud: Cloud Storage, Load Balancing, and Content Delivery Networks (CDN)

Get Started 🚀

Review the project requirements. ✔️
Dive into AWS, Azure, or GCP documentation to familiarize yourself with the services mentioned. 📖
Start building your architecture diagram. 🏗️
Document your progress and implementation steps in a blog or GitHub Readme. 📝
Challenges Faced: Discuss any challenges you encountered and how you overcame them. 🤔
Key Takeaways: Share what you learned from this project. 🧐
Resources: List any helpful resources or references you used. 📚

Reference resources: ✅

💡 If you are an absolute beginner to the cloud and CICD, get yourself. Familiarize with the concepts, you can refer to the below documentation and study material:

For AWS
https://docs.aws.amazon.com/AmazonS3/latest/userguide/HostingWebsiteOnS3Setup.html
https://docs.aws.amazon.com/AmazonS3/latest/userguide/website-hosting-cloudfront-walkthrough.html

For Azure

https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website-how-to?tabs=azure-portal
https://learn.microsoft.com/en-us/azure/storage/blobs/storage-custom-domain-name?tabs=azure-portal

For GCP
https://cloud.google.com/storage/docs/hosting-static-website
https://www.cloudskillsboost.google/focuses/1251?parent=catalog

If you are a visual learner, feel free to check out the video solution

Architecture Diagram for AWS by Ankit

Blog containing step-by-step instructions for AWS

Blog by Ankit

Architecture Diagram for Azure by Nishant

Blog By Nishant containing the step-by-step instructions for Azure

Blog by Nishant

Project #2: 3-tier architecture

Difficulty Level: Intermediate
Focus Area: Cloud Infrastructure, Networking, 3-tier application

For this project, your objective is to design a 3-tier architecture in a cloud platform, such as AWS, Azure, or GCP, with a focus on the following key considerations:

High Availability: Ensure your architecture is highly available, capable of withstanding failures, and can provide uninterrupted service.
Scalability: Design your architecture to be highly scalable, allowing for easy and efficient resource scaling as your application's demands increase.
Fault Tolerance: Implement fault-tolerant mechanisms to minimize downtime and service interruptions during failures.
Custom VPC/VNET: Consider using a custom Virtual Private Cloud (VPC) for AWS or GCP, a Virtual Network (VNET) for Azure, or a similar network customization instead of relying on the default configurations.
Security: Prioritize security by adhering to best practices for Identity and Access Management (IAM) and implementing robust security measures.

Your design should consider these considerations, resulting in a well-structured, efficient, and secure 3-tier architecture.
You can use your chosen cloud platform to implement this architecture effectively.

Reference resources: ✅

💡 To ensure you're ready to take on this challenge, it's essential to have a solid understanding of networking concepts. Check out the following resources for guidance:

👉 AWS: Click Here

👉 Azure: Click Here

👉 GCP: Click Here

🚀IP address calculation/CIDR and Subnet Masks: Click here

If you are a video person, feel free to check out the below video for end to end solution:👇

Architecture Diagram for AWS

Image Source

Architecture Diagram for Azure

Image Source

Detailed workshop for AWS with step-by-step instructions:

Workshop link

Detailed blog for Azure for step-by-step instructions

Blog by Nishant Singh

Project 3: Implement a 2-tier architecture in AWS, Azure, or GCP using Terraform 🚀

Difficulty Level: Intermediate
Focus Area: Iac using Terraform, Custom modules, Infrastructure.

This project aims to leverage the best practices of infrastructure as code (IaC) to create a reusable and shareable infrastructure setup. Our focus is on promoting modularity, flexibility, and maintainability.

Key Guidelines

1. Leverage Custom Modules

Build custom modules to break your infrastructure code into reusable and shareable components. This approach organizes your code and allows other team members to incorporate and adapt the components easily for their specific needs.

2. Use Variables and Data Sources

Please implement variables and data sources in your IaC code to enhance flexibility and maintainability. Variables make it easier to adapt and modify configuration settings, while data sources allow you to retrieve information from external sources to inform your infrastructure.

3. Remote State File

You can store your state file remotely. This practice enhances collaboration, security, and version control of your IaC code. Consider using your infrastructure's remote state storage service, such as Terraform Cloud or AWS S3.

4. Security First

Keep security in mind throughout your IaC development. Ensure your infrastructure is configured with appropriate security measures and adhere to best practices for secure and compliant deployments.

If you are a visual learner, feel free to check out the video solution

Repository containing the code for AWS Terraform 👉Click here👈

Architecture Diagram for AWS by Mahesh Upreti

Blog containing step-by-step instructions for AWS

Blog by Mahesh Upreti

Architecture Diagram for Azure by Joel

Blog containing the step-by-step instructions for Azure

Blog by Joel

AWS Networking - AWS VPC, Subnets, Security Groups, NAT Gateway & IP Addresses

piyushsachdeva — Wed, 22 Feb 2023 21:21:59 +0000

AWS Networking can be a complicated topic, but it's an essential part of building and managing resources on the AWS Cloud. In this comprehensive guide, we'll take a deep dive into the key components of AWS Networking and explore everything you need to know to get started.

let the game begins...

Virtual Private Clouds (VPCs)
At the heart of AWS Networking is the Virtual Private Cloud (VPC). A VPC is a virtual network that enables you to launch AWS resources into a virtual network that you define.It provides you with complete control over your virtual networking environment, including the selection of IP address ranges, subnets, and configuration of route tables and network gateways.

Let's look at each of these component in detail.

Resources that you create in AWS resides pysically in one or more Data Centres usually 100 miles apart from each other.
Collection of multiple data centres are referred to as an Availability Zone such as ca-central-1a , ca-central1b.
Collection of multiple Availability zones in a geographical location is referred to as an *AWS Region such as ca-central-1.

Below diagram shows an AWS Region(ca-central-1) that consists of two Availability Zones (ca-central-1a and ca-central-1b) that are part of VPC A with the CIDR range(10.0.0.0/16)

Subnets

A subnetwork or subnet is a logical subdivision of an IP network.
It further divides a VPC into multiple small networks so that they can be managed seperately.
The practice of dividing a network into two or more networks is called subnetting.

For example, a VPC having 10.0.0.0/16 = 65,536 IPs can be broken down into 4 subnets:

10.0.1.0/24 = 256 IPs
10.0.2.0/24 = 256 IPs
10.0.3.0/24 = 256 IPs
10.0.4.0/24 = 256 IPs

5 IPs per CIDR are reserved by AWS and rest of them will be available for further use.

Types of Subnets:
There are two types of subnets: Private and Public.
Public Subnets: If you want your instance in a public subnet to communicate with the internet then you use public subnet. Generally, web facing instances are placed in Public subnets.

Private Subnet: If a subnet doesn't have a route to the internet gateway, the subnet is known as a private subnet.
Generally, your DB servers are places in private subnets.

In the below diagram, we have added 1 Public and 1 Private subnet in each of the Availability zones.

Internet Gateway

Internet Gateway allows communication between your VPC and the internet. Only one IGW can be attached to one VPC and vice-versa.

In the below diagram, we have attached Internet Gateway to the VPC.

Internet gateway itself doesn’t provide access to the internet
Route table must be associated with the subnets and routes should be defined.

A route table contains a set of rules, called routes, that are used to determine where network traffic from your subnet or gateway is directed.

Public Subnet should have a route to the internet gateway while , Private subnet should have a route to the local network. As shown in the below diagram:

After attaching the route tables with the subnets, our diagram will look something like below

Now, lets talk about another important feature of VPC:
Security Groups

A security group acts as a virtual firewall for your EC2 instances to control incoming and outgoing traffic.
Inbound rules control the incoming traffic to your instance, and outbound rules control the outgoing traffic from your instance.
In the rules, you define, what type of traffic is allowed/denied from which source. e.g allow http traffic from 0.0.0.0 to the EC2.

Once you attach these security groups with your EC2 instances, the diagram will look something like below:

NACL(Network Access Control List) :
Like Security groups acts as a firewall on EC2/host level, NACL or Network Access control list acts as an additional layer of firewall on subnet level.

Default NACL allows all inbound and outbound traffic to your subnets.You can create a custom network ACL and associate it with a subnet.
A network ACL contains rules and a priority assigned to each rule, rules are evaluated based
on their priority, lower the number, higher the priority.

Below is a sample NACL Rule:

After adding the NACL rules to your subnets, our diagram will look something like below:

Now, there are a lot of usecases where your instances in private subnet needs access to the internet. For instance, Database instance needs regular updates/patching to be done by downloading updates from the internet. This can be done securely using NAT Gateway which allows instances in the private subnet to connect to the internet via a secure route.

Nat Gateways should be launched in Public Subnets (One per AZ). Something like below:

Conclusion
AWS Networking can be a complicated topic, but by following best practices and using the tools provided by AWS, you can build a secure and efficient network for your resources.

Want to see all the networking components with detailed explanation? Check out the below tutorial for the same:

References 📚:
https://docs.aws.amazon.com/vpc/latest/userguide/amazon-vpc-limits.html
https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-security-groups.html
https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html
https://aws.amazon.com/about-aws/global-infrastructure/
https://docs.aws.amazon.com/vpc/latest/userguide/default-vpc.html
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-instance-addressing.html

Boost Your Productivity with ChatGPT - A Tutorial for DevOps/Cloud

piyushsachdeva — Fri, 17 Feb 2023 03:03:09 +0000

If you’re an IT professional or a student, you must have come across the term *ChatGPT * at least once in the past few months. ChatGPT is an AI-based tool that allows users to have conversations and answer queries similar to that of a human being. The goal is to make it easier for people to communicate with computers and access information effortlessly. With its rising popularity, ChatGPT has become a vital tool in the field of DevOps and Cloud computing. In this tutorial, we will discuss how ChatGPT can help boost your productivity exponentially.

To get started, sign up for the free version of ChatGPT. The user-friendly interface is similar to that of a phone’s messaging application.

Simply visit the URL : https://chat.openai.com/ aand signup using your email account. Once logged in, it will look something like below:

you can ask ChatGPT to generate prompts, and it will provide you with a list of results. For example, if you’re a beginner in DevOps, you can ask ChatGPT to provide you with sample web applications that you could use in your CI/CD project.

As soon as you hit enter, it will generate the results similar to below:

If you’re a beginner and looking for more ideas, you can ask ChatGPT to generate more ideas by entering the prompt as below:

If you think that this is not ideal for a beginner in programming, you can enter another prompt to generate ideas for a beginner in coding as below:

Now that we have a lot of beginner friendly project ideas, let's pick one from the list and start working on that.For Instance, I have chosen : A simple calculator app.

You can prompt chatGPT to generate the code in any specific programming language or let it choose the language for you. Code generate by ChatGPT is not always 100% accurate but you can quickly fix that if you know code debugging a little bit. Below are the results:

Once you have the code, you can prompt ChatGPT to dockerize the web application.

It would have generated the dockerfile along with the steps to build the image and deploy the container.

Now you can take the usecase one step further and prompt ChatGPT to create a Jenkins CI CD pipeline using the above details.

The same way, it has generate the pipeline code along with all the prerequisites steps and details of each line of the code. Pretty Neat Eh!

You can perform a lot more tasks such as:

Generating documentation
Adding comments in your code using prompt
convert code from one language to another
generate readme file for your application code
troubleshooting the issue by pasting errors as a prompt and the usecases are numerous.

If you're interested in improving your productivity using ChatGPT, be sure to check out the full tutorial for DevOps/Cloud on YouTube! 🎥👨‍💻

Introducing the AWS Transfer Family Service

piyushsachdeva — Sun, 08 Jan 2023 04:26:12 +0000

Are you tired of slow, insecure data transfer processes? 💤 The AWS Transfer Family is here to help! 🚀

What is the AWS Transfer Family, you ask? It's a group of services that enables you to quickly and securely move data to and from the cloud. 🌩️ The family includes AWS Transfer for SFTP, AWS Transfer for FTP, and AWS Transfer for FTPS.

But what makes the AWS Transfer Family so great? 🤔 For starters, it's super easy to use. Setting up and managing your data transfer process is a breeze, thanks to the user-friendly interface and integration with other AWS services such as Amazon S3 and Amazon EC2. 💻 This saves you time and resources, so you can focus on running your business.

But that's not all! The AWS Transfer Family is also extremely secure. 🔒 All data transfer activities are encrypted, ensuring that your sensitive information is always protected. Plus, you have the option to enable multifactor authentication for an extra layer of security. 🛡️

But how does it all work? 🤔 It's actually quite simple. First, you choose the service that best fits your needs - SFTP, FTP, or FTPS. Next, you set up your server and configure your data transfer process, including the location of your Amazon S3 bucket. Finally, you can test your data transfer by uploading and downloading files to and from your S3 bucket. 📤📥

Checkout this cool animated video to see how it works!

Want to see the AWS Transfer Family in action? 💪 Check out our demo of how to set up an SFTP server and test the data transfer process:

And that's it! You've successfully set up an SFTP server using AWS Transfer Family and tested the data transfer process. 🎉 As you can see, it's a quick and easy way to move data to and from the cloud. 🌩️ So why not give the AWS Transfer Family a try today? 🚀

🙏Thank you for following along with the tutorial so far. If you found this blog to be helpful, please be sure to follow me and consider subscribing to my YouTube channel. Good luck 👍 on your cloud journey!

How To Install Multiple Gitlab Runners On a Single AWS EC2 machine

piyushsachdeva — Mon, 05 Dec 2022 17:19:52 +0000

In this quick tutorial, I am going to show you how you can install multiple Gitlab Runners on a single AWS EC2 machine.

Step 1

Provision a EC2 server

Give it a name such as Gitlab-Runner and select a suitable Linux Image. In this demo, I have selected an Ubuntu image

Then scroll down and select the key-pair that you would want to use for this instance or you can create a new key-pair. Once the key-pair is selected, you can create a new security group with the wizard and open access to port 22(SSH) from your local machine or a CIDR range and for your application server so that gitlab runner can ssh into your application server for code deployment. You can optionally use an existing security group as well.

Scroll down to the advance details section and keep everything as default except the user data section. Add the below user data script and hit Launch instance.



#!/bin/bash
sudo apt-get update -y
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash

sudo apt-get install gitlab-runner -y

#install docker 
curl -fsSL https://get.docker.com -o get-docker.sh 
sudo bash get-docker.sh

cd /var/run/ 
sudo chmod 777 docker.pid docker.sock

Step 2

Register the Runner

Once server is running, login to the server using EC2 instance connect or ssh client such as Putty and enter the command

gitlab-runner register

then enter your gitlab running when prompted

Go to your Gitlab UI --> Settings --> CICD and Expand the section in front of Runners

Copy the registration token from here and paste the same in the terminal prompt

Enter the details as below to register the runner

Step 3

Start the Runner

Start the runner using the below command: sudo gitlab-runner start

once it is started, hit refresh on the Gitlab page and you would see the runner as active

Give yourself a pat on the back! you were able to perform the task successfully!

Step4

Register second runner(Optional)
Now , if you would like to register multiple runners on the same ec2 server, using can do so by following the same method. Give a different name and tag to the runner as follows

Now, refresh the page to see the new runner as well

You can use the runners based on the tags or the names.

That's all folks. If you have enjoyed the blog and learned something new today. Feel free to hit the like button and follow me for more such content.

I have also published a full end to end tutorial for Gitlab CI CD Pipeline, feel free to check that out using the below link

How To Mount AWS S3 Bucket On Amazon EC2 Linux Using S3FS

piyushsachdeva — Wed, 31 Aug 2022 17:16:21 +0000

In this blog, we will learn how to use S3 as a filesystem on EC2 Linux machine.

Let's start!

1) Create an EC2 Linux (I have used Ubuntu in this demo) instance

Keep everything as default and add the below user data script to install awscli and s3fs utlity from advance section of wizard



sudo apt-get update -y
sudo apt-get install awscli -y
sudo apt-get install s3fs -y

2) Create an IAM user for s3fs

3) Give the user a unique name and enable programmatic access

Set permission --> create a new policy

Select the service as S3 and include below access levels

Give the policy a unique name and click Create policy

Once the policy is created, go back to the IAM tab and hit refresh so that newly created policy is included in the list
, filter by policy name and hit the enable checkbox to add the policy to our IAM user.

Hit create user

Once the user is created, download the credentials. We are going to use it later.

4) Login to your Ec2 Instance

Go to your home directory and run below commands to create a new directory and to generate some sample files



mkdir /home/ubuntu/bucket; cd $HOME/bucket ;touch test1.txt test2.txt test3.txt

Next step is to create an S3 bucket.
5) Go to S3 service and create a new bucket

give it a unique name and leave reast of the settings as default.
Block public access to this bucket should be enabled by default

Hit create bucket.

6) Once the bucket is created, go to the ssh session and configure our AWS credentails for authentication using the IAM account that we have created.

Use the command



aws configure

and provide the credential details that we have downloaded before

7) Now run the below command to sync local directory with the S3 bucket



aws s3 sync path_on_filesystem s3://bucketname

For example,



aws s3 sync /home/ubuntu/bucket s3://test-s3fs-101

8) create the credential file for s3fs

s3fs supports the standard AWS credentials file stored in ${HOME}/.aws/credentials. Alternatively, s3fs supports a custom passwd file.
The default location for the s3fs password file can be created:
using a .passwd-s3fs file in the users home directory (i.e. ${HOME}/.passwd-s3fs)

file should have the below content:



$AWS_ACCESS_KEY_ID:$AWS_SECRET_KEY_ID

You can run the below command as well:



echo "AKIAQSCIQUH6XXYQMGDA:T5qM7rZmSaU3p/Y0xmuZyWv1/KUnT0Oc58sdCJ3t" > ${HOME}/.passwd-s3fs;
chmod 600 ${HOME}/.passwd-s3fs

9) Now you can run the command to mount S3 bucket as a filesystem



sudo s3fs bucketname path  -o passwd_file=$HOME/.passwd-s3fs,nonempty,rw,allow_other,mp_umask=002,uid=$UID,gid=$UID -o url=http://s3.aws-region.amazonaws.com
,endpoint=aws-region1,use_path_request_style

for example:



sudo s3fs s3fs-test-101 /home/ubuntu/bucket  -o passwd_file=$HOME/.passwd-s3fs,nonempty,rw,allow_other,mp_umask=002,uid=1000,gid=1000 -o url=http://s3.ca-central-1.amazonaws.com
,endpoint=ca-central-1,use_path_request_style

10) Once it is mounted successfully, you can verify by running the command



mount|grep s3fs

11) Add the entry in fstab using the below command so that the changes become persistent after the server reboot as well:



bucketname directoryonfs fuse.s3fs _netdev,allow_other 0 0

For example:



s3fs-test-101 /home/ubuntu/bucket fuse.s3fs _netdev,allow_other 0 0

12) Now the moment of truth, go to your S3 bucket and hit refresh, you should see the files that were present in your file system

13) Let's now verify whether it's getting synced properly after a object delete/addition

Go to your S3 bucket, and upload a new file

Go to your ssh session and do ls in the same directory

Eureka! The file that you just uploaded in your S3 bucket appears in your FileSystem.

Same way you can test the delete file operation. And it works both ways i.e if you perform any file operation on your filesystem, it will sync to your S3 bucket as well.

Feel free to checkout the below hands-on demo of what we have learned so far:

References:
https://github.com/s3fs-fuse/s3fs-fuse
https://aws.amazon.com/
https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

Limitations

Generally S3 cannot offer the same performance or semantics as a local file system. More specifically:

random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy
metadata operations such as listing directories have poor performance due to network latency
non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020)
no atomic renames of files or directories
no coordination between multiple clients mounting the same bucket
no hard links inotify detects only local modifications, not external ones by other clients or tools