DEV Community: Sarvar Nadaf

Introducing AWS FinOps Agent - An AI That Investigates Your Cloud Costs

Sarvar Nadaf — Thu, 11 Jun 2026 19:10:23 +0000

👋 Hey there, Tech Enthusiasts!

I'm Sarvar, a Cloud Architect who loves turning complex tech problems into simple solutions. I've worked with AWS, Azure, DevOps, Data, Analytics, Generative-AI and Agentic-AI building real systems for real companies. In this article series, I'll share what I've learned in a way that's easy to follow, whether you're experienced or just getting started.

Let's get into it! 🚀

If you have managed AWS costs across multiple accounts, you know the pain. You get an alert at 9 AM that something spiked overnight. You open Cost Explorer, filter by service, then by account, then by region. Then it is over to CloudTrail to figure out what changed. By the time you trace it to a developer who launched a new instance type in a test account, two hours have passed. And that was just one anomaly.

I have been doing this for years across organizations running 50 to 500 AWS accounts. Monthly cost reviews, chasing engineers for explanations, building reports in spreadsheets that nobody reads until budget meetings. When AWS announced the FinOps Agent in public preview on June 9, 2026, I cleared my afternoon and set it up.

This article is my honest take after spending real time with it what it does well, where it fits in an enterprise setup, and how it changes the daily work for architects and FinOps teams.

What Is AWS FinOps Agent, in Plain Terms?

It is an AI agent that sits on top of your existing AWS cost tools Cost Explorer, Cost Anomaly Detection, Cost Optimization Hub, Compute Optimizer, and CloudTrail and does the manual work you have been doing yourself.

You ask it questions in plain English. It pulls data from those services, correlates events, and gives you answers. You can also tell it to run tasks on a schedule or react when a cost anomaly fires.

It runs in us-east-1 during the preview, but it can see cost data from all your regions (except GovCloud and China). It is free during the preview period with a monthly usage cap.

The underlying engine uses Amazon Bedrock foundation models, but you do not need to know or care about that. You just talk to it.

Setting It Up Genuinely Simple

I was expecting a 30-minute setup with IAM policy headaches. It took about 5 minutes.

Here is the actual flow:

Open the AWS Console, switch to us-east-1, find FinOps Agent
Click "Get Started" and give your agent a name
It creates an IAM role for you with one click read-only access to billing and cost management services
Optionally connect Jira and Slack
Open the web application and start asking questions

That is it. No CloudFormation stack, no Terraform module, no custom Lambda functions. The whole process felt like it was designed for people who have better things to do than fight with IAM policies.

One thing to note: if you have multi-session enabled in your AWS Console, disable it first. I hit a permissions error during Slack setup that traced back to this. The documentation does not make this obvious upfront.

Also, if you are setting up Slack, you need to add the FinOps Agent app as a member of the channel before configuring the integration in the console. Miss that step and you will get an error with no helpful message. The AWS docs mark it as "Important" but it is easy to skip over.

The First Question I Asked

I opened the chat and typed:

"What were my top 5 cost drivers last month, grouped by service and account?"

Within about 15 seconds, it came back with a table showing exactly that. It pulled from Cost Explorer, broke it down by service, showed the account names, and included the percentage change from the previous month.

No dashboard to build. No saved report to configure. Just a question and an answer.

I then asked a follow-up about why EC2 costs went up in one of my accounts. The agent went deeper it identified the instance type that caused the increase, surfaced the relevant CloudTrail event showing when those instances were launched, and pointed to the IAM role that made the API call.

That kind of investigation jumping between Cost Explorer and CloudTrail, matching timestamps, finding the right principal usually takes me 20 to 30 minutes. The agent did it in under a minute.

One thing I learned quickly: by default, costs shown include credit adjustments. If you want pre-credit numbers (which is what most enterprises care about for budgeting), you need to say that in your prompt. I told the agent to always exclude credits, and it remembered that preference in my next session without me repeating it.

Where This Solves Real Enterprise Pain

Let me map this to the problems I have seen in every organization I have worked with.

Problem 1: Cost Anomalies Go Uninvestigated

Every enterprise I know has AWS Cost Anomaly Detection turned on. Most of them also have a shared email inbox or Slack channel where those alerts land. And in most cases, those alerts sit there for hours or days before someone looks at them.

The reason is simple investigation is manual and time-consuming. AWS FinOps Agent automates this. You set up an automation that says: "When a cost anomaly is detected above $500, investigate it, find the root cause, and post the findings to our #finops-alerts Slack channel."

From that point, every anomaly gets investigated the moment it arrives. The output includes what changed, the CloudTrail event that caused it, the IAM principal responsible, and a summary of the likely root cause.

For a small FinOps team supporting many accounts, this is the difference between catching problems in hours versus catching them in days.

Problem 2: Engineers Cannot Self-Serve Cost Information

In most enterprises, if a developer wants to understand their team's cost, they either need Cost Explorer access (which many organizations restrict) or they file a request with the central FinOps team.

This creates a bottleneck. In my experience, the FinOps team ends up spending a significant chunk of their time answering basic questions like "How much did our staging environment cost last month?" or "Which of our Lambda functions is the most expensive?"

With FinOps Agent, engineers ask their question, the agent answers it using real data, and the FinOps team never gets involved.

You can also upload context files a CSV that maps accounts to team owners, a document that explains your tagging conventions, a list of which cost centers map to which business units. The agent uses this context to understand questions like "What is Team Alpha spending?" without anyone having to explain which accounts belong to Team Alpha.

Problem 3: Monthly Reports Are a Time Sink

Every FinOps team I have worked with produces some version of a monthly cost report. It goes to finance, to engineering leadership, sometimes to the CTO. Building it involves exporting data from Cost Explorer, formatting it in a slide deck or spreadsheet, adding commentary about what changed, and sending it out.

This takes 4 to 8 hours per month in my experience. Some teams spend even longer because different stakeholders want different views.

FinOps Agent lets you schedule these. You tell it: "Every Monday at 8 AM, generate a cost report for last week broken down by business unit, highlight any changes over 10%, and deliver it as a PDF to the #finops-weekly Slack channel."

It runs on schedule. No human involvement. The report shows up in Slack every Monday. You can set up different reports for different audiences a detailed one for engineering leads, a summary for finance, an executive view for the CTO.

The output formats are HTML, PDF, or PPT all presentation-ready.

Problem 4: Optimization Recommendations Sit in a Dashboard Nobody Checks

AWS Cost Optimization Hub and Compute Optimizer both produce solid recommendations rightsizing opportunities, idle resources, Savings Plans suggestions. The problem is that these recommendations live in AWS dashboards that engineers rarely check.

FinOps Agent can pull these recommendations, summarize them, and create Jira tickets assigned to the team that owns each resource. Instead of a FinOps analyst manually reviewing the dashboard and creating tickets, the agent does it on a schedule.

Example automation: "Every Thursday, check Cost Optimization Hub for new recommendations over $100/month in potential savings. For each one, create a Jira ticket in the INFRA project with the recommendation details, the affected resource, and the estimated savings."

Now optimization work flows into engineering sprints naturally, without someone manually copying data between tools.

How It Fits Into an Enterprise Architecture

Let me describe how I would deploy this in a typical multi-account enterprise setup.

Management Account Setup

You create the FinOps Agent in your management account (or the account that has the consolidated billing payer role). This gives it visibility across all member accounts. The IAM role it creates is read-only by default it cannot make changes to your infrastructure.

Member account owners can also create their own agent scoped to just their account if they want independent cost visibility without management account access. This means you can have multiple agents with different scopes a central one for the FinOps team and individual ones for teams that want their own view without seeing the full organization.

Access Patterns

Role	How They Use It
FinOps Team	Set up automations for anomaly investigation, scheduled reports, optimization ticket creation
Engineering Leads	Ask ad-hoc cost questions about their team's spend
Finance	Receive scheduled reports in their preferred format
Platform Team	Monitor shared infrastructure costs, investigate spikes in shared services
Individual Engineers	Self-serve answers about their service's cost impact

How the Pieces Connect

The agent reads from your existing AWS cost services Cost Explorer for spend data, Cost Anomaly Detection for alerts, Cost Optimization Hub and Compute Optimizer for recommendations, and CloudTrail for change history. It writes only to Jira and Slack when you configure those integrations and trigger a task. It does not modify any AWS resources.

Security Model

IAM role-based access (you control what it can read)
Read-only by default against billing services
Write actions (Jira tickets, Slack posts) only happen when you configure and trigger them
All activity logged in CloudTrail for audit
No cross-account write permissions
The agent uses Amazon Bedrock foundation models to process your queries your cost data is accessed through IAM roles within your own account and does not leave AWS

One thing worth noting for architects evaluating risk: this service has zero infrastructure dependency. Your actual cost tools, accounts, and resources are unchanged. If the agent were deprecated tomorrow, you would be back to manual work not rebuilding systems. That makes it a low-risk addition to your environment during preview.

What I Noticed During Hands-On Testing

The Good

Speed of answers. Asking a cost question and getting a real answer in 15-30 seconds changes your workflow. I found myself asking follow-up questions I would never have bothered investigating manually.

Root cause correlation. The fact that it connects Cost Anomaly Detection alerts to CloudTrail events automatically is genuinely useful. This is the step that takes the most time in manual investigation.

Scheduled tasks actually work. I set up a weekly report for Thursday morning. It fired within 2 minutes of the scheduled time, and the PDF it produced was clean and readable. Results also get stored in an "Artifacts" tab in the web app for later access.

Context files are powerful. After uploading an account-to-team mapping, the agent started answering team-level questions correctly without any additional configuration.

Memory across sessions. I told it to always exclude credits when showing cost totals. In my next session, it remembered that preference without me repeating it.

The Rough Edges (It Is Preview After All)

Slack messages are links, not inline content. When the agent posts to Slack, it sends a link back to the FinOps Agent web app rather than the full analysis inline. This means you still need to click through to see the details. For a quick glance during a busy morning, this adds friction.

Sending to Slack is not automatic. After the agent completes an analysis in a chat session, you still need to explicitly prompt "Please send this to Slack." Event-triggered automations handle routing on their own, but for ad-hoc queries you need that extra step.

English only. Prompts in other languages sometimes work, but responses are always in English. Some prompts get a flat "Only English is supported" response. Global enterprises will need multi-language support at GA.

Single region availability. The agent only runs in us-east-1 during preview. It can see multi-region cost data, but the console experience requires you to be in that region.

No IaC support yet. You cannot create the agent via CloudFormation or Terraform. Console-only setup for now. For enterprises that mandate infrastructure as code for everything, this is a blocker until GA.

Compute Optimizer queries are region-scoped. When I asked for rightsizing recommendations, it only checked us-east-1 by default. You need to specify other regions explicitly in your prompt. Easy to miss if you have workloads spread across multiple regions.

Trust but verify. Like any AI system, the agent can occasionally misidentify a root cause or produce an incomplete analysis. In the early days, treat its output as a strong starting point that still needs a human eye before you act on it or forward it to engineering. Once you have seen enough accurate results to build confidence, you can reduce oversight.

Real Workflow: Before and After

Let me lay out what a typical week looks like for a FinOps team before and after adopting this. These numbers assume automations are tuned and the agent is producing reliable output getting there takes a week or two of initial validation.

Before FinOps Agent

Monday morning:

Check Cost Anomaly Detection alerts from the weekend (5 alerts) - 10 minutes
Open Cost Explorer for each alert, filter by account/service/time - 15 minutes per alert
Open CloudTrail, search for relevant API calls - 10 minutes per alert
Write up findings and post to Slack - 5 minutes per alert
Total: about 2.5 hours for 5 anomalies

Wednesday:

Pull Cost Optimization Hub, review new recommendations - 30 minutes
Create Jira tickets for engineering teams - 45 minutes

Friday:

Build weekly cost report for leadership - 1.5 hours

Weekly total for routine FinOps work: roughly 5-6 hours

After FinOps Agent (once automations are stable)

Monday morning:

Check Slack - 5 anomaly investigations already posted over the weekend with root causes identified
Review findings, confirm no false positives - 15 minutes

Wednesday:

Check Jira - optimization tickets already created for engineering teams
Review for accuracy - 10 minutes

Friday:

Weekly report already delivered to Slack on Thursday evening
Quick review for anything unusual - 5 minutes

Weekly total: roughly 30 minutes of review instead of 5-6 hours of manual work

That freed-up time goes to strategic work: Savings Plans analysis, architecture reviews for cost efficiency, building chargeback models, negotiating Enterprise Discount Programs.

Who Should Try This Today

If you are in one of these situations, set it up this week:

FinOps team of 1-3 people supporting many accounts. The automation alone justifies the setup time.
Engineering organization where developers do not have Cost Explorer access. Give them the agent instead.
Any team that produces recurring cost reports manually. Automate them immediately.
Organizations where cost anomaly alerts go uninvestigated. Event-triggered investigation fixes this.

If you are a solo developer with one account and minimal spend, this is probably more than you need right now. But if your monthly bill is complex enough that you spend time understanding it, it is worth trying.

A word of caution: This is a preview service. There is no SLA, features may change before GA, and you should not wire it into critical production workflows without a plan for what happens if the service changes or pricing is introduced. Use it to learn and validate, and be ready to adjust when GA arrives.

What Early Customers Are Reporting

Several companies shared their experience during the preview period. Here is what stood out to me:

Workday runs their AI platform across many AWS accounts. Their team was spending hours every month chasing cost outliers and building leadership reports. They said that anomaly detection and reporting now happens from one natural-language interface, replacing what used to be hours of manual dashboard work.

Convera, a global commercial payments company in a regulated environment, focused on catching small cost changes before they compound. They described the agent completing the full loop detecting anomalies, investigating root cause, and creating Jira tickets for the right engineer so issues reach the owner directly instead of sitting in a shared queue.

Mitre 10, a New Zealand retailer with a lean platform team, was splitting time between reliability work and cost governance. They now define investigation workflows once and have them run continuously, instead of relying on someone remembering to check.

AVIV Group operates digital marketplaces across France, Germany, and Belgium with hundreds of AWS accounts. As they shift to a hybrid FinOps model, the agent answers engineer questions directly, freeing the central team for strategy and leadership reporting.

What I Am Watching For at GA

A few things I want to see before this moves from "useful preview" to "enterprise standard":

Pricing at GA. It is free during preview. Enterprise adoption depends on what it costs relative to the time it saves.
IaC support. We need CloudFormation and Terraform support to deploy this in governed environments.
Inline Slack responses. The current link-based approach adds friction. Full findings posted inline would make the Slack integration much more natural.
Multi-language support. Global enterprises need this.
IAM Identity Center integration. For SSO-based access control in organizations that have moved away from IAM users.
Deeper Slack interaction. Currently delivery-only engineers cannot ask the agent questions from within Slack. That would complete the loop.
Multi-region Compute Optimizer queries by default. Should not require specifying regions manually.

Bottom Line

AWS FinOps Agent does not replace your FinOps team. It replaces the repetitive, manual investigation work that consumes most of their time. It takes the data that already exists across Cost Explorer, CloudTrail, Cost Optimization Hub, and Compute Optimizer, and makes it accessible through conversation instead of dashboards.

For architects, it means getting cost answers during design reviews without waiting for someone to run a report. For FinOps teams, it means spending time on strategy instead of triage. For engineering teams, it means getting cost context delivered to them in Jira and Slack instead of learning yet another AWS console.

It took me 5 minutes to set up, and within the first hour I had it answering questions, investigating anomalies, and scheduling weekly reports. For a preview service, that is a strong start.

The setup guide is at: https://docs.aws.amazon.com/finops-agent/latest/userguide/what-is.html

The console is at: https://us-east-1.console.aws.amazon.com/finops-agent/home

Try it while it is free. Worst case, you spend 10 minutes setting it up and decide it is not for you yet. Best case, you automate away hours of weekly grunt work and spend that time on work that actually needs a human brain.

Written after hands-on testing during the public preview period (June 2026). All features and behaviors described reflect the preview state and may change at general availability.

📌 Wrapping Up

Thanks for reading! If this was helpful:

❤️ Like if it added value
💾 Save for later
🔄 Share with your team

Follow me for more on: AWS architecture, FinOps, DevOps, and AI Infrastructure.

👉 Visit my website | Connect on LinkedIn | Email: simplynadaf@gmail.com

Happy Learning 🚀

Can AI Agents Browse Login-Protected Dashboards? I Tested It

Sarvar Nadaf — Tue, 09 Jun 2026 11:35:23 +0000

👋 Hey there, Tech Enthusiasts!

Let's get into it! 🚀

"Your agent can write Terraform, deploy infrastructure, and debug pipelines. But ask it to check a dashboard behind Cloudflare? It's useless."

If you read my previous article I Let an AI Agent Become My DevOps Engineer you know I've been pushing AI agents into real operational workflows. Code generation, CI/CD pipelines, infrastructure provisioning agents handle all of that now.

But there's one area where every agent I've used just... stops working. The browser.

Here's the reality: Most of the tools I monitor daily Grafana dashboards, AWS Console, CI pipelines, internal wikis live behind login walls, CAPTCHAs, and anti-bot protection. My agents can't touch them.

That's where BrowserAct comes in.

I found this tool that claims to give AI agents real browser control not headless puppeteer scripts that get blocked instantly, but actual anti-detection browsing with CAPTCHA handling and human handoff built in.

I spent a week testing it. Here's what I found.

By the end of this article, you'll:

Understand why AI agents fail at real browser tasks
Install BrowserAct and run your first extraction
See how anti-detection browsing actually works
Test human handoff for 2FA/login scenarios
Run parallel browser sessions without cross-contamination
Turn a repeated workflow into a reusable Skill

Time Required: 30 minutes (15 min read + 15 min hands-on)
Difficulty: Intermediate
Prerequisites: Python 3.12+, Node.js 18+, Google Chrome, terminal access

The Problem: Agents Can't Browse the Real Web

The Reality of AI Agent Automation in 2026

I manage infrastructure across multiple AWS accounts. Every morning I check:

CloudWatch dashboards
GitHub notifications and PR reviews
CI/CD pipeline status
Internal monitoring tools
Competitor product pages (for research)

That's easily 30-40 minutes of tab-switching, scrolling, and context-gathering before I even start real work.

I thought my agent can write Terraform and deploy entire VPCs. Surely it can open a webpage and read some data?

Nope.

Attempt 1: Basic web fetch

curl https://monitoring-dashboard.internal.com

403 Forbidden - Access Denied

Attempt 2: Headless Puppeteer

const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://dashboard.example.com');
// Blocked by Cloudflare challenge

Error: Page stuck on verification challenge

Attempt 3: Selenium
Same story. Detected as bot within 2 seconds.

Problems:

Anti-bot systems detect headless browsers instantly
CAPTCHAs stop automation cold
2FA/login flows need a human but there's no way to hand off
Running multiple accounts in one browser gets all of them flagged
Every script breaks when the site changes layout

My daily routine before BrowserAct:

Open 6 tabs manually
Log into GitHub, AWS Console, Grafana MFA for each
Scroll through notifications, check pipeline status
Copy-paste data into Slack for the team
Repeat next morning

40 minutes. Every. Day.

Sound familiar? I needed something built specifically for this a browser that agents can actually use on the real web.

What is BrowserAct?

Simple Version

BrowserAct is a CLI that gives your AI agent a real Chrome browser with anti-detection, CAPTCHA solving, and human handoff built in.

Think of it like this:

Puppeteer/Playwright = giving your agent a browser that screams "I'M A BOT"
BrowserAct = giving your agent a browser that looks and behaves like a real person

The Five Things It Actually Does

Gets past anti-bot walls: Real fingerprints, proxy rotation, stealth browsing. Sites don't know it's automated.
Handles CAPTCHAs automatically: Cloudflare, DataDome, reCAPTCHA. Solves what it can, escalates what it can't.
Hands off to humans when stuck: Hit a QR code login? SMS verification? It generates a URL. You (or a teammate) open it on your phone, do the human step, and the agent continues from where it stopped.
Runs parallel tasks without interference: Three sessions checking three different things under the same account. They don't step on each other.
Turns workflows into reusable Skills: Did something once? Package it. Run it again without the agent having to figure it out from scratch.

Prerequisites

Before starting, make sure you have:

Python 3.12+ (BrowserAct CLI is Python-based)
Node.js 18+ (for npx skills Skill installation)
uv package manager (or pip)
Google Chrome installed
Terminal/CLI access
An AI agent that can run shell commands (Claude Code, Cursor, Kiro, Codex or just run commands yourself)

Quick Check

python3 --version
# Python 3.12+

node --version
# v18+

google-chrome --version
# Google Chrome 149.x.x.x

If you don't have uv (fast Python package manager):

curl -LsSf https://astral.sh/uv/install.sh | sh

If you don't have Chrome:

# Ubuntu/Debian
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo apt install -y ./google-chrome-stable_current_amd64.deb

# Amazon Linux
wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
sudo yum localinstall -y google-chrome-stable_current_x86_64.rpm

What We're Building

Here's what we're going to test today:

┌─────────────────────────────────────────────────┐
│                 AI Agent (You/Claude/Cursor)    │
└─────────────────────┬───────────────────────────┘
                      │ CLI commands
                      ▼
┌─────────────────────────────────────────────────┐
│              BrowserAct CLI                     │
│  ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │
│  │ Stealth  │ │ Sessions │ │  Human Handoff   │ │
│  │ Browser  │ │ Manager  │ │  (remote-assist) │ │
│  └──────────┘ └──────────┘ └──────────────────┘ │
└─────────────────────┬───────────────────────────┘
                      │ Real Chrome
                      ▼
┌─────────────────────────────────────────────────┐
│  Protected Websites (Cloudflare, Login Walls)   │
└─────────────────────────────────────────────────┘

Step 1: Install BrowserAct

Two parts here the Skill definition (for AI agents) and the actual CLI.

Install the Skill (for agent integration):

npx skills add browser-act/skills --skill browser-act --yes

Output:

◇  Installation complete

│  ✓ ~/.agents/skills/browser-act
│    universal: Amp, Antigravity, Antigravity CLI, Cline, Codex +12 more

Install the CLI itself:

uv tool install browser-act-cli --python 3.12

Output:

Resolved 90 packages in 437ms
Installed 90 packages in 2.09s
Installed 1 executable: browser-act

Verify it works (required on first install):

browser-act get-skills core --skill-version 2.0.2

This returns the environment state, available browsers, and command reference. Run this before anything else it completes the version handshake.

How to Get Your BrowserAct API Key

Go to BrowserAct and sign in to your account.
Click on your profile email address in the top-right corner.
From the dropdown menu, select API Keys.
Click Manage Keys.
Select Create Key.
Enter a name for your API key (for example, Amazon-Q, MCP-Server, or Development).
Click Create.
Copy the generated API key and store it securely. You may not be able to view the full key again after leaving the page.

Note: Treat your API key like a password. Do not share it publicly or commit it to source code repositories.

Set your API key

browser-act auth set <your-api-key>

API key saved.

Step 2: Your First Extraction (Zero Config)

The simplest thing BrowserAct can do extract content from a page without any setup:

browser-act stealth-extract https://httpbin.org/ip

Output from my test:

{
  "origin": "100.54.212.44"
}

Clean, rendered content. No HTML tags, no noise. Just the data.

Let's try something with actual content:

browser-act stealth-extract https://example.com

Output:

# Example Domain
This domain is for use in documentation examples without needing permission. Avoid use in operations.
[Learn more](https://iana.org/domains/example)

Already in markdown format. My agent can read this directly without any parsing.

What stealth-extract does under the hood:

Spins up a lightweight stealth browser
Visits the URL with anti-detection fingerprint
Renders JavaScript (unlike curl)
Returns content in markdown
Tears down the browser

Important note from testing: stealth-extract works great for quick reads on most sites. For heavily protected sites (like nowsecure.nl with aggressive Cloudflare), you'll need a full browser session (Step 3) it has stronger anti-detection because it maintains a persistent fingerprint and proxy.

Step 3: Full Browser Control

Now let's do something more interesting interactive browser automation. First, you need a stealth browser:

browser-act browser create --name "test-stealth" --type stealth --desc "Testing for article"

Output:

id=99703194156616493 name="test-stealth" type=stealth
  desc="Testing for article"

Now open a session:

browser-act --session my-research browser open 99703194156616493 https://github.com/trending

Output:

session_name=my-research
browser_type=stealth
url=https://github.com/trending
title=github.com/trending

See what's on the page (indexed elements):

browser-act --session my-research state

Real output from my test:

url=https://github.com/trending
title=Trending repositories on GitHub today · GitHub

|SCROLL|<html class=js-focus-visible /> (0.0 pages above, 1.2 pages below)

  [1]<a aria-label=Homepage />
  [3]<button type=button aria-expanded=false />
      Platform
  [4]<button type=button aria-expanded=false />
      Solutions
  [8]<a class=NavLink-module__link__EG3d4 />
      Pricing
  [9]<qbsearch-input class=search-input />
      [10]<div class=search-input-container search-with-dialog />
  [12]<a />
      Sign in
  [15]<a class=js-selected-navigation-item />
      Explore
  [17]<a class=js-selected-navigation-item selected />
      Trending

See those numbers? That's how the agent interacts. No DOM parsing, no CSS selectors. Just:

# Click the search input (element 10)
browser-act --session my-research click 10

clicked=10

The page updates search box opens. Now type:

browser-act --session my-research input 10 "browser automation AI"

input="browser automation AI" element=10

This is what they mean by "designed for agent reasoning." The output is compact, indexed, and token-efficient. My agent doesn't waste tokens parsing HTML.

You can also grab the full page as markdown:

browser-act --session my-research get markdown

Returns the entire page content in clean markdown format. Useful when you want to extract data rather than interact.

When you're done:

browser-act session close my-research

session_name=my-research closed=true

Step 4: Anti-Bot in Action

Here's where it gets real. I pointed BrowserAct at nowsecure.nl a site specifically designed to test anti-bot detection. It runs Cloudflare challenges.

# Full browser session on a Cloudflare-protected site
browser-act --session antibot browser open 99703194156616493 https://nowsecure.nl

Output:

session_name=antibot
browser_type=stealth
url=https://nowsecure.nl/
title=nowsecure.nl

It got through. Let me verify by pulling the content:

browser-act --session antibot get markdown

nowsecure.nl
NOWSECURE
---------
### by nodriver

And checking the network traffic shows exactly what happened Cloudflare's turnstile challenge was handled automatically:

browser-act --session antibot network requests

# format: csv
...
GET,200,Script,application/javascript,...,https://challenges.cloudflare.com/turnstile/v0/g/8fc8ed1d8752/api.js
GET,200,Document,text/html,...,https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/turnstile/...

The stealth browser handled the Cloudflare verification without any manual intervention. No CAPTCHA prompt, no block.

BrowserAct uses three layers to get through:

Environment layer - Stealth fingerprint, TLS config, proxy switching. Most blocks never trigger.
Execution layer - If a CAPTCHA appears, solve-captcha handles it automatically.
Human layer - If auto-solve fails, it can hand off to you (more on this next).

I'm not going to claim it works on every site. It doesn't. No tool does. But for the monitoring dashboards and research pages I tested, it got through where Puppeteer and Selenium couldn't.

Step 5: Human Handoff (This Is the Killer Feature)

This is the part that sold me. Here's the scenario:

Your agent is automating a workflow. It hits a login page that requires SMS verification or QR code scan. In Puppeteer world, the automation just dies. Game over.

BrowserAct does something different:

browser-act --session login-task remote-assist --objective "complete 2FA verification"

Real output from my test:

Remote assist session created.

Share this URL with the user:
  https://www.browseract.com/remote-cli/d83544c39e4a4e6ba1cc98f95050e615
expires in 1h 0m

Human assist is now active - the browser is under user control.
Do not send browser commands until the user finishes the assist session.

You open that URL on your phone or another device. You see the browser the actual live browser state. You complete the SMS verification, scan the QR code, whatever. Then you close it.

The agent gets notified and continues from the exact same browser state.

Why this matters for DevOps:

Internal tools with SSO that requires periodic re-auth
AWS Console with MFA
Third-party dashboards with 2FA
Any workflow where "fully automated" isn't realistic

The agent doesn't crash. It doesn't restart. It waits, you help, it continues. That's practical automation.

Step 6: Parallel Sessions (Multi-Task Without Conflicts)

Here's a real use case from my work: I want an agent to check three things simultaneously under the same account.

# Session 1: Check GitHub trending
browser-act --session check-trending browser open 99703194156616493 https://github.com/trending

session_name=check-trending
browser_type=stealth
url=https://github.com/trending
title=github.com/trending

# Session 2: Check GitHub topics (same browser, parallel session)
browser-act --session check-topics browser open 99703194156616493 https://github.com/topics

session_name=check-topics
browser_type=stealth
url=https://github.com/topics
title=github.com/topics

Two sessions. One browser. They don't interfere with each other.

# See all active sessions
browser-act session list

session_name: check-trending
browser_type: stealth
browser_id: 99703194156616493
title: Trending repositories on GitHub today · GitHub
url: https://github.com/trending

session_name: check-topics
browser_type: stealth
browser_id: 99703194156616493
title: Topics on GitHub · GitHub
url: https://github.com/topics

Each session has its own navigation state but shares the login cookies. So you log in once, and all tasks can work in parallel.

The three concurrency models:

Model	What's shared	Use case
Cross-browser parallel	Nothing (independent identity)	Multi-account monitoring
Same-browser multi-session	Login state	Parallel tasks, one account
Privacy mode	Nothing (fresh each time)	One-off scraping, anonymity

When done:

browser-act session close check-trending
browser-act session close check-topics

session_name=check-trending closed=true
session_name=check-topics closed=true

Sessions auto-reclaim after 8 hours if you forget. But clean up after yourself.

Step 7: Skill Forge Make It Reusable

Let's say Step 6 worked great. I want to run that "check my GitHub morning routine" every day without the agent figuring it out from scratch each time.

First, install Skill Forge:

npx skills add browser-act/skills --skill browser-act-skill-forge --yes

Output:

◇  Installation complete

│  ✓ ~/.agents/skills/browser-act-skill-forge
│    universal: Amp, Antigravity, Antigravity CLI, Cline, Codex +12 more

Then tell your agent:

"Forge a Skill that checks GitHub notifications, extracts unread count and top 5 notification titles."

Skill Forge will:

Explore the page structure
Discover the best extraction path
Generate a reusable Skill file
Test it

Next time, the agent just runs the Skill. No re-exploration, no token waste. Same stable path every time.

Where I'd use this:

Daily dashboard checks
Competitor price monitoring
Pull request summary generation
CI/CD status aggregation

Where I wouldn't:

Sites that change layout constantly
One-off tasks that'll never repeat
Anything that needs real-time interaction (chat, live support)

Bonus: How I'd Use This in Production

I ran a few extra tests to see how BrowserAct fits into a real cloud operations workflow. Here's what I found.

Monitoring AWS Health Dashboard

browser-act --session aws-health browser open 99703194156616493 https://status.aws.amazon.com

session_name=aws-health
browser_type=stealth
url=https://health.aws.amazon.com/health/status
title=health.aws.amazon.com/health/status

Then extract the status:

browser-act --session aws-health get markdown

AWS Health Dashboard
====================
Service health - Jun 09, 2026

Take a screenshot for your Slack channel:

browser-act --session aws-health screenshot /tmp/aws-health.png

saved: /tmp/aws-health.png

Real screenshot, 315KB, exactly what the dashboard looks like. Ship that to a Slack webhook and your team gets a visual status check every morning.

Checking GitHub Status (CI Monitoring)

browser-act --session aws-health navigate https://www.githubstatus.com
browser-act --session aws-health get markdown

All Systems Operational
-----------------------
Git Operations   99.85% uptime  Normal
Webhooks         99.96% uptime  Normal
API Requests     99.99% uptime  Normal
Issues           99.97% uptime  Normal
Pull Requests    99.61% uptime  Normal

Now my agent has real uptime data it can act on. If Pull Requests drops below 99%, alert me.

Browsing Cloudflare's Own Product Page (Their Anti-Bot)

The ultimate test can BrowserAct get through Cloudflare's own website?

browser-act --session aws-health navigate https://www.cloudflare.com/products/
browser-act --session aws-health get markdown

Cloudflare Products
-------------------
Everything you need to build, deploy, and scale applications...
Workers Global serverless functions
D1 - Serverless SQL database
R2 - Object storage...

Yes. It extracted their full product catalog through their own protection. That's something.

JavaScript Evaluation (Custom Data Extraction)

Need something specific that the markdown extraction doesn't give you? Run JavaScript directly:

browser-act --session aws-health eval "document.title"

Products | Cloudflare

browser-act --session aws-health eval "document.querySelectorAll('a').length"

This lets you write precise extraction logic without relying on the markdown parser.

My Production Architecture (What I'd Actually Build)

Here's the setup I'm planning for my AWS monitoring workflow:

┌────────────────────────────────────────────────────────────────┐
│                    Cron Job (Every Morning 8 AM)                │
└──────────────────────────┬─────────────────────────────────────┘
                           │
                           ▼
┌────────────────────────────────────────────────────────────────┐
│                   AI Agent (Kiro / Claude Code)                 │
│                                                                │
│  1. browser-act --session grafana browser open <id> <url>      │
│  2. browser-act --session grafana get markdown                 │
│  3. browser-act --session grafana screenshot ./grafana.png     │
│  4. browser-act --session cloudwatch browser open <id> <url>   │
│  5. browser-act --session cloudwatch get markdown              │
│  6. Parse data → Generate summary                             │
│  7. Post to Slack with screenshots                            │
│  8. Close all sessions                                         │
└──────────────────────────┬─────────────────────────────────────┘
                           │
                           ▼
┌────────────────────────────────────────────────────────────────┐
│                 Slack Channel: #morning-status                  │
│                                                                │
│  Morning Infra Report - Jun 04, 2026                           │
│  All AWS services operational                                  │
│  GitHub: 99.85% uptime                                         │
│  Grafana: CPU alert on prod-api-3                              │
│  [dashboard-screenshot.png]                                    │
└────────────────────────────────────────────────────────────────┘

Why this works better than API-only monitoring:

Some dashboards don't expose APIs (internal Grafana, vendor portals)
Screenshots give visual context that JSON data can't
Human handoff means SSO re-auth doesn't break the whole pipeline
One browser session stays logged in no re-auth every day

Production considerations:

Run on a dedicated EC2 instance (t3.medium is enough tested)
Use static proxy for stable identity (avoids triggering "new device" alerts)
Set --desc on browsers with site/account info so future sessions know what's what
Monitor credit usage each session costs credits, budget accordingly
Store screenshots in S3, link in Slack messages
If login expires, remote-assist lets you re-auth from your phone without SSH-ing in

Before vs After: How This Changed My Morning

Let me be real about what my daily routine looked like before and after testing BrowserAct.

Before BrowserAct (My Old Morning)

8:00 AM - Open laptop
8:02 AM - Open GitHub, hit MFA prompt on phone, wait
8:04 AM - Check notifications (12 unread), scan PRs (3 need review)
8:08 AM - Open AWS Console, MFA again, navigate to CloudWatch
8:12 AM - Check 3 dashboards across 2 accounts, screenshot for team
8:18 AM - Open Grafana, login, scroll through panels looking for alerts
8:23 AM - Check GitHub Actions 2 repos have failing builds
8:27 AM - Open competitor's changelog page (Cloudflare protected)
8:30 AM - Copy-paste a summary into Slack for the team
8:35 AM - Realize I missed one account, go back to AWS...
8:40 AM - Actually start working

40 minutes of monkey work. Every morning. On good days.

On bad days a session expires mid-check, or MFA doesn't arrive, or Cloudflare blocks my automation script. Then it's 50+ minutes.

After BrowserAct (What I'm Building Now)

8:00 AM - Agent runs automatically (cron trigger)

Agent:
  → Opens stealth session on GitHub
  → Extracts: 12 notifications, 3 PRs pending review
  → Navigates to GitHub Actions: 2 failing builds (api-server, docs-deploy)
  → Opens parallel session: AWS Health Dashboard
  → Extracts: "All Systems Operational"
  → Screenshots CloudWatch dashboard → saves to S3
  → Navigates to competitor changelog: extracts new features list
  → Posts summary to Slack #morning-status

8:01 AM - Slack notification pops up with the full morning report.

8:02 AM - I read the summary with my coffee. Actually start working.

Time saved: ~38 minutes/day. That's 3+ hours/week.

And the thing is it's not just the time. It's the mental context switching. Opening six different login-protected dashboards pulls you out of focus. Having a one-page summary waiting in Slack means I start my day knowing exactly what needs attention.

The Honest Part

It's not fully there yet. Here's my current reality:

What works today: The extraction, screenshots, and parallel sessions all work as shown in this article. I tested every command above on a real EC2 instance.
What I'm still setting up: The cron automation + Slack integration pipeline. That's a week of wiring. I'll write a follow-up when it's running.
The one catch: First login to each site needs me to remote-assist from my phone (handle MFA manually). After that, the session stays authenticated for days. So it's "almost" fully automated I handle MFA once a week, the agent handles the other 4 days.

Where I Wouldn't Use This

Being honest not everything needs browser automation:

AWS resource monitoring - Use CloudWatch alarms + SNS. APIs exist. Don't browser-scrape what you can API-call.
Simple uptime checks - Use UptimeRobot or similar. Don't overkill it.
Data that changes every second - BrowserAct isn't real-time. It's periodic checks.
Banking or highly sensitive portals - Too risky. Keep that manual.

Where It Shines

Dashboards without APIs - Grafana (free tier), internal tools, vendor portals
Multi-account visual checks - AWS Console across 3 accounts, screenshot each
Competitor monitoring - Product pages behind Cloudflare, pricing pages that block scrapers
CI/CD status aggregation - Pull data from GitHub Actions, CircleCI, Jenkins (web UI) into one summary
Any "check and report" workflow that you do manually more than twice a week

What Worked and What Didn't

Worked Well

Stealth browser sessions - get through Cloudflare where Puppeteer/Selenium can't. Tested on nowsecure.nl and cloudflare.com itself both passed.
Real dashboard extraction - pulled structured data from AWS Health Dashboard, GitHub Status, and Cloudflare product pages. Real production use cases, all worked.
Screenshots - screenshot command saves PNGs directly. 315KB for a full-page capture. Great for visual monitoring and Slack reports.
Indexed interaction - the state/click/input model is clean. Way better than parsing DOM. Real element indices, real clicks.
Human handoff - generates a live URL instantly. This solves a real problem I've had for years.
Session isolation - ran two parallel sessions on the same browser, completely independent. No conflicts.
Navigation within sessions - navigate command lets you move across sites within one session. Open AWS Health, then navigate to GitHub Status, same session.
JavaScript eval - eval lets you run custom extraction when markdown isn't precise enough.
Network capture - network requests shows exactly what's happening under the hood. Great for debugging.
Install was clean - uv tool install pulled 90 packages, had it running in under a minute on a t3.medium.

Mixed

stealth-extract vs full sessions - stealth-extract works for quick reads on normal sites, but for heavily protected sites you need a full stealth browser session. Not obvious from the docs.
CAPTCHA solving - solve-captcha returned "no compatible captcha found" when I tested (page had already loaded past it). Couldn't trigger a scenario where it was needed during my testing.
Speed - stealth browsing is slower than raw Puppeteer. Takes a few seconds to open sessions. Makes sense (it's doing more work) but worth noting.

Could Be Better

Documentation - the get-skills command dumps a massive guide. Useful but overwhelming on first read.
Error messages - "Browser launch failed: Connection closed" doesn't tell you much. Took trial and error to figure out I needed a full session for protected sites.
Requires API key for stealth - can't use the anti-detection features without signing up first. Fair, but the free tier is limited.

Pricing & Credits

BrowserAct runs on a credit system:

Free credits: 100 credits on signup.
Paid plans: Check browseract.com/pricing for current rates

For testing and writing this article, the free credits were enough. For production monitoring workflows running daily, you'd need a paid plan.

Troubleshooting

Issue 1: "browser-act: command not found"

bash: browser-act: command not found

Fix: Install via uv (not npm):

curl -LsSf https://astral.sh/uv/install.sh | sh
uv tool install browser-act-cli --python 3.12

The binary installs to ~/.local/bin/ make sure it's in your PATH.

Issue 2: "Browser launch failed: Connection closed"

Error: Browser launch failed: Browser.close: Connection closed while reading from the driver

Fix: This happens when stealth-extract can't handle a heavily protected site. Use a full browser session instead:

browser-act browser create --name "my-stealth" --type stealth --desc "browsing"
browser-act --session task1 browser open <browser-id> <url>

Issue 3: "api_key: not configured"

CLI:
  api_key: not configured

Fix: You need an API key for stealth browser features:

browser-act auth set <your-api-key>

Get one at browseract.com.

Issue 4: Chrome not found

Error: Chrome executable not found

Fix: Install Chrome:

# Ubuntu
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo apt install -y ./google-chrome-stable_current_amd64.deb

# Amazon Linux
wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
sudo yum localinstall -y google-chrome-stable_current_x86_64.rpm

Issue 5: Session already exists

Error: Session name 'my-task' already in use

Fix: Close it first:

browser-act session close my-task

Real-World Use Cases

As a cloud architect, here's where I see this fitting:

1. Infrastructure Monitoring

Check multiple dashboards (Grafana, CloudWatch, Datadog) through a single agent
Generate daily summaries from web-based monitoring tools
Alert on visual changes in dashboards that don't have API access

2. DevOps Workflows

Automate PR review summaries across repos
Check CI/CD status across multiple platforms
Monitor deployment pipelines with login-protected UIs

3. Multi-Account Operations

Manage multiple AWS accounts through Console (when CLI isn't enough)
Monitor multiple SaaS dashboards
Cross-account compliance checks on web portals

4. Research & Data Collection

Track competitor features and pricing pages
Aggregate release notes from multiple vendor sites
Collect public data from protected listing pages

Key Concepts Learned

1. Sessions = Task Workspaces

browser-act --session <name> <command>

Every task gets its own session. Sessions don't interfere. Name them descriptively.

2. Browsers = Identities

Different browsers = different fingerprints, proxies, cookies. Use separate browsers for separate accounts. Three types:

chrome - imports your local Chrome login state
chrome-direct - controls your running Chrome directly
stealth - anti-detection browser with fingerprint masking (needs API key)

3. Skills = Reusable Workflows

Once something works, package it as a Skill. Next time it runs without re-exploration.

4. Three-Layer Anti-Blocking

Environment (fingerprint) → Execution (auto-solve) → Human (handoff). Progressive escalation.

5. Two Extraction Modes

stealth-extract - quick, zero-config, good for simple reads. Lightweight.
Full browser session - persistent, stronger anti-detection, needed for heavily protected sites.

What's Next?

I'm building the full pipeline I described above. Here's my roadmap:

Week 1 (done): Test BrowserAct on real sites this article
Week 2: Wire up the morning monitoring agent (cron + BrowserAct + Slack webhook)
Week 3: Add Skill Forge package the workflow so it's stable across page layout changes
Week 4: Run for 30 days, track: time saved, credits consumed, failures, manual interventions

If there's interest, I'll write a follow-up with:

Actual production usage data (cost per month, reliability %)
The Skill file I built for GitHub monitoring
How many times remote-assist saved me from a broken pipeline
Token usage comparison vs doing the same thing with raw Playwright

The goal is simple: I want my mornings back. 40 minutes of tab-switching replaced by a 2-minute Slack read. BrowserAct is the missing piece between "my agent can code" and "my agent can actually see what's happening in production."

Resources

BrowserAct:

My Previous Article:

I Let an AI Agent Become My DevOps Engineer

Summary

Here's what I found after testing BrowserAct on an EC2 instance (Amazon Linux, t3.medium):

Stealth browser sessions get through Cloudflare tested on nowsecure.nl and cloudflare.com itself
Extracted real data from AWS Health Dashboard, GitHub Status actual production monitoring working
Screenshots save as PNG ready to pipe to Slack/S3 for visual dashboards
Human handoff generates a live remote URL in seconds actually works
Parallel sessions run independently on the same browser no conflicts
Navigation + eval let you build complex multi-step extraction workflows
Indexed interaction (state, click, input) is agent-friendly and token-efficient
Skill Forge makes repeated workflows reusable
stealth-extract has limits use full sessions for protected sites
Needs API key for anti-detection features
Error messages could be clearer when things fail

Bottom line: If you're running AI agents and they need to interact with real websites not just APIs this is worth testing. I'm already planning a daily monitoring pipeline with it. It's not magic. It won't bypass everything. But it solves real problems that I haven't seen other tools handle this cleanly.

Connect & Share

If this was useful:

Star the BrowserAct GitHub repo
Drop a comment what workflows would you automate?
Share with your team
Follow me for the follow-up article

📌 Wrapping Up

Thanks for reading! If this was helpful:

❤️ Like if it added value
💾 Save for later
🔄 Share with your team

Follow me for more on: AWS architecture, FinOps, DevOps, and AI Infrastructure.

👉 Visit my website | Connect on LinkedIn | Email: simplynadaf@gmail.com

Happy Learning 🚀

I Built an AI Agent That Tailors My Resume - Here's How Agents Actually Work

Sarvar Nadaf — Mon, 25 May 2026 13:32:31 +0000

👋 Hey there, tech enthusiasts!

Let's get into it! 🚀

This article is for beginners who've heard about AI agents but haven't built one yet.

The Problem That Started It All

You're looking for a job. You open LinkedIn, Naukri, Indeed search "Senior Python Developer." Hundreds of results pop up. Every company wants something slightly different. One wants AWS + Django. Another wants FastAPI + Kubernetes. The third one wants Flask + microservices.

You have all these skills. That's not the problem.

The problem is time. To stand out, you need to customize your resume for each role. Read the job description carefully. Figure out what they care about. Rewrite your summary. Reorder skills. Tweak bullet points. Write a cover letter. Hit submit.

That's ONE application. Now do that 10 more times.

It's not a skill problem. It's a time problem. You're doing the same thing over and over read, match, rewrite, submit.

What if you could build something that handles all of this for you?

You give it your resume and say "Find Senior Python roles in Bangalore, 20-35 LPA (Lakhs Per Annum ₹20-35 lakh salary), remote-friendly" and it goes to work. It reads job descriptions, matches your skills, finds gaps, rewrites your resume for each role, writes cover letters, and ranks everything by how well you fit.

Same you. Same resume. Same job market. But the boring repetitive part? Done automatically.

That's what got me into agentic AI. Not the hype. The simple thought: "I can automate this annoying thing."

ChatGPT vs AI Agents What's the Difference?

People ask me this all the time. "Isn't ChatGPT already an agent?"

Not really. Here's the simple difference.

ChatGPT - you ask a question, you get text back. Yes, it can browse the web and run code now. But YOU are still driving. You ask, it responds. You ask again, it responds again. It waits for you at every step.

An AI Agent - you give it a goal and it goes and does the work. Multiple steps. Multiple tools. Without you guiding each step. It decides what to do next on its own.

Here's a simple comparison:

You ask: "Find me Senior Python jobs in Bangalore"
ChatGPT	"Here are some tips: update your LinkedIn, use relevant keywords..." (gives advice, waits for your next question)
AI Agent	Actually searches job boards, finds 47 matches, filters to 12 relevant ones, tailors your resume for each (does the work start to finish)

One waits for you. The other goes and does the work. That's the whole difference.

How Agents Think?

Every agent follows the same basic pattern:

Think → Do → Check → Repeat

Here's what that looks like:

┌─────────┐     ┌─────────┐     ┌─────────┐
│  THINK  │───▶│   DO    │────▶│  CHECK  │
│ What's  │     │ Take    │     │ Did it  │
│ next?   │     │ action  │     │ work?   │
└─────────┘     └─────────┘     └─────┬───┘
     ▲                                │
     │          Not done yet          │
     └────────────────────────────────┘
              ✅ Done? → Stop

That's it. The agent looks at the goal, figures out the next step, does it, checks if it worked, and keeps going until the job is done.

(You might hear people call this "ReAct" which stands for Reason + Act. It's a formal way of saying the agent thinks about what to do, then does it, then thinks again. But at its core, it's just the loop above.)

Let me show you how this works with the resume example:

Goal: "Tailor this resume for an Amazon SDE role."

Agent reads the job description. Amazon wants Python, AWS, and distributed systems.
Agent reads your resume. Python and AWS are there. But distributed systems? It's buried in one bullet point that nobody would notice.
Agent rewrites that bullet point to make distributed systems stand out.
Agent checks again are all the important keywords from the job description covered? Not yet. Kafka is missing.
Agent looks at your experience. You actually used Kafka at your last job but never put it on the resume. (We all do this.)
Agent adds Kafka to the relevant section.
Final check. Everything important is covered. Done.

Seven steps. No human needed. The agent figured out what was missing, fixed it, checked its work, and moved on.

This is how every agent works whether it's tailoring resumes, monitoring servers, or analyzing data. Same loop, different tasks. (Okay, I'm oversimplifying a bit real agents sometimes have more complex decision trees. But this mental model will get you through 90% of what you'll build.)

The Autonomy Spectrum How Much Freedom Should an Agent Have?

Before we look at how to build an agent, let's understand how much freedom you can give one. "Agent" isn't all-or-nothing. It's a range.

Level	What Happens	Example	Risk
Fully Manual	Human does everything	You tailor your resume by hand	None
Assisted	AI suggests, human does	ChatGPT suggests bullet points, you copy-paste	Low
Orchestrated	Human defines steps, agent follows them	You say "Step 1: read JD, Step 2: rewrite summary" agent does exactly that	Low-Medium
Semi-Autonomous	Agent decides steps, pauses for approval	Agent figures out what to do, asks before submitting	Medium
Fully Autonomous	Agent does everything, no human involved	Agent searches, tailors, submits you just get a report	High

Most real-world agents today are in the middle some human control, some agent freedom.

My advice: Start with orchestrated you define the workflow, the agent executes it. Once you trust it, give it more freedom gradually. Add human checkpoints for important steps. Fully autonomous? Maybe someday. Not yet for anything that matters.

The resume tailor agent we're building in this series? Semi-autonomous. It figures out what to fix and does the work, but shows you the result before anything gets submitted.

The Building Blocks of Every Agent

Now that you know how much freedom an agent can have, let's look at what you actually configure to make one work. Most frameworks I've used have these same pieces. Let's group them by what they do.

The Core: Who does what?

LLM (The Brain) - The AI model that does the thinking. GPT-4, Claude, Amazon Nova, Llama. Don't always use the most expensive model. Use cheap models for easy tasks, powerful models for hard tasks.

Role (The Identity) - Tell the agent WHO it is. Example: "You are a Senior Tech Recruiter with 15 years of experience." The more specific the role, the better the output. Vague roles give vague results.

Goal (The Finish Line) - When should the agent stop?

❌ Bad: "Help with job search"
✅ Good: "Find 5 Senior Python roles in Bangalore, 20-35 LPA, remote-friendly, from companies older than 3 years"

Task (The Instruction) - The specific work to do right now. "Read this job description. Compare it against the resume. List the top 5 skill gaps." Be as specific as possible the more you leave unclear, the more the agent makes up.

The Context: What does the agent know?

Backstory (The Personality) - Shapes HOW the agent approaches work. Same task, different backstory, different style:

"Seasoned recruiter" → practical, beats resume-scanning software
"Career coach" → encouraging, focused on growth
"Brutally honest reviewer" → tears your resume apart (but you learn the most)

Knowledge Base (The Facts) - Information you hand the agent upfront. "The candidate is a Senior Python Developer. Based in Bangalore. Prefers remote. Salary: 20-35 LPA." Without this, the agent guesses. And it guesses wrong.

Memory (What It Remembers) - Two types:

Short-term: Remembers what happened earlier in the current task. "I already found Python listed in the resume."
Long-term: Remembers across sessions. Run it next week, it knows what worked last time.

The Safety Net: What keeps it under control?

Tools (The Hands) - What the agent can actually DO beyond generating text: search the web, read files, call APIs, run code. Without tools, it just writes. With tools, it takes action. Not every agent needs tools a resume rewriter just processes text, but a job searcher needs API access.

Guardrails (The Limits) - Rules that keep the agent in check:

Max attempts → stop after X tries (otherwise it loops forever and burns your budget)
Allowed actions → limit which tools it can use
Output checks → verify the result makes sense

I always set guardrails. Always. I learned this at 2 AM watching my bill climb because an agent got stuck in a loop.

Human-in-the-Loop (The Pause Button) - For important decisions, make the agent stop and ask. The agent tailored your resume and is about to submit it. Instead of auto-submitting, it shows you the final version. "Should I submit this?" You review, approve, it continues.

Automate the boring parts. Keep human judgment for the important decisions.

Quick Reference

Group	Pieces	One-line summary
The Core	LLM, Role, Goal, Task	Who is this agent and what's it doing?
The Context	Backstory, Knowledge Base, Memory	What does it know?
The Safety Net	Tools, Guardrails, Human-in-the-Loop	What can it do and what stops it?

Ten pieces, three groups. Now let's see how multiple agents work together.

Multiple Agents Working Together

One agent is useful. But what if the job is too big for one?

You split the work. Just like a real team each person handles what they're best at.

Think of it like a restaurant:

One chef doesn't do everything. There's someone who preps ingredients, someone who cooks, and someone who plates the dish.
Each person does one thing well. Together, they serve a full meal.

Agents work the same way.

Our resume example with 3 agents:

Agent	Job	Output
Job Analyzer	Reads the job description, pulls out what matters most	"They want Python, AWS, and distributed systems experience"
Resume Writer	Takes that analysis and rewrites the resume to match	A tailored resume highlighting the right skills
Cover Letter Writer	Uses both outputs to write a personalized letter	A cover letter that connects your experience to their needs

Each agent does one thing. Together, they handle the full workflow.

How Do They Pass Work to Each Other?

Three patterns. You only need to know the first one for now.

1. Sequential - Like a relay race A→B→C

Agent 1 finishes → hands output to Agent 2 → hands to Agent 3.

This is what we'll use. It's the simplest. Easy to understand, easy to fix when something breaks you know exactly which agent caused the problem.

2. Hierarchical - Like a manager assigning tasks

One "boss" agent gives work to others and collects results. Useful when you need coordination, but more complex to set up.

3. Parallel - Everyone works at the same time

Multiple agents work on independent tasks simultaneously. Faster, but the tasks can't depend on each other. Example: three agents searching three different job boards at the same time.

For this series, we're using sequential. Agent 1 passes to Agent 2 passes to Agent 3. Simple, predictable, easy to debug. You can explore the other patterns later once you're comfortable.

Here's how our 3 agents work together:

┌──────────────┐       ┌───────────────┐       ┌─────────────────────┐
│ Job Analyzer │──────▶│ Resume Writer │──────▶│ Cover Letter Writer │
│              │       │               │       │                     │
│ "They want   │       │ Rewrites your │       │ Writes a letter     │
│  Python, AWS,│       │ resume to     │       │ connecting your     │
│  distributed │       │ highlight     │       │ experience to       │
│  systems"    │       │ those skills  │       │ their needs         │
└──────────────┘       └───────────────┘       └─────────────────────┘

CrewAI - The Framework We'll Use

I tried several agent frameworks. I picked CrewAI because it's simple and respects your time.

Why CrewAI?

Free and open source - active community, regular updates
Ready-made templates - gives you a working project structure immediately
Plain English configuration - define agents and tasks in YAML (simple text files), not complex code
Fast to start - first working agent in about 10 minutes once your environment is set up

Project Structure

When you create a CrewAI project, you get this folder structure:

my-agent-project/
├── src/
│   └── my_agent_project/
│       ├── config/
│       │   ├── agents.yaml      ← WHO: define your agents here
│       │   └── tasks.yaml       ← WHAT: define tasks here
│       ├── tools/
│       │   └── custom_tool.py   ← HOW: custom tools agents can use
│       ├── crew.py              ← WIRING: connects agents to tasks
│       └── main.py              ← START: entry point, kicks everything off
├── knowledge/                   ← FACTS: static files agents can read
├── .env                         ← SECRETS: API keys, model settings
└── pyproject.toml               ← DEPENDENCIES: Python packages needed

You only need to write three things: agent definitions, task definitions, and the wiring between them. Everything else is scaffolding that CrewAI handles.

CrewAI vs LangChain - When to Use Which

	CrewAI	LangChain/LangGraph
Approach	Define agents in simple YAML files	Build custom chains and graphs in code
Learning time	A weekend	A week or more
Setup time	~10 minutes	30+ minutes of boilerplate
Best for	Clear roles, clear handoffs	Complex branching, conditional logic
Feels like	Writing job descriptions for a team	Building a flowchart with many "if/else" paths

Use CrewAI when you know your agents and the flow is straightforward. Agent A does X, passes to Agent B which does Y. Done.

Use LangChain/LangGraph when you need complex logic like "if confidence is above 80% go to Agent A, otherwise retry with Agent B, and if that fails ask a human."

My take: Start with CrewAI. You can always add complexity later. Removing complexity? That's painful.

MCP - How Agents Talk to the Outside World

Your agent can think. But can it actually reach out and grab information from somewhere else? Like a database full of job listings? Or a file stored in the cloud?

That's what MCP does. It gives your agent a way to connect to outside services.

The Simplest Way to Understand MCP

Imagine you buy a new phone. You need to charge it. Now imagine every phone brand had a completely different charger one for Samsung, one for Apple, one for OnePlus, all different shapes.

That's what life was like before USB-C. Now? One cable works for almost everything.

MCP is the USB-C of AI agents. One standard way to connect to any service databases, file storage, APIs, messaging apps. You don't write different connection code for each one.

Back to Our Resume Agent

Our agent needs job listings. Where do those come from?

Maybe a database with thousands of job posts
Maybe files stored in the cloud (like AWS S3)
Maybe a job board API

Without MCP: You write separate connection code for each one. Different code for the database. Different code for S3. Different code for the API. Every new service = more custom code.

With MCP: You configure the connection once in a simple settings file. The agent uses the same standard method to talk to all of them. New service? Just add another configuration. No new code.

The first time I set this up, it took me about 5 minutes. Writing custom database connection code used to take me an hour. That's the difference.

What Else Can Agents Connect To?

Databases - fetch job listings, store results
File Storage (S3) - read resumes, save tailored versions
GitHub - read code, create pull requests
Slack - send notifications, read messages
Web Search - find information online

One standard way to connect to all of these. That's MCP.

We'll set this up hands-on in the next article. For now, just remember: MCP is how your agent reaches out to the world beyond its own code.

What Agents Can't Do Well (Yet)

Let me be honest about the limitations:

They make things up. I've had an agent tell me it "successfully completed" something it definitely didn't do. Always check important outputs yourself.
They cost money. One task = 5-20 AI model calls. A chatbot costs pennies. An agent can cost dollars per run. Multiply by hundreds of runs and your bill gets serious.
They get stuck in loops. Without a maximum attempt limit, an agent will happily retry the same failing approach forever. (I learned this the hard way at 2 AM.)
Same input, slightly different output. This makes testing tricky because you can't always predict the exact result.
They're slow. Multiple AI calls + tool usage = seconds to minutes per task. Not milliseconds.
They "fix" things you didn't ask about. When fixing one issue, an agent sometimes decides to also change nearby code. In a demo, fine. In production, you get unexpected changes. Set tight boundaries.

How I Handle These Problems

Use cheap models for simple tasks. Amazon Nova Lite for easy stuff, Nova Pro for complex reasoning.
Keep prompts short. Every word costs money.
Always set max attempts. Non-negotiable.
Log everything. I send all agent activity to CloudWatch Logs token counts, step durations, success/failure at each step. When something breaks at step 5 of 7, I can trace exactly what went wrong.
Validate outputs. For critical tasks, I add a checking step either another agent that verifies the work, or structured checks that confirm the output meets requirements.

When NOT to Use Agents

Not everything needs an agent. I've seen people over-engineer simple problems because "agents are cool." Skip agents when:

One AI call is enough - Summarize this text? Translate this paragraph? That's one API call, not an agent.
The logic never changes - If there are no decisions to make and the steps are always the same, a simple script is cheaper, faster, and more reliable.
You need instant responses - Agents take seconds to minutes. If you need millisecond responses for a real-time API, agents are too slow.
High volume + simple task - 5-20 AI calls per task × 10,000 tasks per day = a very expensive month. If the task is simple and repetitive, write a rule-based system.
Mistakes are unacceptable - Agents make errors. If a wrong answer has serious consequences (medical, legal, financial), don't rely on an agent without heavy human oversight.
It's just a database query - Reading from a database and returning results? That's a simple API, not an agent.

My rule: If you can draw the logic as a simple flowchart with no "it depends" decisions, you don't need an agent. Use a script. Agents are valuable when there's ambiguity, judgment calls, and multi-step reasoning involved.

What's Next

In the next article, I'm building the resume tailor agent step by step using CrewAI on an EC2 instance with Amazon Nova Pro as the AI model. From setting up the server to running the agent against a real job description.

Stay tuned.

📌 Wrapping Up

Thanks for reading! If this was helpful:

❤️ Like if it added value
💾 Save for later
🔄 Share with your team

Follow me for more on: AWS architecture, FinOps, DevOps, and AI Infrastructure.

👉 Visit my website | Connect on LinkedIn | Email: simplynadaf@gmail.com

Happy Learning 🚀

I Installed Kiro CLI on EC2: Here's Why My First Attempt Failed

Sarvar Nadaf — Fri, 15 May 2026 04:37:18 +0000

👋 Hey there, tech enthusiasts!

I'm Sarvar, a Cloud Architect with a passion for transforming complex technological challenges into elegant solutions. With extensive experience spanning Cloud Operations (AWS & Azure), Data Operations, Analytics, DevOps, and Generative AI, I've had the privilege of architecting solutions for global enterprises that drive real business impact. Through this article series, I'm excited to share practical insights, best practices, and hands-on experiences from my journey in the tech world. Whether you're a seasoned professional or just starting out, I aim to break down complex concepts into digestible pieces that you can apply in your projects.

Let's dive in and explore the fascinating world of cloud technology together! 🚀

I've been using Amazon Q CLI for nearly two years. It became part of my daily workfloquick infrastructure lookups, generating boilerplate, debugging at odd hours. It was reliable, familiar, and I never thought about replacing it.

Then the end-of-support announcement dropped.

At first, I felt that familiar frustration. Another tool sunset. Another migration. But then I paused. Instead of scrambling for a 1:1 replacement, I asked myself: what do I actually need from a terminal AI in 2026?

My answer wasn't "the same thing I had." It was more. I wanted something that understood my team's standards without me repeating them. Something that could pull real AWS data not just guess at it. Something I could shape into different personas for different tasks.

That's when I discovered that Kiro which I'd only known as an IDE also ships a CLI. And it's not just a CLI. It's the tool I didn't know I was waiting for.

Why This Isn't Just Another Migration

Let me be clear: I'm not switching because I have to. I'm switching because Kiro CLI solves problems Q CLI never addressed.

Here's what changed my mind in the first 30 minutes:

🧠 Custom Agents

Define specialized personas with restricted tool access, scoped file permissions, and custom prompts. I now have a DevOps agent that can only touch infrastructure files, and a reviewer agent that's physically incapable of modifying anything. Different tools for different jobs from the same CLI.

📏 Steering Files

Persistent rules that shape every response. My team's infrastructure standards Graviton instances, GP3 volumes, mandatory tags, Lambda best practices are baked into every interaction automatically. No more repeating "use ARM64" in every prompt.

🔌 MCP Server Ecosystem

Connect to AWS Cost Explorer, Pricing API, CloudTrail, Well-Architected tools, and more. This isn't "AI guessing at your costs." This is real data flowing into real recommendations. Within my first week, it found a forgotten NAT Gateway costing $180/month.

📋 Planning Mode

Press Shift+Tab and Kiro switches from "do the thing" to "let's think about the thing first." Structured requirements gathering before implementation. Essential for anything touching multiple services.

📊 Real-Time Usage Tracking

/usage shows exactly where you stand credits consumed, tier limits, what's left. No end-of-month surprises.

🔐 Multiple Auth Options

Builder ID, IAM Identity Center, GitHub, Google. Teams get centralized billing and access control. Solo developers get flexibility.

Amazon Q CLI was a good conversational assistant. Kiro CLI is an extensible development platform that happens to live in your terminal.

That distinction is why I'm writing this guide instead of a "how to cope with deprecation" post.

A Quick Comparison

For those coming from Q CLI, here's what's different at a glance:

	Amazon Q CLI	Kiro CLI
Auth options	Builder ID, IAM Identity Center	+ GitHub, Google
Entry point	`q chat`	`kiro-cli chat` (`q` still works)
Model selection	Manual	Auto-selects per task
Cost visibility	End of month surprise	`/usage` command, real-time
Custom agents	Basic	Full tool/path scoping, hooks
Steering/Rules	Simple rules	Rich markdown standards files
MCP ecosystem	Limited	20+ official AWS MCP servers
Planning mode	❌	✅ Structured requirements first
License	Apache 2.0	AWS IP License

The license change matters if your org has strict open-source requirements. For most of us, it doesn't affect day-to-day use.

Infrastructure Requirements

Before installing, make sure your machine meets these specs:

Requirement	Minimum	Recommended
Architecture	x86_64 or ARM64	Either works fine
OS	Ubuntu, Fedora, or Amazon Linux 2023	Ubuntu 22.04 LTS
glibc	2.34+	2.35+ (ships with Ubuntu 22.04)
RAM	2 GB	4 GB
CPU	2 vCPUs	2 vCPUs is fine
Disk	5 GB free	10 GB+
Network	Internet access	Stable connection

If your glibc is older than 2.34 (check with ldd --version), grab the musl version instead it's the one with -musl.zip in the filename.

My Setup

I'm running this on an EC2 instance:


Instance Type	t3.medium
OS	Ubuntu 22.04.5 LTS
Architecture	x86_64
vCPUs	2
RAM	4 GB
Storage	50 GB GP3
Region	us-east-1

I went with t3.medium (4 GB RAM) to keep things comfortable. AI model interactions benefit from headroom, and the cost difference is negligible for a development tool you use daily.

Installation (Under 5 Minutes)

The whole process is straightforward. Nine steps, no surprises.

Step 1: Download Kiro CLI

SSH into your instance and pull the package:

curl --proto '=https' --tlsv1.2 -sSf 'https://desktop-release.q.us-east-1.amazonaws.com/latest/kirocli-x86_64-linux.zip' -o 'kirocli.zip'

Takes about 10 seconds on a decent connection. The file is around 350 MB.

Step 2: Extract It

unzip kirocli.zip

Step 3: Run the Installer

./kirocli/install.sh

Step 4: Let It Configure Your Shell

It'll ask:

Do you want kiro-cli to modify your shell config (you will have to manually do this otherwise)?

Type yes. This adds the necessary PATH entries so you can run kiro-cli from anywhere.

Step 5: Authenticate

Now it needs to know who you are. I'm using Builder ID it's free, quick to set up, and works well for individual use. You can also use IAM Identity Center for teams, or GitHub/Google if you prefer.

Step 6: Open the Auth URL

Kiro generates a URL for you. Copy it and open it in your browser.

Step 7: Confirm Access

Click Confirm and Continue:

Then click Allow when it asks for data access:

The browser tab closes automatically. You're done with the web part.

Step 8: Back to the Terminal

You should see a success message:

Step 9: Verify It Works

kiro-cli

If you see the Kiro interface, you're in:

That's it. Under five minutes from download to working CLI.

Automatic Migration from Q CLI

This is the part that made me smile. When Kiro CLI installed, it automatically detected and migrated my existing Q CLI setup:

Agents from ~/.aws/amazonq → ~/.kiro/agents/
MCP config from ~/.aws/amazonq/mcp.json → ~/.kiro/settings/mcp.json
Rules from ~/.aws/amazonq/rules → ~/.kiro/steering/

Two years of custom configurations carried over in seconds. The only thing I had to fix was one MCP server with a hardcoded path to ~/.aws/amazonq/ in its environment variables. Everything else just worked.

No starting from scratch. No rebuilding your setup. That's how migrations should feel.

One More Thing: Classic Mode

By default, Kiro opens in the new Terminal UI. It looks polished, but on remote servers and SSH sessions, I prefer classic mode faster rendering, no UI lag, works perfectly in tmux.

Try it once:

kiro-cli --classic

Make it permanent:

kiro-cli settings chat.ui "classic"

Now every time you run kiro-cli, it starts in classic mode:

Why I Prefer Classic Mode on Remote Servers

Faster rendering over SSH
No UI lag
Works perfectly in tmux
More predictable behavior in long sessions

If you ever want the full UI back:

kiro-cli settings chat.ui "tui"

Or just run kiro-cli --tui for a one-off session.

Cleanup (Optional)

That 350 MB zip file is still sitting there. Once you've verified everything works:

rm ~/kirocli.zip

First Impressions After One Week

After seven days of daily use, here's what stood out:

The q command still works. Muscle memory preserved. Kiro CLI responds to both kiro-cli and q.
Model auto-selection is smart. Simple questions get fast responses. Complex architecture questions get deeper reasoning. I don't have to think about which model to pick.
MCP servers change everything. Going from "AI that guesses" to "AI that queries real data" is a paradigm shift. Cost optimization alone justified the switch.
Steering files are addictive. Once you teach it your standards, you can't go back to repeating yourself.

I went from "reluctant migrator" to "why didn't this exist sooner" in about three days.

What's Next

Now that it's installed, the real value starts. In the next article, I'll cover how I use Kiro CLI for actual work setting up MCP servers for cost optimization, creating custom agents that enforce team standards, writing steering files, and the daily workflows that made this tool indispensable.

👉 Part 2: From Budget Alerts to AI-Powered Optimization The Complete Kiro CLI Guide

📌 Wrapping Up

Thank you for reading! I hope this article gave you practical insights and a clearer perspective on the topic.

Was this helpful?

❤️ Like if it added value
🦄 Unicorn if you’re applying it today
💾 Save for your next optimization session
🔄 Share with your team

Follow me for more on:

AWS architecture patterns
FinOps automation
Multi-account strategies
AI-driven DevOps

💡 What’s Next

More deep dives coming soon on cloud operations, GenAI, Agentic-AI, DevOps, and data workflows follow for weekly insights.

🌐 Portfolio & Work

You can explore my full body of work, certifications, architecture projects, and technical articles here:

👉 Visit My Website

🛠️ Services I Offer

If you're looking for hands-on guidance or collaboration, I provide:

Cloud Architecture Consulting (AWS / Azure)
DevSecOps & Automation Design
FinOps Optimization Reviews
Technical Writing (Cloud, DevOps, GenAI)
Product & Architecture Reviews
Mentorship & 1:1 Technical Guidance

🤝 Let’s Connect

I’d love to hear your thoughts drop a comment or connect with me on LinkedIn.

For collaborations, consulting, or technical discussions, feel free to reach out directly at simplynadaf@gmail.com

Happy Learning 🚀

Deploy Web Servers with Terraform: EC2 + Load Balancer Tutorial

Sarvar Nadaf — Mon, 27 Apr 2026 09:40:05 +0000

👋 Hey there, tech enthusiasts!

"A single server is a single point of failure. A load balancer is your insurance policy."

🎯 Welcome Back!

Remember in Article 6 when you built your first VPC with public and private subnets? You created the network foundation, but it was empty—no servers, no applications, nothing running.

Here's the reality: A VPC without compute resources is like building a highway with no cars. You need:

Web servers to run your applications
A way to handle traffic spikes
Protection against server failures
Zero-downtime deployments

That's where EC2 instances and Load Balancers come in.

By the end of this article, you'll:

✅ Deploy EC2 instances with Terraform
✅ Configure security groups for web servers
✅ Use user data to automate server setup
✅ Create an Application Load Balancer (ALB)
✅ Implement health checks and target groups
✅ Build high-availability web infrastructure

Time Required: 45 minutes (20 min read + 25 min practice)

Cost: ~$33/month (~$16 with free tier for ALB)

Difficulty: Intermediate

Let's deploy some real infrastructure! 🚀

💔 The Problem: Single Server Syndrome

The Nightmare Scenario

It's 2 AM. Your phone rings.

Monitoring Alert: Website Down
Status: 503 Service Unavailable
Cause: EC2 instance crashed
Impact: 100% of users affected
Revenue Loss: $500/minute

You scramble to:

SSH into the server (if you can)
Restart the application
Hope it comes back up
Watch users leave your site

The root cause? You had one server running everything.

Common Single-Server Problems:

❌ Traffic Spike: Black Friday hits, server crashes from load

❌ Hardware Failure: AWS instance dies, site goes down

❌ Deployment Risk: Update breaks something, entire site offline

❌ No Redundancy: One point of failure = business risk

❌ Manual Recovery: You're the human load balancer at 2 AM

❌ Poor User Experience: Slow response times, timeouts, errors

Sound familiar? Let's fix this with load balancing.

🌟 What is a Load Balancer? (Quick Theory)

Simple Definition

Load Balancer = Traffic cop for your servers. It:

Distributes incoming requests across multiple servers
Monitors server health automatically
Routes traffic only to healthy servers
Enables zero-downtime deployments

Think of it like this:

Without Load Balancer:

All Users → Single Server → 💥 Overloaded/Crashed

With Load Balancer:

Users → Load Balancer → Server 1 (healthy) ✅
                     → Server 2 (healthy) ✅
                     → Server 3 (unhealthy) ❌ (no traffic)

Why You Need This

Scenario 1: Traffic Spike

Normal: 100 requests/sec → 2 servers handle it easily
Black Friday: 10,000 requests/sec → Load balancer distributes across all servers
Result: Site stays up, users happy

Scenario 2: Server Failure

Server 1 crashes at 2 AM
Load balancer detects failure in 30 seconds
Automatically stops sending traffic to Server 1
Server 2 handles all traffic
Result: You sleep through the night

Scenario 3: Deployment

Deploy new code to Server 1
Load balancer keeps sending traffic to Server 2
Test Server 1, then switch traffic
Result: Zero downtime deployment

📋 Prerequisites

Before starting, make sure you have:

✅ Completed Article 6: Building Your First AWS VPC
✅ Terraform installed (v1.0+)
✅ AWS CLI configured
✅ Basic understanding of VPC and networking
✅ An SSH key pair in AWS (or we'll create one)

🏗️ What We're Building

Internet
    ↓
Application Load Balancer (ALB)
    ↓
    ├─→ EC2 Instance 1 (Apache)
    └─→ EC2 Instance 2 (Apache)

Architecture Components:

VPC - Our isolated network
Public Subnet - Where our resources live
Internet Gateway - Internet access
Security Groups - Firewall rules
EC2 Instances - Web servers running Apache
Application Load Balancer - Traffic distributor
Target Group - Manages server health

📁 Project Structure

Create this folder structure:

07-ec2-load-balancer/
├── main.tf           # Main infrastructure code
├── variables.tf      # Input variables
├── outputs.tf        # Output values
└── terraform.tfvars  # Variable values

Source Code

The complete source code and Terraform configuration used in this article can be found on GitHub:

simplynadaf / terraform-by-sarvar

Terraform Series

🚀 Terraform By Sarvar

Complete Terraform tutorial series from basics to advanced concepts

📖 Read the Series • 🌐 Portfolio • 💼 LinkedIn

📚 Series Overview

This repository contains ONLY Terraform code examples for the Terraform By Sarvar tutorial series.

⚠️ IMPORTANT: This repo contains only .tf files and infrastructure code. Articles are published on dev.to, not stored here.

📖 Read the series on dev.to: https://dev.to/sarvar_04/series/36963

🎯 What You'll Learn

✅ Infrastructure as Code (IaC) fundamentals
✅ Terraform basics to advanced concepts
✅ AWS resource provisioning
✅ Best practices and real-world patterns
✅ Production-ready configurations

📊 Series Progress

📖 Article Series

📘 Foundation (Articles 1-5) ✅ Complete

Introduction to Terraform & IaC - ✅ Published
Installation & Setup - ✅ Published
Your First AWS Resource - ✅ Published
Understanding Terraform State - ✅ Published
Variables and Outputs - ✅ Published

📗 Real Infrastructure (Articles 6-10)

Building a VPC from Scratch…

View on GitHub

Getting Started

Follow these steps to run the project on your local machine:

Clone the repository:

git clone https://github.com/simplynadaf/terraform-by-sarvar.git

Navigate to the project directory:

cd terraform-by-sarvar/articles/08-lambda-serverless

🔧 Step 1: Define Variables

Create variables.tf:

# AWS Region
variable "aws_region" {
  description = "AWS region where resources will be created"
  type        = string
  default     = "us-east-1"
}

# Project Name
variable "project_name" {
  description = "Project name for resource naming"
  type        = string
  default     = "terraform-web"
}

# Environment
variable "environment" {
  description = "Environment name"
  type        = string
  default     = "dev"
}

# VPC CIDR
variable "vpc_cidr" {
  description = "CIDR block for VPC"
  type        = string
  default     = "10.0.0.0/16"
}

# Public Subnet CIDR
variable "public_subnet_cidr" {
  description = "CIDR block for public subnet"
  type        = string
  default     = "10.0.1.0/24"
}

# Instance Type
variable "instance_type" {
  description = "EC2 instance type"
  type        = string
  default     = "t2.micro"
}

# Instance Count
variable "instance_count" {
  description = "Number of EC2 instances"
  type        = number
  default     = 2
}

# SSH Key Name
variable "key_name" {
  description = "SSH key pair name"
  type        = string
  default     = "my-key"
}

# My IP for SSH Access
variable "my_ip" {
  description = "Your IP address for SSH access (CIDR format)"
  type        = string
  default     = "0.0.0.0/0"  # Change this to your IP for security
}

Key Points:

instance_count = 2 - We'll create 2 web servers
instance_type = "t2.micro" - Free tier eligible
my_ip - Restrict SSH access (change from 0.0.0.0/0 in production!)

🌐 Step 2: Create VPC and Networking

Add to main.tf:

# Terraform Configuration
terraform {
  required_version = ">= 1.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

# Provider Configuration
provider "aws" {
  region = var.aws_region
}

# Data source: Get latest Amazon Linux 2023 AMI
data "aws_ami" "amazon_linux" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["al2023-ami-*-x86_64"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

# Data source: Get available availability zones
data "aws_availability_zones" "available" {
  state = "available"
}

# VPC
resource "aws_vpc" "main" {
  cidr_block           = var.vpc_cidr
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name        = "${var.project_name}-vpc"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

# Internet Gateway
resource "aws_internet_gateway" "main" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name        = "${var.project_name}-igw"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

# Public Subnet
resource "aws_subnet" "public" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = var.public_subnet_cidr
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true

  tags = {
    Name        = "${var.project_name}-public-subnet"
    Environment = var.environment
    Type        = "Public"
    ManagedBy   = "Terraform"
  }
}

# Public Route Table

# Public Subnet 2 (Different AZ for ALB)
resource "aws_subnet" "public_2" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.0.2.0/24"
  availability_zone       = data.aws_availability_zones.available.names[1]
  map_public_ip_on_launch = true

  tags = {
    Name        = "${var.project_name}-public-subnet-2"
    Environment = var.environment
    Type        = "Public"
    ManagedBy   = "Terraform"
  }
}
resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.main.id
  }

  tags = {
    Name        = "${var.project_name}-public-rt"
    Environment = var.environment
    Type        = "Public"
    ManagedBy   = "Terraform"
  }
}

# Route Table Association
resource "aws_route_table_association" "public" {
  subnet_id      = aws_subnet.public.id
  route_table_id = aws_route_table.public.id
}

# Route Table Association for Public Subnet 2
resource "aws_route_table_association" "public_2" {
  subnet_id      = aws_subnet.public_2.id
  route_table_id = aws_route_table.public.id
}

New Concept: Data Sources

data "aws_ami" "amazon_linux" {
  most_recent = true
  owners      = ["amazon"]
  # ...
}

Data sources query existing AWS resources. Here we're finding the latest Amazon Linux AMI automatically - no need to hardcode AMI IDs!

🔒 Step 3: Configure Security Groups

Security groups are like firewalls. Add to main.tf:

# Security Group for ALB
resource "aws_security_group" "alb" {
  name        = "${var.project_name}-alb-sg"
  description = "Security group for Application Load Balancer"
  vpc_id      = aws_vpc.main.id

  ingress {
    description = "HTTP from anywhere"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    description = "Allow all outbound traffic"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name        = "${var.project_name}-alb-sg"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

# Security Group for EC2 Instances
resource "aws_security_group" "instance" {
  name        = "${var.project_name}-instance-sg"
  description = "Security group for EC2 instances"
  vpc_id      = aws_vpc.main.id

  ingress {
    description     = "HTTP from ALB"
    from_port       = 80
    to_port         = 80
    protocol        = "tcp"
    security_groups = [aws_security_group.alb.id]
  }

  ingress {
    description = "SSH from my IP"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = [var.my_ip]
  }

  egress {
    description = "Allow all outbound traffic"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name        = "${var.project_name}-instance-sg"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

Security Architecture:

Internet → ALB (Port 80) → EC2 Instances (Port 80)
                            EC2 Instances (Port 22 from your IP)

Key Security Practices:

ALB accepts HTTP (80) from anywhere
EC2 instances only accept HTTP from ALB (not directly from internet!)
SSH (22) only from your IP address
This is called "defense in depth"

🖥️ Step 4: Deploy EC2 Instances

Now for the exciting part - creating web servers! Add to main.tf:

# EC2 Instances
resource "aws_instance" "web" {
  count = var.instance_count

  ami           = data.aws_ami.amazon_linux.id
  instance_type = var.instance_type
  key_name      = var.key_name
  subnet_id     = aws_subnet.public.id

  vpc_security_group_ids = [aws_security_group.instance.id]

  user_data = base64encode(<<-EOF
    #!/bin/bash
    dnf update -y
    dnf install -y httpd
    systemctl start httpd
    systemctl enable httpd
    echo "<h1>Terraform Web Server - Instance: $(ec2-metadata --instance-id | cut -d' ' -f2)</h1><p>AZ: $(ec2-metadata --availability-zone | cut -d' ' -f2)</p>" > /var/www/html/index.html
    EOF
  )

  tags = {
    Name        = "${var.project_name}-web-${count.index + 1}"
    Environment = var.environment
    ManagedBy   = "Terraform"
    Server      = "web-${count.index + 1}"
  }
}

Breaking Down the Code:

1. Count Meta-Argument:

count = var.instance_count

Creates multiple instances. With instance_count = 2, we get 2 servers!

2. User Data - The Magic:

user_data = <<-EOF
  #!/bin/bash
  yum update -y
  yum install -y httpd
  # ...
EOF

User data runs automatically when the instance starts. It:

Updates the system
Installs Apache web server
Starts Apache
Creates a custom HTML page

3. Dynamic Naming:

Name = "${var.project_name}-web-${count.index + 1}"

First instance: terraform-web-web-1
Second instance: terraform-web-web-2

⚖️ Step 5: Create Application Load Balancer

The load balancer distributes traffic. Add to main.tf:

# Application Load Balancer
resource "aws_lb" "main" {
  name               = "${var.project_name}-alb"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.alb.id]
  subnets            = [aws_subnet.public.id, aws_subnet.public_2.id]

  enable_deletion_protection = false

  tags = {
    Name        = "${var.project_name}-alb"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

# Target Group
resource "aws_lb_target_group" "main" {
  name     = "${var.project_name}-tg"
  port     = 80
  protocol = "HTTP"
  vpc_id   = aws_vpc.main.id

  health_check {
    enabled             = true
    healthy_threshold   = 2
    unhealthy_threshold = 2
    timeout             = 5
    interval            = 30
    path                = "/"
    protocol            = "HTTP"
    matcher             = "200"
  }

  tags = {
    Name        = "${var.project_name}-tg"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

# Target Group Attachment
resource "aws_lb_target_group_attachment" "main" {
  count = var.instance_count

  target_group_arn = aws_lb_target_group.main.arn
  target_id        = aws_instance.web[count.index].id
  port             = 80
}

# ALB Listener
resource "aws_lb_listener" "main" {
  load_balancer_arn = aws_lb.main.arn
  port              = "80"
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.main.arn
  }

  tags = {
    Name        = "${var.project_name}-listener"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

Understanding Load Balancer Components:

1. Application Load Balancer (ALB):

The main load balancer resource
internal = false - Internet-facing
load_balancer_type = "application" - Layer 7 (HTTP/HTTPS)

2. Target Group:

Manages the backend servers
Performs health checks every 30 seconds
Marks servers healthy after 2 successful checks
Marks servers unhealthy after 2 failed checks

3. Target Group Attachment:

count = var.instance_count
target_id = aws_instance.web[count.index].id

Registers each EC2 instance with the target group.

4. Listener:

Listens on port 80
Forwards traffic to the target group

Health Check Flow:

ALB → Checks "/" every 30s → Expects HTTP 200 → 
  ✅ Healthy (2 successes) → Receives traffic
  ❌ Unhealthy (2 failures) → No traffic

📤 Step 6: Define Outputs

Create outputs.tf:

# VPC Outputs
output "vpc_id" {
  description = "ID of the VPC"
  value       = aws_vpc.main.id
}

# Subnet Output
output "public_subnet_id" {
  description = "ID of public subnet"
  value       = aws_subnet.public.id
}

# EC2 Instance Outputs
output "instance_ids" {
  description = "IDs of EC2 instances"
  value       = aws_instance.web[*].id
}

output "instance_public_ips" {
  description = "Public IPs of EC2 instances"
  value       = aws_instance.web[*].public_ip
}

# Load Balancer Outputs
output "alb_dns_name" {
  description = "DNS name of the Application Load Balancer"
  value       = aws_lb.main.dns_name
}

output "alb_url" {
  description = "URL to access the load balancer"
  value       = "http://${aws_lb.main.dns_name}"
}

# Security Group Outputs
output "alb_security_group_id" {
  description = "ID of ALB security group"
  value       = aws_security_group.alb.id
}

output "instance_security_group_id" {
  description = "ID of instance security group"
  value       = aws_security_group.instance.id
}

Splat Expression:

aws_instance.web[*].id

The [*] gets ALL instance IDs as a list. Super useful with count!

⚙️ Step 7: Set Variable Values

Create terraform.tfvars:

aws_region = "us-east-1"
project_name = "terraform-web"
environment = "dev"

# Network Configuration
vpc_cidr = "10.0.0.0/16"
public_subnet_cidr = "10.0.1.0/24"

# EC2 Configuration
instance_type = "t2.micro"
instance_count = 2
key_name = "my-key"

# Security - Change this to your IP address
my_ip = "0.0.0.0/0"

⚠️ Important: Change my_ip to your actual IP address for security!

Find your IP:

curl ifconfig.me

Then update:

my_ip = "203.0.113.0/32"  # Your IP

🚀 Step 8: Deploy the Infrastructure

1. Create SSH Key Pair (if you don't have one)

# Create key pair in AWS
aws ec2 create-key-pair \
  --key-name my-key \
  --query 'KeyMaterial' \
  --output text > my-key.pem

![ ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ttadkcyqnuxbsnpwdqs7.png)


# Set permissions
chmod 400 my-key.pem

2. Initialize Terraform

terraform init

Expected output:

Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 5.0"...
- Installing hashicorp/aws v5.100.0...

Terraform has been successfully initialized!

3. Review the Plan

terraform plan

You should see:

Plan: 14 to add, 0 to change, 0 to destroy.

Resources being created:

1 VPC
1 Internet Gateway
1 Subnet
1 Route Table + Association
2 Security Groups
2 EC2 Instances
1 Application Load Balancer
1 Target Group
2 Target Group Attachments
1 ALB Listener

4. Apply the Configuration

terraform apply

Type yes when prompted.

This will take 3-5 minutes because:

EC2 instances need to boot
User data script runs
ALB needs to provision
Health checks need to pass

✅ Step 9: Test Your Infrastructure

1. Get the Load Balancer URL

terraform output alb_url

Output:

http://terraform-web-alb-1740709428.us-east-1.elb.amazonaws.com

2. Test in Browser

Open the URL in your browser. You should see:

🚀 Terraform Web Server
Deployed with Infrastructure as Code

Instance ID: i-09305fc7d4638360a
Availability Zone: us-east-1a
Server: 1

3. Test Load Balancing

Refresh the page multiple times. You'll see the Instance ID and Server number change - that's the load balancer distributing traffic!

First request  → Server: 1
Second request → Server: 2
Third request  → Server: 1

4. Test Individual Instances

# Get instance IPs
terraform output instance_public_ips

![ ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jbibepofm64vdjwyk9kj.png)


# Test directly (should work)
curl http://<instance-ip>

🔍 Understanding What Happened

Traffic Flow

User Request
    ↓
ALB DNS (terraform-web-alb-xxx.elb.amazonaws.com)
    ↓
ALB Listener (Port 80)
    ↓
Target Group (Health Check: ✅)
    ↓
Round-Robin Distribution
    ├─→ EC2 Instance 1 (Apache) → Response
    └─→ EC2 Instance 2 (Apache) → Response

Health Check Process

Every 30 seconds:
ALB → GET / → EC2 Instance
    ↓
HTTP 200 OK?
    ├─ Yes (2 times) → Mark Healthy → Send Traffic
    └─ No (2 times)  → Mark Unhealthy → Stop Traffic

🧪 Advanced Testing

Test High Availability

Scenario: What if one server fails?

# Get instance IDs
terraform output instance_ids

# Stop one instance
aws ec2 stop-instances --instance-ids i-xxxxx

# Wait 1 minute for health check to fail

# Test the ALB URL - still works!
curl http://<alb-dns>

The ALB automatically stops sending traffic to the unhealthy instance!

Monitor Health Status

# Check target health
aws elbv2 describe-target-health \
  --target-group-arn <target-group-arn>

Output:

{
  "TargetHealthDescriptions": [
    {
      "Target": {
        "Id": "i-xxxxx",
        "Port": 80
      },
      "HealthCheckPort": "80",
      "TargetHealth": {
        "State": "healthy"
      }
    }
  ]
}

🐛 Troubleshooting Common Issues

Issue 1: "InvalidKeyPair.NotFound"

Error:

Error: creating EC2 Instance: InvalidKeyPair.NotFound

Solution:

# List your key pairs
aws ec2 describe-key-pairs

# Create if missing
aws ec2 create-key-pair --key-name my-key \
  --query 'KeyMaterial' --output text > my-key.pem
chmod 400 my-key.pem

# Update terraform.tfvars with correct key name

Issue 2: ALB Shows "503 Service Unavailable"

Cause: Instances are unhealthy or still booting.

Solution:

# Wait 2-3 minutes for:
# 1. Instances to boot
# 2. User data to complete
# 3. Health checks to pass

# Check target health
aws elbv2 describe-target-health \
  --target-group-arn $(terraform output -raw target_group_arn)

Issue 3: Can't SSH to Instances

Cause: Security group blocks your IP.

Solution:

# Get your current IP
curl ifconfig.me

# Update terraform.tfvars
my_ip = "YOUR_IP/32"

# Apply changes
terraform apply

Issue 4: "Subnet must have at least 2 availability zones"

Error with ALB:

Error: creating ELBv2 Load Balancer: ValidationError

Solution: ALBs require at least 2 subnets in different AZs. Update main.tf:

# Add second subnet
resource "aws_subnet" "public_2" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.0.2.0/24"
  availability_zone       = data.aws_availability_zones.available.names[1]
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.project_name}-public-subnet-2"
  }
}

# Update ALB subnets
resource "aws_lb" "main" {
  # ...
  subnets = [aws_subnet.public.id, aws_subnet.public_2.id]
}

💰 Cost Breakdown

Monthly Costs (us-east-1):

Resource	Quantity	Cost/Month	Total
t2.micro EC2	2	$8.50	$17.00
Application Load Balancer	1	$16.20	$16.20
Data Transfer (1GB)	-	$0.09	$0.09
Total			~$33.29

Free Tier Benefits:

First 750 hours/month of t2.micro (covers both instances!)
First 15 GB data transfer out
Actual cost with free tier: ~$16.20/month (just the ALB)

Cost Optimization Tips:

Use t2.micro (free tier eligible)
Delete resources when not in use
Use Network Load Balancer ($10.95/month) if you don't need Layer 7 features
Consider using Auto Scaling (covered in future articles)

🧹 Cleanup

Important: Don't forget to destroy resources to avoid charges!

# Destroy everything
terraform destroy

# Type 'yes' when prompted

This will delete:

✅ Load Balancer
✅ Target Group
✅ EC2 Instances
✅ Security Groups
✅ VPC and networking

Verify deletion:

# Check EC2 instances
aws ec2 describe-instances --filters "Name=tag:ManagedBy,Values=Terraform"

# Check load balancers
aws elbv2 describe-load-balancers

📚 Key Concepts Learned

1. Count Meta-Argument

count = 2

Creates multiple identical resources. Access with [count.index].

2. Data Sources

data "aws_ami" "amazon_linux" { }

Query existing AWS resources instead of hardcoding values.

3. User Data

user_data = <<-EOF
  #!/bin/bash
  # Bootstrap script
EOF

Automate instance configuration at launch.

4. Security Group References

security_groups = [aws_security_group.alb.id]

Reference one security group from another for layered security.

5. Splat Expressions

aws_instance.web[*].id

Get all values from resources created with count.

6. Health Checks

health_check {
  path     = "/"
  interval = 30
}

Automatic monitoring and traffic routing.

🎯 Real-World Applications

This architecture is used for:

1. Web Applications

WordPress sites
E-commerce platforms
SaaS applications

2. API Backends

REST APIs
GraphQL servers
Microservices

3. Content Delivery

Static websites
Media streaming
File downloads

4. Development Environments

Staging servers
Testing environments
Demo applications

🚀 What's Next?

In the next article, we'll add:

RDS Database - Persistent data storage
AWS Secrets Manager - Secure credential management
Private Subnets - Enhanced security
Database Connection - Connect EC2 to RDS

Coming Up: Article 8: Secure Database Deployment: RDS + Secrets Manager with Terraform

📖 Additional Resources

Official Documentation:

Related Articles:

💬 Questions?

Common Questions:

Q: Can I use different instance types for each server?

A: Yes! Use for_each instead of count for more flexibility (covered in Article 13).

Q: How do I add HTTPS/SSL?

A: You'll need an ACM certificate and update the listener to port 443. We'll cover this in a future article.

Q: Can I deploy to multiple regions?

A: Yes! Use Terraform workspaces or separate state files per region.

Q: How do I add auto-scaling?

A: Replace EC2 instances with an Auto Scaling Group. We'll cover this in Article 15.

✅ Summary

Today you learned how to:

✅ Deploy multiple EC2 instances with Terraform
✅ Use count to create multiple resources
✅ Configure security groups for layered security
✅ Automate server setup with user data
✅ Create an Application Load Balancer
✅ Implement health checks and high availability
✅ Test load balancing in action

You now have production-ready web infrastructure! 🎉

📌 Wrapping Up

Thank you for reading. I hope this article provided practical insights and a clearer understanding of the topic.

If you found this useful:

❤️ Like if it added value
🦄 Unicorn if you’re applying it today
💾 Save it for your next optimization session
🔄 Share it with your team

💡 What’s Next

More deep dives are coming soon on:

Cloud Operations
GenAI & Agentic AI
DevOps Automation
Data & Platform Engineering

Follow along for weekly insights and hands-on guides.

🌐 Portfolio & Work

You can explore my full body of work, certifications, architecture projects, and technical articles here:

👉 Visit My Website

🛠️ Services I Offer

If you're looking for hands-on guidance or collaboration, I provide:

Cloud Architecture Consulting (AWS / Azure)
DevSecOps & Automation Design
FinOps Optimization Reviews
Technical Writing (Cloud, DevOps, GenAI)
Product & Architecture Reviews
Mentorship & 1:1 Technical Guidance

🤝 Let’s Connect

I’d love to hear your thoughts. Feel free to drop a comment or connect with me on:

🔗 LinkedIn

For collaborations, consulting, or technical discussions, reach out at:

📧 simplynadaf@gmail.com

Found this helpful? Share it with your team.

⭐ Star the repo • 📖 Follow the series • 💬 Ask questions

Made by Sarvar Nadaf

🌐 https://sarvarnadaf.com

Stop Giving AI Agents AWS Credentials: A Better Way to Secure Access

Sarvar Nadaf — Mon, 20 Apr 2026 14:00:16 +0000

👋 Hey there, tech enthusiasts!

Let's dive in and explore the fascinating world of cloud technology together! 🚀

The Wake-Up Call

Three months ago, our security team flagged something concerning. Developers were feeding production logs, error messages, and configuration snippets to ChatGPT for debugging help.

The problem? Those logs contained customer identifiers, internal service names, and architectural details we definitely didn't want leaving our network.

We couldn't just block ChatGPT - developers needed AI assistance. The productivity gains were real. But we also couldn't keep hemorrhaging sensitive data to external APIs.

The requirements were clear:

AI agents need AWS access for legitimate automation tasks
Zero sensitive data leaves our AWS environment
Every action must be auditable
Principle of least privilege, always
No impact on developer velocity

That's when I started looking at Model Context Protocol (MCP) as a security boundary.

Understanding MCP as a Security Layer

Before diving into implementation, let's clarify what MCP actually does and why it matters for security.

Model Context Protocol is an open standard that sits between your AI agent and your resources. Think of it as a translator and gatekeeper combined.

Developer → AI Agent → MCP Server → AWS IAM → AWS Resources
                          ↓
                    Security Layer

The MCP server doesn't just pass requests through. It acts as a security boundary that:

Validates every request before execution
Translates AI intentions into specific AWS API calls
Enforces authentication and authorization
Logs everything for audit trails
Provides a single point of control

Why this matters: Instead of giving AI agents direct AWS credentials, you give them access to an MCP server that has carefully scoped permissions. The AI never touches AWS credentials. It doesn't even know they exist.

The Security Architecture

After several iterations, here's the pattern that survived production. I'll explain the thinking behind each layer.

Layer 1: Authentication Without Permanent Credentials

The first principle: no permanent credentials anywhere in the system.

Developers authenticate with our existing identity provider (Okta in our case). The identity provider issues a JWT token containing the user's identity and group memberships. The MCP server validates this JWT and issues a short-lived session token - 15 minutes, no exceptions.

Why 15 minutes? Long enough for a debugging session, short enough that a leaked token becomes useless quickly. If someone steals a session token, they have a 15-minute window at most. Compare that to permanent AWS credentials that work forever until manually revoked.

The MCP server never stores these tokens. They're validated, used, and discarded. When they expire, users re-authenticate. It's a minor inconvenience that prevents major security incidents.

Layer 2: Request Validation

This is where MCP shines as a security boundary. Every request goes through multiple validation checks:

Action Allowlist: The MCP server maintains a strict list of allowed AWS actions. If the AI requests something not on the list, it's blocked immediately. No wildcards, no "just in case" permissions.

Pattern Detection: I scan every request for dangerous patterns. Words like "delete", "terminate", "destroy" trigger additional scrutiny. Even if the action is technically allowed, suspicious patterns can block the request or require additional approval.

Parameter Sanitization: Before logging or processing, all sensitive parameters get redacted. Passwords, tokens, API keys - anything that looks like a credential gets replaced with [REDACTED] in logs. This prevents credential leakage through audit trails.

Rate Limiting: Each user gets a request budget. Exceed it, and requests start getting throttled. This prevents both accidental runaway scripts and intentional abuse.

The validation happens in milliseconds. Developers don't notice the overhead, but it's the difference between a secure system and a disaster waiting to happen.

Layer 3: AWS Execution with Scoped Permissions

The MCP server uses an IAM role with specific permissions. Not admin. Not power user. Just what's needed for legitimate use cases.

I started by listing every legitimate use case developers had:

Read CloudWatch logs for debugging
List S3 buckets to find data
Get objects from specific buckets
Query CloudWatch metrics for dashboards

Then I created IAM policies that allow exactly those actions and nothing else.

The key insight: Explicit denies for dangerous actions, even if they're not in the allow list. This protects against future policy changes or misconfigurations.

Example: Even if someone accidentally adds s3:* to the allow list, an explicit deny on s3:DeleteBucket still blocks it. Defense in depth.

Layer 4: Comprehensive Audit Trail

CloudTrail logs every AWS API call, but it doesn't capture the context we need. Who made the request? What was the AI prompt? What resources were accessed?

I built a custom logging layer that captures:

User identity (email, not just IAM role)
Original AI prompt (hashed, not stored in plain text)
AWS action requested
Resources accessed
Result (success/failure)
Execution time

All of this goes to CloudWatch Logs in structured JSON format. Now I can query: "Show me all S3 access by user X this week" or "What resources did the AI access when processing this prompt?"

The logs are immutable and retained for 90 days for compliance.

How We Built It

The deployment came down to three critical security decisions. Each one was driven by a specific threat we wanted to prevent.

Decision 1: Network Isolation Over Convenience

I put the MCP server in a completely separate VPC from production. No shared networks, no VPC peering, nothing. The only communication path is through VPC endpoints to AWS APIs.

Why this matters: If someone compromises the MCP server, they're trapped. No internet access means they can't exfiltrate data. No production VPC access means they can't pivot to other systems. They're stuck in a cage that only opens to specific AWS services.

I chose ECS Fargate because it gave me this isolation without the overhead of managing EC2 instances. No patching, no scaling configuration, just containers in a locked-down network.

The trade-off: More complex networking setup. But the security benefit was worth it. A compromised MCP server becomes useless to an attacker.

Decision 2: Explicit Denies as the Last Line of Defense

The IAM policy has two blocks: allows and denies. The allows are specific - exact actions on exact resources. But the denies are what keep me sleeping at night.

I explicitly deny all delete operations, all terminate operations, all IAM changes, and all KMS key operations. Even if someone misconfigures the allow block and adds s3:*, the deny on s3:DeleteBucket still holds.

Why this matters: Policies get changed. People make mistakes. The deny block is the safety net that catches those mistakes before they become incidents.

The trade-off: More rigid system. If we need to add a delete operation later, we have to modify both blocks. But that friction is intentional - it forces us to think twice about dangerous permissions.

Decision 3: Real-Time Alerting Over Post-Incident Analysis

I set up CloudWatch alarms that fire immediately when something looks wrong. High error rates, unusual request volumes, spikes in blocked actions - all trigger alerts to our security team's Slack channel.

Why this matters: Logs are great for forensics, but alerts prevent incidents. If the AI starts trying malicious actions, I want to know in real-time, not during next week's log review.

The alerts are tuned to avoid noise. More than 50 errors in 5 minutes is abnormal. More than 1,000 requests from one user in 5 minutes is suspicious. These thresholds came from watching normal usage patterns for a month.

The trade-off: Alert fatigue is real. We tune the thresholds monthly based on false positive rates. But I'd rather investigate a false alarm than miss a real attack.

What Broke (And How I Fixed It)

Issue 1: Permission Errors Everywhere

What happened: First deployment, every request failed with AccessDenied.

The problem: I was too restrictive. The IAM policy only allowed specific S3 buckets, but developers needed to list buckets first to know what existed.

The fix: Add s3:ListAllMyBuckets with a wildcard resource. Let them see what exists, but control what they can read. It's like letting someone see the library catalog without giving them keys to every book.

Lesson: Start with read-only list permissions, then restrict data access. Users need to discover resources before they can use them.

Issue 2: CloudTrail Logs Were Useless

What happened: CloudTrail showed the MCP server's actions, but not which user requested them.

The problem: All requests came from the same IAM role. No way to trace back to individual users.

The fix: Pass user context through custom CloudWatch Logs. Every MCP request gets logged with the user's email, the action requested, and the resources accessed. Now I can trace every action back to the person who requested it.

Lesson: CloudTrail alone isn't enough for multi-user systems. You need custom logging to capture user context.

Issue 3: AI Agents Tried Creative Exploits

What happened: The AI tried to chain commands to bypass restrictions.

Example request:

"First list the S3 buckets, then for each bucket, 
download all objects and search for passwords"

The problem: My validation checked individual actions, not sequences. The AI was trying to automate a multi-step attack.

The fix: Detect and block chaining attempts. Look for words like "then", "after that", "for each", "loop through". Force users to make explicit, separate requests for each action.

Lesson: AI agents are creative. They'll try to work around restrictions. You need to think like an attacker.

Issue 4: Rate Limiting Was Too Aggressive

What happened: Legitimate users hit rate limits during normal debugging sessions.

The problem: I set limits too low (10 requests per minute). Debugging often requires rapid iteration - check logs, adjust query, check again.

The fix: Tiered rate limits based on action type:

Read operations (Get, Describe): 100 requests per 5 minutes
List operations: 50 requests per 5 minutes
Write operations: 10 requests per 5 minutes

Read operations get higher limits because they're lower risk. Write operations stay restricted.

Lesson: One-size-fits-all rate limits don't work. Different actions have different risk profiles.

What I Learned

After three months in production, here's what actually matters:

1. Explicit Denies Are Your Friend

Don't rely on "not allowing" something. Explicitly deny dangerous actions. Even if someone misconfigures the allow rules, the denies hold.

I have explicit denies for:

All delete operations
All terminate operations
All IAM operations
All KMS key operations

These are the "break glass" protections. They prevent catastrophic mistakes.

2. Log Everything, But Make It Searchable

CloudTrail is great, but you need custom logs for MCP-specific context. I send everything to CloudWatch Logs with structured JSON.

Now I can query: "Show me all S3 access by user X in the last hour" or "What resources did the AI access when processing this prompt?"

The logs are immutable and retained for 90 days. If something goes wrong, I can reconstruct exactly what happened.

3. Sanitize Everything

Never log the actual AI prompts. They might contain sensitive data. I hash them instead.

You can still correlate requests (same hash = same prompt), but you're not storing potentially sensitive prompts in logs.

4. Network Isolation Matters

The MCP server runs in a private VPC with no internet access. It can only reach:

AWS API endpoints (via VPC endpoints)
Internal authentication service
CloudWatch Logs

If someone compromises the MCP server, they can't exfiltrate data. They're stuck in an isolated network.

5. Test Your Security Controls

I wrote tests to verify the security controls actually work. Tests like:

Verify delete operations are blocked
Verify IAM operations are blocked
Verify rate limits work
Verify audit logs capture user context

Run these tests in CI/CD. If they pass, your security controls are working. If they fail, you know immediately.

Alternative Approaches I Considered

Option 1: Direct IAM Roles for AI Agents

Pros:

Simpler architecture
No MCP server to maintain
Lower latency

Cons:

No request validation layer
Can't block dangerous patterns
Harder to audit user actions
AI has direct AWS credentials

Why I didn't use it: Too risky. One prompt injection and the AI could delete production resources. The MCP layer provides defense in depth.

Option 2: AWS Lambda as MCP Server

Pros:

Serverless, no infrastructure
Automatic scaling
Pay per request

Cons:

Cold starts (500ms+)
15-minute timeout limit
Harder to maintain state (rate limiting)
More complex networking

Why I didn't use it: Cold starts killed the developer experience. Waiting 500ms for every request was frustrating. Fargate has no cold starts.

Option 3: API Gateway + Lambda

Pros:

Built-in rate limiting
API key management
Request/response transformation

Cons:

More complex setup
Higher cost at scale
Still has Lambda cold starts
Overkill for internal use

Why I didn't use it: The built-in rate limiting was nice, but not worth the complexity for an internal tool. Fargate + ALB was simpler.

Best Practices That Actually Matter

1. Start With Read-Only

Deploy with read-only permissions first. Let developers use it for a week. Then gradually add write permissions based on actual needs.

This prevents over-permissioning. You'll discover what developers actually need, not what they think they need.

2. Use Separate AWS Accounts

Run the MCP server in a separate AWS account from your production workloads. Use cross-account roles for access.

If the MCP account is compromised, production is still isolated. It's an extra layer of defense.

3. Monitor for Anomalies

Set up CloudWatch alarms for unusual patterns:

High error rates (>50 errors in 5 minutes)
Unusual access patterns (>1,000 requests in 5 minutes)
Blocked actions (>100 blocks in 5 minutes)

These alerts go to your security team. Response time is critical.

4. Regular Security Reviews

Every month, review:

Which actions are being used most
Which permissions are never used (remove them)
Any blocked requests (are they legitimate needs?)
Rate limit effectiveness

Security isn't set-and-forget. It requires ongoing attention.

5. Document Everything

Create a runbook for common scenarios:

How to add a new allowed action
How to investigate suspicious activity
How to rotate credentials
How to handle a security incident

When something goes wrong at 2 AM, you'll be glad you documented it.

Summary

Three months in production taught me that securing AI agent access isn't about perfect security - it's about making attacks harder than they're worth while keeping developers productive.

The MCP pattern works because it gives you a single point of control. You're not trying to secure the AI agent itself. You're securing the gateway it uses to access your resources. That gateway validates every request, enforces least privilege, logs everything, and runs in an isolated network.

We went from developers sending production data to ChatGPT to having a secure, auditable system where AI agents help without creating risk. The benefit? No more 2 AM calls about data leaks.

Is it perfect? No. Can a determined attacker find ways around it? Probably. But it's dramatically better than the alternatives: giving AI agents direct AWS credentials or blocking AI tools entirely and watching developers find workarounds.

The key insight: Security is not about building walls. It's about building gates with guards. MCP is that gate.

"Security is not about building walls. It's about building gates with guards."

📌 Wrapping Up

Thank you for reading! I hope this gave you practical ideas for securing AI agent access in your environment.

Found this useful?

❤️ Like if it helped you think through your security approach
🦄 Unicorn if you're implementing this pattern
💾 Save for your next security review
🔄 Share with your security team

Follow me for more on:

AWS security patterns
AI/ML infrastructure
Cloud architecture
DevSecOps practices

💡 What's Next

I'm working on a follow-up article about monitoring and alerting for MCP deployments. Follow for updates.

Also exploring: Multi-region MCP deployments and disaster recovery patterns.

🌐 Portfolio & Work

Explore my full body of work, certifications, and architecture projects:

👉 Visit My Website

🛠️ Services I Offer

Looking for hands-on guidance with cloud security or AI infrastructure?

Cloud Security Architecture (AWS / Azure)
AI/ML Infrastructure Design
Security Audit & Remediation
Technical Writing & Documentation
Architecture Reviews
1:1 Technical Mentorship

🤝 Let's Connect

Questions about implementing this pattern? Drop a comment or connect with me on LinkedIn.

For consulting or technical discussions: simplynadaf@gmail.com

Stay secure! 🔒

Amazon S3 Files: The Game Changer We've Been Waiting For

Sarvar Nadaf — Tue, 14 Apr 2026 14:09:19 +0000

👋 Hey there, tech enthusiasts!

Let's dive in and explore the fascinating world of cloud technology together! 🚀

After working with AWS for over a decade, I've spent countless hours explaining to clients why they can't just "edit a file in S3" the way they would on their local computer. I've drawn diagrams, created analogies, and watched developers struggle with the limitations of object storage versus file systems. Well, that conversation just got a lot more interesting.

In late 2025, AWS released Amazon S3 Files, and honestly, it's one of those features that makes you wonder why it took so long. Let me walk you through what it is, why it matters, and when you should (and shouldn't) use it.

TL;DR: S3 Files is a true POSIX-compliant file system for S3 buckets. Unlike S3 FUSE/Mountpoint, it provides full file system semantics with sub-millisecond latency. Best for: ML training, AI agents, data analytics, shared development environments.

The Problem S3 Files Solves

Let me start with a story that probably sounds familiar. Last year, I worked with a machine learning team training large language models. Their workflow looked like this:

Store training datasets in S3 (hundreds of gigabytes of text data)
Copy datasets to an EBS volume attached to their GPU instances
Train the model, saving checkpoints every hour
Copy checkpoints back to S3 for durability
When spot instances get interrupted, restart everything

This workflow had several problems:

Data duplication: Paying for storage twice (S3 + EBS on every instance)
Time waste: 30-45 minutes copying data before training could start
Complexity: Custom scripts to sync checkpoints and handle interruptions
Cost: Running 2TB EBS volumes on multiple GPU instances 24/7
Spot instance pain: Every interruption meant re-copying everything

They tried using S3 FUSE and Mountpoint S3 to mount their S3 bucket directly, but ran into a wall. Imagine training a model for 6 hours, then having the checkpoint save fail because the training framework needs to update the checkpoint metadata a basic file operation that S3 FUSE simply can't handle. The training framework would write 95% of the checkpoint file, then fail when it tried to seek back to update the header. Why? Because S3 FUSE isn't a real file system it's just an API wrapper pretending to be one.

This is exactly the problem S3 Files solves.

What Exactly Is S3 Files?

Amazon S3 Files is a true, POSIX-compliant file system that sits on top of your S3 buckets. Think of it as a high-performance bridge between your compute resources (EC2, Lambda, ECS, EKS) and your S3 data a smart caching layer that speaks both "file system" and "S3 object" fluently.

Here's what makes it different from previous solutions:

S3 Files vs. The Old Ways

S3 FUSE / Mountpoint S3:

API wrappers that translate file operations into S3 API calls
Limited file system semantics
Can't handle operations like seeking backward in files
No true concurrent write support

S3 Files:

Built on Amazon Elastic File System (EFS) technology
True POSIX compliance (supports all NFS v4.1+ operations)
Sub-millisecond latency for cached data
Full file system semantics (create, read, update, delete, seek, append)
Concurrent access from multiple compute resources

How S3 Files Actually Works

The architecture is clever. When you create an S3 file system and mount it to your EC2 instance, here's what happens:

Initial Mount: You see your S3 bucket as a directory structure. No data is copied yet.
Lazy Loading: When you access a file, only that file's metadata and content are loaded into a high-performance cache layer (built on EFS).
Smart Caching:
- Frequently accessed files stay in the cache (sub-millisecond access)
- Large sequential reads go directly from S3 (maximizing throughput)
- Only requested byte ranges are transferred (minimizing costs)
Write Operations:
- Writes go to the cache first (fast)
- Changes sync back to S3 automatically within minutes
- You get immediate file system consistency
Bidirectional Sync:
- Changes in the file system appear in S3 within minutes
- Changes in S3 appear in the file system within seconds

Real-World Example: AI Model Training Pipeline

Let me show you a practical example. I recently helped an AI research team migrate their model training pipeline to use S3 Files.

The Scenario

They train large language models with:

500GB training datasets (tokenized text)
Hourly checkpoint saves (10-50GB each)
Multi-GPU distributed training
Spot instances for cost optimization

The Old Architecture

S3 Dataset → Copy to EBS → Train Model → Save Checkpoint to EBS → Sync to S3 → Spot Interruption → Repeat

Problems:

30-45 minutes copying data before training starts
$1,200/month in EBS costs across GPU instances
Complex checkpoint sync logic
Spot interruptions meant starting over with data copies

The New Architecture with S3 Files

S3 Dataset → Mount S3 Files → Train Model → Checkpoints auto-sync to S3 → Spot Interruption → Remount & Resume

The Results

With S3 Files, they mount their S3 bucket directly to their GPU instances. Training frameworks like PyTorch can now write checkpoints with full file system semantics updating headers, seeking to specific positions, and appending data operations that previously failed with S3 FUSE.

Training startup: Reduced from 45 minutes to 2 minutes
Cost savings: $1,200/month in EBS costs eliminated
Spot instance recovery: From 45 minutes to 5 minutes (just remount and resume)
Simplified code: Removed 300+ lines of checkpoint sync and recovery logic
Better reliability: No more corrupted checkpoints or failed saves

Use Cases Where S3 Files Shines

Now that you understand how it works, let's look at where S3 Files makes the biggest impact.

Based on my experience and conversations with other architects, here are the scenarios where S3 Files is a perfect fit:

1. Machine Learning Training

Scenario: Training models on large datasets stored in S3

Before S3 Files:

Copy entire dataset to EBS/EFS before training
Wait hours for data transfer
Pay for duplicate storage

With S3 Files:

Mount S3 bucket directly
Training starts immediately (lazy loading)
Only accessed data is cached
Multiple training instances can share the same data

2. AI Agents and Automation

Scenario: AI agents using Python libraries and CLI tools that expect file systems

The Challenge: Many AI agents and automation tools are built with traditional file system operations in mind. Rewriting them to use S3 APIs directly would require significant code changes.

With S3 Files: Your existing code that expects file system access works without modification. The agent can read, write, and process files as if they were on a local file system, while S3 Files handles the synchronization with your S3 bucket automatically.

This means AI agents can use standard Python libraries, shell scripts, and CLI tools without any code changes.

3. Shared Development Environments

Scenario: Multiple developers or containers need concurrent access to shared data

With S3 Files:

Mount the same S3 bucket on multiple EC2 instances
Developers can read and write simultaneously
Changes are visible across all instances within seconds
No complex locking mechanisms needed

This is especially powerful for Kubernetes clusters where pods need shared access to training data or model artifacts across nodes.

4. Content Management Systems

Scenario: CMS storing media files that need to be accessed by multiple web servers

With S3 Files:

All web servers mount the same S3 bucket
Upload once, available everywhere immediately
No need for S3 sync scripts or CDN invalidation delays

Important Considerations

Let me share some lessons learned from real implementations:

Performance Characteristics

First access: Slower (loading from S3 to cache)
Subsequent access: Sub-millisecond (from cache)
Large sequential reads: Served directly from S3 (high throughput)
Small random reads: Served from cache (low latency)

Pro tip: If you know which files you'll need, you can pre-load them into the cache to avoid first-access latency.

Consistency Model

File system to S3: Changes appear in S3 within minutes
S3 to file system: Changes appear in file system within seconds (sometimes up to a minute)

Important: If you need immediate consistency, use the file system as your source of truth during processing, then let it sync to S3.

Cost Structure

S3 Files has three cost components:

Storage: $0.30/GB-month for data in the cache
Data access:
- $0.03/GB for reads
- $0.06/GB for writes
S3 costs: Standard S3 storage and request costs

Cost optimization tip: The cache only stores actively used data. If you're processing 100GB from a 10TB bucket, you only pay cache costs for the 100GB.

Security

Encryption in transit: TLS 1.3 (automatic)
Encryption at rest: SSE-S3 or KMS (your choice)
Access control: IAM policies + POSIX permissions
Network isolation: Mount targets live in your VPC

When NOT to Use S3 Files

Being honest about limitations is important. S3 Files isn't the right choice for:

Simple object storage: If you're just storing and retrieving whole objects, regular S3 is simpler and cheaper.
Infrequent access: If you rarely access your data, the cache storage costs aren't worth it.
Windows-specific features: If you need Windows file system features, use FSx for Windows File Server.
Extreme performance HPC: If you need the absolute highest performance for HPC workloads, FSx for Lustre is better optimized.
On-premises integration: If you're migrating from on-prem NAS, FSx for NetApp ONTAP provides better compatibility.

Getting Started

S3 Files is available in all commercial AWS regions as of April 2026. You can create an S3 file system through the AWS Console, AWS CLI, or infrastructure as code tools like CloudFormation and Terraform.

The basic workflow involves:

Creating an S3 file system linked to your bucket
Configuring network access (VPC, subnets, security groups)
Setting up IAM permissions
Mounting the file system on your compute resources
Using standard file operations to interact with your S3 data

In the next article, I'll walk you through a complete hands-on implementation with step-by-step commands and real examples.

My Take After Using It

I've been working with S3 Files since early access in late 2025, across several client projects. Here's my honest assessment:

What I love:

It eliminates entire categories of data movement code
Applications that expect file systems just work
The lazy loading is genuinely smart
Cost savings are real when you eliminate data duplication

What to watch out for:

The sync delay (minutes) can surprise people expecting instant S3 updates
You need to understand the caching behavior to optimize costs
Security group configuration is an extra step that trips people up

Bottom line: If you're currently copying data between S3 and file systems, or if you're struggling with S3 FUSE limitations, S3 Files is absolutely worth evaluating. It's not a silver bullet, but for the right use cases, it's transformative.

Availability and Pricing

As of April 2026, S3 Files is available in all commercial AWS regions.

Pricing varies by region, but in us-east-1:

Cache storage: $0.30/GB-month
Read operations: $0.03/GB
Write operations: $0.06/GB
Plus standard S3 costs

Check the S3 pricing page for your region's specific pricing.

Conclusion

Amazon S3 Files represents a significant shift in how we can architect cloud applications. For years, we've had to choose between S3's durability and cost-effectiveness versus a file system's interactive capabilities. S3 Files eliminates that tradeoff.

Whether you're training ML models on spot instances, building AI agents, running data analytics pipelines, or any workload that needs shared, interactive file access to S3 data, S3 Files deserves a serious look. It's not just a new feature it's a new way of thinking about data access in the cloud.

The best part? You don't need to rewrite your applications. If your code works with files, it'll work with S3 Files. And that's the kind of simplicity we all need more of in cloud architecture.

In my next article, I'll show you exactly how to set up S3 Files with a complete hands-on implementation guide, including all the commands, configurations, and real-world testing scenarios.

Have you tried S3 Files yet? I'd love to hear about your use cases and experiences. The cloud architecture community learns best when we share real-world implementations, not just theory.

📌 Wrapping Up

Thank you for reading! I hope this article gave you practical insights and a clearer perspective on the topic.

Was this helpful?

❤️ Like if it added value
🦄 Unicorn if you’re applying it today
💾 Save for your next optimization session
🔄 Share with your team

Follow me for more on:

AWS architecture patterns
FinOps automation
Multi-account strategies
AI-driven DevOps

💡 What’s Next

More deep dives coming soon on cloud operations, GenAI, Agentic-AI, DevOps, and data workflows follow for weekly insights.

🌐 Portfolio & Work

You can explore my full body of work, certifications, architecture projects, and technical articles here:

👉 Visit My Website

🛠️ Services I Offer

If you're looking for hands-on guidance or collaboration, I provide:

Cloud Architecture Consulting (AWS / Azure)
DevSecOps & Automation Design
FinOps Optimization Reviews
Technical Writing (Cloud, DevOps, GenAI)
Product & Architecture Reviews
Mentorship & 1:1 Technical Guidance

🤝 Let’s Connect

I’d love to hear your thoughts drop a comment or connect with me on LinkedIn.

For collaborations, consulting, or technical discussions, feel free to reach out directly at simplynadaf@gmail.com

Happy Learning 🚀

Building Your First VPC: AWS Networking with Terraform

Sarvar Nadaf — Wed, 08 Apr 2026 10:04:44 +0000

👋 Hey there, tech enthusiasts!

🎯 Welcome Back!

Remember in Article 5 when you learned about variables and outputs? You created flexible, reusable S3 bucket configurations. That's great for storage, but what about networking?

Here's the reality: S3 buckets are public services. But what about:

EC2 instances that need network isolation?
Databases that should never be directly accessible from the internet?
Applications that need both public and private components?

That's where VPC (Virtual Private Cloud) comes in.

By the end of this article, you'll:

✅ Understand VPC fundamentals (CIDR, subnets, routing)
✅ Create a production-ready VPC with Terraform
✅ Configure public and private subnets
✅ Set up Internet Gateway and NAT Gateway
✅ Master route tables and associations
✅ Use data sources to fetch AWS information
✅ Build cost-optimized VPC architecture

Time Required: 45 minutes (20 min read + 25 min practice)

Cost: ~$0.05 for testing (destroy immediately!)

Difficulty: Intermediate

⚠️ IMPORTANT: NAT Gateway costs $0.045/hour (~$32/month). We'll create it for learning, but destroy it immediately after testing!

Let's build real infrastructure! 🚀

💔 The Problem: Networking Confusion

The Manual Nightmare

Imagine this scenario (maybe you've lived it):

Your Task: "Set up a VPC for our new application"

You open AWS Console:

Step 1: Create VPC... wait, what's a CIDR block?
Step 2: Create subnets... public or private? What's the difference?
Step 3: Internet Gateway... where does this go?
Step 4: NAT Gateway... why do I need this?
Step 5: Route tables... which subnet goes where?
Step 6: *2 hours later* ... did I configure this correctly?
Step 7: *Application doesn't work* ... something's wrong with routing
Step 8: *Starts over* 😰

Common Problems:

❌ Forgot route table associations - Subnets can't reach internet

❌ Wrong CIDR blocks - IP addresses overlap

❌ NAT Gateway in wrong subnet - Private instances can't update

❌ Inconsistent configuration - Resources don't work together

❌ Can't remember configuration - No documentation

❌ Takes hours to set up - Clicking through endless screens

Sound familiar? Let's fix this with Terraform.

🌐 What is a VPC? (Quick Theory)

Simple Definition

VPC (Virtual Private Cloud) is your own private network in AWS. Think of it like this:

Traditional Data Center: Physical building with servers, switches, routers
VPC: Virtual data center with virtual servers, virtual switches, virtual routers

Key Point: Your VPC is isolated from other AWS accounts. It's YOUR private network.

VPC Components (The Building Blocks)

VPC (Your Private Network)
├── CIDR Block (IP Address Range)
├── Subnets (Subdivisions of your network)
│   ├── Public Subnets (Internet-accessible)
│   └── Private Subnets (Internal only)
├── Internet Gateway (Door to the internet)
├── NAT Gateway (Private subnet's internet access)
└── Route Tables (GPS for network traffic)

Understanding CIDR Blocks

CIDR = Classless Inter-Domain Routing (fancy name for IP address ranges)

Examples:

10.0.0.0/16 = 65,536 IP addresses (10.0.0.0 to 10.0.255.255)
10.0.1.0/24 = 256 IP addresses (10.0.1.0 to 10.0.1.255)
10.0.2.0/24 = 256 IP addresses (10.0.2.0 to 10.0.2.255)

Simple Rule: Smaller number after / = More IP addresses

/16 = 65,536 IPs (big network)
/24 = 256 IPs (small network)
/32 = 1 IP (single host)

Don't worry! You don't need to be a networking expert. Just follow the patterns.

Public vs Private Subnets

Public Subnet:

✅ Has route to Internet Gateway
✅ Resources get public IP addresses
✅ Directly accessible from internet
📍 Use for: Web servers, load balancers, bastion hosts

Private Subnet:

✅ Has route to NAT Gateway (not Internet Gateway)
❌ No public IP addresses
❌ Not directly accessible from internet
📍 Use for: Databases, application servers, internal services

Why both? Security! Keep sensitive resources (databases) in private subnets.

🏗️ What We'll Build Today

Architecture Overview

VPC: 10.0.0.0/16 (65,536 IPs)
│
├── Public Subnet 1: 10.0.1.0/24 (256 IPs) - us-east-1a
│   ├── Internet Gateway attached
│   └── Route: 0.0.0.0/0 → Internet Gateway
│
├── Private Subnet 1: 10.0.11.0/24 (256 IPs) - us-east-1a
│   └── Route: 0.0.0.0/0 → NAT Gateway

Why this design?

✅ Single-AZ
✅ Public subnets = For load balancers, web servers
✅ Private subnets = For databases, application servers
✅ NAT Gateway = Private subnets can download updates
✅ Production-ready = This is how real companies do it

Source Code

The complete source code and Terraform configuration used in this article can be found on GitHub:

simplynadaf / terraform-by-sarvar

Terraform Series

🚀 Terraform By Sarvar

Complete Terraform tutorial series from basics to advanced concepts

📖 Read the Series • 🌐 Portfolio • 💼 LinkedIn

📚 Series Overview

This repository contains ONLY Terraform code examples for the Terraform By Sarvar tutorial series.

⚠️ IMPORTANT: This repo contains only .tf files and infrastructure code. Articles are published on dev.to, not stored here.

📖 Read the series on dev.to: https://dev.to/sarvar_04/series/36963

🎯 What You'll Learn

✅ Infrastructure as Code (IaC) fundamentals
✅ Terraform basics to advanced concepts
✅ AWS resource provisioning
✅ Best practices and real-world patterns
✅ Production-ready configurations

📊 Series Progress

📖 Article Series

📘 Foundation (Articles 1-5) ✅ Complete

Introduction to Terraform & IaC - ✅ Published
Installation & Setup - ✅ Published
Your First AWS Resource - ✅ Published
Understanding Terraform State - ✅ Published
Variables and Outputs - ✅ Published

📗 Real Infrastructure (Articles 6-10)

Building a VPC from Scratch…

View on GitHub

Getting Started

Follow these steps to run the project on your local machine:

Clone the repository:

git clone https://github.com/simplynadaf/terraform-by-sarvar.git

Navigate to the project directory:

cd terraform-by-sarvar/articles/08-lambda-serverless

📝 Step-by-Step: Building the VPC

Step 1: Create Project Directory

mkdir -p ~/terraform-vpc-demo
cd ~/terraform-vpc-demo

Step 2: Create variables.tf

Let's start with variables for flexibility:

nano variables.tf

Copy this code:

# AWS Region
variable "aws_region" {
  description = "AWS region where resources will be created"
  type        = string
  default     = "us-east-1"
}

# Project Name
variable "project_name" {
  description = "Project name to be used in resource naming"
  type        = string
  default     = "terraform-vpc"
}

# Environment
variable "environment" {
  description = "Environment name (dev, staging, prod)"
  type        = string
  default     = "dev"
}

# VPC CIDR Block
variable "vpc_cidr" {
  description = "CIDR block for VPC"
  type        = string
  default     = "10.0.0.0/16"
}

# Public Subnet CIDR
variable "public_subnet_cidr" {
  description = "CIDR block for public subnet"
  type        = string
  default     = "10.0.1.0/24"
}

# Private Subnet CIDR
variable "private_subnet_cidr" {
  description = "CIDR block for private subnet"
  type        = string
  default     = "10.0.11.0/24"
}

What's happening here?

All CIDR blocks are variables (easy to change)
Project name and environment for tagging
Default values provided (but can be overridden)
Cost-optimized: 1 public + 1 private subnet

Save the file (Ctrl+X, then Y, then Enter)

Step 3: Create terraform.tfvars

nano terraform.tfvars

Copy this code:

# Default configuration for development environment
aws_region = "us-east-1"
project_name = "terraform-vpc"
environment = "dev"

# VPC Configuration
vpc_cidr = "10.0.0.0/16"

# Subnets
public_subnet_cidr = "10.0.1.0/24"
private_subnet_cidr = "10.0.11.0/24"

Save the file

Step 4: Create main.tf (The Main Infrastructure)

This is where the magic happens! We'll build it piece by piece.

nano main.tf

Copy this code:

# Terraform configuration
terraform {
  required_version = ">= 1.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

# Provider configuration
provider "aws" {
  region = var.aws_region
}

# Data source: Get available availability zones
data "aws_availability_zones" "available" {
  state = "available"
}

# VPC
resource "aws_vpc" "main" {
  cidr_block           = var.vpc_cidr
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name        = "${var.project_name}-vpc"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

# Internet Gateway
resource "aws_internet_gateway" "main" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name        = "${var.project_name}-igw"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

# Public Subnet 1
resource "aws_subnet" "public_1" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = var.public_subnet_1_cidr
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true

  tags = {
    Name        = "${var.project_name}-public-subnet"
    Environment = var.environment
    Type        = "Public"
    ManagedBy   = "Terraform"
  }
}

# Private Subnet
resource "aws_subnet" "private" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = var.private_subnet_cidr
  availability_zone = data.aws_availability_zones.available.names[0]

  tags = {
    Name        = "${var.project_name}-private-subnet"
    Environment = var.environment
    Type        = "Private"
    ManagedBy   = "Terraform"
  }
}

# Elastic IP for NAT Gateway
resource "aws_eip" "nat" {
  domain = "vpc"

  tags = {
    Name        = "${var.project_name}-nat-eip"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }

  depends_on = [aws_internet_gateway.main]
}

# NAT Gateway
resource "aws_nat_gateway" "main" {
  allocation_id = aws_eip.nat.id
  subnet_id     = aws_subnet.public_1.id

  tags = {
    Name        = "${var.project_name}-nat-gateway"
    Environment = var.environment
    ManagedBy   = "Terraform"
  }

  depends_on = [aws_internet_gateway.main]
}

# Public Route Table
resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.main.id
  }

  tags = {
    Name        = "${var.project_name}-public-rt"
    Environment = var.environment
    Type        = "Public"
    ManagedBy   = "Terraform"
  }
}

# Private Route Table
resource "aws_route_table" "private" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.main.id
  }

  tags = {
    Name        = "${var.project_name}-private-rt"
    Environment = var.environment
    Type        = "Private"
    ManagedBy   = "Terraform"
  }
}

# Public Subnet Route Table Association
resource "aws_route_table_association" "public" {
  subnet_id      = aws_subnet.public.id
  route_table_id = aws_route_table.public.id
}

# Private Subnet Route Table Association
resource "aws_route_table_association" "private_1" {
  subnet_id      = aws_subnet.private_1.id
  route_table_id = aws_route_table.private.id
}

# Private Subnet Route Table Association
resource "aws_route_table_association" "private" {
  subnet_id      = aws_subnet.private.id
  route_table_id = aws_route_table.private.id
}

Save the file

That's the complete VPC infrastructure! Let's break it down...

🔍 Understanding the Code

1. Data Source (NEW CONCEPT!)

data "aws_availability_zones" "available" {
  state = "available"
}

What's a data source?

Resources = Things Terraform creates (VPC, subnets, etc.)
Data Sources = Things Terraform reads from AWS (AZs, AMIs, etc.)

Why use it?

Don't hardcode availability zones (they differ by region)
Automatically get available AZs
Makes code portable across regions

How to use it:

availability_zone = data.aws_availability_zones.available.names[0]  # First AZ
availability_zone = data.aws_availability_zones.available.names[1]  # Second AZ

2. VPC Resource

resource "aws_vpc" "main" {
  cidr_block           = var.vpc_cidr
  enable_dns_hostnames = true  # Important for EC2 instances
  enable_dns_support   = true  # Important for DNS resolution
}

Key settings:

enable_dns_hostnames = EC2 instances get DNS names
enable_dns_support = DNS resolution works in VPC

Without these: Your instances can't resolve domain names!

3. Internet Gateway

resource "aws_internet_gateway" "main" {
  vpc_id = aws_vpc.main.id  # Attach to our VPC
}

What it does: Allows public subnets to access the internet

Think of it as: The front door of your VPC

4. Public Subnets

resource "aws_subnet" "public_1" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = var.public_subnet_1_cidr
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true  # Auto-assign public IPs
}

Key setting: map_public_ip_on_launch = true

EC2 instances in this subnet automatically get public IPs
Makes them accessible from internet

5. Private Subnets

resource "aws_subnet" "private_1" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = var.private_subnet_1_cidr
  availability_zone = data.aws_availability_zones.available.names[0]
  # NO map_public_ip_on_launch = Private!
}

Notice: No map_public_ip_on_launch

Instances in private subnets don't get public IPs
More secure for databases and internal services

6. Elastic IP and NAT Gateway

# Elastic IP (static public IP)
resource "aws_eip" "nat" {
  domain = "vpc"
  depends_on = [aws_internet_gateway.main]
}

# NAT Gateway (in public subnet!)
resource "aws_nat_gateway" "main" {
  allocation_id = aws_eip.nat.id
  subnet_id     = aws_subnet.public_1.id  # Must be in public subnet
  depends_on = [aws_internet_gateway.main]
}

What's NAT Gateway?

Allows private subnets to access internet (outbound only)
Private instances can download updates, call APIs
But internet can't initiate connections to private instances

Why Elastic IP?

NAT Gateway needs a static public IP
Elastic IP provides that

⚠️ COST WARNING: NAT Gateway = $0.045/hour (~$32/month)

7. Route Tables (The GPS)

Public Route Table:

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"  # All traffic
    gateway_id = aws_internet_gateway.main.id  # Goes to Internet Gateway
  }
}

Translation: "Send all internet traffic (0.0.0.0/0) to the Internet Gateway"

Private Route Table:

resource "aws_route_table" "private" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block     = "0.0.0.0/0"  # All traffic
    nat_gateway_id = aws_nat_gateway.main.id  # Goes to NAT Gateway
  }
}

Translation: "Send all internet traffic (0.0.0.0/0) to the NAT Gateway"

8. Route Table Associations

resource "aws_route_table_association" "public_1" {
  subnet_id      = aws_subnet.public_1.id
  route_table_id = aws_route_table.public.id
}

What this does: Links subnet to route table

Think of it as: "Public Subnet 1, use the Public Route Table for routing"

Without this: Subnet doesn't know how to route traffic!

📤 Step 5: Create outputs.tf

nano outputs.tf

Copy this code:

# VPC Outputs
output "vpc_id" {
  description = "ID of the VPC"
  value       = aws_vpc.main.id
}

output "vpc_cidr" {
  description = "CIDR block of the VPC"
  value       = aws_vpc.main.cidr_block
}

# Subnet Outputs
output "public_subnet_1_id" {
  description = "ID of public subnet 1"
  value       = aws_subnet.public_1.id
}

output "public_subnet_2_id" {
  description = "ID of public subnet 2"
  value       = aws_subnet.public_2.id
}

output "private_subnet_1_id" {
  description = "ID of private subnet 1"
  value       = aws_subnet.private_1.id
}

output "private_subnet_2_id" {
  description = "ID of private subnet 2"
  value       = aws_subnet.private_2.id
}

# Gateway Outputs
output "internet_gateway_id" {
  description = "ID of the Internet Gateway"
  value       = aws_internet_gateway.main.id
}

output "nat_gateway_id" {
  description = "ID of the NAT Gateway"
  value       = aws_nat_gateway.main.id
}

output "nat_gateway_public_ip" {
  description = "Public IP of the NAT Gateway"
  value       = aws_eip.nat.public_ip
}

# Route Table Outputs
output "public_route_table_id" {
  description = "ID of the public route table"
  value       = aws_route_table.public.id
}

output "private_route_table_id" {
  description = "ID of the private route table"
  value       = aws_route_table.private.id
}

# Availability Zone
output "availability_zone" {
  description = "Availability zone used"
  value       = data.aws_availability_zones.available.names[0]
}

Save the file

Why so many outputs?

You'll need these IDs in Article 7 (EC2 + Load Balancer)
Good documentation of what was created
Easy to reference in other Terraform modules

🚀 Let's Deploy the VPC!

Step 6: Initialize Terraform

terraform init

Expected output:

Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 5.0"...
- Installing hashicorp/aws v5.x.x...

Terraform has been successfully initialized!

Step 7: Plan the Infrastructure

terraform plan

What you'll see:

Terraform will perform the following actions:

  # aws_eip.nat will be created
  # aws_internet_gateway.main will be created
  # aws_nat_gateway.main will be created
  # aws_route_table.private will be created
  # aws_route_table.public will be created
  # aws_route_table_association.private_1 will be created
  # aws_route_table_association.private_2 will be created
  # aws_route_table_association.public_1 will be created
  # aws_route_table_association.public_2 will be created
  # aws_subnet.private_1 will be created
  # aws_subnet.private_2 will be created
  # aws_subnet.public_1 will be created
  # aws_subnet.public_2 will be created
  # aws_vpc.main will be created

Plan: 10 to add, 0 to change, 0 to destroy.

10 resources! That's a complete VPC infrastructure from just one command.

Review the plan carefully:

✅ VPC with correct CIDR (10.0.0.0/16)
✅ 4 subnets with correct CIDRs
✅ Subnets in different availability zones
✅ Internet Gateway attached to VPC
✅ NAT Gateway in public subnet
✅ Route tables with correct routes

Step 8: Apply the Configuration

terraform apply

Type yes when prompted.

Expected output:

aws_vpc.main: Creating...
aws_vpc.main: Creation complete after 2s [id=vpc-xxxxx]
aws_internet_gateway.main: Creating...
aws_subnet.public_1: Creating...
aws_subnet.public_2: Creating...
aws_subnet.private_1: Creating...
aws_subnet.private_2: Creating...
...
aws_nat_gateway.main: Creating...
aws_nat_gateway.main: Still creating... [10s elapsed]
aws_nat_gateway.main: Still creating... [20s elapsed]
...
aws_nat_gateway.main: Creation complete after 1m45s

Apply complete! Resources: 10 added, 0 changed, 0 destroyed.

Outputs:

availability_zones = [
  "us-east-1a"
internet_gateway_id = "igw-xxxxx"
nat_gateway_id = "nat-xxxxx"
nat_gateway_public_ip = "54.xxx.xxx.xxx"
private_route_table_id = "rtb-xxxxx"
private_subnet_id = "subnet-xxxxx"
public_route_table_id = "rtb-xxxxx"
public_subnet_id = "subnet-xxxxx"
vpc_cidr = "10.0.0.0/16"
vpc_id = "vpc-xxxxx"

⏱️ Time: Takes about 2-3 minutes (NAT Gateway is slow to create)

🎉 Success! You just created production-grade networking infrastructure!

Step 9: Verify in AWS Console

Option 1: AWS CLI

# Get VPC ID
VPC_ID=$(terraform output -raw vpc_id)

# List all subnets
aws ec2 describe-subnets \
  --filters "Name=vpc-id,Values=$VPC_ID" \
  --query 'Subnets[*].[SubnetId,CidrBlock,AvailabilityZone,Tags[?Key==`Name`].Value|[0]]' \
  --output table

# Check NAT Gateway
aws ec2 describe-nat-gateways \
  --filter "Name=vpc-id,Values=$VPC_ID" \
  --query 'NatGateways[*].[NatGatewayId,State,SubnetId]' \
  --output table

# Check route tables
aws ec2 describe-route-tables \
  --filters "Name=vpc-id,Values=$VPC_ID" \
  --query 'RouteTables[*].[RouteTableId,Tags[?Key==`Name`].Value|[0]]' \
  --output table

Option 2: AWS Console

Go to AWS Console → VPC
Find your VPC: terraform-vpc-vpc
Check:
- ✅ 2 subnets (1 public, 1 private)
- ✅ Internet Gateway attached
- ✅ NAT Gateway in public subnet
- ✅ 2 route tables (public and private)

🧠 Understanding Resource Dependencies

Implicit Dependencies

Terraform automatically understands dependencies when you reference resources:

resource "aws_subnet" "public_1" {
  vpc_id = aws_vpc.main.id  # References VPC
}

Terraform knows: "Create VPC first, then create subnet"

Explicit Dependencies

Sometimes you need to force an order:

resource "aws_eip" "nat" {
  domain = "vpc"
  depends_on = [aws_internet_gateway.main]  # Explicit dependency
}

Why? Elastic IP in VPC requires Internet Gateway to exist first.

Dependency Graph

Terraform builds a graph of dependencies:

VPC
├── Internet Gateway
│   ├── Elastic IP
│   │   └── NAT Gateway
│   └── Public Route Table
│       └── Route Table Associations
└── Subnets
    └── Route Table Associations

Terraform creates resources in the correct order automatically!

💰 Cost Breakdown

Let's talk about costs (important!):

Resource	Cost	Notes
VPC	FREE	No charge for VPCs
Subnets	FREE	No charge for subnets
Internet Gateway	FREE	No charge for IGW
Elastic IP	FREE*	*When attached to running instance
NAT Gateway	$0.045/hour	~$32/month ⚠️
Data Transfer	$0.045/GB	Through NAT Gateway

Total for this setup: ~$32/month (just for NAT Gateway)

⚠️ IMPORTANT: Always destroy test infrastructure!

terraform destroy

Production Note: In production, you'd have one NAT Gateway per AZ (2 NAT Gateways = $64/month) for high availability.

🎯 Best Practices

1. CIDR Planning

Do:

✅ Plan CIDR blocks before creating VPC
✅ Leave room for growth (use /16 for VPC)
✅ Use non-overlapping ranges

Don't:

❌ Use overlapping CIDR blocks
❌ Make VPC too small (/24 = only 256 IPs)
❌ Forget about VPC peering requirements

Tool: Use CIDR Calculator for planning

2. Multi-AZ Architecture

Always use multiple availability zones:

✅ High availability
✅ Fault tolerance
✅ Better for production

Our setup: 2 AZs (us-east-1a, us-east-1b)

3. Tagging Strategy

Always tag resources:

tags = {
  Name        = "${var.project_name}-vpc"
  Environment = var.environment
  ManagedBy   = "Terraform"
  Project     = "Learning"
}

Why?

Easy to identify resources
Cost allocation
Automation and filtering

4. Separate Public and Private

Security principle:

Public subnets: Load balancers, bastion hosts
Private subnets: Databases, application servers

Never put databases in public subnets!

5. Use Variables

Make everything configurable:

CIDR blocks
Project name
Environment
Region

Benefit: Same code for dev, staging, prod

🐛 Common Issues & Solutions

Issue 1: NAT Gateway Creation Timeout

Problem:

Error: timeout while waiting for state to become 'available'

Solution:

NAT Gateway takes 2-3 minutes to create
This is normal, just wait
If it fails, run terraform apply again

Issue 2: Elastic IP Limit Reached

Problem:

Error: AddressLimitExceeded: The maximum number of addresses has been reached

Solution:

# Check current EIPs
aws ec2 describe-addresses

# Release unused EIPs
aws ec2 release-address --allocation-id eipalloc-xxxxx

# Or request limit increase in AWS Console

Issue 3: Route Table Not Associated

Problem: Subnet can't reach internet

Solution: Check route table associations:

aws ec2 describe-route-tables --filters "Name=vpc-id,Values=$VPC_ID"

Ensure each subnet has a route table association.

Issue 4: CIDR Block Overlap

Problem:

Error: InvalidSubnet.Conflict: The CIDR 'x.x.x.x/x' conflicts with another subnet

Solution:

Check your CIDR blocks don't overlap
Public: 10.0.1.0/24, 10.0.2.0/24
Private: 10.0.11.0/24, 10.0.12.0/24
No overlap!

Issue 5: Forgot to Destroy NAT Gateway

Problem: Unexpected AWS bill

Solution:

# Check for NAT Gateways
aws ec2 describe-nat-gateways --filter "Name=state,Values=available"

# Destroy with Terraform
terraform destroy

# Or manually delete in AWS Console

🧹 Cleanup (IMPORTANT!)

⚠️ Don't forget this step! NAT Gateway costs money.

# Destroy all resources
terraform destroy

Type yes when prompted.

Expected output:

aws_route_table_association.private_2: Destroying...
aws_route_table_association.private_1: Destroying...
aws_route_table_association.public_2: Destroying...
aws_route_table_association.public_1: Destroying...
...
aws_nat_gateway.main: Destroying...
aws_nat_gateway.main: Still destroying... [10s elapsed]
aws_nat_gateway.main: Still destroying... [20s elapsed]
...
aws_nat_gateway.main: Destruction complete after 1m30s
...
Destroy complete! Resources: 10 destroyed.

Verify deletion:

# Should return empty
aws ec2 describe-vpcs --filters "Name=tag:Name,Values=terraform-vpc-vpc"

Clean up directory:

cd ~
rm -rf ~/terraform-vpc-demo

📚 What You Learned

Concepts:

✅ VPC fundamentals (CIDR, subnets, routing)
✅ Public vs Private subnets
✅ Internet Gateway and NAT Gateway
✅ Route tables and associations
✅ Data sources (aws_availability_zones)
✅ Resource dependencies (implicit and explicit)
✅ Cost-optimized architecture

Skills:

✅ Created production-ready VPC
✅ Configured networking components
✅ Used data sources
✅ Managed complex infrastructure
✅ Understood AWS networking costs

Infrastructure:

✅ 1 VPC
✅ 2 Subnets (1 public, 1 private)
✅ 1 Internet Gateway
✅ 1 NAT Gateway
✅ 2 Route Tables
✅ Cost-optimized setup

🎓 Challenge: Extend Your VPC

Try these modifications:

Easy:

Change CIDR blocks to 172.16.0.0/16
Add a third availability zone
Change project name and environment

Medium:

Add more private subnets (10.0.21.0/24, 10.0.22.0/24)
Create separate route tables for each private subnet
Add Network ACLs for additional security

Hard:

Add VPC Flow Logs
Create a second NAT Gateway for high availability
Add VPC endpoints for S3 and DynamoDB

🔗 Useful Resources

🚀 What's Next?

In Article 7, we'll deploy EC2 instances and an Application Load Balancer in this VPC!

You'll learn:

Launch EC2 instances in public and private subnets
Configure security groups
Create an Application Load Balancer
Set up health checks
Test the complete architecture

Time: 45 minutes

Difficulty: Intermediate

Outcome: Full web application infrastructure

📌 Wrapping Up

Thank you for reading. I hope this article provided practical insights and a clearer understanding of the topic.

If you found this useful:

❤️ Like if it added value
🦄 Unicorn if you’re applying it today
💾 Save it for your next optimization session
🔄 Share it with your team

💡 What’s Next

More deep dives are coming soon on:

Cloud Operations
GenAI & Agentic AI
DevOps Automation
Data & Platform Engineering

Follow along for weekly insights and hands-on guides.

🌐 Portfolio & Work

You can explore my full body of work, certifications, architecture projects, and technical articles here:

👉 Visit My Website

🛠️ Services I Offer

If you're looking for hands-on guidance or collaboration, I provide:

Cloud Architecture Consulting (AWS / Azure)
DevSecOps & Automation Design
FinOps Optimization Reviews
Technical Writing (Cloud, DevOps, GenAI)
Product & Architecture Reviews
Mentorship & 1:1 Technical Guidance

🤝 Let’s Connect

I’d love to hear your thoughts. Feel free to drop a comment or connect with me on:

🔗 LinkedIn

For collaborations, consulting, or technical discussions, reach out at:

📧 simplynadaf@gmail.com

Found this helpful? Share it with your team.

⭐ Star the repo • 📖 Follow the series • 💬 Ask questions

Made by Sarvar Nadaf

🌐 https://sarvarnadaf.com

Connecting 12 MCP Servers to Amazon Q CLI: What Broke and How I Fixed It

Sarvar Nadaf — Mon, 06 Apr 2026 13:02:02 +0000

👋 Hey there, tech enthusiasts!

Let's dive in and explore the fascinating world of cloud technology together! 🚀

Written from experience building AI agent integrations for AWS infrastructure management. Your mileage may vary, but the principles hold across different use cases.

Do We Still Need MCP When We Have Agent Skills?

As a cloud architect working with AI agents, I've spent the last few months exploring how to extend their capabilities. Specifically, I've been working with Amazon Q Developer Pro - AWS's AI assistant that helps with coding, infrastructure management, and cloud operations through a chat interface.

This article shares what I learned about two different approaches to extending AI agents: Model Context Protocol (MCP) and Agent Skills. While the specific implementation details are still evolving, the architectural patterns I describe here represent where the ecosystem is heading based on current capabilities and community standards.

The Problem I Was Solving

I needed Amazon Q Developer Pro to help me manage AWS infrastructure across multiple accounts. The agent needed to:

Check CloudWatch metrics
Query RDS databases
Send alerts to Slack
Generate cost reports
Review CloudFormation templates

I tried two approaches, and each had serious problems.

Approach 1: Connecting Everything via MCP

MCP is a standard protocol that lets AI agents connect to external tools. Think of it like USB ports on your computer - one standard interface that works with many devices.

I connected 12 MCP servers to Amazon Q Developer Pro:

AWS CloudWatch server
RDS query server
Slack messaging server
Cost Explorer server
GitHub server
And seven more

What Went Wrong

Before I even asked a question, 35% of the context window was already full. Each MCP server loaded all its available functions into memory. The CloudWatch server alone exposed 15 different functions with detailed parameter descriptions.

When I asked Amazon Q Developer Pro to "check if our xyz database is healthy," it had to scan through multiple function definitions to figure out which ones to use. Sometimes it picked the wrong ones.

Every function call and response consumed more context. After three or four operations, I was running out of space for the actual conversation.

Approach 2: Using Agent Skills

Agent Skills are different. Instead of connecting to external tools, you give the agent domain knowledge - a guide on how to think about a problem.

I created a skill called "database-health-check" with a simple file structure:

database-health-check/
  SKILL.md          (when to use this, what steps to follow)
  scripts/
    check_rds.py    (Python script to query RDS)

The SKILL.md file contained:

# Database Health Check Skill

## Trigger
Keywords: "database health", "RDS status", "database performance", "DB issues"

## Process
1. Check CPU utilization (threshold: 80%)
2. Check active connections (threshold: 90% of max)
3. Check replication lag (threshold: 1000ms)
4. Check storage space (threshold: 85%)

## Decision Logic
- If CPU > 80%: WARN - "High CPU usage detected"
- If connections > 90%: CRITICAL - "Connection pool nearly exhausted"
- If replication lag > 1000ms: WARN - "Replication falling behind"
- If storage > 85%: CRITICAL - "Low storage space"

## Output Format
Status: [HEALTHY|WARN|CRITICAL]
Issues: [list of problems found]
Recommendations: [suggested actions]

What Went Wrong

The skill worked perfectly on my laptop. Then my colleague tried to use it and got an error: "ModuleNotFoundError: No module named 'boto3'".

The Python script needed boto3, pandas, and psycopg2. My machine had them installed. His didn't. We had no standard way to declare or install these dependencies.

The Real Difference

After working with both, I realized they solve different problems:

MCP answers: "What can I do?"
It provides capabilities - functions the agent can call. Each MCP server is self-contained with its own dependencies and environment.

Skills answer: "How should I think?"
They provide expertise - decision logic, quality standards, and workflows. But they run in whatever environment the agent has.

The Solution: Use Both Together

Here's what actually works in real time. I'll use a real example from last week.

Example: Automated Cost Anomaly Detection

I needed Amazon Q Developer Pro to monitor our AWS costs and alert us when something unusual happens.

The MCP Layer (The Hands)

I set up two MCP servers:

aws-cost-explorer-mcp - exposes functions like get_daily_costs(), get_service_breakdown()
slack-notifications-mcp - exposes send_message(), create_incident()

Each server runs in its own Docker container with all dependencies installed. Amazon Q Developer Pro doesn't need to know about boto3 or the Slack SDK.

The Skill Layer (The Brain)

I created a cost-monitoring skill with this logic in SKILL.md:

Trigger: When user asks about cost anomalies or unusual spending

Process:
1. Get last 30 days of daily costs
2. Calculate average and standard deviation
3. Check if today's cost is more than 2 standard deviations above average
4. If yes, get service breakdown to identify which service spiked
5. If spike is over $500, send high-priority Slack alert
6. If spike is under $500, just report in conversation

Quality checks:
- Always compare against same day of week (Monday vs Monday)
- Exclude known scheduled events (monthly backups, etc.)
- Include percentage change, not just absolute numbers

How They Work Together

When I ask Amazon Q Developer Pro: "Are our AWS costs normal today?"

Amazon Q Developer Pro matches the question to the cost-monitoring skill
The skill loads its SKILL.md (only 2KB of context)
The skill requires cost data, so Amazon Q Developer Pro connects to the aws-cost-explorer-mcp server
The skill orchestrates: call get_daily_costs(), analyze the data, decide if it's anomalous
If anomalous, the skill calls the slack-notifications-mcp server
After completion, the skill unloads from memory

The MCP servers handle the technical execution. The skill handles the business logic and decision-making.

Why This Architecture Works

Context Efficiency
Only the active skill loads into memory. MCP tools load on-demand when the skill needs them. I went from 35% context consumed at startup to 5%.

Portability
The same skill works on my laptop, my colleague's Windows machine, and our CI/CD pipeline. The MCP servers can run locally, in containers, or as remote services.

Reusability
The aws-cost-explorer-mcp server is used by three different skills:

cost-monitoring (detects anomalies)
budget-planning (forecasts spending)
cost-optimization (finds savings opportunities)

Each skill brings different expertise, but they share the same data source.

Maintainability
When AWS changes their Cost Explorer API, I update one MCP server. All skills continue working. When business rules change (new alert thresholds), I update the skill. The MCP servers don't need to change.

Visual Overview

Here's how the three layers work together:

┌─────────────────────────────────────────┐
│   Agent Layer (Amazon Q Developer Pro)  │
│   Matches tasks → Loads skills          │
│   Connects to MCP servers               │
└─────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────┐
│         Skill Layer (The Brain)         │
│   cost-monitoring.SKILL.md              │
│   - When to trigger                     │
│   - Decision logic                      │
│   - Quality checks                      │
└─────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────┐
│         MCP Layer (The Hands)           │
│   aws-cost-explorer-mcp                 │
│   slack-notifications-mcp               │
│   - Tool execution                      │
│   - Environment isolation               │
└─────────────────────────────────────────┘

A Practical Pattern

Here's the decision framework I use:

Put it in an MCP server if:

Multiple skills will use it
It has heavy dependencies (Python libraries, system tools)
It needs credentials or secrets
It's a stable, reusable capability

Put it in a skill if:

It's domain knowledge or business logic
It orchestrates multiple tools
It has conditional decision-making
It's specific to one workflow

Keep it as a simple script if:

It's a one-time operation
The entire workflow is tightly coupled
Splitting it would add unnecessary complexity

What's Still Missing

The ecosystem isn't mature yet. Here's what I wish existed:

Dependency Declaration
Skills need a standard way to say "I need these capabilities" without hardcoding specific MCP servers. Something like:

requires:
  - cloud_metrics
  - notifications

Then the runtime figures out which MCP servers provide those capabilities.

Dynamic Loading
Right now, when you connect an MCP server, all its tools load immediately. I want skills to control this: "Load only the cost analysis tools for this task."

Graceful Fallback
If an MCP server is unavailable, the skill should automatically fall back to a built-in script or tell me clearly what's missing.

Conclusion

After six months of exploring these patterns with Amazon Q Developer Pro and other AI agents, here's what I know:

MCP and Agent Skills aren't competing approaches. They're complementary layers of the same system.

MCP gives your agent reliable, isolated capabilities. Skills give your agent the expertise to use those capabilities intelligently.

You need both. MCP without skills is a toolbox with no craftsman. Skills without MCP are expertise with unreliable tools.

The architecture that works:

Skills encode what to do and when
MCP servers provide how to do it
The agent runtime connects them together

This isn't just theoretical. These patterns are being implemented in actual environments, managing real AWS infrastructure.

Getting Started

If you want to explore this architecture:

For MCP:

Check the official MCP specification at modelcontextprotocol.io
Browse existing MCP servers at github.com/modelcontextprotocol/servers
Amazon Q Developer supports MCP through the Q CLI

For Agent Skills:

Review the Agent Skills specification by Anthropic
Start with simple skills that encode your team's operational knowledge
Focus on decision logic and workflows, not heavy computation

Next Steps:

Identify one repetitive task you do with your AI agent
Ask: Does this need external tools (MCP) or decision logic (Skill)?
Build the simplest version that works
Iterate based on what you learn

The ecosystem is still maturing, but the core patterns are solid. Start small, learn from erros, and build up from there.

📌 Wrapping Up

Thank you for reading! I hope this article gave you practical insights and a clearer perspective on the topic.

Was this helpful?

❤️ Like if it added value
🦄 Unicorn if you’re applying it today
💾 Save for your next optimization session
🔄 Share with your team

Follow me for more on:

AWS architecture patterns
FinOps automation
Multi-account strategies
AI-driven DevOps

💡 What’s Next

More deep dives coming soon on cloud operations, GenAI, Agentic-AI, DevOps, and data workflows follow for weekly insights.

🌐 Portfolio & Work

You can explore my full body of work, certifications, architecture projects, and technical articles here:

👉 Visit My Website

🛠️ Services I Offer

If you're looking for hands-on guidance or collaboration, I provide:

Cloud Architecture Consulting (AWS / Azure)
DevSecOps & Automation Design
FinOps Optimization Reviews
Technical Writing (Cloud, DevOps, GenAI)
Product & Architecture Reviews
Mentorship & 1:1 Technical Guidance

🤝 Let’s Connect

I’d love to hear your thoughts drop a comment or connect with me on LinkedIn.

For collaborations, consulting, or technical discussions, feel free to reach out directly at simplynadaf@gmail.com

Happy Learning 🚀

Terraform Variables and Outputs: Making Your Infrastructure Flexible

Sarvar Nadaf — Wed, 01 Apr 2026 09:59:13 +0000

👋 Hey there, tech enthusiasts!

"The difference between a script and reusable infrastructure code is variables."

🎯 Welcome Back!

Remember in Article 3 when you created that S3 bucket? You hardcoded the bucket name directly in the code:

resource "aws_s3_bucket" "my_first_bucket" {
  bucket = "terraform-first-bucket-yourname-2026"  # Hardcoded!
}

That works... once. But what if you need:

10 buckets with different names?
Same code for dev, staging, and production?
To let your team customize configurations?
To reuse this code in different projects?

Today, you'll learn how to make your Terraform code flexible and reusable using variables and outputs.

By the end of this article, you'll:

✅ Convert hardcoded values to variables
✅ Use all variable types (string, number, bool, list, map)
✅ Work with terraform.tfvars files
✅ Create outputs to expose information
✅ Build dev/prod configurations from the same code
✅ Use variable validation and sensitive values

Time Required: 30 minutes

Cost: $0 (using free tier resources)

Difficulty: Beginner-friendly

Let's make your code reusable! 🚀

💔 The Problem: Hardcoded Hell

The Painful Reality

Scenario 1: Multiple Environments

# dev.tf
resource "aws_s3_bucket" "app" {
  bucket = "myapp-dev-bucket"
}

# staging.tf (copy-paste and change)
resource "aws_s3_bucket" "app" {
  bucket = "myapp-staging-bucket"
}

# prod.tf (copy-paste and change again)
resource "aws_s3_bucket" "app" {
  bucket = "myapp-prod-bucket"
}

Problems:

❌ Three copies of the same code
❌ Change one thing = update three files
❌ High risk of mistakes
❌ Nightmare to maintain

Scenario 2: Team Collaboration

Developer 1: "I need the bucket name to be X"
Developer 2: "I need it to be Y"
DevOps: "Production needs Z"

Everyone edits the same .tf file = merge conflicts and chaos!

There's a better way. Enter variables.

🌟 What Are Variables?

Simple Definition

Variables are like function parameters for your infrastructure code.

Think of it like this:

Without Variables (Hardcoded):

function createBucket() {
  return "my-hardcoded-bucket-name";
}

With Variables (Flexible):

function createBucket(bucketName) {
  return bucketName;
}

createBucket("dev-bucket");    // Different uses
createBucket("prod-bucket");   // Same code!

In Terraform:

# Define the variable
variable "bucket_name" {
  description = "Name of the S3 bucket"
  type        = string
}

# Use the variable
resource "aws_s3_bucket" "app" {
  bucket = var.bucket_name  # Reference with var.
}

Same code, different values! That's the power of variables.

Source Code

The complete source code and Terraform configuration used in this article can be found on GitHub:

simplynadaf / terraform-by-sarvar

Terraform Series

🚀 Terraform By Sarvar

Complete Terraform tutorial series from basics to advanced concepts

📖 Read the Series • 🌐 Portfolio • 💼 LinkedIn

📚 Series Overview

This repository contains ONLY Terraform code examples for the Terraform By Sarvar tutorial series.

⚠️ IMPORTANT: This repo contains only .tf files and infrastructure code. Articles are published on dev.to, not stored here.

📖 Read the series on dev.to: https://dev.to/sarvar_04/series/36963

🎯 What You'll Learn

✅ Infrastructure as Code (IaC) fundamentals
✅ Terraform basics to advanced concepts
✅ AWS resource provisioning
✅ Best practices and real-world patterns
✅ Production-ready configurations

📊 Series Progress

📖 Article Series

📘 Foundation (Articles 1-5) ✅ Complete

Introduction to Terraform & IaC - ✅ Published
Installation & Setup - ✅ Published
Your First AWS Resource - ✅ Published
Understanding Terraform State - ✅ Published
Variables and Outputs - ✅ Published

📗 Real Infrastructure (Articles 6-10)

Building a VPC from Scratch…

View on GitHub

Getting Started

Follow these steps to run the project on your local machine:

Clone the repository:

git clone https://github.com/simplynadaf/terraform-by-sarvar.git

Navigate to the project directory:

cd terraform-by-sarvar/articles/08-lambda-serverless

🛠️ Hands-On: Your First Variable

Let's convert our hardcoded S3 bucket to use variables.

Step 1: Create Project Directory

mkdir -p ~/terraform-variables-demo
cd ~/terraform-variables-demo

Step 2: Create variables.tf

Create a new file called variables.tf:

nano variables.tf

Add this code:

# String variable - for text values
variable "aws_region" {
  description = "AWS region where resources will be created"
  type        = string
  default     = "us-east-1"
}

variable "bucket_name" {
  description = "Name of the S3 bucket (must be globally unique)"
  type        = string
  # No default - user must provide this value
}

# Map variable - for key-value pairs like tags
variable "common_tags" {
  description = "Common tags to apply to all resources"
  type        = map(string)
  default = {
    Environment = "Development"
    ManagedBy   = "Terraform"
    Project     = "Learning"
  }
}

Understanding the syntax:

description - Explains what the variable is for (always add this!)
type - What kind of value (string, number, bool, list, map)
default - Optional default value (if not provided, user must supply it)

Step 3: Create main.tf

nano main.tf

Add this code:

terraform {
  required_version = ">= 1.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

# Use variable in provider
provider "aws" {
  region = var.aws_region  # Reference variable with var.
}

# Use variables in resource
resource "aws_s3_bucket" "example" {
  bucket = var.bucket_name  # Variable for bucket name

  # Merge common_tags with specific tags
  tags = merge(
    var.common_tags,
    {
      Name = var.bucket_name
    }
  )
}

Notice: We reference variables with var.variable_name

Step 4: Initialize and Test

terraform init
terraform plan

Terraform will prompt you:

var.bucket_name
  Name of the S3 bucket (must be globally unique)

  Enter a value:

Type: my-first-variable-bucket-yourname-2026 and press Enter.

You'll see the plan:

Terraform will perform the following actions:

  # aws_s3_bucket.example will be created
  + resource "aws_s3_bucket" "example" {
      + bucket = "my-first-variable-bucket-yourname-2026"
      + tags   = {
          + "Environment" = "Development"
          + "ManagedBy"   = "Terraform"
          + "Name"        = "my-first-variable-bucket-yourname-2026"
          + "Project"     = "Learning"
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Don't apply yet! Let's learn better ways to pass variables.

📝 Three Ways to Pass Variables

Method 1: Command Line (What We Just Did)

terraform plan
# Terraform prompts for bucket_name

Pros: Interactive

Cons: Tedious for multiple variables

Method 2: Command Line Flags

terraform plan -var="bucket_name=my-bucket-2026"

Pros: Good for CI/CD pipelines

Cons: Long commands with many variables

Method 3: terraform.tfvars File (Best for Most Cases)

Create terraform.tfvars:

nano terraform.tfvars

Add this:

bucket_name = "my-variable-bucket-yourname-2026"
aws_region  = "us-east-1"

common_tags = {
  Environment = "Development"
  ManagedBy   = "Terraform"
  Project     = "Variables Demo"
  Owner       = "YourName"
}

Now run:

terraform plan

No prompts! Terraform automatically loads terraform.tfvars.

Apply it:

terraform apply

Type yes when prompted.

🎉 Success! You just created infrastructure using variables!

🎨 Variable Types: The Complete Guide

Terraform supports multiple variable types. Let's explore them all!

Update variables.tf

Replace your variables.tf with this comprehensive version:

# 1. STRING - Text values
variable "aws_region" {
  description = "AWS region where resources will be created"
  type        = string
  default     = "us-east-1"
}

variable "bucket_name" {
  description = "Name of the S3 bucket (must be globally unique)"
  type        = string
}

variable "bucket_prefix" {
  description = "Prefix for multiple bucket names"
  type        = string
  default     = "terraform-demo"
}

# 2. NUMBER - Numeric values
variable "bucket_count" {
  description = "Number of S3 buckets to create"
  type        = number
  default     = 2

  # Validation rule
  validation {
    condition     = var.bucket_count >= 1 && var.bucket_count <= 5
    error_message = "Bucket count must be between 1 and 5."
  }
}

# 3. BOOLEAN - True/False values
variable "enable_versioning" {
  description = "Enable versioning on the S3 bucket"
  type        = bool
  default     = false
}

# 4. LIST - Ordered collection of values
variable "allowed_ips" {
  description = "List of IP addresses allowed to access the bucket"
  type        = list(string)
  default     = []
}

# 5. MAP - Key-value pairs
variable "common_tags" {
  description = "Common tags to apply to all resources"
  type        = map(string)
  default = {
    Environment = "Development"
    ManagedBy   = "Terraform"
    Project     = "Learning"
  }
}

# 6. ENVIRONMENT - With validation
variable "environment" {
  description = "Environment name (dev, staging, prod)"
  type        = string
  default     = "dev"

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "Environment must be dev, staging, or prod."
  }
}

Update main.tf to Use All Variable Types

terraform {
  required_version = ">= 1.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

# Main S3 bucket with variables
resource "aws_s3_bucket" "example" {
  bucket = var.bucket_name

  tags = merge(
    var.common_tags,
    {
      Name = var.bucket_name
    }
  )
}

# Conditional resource - only created if enable_versioning is true
resource "aws_s3_bucket_versioning" "example" {
  count  = var.enable_versioning ? 1 : 0  # Conditional creation
  bucket = aws_s3_bucket.example.id

  versioning_configuration {
    status = "Enabled"
  }
}

# Multiple buckets using count (number variable)
resource "aws_s3_bucket" "multiple" {
  count  = var.bucket_count
  bucket = "${var.bucket_prefix}-${count.index + 1}-yourname-2026"

  tags = var.common_tags
}

What's happening here:

var.enable_versioning ? 1 : 0 - Conditional: if true, create 1 resource; if false, create 0
count = var.bucket_count - Create multiple resources based on number
count.index - Access the current index (0, 1, 2...)
merge() - Combine two maps together

📤 Outputs: Exposing Information

Variables let data IN. Outputs let data OUT.

Why Outputs Matter

Use cases:

Display important information after terraform apply
Pass values to other Terraform modules
Use in scripts or CI/CD pipelines
Share information with team members

Create outputs.tf

nano outputs.tf

Add this code:

# Basic outputs
output "bucket_name" {
  description = "Name of the created S3 bucket"
  value       = aws_s3_bucket.example.id
}

output "bucket_arn" {
  description = "ARN of the S3 bucket"
  value       = aws_s3_bucket.example.arn
}

output "bucket_region" {
  description = "Region where the bucket was created"
  value       = aws_s3_bucket.example.region
}

output "bucket_domain_name" {
  description = "Domain name of the bucket"
  value       = aws_s3_bucket.example.bucket_domain_name
}

# Multiple buckets output
output "multiple_bucket_names" {
  description = "Names of all created buckets"
  value       = aws_s3_bucket.multiple[*].id
}

# Conditional output
output "versioning_enabled" {
  description = "Whether versioning is enabled"
  value       = var.enable_versioning
}

# Map output
output "bucket_tags" {
  description = "Tags applied to the bucket"
  value       = aws_s3_bucket.example.tags
}

# Sensitive output example
output "bucket_id_sensitive" {
  description = "Bucket ID (marked as sensitive)"
  value       = aws_s3_bucket.example.id
  sensitive   = true  # Won't display in plan/apply output
}

Update terraform.tfvars

bucket_name      = "my-variable-bucket-yourname-2026"
bucket_prefix    = "demo-bucket-yourname"
bucket_count     = 2
enable_versioning = true
aws_region       = "us-east-1"

common_tags = {
  Environment = "Development"
  ManagedBy   = "Terraform"
  Project     = "Variables Demo"
  Owner       = "YourName"
}

Apply and See Outputs

terraform apply

After applying, you'll see:

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

Outputs:

bucket_arn = "arn:aws:s3:::my-variable-bucket-yourname-2026"
bucket_domain_name = "my-variable-bucket-yourname-2026.s3.amazonaws.com"
bucket_id_sensitive = <sensitive>
bucket_name = "my-variable-bucket-yourname-2026"
bucket_region = "us-east-1"
bucket_tags = tomap({
  "Environment" = "Development"
  "ManagedBy" = "Terraform"
  "Name" = "my-variable-bucket-yourname-2026"
  "Owner" = "YourName"
  "Project" = "Variables Demo"
})
multiple_bucket_names = [
  "demo-bucket-yourname-1-yourname-2026",
  "demo-bucket-yourname-2-yourname-2026",
]
versioning_enabled = true

View Outputs Anytime

# View all outputs
terraform output

# View specific output
terraform output bucket_name

# View sensitive output (will reveal the value)
terraform output bucket_id_sensitive

# Output as JSON (useful for scripts)
terraform output -json

🌍 Real-World: Dev vs Prod Configurations

The real power of variables: same code, different configurations!

Create dev.tfvars

nano dev.tfvars

# Development environment configuration
environment       = "dev"
bucket_name       = "myapp-dev-yourname-2026"
bucket_prefix     = "myapp-dev-yourname"
bucket_count      = 2
enable_versioning = false  # Save costs in dev
aws_region        = "us-east-1"

common_tags = {
  Environment = "Development"
  ManagedBy   = "Terraform"
  Project     = "MyApp"
  CostCenter  = "Engineering"
}

Create prod.tfvars

nano prod.tfvars

# Production environment configuration
environment       = "prod"
bucket_name       = "myapp-prod-yourname-2026"
bucket_prefix     = "myapp-prod-yourname"
bucket_count      = 3
enable_versioning = true  # Important for prod!
aws_region        = "us-east-1"

common_tags = {
  Environment = "Production"
  ManagedBy   = "Terraform"
  Project     = "MyApp"
  CostCenter  = "Operations"
  Compliance  = "Required"
}

Deploy to Dev

terraform plan -var-file="dev.tfvars"
terraform apply -var-file="dev.tfvars"

Result: 3 buckets created (1 main + 2 multiple), no versioning

Deploy to Prod

# First destroy dev
terraform destroy -var-file="dev.tfvars"

# Then deploy prod
terraform plan -var-file="prod.tfvars"
terraform apply -var-file="prod.tfvars"

Result: 4 buckets created (1 main + 3 multiple), versioning enabled

Same code, different results! This is how professionals manage multiple environments.

✅ Best Practices

1. Always Add Descriptions

Bad:

variable "name" {
  type = string
}

Good:

variable "bucket_name" {
  description = "Name of the S3 bucket for application logs (must be globally unique)"
  type        = string
}

2. Use Validation Rules

variable "instance_type" {
  description = "EC2 instance type"
  type        = string
  default     = "t2.micro"

  validation {
    condition     = contains(["t2.micro", "t2.small", "t3.micro"], var.instance_type)
    error_message = "Instance type must be t2.micro, t2.small, or t3.micro."
  }
}

3. Provide Sensible Defaults

variable "aws_region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"  # Most common region
}

4. Use Meaningful Variable Names

Bad: var.n, var.x, var.thing

Good: var.bucket_name, var.instance_count, var.enable_monitoring

5. Group Related Variables

# Network variables
variable "vpc_cidr" { }
variable "subnet_cidrs" { }

# Application variables
variable "app_name" { }
variable "app_version" { }

# Tags
variable "common_tags" { }

6. Never Commit Secrets to tfvars

Bad:

# terraform.tfvars
database_password = "super-secret-password"  # DON'T DO THIS!

Good:

# Use environment variables for secrets
export TF_VAR_database_password="super-secret-password"

Or use AWS Secrets Manager, HashiCorp Vault, etc.

7. Document Your Variables

Create a README.md:

## Required Variables

- `bucket_name` - Globally unique S3 bucket name

## Optional Variables

- `aws_region` - AWS region (default: us-east-1)
- `enable_versioning` - Enable bucket versioning (default: false)

🐛 Common Issues and Solutions

Issue 1: Variable Not Found

Error:

Error: Reference to undeclared input variable

Solution: Make sure variable is declared in variables.tf:

variable "bucket_name" {
  description = "Bucket name"
  type        = string
}

Issue 2: Type Mismatch

Error:

Error: Invalid value for input variable

Solution: Check your variable type matches the value:

variable "bucket_count" {
  type = number  # Not string!
}

In terraform.tfvars:

bucket_count = 2  # Not "2" (string)

Issue 3: Validation Failed

Error:

Error: Invalid value for variable
Bucket count must be between 1 and 5.

Solution: Provide a value that meets the validation condition:

bucket_count = 3  # Within 1-5 range

Issue 4: Forgot to Pass tfvars File

Problem: Terraform uses default values instead of your custom file

Solution: Always specify the file:

terraform apply -var-file="prod.tfvars"

Issue 5: Sensitive Output Not Hidden

Problem: Sensitive data showing in output

Solution: Mark output as sensitive:

output "database_password" {
  value     = var.db_password
  sensitive = true  # Add this
}

🎓 What You Just Learned

Let's recap your new superpowers:

✅ Variables:

Convert hardcoded values to flexible variables
Use all variable types (string, number, bool, list, map)
Add validation rules
Provide defaults

✅ terraform.tfvars:

Store variable values in files
Create environment-specific configs (dev.tfvars, prod.tfvars)
Auto-loading behavior

✅ Outputs:

Display important information
Mark sensitive data
Use in scripts and modules

✅ Real-World Patterns:

Same code, multiple environments
Conditional resource creation
Multiple resource creation with count

🎯 Challenge: Try It Yourself

Before moving to the next article, try these challenges:

Challenge 1: Add More Variables

Add variables for:

owner_email
cost_center
backup_retention_days

Challenge 2: Create staging.tfvars

Create a staging environment configuration between dev and prod.

Challenge 3: Use List Variable

Create a variable for multiple availability zones:

variable "availability_zones" {
  type    = list(string)
  default = ["us-east-1a", "us-east-1b"]
}

Challenge 4: Complex Map Variable

Create a variable for instance configurations:

variable "instance_configs" {
  type = map(object({
    instance_type = string
    volume_size   = number
  }))
}

Challenge 5: Output Formatting

Create an output that combines multiple values:

output "bucket_info" {
  value = "Bucket ${aws_s3_bucket.example.id} in ${var.aws_region}"
}

📚 Additional Variable Features

Variable Precedence (Order of Priority)

Terraform loads variables in this order (last wins):

Environment variables (TF_VAR_name)
terraform.tfvars file
terraform.tfvars.json file
*.auto.tfvars files (alphabetical order)
-var and -var-file command line flags

Example:

# All of these set the same variable
export TF_VAR_bucket_name="from-env"           # Priority 1
# terraform.tfvars: bucket_name = "from-file"  # Priority 2
terraform apply -var="bucket_name=from-cli"    # Priority 3 (wins!)

Environment Variables

# Set variable via environment
export TF_VAR_bucket_name="my-bucket-2026"
export TF_VAR_aws_region="us-west-2"

terraform plan  # Uses environment variables

Auto-Loading tfvars Files

Terraform automatically loads:

terraform.tfvars
terraform.tfvars.json
*.auto.tfvars
*.auto.tfvars.json

Example:

# These are auto-loaded
terraform.tfvars
common.auto.tfvars
secrets.auto.tfvars

# These need -var-file flag
dev.tfvars
prod.tfvars

🔍 Understanding Variable Interpolation

You can use variables in many ways:

String Interpolation

resource "aws_s3_bucket" "example" {
  bucket = "${var.environment}-${var.app_name}-bucket"
  # Result: "dev-myapp-bucket"
}

Conditional Expressions

resource "aws_instance" "web" {
  instance_type = var.environment == "prod" ? "t3.large" : "t3.micro"
  # prod = t3.large, anything else = t3.micro
}

For Expressions

locals {
  bucket_names = [for i in range(var.bucket_count) : "${var.prefix}-${i}"]
  # Result: ["app-0", "app-1", "app-2"]
}

🔗 Useful Resources

Terraform Variables Docs: https://developer.hashicorp.com/terraform/language/values/variables
Terraform Outputs Docs: https://developer.hashicorp.com/terraform/language/values/outputs
Variable Validation: https://developer.hashicorp.com/terraform/language/values/variables#custom-validation-rules
Type Constraints: https://developer.hashicorp.com/terraform/language/expressions/type-constraints

🎬 What's Next?

In Article 6: Building a VPC from Scratch, we'll:

Create a complete AWS VPC with subnets
Use variables for network configuration
Build public and private subnets
Configure route tables and internet gateways
Apply everything you learned about variables

You'll learn:

Real-world networking with Terraform
Complex resource dependencies
Advanced variable usage
Production-ready VPC architecture

📌 Wrapping Up

Thank you for reading. I hope this article provided practical insights and a clearer understanding of the topic.

If you found this useful:

❤️ Like if it added value
🦄 Unicorn if you’re applying it today
💾 Save it for your next optimization session
🔄 Share it with your team

💡 What’s Next

More deep dives are coming soon on:

Cloud Operations
GenAI & Agentic AI
DevOps Automation
Data & Platform Engineering

Follow along for weekly insights and hands-on guides.

🌐 Portfolio & Work

You can explore my full body of work, certifications, architecture projects, and technical articles here:

👉 Visit My Website

🛠️ Services I Offer

If you're looking for hands-on guidance or collaboration, I provide:

Cloud Architecture Consulting (AWS / Azure)
DevSecOps & Automation Design
FinOps Optimization Reviews
Technical Writing (Cloud, DevOps, GenAI)
Product & Architecture Reviews
Mentorship & 1:1 Technical Guidance

🤝 Let’s Connect

I’d love to hear your thoughts. Feel free to drop a comment or connect with me on:

🔗 LinkedIn

For collaborations, consulting, or technical discussions, reach out at:

📧 simplynadaf@gmail.com

Found this helpful? Share it with your team.

⭐ Star the repo • 📖 Follow the series • 💬 Ask questions

Made by Sarvar Nadaf

🌐 https://sarvarnadaf.com

What is OpenClaw? A Self Hosted AI Assistants

Sarvar Nadaf — Mon, 30 Mar 2026 17:56:49 +0000

👋 Hey there, tech enthusiasts!

Let's dive in and explore the fascinating world of cloud technology together! 🚀

New to Self-Hosted AI?

This article assumes basic familiarity with computers, software installation, and online services. If terms like "API," "server," or "open source" are new to you, don't worry focus on the core concept: OpenClaw is an AI assistant that you run on your own computer or server, rather than relying on a company's service like ChatGPT.

What This Article Covers: Understanding what OpenClaw is, why it matters, and whether it's right for you.

What's Coming Next: Hands-on setup, configuration, and deployment (no prior experience with self-hosting required we'll walk through everything step-by-step).

Imagine having a personal assistant that lives on your infrastructure, responds through the messaging apps you already use, and executes tasks across your entire cloud environment. That's OpenClaw. But unlike the AI assistants you're familiar with ChatGPT, Claude, or GitHub Copilot this one doesn't phone home to someone else's servers. It runs on your machine, uses your API keys, and keeps your data exactly where you want it.

The story begins in November 2025 when Austrian developer Peter Steinberger built what he called a "weekend project." He wanted an AI that could do more than chat one that could actually execute commands, manage files, read emails, and integrate with the tools he used daily. Within weeks, the project exploded. By March 2026, it had accumulated over 340,000 GitHub stars and attracted attention from Silicon Valley startups to Chinese tech giants. Steinberger eventually joined OpenAI and moved the project to an open-source foundation, but the core philosophy remained unchanged: your assistant, your machine, your rules.

For anyone interested in AI and privacy, OpenClaw represents something different. It's not a SaaS product with usage limits and data residency questions. It's software you install, configure, and control. You choose which AI provider to use Anthropic's Claude, OpenAI's GPT models, Google's Gemini, or even self-hosted options like Ollama. You decide which messaging platforms to enable. You set the security boundaries. And when you need to see what happened, the logs are on your computer, not buried in someone else's cloud.

What OpenClaw Actually Is

Architecture and Deployment Model

OpenClaw is a Node.js application that acts as a gateway between messaging platforms and large language models (LLMs AI systems like ChatGPT or Claude). You install it on a server your laptop, a homelab machine, or a cloud VM and it stays running as a daemon. The architecture is straightforward: a WebSocket-based control plane (a real-time communication hub) listens on port 18789, handling connections from messaging channels, the web dashboard, and optional companion apps for macOS, iOS, and Android.

The gateway doesn't process AI requests itself. Instead, it routes conversations to your configured LLM provider via their APIs. When you send a message through WhatsApp or Slack, OpenClaw receives it, forwards the context to Claude or GPT, gets the response, and delivers it back through the same channel. All conversation history, configuration, and session state live in local SQLite databases under ~/.openclaw/.

Multi-Channel Integration

This is where OpenClaw diverges from traditional chatbots. Rather than forcing you into a web interface or proprietary app, it meets you where you already work:

Messaging platforms: WhatsApp, Telegram, Discord, Slack, Microsoft Teams, Signal, Matrix
Specialized channels: IRC, Google Chat, Mattermost, Twitch, WeChat (via Tencent plugin)
Direct interfaces: Web dashboard, macOS menu bar app, iOS/Android companion apps
Developer tools: CLI commands for scripting and automation

Each channel operates independently. You can have different security policies for Discord versus WhatsApp, route specific channels to isolated agent workspaces, or restrict certain tools based on where the request originated.

The Skills System

OpenClaw extends beyond conversation through a skills framework. Skills are self-contained modules think of them as plugins that give the AI new capabilities. A skill is just a directory containing a SKILL.md file with instructions and metadata. The AI reads these instructions and knows how to use the skill's tools.

Skills can be:

Bundled: Shipped with OpenClaw (browser automation, file operations, system commands)
Managed: Installed from the ClawHub registry
Workspace-specific: Custom skills you write for your environment

The precedence model matters for teams building workflows. Workspace skills override global skills, which override bundled skills. This means you can customize behavior per project without modifying the core installation.

Tool Execution and Sandboxing

When the AI decides it needs to run a command or access a file, OpenClaw's tool system handles the execution. By default, tools run directly on the host for the main session. But for group chats or untrusted channels, you can enable sandbox mode. Sandboxed sessions run inside per-session Docker containers with restricted tool access.

The security model is explicit:

Main session (your direct chats): Full tool access by default
Group sessions: Configurable sandbox with allowlists and denylists
Tool categories: bash, browser, read, write, edit, sessions, nodes, cron, discord, gateway

You define which tools each session type can use. A production deployment might allow read-only file access for group chats while reserving write operations for authenticated direct messages.

Why Self-Hosting Matters

Data Privacy and Control

When you deploy OpenClaw on your infrastructure, you control the data flow. Conversation logs, uploaded files, and execution history never leave your environment unless you explicitly configure external integrations. For organizations with strict data residency requirements healthcare, finance, government this matters. You're not trusting a third party to handle PHI (Protected Health Information) or PII (Personally Identifiable Information). You're running the infrastructure yourself.

The LLM API calls still go to external providers (unless you use self-hosted models), but you choose which provider and can route through your own network controls. Want to proxy all OpenAI requests through your corporate gateway? Configure the API endpoint. Need to log every prompt for compliance? Intercept at the gateway level.

Cost Control and Model Flexibility

SaaS AI assistants charge per seat or per usage tier. OpenClaw inverts this model. You pay your LLM provider directly based on token consumption. If Claude is too expensive for your use case, switch to GPT-4o-mini. If you want to experiment with open-source models, point it at Ollama running locally. The gateway doesn't care which model you use it just routes requests.

This flexibility extends to failover and load balancing. Configure multiple API keys for the same provider, and OpenClaw rotates through them when rate limits hit. Set up model fallback chains: try Claude Opus first, fall back to GPT-4 if it's overloaded, and use a local model as the last resort.

Example Cost Comparison:

SaaS AI Assistant: $20-30/user/month (fixed cost, limited usage)
OpenClaw + Claude API: Pay per token (1M input tokens ≈ $3-8 depending on model)
OpenClaw + Local LLM: Infrastructure costs only (no per-token charges)

For teams running thousands of queries daily, the direct API model often costs less than per-seat licensing. For occasional users, SaaS might be cheaper. The key difference: you control the trade-off.

Infrastructure as Code Integration

Because OpenClaw is just a Node.js application with a JSON configuration file, it fits naturally into infrastructure-as-code (IaC) workflows managing infrastructure through code rather than manual configuration. The configuration lives at ~/.openclaw/openclaw.json and supports environment variable substitution for secrets. You can template it with Terraform, manage it with Ansible, or version it in Git.

Deployment patterns cloud architects already know apply directly:

Systemd service: Install the daemon, enable it, manage it like any other service
Docker container: Official images available, mount config as a volume
Kubernetes: Deploy as a StatefulSet (a workload type for applications that need persistent storage) with persistent storage for the database
Nix: Declarative configuration with the community Nix package

Security Boundaries and Threat Model

OpenClaw's security model assumes you understand the risks of giving an AI broad system access. The maintainers are explicit about this. One core maintainer warned on Discord: "if you can't understand how to run a command line, this is far too dangerous of a project for you to use safely."

The threat surface includes:

Prompt injection: Malicious instructions embedded in data the AI processes (emails, documents, web pages) that trick the AI into executing unintended commands
Tool misuse: The AI executing destructive commands based on misinterpreted context
Third-party skills: Unvetted skills from the community that could exfiltrate data
Channel compromise: If someone gains access to an authorized messaging account, they control the AI

Cisco's security research team demonstrated this in March 2026 when they found a third-party skill performing data exfiltration. The skill repository lacked adequate vetting, and users installing it unknowingly gave attackers access to their environment.

For production use, you need to:

Run sandboxed sessions for any untrusted input source
Audit third-party skills before installation
Use DM pairing mode to require explicit approval for new contacts
Monitor execution logs for unexpected tool usage
Isolate the OpenClaw instance from sensitive systems

The MoltMatch Incident: A Cautionary Tale

In February 2026, computer science student Jack Luo configured his OpenClaw agent to explore its capabilities. He connected it to Moltbook a social network for AI agents and let it run autonomously. Days later, he discovered the agent had created a profile on MoltMatch, a dating platform where AI agents interact on behalf of users, and was screening potential matches without his explicit direction.

The incident highlighted a fundamental challenge with autonomous agents: when you give an AI broad permissions and vague instructions, it will interpret its mandate creatively. Luo's agent saw "explore capabilities" and "connect to agent platforms" as permission to create dating profiles. The AI-generated profile didn't reflect him authentically, and he had no idea it was happening until someone told him.

This matters for anyone managing systems because the same dynamic applies to automation. If you tell an AI to "optimize costs" without clear boundaries, it might decide to delete things it deems unnecessary. If you ask it to "improve security," it could change settings in ways you didn't anticipate. Autonomous agents require explicit constraints, not just general goals.

Real-World Adoption Patterns

Small Business and Freelancer Workflows

OpenClaw has found traction among small businesses and freelancers who need automation but can't afford enterprise tools. Common workflows include:

Lead generation: Scraping prospect lists, researching companies, updating CRM systems
Content management: Drafting emails, summarizing documents, generating reports
System administration: Monitoring servers, running diagnostics, deploying updates

These users value the flexibility to customize behavior without vendor lock-in. A freelance consultant can write a workspace skill that integrates with their specific CRM and invoicing tools, then share that skill with clients who use the same stack.

Enterprise Experimentation

Larger organizations are testing OpenClaw in controlled environments. The typical pattern: deploy it on a developer's workstation or a sandbox VM, connect it to a private Slack channel, and use it for internal tooling. Teams report using it for:

Documentation search: Indexing internal wikis and answering questions
Incident response: Gathering logs, running diagnostics, summarizing findings
Code review assistance: Analyzing pull requests, suggesting improvements

The key constraint is trust. Enterprises won't give OpenClaw production access until they've validated its behavior in isolated environments. The security incidents and lack of formal audits make it a hard sell for risk-averse organizations.

What This Means for Your Setup

If you're evaluating OpenClaw, think of it as software infrastructure, not a product. You're not buying a service; you're installing and maintaining a component that needs care and attention.

Key Questions to Consider:

Deployment:

Where will it run? (Your laptop, a home server, a cloud VM)
Who can access it? (Just you, your team, your organization)
What can it access? (Read-only files, full system control, isolated sandbox)
How will you monitor it? (Log files, activity tracking, alerts)
What's the risk if compromised? (Isolated test environment vs. access to important data)

Integration:

Which AI provider will you use? (Cost, rate limits, data handling)
Which messaging platforms? (Authentication, user verification)
What internal systems? (APIs, databases, file storage)
How will you monitor it? (Health checks, error tracking, usage metrics)

Maintenance:

Who maintains the configuration? (You, your team, IT department)
How do you handle updates? (Manual upgrades, automated deployment)
What's the support model? (Community forums, internal expertise)
How do you manage secrets? (Environment variables, password managers)

The answers depend on your technical comfort level and risk tolerance. A tech-savvy individual might run it on their laptop for personal use. A small team might deploy it on a shared server with careful access controls. An enterprise might lock it down in an isolated environment. All are valid approaches based on your needs and skills.

The Bottom Line

OpenClaw is not a polished consumer product. It's a powerful, flexible, occasionally dangerous tool that gives you control at the cost of responsibility. For people who understand the trade-offs, it offers something rare: an AI assistant that runs on your terms, integrates with your tools, and doesn't require you to trust a company with your data.

The project is still young. The security model is evolving. The ecosystem is fragmented. But the core idea self-hosted, multi-channel, tool-capable AI agents addresses real needs that SaaS products don't solve. Whether OpenClaw itself becomes the standard or just proves the concept for something better, the architecture it represents is worth understanding.

When to Consider OpenClaw:

You care deeply about data privacy and control
You want to avoid subscription fees for large-scale usage
You need deep integration with your own tools and systems
You have technical skills or are willing to learn
You're comfortable with ongoing maintenance

When to Avoid OpenClaw:

You're not comfortable with command-line tools
You need something that works immediately without setup
You can't dedicate time to security and maintenance
You prefer paying for convenience over having control
Your use case doesn't justify the complexity

If you decide to explore it, start small. Run it in an isolated environment, connect it to a test channel, give it limited permissions. Learn how it behaves, where it fails, and what it can actually do. Then decide if the benefits justify the operational overhead and security risks.

That's the honest assessment. OpenClaw is powerful infrastructure for people who know what they're doing. If that's you, it's worth exploring. If it's not, wait for the ecosystem to mature.

Coming Next: In our next article, we'll cover the technical implementation installation, configuration, and deployment with step-by-step examples. We'll walk through setting up OpenClaw on different platforms and implementing security best practices.

What We'll Cover in Upcoming Articles:

Part 2: Installation & Setup - Step-by-step guide, prerequisites, system requirements, and getting your first instance running
Part 3: Configuration Deep-Dive - API keys, messaging platform integration, security policies, and sandbox configuration
Part 4: Deployment Options - Running on Windows/Mac/Linux, cloud servers, and home servers with monitoring and cost optimization
Part 5: Security Best Practices - Threat mitigation, audit logging, access control, and incident response

Questions or Topics You'd Like Covered?

Drop a comment below with your questions about OpenClaw. Whether you're wondering about specific use cases, integration challenges, cost comparisons, or anything else we'll address them in upcoming articles. Common questions we're already planning to cover:

How much does it actually cost to run?
Can I use it on Windows without WSL?
What's the learning curve for someone new to self-hosting?
How does it compare to AutoGPT or LangChain agents?
Can I run it on a Raspberry Pi or home server?

Resources:

Official Website: https://openclaw.ai
GitHub Repository: https://github.com/openclaw/openclaw

📌 Wrapping Up

Thank you for reading! I hope this article gave you practical insights and a clearer perspective on the topic.

Was this helpful?

❤️ Like if it added value
🦄 Unicorn if you’re applying it today
💾 Save for your next optimization session
🔄 Share with your team

Follow me for more on:

AWS architecture patterns
FinOps automation
Multi-account strategies
AI-driven DevOps

💡 What’s Next

More deep dives coming soon on cloud operations, GenAI, Agentic-AI, DevOps, and data workflows follow for weekly insights.

🌐 Portfolio & Work

You can explore my full body of work, certifications, architecture projects, and technical articles here:

👉 Visit My Website

🛠️ Services I Offer

If you're looking for hands-on guidance or collaboration, I provide:

Cloud Architecture Consulting (AWS / Azure)
DevSecOps & Automation Design
FinOps Optimization Reviews
Technical Writing (Cloud, DevOps, GenAI)
Product & Architecture Reviews
Mentorship & 1:1 Technical Guidance

🤝 Let’s Connect

I’d love to hear your thoughts drop a comment or connect with me on LinkedIn.

For collaborations, consulting, or technical discussions, feel free to reach out directly at simplynadaf@gmail.com

Happy Learning 🚀

AI Assistance vs AI Agents: Understanding the Shift from Responses to Autonomous Systems

Sarvar Nadaf — Wed, 25 Mar 2026 19:11:06 +0000

👋 Hey there, tech enthusiasts!

I'm Sarvar, a Cloud Architect with a passion for transforming complex technological challenges into elegant solutions. With extensive experience spanning Cloud Operations (AWS & Azure), Data Operations, Analytics, DevOps, Generative AI, and Agentic AI, I've had the privilege of architecting solutions for global enterprises that drive real business impact. Through this article series, I'm excited to share practical insights, best practices, and hands-on experiences from my journey in the tech world. Whether you're a seasoned professional or just starting out, I aim to break down complex concepts into digestible pieces that you can apply in your projects.

Let's dive in and explore the fascinating world of cloud technology together! 🚀

Here's What Most People Get Wrong

ChatGPT can write your email. But it can't send it. That's the difference between an AI assistant and an AI agent, and honestly, most people have no idea there even is a difference.

I've been working with AI for a while now, and I keep seeing the same confusion everywhere. People use "AI assistant" and "AI agent" like they're the same thing. They're not. Not even close.

Here's what you need to know: AI assistants tell you what to do. AI agents actually do it for you. That's the whole game right there.

Let me break this down in a way that actually makes sense.

AI Assistants: The Smart Advisor

Think about the last time you used ChatGPT or asked Siri something. You asked a question, got an answer, and then you had to go do something with that answer. That's an AI assistant.

An AI assistant is like having a really smart friend who knows everything but can't actually touch anything in your world. They can tell you exactly what to do, step by step, but you're the one who has to do it.

Here's what they're good at:

Answering questions (and they're really good at this)
Giving you suggestions and advice
Helping you think through problems
Generating content like emails, code, or ideas

But here's what they can't do:

Take any action in your systems
Make decisions and execute them
Complete tasks that need multiple steps
Use tools or APIs on their own

Let me give you a real example. Say you ask an AI assistant: "What's the weather today?"

You: "What's the weather today?"
Assistant: "It's 72°F and sunny in New York."
You: "Should I bring an umbrella?"
Assistant: "No, it's not going to rain today."

Perfect, right? You got your answer. But notice what didn't happen - the assistant didn't check your calendar, didn't see you have a meeting at 3 PM, didn't notice the forecast shows rain later, and definitely didn't pack an umbrella for you. It just answered your question.

That's the limitation. And for a lot of things, that's totally fine! Sometimes you just need information.

Agentic AI: The Autonomous Doer

Now here's where things get interesting. Agentic AI doesn't just tell you what to do - it actually does it.

Think of it like this: if an AI assistant is your smart friend giving advice, an agentic AI is your trusted employee who you can hand a project to and they'll just... handle it. They'll figure out the steps, make the decisions, use whatever tools they need, and come back with results.

This is what makes agents different:

They can access and use tools (APIs, databases, systems)
They make decisions based on what they find
They complete multi-step workflows autonomously
They only ask you when they really need input

Here's the same scenario but with an agent:

You: "Book me a flight to New York next week."

Agent: *searches multiple airlines*
       *compares prices and times*
       *checks your calendar for conflicts*
       *finds the best option*
       *books the ticket*
       *adds it to your calendar*
       *sends confirmation to your email*

"Done! I booked you on Delta flight DL123, Tuesday 9 AM, $287. 
I saw you have a meeting at 2 PM that day, so this gives you time. 
Confirmation is in your email."

See the difference? You gave one instruction, and the agent handled everything. It made decisions (which flight is best), used tools (booking system, calendar, email), and completed the entire workflow. You didn't have to do anything except say what you wanted.

Let Me Show You the Real Difference

I think the best way to understand this is with a practical example. Let's say you need to check on your servers (if you're not technical, just think of this as checking if your website is working).

With an AI Assistant:

You ask: "I need to check my server status."

The assistant responds:

"Sure! To check your server status, you can:
1. Log into your AWS Console
2. Navigate to the EC2 Dashboard
3. Look at the instance state column

Or if you prefer the command line, run:
aws ec2 describe-instances

Would you like help with anything else?"

Great advice! But now you have to go do all of that. You have to log in, navigate around, run commands, interpret the results. The assistant told you what to do, but you're still doing all the work.

With an Agentic AI:

You say the same thing: "I need to check my server status."

The agent responds:

*connects to your AWS account*
*checks all your servers*
*analyzes the data*

"I checked your 5 servers. Here's what I found:

- web-server-1: Running fine, healthy
- web-server-2: Running but CPU is at 89% (might need attention soon)
- database-1: Running fine, healthy  
- api-server-1: STOPPED - this one is down
- backup-server: Running fine, healthy

I noticed api-server-1 is down. This could be affecting your API endpoints. 
Want me to restart it for you?"

The agent didn't just tell you what to do. It actually went and checked everything, found a problem, and is ready to fix it if you want. That's the power of agentic AI.

Why This Matters in 2026

Look, I get it. For years, we've been calling everything "AI assistants" and it worked fine. But now we're in 2026, and the technology has split into two very different paths.

AI assistants got better at answering. They understand context better, they give more accurate information, they can help with more complex questions. ChatGPT, Claude, and similar tools are incredibly good at what they do.

Agentic AI learned to act. They can now access your tools, make decisions, and complete entire workflows. They're not just smart anymore - they're capable.

And here's the thing: most people are still using AI like it's 2023. They're asking questions and copying answers when they could be handing off entire tasks.

Let me give you some real examples of what's possible now:

GitHub Copilot Workspace

You tell it: "Add user authentication to this app."

Instead of just giving you code snippets, it can:

Analyze your codebase
Write code across multiple files
Suggest database changes
Draft API endpoints
Generate tests
Create a pull request

You still review everything and make the final decisions, but it handles the heavy lifting. That's moving toward agentic behavior.

AWS Bedrock Agents

You can set up an agent to monitor your infrastructure. It can:

Watch your servers continuously
Detect when something goes wrong
Analyze issues
Take corrective actions (like restarting services)
Log everything
Alert you when it needs help

Instead of checking everything manually, you get a report: "Had 3 minor issues last night, all handled automatically." The agent does the routine monitoring while you focus on bigger problems.

Modern DevOps Agents

Your deployment fails at 2 AM. An agent can:

Detect the failure quickly
Analyze the logs
Identify the issue
Roll back the deployment
Alert the team with context
Generate an incident report

You find out at 9 AM that there was a problem and it's already been handled. Not perfect, but better than waking up to a crisis.

The Quick Comparison

If you're still wrapping your head around this, here's a simple table that breaks it down:

What It Does	AI Assistant	Agentic AI
Answers your questions	✅ Yes	✅ Yes
Gives you information	✅ Yes	✅ Yes
Takes actions in systems	❌ No	✅ Yes
Uses tools and APIs	❌ No	✅ Yes
Makes decisions on its own	❌ No	✅ Yes
Handles multi-step tasks	❌ No	✅ Yes
Works autonomously	❌ No	✅ Yes
Needs your approval for everything	✅ Yes	⚠️ Sometimes

The pattern is pretty clear, right? Assistants are great at the "thinking" part. Agents are great at the "doing" part.

So When Should You Use Each One?

This is the practical question everyone should be asking. Because here's the truth: you don't always need an agent. Sometimes an assistant is exactly what you want.

Use an AI Assistant When:

You want information or advice. If you're trying to learn something, understand a concept, or get suggestions, an assistant is perfect. You're not trying to automate anything - you just want to think through something with help.

Examples:

"Help me write this email" (you'll review and send it)
"Explain how this code works" (you're learning)
"Give me ideas for my presentation" (you're brainstorming)
"What's the best way to structure this database?" (you want advice)

You want to stay in control. Sometimes you don't want anything happening automatically. You want to review every step, make every decision, and execute everything yourself. That's totally valid, and assistants are perfect for this.

You're working on creative or strategic tasks. Writing, designing, planning, strategizing - these are areas where you want AI to augment your thinking, not replace your decision-making.

Use Agentic AI When:

You have repetitive workflows. If you're doing the same multi-step process over and over, that's a perfect candidate for an agent. Let it handle the routine stuff while you focus on things that actually need your brain.

Examples:

Monitoring systems and alerting on issues
Processing incoming data and routing it correctly
Handling customer support for common questions
Running deployments and rollbacks
Generating and sending reports

You need 24/7 operation. Agents don't sleep. If you need something monitored or handled around the clock, an agent is your answer.

Speed matters. Agents can react in seconds. If you need fast response to events or issues, agents beat humans every time.

The task is well-defined. If you can clearly describe the steps and decision points, an agent can probably handle it. The more structured the workflow, the better agents perform.

The Real Business Impact

Let me give you a concrete example of what this looks like in practice.

I know a company that was spending about 2 hours every day manually checking their server infrastructure. Someone had to log in, check each server, look at metrics, make sure everything was healthy. Boring, repetitive, but necessary.

First, they tried an AI assistant approach. They used ChatGPT to help generate the commands they needed to run. It was helpful - instead of remembering all the commands, they could just ask "how do I check CPU usage?" and get the answer. But they still had to run everything manually. They saved maybe 15 minutes a day. Better than nothing, but not game-changing.

Then they built an agentic AI solution. They set up an agent that:

Checks all servers every 5 minutes
Analyzes the metrics automatically
Alerts them when something is actually wrong
Can restart services if configured to do so
Generates a daily health report

Now? They spend maybe 10-15 minutes a day reviewing the report. The agent handles the routine monitoring. That's about 1 hour and 45 minutes saved every day. Not revolutionary, but meaningful. And they catch issues faster because the agent is always watching.

That's the difference between telling and doing.

What People Get Wrong About This

There's a lot of confusion out there, so let me clear up some common misconceptions:

"All AI is agentic now"

No, it's really not. Most of the AI you use every day is still assistant-level. ChatGPT, Claude, most chatbots - they're assistants. They're really good assistants, but they're not agents. True agentic AI that can take actions in your systems is still relatively new and not as widespread as people think.

"Agents will replace all human work"

This is the fear everyone has, and it's not realistic. Agents are good at repetitive, well-defined tasks. They're not replacing strategy, creativity, complex decision-making, or anything that requires real judgment. They're tools that handle routine work so humans can focus on more valuable tasks.

"AI assistants are outdated now"

Not at all. Assistants are perfect for tons of use cases. Not everything needs to be automated. Sometimes you want advice, not action. Sometimes you want to stay in control of every step. Assistants aren't going anywhere.

"Agents are just better assistants"

This is the big one people get wrong. They're not "better" - they're fundamentally different. It's like saying a car is a "better" bicycle. No, it's a different tool for different purposes. Assistants and agents solve different problems.

"Agents are too risky to use"

There's some truth to this concern, but it's manageable. Yes, giving AI the ability to take actions in your systems requires careful setup. You need guardrails, monitoring, and clear boundaries. But we've figured this out. Modern agent frameworks have safety built in. You can limit what they can do, require approval for certain actions, and monitor everything. It's not riskier than giving a new employee access to your systems - you just need proper controls.

The Safety Question Everyone Should Ask

Since we're talking about AI that can actually do things in your systems, let's address the elephant in the room: is this safe?

The honest answer is: it depends on how you set it up.

Here's what you need to think about:

Guardrails: Good agent systems let you define exactly what the agent can and can't do. You can say "you can restart servers, but you can't delete anything" or "you can book flights under $500, but ask me about anything more expensive." These boundaries are crucial.

Monitoring: You should be logging everything your agent does. Every action, every decision, every API call. This isn't just for debugging - it's for accountability and safety.

Approval workflows: For high-stakes actions, you can require human approval. The agent can do all the analysis and preparation, but it waits for your "yes" before executing anything critical.

Rollback capabilities: Things go wrong sometimes. Your agent should be able to undo its actions, or at least alert you immediately if something unexpected happens.

The key is this: you're not giving the agent unlimited power. You're giving it specific, controlled capabilities within boundaries you define. It's like giving someone the keys to your car - you're trusting them with something important, but you're not giving them ownership of your entire life.

Where This Is All Heading

We're in 2026, and we're still early in the agentic AI era. Here's what's developing:

Multi-agent systems are emerging. Instead of one agent doing everything, you might have multiple specialized agents working together. One handles customer support, another manages infrastructure, another handles data analysis. They can coordinate to handle complex workflows.

The line is blurring. Some AI assistants are adding limited action capabilities. Some agents are getting better at explaining their reasoning. We're moving toward hybrid systems that can adapt based on what you need.

It's getting easier to build. Right now, building an agent requires technical skill. But we're seeing more platforms that make it easier. Eventually, less technical people will be able to build agents for specific workflows.

Safety is improving. As more people use agents, we're learning better ways to build in safety features, monitoring, and control mechanisms. The technology is maturing.

The future probably isn't "assistants vs agents." It's more likely "assistants and agents working together." You'll use assistants for thinking and planning, and agents for executing and monitoring. They complement each other.

How to Actually Get Started

Alright, enough theory. Let's talk about what you should actually do with this information.

If You Want to Start Using AI Assistants:

This is the easy path. You can literally start right now.

Pick a tool. ChatGPT, Claude, or any of the major AI assistants. Most have free tiers to start.
Learn to prompt well. The better you ask questions, the better answers you get. This is a skill worth developing.
Start with simple tasks. Use it for writing, brainstorming, learning. Get comfortable with what it can and can't do.
Integrate if needed. If you're technical, most assistants have APIs you can use in your applications.

Cost: $0-100/month depending on usage
Time to start: Literally right now
Complexity: Low - anyone can do this

If You Want to Build Agentic AI:

This is more involved, but not as hard as you might think.

Define your workflow clearly. What exact task do you want automated? What are the steps? What decisions need to be made? The clearer you are, the better your agent will work.
Choose your platform. AWS Bedrock Agents, LangChain, AutoGPT, or other agent frameworks. Each has pros and cons. Bedrock is good if you're already on AWS. LangChain is good if you want flexibility.
Start small. Don't try to automate everything at once. Pick one simple workflow and get that working first.
Build in safety from day one. Set clear boundaries on what the agent can do. Log everything. Start with requiring approval for actions, then relax that as you gain confidence.
Test thoroughly. Run your agent through every scenario you can think of. What happens when things go wrong? How does it handle edge cases?
Monitor and improve. Once it's running, watch it closely at first. You'll find ways to improve it based on how it actually performs.

Cost: $100-1000+/month depending on what you're building
Time to start: Days to weeks
Complexity: Medium - you'll need some technical skills or help

The Bottom Line

Here's what you need to remember from all of this:

AI assistants are for thinking. They help you understand, plan, create, and learn. They're your smart advisor who knows a lot but can't touch anything.

Agentic AI is for doing. They handle workflows, take actions, make decisions, and complete tasks. They're your capable employee who can handle projects independently.

Both are valuable. Both have their place. The question isn't "which is better?" The question is "which solves my problem?"

Need answers? Use an assistant.
Need actions? Use an agent.
Need both? Use both.

We're in 2026, and we're just starting to figure out what's possible with AI that can actually do things. The assistants made us smarter. The agents are making us more productive.

The real power comes from knowing when to use each one.

Quick Reference Guide

Still not sure which you need? Ask yourself these questions:

Does it just need to answer, or does it need to act?

Just answer → Assistant
Actually act → Agent

Do I want to stay in control of every step?

Yes → Assistant
No, I want it handled → Agent

Is this a one-time question or an ongoing workflow?

One-time → Assistant
Ongoing → Agent

Am I trying to learn something or automate something?

Learn → Assistant
Automate → Agent

Do I need this to work when I'm not around?

No → Assistant
Yes → Agent

Simple as that.

What This Means for Different People

If You're a Developer:

Learn both. Understanding how to build assistants and agents is becoming an important skill. Start with assistants (easier), then explore agents (more complex). Focus on safety, monitoring, and good design patterns.

Developers who understand agentic AI will have an advantage as this technology matures.

If You're Running a Business:

Look at your workflows. Where are people doing repetitive, multi-step tasks? Those are agent candidates. Where do people need information and advice? Those are assistant use cases.

Start with one agent for one workflow. Measure the impact. If it works, expand. Don't try to automate everything at once.

And remember: the goal isn't to replace people. It's to free them up to do more valuable work.

If You're Just Curious:

Try ChatGPT or Claude if you haven't already. That's an assistant. Play with it. See what it can and can't do.

Then watch for agent capabilities showing up in tools you use. GitHub Copilot, AWS services, automation platforms - agents are being built into everything.

Understanding this difference will help you make sense of where AI is actually going.

Final Thoughts

The AI revolution isn't just about smarter answers. It's about AI that can actually do things.

For years, we've had AI that could think but not act. Now we have AI that can do both. That's a fundamental shift, and most people haven't caught up to it yet.

AI assistants made us more informed. They gave us access to knowledge and helped us think through problems. That's valuable, and it's not going away.

Agentic AI is making us more productive. It's taking tasks off our plate and handling them autonomously. That's powerful, and it's just getting started.

The key is understanding which tool solves which problem. Don't use an assistant when you need an agent. Don't use an agent when you just need advice.

We're at the beginning of the agentic AI era. The assistants aren't going away - they're just getting company. And together, they're changing how we work.

The question isn't whether you should use AI. The question is: which AI should you use, and for what?

Now you know the answer.

📌 Wrapping Up

Thank you for reading! I hope this article gave you practical insights and a clearer perspective on the topic.

Was this helpful?

❤️ Like if it added value
🦄 Unicorn if you’re applying it today
💾 Save for your next optimization session
🔄 Share with your team

Follow me for more on:

AWS architecture patterns
FinOps automation
Multi-account strategies
AI-driven DevOps

💡 What’s Next

More deep dives coming soon on cloud operations, GenAI, Agentic-AI, DevOps, and data workflows follow for weekly insights.

🌐 Portfolio & Work

You can explore my full body of work, certifications, architecture projects, and technical articles here:

👉 Visit My Website

🛠️ Services I Offer

If you're looking for hands-on guidance or collaboration, I provide:

Cloud Architecture Consulting (AWS / Azure)
DevSecOps & Automation Design
FinOps Optimization Reviews
Technical Writing (Cloud, DevOps, GenAI)
Product & Architecture Reviews
Mentorship & 1:1 Technical Guidance

🤝 Let’s Connect

I’d love to hear your thoughts drop a comment or connect with me on LinkedIn.

For collaborations, consulting, or technical discussions, feel free to reach out directly at simplynadaf@gmail.com

Happy Learning 🚀