DEV Community: Jatin Mehrotra

AWS Now Gives You a Free Sandbox Account - No Credit Card, No Cost, 8 Hours to Build (2026)

Jatin Mehrotra — Thu, 09 Jul 2026 08:59:39 +0000

AWS just announced free Sandbox environments which lets any AWS Builder Center user to provision time limited free AWS account without any credit card, personal account or any sort of cost involved. ABSOLUTELY FREE!!!

In this short blog i will walk through how you can use Sandbox Account and start learning on AWS Absolutely free through Workshops in just 5 steps.

Don't miss the important caveats around Sandbox environment in the last section.

Step 1 Log in to Builder Center

Log into Builder Center.

Note: You need to have AWS Builder Center Account in order to access Free Sandbox AWS Account.

Step 2 Access Workshops for Sandbox Env

In order to access sandbox AWS Account, you need to navigate to Workshops

You can choose which category, level, type of workshop based on your preference under Discover

Step 3 Choose Workshops

Once you selected your choice of workshop, click on Request free sandbox environment

Note: At a time you can only choose 1 Workshop as Sandbox environment is limited to 1 at a time

In my case i chose Building and scaling Agentic AI workflows
Click on Request

You will get 2 notification of environment being prepared and when it is ready to use.

Step 4 Accessing workshop

Under Your Workshops click on Go to Free sandbox environment
Complete the steps of joining workshop (accept the terms), you will be asked for email or builder center ID to recieve a code

Step 5 Access AWS Account through Console

Once you recieve code for workshop, you will see workshop screen, explaining about the workshop and how to proceed with workshop like actual building steps like clicking on Lab 1 to explain different modules
And from there you can access the AWS account through Console

Step 6 Go build on AWS

Once you are inside AWS console, you can follow the steps mentioned in the workshop and start learning and building on AWS

You can see resources are already created for you in your account for the workshop. This happened while account was being prepared

Few things to Note about sandbox accounts

At the time of launch you can only create, read, modify and delete resources mentioned in the workshop
Account permissions are scoped only to those resources
Your AWS account is time-boxed to 8 hours, starting from when the environment is prepared
One sandbox environment per week limit: You can request 1 sandbox environment per week. The weekly limit resets every Sunday.

I share such amazing AWS updates on DevOps, Kubernetes and GenAI daily over Linkedin, X. Follow me over there so that I can make your life more easy.

I Built PR Preview Environments With AWS Lambda MicroVMs and Cut Staging Costs by 78%

Jatin Mehrotra — Tue, 07 Jul 2026 04:15:07 +0000

On June 22, 2026, AWS released Lambda MicroVMs and I also covered how it works and how your agents can use it

That got me thinking, if Lambda MicroVMs suspend when idle and cost $0 while paused, why not use them as pull request preview environments?

Your typical staging runs 730 hours/month while reviewers use it for 50 minutes. You're paying for 93% idle time. Lambda MicroVMs flip this. Suspend when idle, resume in under 2 seconds, pay $0 while paused. The result: saves 78% vs standalone staging and 93% vs EKS, with zero infrastructure to manage, no cluster upgrades, no ALB configuration.

In this blog, I'll show you how to build dynamic PR environments with Lambda MicroVMs, real cost breakdown, and share the production lessons I learned building this solution.

For the people who want to jump straight in to trying the solution, follow my repo here

The Staging Bottleneck Problem
Architecture
The Solution
How to Test This
Cost: Lambda MicroVMs vs Shared Staging vs EKS
Production Tips for Lambda MicroVMs
From DevOps Perspective

The Staging Bottleneck Problem

Most enterprise teams have the same problem where staging environments are a bottleneck.

Developer A deploys their feature and QA starts testing. Developer B has to wait, then Developer C deploys another change. Suddenly, reviewers are testing multiple developers' code instead of a single pull request. Debugging becomes harder, feedback slows down, and the staging environment keeps running 24/7 even when nobody is using it.

Existing solutions all had trade-offs; shared staging, Kubernetes namespaces with always-on costs, or frontend-only preview platforms like Vercel and Netlify.

What I wanted:

Each PR gets its own isolated environment (VM-level, not container)
Reviewers click a link in the PR comment and see the app running with that PR's code
PR closes, environment dies (data cleaned up, nothing left running)
$0 compute cost while idle

Lambda MicroVMs gave me exactly this. Suspend/resume is the killer feature here. The MicroVM suspends after 5 minutes of no traffic and resumes in under 2 seconds when the reviewer comes back. While suspended: $0 compute.

Architecture

Every pull request creates its own MicroVM, receives a unique preview URL, automatically suspends when idle, and is destroyed when the PR closes.

The Solution

Project Structure

dynamic-pr-environment/
├── microvm-image/
│   ├── app.py              # Flask app + lifecycle hooks
│   ├── Dockerfile          # Based on al2023-minimal
│   └── requirements.txt    # flask, gunicorn, boto3
├── proxy/
│   └── handler.py          # Auth proxy Lambda (token validation + JWE forwarding)
├── infra/
│   ├── main.tf             # S3, DynamoDB, OIDC, IAM roles
│   ├── proxy.tf            # Proxy Lambda + Function URL + permissions
│   ├── outputs.tf
│   └── variables.tf
└── .github/workflows/
    ├── pr-deploy.yml        # Deploy on PR open/push
    └── pr-cleanup.yml       # Cleanup on PR close

Every pull request creates its own MicroVM, receives a unique preview URL, automatically suspends when idle, and is destroyed when the PR closes.

The App

The demo application is a full-stack task manager. A Flask web app serving HTML on the frontend with a REST API (POST/DELETE) that persists tasks to DynamoDB.

Each PR gets its own DynamoDB partition (PR#), so reviewers see only that PR's data. Reviewers can add and delete tasks directly in the preview environment. Once the PR is closed, all DynamoDB data for that partition is automatically cleaned up

App code

Two Flask servers in one process:

Port 9000 (hooks server, started first in background thread): Handles /ready, /run, /suspend, /resume, /terminate hooks
Port 8080 (app server, main thread): Serves the actual web app

# First Hooks server starts FIRST (Lambda sends /ready here during image build)
hooks_thread = threading.Thread(target=start_hooks_server, daemon=True)
hooks_thread.start()

# Second Then app server
app.run(host="0.0.0.0", port=8080, debug=False)

Why Lifecycle Hooks?

Lambda MicroVMs are snapshot-based. The image is frozen during build and restored whenever a MicroVM starts. Hooks let you customize each lifecycle stage.

Role of hooks in this solution:

/ready (during image build) - Flask app starts, responds 200, Lambda snapshots it. Build fails without this.
/run (MicroVM starts) - Receives PR number + branch + author, writes config to disk. One image, every PR.
/suspend (5 min idle, no reviewer) - MicroVM pauses. No reviewer = no cost.
/resume (reviewer comes back) - Reload config from disk. App resumes in under 2s.
/terminate (PR closed) - Delete all tasks in PR# from DynamoDB. Zero leftovers.

Without hooks, no way to configure a generic image per-PR at runtime or clean up data on shutdown.

@hooks_app.route("/aws/lambda-microvms/runtime/v1/run", methods=["POST"])
def hook_run():
    body = request.json or {}
    payload = json.loads(body.get("runHookPayload", "{}"))

    config = {
        "pr_number": payload.get("pr_number"),
        "branch": payload.get("branch"),
        "dynamodb_table": payload.get("dynamodb_table"),
    }

    with open(CONFIG_PATH, "w") as f:
        json.dump(config, f)

    return jsonify({"status": "ready"}), 200

The Auth Proxy (proxy/handler.py)

A proxy Lambda behind a Function URL. Validates the access token, generates a short-lived JWE auth token, and forwards the request to the MicroVM.

# 1. Validate access token from DynamoDB
result = table.get_item(Key={"PK": f"TOKEN#{microvm_id}", "SK": "ACCESS"})
if token != result["Item"]["token"]:
    return response(403, "text/plain", "Access denied")

# 2. Generate JWE auth token for MicroVM
token_resp = lambda_client.create_microvm_auth_token(
    microvmIdentifier=microvm_id,
    expirationInMinutes=5,
    allowedPorts=[{"port": 8080}],
)

# 3. Forward request with auth
req = urllib.request.Request(target_url, headers={"X-aws-proxy-auth": token})

For the demo, each PR gets a random access token stored in DynamoDB.

For production, replace this with GitHub OAuth or another identity provider to verify reviewers.

Note: For the demo i am using Lambda Function URL for simplicity but you can use API Gateway + Lambda Authorizer or CloudFront + Lambda@Edge if you need custom domains, rate limiting.

The GHA Workflow

.github/workflows/pr-deploy.yml is Triggered on pull_request: [opened, synchronize] with path filter on Blog/dynamic-pr-environment/**.

Deployment Workflow:

OIDC authenticate to AWS (no long-lived secrets)
Zip and upload code to S3
update-microvm-image (triggers new snapshot build)
Wait for UPDATED state
Terminate old MicroVM
run-microvm with runHookPayload containing PR metadata
Generate access token, store in DynamoDB
Post preview URL with token to PR comment

.github/workflows/pr-cleanup.yml does the cleanup.

Terraform code

Terraform creates

S3 bucket (code artifacts)
DynamoDB table (pr-environments, PAY_PER_REQUEST, partition key PK, sort key SK)
GitHub OIDC provider + IAM role
MicroVM build role + execution role
Auth proxy Lambda + Function URL

How to Test This

Prerequisites

Understanding of Lambda MicroVMs, covered in my previous blog
AWS account with Lambda MicroVMs access
Terraform >= 1.9
GitHub repo with Actions enabled

Step 1: Deploy Infrastructure

Edit terraform.tfvars with your values for github org and repo

cd infra
cp terraform.tfvars.example terraform.tfvars
terraform init && terraform apply

Step 2: Configure GitHub Secrets

From Terraform outputs, set these in your repo's Settings > Secrets:

Secret	Value
`AWS_ROLE_ARN`	GitHub Actions IAM role ARN
`S3_BUCKET`	Artifacts bucket name
`MICROVM_BUILD_ROLE_ARN`	Build role ARN
`MICROVM_EXECUTION_ROLE_ARN`	Execution role ARN
`PROXY_URL`	Auth proxy Function URL

Step 3: Create a PR

Make any change to a file under Blog/dynamic-pr-environment/ and open a PR. The workflow will:

Build the MicroVM image (~2-3 min first time, faster after)

Run the MicroVM
Post a comment with the preview URL

Step 4: Click the Preview URL

You should see your app running with the PR banner showing the branch name and PR number. Any code changes you push will trigger a new deployment automatically.

You can also raise parallel PR with different feature

You can also review them together as each microvm is isolated per PR

Step 5: Close the PR

The cleanup workflow terminates the MicroVM

deletes all DynamoDB data for that PR, and removes S3 artifacts.

Cost: Lambda MicroVMs vs Shared Staging vs EKS

Scenario

Region: us-east-1
Developers: 5
Pull Requests: 40/month
PR Lifetime: 2 days
Active Usage: 50 minutes per PR (3 review sessions)
Application: Flask (2 GB RAM, 1 vCPU baseline)

Note: t4g.small is the smallest Graviton instance with 2 GB RAM, matching the MicroVM's 2 GB memory baseline.

Per raw compute-hour, Lambda MicroVMs cost more than equivalent EC2. But with MicroVMs you're not managing instances, patching AMIs, configuring ALBs, debugging security groups, or upgrading K8s clusters. One API call to create, one to destroy.

The operational overhead of EC2/EKS staging is invisible on the invoice but real on your team's time.

Where the savings actually come from; pay for active minutes, not uptime

Here's the part that matters: a PR sits open for 2 days, but reviewers only interact with it for ~50 minutes total. Lambda MicroVMs suspend after 5 minutes idle and bill $0 while suspended, so you're paying for 33.3 active hours across the whole month (40 PRs x 50 min), not 730.

Monthly Cost Breakdown

All three solutions use the same app (Flask, 2GB RAM) with DynamoDB and S3. The difference is how the preview environment is hosted: MicroVM (suspend/resume, Function URL) vs EC2 (always-on, ALB) vs EKS (always-on, ALB, control plane).

Resource	Lambda MicroVMs	Shared Staging	EKS
Compute	$4.20 (33.3 active hrs)	$12.26 (t4g.small 24/7)	$12.26 (t4g.small 24/7)
Snapshot (storage + read/write)	$2.63	-	-
Load Balancer / Proxy	~$0.01 (Lambda + Function URL)	$16.93 (ALB)	$16.93 (ALB)
EBS Storage (20GB gp3)	-	$2.40	$2.40
EKS Control Plane	-	-	$73.00
DynamoDB	~$0.01	~$0.01	~$0.01
S3	~$0.05	~$0.05	~$0.05
Total	$6.90	$31.65	$104.65
Savings vs MicroVM	-	78%	93%

Note: These numbers are specific to this scenario. Your savings will vary based on active usage time, number of PRs, and suspend/resume frequency. The breakeven guidance below helps you estimate for your own workload.

When does this stop making sense? MicroVMs break even with shared staging at ~290 min active per PR (~5 hrs), and with EKS at ~1000 min (~16 hrs).If reviewers spend 5+ active hours per PR, always-on staging becomes cheaper. Typical PR reviews (30–60 minutes) stay well within the MicroVM sweet spot.

The takeaway: Lambda MicroVMs don't win because they're cheap compute, they're actually pricier per vCPU-hour than plain EC2. They win because the billing model matches the actual usage pattern of a PR preview: mostly idle, briefly active. Shared staging and EKS pay the same 24/7 whether anyone's looking or not. And you're not paying for an ALB or EBS volume that sits idle all weekend.

Production Tips for Lambda MicroVMs

Hooks must run on a separate port: Lambda routes user traffic to port 8080 by default. Lifecycle hooks (/ready, /run, /suspend, /resume, /terminate) should run on a separate port such as 9000, otherwise the image build can fail.
Treat deployments as immutable: Every new commit creates a new MicroVM image snapshot. Instead of updating a running MicroVM, terminate the old one and start a new instance from the latest snapshot.
Avoid network calls in the /run hook: The hook has a strict timeout of 1–60 seconds. DynamoDB cold connections can exceed that, so initialize data lazily after the application starts serving requests.
Use runHookPayload for per-PR configuration: Pass values such as the PR number, branch name, or environment settings when starting the MicroVM instead of baking them into the image.
readyTimeoutInSeconds can cause build failures: Setting this too aggressively may cause image builds to fail before the application is ready. Omitting it uses Lambda's default timeout, which is usually sufficient.
Default quotas are low: Lambda MicroVMs have conservative limits on concurrent MicroVMs. Small teams are unlikely to notice, but larger teams should request a quota increase.
ARM only (Graviton): Lambda MicroVMs currently support only arm64. Most Python, Node.js, and Go applications work without changes, but workloads with x86-only binaries will need to be ported.
Use relative URLs behind the proxy: If your application is served from a path such as /microvm-123/, use relative paths (fetch("tasks")) instead of absolute ones (fetch("/tasks")) so requests continue to work correctly.

From DevOps Perspective

In this blog we saw how Lambda MicroVMs turn preview environments into an on-demand resource, every PR gets an isolated environment, reviewers test changes independently, and compute costs drop to $0 while idle. Combined with lifecycle hooks, they provide a surprisingly practical foundation for production-grade preview environments.

Curious whether teams running Kubernetes-based preview environments today would actually switch, or if the 8-hour runtime cap on MicroVMs is a dealbreaker for long-lived PRs. What's blocking you?

I share such amazing AWS updates on DevOps, Kubernetes and GenAI daily over Linkedin, X. Follow me over there so that I can make your life more easy.

AWS Lambda MicroVMs: Your AI Agent Can Write Code. But Should It Execute It?

Jatin Mehrotra — Fri, 26 Jun 2026 11:54:18 +0000

Amazon Web Services(AWS) launched Lambda MicroVMs a new serverless compute primitive that provides VM level isolation, near-instant startup, suspend/resume, and state preservation for running untrusted or AI-generated code.

This blog will explain why do we need Lambda MicroVMs, how does it work & demo on how to build ephemeral sandboxes for safely executing the code by an AI agent.

People can also jump straight in to the Demo: Build Ephemeral Sandboxes for AI Agent Code Execution plus there is working code as well to build lambda microvm and executing agent code in the microvm

In the end shared Developer Perspective on how this changes the way we build modern application & its a must read.

Architecture
The Developer's Problem in 2026
The trade off before Lambda MicroVMs
What is Lambda MicroVM
Core Concepts for Lambda MicroVMs
Demo: Build Ephemeral Sandboxes for AI Agent Code Execution
Cost and Sizing
Developer Perspective

Architecture

The Developer's Problem in 2026

If you've ever built an application where users run code you didn't write like a data analytics platform, an interactive coding environment, a CI/CD runner, or even a coding assistant, you'll realise these type of applications have one thing in common.

They're executing code that isn't part of your application.

Code supplied by end users, generated by AI, or pulled from external sources. And they all share the same three requirements:

Isolation: One user's execution can't see, affect, or access another's
Speed: Environments need to start instantly because users won't wait
State that survives: Real work is multi-step. A user installs a package in step 1, loads data in step 2, and builds on that in step 3. If state resets between steps, the experience is broken. And when users go idle (switching tabs, taking a break), the environment should pause not die. When they come back, everything should be exactly where they left it.

The trade off before Lambda MicroVMs

Option	Isolation	Startup Speed	State Across Steps	State Preserved on Idle (Suspend/Resume)
EC2 instance per user	Strong (full VM)	30-60 seconds	Yes	Manual (hibernate setup, EBS backed only)
Containers (ECS/Fargate)	Moderate (shared kernel)	5-10 seconds	Within container lifetime	No container stops, state gone
Lambda Functions	Strong (Firecracker)	Sub-second	Stateless between invocations	No execution env frozen/recycled unpredictably
Lambda MicroVMs	Strong (Firecracker)	Sub-second	Up to 8 hours	Auto suspend/resume memory + disk snapshotted

EC2 provides strong isolation but starts slowly and hibernation is a hack.
Containers are fast, but sharing a kernel with untrusted code means significant hardening work.
Lambda Functions are instant and isolated, but they're designed for event driven request response not multi step sessions where step 2 depends on what was installed in step 1.
Lambda MicroVMs give you all four: strong isolation, sub-second start, persistent state across steps, and automatic suspend/resume that preserves everything while you stop paying for compute.

What is Lambda MicroVm

Think of serverless compute environment which provides:

VM-level isolation: Each sandbox runs its own kernel (Firecracker) with full operating system capabilities (for e.g. installing system packages or mounting filesystems). No shared memory, no shared filesystem
snapshot-based rapid startup speeds
State persistence up to 8 hours: Memory, disk, running processes all preserved across interactions within a session.
Auto suspend/resume: When idle, MicroVMs suspend. When traffic returns, they resume with everything intact.
fine-grained control over ingress networking (port access, support for HTTP/2, gRPC, WebSockets) and egress networking (public internet access and VPC access).

Note: It's not a replacement for Lambda Functions, but a sibling designed for a different workload which also means no infrastructure to manage.

Core Concepts for Lambda MicroVMs

AWS Lambda MicroVMs uses several resource types that as a dev we need to create and manage. Understanding the following provides the foundation for building applications with MicroVMs.

MicroVM: isolated compute environment for a single tenant, user session, or job with full operating system capabilities, providing near-instant launch and resume

MicroVM Image: Defines the application environment of a MicroVM. When you create an image, Lambda builds it into a snapshot that enables near instant launch

Snapshot: A freeze of memory + disk + running processes + network connections.

Network Connectors: Controls inbound & outbound traffic for MicroVM. You associate connectors with a MicroVM at run time. Traffic can be flown to VPC with custom connector.

Auth Token: A short-lived JWE token scoped to a specific MicroVM + allowed ports + expiry. Required in X-aws-proxy-auth header for every inbound request.

How the MicroVM Image Build Works

This is the part that makes sub-second startup possible. When you create or update a MicroVM image, Lambda doesn't just store your Dockerfile it runs through the entire build, starts your app, and freezes everything in place.

Here's what happens behind the scenes:

Lambda spins up a fresh VM using the base image you specified (al2023-minimal)
Runs your Dockerfile instructions like installs packages, copies files, configures the environment
Starts your application
Optionally waits for your app to signal it's ready (HTTP 200 on the /ready hook)
Takes a full Firecracker snapshot of memory, disk, running processes, network connections

The resulting snapshot becomes your image. Future MicroVMs restore directly from it instead of booting from scratch, which is why startup is nearly instantaneous.

The gotcha: Since all MicroVMs from the same image start from identical state, anything unique you create during build (UUIDs, secrets, random seeds, DB connections) gets shared across every MicroVM. The fix: don't generate unique stuff at build time. Do it in the /run lifecycle hook which runs after each individual MicroVM starts.

MicroVM Lifecycle States & Hooks [IMP to build reliable applications]

A MicroVM transitions through these states: PENDING → RUNNING → SUSPENDED → RESUME → TERMINATED

Each transition has a corresponding lifecycle hook, places where you plug in your setup or cleanup code:

Hook	When it runs	Use it for
`/run`	After snapshot restore, before accepting traffic	Generate per instance unique IDs, reestablish DB connections, seed random values/payload that shouldn't be shared across MicroVMs from the same image
`/resume`	After waking from SUSPENDED state	Refresh expired tokens, reconnect pools that timed out during suspend
`/suspend`	Before memory+disk gets checkpointed	Flush write buffers, gracefully close connections you don't want frozen mid handshake
`/terminate`	Before resources are released	Persist final results to S3 or external DB, send completion callbacks

Note: If your /run hook fails or times out, the MicroVM goes straight to TERMINATED without ever reaching RUNNING, your users won't see an error. Always add timeout handling and logging inside your hooks.

Demo: Build Ephemeral Sandboxes for AI Agent Code Execution

I have created a repo which has the working script to try the following demo. Choice is yours you can use the console or the script.

┌───────────────────────────────────────────────────────────────┐
│              YOUR APPLICATION                                   │
│                                                                │
│   1. create sandbox  ─── boto3 API call ──▶  Lambda Control   │
│   2. send work       ─── HTTPS POST ──────▶  MicroVM endpoint │
│   3. destroy sandbox ─── boto3 API call ──▶  Lambda Control   │
│                                                                │
└───────────────────────────────────────────────────────────────┘
                                                     │
                     ┌───────────────────────────────▼───────────┐
                     │         Firecracker MicroVMs              │
                     │                                           │
                     │  ┌──────────┐  ┌──────────┐ ┌──────────┐│
                     │  │ User A   │  │ User B   │ │ Job X    ││
                     │  │ (Python  │  │ (Node.js │ │ (CI run) ││
                     │  │  REPL)   │  │  IDE)    │ │          ││
                     │  └──────────┘  └──────────┘ └──────────┘│
                     │  Separate kernel, memory, disk per VM    │
                     └──────────────────────────────────────────┘

For this demo, I'm building an AI coding assistant that creates sandboxes on demand, executes generated code in them, and destroys them after. But same pattern works for any use case like web app backend, CI runner, or interactive platform.

Pre-requisites:

AWS account with access to a supported region (us-east-1, us-east-2, us-west-2, ap-northeast-1, eu-west-1)
Python 3.12+, strands-agents SDK installed (pip install strands-agents)
S3 bucket for the code artifact
Update aws-cli to 2.35.11

Step 1: The Sandbox Image (Dockerfile)

This defines what's available inside every sandbox. Think of it like setting up a workspace with all the tools someone might need. You don't know what they'll build but you know what they'll have access to.

FROM public.ecr.aws/lambda/microvms:al2023-minimal

# System dependencies (curl-minimal is already in the base image)
RUN microdnf install -y python3 python3-pip git && microdnf clean all

# Python packages available to AI-generated code
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Code executor
WORKDIR /sandbox
COPY executor.py .

EXPOSE 8080
CMD ["python3", "executor.py"]

Step 2: The Executor (Runs Inside the MicroVM)

A HTTP server that receives code, runs it as a subprocess, and returns output. This is the "interface" between the outside world and the sandbox interior.

import subprocess
import os
import logging
from flask import Flask, request, jsonify

app = Flask(__name__)
logging.basicConfig(level=logging.INFO)

WORK_DIR = "/workspace"
os.makedirs(WORK_DIR, exist_ok=True)

@app.route("/execute", methods=["POST"])
def execute():
    payload = request.json
    code = payload.get("code", "")
    timeout = payload.get("timeout", 30)

    app.logger.info(f"=== Incoming request: {len(code)} chars, timeout: {timeout}s ===")
    app.logger.info(f"Code:\n{code}")

    code_file = os.path.join(WORK_DIR, "run.py")
    with open(code_file, "w") as f:
        f.write(code)

    try:
        result = subprocess.run(
            ["python3", code_file],
            capture_output=True,
            text=True,
            timeout=timeout,
            cwd=WORK_DIR,
        )
        return jsonify({
            "stdout": result.stdout,
            "stderr": result.stderr,
            "exit_code": result.returncode,
        })
    except subprocess.TimeoutExpired:
        return jsonify({
            "stdout": "",
            "stderr": f"Execution timed out after {timeout}s",
            "exit_code": -1,
        }), 408

@app.route("/health", methods=["GET"])
def health():
    return jsonify({"status": "ready"})

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

The security doesn't come from this code, it comes from the VM boundary around it. Even if submitted code achieves root inside the sandbox, it's root of an isolated Linux with no visibility into other MicroVMs.

Step 3: Build the MicroVM Image

Console way

you need to mention s3 bucket where zip of DOCKERFILE is present.

Note: If you are using hooks then you need to enable ports for these hooks in step 2 in your application code

CLI method

Package, upload, and build:

# Package
zip sandbox-code.zip Dockerfile executor.py requirements.txt

# Upload to S3
aws s3 cp sandbox-code.zip s3://my-bucket/sandbox-code.zip

# Create the MicroVM image
aws lambda-microvms create-microvm-image \
  --name python-sandbox \
  --code-artifact uri=s3://my-bucket/sandbox-code.zip \
  --base-image-arn arn:aws:lambda:us-east-1:aws:microvm-image:al2023-1 \
  --build-role-arn arn:aws:iam::123456789:role/MicroVMBuildRole \
  --region us-east-1

Lambda builds the container, starts Flask, confirms it's listening, and takes the Firecracker snapshot. Build logs go to CloudWatch under /aws/lambda-microvms/python-sandbox.

Once status is SUCCESSFUL, every future MicroVM launched from this image starts preinitialized.

Step 4: Wire It Into Your Application

Console

You can run microvm manually as well, configure it as per your needs and even pass payload during /run hook. For our demo i am controlling all this aspect within agent sandbox code.

Note: Execution role decide what the microvm can access within AWS. For the blog my role provide ADMIN priveleges but for production permission needs to be trimmed down.

CLI/Code way

Here's where your application creates, uses, and destroys sandboxes. I'm using Strands Agents SDK for this demo to build the agent directly manages MicroVM lifecycle through three tool functions.

# agent.py
import boto3
import requests
from strands import Agent
from strands.tools import tool

# --- Configuration ---
REGION = "us-east-1"
IMAGE_NAME = "python-sandbox"
STACK_NAME = "microvm-sandbox-roles"

# Auto fetch ARNs from AWS
sts_client = boto3.client("sts", region_name=REGION)
cfn_client = boto3.client("cloudformation", region_name=REGION)

ACCOUNT_ID = sts_client.get_caller_identity()["Account"]
SANDBOX_IMAGE_ARN = f"arn:aws:lambda:{REGION}:{ACCOUNT_ID}:microvm-image:{IMAGE_NAME}"

stack_outputs = cfn_client.describe_stacks(StackName=STACK_NAME)["Stacks"][0]["Outputs"]
SANDBOX_ROLE_ARN = next(
    o["OutputValue"] for o in stack_outputs if o["OutputKey"] == "ExecutionRoleArn"
)

lambda_client = boto3.client("lambda-microvms", region_name=REGION)


@tool
def create_sandbox() -> dict:
    """Create an isolated MicroVM sandbox for executing code.
    Returns sandbox_id and endpoint URL."""
    response = lambda_client.run_microvm(
        imageIdentifier=SANDBOX_IMAGE_ARN,
        executionRoleArn=SANDBOX_ROLE_ARN,
        idlePolicy={
            "maxIdleDurationSeconds": 300,
            "suspendedDurationSeconds": 1800,
            "autoResumeEnabled": True,
        },
    )
    return {
        "sandbox_id": response["microvmId"],
        "endpoint": f"https://{response['endpoint']}",
    }


@tool
def run_code(sandbox_id: str, endpoint: str, code: str) -> dict:
    """Execute Python code in the isolated sandbox.
    State persists between calls files and packages carry over."""
    token_response = lambda_client.create_microvm_auth_token(
        microvmIdentifier=sandbox_id,
        expirationInMinutes=5,
        allowedPorts=[{"port": 8080}],
    )
    token = token_response["authToken"]["X-aws-proxy-auth"]

    resp = requests.post(
        f"{endpoint}/execute",
        headers={
            "X-aws-proxy-auth": token,
            "X-aws-proxy-port": "8080",
        },
        json={"code": code, "timeout": 60},
    )
    return resp.json()


@tool
def destroy_sandbox(sandbox_id: str) -> str:
    """Terminate the sandbox. All memory, disk, and processes destroyed."""
    lambda_client.terminate_microvm(microvmIdentifier=sandbox_id)
    return "Sandbox terminated. All state destroyed."


# The agent — decides what code to write based on user's question
agent = Agent(
    model="us.anthropic.claude-sonnet-4-5-20250929-v1:0",
    system_prompt="""You are a data analysis assistant. When the user asks you
    to analyze data, write code, or compute anything:
    1. Create a sandbox with create_sandbox()
    2. Write Python code and execute it with run_code()
    3. You can call run_code() multiple times state persists
    4. When finished, destroy the sandbox with destroy_sandbox()
    Always run code in the sandbox. Never just describe what code would do.""",
    tools=[create_sandbox, run_code, destroy_sandbox],
)

Note: This same pattern without an agent would be a direct boto3 call from your web backend, the three API calls are the same regardless of what's triggering them.

Step 5: See It Work (Multi-Step State Persistence)

Try these 3 prompts each one validates a different MicroVM capability:

Prompt 1: Prove it's running in an isolated VM (not locally)

You: Run code to print os.uname() and the contents of /etc/os-release

You can also see the logs of what code is being executed using the microvm image logs

This proves VM-level isolation. The response will shows Linux aarch64 Amazon Linux 2023 instead of your local macOS.

Prompt 2: Prove filesystem state persists between calls

You: Write a file /workspace/state.txt with the current timestamp

     ... wait ~70 seconds (MicroVM auto-suspends after 60s idle) ...

You: Read /workspace/state.txt and print its contents

Micromvm automatically suspends after maxIdleDurationSeconds

When user sends prompt again it resumes instaneously.

After suspension the microvm resumes for the above request

MicroVM wrote a file, went idle, auto suspended (memory + disk snapshotted), and when you came back with a new request it auto resumed with the file still there.

Note: This uses agent_suspend_demo.py which has a 1 minute idle timeout (instead of the default 5 minutes) so you can see the suspend/resume cycle quickly. Run it with python agent_suspend_demo.py.

Prompt 3: Prove packages are pre installed from the snapshot

You: Create a CSV file at /workspace/sales.csv with columns: product, revenue, region for 10 sample rows. Then read it back with pandas and tell me the top product by total revenue.

cw logs

This proves snapshot based startup. Pandas is immediately available because it was installed during image build and captured in the Firecracker snapshot.

After each prompt the agent calls destroy_sandbox() proving lifecycle control (one API call, all state gone, no residual data).

Cost and Sizing

In our demo we used the default MicroVM size (2 GB / 1 vCPU). Lambda MicroVMs uses a baseline peak sizing model. You configure a baseline, and the MicroVM can burst up to 4x during peak load:

Baseline is set via the memory parameter when creating your MicroVM image.

Mimimum baseline size starts with 0.5 GB & max memory upto 8 GB

What you actually pay:

MicroVM State	What You Pay
Running (executing code)	Baseline compute rate per second
Burst (CPU exceeds baseline)	Per second charge for resources above baseline
Suspended (idle, state snapshotted)	Snapshot storage only nearly free
Terminated	Nothing

For detailed pricing, see Lambda MicroVMs pricing.

Developer Perspective

In this blog we saw how AWS Lambda MicroVMs is solving the fundamental problem of modern applications; WHERE DOES THAT UNTRUSTED CODE EXECUTE SAFELY?

With the rise of coding assistants, autonomous agents, and platforms that let users run arbitrary scripts this problem isn't going away. It's getting worse. Lambda MicroVMs gives that answer without making you build the isolation layer yourself.

The scary cases haven't gone away, users discovering /proc, AI agents executing rm -rf /, malicious packages attempting container escapes. The difference is these are now AWS's problem. The blast radius is a disposable VM that you terminate with one API call.

The fastest way to build with Lambda MicroVMs with best practices would be to use the skill provided by lambda team wi to your agent like Kiro or Claude

npx skills add https://github.com/aws/agent-toolkit-for-aws/tree/main/skills/specialized-skills/serverless-skills/aws-lambda-microvms --yes --global

Let me know what you think in the comments and if you build something with Lambda MicroVMs, I'd genuinely love to see it.

I share such amazing AWS updates on DevOps, Kubernetes and GenAI daily over Linkedin, X. Follow me over there so that I can make your life more easy.

AWS DevOps Agent is not here to replace you! It's here to compliment you by acting like 24/7 on call engineers finding root cause for issues while you build new feature and innovate. More details in the blog on how to setup with 2 Demo.

Jatin Mehrotra — Mon, 08 Dec 2025 11:56:45 +0000

Jatin Mehrotra for AWS Community Builders

Dec 6 '25

AWS DevOps Agent Explained: Architecture, Setup, and Real Root-Cause Demo (CloudWatch + EKS)

#ai #aws #agents #awsdevopsagent

8 min read

AWS DevOps Agent is not here to replace you! It's here to compliment you by acting like 24/7 on call engineers finding root cause for issues while you build new feature and innovate. More details in the blog on how to setup with 2 Demo.

Jatin Mehrotra — Mon, 08 Dec 2025 10:56:34 +0000

Jatin Mehrotra for AWS Community Builders

Dec 6 '25

AWS DevOps Agent Explained: Architecture, Setup, and Real Root-Cause Demo (CloudWatch + EKS)

#ai #aws #agents #awsdevopsagent

8 min read

AWS DevOps Agent Explained: Architecture, Setup, and Real Root-Cause Demo (CloudWatch + EKS)

Jatin Mehrotra — Sat, 06 Dec 2025 22:19:36 +0000

Amazon Web Services(AWS) launched Frontier agents that are autonoumous systems during re:Invent 2025 which achieve various use cases goals, scale massively to tackle concurrent tasks, and run persistently for hours or days without intervention.

In this blog we will talk about one of the frontier agents i.e AWS DevOps Agent !!!

This blog will explain what is AWS DevOps Agent, its architecture & components, security & demo for investigating Ec2 CPU spike (CW alarm) and EKS Pod error.

People can also jump straight in to investigating Cloudwatch alarms or EKS errors plus there is tf repo as well for EKS

In the end also shared DevOps Engineer Perspective on the fear of being replaced and its a must read.

Architecture
What is AWS DevOps Agent
How to maximize Agent's Effectiveness
AWS DevOps Agent working architecture
Resource Discovery by AWS DevOps Agent
Demo 1: Investigate Cloudwatch Alarm
Demo 2: Investigate EKS errors
DevOps Engineer perspective

Architecture

What is AWS DevOps Agent

Think of AWS DevOps Agent like an 24/7 continuously learning Autonomous on call Engineer which has all the tools(more about it later) primarily to investigate incidents, finds root causes, provides mitigation plan and provide you Prevention as well.

At the moment it cannot fix the incidents on its own. Of course a human will be needed to fix the root cause.

How it investigates and find the root cause

To investigate root cause and give recommendations it needs understanding and relationships of the infrastructure & applications called as topology inside AWS account.

The understanding of infrastructure holistically becomes the CONTEXT for this DevOps Agent.

How to maximize Agent's Effectiveness

While the topology provides important context during investigations, AWS DevOps Agent is not limited to investigating only the resources shown in the topology.

The agent may use additional data sources, such as AWS service APIs or connected observability tools, to investigate resources that are not in the application topology.

And that is why AWS has given option to add capabilities to maximize Agent's effectiveness by :

Connect multiple AWS accounts
Connect CI/CD pipelines through repo like Github/GitLab
MCP servers
Telemetry sources like Datadog, New Relic
Ticketing and chat like serviceNow and slack
Even EKS (demo over here)

Note: We can also provide runbooks as pre-loaded guidance/hints to enhance investigation performance to provide investigation hints and guidance.

IN THE END IT ALL ABOUT UNDERSTANDING RELATIONSHIPS ABOUT YOUR RESOURCES

AWS DevOps Agent working architecture [IMP]

Operates through a dual-console architecture.
Admins uses management Console to create and manage Agent Spaces,configure capabilities & set up access controls.
Operations teams uses AWS Agent web app to interact with agent and start investigation.

DevOps Agent Spaces

DevOps Agent spaces is logical container/boundary that defines what tools and infrastructure AWS DevOps Agent has access to.

When you create an Agent Space, you define which AWS accounts the agent can access, which external tools it can connect to, and which users in your organization can interact with the agent.

Admins configure the Agent Space through the AWS Management Console

Security Aspect of Agent Spaces:

Each Agent Space uses dedicated IAM roles that grant access only to specific AWS accounts and resources
You control which users or groups can access each Agent Space.
Information from one Agent Space is not visible or accessible from another Agent Space

DevOps Agent Web App

Operations team uses web apps for daily incident response activities.

Security Aspect of Web Apps:

IAM identity Center (user Access): centrally manage user access to the DevOps Agent Space web apps even federate with external identity providers. MFA support is included
IAM authentication link (Admin access): direct access to the web app from the AWS Management Console using your existing console session.

Resource Discovery by AWS DevOps Agent

Until this point we understand Agent's context starts with Resource discovery(topology) and it does by 2 ways

CloudFormation stacks: By default Agent will list all of the CloudFormation stacks and their resources. Resources created by CDK is also supported.
Resource Tags: Resources not deployed from CloudFormation (like Console or TF), will be discovered by AWS Tag Key and value pairs to include in topology.

Demo 1: Investigate Cloudwatch Alarm

Pre-requisite

For the brevity of the blog I won't cover how to create agent space. Pretty straightforward and you don't need aws organizations to do this.
Cloudformation basics
Access to us-east-1 region. DevOps agent is available only in this region.
I used single standalone account.

1) Investigate Cloudwatch Alarm

Deploy the cloudformation stack.

This CFN template is creating an security group, key-pair for ssh ec2 instance with startup script to do CPU stress test, CW alarm for CPU utilization, auto shutdown ec2 instances after 2 hours.
After ec2 instance is created wait for 5-10 minutes to trigger cloudwatch alarm
You can also SSH into instance and run the stress test manually as well ./cpu-stress-test.sh

Once stack is deployed DevOps Agent automatically identifies the new resources.

Access Web App

After creating agent spaces click on View Details

Click on Web app or directly Operator Access link

Investigation Root Cause

Once your Cloudwatch Alarm is triggered go to Web app and under incident Response click on latest alarm and hit on Start Investigation

_DevOps Agent is smart enough to fill the completed prompt. Agent will figure out the steps on its own and give you the final ROOT CAUSE
_

Well I also found an interesting finding when the time between 2 alarms was around 40 minutes agent was unable to find the root cause and I had to rerun the investigation.

Mitigation plan

Well since it was user initiated agent was smart enough to give no mitigation plan

Prevention

Since it was a very straightforward demo with user initiated errors so it wasn't enough to generate prevention recommendations

Investigation gaps

One of my favorite feature which even docs does not cover is Investigation gaps.

No investigation can be perfect and that's what investigation gaps fill in to tell when it cannot cover extreme details due to absence of resource at infrastructure level for example in this absence of ssh agent, CloudWatch log groups then it tells those details.

Using Chat

You can ask more detailed questions using chat in natural language

Demo 2: Investigate EKS errors

Adding Capability;Give DevOps Agent access to EKS clusters

Go to Capabilities and click Edit
As we learned earlier Agent space IAM role control the access of AWS resources of Agent, Click view role permissions

Cope the IAM Role ARN. You will need to add an access entry in the EKS cluster with AmazonEKSAdminViewPolicy ( we have tf code for it)

Go to terraform code and replace the arn in terraform.vars
After the terraform code is finished you can see nginx has imagePullBack Error which is intentional

As we learn earlier we are deploying resources by terraform so we need to add tags in the Agent space so that DevOps agent can find the resources.

I am using the same tags in my terraform code as well. Click Save. You will see Agent Space automatically finds the Newly created resources.

Go to Web App

Now we will ask the AWS DevOps Agent what is the cause of this error.

Agent successfully investigated the EKS cluster and find out the root cause

Not only it found out the root cause it also gave the mitigation steps also rollback steps if mitigation cause issues.

DevOps Engineer perspective

We explored how the AWS DevOps Agent reduces MTTR, prevents future incidents through recommendations, and pinpoints root causes.

The Agent’s effectiveness comes from its deep understanding of your infrastructure—both inside AWS accounts and across external systems. It would be interesting to connect MCP server to enhance context even further.

Though Devops Agent is in preview and its free but has some limits

Is it secure? if configured Absolutely yes! because administrators control what DevOps Agent can access in AWS account, Agent IAM permissions control access to its feature and capabilities.

Will it replace you? Absolutely not! because still need an engineer to fix the issues, an engineer to build new features for the infrastructure & need that understanding of infrastructure how to rollback if things goes wrong.

Let me know what do you think?

I share such amazing AWS updates on DevOps, Kubernetes and GenAI daily over Linkedin, X. Follow me over there so that I can make your life more easy.

AWS launched EKS Capabilities to simplify Kubernetes with a quick on/off model. It runs Argo CD, ACK, and kro for you—no installs or upgrades. I tested it by deploying an S3 bucket using ACK + Argo CD using EKS Capabilities. Terraform Repo link inside.

Jatin Mehrotra — Wed, 03 Dec 2025 16:40:44 +0000

Jatin Mehrotra for AWS Community Builders

Dec 1 '25

I Created S3 Buckets Using ArgoCD , ACK with EKS Capabilities—No Controllers Installed.

#aws #kubernetes #eks #reinvent2025

9 min read

I Created S3 Buckets Using ArgoCD , ACK with EKS Capabilities—No Controllers Installed.

Jatin Mehrotra — Mon, 01 Dec 2025 15:08:10 +0000

Amazon Web Services (AWS) EKS team has introduced EKS capabilities on Dec 1, 2025 to help developers for writing their code by offloading Kubernetes management of resources like Argo CD, ACK, kro by a simple turn/off switch.

Amazon EKS Capabilities is a new set of fully managed features designed to make running Kubernetes easier and faster for developers. Think of it as a on-off button on top of your EKS cluster that removes a lot of the heavy lifting for you.
These capabilities give you Kubernetes-native tools for things like:

deploying your apps continuously (Argo CD)

managing AWS resources directly from Kubernetes (ACK)

creating and organizing your Kubernetes objects (kro) And the best part? AWS manages all of this for you.

Instead of installing, updating, and scaling these tools on your worker nodes, EKS now runs them for you. That means less time fighting with cluster operations and more time actually building and scaling the applications.

For the people who wants to jump straight in to trying the solution can follow my repo here

I tested this setup today and here’s how I deployed an S3 bucket with ACK and GitOps using Argo CD and EKS Capabilities.

Bonus
Architecture
Pricing
What is EKS capability
Prerequisites
Terraform Code Walkthrough
DevOps and Platform Engineering perspective

Bonus

I have found a bug which does not allow ArgoCD to get in healthy state, follow along to find out what that bug is and how to fix that bug.

Note: This blog assumes that you have fundamental knowledge on how Argo CD, ACK and kro works conceptually.

Architecture

Available capabilities

ACK: ACK enables the management of AWS resources using Kubernetes APIs, allowing you to create and manage S3 buckets, RDS databases, IAM roles, and other AWS resources using Kubernetes custom resources.
Argo CD: GitOps-based continuous deployment for your applications, using Git repositories as the source of truth for your workloads and system state.
kro: Create custom Kubernetes APIs that compose multiple resources into higher-level abstractions, allowing platform teams to define reusable patterns for common resource combinations-cloud building blocks

Pricing [IMP]

Well the pricing is tricky. You pay for EKS Capabilities based on two components that are both billed hourly:

a base hourly rate for each enabled capability
hourly usage charges based on the quantity of resources managed by each capability.

In short base charge + usage charge.

For Argo CD, you pay hourly for each Argo CD Application managed. For AWS Controllers for Kubernetes (ACK), you pay hourly for each ACK resource managed. For Kubernetes Resource Orchestrator (KRO), you pay hourly for each KRO Resource Graph Definition (RGD) instance managed.

What is EKS Capability?

It is basically an AWS resource managed by AWS including its scaling, lifecycle, security but runs inside your EKS cluster but not on your worker nodes.

Basically Capabilities run in EKS, eliminating the need to install and maintain controllers and other operational components on your worker nodes.

Resources That can be created by Capability

For ArgoCD
- Application
- ApplicationSet
- AppProject
For kro
- ResourceGraphDefinition (RGD)
- Custom resource instances
For ACK
- When you enable the ACK capability, you can create and manage AWS resources using Kubernetes custom resources. ACK provides over 200 CRDs for more than 50 AWS services,

Capability IAM Role [IMP]

Each EKS capability resource has a configured capability IAM role.
The capability role is used to grant AWS service permissions for EKS capabilities to act on your behalf.
For example, to use the EKS Capability for ACK to manage Amazon S3 Buckets, you will grant S3 Bucket administrative permissions to the capability, enabling it to create and manage buckets.

Prerequisites

In order to follow along make sure you have following prerequisites met.

Common to all 3 capabilities

An EKS cluster
An IAM Capability Role with permissions for ACK, ArgoCD, kro
Sufficient IAM permissions to create capability resources on EKS clusters
(For CLI/eksctl) The appropriate CLI tool installed and configured

Argo CD capability specific

AWS Identity Center configured - Required for Argo CD authentication (local users are not supported)
kubectl configured to communicate with your cluster

Terraform Code Walkthrough

Terraform code for this repo lies in this repo.

Just run terraform apply

Demo: Creating S3 buckets with ACK and ArgoCD using EKS Capabilities

The whole process of giving this managed platform capabilities is simple.

Create IAM Capability Role -> Enable Capabilities (ACK and ArgoCD for this Demo) -> Register cluster to ArgoCD -> add ArgoCD applications to track Git repo

We create an capability IAM role with trust policies as mentioned in the docs
We add permissions for the role like if we want to create s3 bucket using ACK we add s3 policy to the role so that ack can talk to AWS Services.

For kro and ArgoCD the role does not need any permissions because when you enable capability adds access entries policies which add permissions for these capabilities to interact with EKS cluster.

The Capability Role

For this we create a trust relationship which allows capabilities to assume role.

{
  "Version": "2012-10-17",               
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "capabilities.eks.amazonaws.com"
      },
      "Action": [
        "sts:AssumeRole",
        "sts:TagSession"
      ]
    }
  ]
}

Now based on the capability you add permissions to it, for the brevity of this blog ACK and ArgoCD capability will use the same role and in the tf code I have added full s3 permissions for ACK to create S3 bucket.

ArgoCD IAM Identity Center Integration

The Argo CD managed capability integrates with AWS Identity Center for authentication and uses built-in RBAC roles for authorization

That is why IAM identity Center is a prerequisite.

This is how permissions work with ArgoCD. When a user accesses Argo CD UI:

They authenticate using AWS Identity Center (which can federate to your corporate identity provider)
AWS Identity Center provides user and group information to Argo CD
Argo CD maps users and groups to RBAC roles based on your configuration
Users see only the applications and resources they have permission to access

This is exactly what I have done in my terraform code, accessing IAM identity center instance and user details and passing to the argoCD capability and configuring RBAC role for my user as ADMIN role.

data "aws_ssoadmin_instances" "main" {}

data "aws_identitystore_user" "admin" {
  identity_store_id = tolist(data.aws_ssoadmin_instances.main.identity_store_ids)[0]

  alternate_identifier {
    unique_attribute {
      attribute_path  = "UserName"
      attribute_value = var.idc_username
    }
  }
}

resource "null_resource" "eks_capability_argocd" {
  depends_on = [module.eks]

  provisioner "local-exec" {
    command = <<-EOT
      aws eks create-capability \
        --region ${local.region} \
        --cluster-name ${local.cluster_name} \
        --capability-name my-argocd \
        --type ARGOCD \
        --role-arn ${aws_iam_role.eks_capability_role.arn} \
        --delete-propagation-policy RETAIN \
        --configuration '{
          "argoCd": {
            "awsIdc": {
              "idcInstanceArn": "${tolist(data.aws_ssoadmin_instances.main.arns)[0]}",
              "idcRegion": "${local.region}"
            },
            "rbacRoleMappings": [{
              "role": "ADMIN",
              "identities": [{
                "id": "${data.aws_identitystore_user.admin.user_id}",
                "type": "SSO_USER"
              }]
            }]
          }
        }' \
        --profile jj
    EOT
  }
}

To restrict for your use- case see built-in RBAC roles

EKS cluster and installing capabilities]

I am using EKS auto mode for this blog. As this is fresh update I am using aws cli commands to enable capabilities.

Note:

1) Capabilities are not ACTIVE instantly so we need to wait until they become ACTIVE for next operation.

2) Behind the scenes after enabling capabilities, an access entry is created for the capability role with access entries like AmazonEKSArgoCDPolicy,AmazonEKSACKPolicy , AmazonEKSArgoCDClusterPolicy

resource "null_resource" "eks_capability_ack" {
  depends_on = [module.eks]

  provisioner "local-exec" {
    command = <<-EOT
      aws eks create-capability \
        --region ${local.region} \
        --cluster-name ${local.cluster_name} \
        --capability-name my-ack \
        --type ACK \
        --role-arn ${aws_iam_role.eks_capability_role.arn} \
        --delete-propagation-policy RETAIN \
        --profile jj
    EOT
  }
}

resource "null_resource" "eks_capability_argocd" {
  depends_on = [module.eks]

  provisioner "local-exec" {
    command = <<-EOT
      aws eks create-capability \
        --region ${local.region} \
        --cluster-name ${local.cluster_name} \
        --capability-name my-argocd \
        --type ARGOCD \
        --role-arn ${aws_iam_role.eks_capability_role.arn} \
        --delete-propagation-policy RETAIN \
        --configuration '{
          "argoCd": {
            "awsIdc": {
              "idcInstanceArn": "${tolist(data.aws_ssoadmin_instances.main.arns)[0]}",
              "idcRegion": "${local.region}"
            },
            "rbacRoleMappings": [{
              "role": "ADMIN",
              "identities": [{
                "id": "${data.aws_identitystore_user.admin.user_id}",
                "type": "SSO_USER"
              }]
            }]
          }
        }' \
        --profile jj
    EOT
  }
}

Registering the eks cluster to ArgoCD to deploy applications

Docs suggest to use argocd cli but it never succeeded as it always timed out so I register the secret using K8 Secret. This is possible because of the new access entry(AmazonEKSArgoCDPolicy) policy which ArgoCD capability to read the K8 secrets

resource "null_resource" "argocd_add_cluster" {
  depends_on = [null_resource.wait_for_argocd]

  provisioner "local-exec" {
    command = <<-EOT
      aws eks update-kubeconfig \
        --name ${local.cluster_name} \
        --region ${local.region} \
        --profile jj

      kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: ${local.cluster_name}-cluster
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
stringData:
  name: ${local.cluster_name}
  server: ${module.eks.cluster_arn}
  project: default
EOF
    EOT
  }
}

Create an ArgoCD Application

This demo uses a public GitHub repository, so no repository configuration is required. For private repositories, configure access using AWS Secrets Manager, CodeConnections, or Kubernetes Secrets

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: eks-capability
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/jatinmehrotra/aws-reinvent-2025
    targetRevision: HEAD
    path: eks-capabilities/ack_yaml
  destination:
    name: reinvent-2025
    namespace: ack
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

Over this path there is yaml file to create S3 bucket using ACK which will be synced and applied to cluster and then ACK will create this bucket in our AWS Account

apiVersion: s3.services.k8s.aws/v1alpha1
kind: Bucket
metadata:
  name: my-test-bucket
  namespace: default
spec:
  name: jj-bucket-name-12345

Fixing the Bug

Even after doing all this or following AWS docs this won't run because the access entries(AmazonEKSArgoCDClusterPolicy) policies created by AWS for ArgoCD Capability lacks permissions to list the cluster resources.

SO added cluster admin permission for the blog

horizontalpodautoscalers.autoscaling is forbidden: 
User "arn:aws:sts::xxxxxxx:assumed-role/eks-capability-role/aws-go-sdk-1764587162523491382" 
cannot list resource "horizontalpodautoscalers" in API group "autoscaling" at the cluster scope

So to fix that we need to import the access entry created by AWS and add the Cluster Admin Access entry.

# Import the access entry created by EKS capability
resource "null_resource" "import_access_entry" {
  depends_on = [null_resource.wait_for_argocd]

  provisioner "local-exec" {
    command = <<-EOT
      terraform import -input=false \
        aws_eks_access_entry.capability_role \
        "${local.cluster_name}:${aws_iam_role.eks_capability_role.arn}" || true
    EOT
  }
}

resource "aws_eks_access_entry" "capability_role" {
  cluster_name  = module.eks.cluster_name
  principal_arn = aws_iam_role.eks_capability_role.arn
  type          = "STANDARD"

  lifecycle {
    ignore_changes = [kubernetes_groups]
  }

  depends_on = [null_resource.import_access_entry]
}

# Add ClusterAdmin policy to fix insufficient ArgoCD permissions
resource "aws_eks_access_policy_association" "capability_role_admin" {
  cluster_name  = module.eks.cluster_name
  principal_arn = aws_iam_role.eks_capability_role.arn
  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"

  access_scope {
    type = "cluster"
  }

  depends_on = [aws_eks_access_entry.capability_role]
}

You can access ArgoCD UI from console -> Capabilities and Under Argo CD

You can login using your IAM identity Center

Argo CD successfully sync the application for Git repository and then ACK creates the S3 bucket

Cleanup

For cleanup before running terraform destroy. Read the delete docs

Delete the s3 bucket first
run the cleanup script

Then run terraform destroy.Because cluster won't be deleted if capabilities are not deleted.

DevOps and Platform Engineering perspective

In this blog we saw the Use case of GitOps for Applications and Infrastructure where Use Argo CD to deploy applications and ACK to provision infrastructure, both from Git repositories. Your entire stack—applications, databases, storage, and networking—is defined as code and automatically deployed

We just push the AWS infra yaml files to Git and ArgoCD deploys the updated application, and ACK provisions a new S3 bucket with the correct configuration.
All changes are auditable, reversible, and consistent across environments.

There can be many use cases for these capabilities like Account and regional Bootstrapping, Modernization of EKS resources

Its very interesting to see how they are compared to self managed versions ack,argocd,kro. Ofcourse there are limitations like ArgoCD right now supper single namespace deployment or Notifications Controller in ArgoCD isn't supported

But overall a great update for Platform engineering which offloads the setup, complexity and management of Kubernetes resources to AWS and lets developers focus on their productivity and applications.

I share such amazing AWS updates on DevOps, Kubernetes and GenAI daily over Linkedin, X. Follow me over there so that I can make your life more easy.

Cut CI/CD Costs by 77% & 2x Deployment Speed with GitHub Actions on EKS Auto Mode

Jatin Mehrotra — Mon, 21 Apr 2025 00:18:08 +0000

On April 5, 2025 I did a live stream on how to run Github Actions Self Hosted Runners on EKS Auto with AWS Heroes Arshad Zackeriya and Jones Zachariah Noel.

Disclaimer: 🐶 No Beagles were harmed. Slightly annoyed, maybe — but unharmed.

The results {Performance, Speed, Cost} were not only astonishing but impeccable and promising enough to adopt this solution at the enterprise level. This solution isn't just AWS agnostic, with the knowledge gained in this blog can be extended to Azure(AKS), google(GKE) or if you are running K8 on your own bare metal servers.

For the people who wants to jump straight in to trying the solution can follow my repo here

This is going to be a bit of a long one, so grab a coffee, get comfy, and make sure you're sitting in your optimal developer position™ — you know the one.

Architecture
Why this solution
Self Hosted Runner Concept on K8
Terraform Code Walkthrough
How to test this solution
Github Large Hosted Runner vs Running Runners on EKS Auto
From a solutions architect perspective

Architecture

Why this solution

I work as a senior DevOps Engineer at Colorkrew where we have a lot of products and to support the development workflow we have lot of GitHub repositories.

As we started to increase our product portfolio, our CI/CD pipelines also started to become more complex, concurrent and frequent leading to the need of more computation power and eventually more robust infrastructure layer which supports our growing needs.

Running Github Actions on default free machines (called as runners) started to become slow and the initial solution was to either use Large hosted runners by GitHub which are paid or run the runners on our Infrastructure like Kubernetes.

So I started to compare both solutions on performance, speed and cost and this lead to the inception of this running GitHub's self hosted runners on EKS Auto.

Self Hosted Runner Concept on K8

Runners: The machines (servers or virtual environments) that actually execute the jobs defined in your workflows. When you manage called as Self Hosted runners otherwise GitHub Hosted runners. Runners are ephemeral in nature.

Runner Scale Sets: Think of it as a logical grouping of runners that are homogeneous in nature which means all runners under a particular group will have same configuration. Can be installed at repository, organization or enterprise level.

If you want heterogeneous setup which means different configuration for runners for different ci/cd jobs then you need a multiple runner scale sets.

Another important thing to know about Runner scale sets are that you configure how many minimum and maximum runners you want all the time.

Name of Runner Scale Sets: Runner scale sets are addressable by their name so remember it because when you need to specify that name of runner scale set in the runs on: property of GHA to assign that workflow on on a particular runner scale set.

Endpoints: ARC talks to two endpoints api.github.com and pipelines.actions.githubusercontent.com. Make sure your organization's firewall, proxies, nat gateway whatever being used to access internet should be configured to allow the above endpoints for ARC controller.
ARC controller: contains of 2 elements/pods.
- controller-manager: First pod that comes online. This has different controllers managing different resources in the cluster. Important one to understand is AutoScalingListener Controller manages the listener pod.
  - Responsible for creating the resources and making sure that match the desired count and state.
- Runner ScaleSet Listener: Manages the decision making about scaling needs. Responsible to decide how many runners to create. Each listener has its own pod which means 1 listener pod per runner scale set. If you have 2 runner scale set then 2 listener pods either on same namespace like controller manager or different namespace ( configurable)

I thought should give easier explanation in my own analogies 😋🫣

Actions Runner Controller (ARC is like the manager of a smart, automated coffee shop — it watches how many customers are coming in (workflows) and instantly hires or releases baristas (runners) as needed.
Instead of having baristas standing around all day just in case, ARC spins up temporary baristas (containers in Kubernetes) only when customers arrive, and lets them go when the work is done. This keeps the system fast, efficient, and cost-effective.

With Runner Scale Sets, you can define the rules for how many baristas you want at any given time, based on how busy your shop is — and ARC handles the rest.

You can read more about detailed end to end workflow over here

Terraform Code Walkthrough

jatin-mehrotra-personal / eks-auto-self-hosted-runners

Run GitHub Actions Self Hosted Runners on EKS Auto Mode

This repository provides infrastructure as code (IaC) to deploy auto-scaling GitHub Actions self-hosted runners on Amazon EKS using GitHub's Actions Runner Controller (ARC).

detailed blog: https://dev.to/aws-builders/cut-cicd-costs-by-77-2x-deployment-speed-with-github-actions-on-eks-auto-2ob2

See it in action: https://www.youtube.com/live/s23QvNz2WuY?si=08n0qCVpMYC1qTKc

Overview

This solution allows you to:

Deploy a fully managed EKS cluster with auto-scaling capabilities
Set up GitHub Actions Runner Controller (ARC) for managing self-hosted runners
Configure auto-scaling runner sets that scale based on workflow demand
Support Docker-in-Docker (DinD) runners for container-based workflows

Architecture

The infrastructure consists of:

Amazon EKS cluster running in a custom VPC
GitHub Actions Runner Controller deployed via Helm
Auto-scaling runner sets configured to scale from 0 to meet demand
Optional Docker-in-Docker (DinD) runner support
Karpenter for node auto-scaling (configured but optional)

Prerequisites

AWS CLI configured with appropriate permissions
Terraform v1.0.0+
kubectl
Helm v3+
A GitHub repository or organization where you want to deploy runners
…

View on GitHub

Setting Up Auto-Scaling GitHub Actions Self-Hosted Runners on Amazon EKS

Introduction

In this walkthrough, we'll set up GitHub Actions Runner Controller (ARC) on Amazon EKS to automatically scale self-hosted runners based on workflow demand.

Project Structure

Here's the structure of our implementation:

eks-auto-self-hosted-runners/
├── README.md
├── architecture/
├── commit_log.txt
├── scripts/
└── terraform/
    ├── base/
    └── modules/
        ├── arc/
        ├── eks/
        ├── karpenter_config/
        └── vpc/

1. Root directory - Contains the main README and architecture diagrams
2. scripts/ - Contains utility scripts for cleanup and performance testing
3. terraform/ - The main infrastructure code
   base/ - The entry point for Terraform deployment
   modules/ - Reusable Terraform modules:
   arc/ - Actions Runner Controller configuration
   eks/ - EKS cluster configuration
   karpenter_config/ - Node auto-scaling configuration
   vpc/ - Network infrastructure configuration

Architecture Overview

Our solution uses the following components:

Amazon EKS Auto: Managed Kubernetes service to host our runner infrastructure with Karpenter for provisioning nodes on-demand for runners compute
GitHub Actions Runner Controller (ARC): Kubernetes controller that manages self-hosted runners
Terraform: Infrastructure as Code tool to deploy and manage all components

The architecture allows GitHub Actions workflows to dynamically request runners, which are provisioned on-demand in our EKS cluster and automatically scaled down when not needed.

Step 1: Setting Up the Infrastructure

VPC Configuration

We start by creating a VPC with public and private subnets. Our VPC configuration uses a CIDR block of 10.0.0.0/16 with subnets spread across two availability zones. The
private subnets host our EKS nodes, while public subnets are used for NAT gateways and load balancers.

The configuration in terraform/base/vpc.tf references the VPC module and sets up all necessary networking components with appropriate tagging for Kubernetes integration.

EKS Cluster Setup

Next, we create an EKS cluster using the EKS module defined in terraform/modules/eks. Our cluster runs Kubernetes version 1.31 and includes a system node group for running
essential cluster services.

The terraform/base/eks.tf file configures the cluster with public endpoint access and places the worker nodes in the private subnets for enhanced security.

Step 2: Configuring EKS Auto's pre-installed Karpenter for Auto-Scaling

EKS Auto comes with Karpenter pre-installed. We leverage Karpenter to provision and auto scale Ec2 spot instances from our desired Ec2 type, capacity and configuration for our GHA runner's compute.

Our Karpenter configuration in terraform/base/karpenter_config.tf uses m7a instance types with spot pricing for cost efficiency. The consolidation policy is set to "WhenEmpty" with a 5-minute timeout, which means nodes will be removed when they're no longer needed.

Key configuration parameters include:
• Instance types: m7a family with 8 CPUs
• Capacity type: Spot instances for cost savings
• Storage: 300GB with 5000 IOPS
• Availability zones: us-east-1a and us-east-1b

Step 3: Deploying Actions Runner Controller (ARC)

Now we deploy the GitHub Actions Runner Controller using the ARC module defined in terraform/modules/arc. The module is referenced in terraform/base/arc.tf with configuration parameters from locals.tf.

The ARC deployment consists of two main components:

Controller: Manages the lifecycle of runner pods
Runner Sets: Define the configuration for the runners

Controller Deployment

The controller is deployed using a Helm chart from the official GitHub Actions Runner Controller repository. The configuration is defined in terraform/modules/arc/controller.tf and uses values from helm/controller_values.yaml.

Runner Sets Configuration

We deploy two types of runner sets:

Standard Runners: For general workflow jobs
Docker-in-Docker (DinD) Runners: For jobs that need to build Docker images
- There is also another type of runner set called Kubernetes mode which should be used when organizations cannot afford to run docker with superuser context as in case DinD runners then use Kubernetes mode ( not covered in this blog)

The runner sets are defined in terraform/modules/arc/runner_sets.tf and use values from helm/arc_listener_values.yaml and helm/arc_listener_values_dind.yaml respectively.

Note: This is very important that Controller pod and runner scale set pod should be configured to be deployed on on-demand ec2 instance for reliability and ephermal runner pods should be configured to run spot instances.

# controller config for on demand node

nodeSelector:
  karpenter.sh/nodepool: system

tolerations:
- key: "CriticalAddonsOnly"
  operator: "Exists"

# listener config for on demand node

listenerTemplate:
  spec:
    nodeSelector:
      karpenter.sh/nodepool: system
    tolerations:
    - key: "CriticalAddonsOnly"
      operator: "Exists"

Step 4: GitHub Authentication Setup

ARC authenticates with GitHub using a GitHub App. The credentials are stored in Kubernetes secrets and used by the runner controller to authenticate with GitHub.

To set up authentication:

Create a GitHub App with the necessary permissions
Generate a private key for the app
Store the app ID, installation ID, and private key in the terraform/modules/arc/secrets (Don't forget to create it, gitignored) the directory
The terraform/modules/arc/secrets.tf file creates a Kubernetes secret with these credentials

Step 5: Namespace Management

We create dedicated namespaces for ARC components:

arc-systems: For the controller components
arc-runners: For the runner pods

These namespaces are defined in terraform/modules/arc/namespaces.tf.

Step 6: Cleanup Handling

To ensure proper cleanup of resources, we've included a cleanup script in scripts/cleanup-finalizers.sh that removes finalizers from Kubernetes resources. This script is called during the Terraform destroy process to ensure resources are properly cleaned up.

The script handles various resource types including:
• AutoscalingRunnerSet
• EphemeralRunnerSet
• AutoscalingListener
• ServiceAccounts
• RoleBindings
• Roles

How It Works

When a GitHub Actions workflow runs, it requests a runner with specific labels
The ARC controller detects this request and creates a runner pod in the EKS cluster
If needed, Karpenter provisions a new EC2 instance to host the runner pod
The runner registers with GitHub and runs the workflow job
After the job completes, the runner pod is terminated
When no more runners are needed, Karpenter consolidates and removes unused nodes

Benefits of This Approach

Cost Efficiency: Runners are only provisioned when needed and automatically scaled down when idle
Flexibility: Custom runner environments can be defined to meet specific workflow requirements
Scalability: The system can handle large numbers of concurrent workflows
Security: Runners run in isolated Kubernetes pods with defined security contexts
Reliability: Failed runners are automatically replaced, ensuring workflow reliability

How to test this solution

I have created 3 GHA of different types to test different scenarios.

Simple test

The first type is the simple GHA but important point to pay attention is runs-on parameter which uses arc-runner-set label.

The moment you run the first workflow it takes around 1 minute because Karpenter is provisioning an Ec2 instance.

Once it is provisioned and if you run the workflow again because Ec2 instance is already present the action completes its execution immediately.

Karpenter will delete the node if it's unused for at least 5 minutes.

Now if you have latency sensitive requirement of running your actions immediately then set minRunners>0 for the runner scale set configuration

Concurrent job test

In this GHA I am running 5 concurrent jobs running for exactly 1 minute each triggered either through push or manually.

DinD job test

There are times when we want to run microservice in container like redis and do e2e testing.

That's where we need Docker Inside Docker kind of GHA.

Important point to pay attention here is runs-on parameter which uses different runner set configured especially for dind workflows

In this workflow busy box container runs the main job which can talk to redis service container.

Github Large Hosted Runner vs Running Runners on EKS Auto

Performance & Speed

I tried to do the performance and speed comparison of 8 Core CPU vs our solution.

I tried to run the sleep-matrix GHA using auto-commit script for 10 minutes and every commit happening in every 45 seconds effectively simulating a concurrent job situations for both EKS Auto solution and Github large Runners and results were hands down in favor of our solution

Our solutions has stable performance with constant 1 minute 8 seconds Execution time throughout.

Price

Update [21st April 2025] : I did a mistake in EKS Auto node management price earlier but after correction our solution makes it even cheaper

This is little tricky and may not be 100% accurate comparison but still can be considered 70-80% accurate.

Assume over a 30‑day month with 1 hour of concurrent CI workloads per day (12 jobs running in parallel). Lets calculate the cost on our solution and as well on Github's self hosted runners

For our solution

For Github's large runner

There is also a cost Github Teams/Enterprise subscription. I am assuming it's on cheaper side i.e Team's plan for a team of 10 developer

self‑hosting on EKS with m7a.2xlarge Spot nodes costs approximately $170.38/month, while same GitHub’s 8‑core large hosted runners cost $731.20/month, so my solution on EKS Auto is YIELDING A WHOOPING SAVINGS OF 76.7%.

Note: Even if we add Github Team's for EKS solution i.e $40 more which is 170+40 =$210/month which is still a whooping savings of 72%

Lets assume it's not 1 hour, it's 2 hour of concurrent CI workloads per day (12 jobs running in parallel)

yielding a savings of 84%

Note: I know I haven't added the networking components cost like nat gateway even if we add up it will still be cheaper GitHub's large runners :D

From a solutions architect perspective

By combining Amazon EKS Auto and GitHub Actions Runner Controller, we've created a solution that aligns with the AWS Well-Architected Framework:

Operational Excellence:
• Infrastructure as code through Terraform ensures consistent deployments
• Auto-scaling runners eliminate manual capacity management
• Clean separation of components improves maintainability

Security:
• GitHub App authentication with scoped access
• EKS security groups and IAM roles enforce least-privilege
• Private subnets reduce attack surface

Reliability:
• Multi-AZ deployment ensures high availability
• Auto-scaling from zero accommodates varying workloads without manual intervention
• EKS Auto's Karpenter provides intelligent node provisioning, ensuring resources are available when needed

Performance Efficiency:
• On-demand scaling ensures resources are used only when needed
• Spot instances reduce costs while maintaining performance
• Customizable instance types allow optimization for specific workload requirements
• Docker-in-Docker support enables container-based workflows with minimal overhead

Cost Optimization:
• Scale-to-zero capability eliminates costs when no workflows are running
• Spot instances provide significant cost savings (up to 90%) compared to on-demand instances
• Resource consolidation through Karpenter's WhenEmpty policy reduces idle resources

Sustainability:
• Efficient resource utilization through auto-scaling minimizes environmental impact
• Scale-to-zero capability reduces energy consumption during idle periods
• Reduced idle resources lower energy consumption

This solution provides organizations with a scalable, cost-effective platform for running GitHub Actions workflows while maintaining full control over their infrastructure. The architecture can easily scale from small development teams to enterprise-level deployments, adapting to changing workflow requirements without compromising on security or performance.

By leveraging AWS managed services like EKS and implementing infrastructure as code through Terraform, this solution reduces operational overhead while providing the flexibility needed
for modern CI/CD pipelines. The result is a robust, efficient platform that enables teams to focus on delivering value rather than managing infrastructure.

Follow The Zacs' Show Talking AWS on YouTube or reach out to them if you want to share your knowledge on AWS.

Have some questions on this solution reach out to me over Linkedin, X.

AWS CodePipeline + EKS Action: The Easiest Way to Deploy Containers Yet!

Jatin Mehrotra — Tue, 25 Feb 2025 06:26:06 +0000

Amazon Web Services (AWS) CodePipeline Team has simplified Developer, DevOps engineers operational overhead and streamlined the Deployment process to EKS by introducing an CodePipeline action to deploy directly to your EKS cluster.

Why does it matter to you?

Previously if I had to deploy resources to EKS using DevOps approach, I had to manage codebuild project, permissions to access eks, kubectl, helm commands, other horrific shell commands and still it wont be perfect in one shot.

I tried this out today so let me show you how you can simplify your deployment pipeline for EKS by 100%, say goodbye to codebuild and remove all the complex process, scripts and command!!!

Architecture

Prerequisites

EKS cluster with public endpoint.
Kubernetes resource like deployment.yaml file
You can also helm chart if you wish.

Note: If you want to try this update with EKS private Endpoint then checkout this blog by AWS Hero Arshad Zackeriya.

Whichever you prefer I have provided both (helm chart and deployment.yaml) in this repository.

Behind the Scenes

In case of kubernetes manifest file

This action logs into eks cluster, set kubeconfig context
Installs kubectl
Applies Kubernetes manifests
Rollout resources

In case of helm chart

This action logs into eks cluster, set kubeconfig context
Instals helm
Instal Helm charts

Create an EKS cluster

Your cluster can be public or private (including those in private VPCs).The pipeline will automatically establish a connection into your private network to deploy your container application, without additional infrastructure needed.

I have used eks auto using terraform eks module which is quick and easy with EKS endpoint as public.

eks.tf

# eks cluster

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.31"

  cluster_name                   = local.cluster_name
  cluster_version                = "1.32"
  cluster_endpoint_public_access = true


  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  cluster_compute_config = {
    enabled    = true
    node_pools = ["general-purpose"]
  }

  # Cluster access entry
  # To add the current caller identity as an administrator
  enable_cluster_creator_admin_permissions = true


  tags = {
    Environment = "dev"
    Terraform   = "true"
  }
}

vpc.tf

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.19.0"

  name = "codepipeline-eks-action"
  cidr = "10.0.0.0/16"

  azs             = ["us-east-1a", "us-east-1b"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets  = ["10.0.4.0/24", "10.0.5.0/24"]

  enable_nat_gateway     = true
  single_nat_gateway     = true
  one_nat_gateway_per_az = false

  enable_dns_hostnames = true
  enable_dns_support   = true

  public_subnet_tags = {
    "kubernetes.io/role/elb" = "1"
  }
  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = "1"
  }

  tags = {
    Environment = "terraform-playground"
  }
}

pod_identity.tf

data "aws_iam_policy_document" "allow_pod_identity" {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["pods.eks.amazonaws.com"]
    }

    actions = [
      "sts:AssumeRole",
      "sts:TagSession"
    ]
  }
}

resource "aws_iam_role" "read_ecr" {
  name               = "read-ecr-role"
  assume_role_policy = data.aws_iam_policy_document.allow_pod_identity.json
}

resource "aws_iam_role_policy_attachment" "read_ecr" {
  role       = aws_iam_role.read_ecr.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
}

resource "aws_eks_pod_identity_association" "read_ecr" {
  cluster_name    = local.cluster_name
  namespace       = "default"
  service_account = "ecr-sa"
  role_arn        = aws_iam_role.read_ecr.arn
}

Note: I am using pod identity with ecr repository permission to obtain ecr image for deployment.yaml.

If you want complete code for cluster, you can follow this repo

Create pipeline with EKS deployment action

Case 1: when kubectl configuration is used.

I am using the source as GitHub using code start connection and code resides in this repo

Select the cluster
Provide the path for your deployment.yaml file (in my case its deployment.yaml file)

Case 2: When Helm chart is used

Enter the release name
Enter the helm chart (in my case its the test-chart)

Important step

Note: Once pipeline is created you need to edit the pipeline service role or update the existing one and add the following permissions to avoid error.

{
    "Statement": [
        {
            "Action": [
                "iam:PassRole"
            ],
            "Resource": "*",
            "Effect": "Allow",
            "Condition": {
                "StringEqualsIfExists": {
                    "iam:PassedToService": [
                        "cloudformation.amazonaws.com",
                        "elasticbeanstalk.amazonaws.com",
                        "ec2.amazonaws.com",
                        "ecs-tasks.amazonaws.com"
                    ]
                }
            }
        },
        {
            "Action": [
                "codecommit:CancelUploadArchive",
                "codecommit:GetBranch",
                "codecommit:GetCommit",
                "codecommit:GetRepository",
                "codecommit:GetUploadArchiveStatus",
                "codecommit:UploadArchive"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "codedeploy:CreateDeployment",
                "codedeploy:GetApplication",
                "codedeploy:GetApplicationRevision",
                "codedeploy:GetDeployment",
                "codedeploy:GetDeploymentConfig",
                "codedeploy:RegisterApplicationRevision"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "codestar-connections:UseConnection"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "elasticbeanstalk:*",
                "ec2:*",
                "elasticloadbalancing:*",
                "autoscaling:*",
                "cloudwatch:*",
                "s3:*",
                "sns:*",
                "cloudformation:*",
                "rds:*",
                "sqs:*",
                "ecs:*"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "lambda:InvokeFunction",
                "lambda:ListFunctions"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "opsworks:CreateDeployment",
                "opsworks:DescribeApps",
                "opsworks:DescribeCommands",
                "opsworks:DescribeDeployments",
                "opsworks:DescribeInstances",
                "opsworks:DescribeStacks",
                "opsworks:UpdateApp",
                "opsworks:UpdateStack"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "cloudformation:CreateStack",
                "cloudformation:DeleteStack",
                "cloudformation:DescribeStacks",
                "cloudformation:UpdateStack",
                "cloudformation:CreateChangeSet",
                "cloudformation:DeleteChangeSet",
                "cloudformation:DescribeChangeSet",
                "cloudformation:ExecuteChangeSet",
                "cloudformation:SetStackPolicy",
                "cloudformation:ValidateTemplate"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "codebuild:BatchGetBuilds",
                "codebuild:StartBuild",
                "codebuild:BatchGetBuildBatches",
                "codebuild:StartBuildBatch"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Effect": "Allow",
            "Action": [
                "devicefarm:ListProjects",
                "devicefarm:ListDevicePools",
                "devicefarm:GetRun",
                "devicefarm:GetUpload",
                "devicefarm:CreateUpload",
                "devicefarm:ScheduleRun"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "servicecatalog:ListProvisioningArtifacts",
                "servicecatalog:CreateProvisioningArtifact",
                "servicecatalog:DescribeProvisioningArtifact",
                "servicecatalog:DeleteProvisioningArtifact",
                "servicecatalog:UpdateProduct"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "cloudformation:ValidateTemplate"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ecr:DescribeImages"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "states:DescribeExecution",
                "states:DescribeStateMachine",
                "states:StartExecution"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "appconfig:StartDeployment",
                "appconfig:StopDeployment",
                "appconfig:GetDeployment"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:us-east-1:xxxx:log-group:/aws/codepipeline/eks-deploy-codepipeline",
                "arn:aws:logs:us-east-1:xxxxx:log-group:/aws/codepipeline/eks-deploy-codepipeline:log-stream:*"
            ]
        },
        {
            "Sid": "EksClusterPolicy",
            "Effect": "Allow",
            "Action": "eks:DescribeCluster",
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "EksVpcClusterPolicy",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeDhcpOptions",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeRouteTables",
                "ec2:DescribeSubnets",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeVpcs"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "ec2:CreateNetworkInterface",
            "Resource": "*",
            "Condition": {
                "StringEqualsIfExists": {
                    "ec2:Subnet": [
                        "arn:aws:ec2:us-east-1:292170836962:subnet/subnet-example1",
                        "arn:aws:ec2:us-east-1:292170836962:subnet/subnet-example2"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": "ec2:CreateNetworkInterfacePermission",
            "Resource": "*",
            "Condition": {
                "ArnEquals": {
                    "ec2:Subnet": [
                        "arn:aws:ec2:us-east-1:xxxx:subnet/subnet-example1",
                        "arn:aws:ec2:us-east-1:xxxx:subnet/subnet-example2"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": "ec2:DeleteNetworkInterface",
            "Resource": "*",
            "Condition": {
                "StringEqualsIfExists": {
                    "ec2:Subnet": [
                        "arn:aws:ec2:us-east-1:xxxxx:subnet/subnet-example1",
                        "arn:aws:ec2:us-east-1:xxxx:subnet/subnet-example2"
                    ]
                }
            }
        }
    ],
    "Version": "2012-10-17"
}

Create a access entry in eks cluster with AmazonEKSClusterAdminPolicy for the above codepipeline service role.

Run the pipeline

Case when kubectl manifest is updated

(base) jatin.mehrotra@CK0662-001 codepipeline-eks-deployment-no-github % kubectl get pods
NAME                         READY   STATUS    RESTARTS   AGE
hello-k8s-74fd98b69b-2pwrv   1/1     Running   0          2m37s
hello-k8s-74fd98b69b-g876f   1/1     Running   0          2m37s
hello-k8s-74fd98b69b-phs77   1/1     Running   0          2m37s

Case when helm chart manifest is updated

(base) jatin.mehrotra@CK0662-001 codepipeline-eks-deployment-no-github % kubectl get pods                             
NAME                    READY   STATUS    RESTARTS   AGE
test-67cbfddc66-2gcvj   1/1     Running   0          10m

From Developer, DevOps perspective

With this update I don't have to manage codebuild projects or any kind of compute environment and manage complex scripts, permissions and tool installations.
Following is the image of the setup which I used to have with codebuild before this update. Such a complex mess.

This is surely a game changer for developers, devops engineers and infra engineers who wants to focus on business problem, their applications and other kubernetes issue like monitoring and scaling.

This is not GitOps approach which is followed by Flux/ArgoCD but the best DevOps approach for EKS cluster in my view.

Will you use this action for your eks clusters? Let me know in the comments!!!

I share such amazing AWS updates on DevOps, Kubernetes and GenAI daily over Linkedin, X. Follow me over there so that I can make your life more easy.

CodePipeline EC2 Action: Simplify your EC2 application deployment pipeline by 90%!

Jatin Mehrotra — Mon, 24 Feb 2025 03:48:57 +0000

Amazon Web Services (AWS) CodePipeline Team has simplified Developer, DevOps engineers operational overhead and streamlined the Deployment process to EC2 by introducing an CodePipeline action to deploy directly to your EC2 instances.

Why does it matter to you?

Previously, if you wanted to deploy to EC2 instances, you had to use CodeDeploy with an AppSpec file to configure the deployment. After this update NO NEED TO MANAGE CODEDEPLOY RESOURCE AND APPSPEC FILE.

I tried this out today so let me show you how you can simplify your deployment pipeline by 90% and remove all the complex process and scripts!!!

Architecture

Prerequisites

supports only Linux instance types
maximum fleet size supported is 500 instances.
Pipeline V2 only supported
SSM Agent must be installed

Behind the Scenes

This action performs a send-command using SSM to execute the script in the instance.

Create an EC2 instance with Apache

Note: If you are creating an instance then you need to create a role with AmazonSSMManagedInstanceCore, AmazonS3ReadOnlyAccess and attach to ec2 instance.

Add ssm agent to the instance manually either using user data or logging into existing instance and then running the commands.
If you are using existing instance then you need to reboot the instance after adding the ssm agent.

I have created EC2 with Apache web server for the brevity of the blog.

sudo su
yum update -y
yum install httpd -y
service httpd start
chkconfig httpd on
echo "codepipeline ec2 deployment action" > /var/www/html/index.html

Create pipeline with Ec2 deployment action

Here we need to select instance using tags; I have used name tag to identify my instance.
Then you need to provide the target directory in ec2 instance where you want to deploy.
Finally path to the executable script file that runs AFTER the Deploy phase.

Note: Once pipeline is created you need to edit the pipeline service role and add the following permissions to avoid error.

        {
            "Effect": "Allow",
            "Action": [
                "ssm:CancelCommand",
                "ssm:DescribeInstanceInformation",
                "ssm:ListCommandInvocations",
                "ssm:SendCommand"
            ],
            "Resource": "*"
        }

Run the pipeline

I am using the source as GitHub using code start connection and code resides in this repo.

Note: The aws guide mentioned about adding AWSSystemsManagerDefaultEC2InstanceManagementRoleeployAction to the ec2 instance role but in my environment I didn't needed that permission

Some advanced options for this action

You can specify number of instances in number or percentages to deploy to in parallel.
You can specify number of instances in number or percentages to stop the task after the task fails.
You can specify load balancer which will block the traffic Toi the instance when deployment is taking for that instance.

From Developer, DevOps perspective

With this update I can have an END-to-END AWS native Continuous deployment on EC2 without managing CodeDeploy resources.
This is surely a game changer for developers and devops engineers who wants to focus on business problem, their applications and not on their complex deployment process.
Of course this experience of simplifying can be augmented even further to 100% if AWS can add necessary permissions for the role.

After reading this, do you wish to migrate to this action and simplifying your deployment process? Let me know in the comments!!!

I share such amazing AWS updates on DevOps, Kubernetes and GenAI daily over Linkedin, X. Follow me over there so that I can make your life more easy.

EKS Auto Mode Unlocked for Existing Clusters with Terraform

Jatin Mehrotra — Fri, 13 Dec 2024 05:59:47 +0000

In the previous blog, I explained that EKS Auto mode is now supported by terraform-eks-module and illustrated how we can create new cluster with EKS Auto Mode.

In this blog, we’ll learn how to enable EKS Auto Mode on existing clusters and migrate workloads from EKS Managed Node Groups to EKS Auto nodes with ZERO DOWNTIME and continued application availability using my terraform code.

I have also added a BONUS section which explains how we can control our pod's deployments on EKS Auto Mode nodes or other compute types.

Motivation

Terraform-provider-aws released a new version v5.8.1 which allows to enabled EKS Auto with built-in NodePools on existing cluster.

Terraform-aws-eks release a new version v20.31.1 which allows to use custom NodeClass/NodePools when EKS Auto is enabled without built-in NodePools.

I want this blog to be really short, crisp and efficient so lets jump into actual steps!

Deploy Terraform cluster without EKS Auto Mode

We want to create the use case where we have an existing cluster WITHOUT EKS Auto Mode using EKS MNG.
Use this repository code to deploy EKS cluster with Managed node group.

Note: I am attaching policies to the node IAM role for EKS MNG - this is too permissive, better to use EKS Pod Identity (or IRSA, but EKS Pod Identity is preferred). Feel free to send a PR to the repo :)

Deploy workload or pods

We will automate this as well using terraform's kubectl_manifest resource, we will deploy workload yaml code using terraform

Note: During cluster creation, test workload(pods) were not deployed because kubectl context was not set locally. So run the following command to set the kubectl context and run terraform apply again once cluster is created.

aws eks --region us-east-1 update-kubeconfig --name eks-existing-cluster-tf-test --profile <your-profile-name> ; terraform apply

Current state of EKS cluster before EKS Auto Mode

Let's verify the current state of eks cluster when EKS Auto mode is not enabled.
EKS Auto mode is disabled.

EKS Auto Managed Node group created by me is running.

Pods are running on EKS managed node group

Enable EKS Auto Mode on Existing Cluster

Uncomment the following code to the eks.tf and terraform apply to enable EKS Auto Mode

bootstrap_self_managed_addons = true

cluster_compute_config = {
   enabled = true
}

bootstrap_self_managed_addons = true is very important otherwise you will face error where terraform tries to recreate the cluster again. I literally cried over this

Current state of EKS cluster after EKS Auto Mode

As expected built-in NodePools are empty

Migrate workload(pods) from EKS MNG to EKS Auto Node

There are couple of ways to smoothly migrate existing workloads from MNG to EKS Auto with minimal disruption while maintaining application’s availability throughout the migration.

Note: Copy the EKS MNG node group name.

Using eksctl tool

The following command will cordon all nodes and all pods are evicted from a nodegroup and EKS will provision pods to node managed by EKS Auto.

eksctl drain nodegroup --cluster=<clusterName> --name=<copiedNodegroupName>  --region us-east-1 --profile=<profile>

eksctl command evicts pod one at a time which I have tested so application availability is maintained.
But if you still want to be 100% sure, you can use the best practice of using pod Disruption budget. We will automate this using terraform so run terraform apply

resource "kubectl_manifest" "test_pdb" {
  yaml_body = <<YAML
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: test-pdb
  labels:
    environment: test
spec:
  minAvailable: 1
  selector:
    matchLabels:
      environment: test
YAML
}

After migrating; If we want to allow scheduling pods to EKS MNG we need to uncordon the EKS MNG or you can delete the Node group

eksctl drain nodegroup --cluster=<clusterName> --name=<copiedNodegroupName>  --region us-east-1 --profile=<profile> --undo

Using kubectl

we can use the following command to drain the nodes using kubectl

kubectl drain --ignore-daemonsets <node name>

Once it returns (without giving an error), you can delete the node or you want to tell Kubernetes that it can resume scheduling new pods onto the node

kubectl uncordon <node name>

[ BONUS ] How to schedule Pods always on EKS Auto Nodes?

There are 2 options to achieve this :

Either delete The NodeGroup and let EKS Auto handle the scheduling on EKS Auto Nodes
Using labels and NodeAffinity

Control if a workload is deployed on EKS Auto Mode nodes

There is concept called mix-mode cluster where you’re running both EKS Auto Mode and other compute types, such as self-managed Karpenter provisioners or EKS Managed Node Groups.
In mix mode clusters by default deployment is deployed to EKS MNG nodes and not EKS Auto Nodes
In such case we can use labels and nodeAffinity.

Using NodeSelector label

Use the label eks.amazonaws.com/compute-type: auto when you want a workload is deployed to EKS Auto Node.
This nodeSelector value is only relevant if you are running a cluster in a mixed mode, node types not managed by EKS Auto Mode

apiVersion: apps/v1
kind: Deployment
spec:
      nodeSelector:
        eks.amazonaws.com/compute-type: auto

I have an added the above configuration in sample_app_on_eks_auto_nodes.tf file. We are automating using Terraform so uncomment and run `terraform apply.

Using nodeAffinity

You can add this nodeAffinity to Deployments or other workloads to require Kubernetes to not schedule them onto EKS Auto Mode nodes

I have added the workload with nodeAffinity in sample_app_not_on_eks_auto_nodes.tf . We are automating using Terraform so uncomment and run terraform apply.

From DevOps, IaC Perspective

We saw how we can enable EKS Auto mode for Existing clusters with built-in NodePools using terraform-eks-module
We also saw how we can migrate our existing workload from EKS Managed Group to EKS Auto Nodes without any down time as EKS Auto node respect PodDisruptionBudget.
We also saw how we can use nodeSelector Labels and nodeAffinity to control deployment of workload in case of mixed-mode EKS clusters.

Currently EKS Auto deploys EC2 of instance type c6a.large which can be also customized using nodeClass and NodePool which we will see in the next blog. Follow me on Linkedin or on dev.to so that you get timely updates of what I share.

Feel free to reach out to me on Linkedin, X if you face any error migrating your Existing workloads to EKS Auto Mode Nodes using terraform.

DEV Community: Jatin Mehrotra

AWS Now Gives You a Free Sandbox Account - No Credit Card, No Cost, 8 Hours to Build (2026)

Step 1 Log in to Builder Center

Step 2 Access Workshops for Sandbox Env

Step 3 Choose Workshops

Step 4 Accessing workshop

Step 5 Access AWS Account through Console

Step 6 Go build on AWS

Few things to Note about sandbox accounts

I Built PR Preview Environments With AWS Lambda MicroVMs and Cut Staging Costs by 78%

Table of Contents

The Staging Bottleneck Problem

Architecture

The Solution

Project Structure

The App

App code

Why Lifecycle Hooks?

The Auth Proxy (proxy/handler.py)

The GHA Workflow

Terraform code

How to Test This

Prerequisites

Step 1: Deploy Infrastructure

Step 2: Configure GitHub Secrets

Step 3: Create a PR

Step 4: Click the Preview URL

Step 5: Close the PR

Cost: Lambda MicroVMs vs Shared Staging vs EKS

Where the savings actually come from; pay for active minutes, not uptime

Monthly Cost Breakdown

Production Tips for Lambda MicroVMs

From DevOps Perspective

AWS Lambda MicroVMs: Your AI Agent Can Write Code. But Should It Execute It?

Architecture

The Developer's Problem in 2026

The trade off before Lambda MicroVMs

What is Lambda MicroVm

Core Concepts for Lambda MicroVMs

How the MicroVM Image Build Works

MicroVM Lifecycle States & Hooks [IMP to build reliable applications]

Demo: Build Ephemeral Sandboxes for AI Agent Code Execution

Step 1: The Sandbox Image (Dockerfile)

Step 2: The Executor (Runs Inside the MicroVM)

Step 3: Build the MicroVM Image

Console way

CLI method

Step 4: Wire It Into Your Application

Console

CLI/Code way

Step 5: See It Work (Multi-Step State Persistence)

Cost and Sizing

Developer Perspective

AWS DevOps Agent is not here to replace you! It's here to compliment you by acting like 24/7 on call engineers finding root cause for issues while you build new feature and innovate. More details in the blog on how to setup with 2 Demo.

AWS DevOps Agent Explained: Architecture, Setup, and Real Root-Cause Demo (CloudWatch + EKS)

AWS DevOps Agent is not here to replace you! It's here to compliment you by acting like 24/7 on call engineers finding root cause for issues while you build new feature and innovate. More details in the blog on how to setup with 2 Demo.

AWS DevOps Agent Explained: Architecture, Setup, and Real Root-Cause Demo (CloudWatch + EKS)

AWS DevOps Agent Explained: Architecture, Setup, and Real Root-Cause Demo (CloudWatch + EKS)

Architecture

What is AWS DevOps Agent

How it investigates and find the root cause

How to maximize Agent's Effectiveness

AWS DevOps Agent working architecture [IMP]

DevOps Agent Spaces

DevOps Agent Web App

Resource Discovery by AWS DevOps Agent

Demo 1: Investigate Cloudwatch Alarm

Pre-requisite

1) Investigate Cloudwatch Alarm

Access Web App

Investigation Root Cause

Mitigation plan

Prevention

Investigation gaps

Using Chat

Demo 2: Investigate EKS errors

Adding Capability;Give DevOps Agent access to EKS clusters

Go to Web App

DevOps Engineer perspective

AWS launched EKS Capabilities to simplify Kubernetes with a quick on/off model. It runs Argo CD, ACK, and kro for you—no installs or upgrades. I tested it by deploying an S3 bucket using ACK + Argo CD using EKS Capabilities. Terraform Repo link inside.