DEV Community: will peixoto

Lambda MicroVMs: shell access, lifecycle hooks and the best practices that matter

will peixoto — Wed, 04 Nov 2026 14:00:00 +0000

Let me tell you about a scene anyone from the infra side will recognize right away. When Lambda showed up back in 2014 and started spreading, infrastructure folks would come over and ask: "OK, but how do I connect to the Lambda to check it's running? Run a ps, a tail, look at the app from the inside?". And the answer was always the same: you don't. A function is a black box, you look at CloudWatch and pray.

Well, with Lambda MicroVMs you can now attach a real shell. =D In this post I show how to open that shell, how to implement in Node the lifecycle hooks that keep the MicroVM healthy, and the official best practices for performance, security and cost.

Validity note: written in October 2026, checking every command, parameter and limit against the official Lambda MicroVMs docs. The service is new and the docs are still moving (I even found two spellings of the same log group on different pages), so I'll keep this post up to date. If something doesn't match what you see in your console, let me know in the comments. Official docs: Best practices for Lambda MicroVMs.

Connecting a shell to a Lambda MicroVM

There are two ways in. In the console, you select the MicroVM and click Connect. From the CLI, you generate a shell token and connect:

aws lambda-microvms create-microvm-shell-auth-token \
  --microvm-identifier <id> --expiration-in-minutes 30

# once inside, list the tasks and exec into one:
ctr task ls
ctr task exec -t --exec-id shell <id> /bin/sh

Shell is opt-in. The MicroVM has to be launched with the SHELL_INGRESS network connector, and if it came up without it, create-microvm-shell-auth-token returns a ValidationException. Connectors are set on run-microvm and don't change while the MicroVM is running, so you decide at launch time which MicroVMs accept that way in.

For the rest of your debugging, three fields help a ton: terminationMessage on get-microvm (for a MicroVM that died), the build logs in CloudWatch under /aws/lambda-microvms/<image-name>, and StateReason when a network connector lands in FAILED.

Lifecycle hooks: what keeps the MicroVM standing

Remember the snapshot caveat (the MicroVM boots from a frozen state, so anything that needs to be fresh can't be frozen)? AWS handles it with lifecycle hooks: four HTTP endpoints your app implements, which the service calls with a POST at the right moment.

/run: post-initialization. This is where you generate UUIDs, secrets and random values, never at image build time, or every MicroVM is born with the same "random". Use a CSPRNG. /run also receives in its body the runHookPayload (per-MicroVM config, such as a tenant ID, in a string of up to 16 KB that you pass with --run-hook-payload on run-microvm) and the microvmId, which Lambda injects.
/suspend: graceful cleanup before suspending (flush pending writes, close connections, release resources).
/resume: revalidates connections and cached state. AWS SDK connections usually recover on their own; custom connections may not, so check them here. The MicroVM stays SUSPENDED while this hook runs and only flips to RUNNING once it returns 200.
/terminate: flushes data before shutting down. It only runs on final termination; before a suspend, /suspend is the one doing the work.

The hooks arrive on the path /aws/lambda-microvms/runtime/v1/<hook>, on a port you configure, and external traffic only starts flowing after /run returns 200. If /run fails or times out, the MicroVM can go straight to TERMINATING without ever reaching RUNNING, so handle errors and timeouts inside the hook.

About the port: the docs don't tell you to split hooks and app, and in the example below both share 8080, the default port for external traffic. Make that call on purpose, looking at which ports the access token opens through allowedPorts.

There are also two hooks for the image build phase. /ready returns 200 once the app has finished starting, and that's when the service takes the disk and memory snapshot (while you're not ready, return 503 right away and Lambda keeps retrying until the timeout). /validate is called after the build, on a fresh MicroVM created from the image: that's where you run test payloads that exercise your code paths, and Lambda tracks which snapshot regions were touched so it can prefetch them and cut startup latency. So /ready makes sure the snapshot captures an app that's ready to serve, and /validate is the one that speeds up boot.

In our Node HTTP server it looks like this:

const http = require("http");
const crypto = require("crypto");

let sessionSecret; // generated in /run, NEVER at image build time

function ok(res, body) {
  res.writeHead(200, { "Content-Type": "application/json" });
  res.end(JSON.stringify(body));
}

const server = http.createServer((req, res) => {
  const HOOK = "/aws/lambda-microvms/runtime/v1";
  switch (req.url) {
    case `${HOOK}/run`:        // randomness and init live here; return 200 before traffic comes in
      sessionSecret = crypto.randomUUID();
      return ok(res, { ready: true });
    case `${HOOK}/suspend`:    // cleanup before freezing
      return ok(res, { cleanedUp: true });
    case `${HOOK}/resume`:     // revalidate connections/state on wake-up
      return ok(res, { revalidated: true });
    case `${HOOK}/terminate`:  // flush data on shutdown
      return ok(res, { flushed: true });
    default:
      // ... your actual application
      return ok(res, { hello: "world" });
  }
});

server.listen(8080, "0.0.0.0");

If the app stops answering after a resume, suspect number one is a missing /resume (or the app not binding to the right port, like 8080, after it wakes up).

Quick proof: the snapshot freezes your "random"

That warning about generating randomness in /run sounds like a detail until you measure it. I put together a small demo that runs locally, on my machine: no AWS, no cost, just the Node stdlib. It simulates the build producing the snapshot and boots two "MicroVMs" from it. The code ships with the demo post, the series finale.

The output, captured for real:

== 3. antes do /run: o servico nao entrega trafego ==
amostra-A  GET /  -> HTTP 503

== 5. depois do /run 200: trafego liberado ==
amostra-A  GET /  -> HTTP 200

== 6. veredito ==
token gerado no BUILD  A vs B: IGUAL     <- colide entre analises
   A: 6fcf9c71-2024-4901-acfb-2593ed5a2fbb
   B: 6fcf9c71-2024-4901-acfb-2593ed5a2fbb
token gerado no /run   A vs B: diferente <- ok
   A: 2011db3c-3120-4b9a-b89d-caa48b034c8f
   B: 496f5bfc-7c7f-4cd7-b39f-2ea3c9b8c281

(The demo prints in Portuguese: "antes do /run" is before /run, "IGUAL" is identical, "diferente" is different.) You can see both things in the same output: the token baked at build time comes out identical in both, and traffic only gets a 200 after /run responds.

Heads up: this demo does not run on Lambda MicroVMs. It exercises the hooks contract locally, with a snapshot.json standing in for the real freeze. It's there to show the behavior. Proof on the real service only comes from running on a MicroVM in AWS, and that's for the demo post, the series finale.

Official best practices, by pillar

Performance. Pick the smallest MicroVM that covers your CPU, memory and bandwidth (bandwidth scales with configured memory). Keep the snapshot lean: strip build dependencies and temp files before the image is captured, which cuts run and resume time. And for latency-sensitive workloads, keep your most-used environments suspended so they resume fast.

Security. Issue short-lived tokens (15 to 30 min) and have refresh logic on the client before they expire. Scope the token to the ports the app actually uses, with the allowedPorts parameter. Sensitive traffic (database, internal API) goes through a VPC egress connector, which goes through your security groups and network ACLs. And apply least-privilege: split the build role (image creation) from the execution role (runtime), each with the bare minimum. Handy detail: /ready and /validate run under the build role, and the runtime hooks run under the execution role.

Cost. Set maxIdleDurationSeconds in the idle policy to suspend idle MicroVMs (a suspended MicroVM pays no compute, only snapshot storage). Put a safety ceiling with the maximum duration (--maximum-duration-in-seconds in the CLI, up to 8 hours, counting running plus suspended time) so a forgotten instance doesn't surprise you on the bill. Use suspendedDurationSeconds to automatically terminate a MicroVM that sat parked for too long. Tag your images for cost allocation in Cost Explorer, and delete image versions you no longer use, because they cost storage too.

Before you decide how much isolation and availability your use case needs, it's worth looking at the price of each step up. I ran those numbers in High availability has a price: resilience is a decision, not a stack.

Browser apps: a proxy in front of the MicroVM

An app that runs in the browser (a VS Code server, a web IDE) runs into a rule in the docs: every request to the MicroVM endpoint needs the JWE token in the X-aws-proxy-auth header, and there's no unauthenticated option. When the user opens a URL, clicks a link or loads an iframe, the browser won't let you set that header. For a WebSocket opened by your own JavaScript, the docs let you pass token and port as subprotocols (lambda-microvms.authentication.<token>), but that doesn't cover regular navigation.

The architecture answer is a reverse proxy of your own in front (the Gateway pattern): a component that takes the browser request already authenticated the way your app does it, fetches or mints the MicroVM token, and injects X-aws-proxy-auth before forwarding. It also needs a user → MicroVM map, because each MicroVM has its own endpoint and there's no load balancing across them. With that map, when the user closes the tab and comes back later, you know which instance to route to. The orchestrator that already calls run-microvm is a natural place to keep that map.

If all you need is a custom domain, AWS published a pattern with an Application Load Balancer rewriting the Host header, but there the client still has to send the token. It doesn't replace the auth proxy.

If what's orchestrating these MicroVMs is an AI agent operating your account, the other side of the coin is permission control, which I covered in AWS MCP Server: which one to use, when, and how to set it up.

The errors you'll see (and what to do)

From the troubleshooting table in the docs, these are the ones I'd keep at hand:

502 Bad Gateway: the app crashed or isn't listening. Check the EXPOSE and CMD in your Dockerfile and the runtime logs. With auto-resume on, it can also be a MicroVM that failed to resume (check its state with get-microvm).
429 Too Many Requests: you went over the rate (account or MicroVM). Retry with exponential backoff and jitter.
Dropping connections: idle timeout. Implement keepalive (ping/pong) or raise maxIdleDurationSeconds.
Build failed (CREATION_FAILED): look at the logs under /aws/lambda-microvms/<image-name>, check your Dockerfile syntax and the build role's S3 permissions. Run docker build locally to reproduce.

And for basic monitoring: when a MicroVM terminates out of nowhere, get-microvm returns a stateReason with the cause. That's the first place to look.

What you take away

You can shell into Lambda now, via the console or create-microvm-shell-auth-token, but only if the MicroVM was launched with the SHELL_INGRESS connector.
The lifecycle hooks (/run, /suspend, /resume, /terminate) are what keep the MicroVM healthy. Generate randomness in /run, revalidate connections in /resume, and use /validate to shorten boot time.
Security: short-lived tokens with refresh, per-port scope, VPC egress, and a build role separate from the execution role.
Cost: suspend when idle, cap the duration, terminate what sat parked, and clean up image versions you don't use.

Sources

Best practices for Lambda MicroVMs
Core concepts (states and transitions)
Running and using MicroVMs (lifecycle hooks, run hook payload, suspend/resume)
MicroVM images (build hooks /ready and /validate)
Security and permissions (IAM roles, tokens, shell)
Networking (authentication, ports, VPC egress)
Troubleshooting
Adding custom domains to AWS Lambda MicroVMs with Application Load Balancer

Originally published on willpeixoto.dev.

Running a MicroVM use case already? Tell me which hook saved you (or burned you for not having it). And if you're from the infra side and always wanted to shell into a Lambda, now's your moment. If this post helped, drop a like and share it around. Thanks a lot!

BUILD. SCALE. REPEAT. =D

Lambda Web Functions: when to switch from Event Functions, and what to do with no South America region

will peixoto — Mon, 02 Nov 2026 14:00:00 +0000

Lambda Web Functions runs your Node.js HTTP server directly on Lambda, with many requests per environment and billing for active CPU. That changes the math for SSR and for APIs that spend the day waiting on a database. The catch: AWS launched it in five regions, and none of them is in South America.

If you've ever put Express on Lambda, you know the ritual: Lambda Web Adapter or serverless-http translating events into req/res, and a bill that charges for every millisecond spent sitting there waiting for Postgres to answer. That's what AWS changed on October 12. And if your users are in Brazil (mine are, I'm writing from São Paulo), the launch comes with a region decision baked in.

Validity note: written in October 2026, six days after the announcement, checking limits, pricing and IaC against the official docs. Freshly launched services pick up regions and runtimes fast, and I'll update this post when that happens (the first update will be a South America region showing up). If something doesn't match what you're seeing, tell me in the comments. The living reference is the official Lambda Web Functions documentation.

What Lambda Web Functions is (and what happened to the Lambda you already use)

Lambda Web Functions is a new function format where your code is a regular HTTP server listening on a port, instead of an (event, context) handler. The AWS News Blog mentions Express, Hono, Fastify, Next.js "or any Node.js framework", with no adapter in between.

The launch also came with a rename. The Lambda you already know is now called Lambda Event Functions, with no code change and no redeploy. Lambda now has three formats, all built on Firecracker: Event Functions, Web Functions and MicroVMs. Each one serves a different programming model (I broke down the third one in my post on Lambda MicroVMs for untrusted code isolation).

The new vocabulary, anchored in what you already use:

Web Function is the top-level resource, the umbrella.
Revision is an immutable snapshot of code and config (.zip package in S3, runtime, timeout, env vars, execution role, per-environment concurrency, network connector). Think of an Event Function version: change any of those fields and a new revision is born.
Endpoint is the HTTPS front door, with its own domain and TLS. It plays the role of a weighted alias: it routes to at most two revisions, with a percentage weight (integer from 1 to 100) on each. Canary and blue/green come built in, no CodeDeploy needed.

What changes in the execution and billing model

With Event Functions, every table gets its own waiter. The waiter takes the order, walks it to the kitchen, then stands next to the table waiting for the dish. You pay that waiter for the whole time standing there, including the time spent staring at the kitchen door. That's the "1 request = 1 environment, billed by wall-clock duration times configured memory" model.

With Web Functions, one waiter covers several tables: while one dish is in the kitchen, they take the order at the next table. And you only pay for the time they're actually working. In the docs, that turns into two changes.

The first is per-environment concurrency: each execution environment serves several requests in parallel. The ceiling is maxConcurrencyPerEnvironment, set on the revision, from 1 to 128, default 64. Below that ceiling, Lambda itself adjusts the limit dynamically based on resource usage.

The second is active CPU billing. The pricing page splits it into three dimensions (us-east-1, Arm):

Dimension	Price
Active vCPU	$0.0000276944 per vCPU-second
Memory	$0.0000036667 per GB-second
Requests	$0.30 per million

Granularity is 1 ms, with data transfer billed separately. Time spent waiting on a query or an API call burns very little CPU, and only active CPU goes on the bill. Memory follows a different rule: the environment pays for the memory it consumes for as long as it exists, including the gaps between requests.

The rest of the execution contract, straight from the docs:

Lambda injects AWS_LAMBDA_HTTP_ENDPOINT (host:port), and your server has up to 10 seconds to listen on exactly that address. As soon as it's listening, Lambda marks the environment as ready and starts sending requests.
On shutdown you get a SIGTERM, and the process has up to 1 second to clean up.
It scales to zero with no traffic. Cold starts happen when a new environment comes up; for small traffic increases, Lambda keeps some headroom of environments ahead of demand.
The environment is long-lived and isn't frozen between requests the way an Event Function is.

Hands-on: the server and the deploy

The minimum code is an HTTP server. Here's a version with node:http, zero dependencies, following the contract in the docs:

// index.js (the runtime loads index.js from the package root by default)
const http = require('node:http');

const [host, port] = process.env.AWS_LAMBDA_HTTP_ENDPOINT.split(':');

// Global scope: process-level things only (SDK client, connection pool, config).
const startedAt = new Date().toISOString();

const server = http.createServer((req, res) => {
  // Everything that belongs to the request stays in here, never in a global.
  const requestId = req.headers['x-request-id'] ?? 'n/a';

  if (req.url === '/health') {
    res.writeHead(200, { 'content-type': 'application/json' });
    return res.end(JSON.stringify({ ok: true, startedAt }));
  }

  res.writeHead(200, { 'content-type': 'application/json' });
  res.end(JSON.stringify({ message: 'Hello from Lambda Web Functions!', requestId }));
});

server.listen(Number(port), host, () => {
  console.log(`listening on ${host}:${port}`);
});

// You get up to 1 s after SIGTERM. Don't count on it to persist data.
process.on('SIGTERM', () => {
  server.close(() => process.exit(0));
  setTimeout(() => process.exit(0), 800).unref();
});

The x-request-id header is illustrative (the right header depends on your front door). The example is really about scope: with up to 128 requests in the same process, a global variable holding the current request's user leaks one customer's data to another.

For deploys, the fastest path is the CLI's aws lambda-web deploy. For IaC, CloudFormation already has three types: AWS::Lambda::WebFunction, AWS::Lambda::WebFunctionRevision and AWS::Lambda::WebFunctionEndpoint. The template below only uses properties that appear in the CloudFormation reference:

AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  CodeBucket:
    Type: String
  ExecutionRoleArn:
    Type: String

Resources:
  WebApp:
    Type: AWS::Lambda::WebFunction
    Properties:
      FunctionName: my-api

  WebAppRevision:
    Type: AWS::Lambda::WebFunctionRevision
    Properties:
      FunctionName: !Ref WebApp           # accepts name or ARN
      Description: first revision
      BuildConfig:
        CodeConfig:
          S3Object:
            Bucket: !Ref CodeBucket
            Key: app.zip
        RuntimeConfig:
          Runtime: nodejs24.x
      ServiceConfig:
        ExecutionRoleArn: !Ref ExecutionRoleArn
        TimeoutSeconds: 30
        MaxConcurrencyPerEnvironment: 32  # 1 to 128, default 64

  WebAppEndpoint:
    Type: AWS::Lambda::WebFunctionEndpoint
    Properties:
      FunctionName: !Ref WebApp
      EndpointName: prod
      EndpointType: HomeRegion            # can't change after creation
      AuthType: IamAuth                   # required for CloudFront with OAC
      RevisionWeights:
        - RevisionId: !GetAtt WebAppRevision.RevisionId
          Weight: 100
      ScalingConfig:
        MaxEnvironments: 20               # optional, minimum 2

Outputs:
  EndpointDomain:
    Value: !GetAtt WebAppEndpoint.DomainName

The CLI deploy creates an ApplicationManaged endpoint (public, no Lambda-level authentication) by default, which is why the template ships with IamAuth. Also, EndpointType and regions are fixed: to change them, you create a new endpoint and move your clients to the new domain. The What's New post also mentions AWS SAM and AWS CDK; I only checked the CloudFormation reference property by property, so the template is plain CloudFormation.

When to switch from Event Functions to Web Functions

The docs list the Web Functions use cases: full-stack web apps, REST and GraphQL APIs, HTTP microservices, SSR and response streaming (LLM output, for example). Queues and file uploads, per the same docs, stay on Event Functions.

My rule of thumb is the ratio between CPU and waiting. If a request takes 200 ms end to end and 50 ms of that is CPU (the rest is database, external API, AI model), on Event Functions you pay for the 200 ms; on Web Functions, you pay for the 50 ms of CPU plus the environment's memory. LLM streaming is the extreme case: seconds of open connection, almost no CPU.

The reverse holds too. If the request is pure CPU (generating a PDF, resizing images, heavy crypto), the advantage disappears and a risk shows up: synchronous CPU blocks the event loop for every in-flight request in that environment. The docs tell you to use worker_threads or lower maxConcurrencyPerEnvironment. That's the Bulkhead pattern applied inside the process: you cap how many requests share the same resource so one slow request doesn't sink the rest.

The trade-offs (it's not all sunshine and rainbows)

Shared state across requests

One request per environment forgave a lot of sins. The official best practices forbid credentials, customer data, response objects or mutable request state in global variables. Connection pools and SDK clients can stay global, as long as you validate stale connections and cap caches. And the 512 MB /tmp is shared across requests, so file names have to be unique.

Node.js only, Arm64 only

At launch it's nodejs24.x only, Arm64 only. Native modules compiled for x86_64 won't run. There are no runtime management controls: you can't pin a patch version. Java, Python or .NET teams stay on Event Functions (or MicroVMs).

Fixed size

Every environment gets 2 GB of memory, no choice, split between heap, buffers, native modules and every in-flight request. The default V8 heap is 1,024 MB; you can raise it with NODE_OPTIONS=--max-old-space-size=..., as long as you leave room for everything else. On Event Functions you pick anything from 128 MB to 10 GB.

Pricier requests

$0.30 per million, versus $0.20 on Event Functions: 50% more. In high-request, low-CPU workloads, the request line adds up: in the official auth middleware example (100 million requests), $30.00 of the $64.56 total is requests alone.

The fine print

HTTP/1.1 only (HTTP/1.0 gets a 505), the Upgrade header is stripped (WebSocket doesn't show up as supported in the docs), and the service metrics have no revision dimension. During a canary, you log the revision ID yourself in the application.

Let's run the numbers (with what's confirmed)

I'll use the official I/O-heavy API example from the pricing page, because every assumption comes from AWS: us-east-1, Arm, 10 million requests a month, 200 ms end to end, 50 ms of active CPU per request, a single environment serving everything for 30 days (2,592,000 seconds), 0.29 GB of memory consumed, no free tier and no data transfer.

CPU: 10,000,000 × 0.05 s = 500,000 vCPU-seconds × $0.0000276944 = $13.85
Memory: 0.29 GB × 2,592,000 s × $0.0000036667 = $2.76
Requests: 10 × $0.30 = $3.00
Total: $19.61

Memory is the open question. The runtime docs say the environment has a fixed 2 GB, and the example uses 0.29 GB. The pricing page settles half of it, because it talks about memory consumed by the environment. It doesn't explain how that consumption is measured (peak, average, sampling). If it were the full 2 GB, the memory line in this example would jump to about $19 and the total would nearly double. That's why I'm not handing you an Event versus Web comparison built on made-up numbers. Measure before you decide:

CPU time per request versus total duration (today's Duration on your Event Function is the ceiling).
Process memory under the concurrency you plan to run, compared with the memory line on the bill.
How many environments stay alive throughout the day, because memory is billed per live environment, even idle ones.
Data transfer, which is left out of every official example and weighs heavily in the cross-region scenario in the next section.

This ties straight into the Cost Optimization pillar of the Well-Architected Framework: measure real demand before picking the pricing model.

The gap: no South America region at launch

The launch regions are us-east-1, us-east-2, us-west-2, ap-northeast-1 (Tokyo) and eu-west-1 (Ireland). Nothing in South America, so São Paulo (sa-east-1) is out, and I couldn't find any official timeline for when it comes in. Brazil is the concrete case I deal with every day, but the same reasoning applies to anyone whose users and data live far from those five regions. If that's you, the question stops being "Web or Event?" and becomes "is it worth crossing a continent to use Web Functions?".

The path AWS left ready is CloudFront. On the same day, CloudFront shipped support for Web Functions as an origin, with Origin Access Control and at no additional cost. The configuration confirmed in the docs:

Endpoint with AuthType: IamAuth.
OAC with origin type lambda and signing behavior always-amz-auth (CloudFront rejects the others).
A resource-based policy on the Web Function allowing lambda:InvokeWebFunctionEndpoint for the cloudfront.amazonaws.com principal, with aws:SourceArn pinned to your distribution.

{
  "Principal": { "Service": "cloudfront.amazonaws.com" },
  "Action": "lambda:InvokeWebFunctionEndpoint",
  "Condition": {
    "StringEquals": {
      "aws:SourceArn": "arn:aws:cloudfront::111122223333:distribution/<distribution-id>"
    }
  }
}

On the Security side, the endpoint domain is still reachable from the internet even behind CloudFront. With IamAuth, only requests signed by someone your policies allow get through; with ApplicationManaged, anyone who knows the domain can call your application. To lock that down across the organization, the docs show an SCP using the lambda:Request/WebFunctionAuthType condition key to deny ApplicationManaged.

Where your data lives drives the architecture. CloudFront brings the edge closer, but the function is still in the United States. If your database is in sa-east-1, every query from your function crosses the continent, and an SSR page with five queries pays that trip five times before returning a single byte, with inter-region data transfer on the bill. There's no official latency number for this, so measure it: the time of one query from us-east-1 to your database, times the number of queries on the route.

There's also a networking limit: the egress network connector, which gives you access to your VPC, only works with a HomeRegion endpoint. It doesn't give you internet access (you need a NAT gateway). Without a connector, the egress IP is managed by Lambda and is no good for an allow list; with a connector, the docs tell you to allow the destination by security group or by the subnets' CIDR, because there's no stable source IP either.

Here's the decision table:

Scenario	Recommendation
I/O-bound API or SSR, data can live in us-east-1, no requirement to keep data in Brazil (from a customer contract or your reading of LGPD, Brazil's data protection law, similar to GDPR)	Web Functions in us-east-1 + CloudFront with OAC, data in the same region
Database has to stay in sa-east-1 (contract, compliance, latency to other systems)	Stay on Event Functions in sa-east-1, behind API Gateway or a Function URL
CPU-bound workload, queues, file uploads	Event Functions, in any region
Java, Python or .NET stack	Event Functions (or MicroVMs)
Authorizer or middleware with lots of requests and little CPU	Run the numbers: the 50% pricier request can eat the CPU savings
Lots of global state in the code, no time to refactor	Stay on Event Functions until the global scope is clean
Need VPC access with a multi-region endpoint, or an egress IP for an allow list	Event Functions; on Web Functions, VPC only with a `HomeRegion` endpoint + connector (and NAT for internet), and validate the egress IP before promising an allow list

I covered this kind of region trade-off, in terms of cost and availability, in high availability has a price. And concentrating everything in us-east-1 carries its own risk, as I explained in the post on why multi-cloud can make you less resilient. From a Reliability standpoint, leaving your home region means trading one dependency for another, so choose with your eyes open.

What you take away

Web Functions runs your Node.js HTTP server on Lambda with no adapter, with up to 128 requests per environment (default 64), billing for active CPU, memory consumed while the environment is alive, and $0.30 per million requests.
The win is in I/O-bound workloads: SSR, APIs that wait on a database, LLM streaming. CPU-bound work, queues and uploads stay on Event Functions.
The price is discipline (zero request state in global variables, no synchronous CPU on the event loop) plus accepting a fixed 2 GB and Node.js 24 on Arm64 only.
Before comparing costs, measure CPU per request, consumed memory and live environments (the docs don't detail how billed memory is measured).
With no South America region, the real decision for a Brazilian workload is: data in us-east-1 with CloudFront + OAC in front, or stay on Event Functions in São Paulo. With the database on the other side of the continent, per-query latency can eat whatever you gained by switching.

Sources

Originally published on willpeixoto.dev.

So, are you running SSR or a Node API on Lambda Web Adapter today? Would you move it to a launch region behind CloudFront, or hold it close to your users until a nearby region shows up? Tell me in the comments, I want to take this conversation to the next AWS User Group São Paulo meetup. If this post helped, drop a like and share it around. Thanks a lot!

BUILD. SCALE. REPEAT. =D

Invisible Coupling: When Your Microservices Become a Distributed Monolith

will peixoto — Fri, 30 Oct 2026 12:00:00 +0000

Black Friday peak. The checkout has microservices, a CI/CD pipeline, containers, serverless functions, every piece with its own deploy and a green dashboard. Then the settlement service, way at the end of the chain, starts responding slowly, and within minutes the whole payment authorization flow is timing out. On paper, every service was independent. The customer still gave up on the purchase while waiting for the approval.

That scene has a name, invisible coupling, and so does the system it produces: a distributed monolith. If this sounds familiar, stick with me, this post is for you. It kicks off a short run of posts on the topic: next I'll show how to break these dependencies by separating the hot path from the side-effects, walk through the patterns that hold that split together, and wrap up hands-on with a prototype on AWS.

Validity note: written in October 2026. The argument here is about architecture and doesn't expire with a release, but the AWS links (Well-Architected, Builders' Library) get reorganized from time to time. If one of them points somewhere odd, let me know in the comments and I'll fix it.

Invisible coupling: what a distributed monolith really is

Coupling is a measure of how much one service depends on another to work. My provocation, right up front: if your microservice only responds because another one responded first, that independence is an illusion. In practice, it's part of a distributed monolith.

Sam Newman explores that term, distributed monolith, in Monolith to Microservices, and he describes exactly the trap: you paid the full price of running separate services (network, deploys, observability, tracing) and didn't collect the prize, which was independence. The danger is the false sense of modernity. All the microservices machinery is there, but when one point fails, the system fails with it. Distributing the system didn't erase the coupling, it just made it harder to see.

The comment that made me write this

A few days ago, in a discussion about serverless, I read an argument that went roughly like this (translated from Portuguese):

"In complex environments, when microservices depend on others, and those depend on third parties, the latency of those requests drives cost way up and can make the model unviable. Also, in compute-heavy scenarios, serverless may not be the best option either because of cost."

This argument shows up all the time, and it touches on something real: dependency chains between microservices are expensive, fragile and slow. So far, I agree.

Except this isn't a serverless problem. It's an architecture problem. Running on AWS Lambda, ECS, EKS or a VM in your server closet, if the integration is poorly designed the outcome is the same: latency, cost and fragility. A synchronous chain doesn't get faster because you swapped the runtime underneath it.

That's why the "serverless vs. containers" debate feels to me like fighting over the wrong dish. Each one has its place (stateful workloads and long-running jobs call for containers; event-driven and spiky traffic is where serverless shines), but that catalog is for another day.

The central point of this post is this:

A microservice's independence is decided by how you design its integration, and the runtime doesn't get a vote.

Temporal coupling and accumulated latency

Temporal coupling happens when a service can only respond if another one is up at that exact moment. The availability and latency of one service start contaminating every other service in the chain.

Then comes the cascade. Each synchronous hop adds a few milliseconds, which looks like nothing at first. Then you add up five calls in series and latency blows up right in the tail percentiles, those slowest requests that make up 5% or 1% of the total. That's where the customer either completes the purchase or abandons the cart.

Let's do some back-of-the-napkin math, with illustrative, hypothetical numbers, just to picture it. Five services in series, each responding in 40ms at the median: the whole chain sits around 200ms. Fine so far. But real requests don't live at the median. If each hop has a 1% chance of hiccuping and taking 2 seconds, in a chain of five the chance that at least one of them hiccups gets close to 5%. One in every twenty transactions will feel it. And the customer who felt it doesn't care that YOUR average looks great. It's the same series-component arithmetic that drags availability down in the post on why multi-cloud can leave you less resilient: every extra link multiplies the odds of something going wrong.

The distributed monolith in action: payments

Back to the payment from the opening, because that's where every millisecond counts. The customer is standing at the checkout waiting for approval, or the e-commerce site needs to confirm before they give up. A delay of a few seconds costs revenue and trust.

And the bill doesn't stop at the merchant: the card network charges too. Mastercard's Transaction Processing Rules (sections 2.3 and 2.4) require the issuing bank to answer the authorization within a maximum time, and if the answer doesn't arrive in time the transaction times out and falls to Stand-In Processing. For Maestro and ATM transactions, an issuer with more than 2% of failures from unavailability, malfunction, or timeout for two consecutive months is already below standard, and anyone who misses the performance standards is subject to a USD 15,000 assessment on the first occurrence, rising to USD 20,000 from the third within 12 months. In payments, latency is part of the contract.

In the traditional synchronous model, an authorization walks down this staircase: the Authorization service calls Fraud, which calls Limits, which calls Settlement. The response only climbs back to the customer once the last one answers. Look at the linear shape of the diagram below: nobody moves until the one below responds.

That's the distributed monolith in practice: several services tied into a single linear flow. Each hop adds latency, multiplies logs, traces and retries, and adds one more point that, if it goes down, takes the whole transaction with it.

At low load, it works. The problem shows up under high concurrency: the Black Friday from the opening, a promotion that goes viral, the daily volume of a digital bank. Average latency even looks acceptable, but p95 and p99 shoot up (remember the napkin math?). Cost grows with it, and retries make the bill worse: the Amazon Builders' Library article on timeouts and retries describes a five-deep stack of service calls where each layer retries three times on its own, and the load on the database at the bottom gets multiplied by 243. In serverless, the cost of waiting is literal: in a synchronous chain of Lambda functions, the functions at the top keep running (and keep being billed for duration) while they wait on the ones below, the anti-pattern AWS itself documents. And a small percentage of slow transactions is enough for the frustration to show.

How we got here

Nobody woke up one day wanting to build a distributed monolith, so it's worth understanding why this pattern spread so widely. During the microservices adoption wave, one situation became routine: "I need data that lives in another microservice". Nothing wrong there, that's normal and acceptable in any distributed system. The problem was the answer, which became something close to a mantra: "then build an API, the other service exposes it and I consume it". That was the by-the-book approach at a lot of companies, and I've watched the pattern repeat itself in projects I worked on and in conversations with people in the field.

And it was attractive. It leaned on good practice: well-documented REST APIs, OpenAPI/Swagger contracts, fast integration. For a while, it worked. Once the domain starts demanding low latency, high availability and resilience, the model shows its seams. Every new synchronous dependency is more temporal coupling. What was born to decouple turned into another kind of coupling, distributed and far more expensive to maintain.

Try this at home

Want to know if this story is yours? Run a thought experiment, no need to break anything in production: pick a service in your system and imagine it going down right now. How many others stop with it? If the answer is "the whole transaction", invisible coupling already lives there.

A useful variation to bring to your next standup: can you deploy this service at peak time without warning the neighboring teams? If the answer is no, that independence only exists on the org chart. And that's exactly Sam Newman's criterion: when a set of services has to go to production together, you have a distributed monolith.

Where independence is actually designed

So the deciding factor is how the services talk to each other. A synchronous chain creates temporal coupling and makes everything fragile. The opposite path has a name and tooling too: Event-Driven Architecture (EDA) to reduce direct dependencies, clear domain boundaries so each service owns its own data, and the split between the hot path (the fast path the user is waiting on) and the side-effects (analytics, notifications, the receipt email, which can run in parallel without holding anyone in line). Werner Vogels summed it up in his re:Invent 2022 keynote with "the world is event-driven": the real world is asynchronous, and your architecture thanks you when you stop pretending it isn't.

The same payment, now event-driven

Go back to the payments staircase. In the redesign, the hot path keeps only what decides the approval: Authorization checks Fraud and Limits, responds to the customer and publishes a "payment authorized" event. Settlement, Notification and Analytics become consumers of that event, running in parallel, each at its own pace. If Settlement is down, Authorization keeps responding, and the event waits in the queue until it comes back. That event can travel over a queue, a bus or a stream, and each choice has its own trade-offs around ordering, replay and cost (I compared the streaming options in the post on Kinesis, Firehose, Flink or MSK).

In the card world, settlement is already asynchronous by nature: authorization happens on the spot, and clearing and settlement come later, usually in batches. The synchronous design we saw earlier was fighting the domain itself.

There's a whole toolbox behind this split (Circuit Breaker, Bulkhead, Outbox, CQRS, Saga, idempotency keys) and, above all, the decision of when to use each one. And none of them comes for free: in the post on the price of high availability I argued that resilience is a decision with a cost, and every one of these patterns goes into that bill. The rest is material for the next posts, where I'll go pattern by pattern. For now, hold on to the core idea: what separates an independent microservice from a hostage is the integration.

This ties straight into the Reliability pillar of the Well-Architected Framework. One of its best practices, implement loosely coupled dependencies, recommends making interactions asynchronous where possible, usually with a durable layer in between (an Amazon SQS queue, a Kinesis stream, AWS Step Functions), precisely so that one component's failure stays isolated instead of taking down the entire chain.

Remember the scene from the beginning, the whole checkout timing out because settlement, at the end of the chain, choked? In the event-driven design, that same slowness becomes a growing queue, and the customer gets the approval. What took the checkout down was the design of a chain built to fall together, and a design can be redone. No rewriting the system from scratch and no runtime migration: the work is to go through the integrations one by one and ask which ones really need an answer right now.

What about that comment from the beginning?

I'm coming back to it because this whole post is an answer. In its first part, the comment got the symptom right: a microservice that depends on another, which depends on a third party, gets slow and expensive. It just blamed the wrong culprit. The latency comes from the synchronous chain, and that chain is just as slow on containers, on a VM or on Lambda. And a good part of the cost it put on serverless's tab is the cost of waiting: Lambda calling Lambda pays for duration while it waits for the answer. Take the waiting out of the path, with a lean hot path and everything else turned into events, and that bill drops along with the latency.

In the second part, about heavy processing, it has a point. A workload with high, constant CPU can come out cheaper on containers, and that runtime choice is worth making case by case. But that's a different conversation. Swapping Lambda for containers in a synchronous chain only changes where the waiting happens.

What you take away from this post

If I could leave you with just a few ideas, these would be the ones:

If your service only responds after another one has responded, it's part of a distributed monolith.
Temporal coupling is the villain: one service's latency and availability contaminate the whole chain, and the bill shows up in p95/p99, far from the average.
Swapping the runtime doesn't fix a synchronous chain. Serverless or containers, what matters is how the integration is designed.
Independence is designed: EDA, clear boundaries, the hot path separated from the side-effects, and the named resilience patterns (Circuit Breaker, Bulkhead, Outbox).

So, to finally answer that comment: the question was never whether to use serverless. What decides whether your system survives Black Friday is how you design the integration between services, and that holds for any runtime.

Originally published on willpeixoto.dev.

In the next post I'll show how to break these dependencies in practice, separating the hot path from the side-effects to cut latency and make your microservices truly independent.

Before that, I want to hear from you: have you ever heard (or said) that serverless gets expensive when one service depends on another? After this post, does that argument still hold up? Tell me in the comments. If this post helped, drop a like and share it around. Thanks a lot!

BUILD. SCALE. REPEAT. =D

AWS MCP Server: self-host or managed, OAuth or SigV4, and how to set it up

will peixoto — Tue, 14 Jul 2026 13:30:16 +0000

🇧🇷 Também disponível em português: AWS MCP Server: qual usar, quando e como configurar

Knowledge MCP vs the managed AWS MCP Server, IAM, and the setup in Claude Code, the Claude app, and Kiro without pasting a key into .env.

Let me tell you a scene. I bet you've lived it.

It's 11pm, you're coding with an AI sitting right next to you. You ask it something and it swears the thing exists: it makes up a service, a name that sounds a lot like something you've built before, even an implementation it promises works, with the full how-to. And you think: huh, didn't know that one, let me check the docs because it makes total sense. And bang, you find out it was hallucinating. It was writing Lambda code calling an AWS service from a function that doesn't exist, wiring up the DynamoDB table, building the IaC (Infrastructure as Code), the whole thing. Then it asks to "take a look at your account's current config" to validate the names, and you hit the wall everyone hits.

The agent, besides hallucinating, has no access to your account. It can't see your resources, it has no idea what's actually there. So it does what it does best: it's creative, it's proactive, and it invents something to make you happy. To it, that thing makes sense to exist, so it assumes it exists. The little guy doesn't even blush: it hands you an ARN (Amazon Resource Name) that doesn't exist, picks a region you don't use, and guesses a table name. There it goes. It's doing exactly what it was built to do, which is make you happy.

And you, tired of fighting with it and just wanting to wrap up, do what? You paste an access key into a .env so it can "just see the account for a second" and give you everything right, with the real names. Who's never done that?

And it's all fine, except for one detail: you forget you did it. And bang, the .env went along in the commit. It's already in the git history, alerts firing everywhere, the usual mess. The key is exposed now. And that's also how your agent, in that moment where it seems to want to punish you or decided to be a little too proactive, gets real access to your account and, who knows, decides to delete the wrong stack. Or it's just the surprise bill of forty bucks in a single day, which turns into a much worse number when nobody's watching. I've been through this. You probably have too.

The good news is that AWS solved this. Actually, it solved it two different ways, with names so similar they confuse a lot of people. This post is the map I wish I'd had: what the two are, which pain each one kills, and when and how to use each. Let's go.

Updated November 2026. The MCP (Model Context Protocol) world moves fast. If something doesn't match the screen in front of you, check the official docs (links at the end) and tell me in the comments so I can fix it.

TL;DR: make the managed AWS MCP Server your default. It already searches the AWS docs with no credentials, and it inspects or operates your account when you authorize it. Keep the Knowledge MCP Server only when you want doc lookup and zero account access. And don't run both at once: duplicated tools confuse the agent. Along the way I show how to connect it in Claude Code, the Claude app, and Kiro, whether you can self-host it, how SigV4 (Signature Version 4) and read-only mode work, the IAM (Identity and Access Management) on the account side, and when NOT to use any of this.

But what is MCP, anyway?

Before talking about the servers, let's level set.

MCP (Model Context Protocol) is the "plug" your AI assistant uses to talk to the outside world: tools, data and services. Anthropic created it, it's under open governance now, and over the last year every assistant worth using (Claude Code, Kiro, Cursor) started speaking MCP.

Want an easy analogy? Think about the API (Application Programming Interface). The API is what lets two applications talk to each other. MCP is the same idea, just for the agent: it's what lets the AI talk to any tool without you having to build a custom integration for every single system. MCP connects the agent to the system through a standard everyone agreed to use. And that's the nice thing about a standard: you learn it once and it works for any tool and any client. So if you want to give an AI access to your system, this is the path to follow.

Should you build your own MCP server or use a managed one?

Use a managed one when what the agent needs is AWS itself: the docs, the APIs, the resources in your account. Build your own when you need to expose your own system (an internal API, a runbook, an alert) that no managed server knows about. What changes is who carries the operation: with the managed AWS MCP Server, AWS runs the hosting and the updates, you sign in with your own IAM identity (OAuth or SigV4), and the calls land in CloudTrail; with your own, hosting, authentication and logging are all on you.

For most AWS cases, you don't even need to build anything. You can build one, sure (on Lambda, on Fargate, on Amazon Bedrock AgentCore Runtime, whatever you prefer), and AWS even has official guidance for that. But AWS already runs managed servers ready to go, so a lot of the time you just plug in and use it.

And it's rarely one or the other. The common setup is both plugged into the same client: the managed one for AWS, yours for the system only your company has.

The two general-purpose AWS MCP servers

AWS actually has a bunch of MCP servers, including specialized ones for specific services and workflows. But the confusion lives in two general-purpose options: the AWS Knowledge MCP Server and the managed AWS MCP Server. And the names don't help at all. Here's everything in one table, then I break down each one:

	AWS Knowledge MCP Server	AWS MCP Server (managed, GA, General Availability)
What it solves	"My agent hallucinates AWS APIs, ARNs and service names."	"My agent needs to see or act on my account, without me leaking a key."
What it accesses	Only AWS docs and knowledge (read-only)	Docs + real services in your account (authenticated)
Credentials	None. You don't even need an AWS account.	AWS browser sign-in (OAuth) or AWS CLI, Command Line Interface (SigV4, Signature Version 4)
Audit	Not applicable	CloudTrail + CloudWatch
Use it when	You want correct syntax, current docs, regional availability	You want the agent inspecting or operating real infra
Risk if misused	Very low: no credentials, it can't touch your account	Real. It's your account. Least-privilege matters.

If you keep one sentence from this post, keep this one:

One server gives your agent knowledge. The other gives it hands.

Figuring out which problem you actually have is half the battle.

Want another analogy? The Knowledge server is the guru: that friend who memorized the entire AWS documentation and clears up your doubt on the spot. The managed one is the doorman checking badges: it lets you into the account for real, but it stands there checking and only opens the doors IAM (Identity and Access Management) authorized. Name doesn't match the badge? You don't get in.

Where does this run? The AWS MCP Server is remote, and what connects to it is the MCP client: Claude Code, the Claude app (Desktop and claude.ai), Kiro, Cursor, or your own agent's code (Strands, SDK, Software Development Kit). That holds even with inference running on Bedrock, because the MCP client is the application. A production agent on AgentCore, though, typically consumes tools through the AgentCore Gateway (which also speaks MCP) or your own MCP server. That production scenario is a topic for another post in this series.

Server #1: AWS Knowledge MCP Server

What this server does is simple: it's remote, fully managed, and it gives the model structured access to the official docs, always current. And that "current" is the point. What the model knows on its own stops at its training date, so it has no clue about what came after and ends up guessing. AWS keeps this server in sync, so it becomes your source of truth: it searches the docs, returns the page as clean markdown, checks whether a service exists in a region, lists the current regions, and now also retrieves AWS agent skills. Read-only, it doesn't write or touch the account.

Why is it almost a no-brainer to turn on? Because there's no credential, and no AWS account needed. Nothing to protect, nothing to leak. The risk is basically zero and the payoff is the agent stops guessing and starts citing the real docs before it spits out the CDK (Cloud Development Kit).

Use it when you're learning a service, designing the architecture you want to build and validating the idea, checking syntax, generating IaC you actually trust, or answering "is this in my region yet?" without opening the browser. Connecting is pasting a URL: add https://knowledge-mcp.global.api.aws as a remote (HTTP) server in your client and that's it, no credentials at all. With it on, the agent checks the live docs before spitting out the CDK instead of guessing from training. Hallucinated ARNs drop off a cliff. Pretty good, right?

Server #2: AWS MCP Server, the managed one (reads and operates the account)

This is the one that cures the .env shame.

The pain is different: the agent needs to see or do something in the account for real. Read the CloudWatch logs of the function that's breaking, list what's actually in the bucket, check the real schema of the DynamoDB table. The old "solution" was to hand it a long-lived credential. That's the part that keeps the security folks up at night. And honestly, it should keep you up too.

What this server does: it's remote, hosted and managed by AWS, and it gives the agent authenticated access to AWS services through a small, fixed set of tools. No local install, automatic updates, and (this part I really like) every call lands in CloudTrail. The agent doesn't get a master key. It authenticates as you, through a real auth flow, on an IAM leash.

Since I first wrote this, the server picked up two more capabilities: running Python scripts in a sandboxed environment and following skills curated by AWS itself (the aws___retrieve_skill tool shows up next to aws___search_documentation). Both also go through your IAM credentials, just like the API calls. And it pairs well with the aws-core plugin I show further down, which also brings the AWS agent skills.

The auth flow in plain English: there are two paths now, and the newer one is the simpler one. Today the server speaks OAuth directly. You add the URL to your client, the first tool call opens the browser on AWS Sign-in, you log in with your usual identity, done. The token lasts 1 hour and refreshes itself for up to 12. No proxy, nothing to install.

The second path is SigV4 with the MCP Proxy for AWS (package mcp-proxy-for-aws-cli), an open source proxy that runs on your machine, takes your local AWS CLI credentials, and signs every call. It still exists and it has its moment: multiple accounts in the same session, read-only mode (hiding write-capable tools from the agent), a fixed default region, an org that blocks the OAuth permissions (signin:AuthorizeOAuth2Access and signin:CreateOAuth2Token), or a client that can't do OAuth but can run a local proxy. The setup guide labels OAuth as "simple" (web clients, a single account, getting started) and SigV4 as "advanced", pointing it at terminal and IDE coding agents like Claude Code, Kiro and Codex. Both work in those clients; I break SigV4 down in its own section below.

Either way the outcome is the same: you don't paste a key anywhere, the agent acts with your identity, and everything respects your IAM. Documentation search, by the way, needs no credentials at all.

The OAuth flow, step by step:

You attach the AWSMCPSignInOAuthAccessPolicy managed policy to your role or user (once).
You add the server URL to your client and fire the first call.
The browser opens on AWS Sign-in, you authorize, and the client keeps the token (1 hour, auto-refresh up to 12).
The server applies the context keys and forwards to the AWS service.
IAM authorizes via your policy and responds.
The whole call is logged to CloudTrail.

The "ohhh, got it" moment is this: ask "why did checkout-prod start throwing 500s after 2pm?" and watch the agent pull the real CloudWatch logs, cross-reference a recent deploy, and point at the actual resource. All inside what IAM allows, all auditable, with no key in any dotfile. And it works with what you already use: Claude Code, Kiro, Cursor, any MCP-compatible client.

Can you self-host the AWS MCP Server, or is it cloud-only?

It's cloud-only. AWS hosts and runs the AWS MCP Server, and your client connects to a regional endpoint over HTTPS. On the SigV4 path, the only thing running on your machine is the MCP Proxy for AWS, which signs the requests and forwards them; the server itself stays remote. The Knowledge MCP Server works the same way: remote, managed, and it needs public internet access.

So what if everything has to run on your side? Then you take on the operation, one of two ways. The first is the open source servers in awslabs/mcp: the older AWS API MCP Server, for example, runs locally via uvx, pip or Docker. The catch is that its own page now marks it as superseded by the managed AWS MCP Server, and you go back to managing credentials on the machine that runs it. The second is building your own, as in the building vs managed section above.

And if your next question is "can I at least keep the traffic private?": as of this update, the official docs list no PrivateLink or VPC (Virtual Private Cloud) endpoint for the managed server. More on that in the security section.

How to connect: Claude Code, the Claude app, and Kiro

Now the practical part. The OAuth path has a single prerequisite: the identity you'll use needs the sign-in managed policy. Attach it once and forget it:

aws iam attach-role-policy \
  --role-name MyRole \
  --policy-arn arn:aws:iam::aws:policy/AWSMCPSignInOAuthAccessPolicy

(If you use an IAM user instead of a role, it's attach-user-policy with --user-name.)

In Claude Code

One line, and that's really it:

claude mcp add aws-mcp https://aws-mcp.us-east-1.api.aws/mcp --transport http

On the first tool call the browser opens, you log in, and the agent sees the account on an IAM leash.

There's also a second route: the aws-core plugin. On top of the AWS MCP Server pre-configured, it brings the AWS agent skills, which are ready-made instruction packs so the agent handles CDK, serverless, containers and billing tasks well. If you want the server plus that extra context in one go, install it from Anthropic's official marketplace, which Claude Code already ships with:

/plugin install aws-core@claude-plugins-official

Heads-up: this command runs inside Claude Code, in the terminal. The "Plugins" section in the Claude Desktop app is a different catalog, so don't go looking for aws-core there (in the app, the path is the connector in the next section). The steering equivalent here is a CLAUDE.md at the project root, with the same rules I show in the Kiro block below.

In the Claude app (Desktop and claude.ai)

Yes, it works in the app, no terminal needed. In Claude Desktop: Settings, Connectors, "Add custom connector", give it a name (I called mine aws-mcp) and paste the URL:

https://aws-mcp.us-east-1.api.aws/mcp?oauth=initialize

The ?oauth=initialize suffix tells the server to kick off the OAuth flow explicitly (Cursor, Kiro IDE, VS Code, Codex and Gemini CLI use the same trick, per the setup guide). Leave the OAuth Client ID and Secret fields empty, the flow handles that. On claude.ai web it's the same URL without the suffix, in the connector settings.

On the first call, the AWS authorization screen opens:

Two warnings here, because I hit them first so you don't have to:

It's not the Directory connector. If you search "AWS" in Claude's connector directory, an "AWS API MCP Server" shows up. That is NOT the new managed one: it's the older aws-api-mcp-server from awslabs (look at its tools, just suggest_aws_commands and call_aws), precisely one of the servers the official docs tell you to replace. The right path is the custom connector with the URL above. And don't run both, that's the exact tool-conflict scenario AWS warns about.

Don't authorize as root. The consent screen offers "Continue with Root or IAM user" and says access follows "your existing AWS permissions". Whatever identity authorizes that connection defines the agent's blast radius. Prefer IAM Identity Center or a federated role with temporary credentials, and use an IAM user only if federation isn't an option where you are. Always with the sign-in managed policy and least-privilege. Root driven by an agent is everything we don't want (I fell for this one too: logged in, worked on the first try, and it was root driving the whole account).

And a practical red flag, with a catch: if the connection worked without you attaching AWSMCPSignInOAuthAccessPolicy, don't jump straight to "I must be root or admin". Your identity may already carry signin:AuthorizeOAuth2Access and signin:CreateOAuth2Token through some other policy or permission set. Careful with the classic aws sts get-caller-identity too: it shows the identity of your local CLI, and that's not necessarily the one you picked in the browser during OAuth. To be sure, check the policies of the identity you actually signed in with, and the authorization event in CloudTrail.

In Kiro

In Kiro CLI (2.11 or later):

kiro-cli mcp add --name aws-mcp --url https://aws-mcp.us-east-1.api.aws/mcp

In Kiro IDE, add it as a remote server with the ?oauth=initialize URL, or via mcp.json (per project at <root>/.kiro/settings/mcp.json, global at ~/.kiro/settings/mcp.json; when both exist, the project one wins).

Config turns the server on. To make Kiro actually prefer these tools, create a steering file at .kiro/steering/aws-mcp.md:

---
inclusion: always
---

# Using AWS via MCP

- Before generating any AWS code or IaC (CDK, CloudFormation, SDK), validate
  service names, syntax and regional availability against the docs via the
  `aws-mcp` server (documentation search tool). Don't guess APIs or ARNs.
- To inspect real account resources (CloudWatch logs, S3 items, DynamoDB
  schema), use the same `aws-mcp` server. Never ask for or use an access key.
- Never run a destructive action (delete, scaling) without confirming first.

inclusion: always makes that rule part of every Kiro conversation in that project.

How does SigV4 authentication work with the AWS MCP Server?

SigV4 (Signature Version 4) is the protocol AWS uses to sign API requests. With the AWS MCP Server, the MCP Proxy for AWS runs locally, reads your AWS CLI credentials and signs each request to the MCP endpoint; the server then acts as that identity, and IAM decides what goes through. Pick it over OAuth when you need several accounts in one session, read-only mode, a default region, or when your org blocks the OAuth sign-in permissions.

If you landed on one of those cases, the setup asks a bit more of your local machine: AWS CLI 2.32+ signed in with aws login (credentials rotate on their own every 15 minutes, sessions up to 12 hours), uv installed, and the proxy in mcp.json:

{
  "mcpServers": {
    "aws-mcp": {
      "command": "uvx",
      "args": [
        "mcp-proxy-for-aws-cli==1.7.0",
        "https://aws-mcp.us-east-1.api.aws/mcp",
        "--metadata", "AWS_REGION=sa-east-1"
      ]
    }
  }
}

Notice the -cli in the name. mcp-proxy-for-aws-cli is the distribution meant to run through uvx, with its whole dependency tree locked to tested versions. mcp-proxy-for-aws, without the suffix, became the library you import into your own Python code (LangChain, Strands Agents and the like).

The official docs use mcp-proxy-for-aws-cli@latest, which always pulls the newest release. It's convenient, but you end up running whatever shipped yesterday without reviewing anything. I'd rather pin the version (your supply chain will thank you) and bump it on purpose, checking PyPI every now and then for a new release.

The endpoint is regional (eight regions today, still no São Paulo); you connect to one of them and operate on resources in whatever region you pass in AWS_REGION. Without AWS_REGION, operations default to us-east-1. Check the current list in the official docs, since this keeps changing.

For multiple accounts, the proxy takes more than one profile in --profile: the first one is the default and the others become available for switching per call. Multi-profile only exists on this path; the docs say outright that OAuth doesn't support it.

Does the AWS MCP Server have a read-only mode?

Yes, but only on the SigV4 path. Start the MCP Proxy for AWS with --read-only and it disables the tools that may require write permissions, based on the readOnlyHint annotation each tool carries. With OAuth there's no such switch in the docs; there, your read-only guardrail is IAM.

In the mcp.json from the previous section, it's one more argument:

{
  "mcpServers": {
    "aws-mcp": {
      "command": "uvx",
      "args": [
        "mcp-proxy-for-aws-cli==1.7.0",
        "https://aws-mcp.us-east-1.api.aws/mcp",
        "--read-only",
        "--metadata", "AWS_REGION=sa-east-1"
      ]
    }
  }
}

Two things before you relax. First, the docs don't publish the list of which tools stay visible in read-only mode, so after turning it on, run /tools or /mcp in your client and look at what's left. Second, read-only mode hides tools from the agent; the boundary that actually holds is still IAM. Pair it with a read-only identity and the aws:ViaAWSMCPService deny I show in the account section, and you get two layers instead of one.

If you're still on the older local AWS API MCP Server, its equivalent is the READ_OPERATIONS_ONLY=true environment variable, which only allows operations whose access level isn't "Write".

The security part you can't ignore

"Everything fails, all the time."

Werner Vogels, Amazon CTO

If there's one part not to skip, it's this one. Werner's right, and with an AI driving your account it's worth taking that literally: assume the agent will mess up sooner or later, and design for it. Giving it access to read and operate your account is as serious as it sounds, so it's worth doing carefully.

First thing: IAM is still the boss. The managed server works inside your permissions. Treat the agent like a new coworker. Or better: like that fresh intern who does everything to the letter, no questions asked, and who's so afraid of getting it wrong that they don't even stop to think. The type who, if you tell them to go find the spark stockroom to store some in a bottle, will actually go looking. Yeah, the agent is just like that too: give it too much access and it'll use it, even when it makes no sense. So, least-privilege, scoped only to what the task needs, nothing beyond that. AWS added standardized IAM context keys for these managed MCP servers, so you can write policy that knows "this call came through the MCP server" and restrict accordingly.

Second: there's no long-lived key anywhere. With OAuth, what exists is a 1-hour token with automatic refresh; on the SigV4 path, temporary AWS CLI credentials that rotate on their own every 15 minutes. So no secret sits around in your shell history, your repo, or your .env, which is exactly where we tend to leak them.

Third: the authenticated calls are auditable. CloudTrail records what the agent requested through the server and CloudWatch gives you the metrics. After an incident you can answer "what did the agent actually do?" with a straight face. If you can't answer that today about your current setup, that alone is reason enough to switch. Just don't confuse visibility with retention: Event History covers the recent stuff, and long-term evidence needs a properly configured trail or CloudTrail Lake, with retention and access control set on purpose (and that side can cost storage and ingestion).

And one limitation to check before production: as I write this, the official docs list no PrivateLink or VPC endpoint support for the managed server. What exists is an open feature request from the community, with nothing committed by AWS. If keeping this traffic inside the network boundary is a hard requirement for you, treat it as a blocker until the docs say otherwise.

My rule of thumb? One dedicated IAM role per agent purpose. If an agent gets compromised or goes sideways, the blast radius stops at that role. Don't reuse your admin identity. And don't hand it * on * "just to unblock the demo", because the demo becomes prod faster than you'd think.

Setting up the AWS account side (the essentials)

On the account side it's less than it seems, because the managed AWS MCP Server doesn't create IAM actions of its own. There's no mcp:Invoke to allow: it signs every call with SigV4 using your credentials and forwards it to the service, which authorizes against your usual policies. If your identity can't call logs:GetLogEvents, neither can the agent. Your current permissions are the boundary.

The quick-start is this: begin with a read-only identity (a dedicated role or an SSO, Single Sign-On, permission set) and use the new context keys, aws:ViaAWSMCPService and aws:CalledViaAWSMCP, to deny destructive actions when the call comes from the agent, even if your role could do them. That way the agent reads all it wants and the dangerous verbs are blocked just for it. Then let CloudTrail show what it actually used and tighten the policy around that.

A simple guardrail, just so you get the idea:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "BlockDestructiveActionsViaMCP",
    "Effect": "Deny",
    "Action": ["dynamodb:DeleteTable", "s3:DeleteBucket", "lambda:DeleteFunction"],
    "Resource": "*",
    "Condition": { "Bool": { "aws:ViaAWSMCPService": "true" } }
  }]
}

This setup deserves a post of its own, done as code. Later in the series I'll publish an "IAM for AI agents on AWS, with CDK" covering the full step by step (CLI, console and CDK), cdk-nag, org-wide SCP (Service Control Policy) and multi-account. I'll link it here once it's out.

Which one to use: the decision changed

When this post first went out, the answer was "depends on the pain". Three weeks later, AWS settled it: the official setup guide now recommends migrating to the AWS MCP Server and removing Knowledge (and the older AWS API MCP) from your configuration. The reason is practical: duplicate tools confuse the agent and hurt performance, and it's the docs themselves telling you to clean up.

Knowledge is still up: GA, credential-free, working. One legitimate case remains: you want ONLY doc lookups, with no AWS account and nothing to authorize (studying a service, say). For everything else, it's one server: the managed one, which already does credential-free doc search and operates the account when you authorize it. Less config, fewer conflicts, same IAM leash.

When NOT to reach for this

Because there's no silver bullet, and someone has to say it.

Don't drop this on a production account with no guardrails. Start in a sandbox and get your IAM boundaries right before the agent can touch anything that bills or deletes. In the Frugal Architect, Werner makes the point that cost is an architecture requirement, not something to find out at the end of the month, and with an agent firing off calls that goes double.

Don't grant broad permissions "for now". There's no "for now", trust me. Scope it from the first connection.

And don't leave anything irreversible on autopilot. Deletes, scaling actions, money movement: keep a human in the loop. The agent proposes, you approve.

Mind your region and compliance

A few things that matter if you operate outside us-east-1 (for me, that's sa-east-1, São Paulo).

First, separate two things: the region of the MCP endpoint and the region of the resources the agent operates. Today the endpoint lives in eight regions across the US, Europe and Asia Pacific, still without sa-east-1. You connect through one of them and the agent still works on your resources wherever they live. Connecting through us-east-1 doesn't mean your DynamoDB table has to move there.

Before pointing the agent anywhere, confirm the services you'll touch are available in your region. And here's the nice part: that's literally a question the MCP server itself answers for you (the doc search does it, no credentials). Check it instead of assuming.

On data residency and privacy laws (LGPD, Brazil's General Data Protection Law, here; GDPR, the EU's General Data Protection Regulation, and friends elsewhere), if the agent is going to touch resources with personal data, scope IAM so it can't read what it shouldn't, and use the CloudTrail trail as evidence of who accessed what. That evidence is a strong piece of your compliance argument, and only a piece: LGPD and GDPR still ask about legal basis, purpose limitation, access control and retention.

And for small teams: that "free" CloudTrail trail is gold when you don't have a dedicated security team. You get a record of everything the agent did without building anything.

To wrap up

AWS has a pile of MCP servers, but the confusion lives in two general-purpose ones, one per pain: knowledge (stop hallucinating) and action (operate the account safely).
The simple path is OAuth now: add the URL, log in through the browser, done. Works in Claude Code, the Claude app (Desktop and claude.ai), and Kiro.
The managed one gets you off .env: a short-lived token (OAuth) or temporary CLI credentials (SigV4), with least-privilege and everything logged to CloudTrail.
The official recommendation became one server: migrate to the AWS MCP Server and drop Knowledge from your config. It's still alive for credential-free doc lookups, but running both confuses the agent.
The managed server is cloud-only. Need it on your side? Build your own or run the open source ones, and own the operation. Need read-only? --read-only on the SigV4 proxy, with IAM still as the real boundary.

If you've been doing the .env hack, relax, everyone has. But the tools to stop are right here, they're managed, and they're auditable. There's no good excuse left to keep handing your account's keys to a robot that occasionally goes off the rails.

This post kicks off a series on MCP and agents on AWS. Next up I go deep on two tracks: setting up IAM as code (CDK) and OpenAI landing on Bedrock. Which one first? Tell me in the comments.

Liked it? Drop a like, tell me what you think in the comments, and share it with your crew to keep the community strong. Thanks a lot for reading this far. See you in the next one?

BUILD. SCALE. REPEAT. =D

Want to go deeper

AWS's open source MCP servers (awslabs/mcp), home of the Knowledge MCP Server
Agent Toolkit for AWS, where the managed AWS MCP Server lives
AWS MCP Server, the official page
Official AWS MCP Server setup (OAuth and SigV4)
MCP Proxy for AWS (flags, including --read-only and --profile)
AWS MCP Server endpoints and quotas
Guidance for deploying MCP servers on AWS
The AWS MCP Server is now GA (official announcement)

Prefer Portuguese? Read the PT version here.

If your agent is going to run code it didn't write, the next step is isolation: AWS Lambda MicroVMs: run untrusted code with VM-level isolation.

Originally published on willpeixoto.dev.

AWS Lambda MicroVMs: run untrusted code with VM-level isolation (no infra to manage)

will peixoto — Wed, 24 Jun 2026 12:44:14 +0000

AWS just shipped Lambda MicroVMs, a new serverless primitive that gives each user or session a VM-level isolated sandbox, with near-instant launch and state preserved for up to 8 hours, all on Firecracker. Here is what it is, when to reach for it instead of a plain Lambda Function, and how to architect on top of it.

🇧🇷 Leia em português.

Let me put you in a situation. You need to run a piece of code you did not write. Maybe it is the script your user pasted into your platform, maybe it is the snippet an AI agent just generated and wants to execute. And then comes the question that keeps anyone working with multi-tenant up at night: how do I run this without handing a stranger the keys to the house?

Until last week you had three paths, each with a catch. A VM gives you strong isolation but takes minutes to boot. A container starts in seconds but shares a kernel, so running untrusted code there takes a pile of hardening. And the Lambda Function was built for short request-response, not for a session that has to keep live state between one interaction and the next (externalizing it to DynamoDB stores the data, not the live runtime: the running process, the loaded packages, the memory). In the end you chose between performance and isolation. No way around it. Or there was.

Container, VM, or Lambda: the trade-off none of them solved alone

This pattern got common: AI coding assistants, interactive code environments, analytics, vulnerability scanners, game servers running player scripts. They all need the same thing: give each user their own environment to run code the team did not write, safely and without lag.

The knot is that real isolation and low latency pull in opposite directions. From a security angle you want a hard boundary between tenants (the Security pillar of the Well-Architected Framework: isolate what is not trusted). From an experience angle you want that environment up the instant the user shows up. Reconciling the two was the expensive work.

And there is a nice irony in this story. We spent years learning to build stateless apps, and now state is a requirement again.

The solution to the future was hiding in the past.

That is a line a friend dropped in a conversation, and it has not left my head since. Ever felt that way? Because I have. And it is roughly what Lambda MicroVMs does: it brings state back, without handing you the weight of a full VM.

What Lambda MicroVMs is

Lambda MicroVMs is a new primitive inside Lambda, built exactly for that gap. Each MicroVM gives a single user or session its own isolated environment that boots fast, keeps memory and disk for the whole session, and pauses to a low cost when the user steps away.

The magic comes from Firecracker, the same lightweight virtualization that already runs over 15 trillion Lambda invocations a month. This is not raw new tech, it is the mature foundation of Lambda itself, exposed in a new way.

The model is image-then-launch:

You build the image once (AWS runs your Dockerfile, initializes the app, and takes a snapshot of memory and disk). After that, every MicroVM you launch resumes from that snapshot instead of cold-booting. That is why launch and resume are near-instant, even for a multi-gigabyte session.

What it is actually for (with examples you will recognize)

The main cue: this only enters the picture if you are building a platform that runs third-party code. If your app does not execute outside code, you do not need it. It is a building block for people who build that kind of product:

Replit, CodeSandbox, "VS Code in the browser": the user types code in the browser and it runs isolated, per user, holding state while the tab is open. That "runs isolated" is the MicroVM.
Code interpreter (like ChatGPT's or Claude's): you ask "plot this CSV", the AI writes Python and runs it to answer you. The runtime that executes that generated code, isolated per conversation, is the use case.
CI/CD runner (and relatives): a job runs the code of a Pull Request that may come from any stranger's fork, untrusted by definition, so you want an isolated, disposable runner per job. Same family: a scanner that runs a suspicious binary, a coding-interview platform (the candidate's code runs isolated), an AI agent that runs shell commands.

The thread tying it all together: each user, session, or job needs its own isolated environment, and the code running there is not code you wrote. That is the cue to use a MicroVM instead of a Lambda Function.

Lambda Function or Lambda MicroVM?

They do not compete, they complete each other. The official comparison:

	Lambda Functions	Lambda MicroVMs
Best for	request-response or event-driven (APIs, data processing, automation)	persistent environments running user or AI-produced untrusted code
Programming model	function handler invoked in a supported runtime	any application: run your own binaries, listen on ports, use Linux OS capabilities
Duration	up to 15 min per invocation; multi-step workflows up to a year with Lambda Durable Functions	up to 8 hours per session; suspend and resume across sessions
Runtime	service-provided runtimes (or customer-provided)	customer-provided MicroVM images
Inbound networking	direct invocations or event-source integrations; response streaming	inbound access to any port using OSI Layer 7 protocols
Concurrency	one request per execution environment at a time	multiple concurrent connections per MicroVM
Environment state	warm starts may reuse the environment, but state may not persist across invocations	memory and disk state preserved on suspend, restored on resume
Scaling	automatic: Lambda creates and destroys environments in response to traffic	developer-controlled: you create, suspend, resume, and terminate via API
Lifecycle	fully managed by Lambda	developer-controlled, with optional idle policies
Pricing	per-request + GB-seconds	per-second of compute while running + snapshot storage while suspended

The most common confusion: people assume the duration is the same as Lambda's. The startup is similar (both resume from a snapshot), but a Function dies at 15 minutes while a MicroVM holds a session for up to 8 hours with state intact. The real design: your app keeps Lambda Functions for the event-driven backbone, and calls MicroVMs only for the steps that need to run untrusted code in isolation.

How it works in practice: from endpoint to orchestration

Three things that trip people up at first, together.

The endpoint has a status. When you call run-microvm, you get an ID and a dedicated HTTPS endpoint for that MicroVM. But it is not ready instantly: it goes through states, from launch to RUNNING (about 2 seconds), and when idle it moves to suspended, coming back on resume. The endpoint is per MicroVM, per session.

One image, many MicroVMs. You build the image once (create-microvm-image) and each MicroVM is a run-microvm. Want two? Call it twice, and you get two independent instances. Idle behavior is governed by the idle-policy: maxIdleDurationSeconds (suspend after X idle) and autoResumeEnabled (the next request wakes the MicroVM on its own, in about 1s, no manual restart). When you are done, terminate-microvm releases everything.

You become the orchestrator. Since the endpoint is per session, something has to decide when to launch and where to route. Typically a Lambda Function in the backbone does it: it keeps a session -> MicroVM map (a store like DynamoDB in production), calls RunMicrovm on a user's first access, stores the ID and endpoint, mints a short-lived token with CreateMicrovmAuthToken, and proxies the request to the MicroVM's endpoint with the X-aws-proxy-auth header. If the instance is suspended and autoResume is on, the request itself wakes it. Add a routine to terminate orphan MicroVMs and you have the skeleton. The backbone code is in the next post in the series. And do not confuse this with Step Functions: MicroVM is the execution environment, Step Functions is an orchestrator, different layers.

Cost, limits, and what is still missing

Cost is a decision, not a detail. Werner Vogels keeps hammering in the Frugal Architect that cost is an architecture requirement, not a number you discover on the bill. The suspend is exactly that in practice: you pay a lot for VM-level isolation, but only while the user is active. When they leave, the MicroVM suspends and the cost drops, with no loss of state. Designing your idle-policy on purpose is a cost decision. The model, from the official table: you pay per second of compute while it runs, and only snapshot storage while it is suspended. Unit prices are on the Lambda pricing page.

Limits: ARM64, up to 16 vCPUs, 32 GB of memory, and 32 GB of disk per MicroVM, and up to 8 hours of total runtime. Provisioning is flexible: you set a baseline and burst up to 4x at peak, paying the baseline while it runs.

IaC: you can use the console, CloudFormation, and CDK.

Why Dockerfile + zip, and not a prebuilt ECR image? Aidan Steele dug into it: Lambda builds two copies of the image, one for Graviton 3 and one for Graviton 4, so it needs the source to recompile. The base comes from ECR Public, but pushing your own prebuilt image from a private ECR as the artifact is not the path. One thing that confuses people coming from containers: ECR does not leave your life. You do not deliver the MicroVM image via ECR, but inside the running MicroVM you can run Docker and docker pull your private ECR images at runtime. ECR is for consumption inside, not for delivering the image itself.

Networking and region: inbound traffic on configurable ports (HTTP/2, gRPC, WebSockets), service-provided JWE auth, outbound to the internet or your VPC. And it is available so far only in US East (N. Virginia, Ohio), US West (Oregon), Europe (Ireland), and Asia Pacific (Tokyo).

When NOT to use it

If the workload is short request-response with no state, it stays a Lambda Function. A MicroVM there is a cannon for a mosquito. And if you just need more than 15 minutes with your own (trusted) code, a MicroVM is also overkill: for a long job, look at Fargate; for a multi-step workflow, Lambda Durable Functions (up to a year, as the table shows). MicroVMs are for when the differentiator is isolating untrusted code, not just going past 15 minutes.

There is also a gotcha AWS itself flags, and it rhymes with the determinism conversation: since the MicroVM boots from a pre-initialized snapshot (the equivalent of Lambda SnapStart, as Aidan Steele confirmed by testing), apps that generate unique content, open connections, or load ephemeral data at init may diverge. The snapshot froze a moment; whatever needs to be fresh per session cannot be frozen along with it. The fix has a name: lifecycle hooks to re-initialize randomness when each MicroVM is created. Map that out before assuming it just works.

Does it kill the container?

No, and the reason is even better.

The hype of the week is "containers are obsolete." They are not. Quite the opposite: Aidan Steele tested it and you can run Docker inside a MicroVM, with OS capabilities enabled. So the MicroVM does not kill the container, it is more isolated and still runs containers inside. The honest cut is different: there is one specific spot, running untrusted code in isolation, where you will no longer want to harden a container by hand. There the MicroVM wins. Everywhere else, the container is still king.

The details the docs leave out

Aidan Steele spent launch day poking at the service and found some really interesting things that are not in the official docs.
I read it and figured it was worth bringing here:

You can get a shell into the MicroVM, via the CreateMicrovmShellAuthToken API, with pty as a first-class citizen (Lambda Functions do not have it). Gold for IDE and coding-agent use cases.
Outbound UDP is blocked by default and DNS is a local stub, so DNS inside a container falls back to 8.8.8.8 and fails. The fix is to run with Lambda's DNS: docker run --dns 169.254.169.253, or go via VPC.
Lambda network connectors: a reified VPC config (subnets, security groups, an IAM role for the ENI) with its own lifecycle. The network team creates it, the developer just consumes it.
Performance (his tests): image build 2-3 min; RunMicrovm to RUNNING about 2s, plus 2s to serve; suspend and resume about 1s each.

What you take away

Lambda MicroVMs fills a real gap: VM-level isolation with near-instant launch and per-session state, which no single service delivered together.
It does not replace the Lambda Function, it complements it. Function in the backbone, MicroVM for the untrusted code.
The idle suspend is a deliberate cost lever, design your idle-policy on purpose.
Before locking in architecture: check the region (no São Paulo yet), the limits (ARM64, 16 vCPU, 32 GB, 8h), and the snapshot caveat.

This post was the map. In the next one in the series I actually spin up a MicroVM and we prove the isolation in practice, launching two MicroVMs and testing whether one can reach the other, with the repo on GitHub for you to run along.

Got a case where you run user or AI code that today is duct-taped onto a container or a hand-rolled VM? Does this primitive fit? Drop a like, share it with whoever is building a multi-tenant platform, and let's talk. Cheers! =D

Originally published on willpeixoto.dev.

High Availability Has a Price: Resilience Is a Decision, Not a Stack

will peixoto — Thu, 23 Oct 2025 16:26:18 +0000

The cost of high availability: resilience starts with strategic decisions, not technology.

🇧🇷 Read in Portuguese

After a major outage like the October 2025 event in AWS's us-east-1 region, the smoke clears and the same questions that haunt CTOs and architects always resurface:

"Should we be multi-region?" Or worse (and a little nostalgic): "Should we go back to on-premises?"

The truth is that the right answer is rarely technical.

It's strategic, first and foremost.

As Werner Vogels (Amazon CTO) likes to put it in his talks:

"Everything fails, all the time."

And that's exactly it. The central question isn't whether it will fail, it's when it will fail and how prepared you'll be when that inevitable moment arrives. Because it will arrive. Whether you're in the cloud, on-premises, or running a complex multi-cloud setup.

What really separates resilient teams isn't the absence of failure. It's the speed, clarity, and effectiveness with which they respond and recover.

And that's where real architectural maturity lives: resilience isn't about choosing "multi-region" or "on-premises." It's about understanding the inherent risk, documenting the choice transparently, and reacting with a plan.

The Context Behind the Question: The Paradox of Visible Failure

Every time there's a big outage, I notice technical teams and executives splitting into two extreme reactions, both driven by fear and pressure:

"We need to go multi-region, now! Cost is secondary!"
"See? The cloud isn't reliable. We should have stayed on-premises, where we had control!"

Both extremes are dangerous shortcuts.

Multi-region is not a vaccine against downtime, and going back to on-premises is not a synonym for control; it just moves the maintenance complexity onto you.

A Crucial Point of Reflection: The cloud doesn't fail more than a traditional data center, it just fails in a way that is more visible, shared, and, ironically, democratic. On AWS, problems scale globally and become trending topics in minutes. On-premises, they hide behind scattered logs, long repair times, and, often, they only hit you. Honestly: do you believe your company has a greater capacity than AWS (or any major cloud provider) to manage physical security, cabling, power, cooling, and, above all, the resilience of infrastructure at global scale?

Migrating or evolving an architecture, at its core, is not about "throwing everything away" or "buying the hype." It's about keeping what's good in the legacy and removing what limits growth.

This isn't a black-and-white fight of "Cloud vs. Data Center." It's a strategic game of Conscious Resilience vs. Comfort Zone.

Cost vs. Continuity: The Economics Behind the 9s

In the world of infrastructure, each additional "9" in your SLA (Service Level Agreement) doesn't just cost more. It costs exponentially more.

To illustrate the real impact of each availability tier, here's the maximum allowed downtime per year:

99% (two 9s): about 3.65 days of downtime per year. Cost and complexity: baseline (1x).
99.9% (three 9s): about 8 hours and 46 minutes of downtime per year. Cost and complexity: 1.5x to 2x the baseline.
99.99% (four 9s): about 52 minutes of downtime per year. Cost and complexity: 2x to 3x. Requires Multi-AZ and strong automation.
99.999% (five 9s): about 5 minutes of downtime per year. Cost and complexity: 3x and up. Requires flawless automation and, often, a Multi-Region architecture.

Each tier jump means more than doubling or tripling infrastructure; it also demands operational review and sophistication. And here's the catch: every additional 9 has to be justified by ROI (Return on Investment), never by technical pride.

📢 The Non-Negotiable Factor: Regulation. For sectors like finance, healthcare, or telecom, the SLA choice isn't always purely economic. Often, the availability requirement (and the data recovery capability, the RPO) is imposed by law or industry rules. In those cases, the debate isn't whether you can afford it, but how to hit the legally mandated SLA at the lowest possible cost and complexity, because the cost of a regulatory fine outweighs any technical saving.

Rule of Thumb for Complexity:

High availability (within a single region): can cost 1.5x to 2x the baseline.
Multi-Region (Active/Passive): can cost 2.5x to 3x.
Multi-Cloud (Active/Active): almost never reduces risk. On the contrary, it usually increases the failure surface and operational complexity.

Conscious Decisions: The Virtue of ADRs

Every architectural choice is a commitment based on a context, and that context is volatile. Without a record, the context is lost, which condemns us to redo decisions, revisit old discussions, and rack up unnecessary cost.

That's where the practice of ADRs (Architecture Decision Records) becomes crucial. These aren't 50-page documents. They're short records that capture the decision, the reason, and the accepted risk at a specific point in time.

Example ADR (focused on the accepted risk):

# ADR-014: Do not use multi-region replication in the MVP

Context:
- Current traffic < 10 req/s.
- Multi-region replication cost is estimated at > 3x current cost.

Decision:
Keep a single-region architecture (using Multi-AZ for intra-region HA),
with a daily cross-region backup.

Review Trigger:
After reaching an average of 100 req/s, or when the current SLA (99.95%)
starts causing business impact.

Accepted Risk / Consequence:
Risk of total service downtime if an outage affects the entire region
(estimated RTO of 4 hours for cross-region recovery).

An ADR doesn't prevent failure. But it prevents failure from catching the team by surprise, because the risk was mapped, accepted, and justified by the business. It's the map for future discussions.

Selective Resilience: Not Everything Needs HA (and That's Fine)

Selective resilience is a virtue of economy and clarity. Not every service needs global redundancy. Spending finite resources (money and engineering attention) on unnecessary redundancy is one of the biggest forms of waste in architecture.

Prioritize High Availability (HA) only for what truly matters:

Direct revenue functions: components critical to the financial transaction (e.g., checkout and payment APIs).
The critical customer journey: functions that block the core value of the product (e.g., login or the main catalog).
Regulatory and legal risk: services where failure triggers legal fines or breaches a penalizing contractual SLA.
Integrity of critical data: where data loss violates an acceptable RPO (e.g., mandatory data retention systems).

Everything else? It can be restored through a well-defined recovery playbook. Batch jobs, internal back-office systems, and dashboards can tolerate minutes (or even hours) of downtime, as long as the reprocessing plan is clear.

High availability with no purpose is like installing an airbag on a bicycle. It's a sophisticated solution to a problem that doesn't exist in that context.

Managed != Failure-Proof: The Serverless Mindset

A common mistake is believing that using serverless services (Lambda, DynamoDB, SQS, EventBridge) is a synonym for immunity to failure. It isn't.

Failure will come, and often from where you least expect it, because the serverless paradigm shifts the risk surface, it doesn't remove it.

The key point is this:

Managed services reduce your operational surface (you don't manage the OS, patching, or capacity), but they don't replace good design and preparation.

During the October 2025 us-east-1 outage, plenty of 100% serverless applications went down. Not because serverless failed them, but because they leaned on a single region. When DNS resolution for the regional DynamoDB endpoint broke, anything pinned to us-east-1 (directly, or indirectly through a global control plane like IAM or STS) broke with it. Multi-AZ wouldn't have saved you here: the endpoint was regional, not zonal. And the applications that recovered slowest were frequently the ones whose code answered the failure with aggressive, unbounded retries, turning one outage into a self-inflicted retry storm.

Real resilience doesn't come from AWS. It comes from the architecture you design on top of it.

The Decision Belongs to the Business, the Clarity to the Architect

The difference between "having an opinion" and "having influence" lies in your ability to translate technical complexity into strategic clarity. Your job isn't to scare the board with jargon. It's to give them the visibility they need to decide consciously.

Experience has taught me that a team's maturity can be measured precisely by its ability to ask the right question:

❓ Where Is Your Team's Maturity?

Immature teams focus on the tool:

They ask: "Which stack solves this?"
They ask: "Should we use K8s or Serverless?"
They ask: "What does Netflix do?"

Mature teams focus on risk and the business:

They ask: "What risk are we willing to accept for this cost?"
They ask: "What RTO/RPO does the end customer require from this service?"
They ask: "What does our business need to survive a disaster?"

The result is that two teams can use the exact same cloud: one scales predictably, the other lives in panic mode. The difference isn't the cloud. It's the level of understanding, documentation, and technical humility behind the decisions made.

The Common Trap: Who hasn't heard an executive say, "Technical decisions are up to the Architecture team"? What they're actually doing is transferring responsibility for defining business risk. Your team defines the HOW (the stack), but the Business defines the HOW MUCH (the acceptable RTO and RPO). It's your job to hand the question back, so the risk decision belongs to the business.

Translating Resilience Concepts for Leadership

(After all, who hasn't heard: "Now translate that so I can understand it!")

1. Multi-Region Failover

   Translation: Insurance against catastrophe. It guarantees that a
                regional disaster won't take us offline for days,
                reducing revenue loss to a few hours.

   Question:    How many hours (or minutes) of downtime can the
                business accept for service X if an entire region goes down?
------------------------------------------------------------------
2. Active-Active Setup

   Translation: Maximum, uninterrupted availability. It lets us perform
                any maintenance or update without ever impacting the
                end customer.

   Question:    Does service X need to be 100% continuous? Can we afford
                a 15-minute maintenance window?
------------------------------------------------------------------
3. RTO / RPO

   Translation: Defining the limit of the damage. These are the numbers
                that tell us what we can lose, and for how long, before
                fines or reputation become unsustainable.

   Question:    How much data (RPO) can we lose, and how long (RTO) does
                the team have to restore the service before the business breaks?
------------------------------------------------------------------
4. SPOF (Single Point of Failure)

   Translation: The Achilles' heel of revenue. It's the weak point that,
                if broken, paralyzes the whole company. This is where the
                risk must be zero.

   Question:    If this component goes down, what's the financial loss
                in 1 hour?

Conclusion

There is no such thing as a failure-proof architecture.

But there is such a thing as an organization that is surprise-proof.

And it starts with conscious decisions, documentation (the ADRs), and the technical humility to accept that error and risk are part of the equation.

Teams that understand the "why" before diving into the "how" build systems that don't just scale. They survive, and grow predictably.

Essential References

For anyone who wants to go deeper on risk decisions and architectural patterns, these are the documents we use as a foundation for resilience on any cloud (with a focus on AWS):

AWS Well-Architected Framework (Reliability Pillar): the fundamental guide to understanding disaster recovery (DR) and high availability (HA) principles. Reliability Pillar
Disaster Recovery of Workloads on AWS: the key document for going deeper on RTO/RPO and choosing between patterns like Pilot Light and Active-Active. DR Whitepaper
DynamoDB Global Tables: an excellent practical case study of HA at the data layer, abstracting away multi-region complexity. DynamoDB Global Tables
EventBridge Resilience Guide: essential for anyone working with serverless, focused on event-based resilience patterns. EventBridge Resilience Guide
AWS Post-Event Summary: Amazon DynamoDB Service Disruption in US-EAST-1 (Oct 19-20, 2025): the primary source on the outage referenced in this article, straight from AWS. Service Disruption Summary

Essential Resilience Glossary

So everyone is on the same page, here are some key terms used in this article, explained simply:

High Availability (HA): the ability of a system to keep operating even when one or more of its components fail. Measured in "9s" (e.g., 99.99%).
Outage: an unplanned interruption of a service: the service goes down.
On-premises: infrastructure and data centers you own, physically located at the company (not in the cloud).
Multi-Region: using data centers in two or more different geographic cloud regions (e.g., US East and São Paulo) for maximum protection against regional disasters.
Multi-AZ (Multi-Availability Zone): using two or more Availability Zones (isolated, nearby data centers) within the same cloud region. This is the baseline HA pattern.
SLA (Service Level Agreement): a formal agreement defining the level of service a provider is expected to deliver (usually measured in uptime).
ROI (Return on Investment): a financial metric measuring the relationship between money earned (or saved) and money invested.
ADR (Architecture Decision Record): a short document recording an architectural decision, the reasoning, and the accepted risk at a specific point in time.
RTO (Recovery Time Objective): the maximum acceptable time a system can be down after a failure.
RPO (Recovery Point Objective): the amount of data (measured in time, e.g., 5 minutes) that can be lost during a disaster event.
Serverless: a cloud computing model where the provider manages all the infrastructure and the developer focuses only on the code, paying only for usage.
Circuit Breaker: a software pattern that, when a dependency starts failing repeatedly, "opens" the circuit to protect the rest of the application from cascading failures.

Going deeper

Three posts on the blog that carry this conversation into different corners:

Is multi-cloud more resilient? The math says no: what happens to your availability math once you put two clouds in series.
Data Streaming on AWS: Kinesis, Firehose, Flink, or MSK?: where RPO stops being a number on a slide and becomes a service choice.
AWS Lambda MicroVMs: run untrusted code with VM-level isolation: when isolation moves from implementation detail to architectural requirement.

And I would like to hear from you: how did your architecture hold up during the last regional outage, and which trade-offs had you already written down before it happened? Tell me in the comments, or find me on LinkedIn and let's compare notes.

BUILD. SCALE. REPEAT. =D