DEV Community: aws

Desert Hours: Mapping When Transit Disappears With Amazon Location Service

xbill — Sat, 31 Oct 2026 19:05:23 +0000

This article provides a step by step guide to building an hourly transit reach map with Amazon Location Service. Walking areas come from Pedestrian isolines, journey times come from Transit routing, and the finished map runs as static files on Amazon S3 and Amazon CloudFront.

Repository: https://github.com/xbill9/desert-hours

Live map: https://d35zkoavyfqlat.cloudfront.net/

Demo video: https://youtu.be/jfC_d6nq2EM

What Is a Transit Desert?

A transit desert is a place where demand for transit outruns the service on offer. Published maps usually score that gap once per area. Service changes by the hour, so this project scores it for every hour of a weekday.

The study area is Mercer County, New Jersey. It holds Trenton, Princeton and a ring of suburbs, and 16,114 of its 144,114 households have no car.

What You Can Explore

Drag the clock from 5 AM to 4 AM and the blue walkable service area shrinks across the county. The running total shows how many car-free households are out of reach at that hour.

Click a census tract for its day: car-free households, the share of housing that can reach service, departures by hour, and transit times to three destinations at 8 AM, 5 PM, 11 PM and 2 AM beside the driving time.

At This Point You Should Have

An AWS account with access to Amazon Location Service
The AWS CLI, version 2
Python 3 with pyshp and shapely
A GTFS feed for your area, a car ownership table and Census block boundaries

Step 1 — Gather the Data

Source	What it holds	Use
NJ TRANSIT GTFS	Scheduled bus and rail departures	Departures per stop, per hour
Census ACS table B08201	Households by vehicles available	Car-free households per tract
2020 Census blocks	Housing units per block	Where households sit in a tract
Mercer County route layer	Route 130 Connection stops	A county service outside the feed

Every published figure is a county, municipal or tract total, so the map holds no personal data.

Step 2 — Count Departures by Hour

pipeline/build_data.py picks one Wednesday in the feed and counts departures per stop in each hour. It keeps stops inside the county and within 1.5 km of it, because people cross county lines to catch a bus. That leaves 1,012 stops.

Step 3 — Ask Amazon Location for a 10-Minute Walk

Each stop gets a Pedestrian isoline along the street network.

aws geo-routes calculate-isolines --region us-east-1 \
  --origin -74.7548 40.2186 --travel-mode Pedestrian \
  --thresholds '{"Time":[600]}' --isoline-geometry-format Simple

The shape depends on the streets around the stop. Both of these reach about 560 m at their farthest point:

Location	10-minute walk area	Farthest point
Trenton station	0.33 km²	562 m
Hamilton suburb	0.09 km²	560 m

The median across all 1,012 stops is 0.26 km², against 0.50 km² for a 400 m circle.

🔎 Tip: Place Households Where Housing Is

A tract includes parks, rail yards and farmland. Spreading its households evenly across that area counts people who live nowhere near those spaces.

Census blocks carry a housing unit count, so each tract's car-free households are shared across its blocks by housing units. A block counts as served in an hour when its interior point falls inside that hour's service area.

Step 4 — Build the Hourly Service Area

For each hour, the pipeline joins the walking areas of the stops with at least one departure in that hour. Testing every block against that shape gives the hourly total.

Code does the counting and the page quotes it:

Hour	Car-free households out of reach	Share
8 AM	6,388	39.6%
11 PM	8,129	50.4%
3 AM	14,095	87.5%

Step 5 — Ask for Real Journeys

CalculateIsolines covers walking and driving modes. CalculateRoutes adds a Transit mode with timetable departures.

aws geo-routes calculate-routes --region us-east-1 \
  --origin -74.7548 40.2186 --destination -74.6672 40.3495 \
  --travel-mode Transit --departure-time 2026-11-04T02:00:00-05:00

A request for 2 AM from Trenton to Princeton returns a timed journey:

Leg	Departs	Arrives
Walk to Trenton Transit Center	3:56 AM	4:11 AM
Train to Princeton Junction	4:11 AM	4:24 AM
Train to Princeton	4:40 AM	4:45 AM
Walk to the destination	4:45 AM	5:09 AM

The trip takes 3 hours 9 minutes from the moment you wanted to leave, and most of that is waiting for the first train. The pipeline stores that full wait.

Step 6 — Ship It as Static Files

The results are GeoJSON and HTML in a private S3 bucket behind CloudFront. Precomputing everything leaves the page with one Amazon Location key, limited to map actions and the site's domain.

aws cloudformation deploy --region us-east-1 \
  --stack-name desert-hours --template-file deploy/stack.yaml

The findings sit in the page's HTML, so they read without JavaScript. MapLibre draws the tracts and service areas over an Amazon Location base map.

One Tract, One Day

Trenton Tract 1 is a strip along the Delaware River with about 445 car-free households of 1,332, with a margin of ±227.

Measure	Result
Housing within a 10-minute walk of service, 8 AM	73%
Housing within a 10-minute walk of service, 11 PM	0%
Drive to Capital Health Regional Medical Center	11 minutes
Transit to the hospital, leaving at 8 AM	52 minutes
Transit to the hospital, leaving at 2 AM	3 hours 29 minutes

Compare and Contrast

Approach	Walking area	Time of day	Journey times
400 m circles	Same everywhere	Any	None
Static desert score	Varies by method	One score	None
Desert Hours	Street network	Every hour	Timetable by Amazon Location

What Did It Cost?

Item	Count	List price	Total
Isolines	1,012	$1.50 per thousand	$1.52
Journeys	1,260	$0.50 per thousand	$0.63

Totals are arithmetic from AWS Price List rates. Results are cached, so a rebuild makes no new calls.

So, Which One?

Use isolines when reach depends on the street network, which it does in any suburb. Use Transit routing for the stories a map shape cannot tell, such as how long a 2 AM trip takes. Keep both in a batch pipeline, and the live page never spends money on routing.

Summary

The goal of this article was to map how transit reach changes through a weekday. The key to the solution was Amazon Location Pedestrian isolines joined by hour and tested against Census blocks. The Mercer County results were:

🟢 6,388 car-free households are out of walking reach at 8 AM, 39.6% of the county's total
❌ 14,095 are out of reach at 3 AM, 87.5% of the total
🟢 Street-based walking areas separate a city grid from a suburb at the same distance
⚠️ Trenton Tract 1 falls from 73% of housing in reach at 8 AM to 0% at 11 PM

Scope: scheduled service for one weekday, which may be incomplete, inaccurate or out of date. TigerTransit, the Princeton municipal shuttle, the Z-Line and on-demand services are not included, so reach is understated where they run. Route 130 Connection stops use a September 2023 timetable and count in every hour with a departure. Car-free households are assumed to be spread like housing within a tract, and tract estimates carry wide margins. Journey times come from one point per tract. Schedule data comes from NJ TRANSIT's published GTFS through the Mobility Database archive, and this project is independent of NJ TRANSIT.

The strategy for using Amazon Location Service for an hourly transit map was validated with an incremental step by step approach.

References

AWS FinOps Field Notes (2026-10-31): Automating Orphaned EBS & EC2 Cleanup

Ace7-coder — Sat, 31 Oct 2026 18:12:10 +0000

While reviewing real-world cloud cost audits this week, the same infrastructure leak kept surfacing across multiple engineering teams: unattached EBS volumes and idle resources quietly inflating the monthly bill.

Where the Waste Hides

The Silent Accumulation: As Tim O'Brien recently highlighted, fifty abandoned 100 GB unattached EBS volumes cost roughly $500/month ($6,000/year). Clearing out forgotten storage isn't just a one-off cleanup—it requires continuous ownership tracking because unattached volumes persist indefinitely unless removed.
Orphaned Post-Termination Storage: Aman Maheshwari pointed out that in environments running hundreds of EC2 instances, attached EBS volumes often survive instance termination (if DeleteOnTermination was unset) and continue billing unnoticed.
The Extended Support Trap: Antonio Augusto Maguelniski shared a case where a client's AWS bill doubled without any new deployments—their RDS costs surged from $30 to $330/month simply because two PostgreSQL 12 instances slipped out of standard support into AWS Extended Support.

Auditing Unattached Volumes with Python & Boto3

Instead of manually clicking through region tabs in the AWS Console, you can query for detached volumes in available state across your infrastructure with a simple Boto3 inspection script:

import boto3

def audit_unattached_ebs(region='us-east-1'):
    ec2 = boto3.client('ec2', region_name=region)

    # Query volumes that are not attached to any running instance
    response = ec2.describe_volumes(
        Filters=[{'Name': 'status', 'Values': ['available']}]
    )

    orphaned_volumes = response.get('Volumes', [])
    print(f"[{region}] Found {len(orphaned_volumes)} unattached EBS volume(s):")

    total_wasted_gb = 0
    for vol in orphaned_volumes:
        vol_id = vol['VolumeId']
        size = vol['Size']
        vol_type = vol['VolumeType']
        total_wasted_gb += size
        print(f" - {vol_id} | {size} GiB ({vol_type}) | Created: {vol['CreateTime'].strftime('%Y-%m-%d')}")

    print(f"Total unattached storage: {total_wasted_gb} GiB")

if __name__ == '__main__':
    audit_unattached_ebs('us-east-1')

Continuous FinOps Automation
Manually deleting disks risks wiping data that still needs an owner review, while leaving them alone guarantees continuous waste. Automated governance requires regular multi-account audits and programmatic reporting before executing snapshots or deletions.

👉 Grab the free open-source CLI on GitHub: aws-finops-auditor-lite
👉 Need multi-account AWS Organization (sts:AssumeRole) scanning and automated CI/CD schedules? Get the Enterprise AWS FinOps Auditor ($99 one-time)

Building `aws-ec2-check`: A Small Open-Source AWS Utility with Node.js

Sahinur — Sat, 31 Oct 2026 18:00:00 +0000

As I started working more deeply with AWS, I noticed a small but recurring workflow:

When debugging an EC2-based application, I often need to quickly answer a few basic questions:

Is the instance running?
What type of instance is it?
Which region and Availability Zone is it in?
What are its public and private IPs?
Are the EC2 status checks healthy?

The AWS Console provides all of this information, but sometimes I just want a quick answer from my terminal.

So I built aws-ec2-check, a small open-source CLI for inspecting AWS EC2 instances directly from the command line.

The project is now publicly available on GitHub:

GitHub: https://github.com/devSahinur/aws-ec2-check

The repository is written in TypeScript and is published under the MIT License.

The problem

When working with EC2-based applications, a lot of troubleshooting starts with basic infrastructure information.

For example:

Is the instance running?

Is the instance healthy?

Which Availability Zone is it using?

What's the public IP?

What's the instance type?

Are the EC2 status checks passing?

I could open the AWS Console and look these up.

But when I'm already working in the terminal, switching between tools feels unnecessary for such a small task.

That was the motivation behind the project.

What `aws-ec2-check` does

The idea is simple:

Terminal
   ↓
aws-ec2-check
   ↓
AWS SDK for JavaScript
   ↓
Amazon EC2

For a specific instance, the CLI is designed to provide useful EC2 information such as instance state, metadata, and health/status information.

A typical command looks like:

aws-ec2-check i-0123456789abcdef0

The project also supports a CLI-first workflow rather than requiring users to interact with the AWS Console.

Why build another AWS tool?

There is already an official AWS CLI.

So I'm not trying to replace it.

The goal of this project is much smaller:

Make one common EC2 inspection workflow convenient from the terminal.

That distinction is important.

I don't think every open-source project needs to become a massive platform.

Sometimes a small developer utility is enough to solve a real annoyance.

How it's built

I chose TypeScript because it fits naturally with my Node.js backend background while giving the project strong typing and a better development experience.

The repository is currently categorized around:

AWS
CLI
DevOps
EC2
Node.js
TypeScript

and is publicly available as devSahinur/aws-ec2-check.

At a high level, the architecture is:

Developer
    ↓
CLI
    ↓
TypeScript
    ↓
AWS SDK for JavaScript
    ↓
EC2 API

The CLI acts as a small abstraction over the EC2 APIs needed for inspection.

AWS SDK instead of the Console

One thing I wanted to learn from this project was what happens underneath the AWS Console.

When I use the Console, a lot of AWS operations feel like buttons and tables.

When building a CLI with the AWS SDK, those same workflows become explicit API calls.

That makes the architecture easier to reason about:

CLI command
    ↓
Application logic
    ↓
AWS SDK
    ↓
EC2 API
    ↓
Response
    ↓
Formatted terminal output

For me, this was a useful way to connect AWS infrastructure knowledge with backend development.

Security matters

Because this tool interacts with AWS, credential handling was one of the parts I paid close attention to.

I don't want AWS credentials embedded in the source code.

The tool should work with AWS's standard credential resolution mechanisms rather than inventing its own authentication approach.

That means developers can use their existing AWS configuration or appropriate AWS roles.

Just as importantly, the tool is intended for inspection, not unnecessary infrastructure modification.

That led to another important design principle:

A small tool should ask for small permissions.

There's no reason for an EC2 inspection tool to require administrator access.

IAM and least privilege

AWS permissions can become dangerous when developers solve every AccessDenied problem by attaching broad policies.

For this project, I wanted the permissions model to follow the same principle I discussed in my earlier IAM article:

Required action
      ↓
Required resource
      ↓
Minimum permission

Rather than:

Required action
      ↓
AdministratorAccess

This is a small CLI, but the security principle is the same as it is for a large production system.

Keeping the CLI simple

I wanted the command structure to stay easy to understand.

The core idea is:

aws-ec2-check <instance-id>

And as the project evolves, additional commands and options can support common workflows without making the CLI feel complicated.

For example:

aws-ec2-check --help

and:

aws-ec2-check --version

make the tool easier to discover and use.

Why TypeScript?

For a small project like this, TypeScript isn't required.

But I still prefer it because it gives me:

Better type safety
Better editor support
Easier refactoring
Clearer function contracts
Better maintainability as the project grows

It also keeps the project close to the technologies I already use in backend development.

Testing the project

Another thing I wanted to avoid was creating a CLI that only works on my machine.

A useful open-source project needs to be reproducible.

So the project is structured around the usual development workflow:

npm install
npm run build
npm test
npm run lint
npm run typecheck

The goal is to keep the project easy to validate locally and suitable for automated checks through GitHub.

What I learned building it

The interesting part of building this project wasn't only calling the EC2 API.

It made me think more carefully about the difference between:

"I can write the code."

and:

"I can publish something other developers can actually use."

That difference includes a lot of small details.

CLI design

A CLI needs clear commands and understandable output.

The user shouldn't have to understand the internal implementation just to get an EC2 status.

AWS authentication

AWS credential handling shouldn't be reinvented.

Using established AWS mechanisms is safer and easier to maintain.

Error handling

AWS errors can contain useful technical information, but exposing raw errors to users isn't always the best experience.

A good CLI should turn those failures into useful messages.

Documentation

Open source isn't just about code.

A developer should be able to discover:

What does this tool do?
How do I install it?
How do I authenticate?
What permissions are required?
What commands are available?
What should I do when something fails?

If the README can't answer those questions, the project isn't finished.

What I'd like to add next

The first version is intentionally focused.

But there are several areas that can make the tool more useful over time:

Instance filtering
Tag-based filtering
Multi-region support
JSON output improvements
More detailed health information
Additional EC2 inspection commands
Better automation support

The important part is adding features based on real developer needs rather than adding functionality just to make the project bigger.

Why open source?

One of my goals with this project was to move beyond simply learning AWS privately.

I wanted to follow a cycle like:

Learn
  ↓
Build
  ↓
Document
  ↓
Publish
  ↓
Get feedback
  ↓
Improve

That's one of the reasons I'm interested in building in public.

A project becomes more valuable when other developers can inspect it, use it, report issues, and contribute ideas.

Try it

The repository is available here:

https://github.com/devSahinur/aws-ec2-check

It's a public GitHub repository built with TypeScript and focused on AWS EC2 inspection.

You can explore the implementation, open an issue, suggest an improvement, or contribute to the project.

Final takeaway

I didn't want to build a huge AWS platform.

I wanted to solve one small problem:

Make EC2 inspection easier from the terminal.

That small problem turned into a useful learning project covering:

TypeScript + Node.js + AWS SDK + EC2 + IAM + CLI development + Open Source

For me, that's the real value of building small developer tools.

You don't always need a huge project to learn something meaningful.

Sometimes you just need to take one annoying workflow and make it a little better.

Build something small. Make it useful. Publish it. Then keep improving it.

Honeytokens in Practice: Catching Credential Theft When Malware Uses LLMs to Triage Your Secrets

EME GUG — Sat, 31 Oct 2026 17:13:15 +0000

Tuần này trên Hacker News có một bài khá đáng lo: malware mới dùng LLM để xếp hạng nạn nhân theo giá trị. Thay vì exfiltrate cả ổ cứng rồi để người ngồi lọc, con malware gom file .env, ~/.aws/credentials, lịch sử shell, config Kubernetes, rồi đưa cho một LLM đánh giá: máy này có đáng khai thác không, key nào là production, wallet nào có tiền. Tức là attacker giờ lọc chiến lợi phẩm nhanh hơn bạn kịp phát hiện mình bị dính.

Tin tốt là chính chuyện đó lại tạo ra một điểm yếu để mình tận dụng: càng tự động, attacker càng dễ ăn phải mồi. Một LLM thấy file prod-admin.env có AWS_ACCESS_KEY_ID thì gần như chắc chắn sẽ chấm điểm cao, và script phía sau sẽ thử dùng key đó. Đó là lúc honeytoken (hay canary token) phát huy tác dụng. Bài này mình chia sẻ cách tự dựng honeytoken cho máy dev và server, không phụ thuộc dịch vụ bên thứ ba.

Honeytoken là gì và vì sao nó hợp với thời LLM

Honeytoken là một credential, file hoặc URL giả nhưng trông thật, không ai trong team có lý do hợp lệ để chạm vào. Hễ nó bị dùng là có chuyện. Khác với IDS hay EDR phải đoán xem hành vi nào là xấu, honeytoken gần như không có false positive: không ai dùng AWS key tên backup-admin-legacy cả, nên nó mà được gọi thì chắc chắn có người đang lục máy bạn.

Với malware dùng LLM để triage, có ba điểm hay:

LLM được prompt để ưu tiên thứ trông có giá trị, nên bạn có thể thiết kế mồi đúng gu nó: tên file có prod, admin, root, comment kiểu # DO NOT COMMIT.
Pipeline tự động thường validate key ngay để chấm điểm, nên bạn biết sớm, nhiều khi chỉ vài phút sau khi bị lấy.
Chi phí gần như bằng 0: một IAM user không có quyền, một endpoint HTTP, vài dòng bash.

flowchart LR
    A[Malware trên máy dev] --> B[Gom .env, ~/.aws, kubeconfig]
    B --> C[LLM chấm điểm giá trị]
    C --> D{Key trông như production?}
    D -->|Có| E[Script thử gọi API bằng key]
    E --> F[CloudTrail / Webhook ghi nhận]
    F --> G[Alert Telegram/Slack]
    G --> H[Rotate key thật, cách ly máy]

Lớp 1: AWS key mồi với IAM user không có quyền gì

Đây là loại mồi hiệu quả nhất vì AWS key là thứ attacker thèm nhất. Ý tưởng: tạo một IAM user thật, gắn policy deny toàn bộ, lấy access key, rồi cài vào các chỗ attacker hay tìm.

# AWS CLI v2.x
USER=backup-admin-legacy

aws iam create-user --user-name $USER

# Deny tất cả, key có tồn tại nhưng không làm được gì
aws iam put-user-policy --user-name $USER \
  --policy-name deny-all \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Deny", "Action": "*", "Resource": "*"}]
  }'

aws iam create-access-key --user-name $USER \
  --query 'AccessKey.[AccessKeyId,SecretAccessKey]' --output text

Sau đó thêm vào ~/.aws/credentials một profile tên hấp dẫn:

[prod-admin]
aws_access_key_id = AKIA...
aws_secret_access_key = ...
region = ap-southeast-1

Để phát hiện, bạn cần một CloudTrail trail đẩy log về CloudWatch Logs, rồi tạo metric filter theo access key ID:

aws logs put-metric-filter \
  --log-group-name cloudtrail-logs \
  --filter-name honeytoken-prod-admin \
  --filter-pattern '{ $.userIdentity.accessKeyId = "AKIA_YOUR_CANARY_ID" }' \
  --metric-transformations \
    metricName=HoneytokenUsed,metricNamespace=Security,metricValue=1

Gắn một CloudWatch Alarm với ngưỡng >= 1 rồi đẩy qua SNS là xong. Có hai lưu ý thực tế. Thứ nhất, đừng đặt honeytoken trong account production, hãy dùng một account riêng cho security để không ai lỡ tay xoá. Thứ hai, attacker có kinh nghiệm biết vài cách kiểm tra key mà không để lại nhiều log. Vì vậy honeytoken là một lớp cảnh báo sớm, không phải lá chắn duy nhất.

Lớp 2: Canary URL trong file .env và tài liệu nội bộ

Không phải cái gì cũng là AWS key. Pipeline LLM thường đọc cả file .env, README nội bộ, note trong Obsidian. Mình nhét vào đó một URL độc nhất, bất kỳ request nào tới URL đó đều là tín hiệu. Đây là một webhook receiver tối giản viết bằng Python 3.12 + Flask 3.0:

# canary_server.py
import os, json, time, secrets
import requests
from flask import Flask, request, abort

app = Flask(__name__)
TG_TOKEN = os.environ['TG_TOKEN']
TG_CHAT = os.environ['TG_CHAT']

# Mỗi token gắn với một vị trí cài mồi, để biết máy nào bị lộ
TOKENS = json.load(open('tokens.json'))  # {'a1b2...': 'laptop-andy:.env.production'}

def alert(where: str):
    msg = (f'HONEYTOKEN TRIGGERED\n'
           f'Vị trí: {where}\n'
           f'IP: {request.headers.get("X-Forwarded-For", request.remote_addr)}\n'
           f'UA: {request.headers.get("User-Agent", "-")}\n'
           f'Time: {time.strftime("%Y-%m-%d %H:%M:%S")}')
    requests.post(f'https://api.telegram.org/bot{TG_TOKEN}/sendMessage',
                  json={'chat_id': TG_CHAT, 'text': msg}, timeout=5)

@app.route('/v1/<token>', methods=['GET', 'POST'])
def hit(token):
    where = TOKENS.get(token)
    if where:
        alert(where)
    # Trả 401 cho giống API thật, attacker tưởng key sai chứ không nghĩ là bẫy
    abort(401)

def new_token(location: str) -> str:
    t = secrets.token_hex(16)
    TOKENS[t] = location
    json.dump(TOKENS, open('tokens.json', 'w'), indent=2)
    return t

Rồi cài vào file mồi:

# .env.production (file mồi, KHÔNG phải file thật)
INTERNAL_BILLING_API=https://api-billing.yourdomain.dev/v1/a1b2c3d4e5f6...
INTERNAL_BILLING_KEY=sk_live_51Hx...fake

Mẹo nhỏ: đặt domain trông như internal API (api-billing, vault-internal), đừng dùng tên kiểu canary.yourdomain.dev. LLM đọc được chữ canary và có thể được prompt để bỏ qua. Bạn đang chơi trò tâm lý với một model, nên đặt tên mồi cho đúng kiểu thứ nó được dặn đi tìm.

Lớp 3: Theo dõi việc đọc file mồi trên chính máy

Hai lớp trên chỉ báo khi mồi được dùng. Muốn biết sớm hơn, lúc file vừa bị đọc, trên Linux dùng inotifywait (gói inotify-tools 3.22+):

#!/usr/bin/env bash
# watch_decoys.sh: chạy bằng systemd user service
DECOYS=(
  "$HOME/.aws/credentials.bak"
  "$HOME/projects/legacy/.env.production"
  "$HOME/.ssh/id_rsa_prod_old"
)

inotifywait -m -e open,access --format '%w %e' "${DECOYS[@]}" |
while read -r file event; do
  PROC=$(lsof -t "$file" 2>/dev/null | head -1)
  CMD=$(ps -o comm= -p "$PROC" 2>/dev/null)
  curl -s -X POST "https://api.telegram.org/bot${TG_TOKEN}/sendMessage" \
    -d chat_id="$TG_CHAT" \
    -d text="Decoy opened: $file ($event) by ${CMD:-unknown} on $(hostname)"
done

Trên server production, mình khuyên dùng auditd thay vì inotify vì nó ghi được PID, UID và executable đầy đủ hơn:

sudo auditctl -w /opt/app/.env.backup -p r -k honeytoken
sudo ausearch -k honeytoken --format text

Có một cái bẫy hay gặp: backup tool, Spotlight hay indexer của IDE cũng đọc file và gây false positive. Đặt file mồi ở thư mục bị exclude khỏi backup và indexer, hoặc lọc theo tên process trong script.

sequenceDiagram
    participant M as Malware
    participant F as File mồi
    participant W as inotify/auditd
    participant API as Canary API / AWS
    participant A as Alert
    M->>F: Đọc .env.production
    F-->>W: Event open/read
    W->>A: Cảnh báo sớm (giây)
    M->>API: Thử dùng key/URL
    API->>A: Cảnh báo xác nhận

Kết luận

Malware dùng LLM giúp attacker nhanh hơn, nhưng cũng làm chúng đoán trước được hơn: chúng ưu tiên đúng những thứ bạn có thể làm giả. Một số việc nên làm ngay trong tuần này:

Tạo 1 AWS honeytoken trong một account riêng, policy deny-all, gắn CloudWatch alarm. Mất khoảng 20 phút.
Cài mồi lên máy của từng dev, mỗi máy một token riêng để khi có alert là biết ngay máy nào bị lộ.
Đặt tên mồi theo gu của LLM: có prod, admin, legacy, backup, tuyệt đối không dùng chữ canary hay honey.
Thêm inotify/auditd cho file mồi để có cảnh báo từ lúc bị đọc, không phải đợi tới lúc bị dùng.
Viết sẵn runbook: có alert thì rotate key thật nào, cách ly máy ra sao, ai chịu trách nhiệm. Alert mà không ai biết phải làm gì thì cũng vô ích.

Honeytoken không thay được secret manager, MFA hay việc không để credential nằm plaintext trên laptop. Nhưng nó là lớp phòng thủ rẻ nhất mà mình biết, nó báo đúng tới mức bạn có thể tin từng cái alert, và nó biến chính sự tự động của attacker thành thứ tố cáo chúng.

Test Step Functions with the TestState API

Sascha Mirow-Stanitzok — Sat, 31 Oct 2026 16:50:03 +0000

For a long time, I treated the question of how to test Step Functions as something for “later”. You check the happy path anyway, because you see it while trying things out. The Retry and Catch blocks, however, are a piece of JSON that only really runs for the first time when something goes wrong in production. That is exactly the wrong place to find out that an error name has a typo.

With the TestState API, there is now a tool I would use in every Step Functions project. It lets you test a single state of your state machine in the cloud without deploying it. Since mocks are supported, you do not even need real Lambda functions or an IAM role. In this article, we build the whole thing with a small example: a bike rental with a retry for throttling, a catch for “no bike available” and tests with node:test in TypeScript.

In short

The TestState API runs exactly one state of your Step Functions definition and reports whether it succeeded, would be retried or ends up in a catch. With mocked results and errors, you can test retry and catch paths in a targeted way, without a deployment and without an IAM role. The biggest pitfall: the API only ever tests one state, so for a run through the whole machine with real permissions you also need a short test after deployment.

What you will learn in this article

What the TestState API can do and where its limits are
How to check retry, backoff and catch with mocked errors
How tests and the CDK stack share the same ASL file
Which guardrail tests run without any AWS access
Which typical mistakes happen when testing

Why teams rarely test Step Functions properly

AWS Step Functions is the orchestration service of AWS. You describe a workflow as a state machine in the Amazon States Language (ASL), a JSON format, and Step Functions runs the steps, waits, branches and retries. This is practical, because error handling lives declaratively in the definition instead of being scattered across code. However, that is also where the testing problem starts.

A Retry block is configuration. For example, it says: on Lambda.TooManyRequestsException, try three times, start with a two second pause and double the wait each time. Whether the error names are right, whether the order of the catchers fits and whether the error ends up in the right next state is something you will not notice during normal testing. After all, the error case simply does not happen.

For years, the answer was Step Functions Local, an emulator in a Docker container. AWS now states very clearly in the documentation that Step Functions Local does not provide feature parity and is unsupported. Instead, AWS explicitly recommends the TestState API. That matches my view: an emulator that lags behind the real service creates false confidence. A test against the real execution logic in the cloud is more honest.

What the TestState API does

The TestState API takes an ASL definition and runs exactly one state. You either pass only that single state or the complete state machine plus the name of the state in the stateName field. No state machine is created or changed. In response, you get the status, the output, the next state and, if you ask for it, details about data processing.

The four statuses at a glance

The status is the heart of your tests. It tells you what Step Functions would do with the result of the state:

Status	Meaning	Typical test
`SUCCEEDED`	The state completed, `nextState` names the following state.	Happy path, shaping data with `ResultSelector` and `ResultPath`
`RETRIABLE`	The error matches a retrier and attempts are left.	Throttling, short service errors
`CAUGHT_ERROR`	A catcher handled the error.	Business errors, exhausted retries
`FAILED`	Neither retry nor catch apply, the execution would fail.	Spotting missing error handling

With inspectionLevel, you control the level of detail. INFO is the default and only returns the result. With DEBUG, you also see the intermediate values after InputPath, Parameters, ResultSelector and ResultPath, and in case of an error the fields retryIndex, retryBackoffIntervalSeconds and catchIndex. TRACE adds the full HTTP request and response for HTTP tasks.

Mocks instead of real Lambda calls

The most important building block is the mock field. For Task, Map or Parallel states, it lets you provide either a result (mock.result) or an error (mock.errorOutput with error and cause). Both at once are not allowed. As soon as a mock is set, roleArn becomes optional according to the documentation. You then test pure workflow logic, and no service is called at all.

In addition, Step Functions validates the mock against the API model of the service. With fieldValidationMode, you choose how strict this is: STRICT (default) checks names, types and required fields, PRESENT only checks the fields in the mock, and NONE turns validation off. I would stay with STRICT, because a mock that can never exist in reality tests nothing.

For retry tests, there is also stateConfiguration.retrierRetryCount. It simulates how many retries have already happened for the matching retrier. For Map and Parallel states, errorCausedByState names the inner state the error came from.

Fast and cheap on the left, realistic on the right: each layer catches different mistakes.

The example: a bike rental with retry and catch

As an example, we take renting a bike, a deliberately small, fictional domain. The state machine first checks whether the rider is verified. After that, a Lambda function reserves a bike at the station, and a second function unlocks it. Two kinds of errors are interesting:

Throttling by Lambda (Lambda.TooManyRequestsException) is temporary and should be retried with backoff.
The business error BikeUnavailable (“no bike available”) should not be retried, but lead directly to a hint about another station.

Everything else ends up in the RentalFailed state via States.ALL. The placeholders ${ReserveBikeFunctionArn} and ${UnlockBikeFunctionArn} are replaced by CloudFormation through DefinitionSubstitutions on deployment, while the tests replace them with dummy ARNs. This way, tests and infrastructure share the same file.

{
  "Comment": "Bike rental: check rider, reserve bike, unlock bike",
  "StartAt": "CheckRider",
  "States": {
    "CheckRider": {
      "Type": "Choice",
      "Choices": [
        { "Variable": "$.rider.verified", "BooleanEquals": true, "Next": "ReserveBike" }
      ],
      "Default": "RejectRental"
    },
    "ReserveBike": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "${ReserveBikeFunctionArn}",
        "Payload.$": "$"
      },
      "ResultSelector": { "reservationId.$": "$.Payload.reservationId" },
      "ResultPath": "$.reservation",
      "Retry": [
        {
          "ErrorEquals": ["Lambda.TooManyRequestsException", "Lambda.ServiceException"],
          "IntervalSeconds": 2,
          "MaxAttempts": 3,
          "BackoffRate": 2
        }
      ],
      "Catch": [
        { "ErrorEquals": ["BikeUnavailable"], "ResultPath": "$.error", "Next": "OfferOtherStation" },
        { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "RentalFailed" }
      ],
      "Next": "UnlockBike"
    },
    "UnlockBike": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "${UnlockBikeFunctionArn}",
        "Payload.$": "$"
      },
      "ResultPath": null,
      "Retry": [
        {
          "ErrorEquals": ["Lambda.TooManyRequestsException", "Lambda.ServiceException"],
          "IntervalSeconds": 1,
          "MaxAttempts": 2,
          "BackoffRate": 2
        }
      ],
      "Catch": [
        { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "RentalFailed" }
      ],
      "End": true
    },
    "OfferOtherStation": {
      "Type": "Pass",
      "Parameters": {
        "status": "NO_BIKE",
        "stationId.$": "$.stationId",
        "hint": "Try a nearby station"
      },
      "End": true
    },
    "RejectRental": {
      "Type": "Fail",
      "Error": "RiderNotVerified",
      "Cause": "The rider has not been verified yet"
    },
    "RentalFailed": {
      "Type": "Fail",
      "Error": "RentalFailed",
      "Cause": "Reservation or unlock failed"
    }
  }
}

One detail that is often overlooked: States.ALL must always be the last catcher. Step Functions checks the catchers in order, so an early catch-all would swallow the specific BikeUnavailable case. We will check exactly that automatically in a moment.

🎬 How a TestState test runs – this animated diagram only runs on my site: watch it on sascha-ms.de

How to test Step Functions step by step with TypeScript

Now let us build the tests. I use the built-in test runner of Node.js (node:test) and the AWS SDK for JavaScript v3. Node.js 22.18 and later runs TypeScript files directly as long as they only contain type annotations. So the tests need no extra build step.

Prerequisites

You need Node.js 22.18 or later, an AWS account with credentials on the command line (for example via aws sso login) and the AWS CDK for deployment. The identity you test with needs the states:TestState permission. For tests without a mock, you also need iam:PassRole for the execution role.

Step 1: Create the project

mkdir bike-rental && cd bike-rental
npm init -y
npm pkg set type=module
npm install @aws-sdk/client-sfn aws-cdk-lib constructs
npm install -D typescript @types/node aws-cdk esbuild
mkdir statemachine src test lib bin

Then save the ASL file from above as statemachine/bike-rental.asl.json. The tsconfig.json is only used for type checking, which is why noEmit is set to true:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "nodenext",
    "moduleResolution": "nodenext",
    "strict": true,
    "noEmit": true,
    "allowImportingTsExtensions": true,
    "verbatimModuleSyntax": true,
    "types": ["node"],
    "skipLibCheck": true
  },
  "include": ["bin", "lib", "src", "test"]
}

Step 2: Guardrail tests without AWS access

Before we ask the cloud, a quick look at the structure pays off. These tests run completely offline and finish in milliseconds. They check rules I want to see in every review: every Lambda task retries throttling, has a catch-all, and that catch-all comes last. In addition, all transitions point to existing states.

// test/asl-guardrails.test.ts
import { test } from "node:test";
import assert from "node:assert/strict";
import { loadDefinition } from "./test-state.ts";

interface AslState {
  Type: string;
  Resource?: string;
  Retry?: { ErrorEquals: string[]; MaxAttempts?: number }[];
  Catch?: { ErrorEquals: string[]; Next: string }[];
  Next?: string;
  Default?: string;
  Choices?: { Next: string }[];
}

const definition = JSON.parse(loadDefinition()) as { StartAt: string; States: Record<string, AslState> };
const states = Object.entries(definition.States);

test("every Lambda task retries throttling and has a catch-all", () => {
  for (const [name, state] of states.filter(([, s]) => s.Type === "Task")) {
    const retried = (state.Retry ?? []).flatMap((r) => r.ErrorEquals);
    assert.ok(retried.includes("Lambda.TooManyRequestsException"), `${name}: Retry for throttling is missing`);
    const caught = (state.Catch ?? []).flatMap((c) => c.ErrorEquals);
    assert.ok(caught.includes("States.ALL"), `${name}: Catch with States.ALL is missing`);
  }
});

test("States.ALL is always the last catcher", () => {
  for (const [name, state] of states) {
    const catchers = state.Catch ?? [];
    const index = catchers.findIndex((c) => c.ErrorEquals.includes("States.ALL"));
    if (index >= 0) assert.equal(index, catchers.length - 1, `${name}: States.ALL must come last`);
  }
});

test("every transition points to an existing state", () => {
  const known = new Set(Object.keys(definition.States));
  assert.ok(known.has(definition.StartAt));
  for (const [name, state] of states) {
    const targets = [state.Next, state.Default, ...(state.Choices ?? []).map((c) => c.Next), ...(state.Catch ?? []).map((c) => c.Next)];
    for (const target of targets.filter(Boolean)) {
      assert.ok(known.has(target as string), `${name}: unknown target ${target}`);
    }
  }
});

test("all placeholders are replaced", () => {
  assert.doesNotMatch(loadDefinition(), /\$\{\w+\}/);
});

Step 3: A small helper for the TestState API

The helper loads the definition, replaces the placeholders and calls TestStateCommand. Without a mock, it requires a role from the SFN_TEST_ROLE_ARN environment variable, so nobody calls real Lambda functions by accident.

// test/test-state.ts
import { readFileSync } from "node:fs";
import { SFNClient, TestStateCommand, type TestStateCommandOutput } from "@aws-sdk/client-sfn";

const client = new SFNClient({});

// Replace the CDK placeholders with dummy ARNs, just like DefinitionSubstitutions does on deploy.
const placeholders: Record<string, string> = {
  ReserveBikeFunctionArn: "arn:aws:lambda:eu-central-1:123456789012:function:reserve-bike",
  UnlockBikeFunctionArn: "arn:aws:lambda:eu-central-1:123456789012:function:unlock-bike",
};

export function loadDefinition(file = "statemachine/bike-rental.asl.json"): string {
  const raw = readFileSync(file, "utf8");
  return raw.replace(/\$\{(\w+)\}/g, (match, key: string) => placeholders[key] ?? match);
}

export const hasAwsCredentials = Boolean(
  process.env.AWS_PROFILE || process.env.AWS_ACCESS_KEY_ID || process.env.AWS_WEB_IDENTITY_TOKEN_FILE,
);

export interface RunOptions {
  input: unknown;
  mockResult?: unknown;
  mockError?: { error: string; cause: string };
  retrierRetryCount?: number;
}

// Runs exactly one state of the full definition via the TestState API.
export async function runState(stateName: string, opts: RunOptions): Promise<TestStateCommandOutput> {
  const mocked = opts.mockResult !== undefined || opts.mockError !== undefined;
  const roleArn = process.env.SFN_TEST_ROLE_ARN;
  if (!mocked && !roleArn) {
    throw new Error(`State ${stateName} is tested without a mock: please set SFN_TEST_ROLE_ARN`);
  }
  return client.send(
    new TestStateCommand({
      definition: loadDefinition(),
      stateName,
      input: JSON.stringify(opts.input),
      inspectionLevel: "DEBUG",
      roleArn: mocked ? undefined : roleArn,
      mock: mocked
        ? {
            result: opts.mockResult !== undefined ? JSON.stringify(opts.mockResult) : undefined,
            errorOutput: opts.mockError,
          }
        : undefined,
      stateConfiguration:
        opts.retrierRetryCount !== undefined ? { retrierRetryCount: opts.retrierRetryCount } : undefined,
    }),
  );
}

Step 4: Test retry and catch

Now for the actual tests. Each test simulates a result or an error of the Lambda function and checks what Step Functions does with it. The second test is my favourite: it recalculates the backoff. With IntervalSeconds 2, BackoffRate 2 and one retry already done, the next pause is 2 × 2¹ = 4 seconds. This calculation matches the example in the AWS documentation.

// test/reserve-bike.test.ts
import { test } from "node:test";
import assert from "node:assert/strict";
import { hasAwsCredentials, runState } from "./test-state.ts";

const input = { rentalId: "r-42", stationId: "harbour-3", rider: { id: "u-7", verified: true } };
const skip = hasAwsCredentials ? false : "no AWS credentials – TestState runs in the cloud";

test("happy path: reservation lands in $.reservation", { skip }, async () => {
  const res = await runState("ReserveBike", {
    input,
    mockResult: { StatusCode: 200, Payload: { reservationId: "res-1001" } },
  });
  assert.equal(res.status, "SUCCEEDED");
  assert.equal(res.nextState, "UnlockBike");
  assert.deepEqual(JSON.parse(res.output ?? "{}").reservation, { reservationId: "res-1001" });
});

test("throttling is retried with backoff", { skip }, async () => {
  const res = await runState("ReserveBike", {
    input,
    mockError: { error: "Lambda.TooManyRequestsException", cause: "Rate exceeded" },
    retrierRetryCount: 1,
  });
  assert.equal(res.status, "RETRIABLE");
  assert.equal(res.inspectionData?.errorDetails?.retryIndex, 0);
  // IntervalSeconds 2 x BackoffRate 2 ^ 1 retry so far = 4 seconds
  assert.equal(res.inspectionData?.errorDetails?.retryBackoffIntervalSeconds, 4);
});

test("after the last retry the catch-all takes over", { skip }, async () => {
  const res = await runState("ReserveBike", {
    input,
    mockError: { error: "Lambda.TooManyRequestsException", cause: "Rate exceeded" },
    retrierRetryCount: 3,
  });
  assert.equal(res.status, "CAUGHT_ERROR");
  assert.equal(res.nextState, "RentalFailed");
  assert.equal(res.inspectionData?.errorDetails?.catchIndex, 1);
});

test("no bike available leads to another station, without retries", { skip }, async () => {
  const res = await runState("ReserveBike", {
    input,
    mockError: { error: "BikeUnavailable", cause: "Station harbour-3 is empty" },
  });
  assert.equal(res.status, "CAUGHT_ERROR");
  assert.equal(res.nextState, "OfferOtherStation");
  const output = JSON.parse(res.output ?? "{}");
  assert.equal(output.error.Error, "BikeUnavailable");
  assert.equal(output.rentalId, "r-42");
});

test("unverified riders are rejected", { skip: hasAwsCredentials && process.env.SFN_TEST_ROLE_ARN ? false : "needs SFN_TEST_ROLE_ARN" }, async () => {
  const res = await runState("CheckRider", { input: { ...input, rider: { id: "u-8", verified: false } } });
  assert.equal(res.status, "SUCCEEDED");
  assert.equal(res.nextState, "RejectRental");
});

The Choice check in the last test has no mock, because mocks only exist for Task, Map and Parallel states. The examples in the AWS documentation pass a role for it, so this test only runs when you set SFN_TEST_ROLE_ARN. Since a Choice state calls no service, a role without further service permissions should be enough.

Step 5: Run the tests

Add the scripts to package.json and start the tests. The region matters, otherwise the SDK stops with “Region is missing”.

npm pkg set scripts.typecheck="tsc --noEmit" scripts.test="node --test test/*.test.ts"
export AWS_REGION=eu-central-1
npm run typecheck
npm test

With valid credentials, all tests run against the TestState API. Without credentials, the guardrail tests still run and the cloud tests are skipped cleanly. The expected result then looks like this:

ok 1 - every Lambda task retries throttling and has a catch-all
ok 2 - States.ALL is always the last catcher
ok 3 - every transition points to an existing state
ok 4 - all placeholders are replaced
ok 5 - happy path: reservation lands in $.reservation # SKIP no AWS credentials – TestState runs in the cloud
...
# pass 4
# fail 0
# skipped 5

Step 6: Deploy the same definition with CDK

For deployment, the CDK stack reads exactly the file the tests use. DefinitionBody.fromFile uploads it as an asset, and definitionSubstitutions inserts the real function ARNs. The Lambda functions are deliberately simple. What matters is that BikeUnavailable arrives as the error name, which is why the error class sets this.name.

// src/reserve-bike.ts
class BikeUnavailable extends Error {
  constructor(stationId: string) {
    super(`No bike available at station ${stationId}`);
    this.name = "BikeUnavailable"; // becomes the error name in Step Functions
  }
}

interface RentalRequest {
  rentalId: string;
  stationId: string;
}

// Demo stock instead of a real database
const stock: Record<string, number> = { "harbour-3": 2, "old-town-1": 0 };

export const handler = async (event: RentalRequest) => {
  const free = stock[event.stationId] ?? 0;
  if (free < 1) throw new BikeUnavailable(event.stationId);
  return { reservationId: `res-${event.rentalId}` };
};

// src/unlock-bike.ts
export const handler = async (event: { rentalId: string; reservation: { reservationId: string } }) => {
  console.log(JSON.stringify({ msg: "unlock", rentalId: event.rentalId, reservationId: event.reservation.reservationId }));
  return { unlocked: true };
};

// lib/bike-rental-stack.ts
import * as path from "node:path";
import { Duration, Stack, type StackProps } from "aws-cdk-lib";
import { Runtime } from "aws-cdk-lib/aws-lambda";
import { NodejsFunction } from "aws-cdk-lib/aws-lambda-nodejs";
import { DefinitionBody, StateMachine, StateMachineType } from "aws-cdk-lib/aws-stepfunctions";
import type { Construct } from "constructs";

export class BikeRentalStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const common = { runtime: Runtime.NODEJS_22_X, memorySize: 256, timeout: Duration.seconds(10) };
    const reserveBike = new NodejsFunction(this, "ReserveBike", { ...common, entry: "src/reserve-bike.ts" });
    const unlockBike = new NodejsFunction(this, "UnlockBike", { ...common, entry: "src/unlock-bike.ts" });

    const rental = new StateMachine(this, "BikeRental", {
      stateMachineType: StateMachineType.STANDARD,
      // The same ASL file that the tests read
      definitionBody: DefinitionBody.fromFile(path.join("statemachine", "bike-rental.asl.json")),
      definitionSubstitutions: {
        ReserveBikeFunctionArn: reserveBike.functionArn,
        UnlockBikeFunctionArn: unlockBike.functionArn,
      },
    });

    reserveBike.grantInvoke(rental);
    unlockBike.grantInvoke(rental);
  }
}

// bin/app.ts
import { App } from "aws-cdk-lib";
import { BikeRentalStack } from "../lib/bike-rental-stack.ts";

const app = new App();
new BikeRentalStack(app, "BikeRentalStack");

echo '{ "app": "node bin/app.ts" }' > cdk.json
npx cdk synth
npx cdk deploy

In the synthesized template, you will find the AWS::StepFunctions::StateMachine resource with DefinitionS3Location and both substitutions. If you prefer the Serverless Framework, the serverless-step-functions plugin gets you to the same place. The tests stay unchanged, because they only know the ASL file.

Typical mistakes when testing with TestState

A few pitfalls stood out while I worked through the documentation and built the example:

Missing region: The SDK needs a region. Set AWS_REGION or add it to your profile.
No mock, no role: Without a mock, the TestState API has to call real services and needs a role with matching permissions. The helper above deliberately stops with a clear message in that case.
Mock does not match the API model: In STRICT mode, Step Functions rejects mocks with wrong field names or types. For .sync integrations, the mock must also match the response of the polling API, so for startExecution.sync:2 it must look like DescribeExecution.
Map and Parallel: A mock is required here. The test covers input and output processing of the outer state, not the individual iterations. You test the inner state separately via stateName.
Console instead of code: According to the documentation, the console does not support mocks or Map and Parallel tests yet. For everything in this article, you need the CLI or an SDK.
Exhausted retries: retrierRetryCount counts the retries that have already happened. If the value equals MaxAttempts, no attempts are left and the catcher takes over. If your test shows something else, look at the inspection data with DEBUG.

Cost and security when testing

The TestState API creates no state machine and starts no execution. The Step Functions pricing page does not list TestState as a separate item. What it bills are state transitions of real executions: for Standard workflows in US East (N. Virginia), 0.000025 US dollars per transition, with 4,000 free transitions per month. Still, check the current pricing page for your region before using it in large CI pipelines.

For security, the rule is simple: mocked tests only need states:TestState. Give your CI role exactly this permission and no iam:PassRole as long as you only work with mocks. Only tests with real calls need an execution role, which you should scope tightly as well. Leave out the revealSecrets parameter in CI pipelines, so no credentials from HTTP tasks end up in test logs.

My take

For me, the TestState API with mocks is the most useful testing tool Step Functions has received so far. It tests the real execution logic instead of a replica, and it makes exactly those parts testable that otherwise only run in an emergency. Especially with patterns like the saga pattern with Step Functions, correctness depends on catch rules and compensations. Those should not only be right on a whiteboard.

Still, TestState does not replace a test after deployment. The API checks one state after another, mocked and without real IAM permissions. Whether the role of the state machine is really allowed to invoke the Lambda function is something you will not see this way. So my split would be: guardrail tests offline on every commit, TestState tests with mocks in CI and, after deployment, a single real execution as a smoke test.

One point where I disagree with some teams: they try to test every combination of errors and states. That quickly turns into a test suite nobody understands anymore. Instead, test the decisions that matter for the business. For the bike rental, these are “throttling is retried”, “no bike available goes to another station” and “everything else fails in a controlled way”. By the way, I wrote about how long the pauses between retries should be in more detail in exponential backoff with jitter.

FAQ

Can I test Step Functions locally?

Step Functions Local still exists, but AWS calls it unsupported and without feature parity. For tests before deployment, AWS recommends the TestState API. You can additionally check the structure of the ASL file offline with your own tests.

Does the TestState API need an IAM role?

Only without a mock. As soon as you mock a result or an error, roleArn is optional. The caller always needs the states:TestState permission.

Can TestState run a whole state machine?

No, each call tests exactly one state. However, you can chain tests by using the output and nextState of one call as the input for the next.

Does TestState work with Map and Parallel states?

Yes, with a mock. The test covers input and output processing of the outer state. For Parallel, the mock must be an array with one element per branch, and for errors you name the inner state that caused them with errorCausedByState.

Your next step

Pick the state machine in your setup with the most error handling. Write exactly one test with a mocked error for each catcher and run it in CI. If you are still deciding whether Step Functions is the right service for your workflow at all, the AWS service finder helps you sort it out.

And now I am curious: how do you test your Step Functions workflows today, and which error path surprised you most recently in production?

Sources

AWS Step Functions API Reference: TestState
AWS Step Functions: Using TestState API to test a state
AWS Step Functions API Reference: TestStateConfiguration
AWS Step Functions: Step Functions Local (unsupported)
AWS CloudFormation: AWS::StepFunctions::StateMachine (DefinitionSubstitutions)
AWS Step Functions: Pricing
Checked on 10/28/2026 with Node.js 22.23, TypeScript 7.0.2, aws-cdk-lib 2.275.0, AWS CDK CLI 2.1146 and @aws-sdk/client-sfn 3.1158: tsc, 4 offline tests and cdk synth. The TestState tests are written against the documentation, and the ARNs are placeholders.

This post was created with AI assistance.

👉 Next step: Working with AWS CDK? Grab my printable AWS CDK cheat sheet with commands, concepts and a stack skeleton.

Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en

Event-Driven Architecture on the Cloud

Sri Balaji — Sat, 31 Oct 2026 15:55:14 +0000

⚡ TL;DR: Replace brittle synchronous calls with events, and learn to pick the right building block: queues, pub/sub, or streams. Because delivery is at-least-once, idempotency is the non-negotiable price of admission.

When one slow call takes down everything
The mental model: a newsroom, not a phone tree
The picture: one event, many reactions
Queues vs pub/sub vs streams
Publishing and consuming an event
Delivery guarantees: why idempotency is non-negotiable
Common mistakes that cost hours
When event-driven beats a synchronous call
Where to go next

When one slow call takes down everything

You ship an order service. It takes a payment, then calls the email service, then the analytics service, then the warehouse service, one after another, in the same request. It works great in the demo. Then the email provider has a bad afternoon, its API hangs for 30 seconds, and suddenly customers can't place orders at all. Nobody is buying email. They're buying products. But because checkout calls email synchronously, a hiccup in a non-critical service became an outage in your most critical one.

The fix isn't a faster email service. It's changing the shape of the conversation. Instead of checkout calling everyone and waiting, checkout announces "an order was placed" and walks away. Whoever cares, email, analytics, the warehouse, reacts on their own time. That announcement is an event, and designing around it is event-driven architecture.

📌 Who this is for: Developers and junior cloud engineers who can already build a service that calls another service over HTTP, and are starting to feel the pain, cascading failures, slow requests, services that have to know too much about each other. No prior messaging experience needed. We use AWS, GCP, and Azure names, but the ideas are identical everywhere.

The mental model: a newsroom, not a phone tree

Event-driven architecture means services communicate by emitting and reacting to events, facts about what happened, instead of directly commanding each other.

The one-sentence version

Synchronous calls are a phone tree. To tell five teams something, you phone each one, wait for them to pick up, and you're stuck on the line until the last call ends. If one person doesn't answer, you're frozen mid-tree.

Event-driven is a newsroom. A reporter publishes a story. They don't know, or care, who reads it. Subscribers (the sports desk, the weather team, an archive bot) each pick it up and do their own thing, at their own pace. The reporter's job is done the moment the story is filed. Add a tenth subscriber tomorrow and the reporter's code never changes.

In the real world	In tech
A reporter files a story	A producer publishes an event
The story itself (a fact that happened)	The event / message payload
The newswire everyone reads from	The topic / queue / stream
The sports & weather desks	Independent consumers / subscribers
Anyone can start reading the wire tomorrow	Add a consumer without touching the producer

The newsroom maps cleanly onto the building blocks.

The picture: one event, many reactions

Here's the order flow rebuilt around an event. Checkout publishes once to a topic; the topic fans the event out to every interested consumer. Checkout finishes in milliseconds and is completely unaffected if a downstream consumer is slow or down.

A producer publishes once; the topic fans the event out to independent consumers (fan-out).

Customer clicks Buy: Checkout charges the card and persists the order. That part is still synchronous, it must succeed before we promise anything.
Checkout publishes one event: It sends a single OrderPlaced message to the topic with the order id and details, then returns 200 to the customer. Total added latency: a few milliseconds.
The topic fans out: The messaging service delivers a copy to every subscriber, email, warehouse, analytics. Checkout has no idea who they are.
Consumers react independently: Email sends a receipt. The warehouse reserves stock. Analytics records the sale. If email is down, the warehouse and analytics are unaffected, and email retries later.

Queues vs pub/sub vs streams

"Messaging" is three different patterns wearing the same coat. Picking the wrong one is the most common early mistake, so anchor on these distinctions before you reach for a service.

Queue, one message, one worker. The message is consumed and gone. Use it to distribute work: ten workers pull from one queue and each grabs different items.
Pub/Sub, one message, every subscriber gets a copy. Use it to broadcast a fact so multiple systems can react (this is fan-out).
Stream, an append-only log you can replay. Messages aren't deleted on read; consumers track their own position. Use it for ordered history, replay, and analytics.

	Queue	Pub/Sub	Stream
Delivery	1 message → 1 consumer	1 message → all subscribers	1 log → many readers, each at own offset
Ordering	Best-effort (FIFO variants exist)	Usually unordered	Strong, per-partition order
Fan-out	No (work is split, not copied)	Yes, the whole point	Yes (each consumer reads the full log)
Replay	No, read = gone	No, miss it, miss it	Yes, rewind to any offset
Best for	Background jobs, task distribution	Broadcasting events, decoupling	Event sourcing, metrics, audit trails
Managed examples	SQS, Pub/Sub (pull), Service Bus queues	SNS, EventBridge, Pub/Sub, Service Bus topics	Kinesis, Kafka / MSK, Event Hubs

The same event needs different plumbing depending on what you want from it.

💡 Queue + pub/sub is the classic combo: On AWS the textbook fan-out is SNS → SQS: SNS broadcasts the event, and each consumer has its own SQS queue subscribed to the topic. You get broadcast AND a durable buffer per consumer, so a slow consumer never blocks the others. EventBridge plays the same role with richer routing rules.

Publishing and consuming an event

Concretely, publishing is a one-liner and consuming is a small loop. Here's the SNS → SQS fan-out from the diagram, in Python with boto3. First, checkout publishes the event:

publish_order.py

import json
import boto3

sns = boto3.client("sns")
TOPIC_ARN = "arn:aws:sns:eu-west-1:123456789012:OrderPlaced"

def publish_order_placed(order_id: str, total: float, email: str) -> None:
    event = {
        "type": "OrderPlaced",
        "order_id": order_id,
        "total": total,
        "customer_email": email,
    }
    sns.publish(
        TopicArn=TOPIC_ARN,
        Message=json.dumps(event),
        # idempotency: a stable id lets consumers dedupe
        MessageAttributes={
            "event_id": {"DataType": "String", "StringValue": order_id},
        },
    )
    # checkout returns to the customer right here, no waiting on consumers

Each consumer owns an SQS queue subscribed to that topic. The email worker just polls its queue, does its job, and deletes the message to acknowledge it:

email_consumer.py

import json
import boto3

sqs = boto3.client("sqs")
QUEUE_URL = "https://sqs.eu-west-1.amazonaws.com/123456789012/email-queue"

while True:
    resp = sqs.receive_message(
        QueueUrl=QUEUE_URL,
        MaxNumberOfMessages=10,
        WaitTimeSeconds=20,  # long polling, cheaper, less spin
    )
    for msg in resp.get("Messages", []):
        envelope = json.loads(msg["Body"])        # SNS wraps the payload
        event = json.loads(envelope["Message"])   # our actual event

        if not already_processed(event["order_id"]):
            send_receipt(event["customer_email"], event["order_id"])
            mark_processed(event["order_id"])

        # delete = acknowledge. Only do this AFTER the work succeeded.
        sqs.delete_message(QueueUrl=QUEUE_URL, ReceiptHandle=msg["ReceiptHandle"])

Notice the already_processed / mark_processed guard, and that we delete the message only after the work succeeds. That's not optional decoration, it's the heart of running this safely. Here's why.

Delivery guarantees: why idempotency is non-negotiable

Almost every managed messaging service gives you at-least-once delivery. Read that carefully: at least once. Not exactly once. The same event can, and eventually will, be delivered to your consumer more than once. This isn't a bug; it's the honest trade-off that makes the system reliable.

It happens for boring, unavoidable reasons. A consumer processes a message, but its acknowledgement gets lost on the network. The broker never hears "done," so after a visibility timeout it redelivers, and now you've sent two receipt emails for one order. The number of guarantees in plain terms:

At-most-once, fire and forget. Fast, but you can silently lose messages. Rarely what you want.
At-least-once, never lost, sometimes duplicated. The realistic default for SQS, SNS, Pub/Sub, and Kinesis.
Exactly-once, the dream. A few services offer it in narrow conditions (Kafka transactions, SQS FIFO dedup windows), but it's limited and costs throughput. Don't architect around assuming it.

⚠️ Design for duplicates, not against them: Since you'll get duplicates, make your consumers idempotent: processing the same event twice has the same effect as processing it once. Use the event's stable id as a dedup key, record "I've handled order_id X," and skip it if it shows up again. Idempotency turns at-least-once from a liability into a non-issue.

The same instinct covers the other failure: a message your consumer can never process (bad data, a permanent bug). Without a backstop it gets redelivered forever, a "poison pill." Configure a dead-letter queue (DLQ) so a message that fails N times moves aside for inspection instead of blocking the line.

Common mistakes that cost hours

Treating at-least-once as exactly-once. No dedup guard means double-charged cards and duplicate emails the first time a network blip causes a redelivery. Make consumers idempotent from day one.
Acknowledging before the work is done. Delete or ack the message only after processing succeeds. Ack first and crash, and the event is gone forever.
No dead-letter queue. A single malformed message retries endlessly, drowns your logs, and can stall the whole queue. Always wire a DLQ with a sane retry count.
Putting commands in events. An event states a fact (OrderPlaced), not an instruction (SendEmail). If the producer is telling a specific consumer what to do, you've just rebuilt a synchronous call with extra steps and lost the decoupling.
Reaching for a stream when you needed a queue. Kafka/Kinesis are powerful and operationally heavy. If you just need background jobs, a plain queue (SQS) is simpler, cheaper, and enough.
Forgetting ordering isn't free. Standard queues and pub/sub don't guarantee order. If OrderShipped can arrive before OrderPlaced, either use a FIFO/partitioned option or make consumers tolerant of out-of-order events.

When event-driven beats a synchronous call

Event-driven isn't free, you trade the simplicity of a function call for eventual consistency, harder debugging, and new infrastructure. So don't make everything an event. Reach for it when the trade pays off:

Multiple consumers care about the same thing, fan-out beats calling each one yourself.
The reaction can happen later, receipts, analytics, and indexing don't need to finish before you answer the user.
You want failure isolation, a down consumer shouldn't take down the producer.
Load is spiky, a queue absorbs bursts and lets workers drain at their own pace.

Keep it synchronous when the caller genuinely needs the answer now to continue, reading a user's profile to render a page, or checking inventory before confirming a price. A request that can't proceed without the response shouldn't be fire-and-forget.

The whole article in seven lines

Event-driven = services emit and react to facts, instead of commanding each other directly.
Think newsroom (publish, anyone subscribes), not phone tree (call everyone, wait).
Queue = work split across workers. Pub/Sub = broadcast a copy to all. Stream = replayable ordered log.
Fan-out (one event, many consumers) is the superpower, add consumers without touching the producer.
Delivery is at-least-once, so duplicates are guaranteed, make every consumer idempotent.
Always add a dead-letter queue so a poison message can't block the line.
Use events for fan-out, deferrable work, and failure isolation; stay synchronous when the caller needs the answer to continue.

Where to go next

You now have the vocabulary and the mental model. The fastest way to make it stick is to wire a real producer and consumer, then design a system where events change how it scales.

Go deeper on the worker side: Async Processing: Queues & Workers covers retries, visibility timeouts, and worker pools in detail.
See how decoupling shapes growth: Scalability Principles shows why queues and fan-out are the backbone of systems that scale.
Ready to build the rest of the stack? The Cloud Engineer path walks you from networking to compute to event-driven systems, level by level.

Originally published on TheSimplifiedTech, where this guide is interactive, with in-browser terminal labs and diagrams. Learn cloud and DevOps by doing, no videos.

Saga pattern with Step Functions: rolling back cleanly

Sascha Mirow-Stanitzok — Sat, 31 Oct 2026 15:55:03 +0000

The saga pattern with Step Functions solves a problem every distributed system runs into sooner or later: a business process spans several services, and something breaks halfway through. The booking exists, the bike is reserved, but the payment gets declined. However, there is no database transaction covering all three steps. What now?

This topic keeps coming up for me because architecture discussions often swing between two extremes. One camp wants to force everything into a single database so it becomes “properly” transactional again. The other camp, on the other hand, simply hopes nothing goes wrong. In short, neither gets you very far. So in this final part of the series, I will show you step by step how to build a workflow with AWS Step Functions that cleans up after itself when something fails, with complete CDK code in TypeScript.

In short

A saga splits a distributed process into local steps and pairs each step with a compensating action that runs in reverse when an error occurs. Step Functions acts as the conductor and, as a result, makes retries, catches and ordering visible. The biggest pitfall is not the state machine itself but compensations that are not idempotent or that are allowed to fail.

What you will learn in this article

When a saga makes sense and when a simple transaction is enough
How to wire forward steps and compensations in Step Functions with CDK
Why every compensation has to be idempotent and what that looks like in code
How to configure Retry and Catch so only sensible errors are retried
How to test the saga locally and make it fail on purpose after deployment

What is the saga pattern?

A saga is a sequence of local transactions. Each step changes data in exactly one service and is therefore complete on its own. For every step that changes something, there is a matching compensation: a business-level counter-action that undoes its effect. If step three hits an error, the saga runs the compensations for steps two and one in reverse order.

The word “business-level” matters here. After all, a compensation is not a technical rollback that pretends nothing ever happened. For example, a charged payment gets refunded; it does not vanish. Anyone looking at the data in the meantime briefly sees a state that will be corrected later. This is also called eventual consistency.

Orchestration or choreography

There are two ways to coordinate a saga. With choreography, each service reacts to events from the others, and there is no central flow. With orchestration, however, a conductor (the orchestrator) tells each participant what happens next. Choreography works well when a few services should stay loosely coupled, and it builds on patterns such as the outbox pattern. Once four or more steps with compensations come together, though, pure choreography quickly makes me lose track of which service is currently undoing what.

That is why I prefer an orchestrator for flows like that. On AWS, Step Functions is the obvious choice: the flow lives in one place as a state machine, every execution can be traced in the console, and Retry and Catch are built into the language. Moreover, AWS Prescriptive Guidance describes exactly this variant with Step Functions as the orchestrator.

When you do not need a saga

Before you start, here is an honest shortcut. If all the affected data lives in one DynamoDB table or a few tables in the same account, TransactWriteItems is often enough. It also writes several items atomically, with no compensations at all. A saga only pays off when steps cross boundaries that share no transaction: an external payment provider, another team with its own database, or a legacy system.

The example: a bike rental booking

For example, picture a fictional bike rental shop. A booking consists of four steps:

CreateBooking creates the booking with status PENDING.
ReserveBike reserves the bike if it is available.
ChargePayment charges the amount with the (simulated) payment provider.
ConfirmBooking sets the booking to CONFIRMED.

Three compensations belong to it, running in reverse: RefundPayment refunds the payment, ReleaseBike frees the bike, and MarkBookingFailed sets the booking to FAILED. Deleting the booking would work too. I prefer to keep it with a status because you will want to understand later why a booking failed.

Forward steps on top, compensations below: each Catch jumps to the point from which the saga has to roll back.

Also notice where the catches jump to. If the reservation fails, there is nothing to refund and no bike to release, so the flow goes straight to MarkBookingFailed. If the payment or the confirmation fails, the chain starts at RefundPayment. The compensations are chained so that every entry point automatically includes all later cleanup steps.

🎬 Flow of the booking saga with compensation – this animated diagram only runs on my site: watch it on sascha-ms.de

Preparing the saga pattern with Step Functions

For this guide you need:

Node.js 22 or newer and npm
an AWS account with configured credentials (aws configure or SSO)
the AWS CDK, here as a local dependency (tested with aws-cdk-lib 2.275)
an account prepared with cdk bootstrap and a Region, for example eu-central-1

This time I build the infrastructure with CDK, since the previous part used the Serverless Framework. Two things matter to me here. Firstly, the DynamoDB steps run as direct Step Functions service integrations, with no Lambda in between. As a result, it saves code, cold starts and money. Secondly, Lambda only shows up where real logic is needed, which here is the payment provider.

Step 1: Create the project

mkdir bike-booking-saga && cd bike-booking-saga
npm init -y
npm install aws-cdk-lib constructs @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb
npm install -D typescript tsx esbuild aws-cdk @types/node @types/aws-lambda
mkdir src lib bin

Create a tsconfig.json and a cdk.json. In addition, the cdk.json tells the CDK how to start your app. You need esbuild locally so that NodejsFunction can bundle the handlers without Docker.

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "noEmit": true,
    "types": ["node"]
  },
  "include": ["src", "lib", "bin"]
}

{
  "app": "npx tsx bin/app.ts"
}

Idempotent compensations in code

Before we build the state machine, let us look at the place where sagas most often fail in practice: the participants themselves. Step Functions retries steps on errors, and a network timeout can also cause a call to arrive twice. Every action and every compensation therefore has to be idempotent. In other words, a second call with the same booking ID must not cause anything new. You can find more on the basics in my article on idempotency in AWS Lambda.

Step 2: Write the payment logic

The payment logic is deliberately simple: the simulated bank declines amounts over 300 euros. As a result, we get a reproducible failure case. Storage sits behind a small interface so we can test the logic without AWS.

// src/payment.ts
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import {
  DynamoDBDocumentClient,
  GetCommand,
  PutCommand,
  UpdateCommand,
} from "@aws-sdk/lib-dynamodb";

export interface BookingInput {
  bookingId: string;
  bikeId: string;
  customerId: string;
  amountCents: number;
}

export interface PaymentRecord {
  bookingId: string;
  amountCents: number;
  status: "CHARGED" | "REFUNDED";
}

// Business error: its name shows up in Step Functions as "Error" in the Catch.
export class PaymentDeclinedError extends Error {
  constructor(message: string) {
    super(message);
    this.name = "PaymentDeclinedError";
  }
}

export interface PaymentStore {
  // Creates the payment only if the booking does not have one yet.
  createIfAbsent(record: PaymentRecord): Promise<PaymentRecord>;
  get(bookingId: string): Promise<PaymentRecord | undefined>;
  markRefunded(bookingId: string): Promise<void>;
}

// Demo rule instead of a real payment provider: the "bank" declines anything over 300 euros.
const LIMIT_CENTS = 30_000;

export async function chargePayment(
  input: BookingInput,
  store: PaymentStore,
): Promise<PaymentRecord> {
  if (input.amountCents <= 0) {
    throw new PaymentDeclinedError("Amount must be positive");
  }
  if (input.amountCents > LIMIT_CENTS) {
    throw new PaymentDeclinedError(
      `Amount of ${input.amountCents} cents exceeds the limit`,
    );
  }
  // Idempotent: a second call with the same bookingId does not charge twice.
  return store.createIfAbsent({
    bookingId: input.bookingId,
    amountCents: input.amountCents,
    status: "CHARGED",
  });
}

export async function refundPayment(
  input: Pick<BookingInput, "bookingId">,
  store: PaymentStore,
): Promise<{ bookingId: string; refunded: boolean }> {
  const existing = await store.get(input.bookingId);
  // Never charged or already refunded: the compensation is a no-op.
  if (!existing || existing.status === "REFUNDED") {
    return { bookingId: input.bookingId, refunded: false };
  }
  await store.markRefunded(input.bookingId);
  return { bookingId: input.bookingId, refunded: true };
}

export class DynamoPaymentStore implements PaymentStore {
  constructor(
    private readonly tableName: string,
    private readonly doc = DynamoDBDocumentClient.from(new DynamoDBClient({})),
  ) {}

  async createIfAbsent(record: PaymentRecord): Promise<PaymentRecord> {
    try {
      await this.doc.send(
        new PutCommand({
          TableName: this.tableName,
          Item: record,
          ConditionExpression: "attribute_not_exists(bookingId)",
        }),
      );
      return record;
    } catch (err) {
      if ((err as Error).name === "ConditionalCheckFailedException") {
        const existing = await this.get(record.bookingId);
        if (existing) return existing;
      }
      throw err;
    }
  }

  async get(bookingId: string): Promise<PaymentRecord | undefined> {
    const res = await this.doc.send(
      new GetCommand({ TableName: this.tableName, Key: { bookingId } }),
    );
    return res.Item as PaymentRecord | undefined;
  }

  async markRefunded(bookingId: string): Promise<void> {
    await this.doc.send(
      new UpdateCommand({
        TableName: this.tableName,
        Key: { bookingId },
        UpdateExpression: "SET #s = :refunded",
        ConditionExpression: "attribute_exists(bookingId)",
        ExpressionAttributeNames: { "#s": "status" },
        ExpressionAttributeValues: { ":refunded": "REFUNDED" },
      }),
    );
  }
}

Three details matter. Firstly, the business error gets its own name. Exactly this name shows up in Step Functions as the error type, and you can react to it specifically in a Catch. Secondly, the charge uses a conditional write (attribute_not_exists), so a second call returns the existing payment instead of charging again. Finally, the refund is a no-op if there never was a payment. That is not an edge case; it is the normal case when the charge was declined.

Step 3: Lambda handlers

The handlers are just thin wrappers around the logic. Furthermore, the table name comes from an environment variable that the CDK stack sets.

// src/handlers.ts
import {
  type BookingInput,
  chargePayment,
  DynamoPaymentStore,
  refundPayment,
} from "./payment.js";

const store = new DynamoPaymentStore(process.env.PAYMENTS_TABLE ?? "");

export const charge = async (event: BookingInput) => chargePayment(event, store);

export const refund = async (event: Pick<BookingInput, "bookingId">) =>
  refundPayment(event, store);

Step 4: Test the logic

With the built-in Node.js test runner, you check the most important properties before anything runs in AWS: duplicate calls, declines and idempotent refunds.

// src/payment.test.ts
import { test } from "node:test";
import assert from "node:assert/strict";
import {
  chargePayment,
  PaymentDeclinedError,
  type PaymentRecord,
  type PaymentStore,
  refundPayment,
} from "./payment.js";

class InMemoryPaymentStore implements PaymentStore {
  readonly items = new Map<string, PaymentRecord>();
  async createIfAbsent(record: PaymentRecord) {
    const existing = this.items.get(record.bookingId);
    if (existing) return existing;
    this.items.set(record.bookingId, record);
    return record;
  }
  async get(bookingId: string) {
    return this.items.get(bookingId);
  }
  async markRefunded(bookingId: string) {
    const item = this.items.get(bookingId);
    if (item) item.status = "REFUNDED";
  }
}

const booking = {
  bookingId: "b-1",
  bikeId: "bike-7",
  customerId: "k-42",
  amountCents: 4_500,
};

test("charges a valid amount", async () => {
  const store = new InMemoryPaymentStore();
  const result = await chargePayment(booking, store);
  assert.equal(result.status, "CHARGED");
  assert.equal(store.items.size, 1);
});

test("a duplicate call does not charge twice", async () => {
  const store = new InMemoryPaymentStore();
  await chargePayment(booking, store);
  await chargePayment(booking, store);
  assert.equal(store.items.size, 1);
});

test("declines amounts that are too high with PaymentDeclinedError", async () => {
  const store = new InMemoryPaymentStore();
  await assert.rejects(
    chargePayment({ ...booking, amountCents: 50_000 }, store),
    (err: Error) => err instanceof PaymentDeclinedError && err.name === "PaymentDeclinedError",
  );
  assert.equal(store.items.size, 0);
});

test("a refund without a prior payment is a no-op", async () => {
  const store = new InMemoryPaymentStore();
  const result = await refundPayment({ bookingId: "unknown" }, store);
  assert.deepEqual(result, { bookingId: "unknown", refunded: false });
});

test("the refund is idempotent", async () => {
  const store = new InMemoryPaymentStore();
  await chargePayment(booking, store);
  const first = await refundPayment(booking, store);
  const second = await refundPayment(booking, store);
  assert.equal(first.refunded, true);
  assert.equal(second.refunded, false);
  assert.equal(store.items.get("b-1")?.status, "REFUNDED");
});

npx tsc --noEmit
npx tsx --test src/payment.test.ts

Expected result: # pass 5 and # fail 0 at the end of the output.

Building the state machine with CDK

Now for the orchestrator. The stack creates three DynamoDB tables (bookings, bikes, payments), two Lambda functions and the state machine. It is easiest to read the code from the bottom up: the forward chain sits at the end and the compensations come before it, because every Catch has to point to a state that is already defined.

Step 5: Write the stack

// lib/bike-booking-saga-stack.ts
import * as path from "node:path";
import { CfnOutput, Duration, RemovalPolicy, Stack, type StackProps } from "aws-cdk-lib";
import * as dynamodb from "aws-cdk-lib/aws-dynamodb";
import * as lambda from "aws-cdk-lib/aws-lambda";
import { NodejsFunction } from "aws-cdk-lib/aws-lambda-nodejs";
import * as sfn from "aws-cdk-lib/aws-stepfunctions";
import * as tasks from "aws-cdk-lib/aws-stepfunctions-tasks";
import type { Construct } from "constructs";

// Errors that are worth retrying (throttling, timeouts).
const TRANSIENT_DYNAMO_ERRORS = [
  "DynamoDB.ProvisionedThroughputExceededException",
  "DynamoDB.RequestLimitExceeded",
  "DynamoDB.ThrottlingException",
  "States.Timeout",
];

export class BikeBookingSagaStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const table = (name: string, key: string) =>
      new dynamodb.Table(this, name, {
        partitionKey: { name: key, type: dynamodb.AttributeType.STRING },
        billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
        removalPolicy: RemovalPolicy.DESTROY, // example only
      });

    const bookings = table("Bookings", "bookingId");
    const bikes = table("Bikes", "bikeId");
    const payments = table("Payments", "bookingId");

    const fn = (name: string, handler: string) =>
      new NodejsFunction(this, name, {
        entry: path.join(__dirname, "../src/handlers.ts"),
        handler,
        runtime: lambda.Runtime.NODEJS_22_X,
        architecture: lambda.Architecture.ARM_64,
        memorySize: 256,
        timeout: Duration.seconds(10),
        environment: { PAYMENTS_TABLE: payments.tableName },
      });

    const chargeFn = fn("ChargeFn", "charge");
    const refundFn = fn("RefundFn", "refund");
    payments.grantReadWriteData(chargeFn);
    payments.grantReadWriteData(refundFn);

    const bookingIdAttr = tasks.DynamoAttributeValue.fromString(
      sfn.JsonPath.stringAt("$.bookingId"),
    );

    // ---------- Final states ----------
    const bookingConfirmed = new sfn.Succeed(this, "BookingConfirmed");
    const bookingFailed = new sfn.Fail(this, "BookingFailed", {
      error: "BookingFailed",
      cause: "Booking was rolled back",
    });
    const compensationFailed = new sfn.Fail(this, "CompensationFailed", {
      error: "CompensationFailed",
      cause: "Compensation failed, check manually",
    });

    // ---------- Compensations (in reverse) ----------
    const markBookingFailed = new tasks.DynamoUpdateItem(this, "MarkBookingFailed", {
      table: bookings,
      key: { bookingId: bookingIdAttr },
      updateExpression: "SET #s = :failed",
      expressionAttributeNames: { "#s": "status" },
      expressionAttributeValues: {
        ":failed": tasks.DynamoAttributeValue.fromString("FAILED"),
      },
      resultPath: sfn.JsonPath.DISCARD,
    });
    markBookingFailed
      .addRetry({ errors: TRANSIENT_DYNAMO_ERRORS, maxAttempts: 5, backoffRate: 2 })
      .addCatch(compensationFailed, { resultPath: "$.compensationError" })
      .next(bookingFailed);

    const releaseBike = new tasks.DynamoUpdateItem(this, "ReleaseBike", {
      table: bikes,
      key: { bikeId: tasks.DynamoAttributeValue.fromString(sfn.JsonPath.stringAt("$.bikeId")) },
      updateExpression: "REMOVE reservedBy",
      // Only release our own reservation, never one held by another booking.
      conditionExpression: "reservedBy = :b",
      expressionAttributeValues: { ":b": bookingIdAttr },
      resultPath: sfn.JsonPath.DISCARD,
    });
    releaseBike
      .addRetry({ errors: TRANSIENT_DYNAMO_ERRORS, maxAttempts: 5, backoffRate: 2 })
      // Bike is not (or no longer) ours: nothing to release, continue with the next step.
      .addCatch(markBookingFailed, {
        errors: ["DynamoDB.ConditionalCheckFailedException"],
        resultPath: sfn.JsonPath.DISCARD,
      })
      .addCatch(compensationFailed, { resultPath: "$.compensationError" })
      .next(markBookingFailed);

    const refundPayment = new tasks.LambdaInvoke(this, "RefundPayment", {
      lambdaFunction: refundFn,
      payload: sfn.TaskInput.fromObject({ bookingId: sfn.JsonPath.stringAt("$.bookingId") }),
      resultSelector: { "refunded.$": "$.Payload.refunded" },
      resultPath: "$.refund",
    });
    refundPayment
      .addRetry({ errors: ["States.ALL"], interval: Duration.seconds(2), maxAttempts: 5, backoffRate: 2 })
      .addCatch(compensationFailed, { resultPath: "$.compensationError" })
      .next(releaseBike);

    // ---------- Forward steps ----------
    const createBooking = new tasks.DynamoPutItem(this, "CreateBooking", {
      table: bookings,
      item: {
        bookingId: bookingIdAttr,
        bikeId: tasks.DynamoAttributeValue.fromString(sfn.JsonPath.stringAt("$.bikeId")),
        customerId: tasks.DynamoAttributeValue.fromString(sfn.JsonPath.stringAt("$.customerId")),
        status: tasks.DynamoAttributeValue.fromString("PENDING"),
      },
      resultPath: sfn.JsonPath.DISCARD,
    });
    createBooking.addRetry({ errors: TRANSIENT_DYNAMO_ERRORS, maxAttempts: 3 });

    const reserveBike = new tasks.DynamoUpdateItem(this, "ReserveBike", {
      table: bikes,
      key: { bikeId: tasks.DynamoAttributeValue.fromString(sfn.JsonPath.stringAt("$.bikeId")) },
      updateExpression: "SET reservedBy = :b",
      // Semantic lock: free, or already reserved by exactly this booking.
      conditionExpression:
        "attribute_exists(bikeId) AND (attribute_not_exists(reservedBy) OR reservedBy = :b)",
      expressionAttributeValues: { ":b": bookingIdAttr },
      resultPath: sfn.JsonPath.DISCARD,
    });
    reserveBike
      .addRetry({ errors: TRANSIENT_DYNAMO_ERRORS, maxAttempts: 3 })
      .addCatch(markBookingFailed, { resultPath: "$.error" });

    const chargePayment = new tasks.LambdaInvoke(this, "ChargePayment", {
      lambdaFunction: chargeFn,
      payload: sfn.TaskInput.fromObject({
        bookingId: sfn.JsonPath.stringAt("$.bookingId"),
        bikeId: sfn.JsonPath.stringAt("$.bikeId"),
        customerId: sfn.JsonPath.stringAt("$.customerId"),
        amountCents: sfn.JsonPath.numberAt("$.amountCents"),
      }),
      resultSelector: { "status.$": "$.Payload.status" },
      resultPath: "$.payment",
    });
    chargePayment
      .addRetry({ errors: ["Lambda.TooManyRequestsException"], interval: Duration.seconds(1), maxAttempts: 3, jitterStrategy: sfn.JitterType.FULL })
      // Declined or unclear: nothing charged yet, or refundable idempotently.
      .addCatch(refundPayment, { resultPath: "$.error" });

    const confirmBooking = new tasks.DynamoUpdateItem(this, "ConfirmBooking", {
      table: bookings,
      key: { bookingId: bookingIdAttr },
      updateExpression: "SET #s = :confirmed",
      conditionExpression: "#s = :pending",
      expressionAttributeNames: { "#s": "status" },
      expressionAttributeValues: {
        ":confirmed": tasks.DynamoAttributeValue.fromString("CONFIRMED"),
        ":pending": tasks.DynamoAttributeValue.fromString("PENDING"),
      },
      resultPath: sfn.JsonPath.DISCARD,
    });
    confirmBooking
      .addRetry({ errors: TRANSIENT_DYNAMO_ERRORS, maxAttempts: 3 })
      .addCatch(refundPayment, { resultPath: "$.error" });

    const definition = createBooking
      .next(reserveBike)
      .next(chargePayment)
      .next(confirmBooking)
      .next(bookingConfirmed);

    const stateMachine = new sfn.StateMachine(this, "BikeBookingSaga", {
      definitionBody: sfn.DefinitionBody.fromChainable(definition),
      stateMachineType: sfn.StateMachineType.STANDARD,
      timeout: Duration.minutes(5),
      tracingEnabled: true,
    });

    new CfnOutput(this, "StateMachineArn", { value: stateMachine.stateMachineArn });
    new CfnOutput(this, "BikesTable", { value: bikes.tableName });
    new CfnOutput(this, "BookingsTable", { value: bookings.tableName });
  }
}

// bin/app.ts
import { App } from "aws-cdk-lib";
import { BikeBookingSagaStack } from "../lib/bike-booking-saga-stack.js";

const app = new App();
new BikeBookingSagaStack(app, "BikeBookingSagaStack");

Getting Retry and Catch right

Step Functions defines the evaluation order clearly: the service checks the Retry list first, and only when no retrier matches or the attempts are used up does the Catch list apply. Without a matching Catch, however, the whole execution fails. Therefore, I derive two rules from that.

Firstly, in forward steps you only retry errors where a second attempt has a chance of succeeding, such as throttling or timeouts. After all, a declined payment will not get friendlier on the third attempt. By the way, LambdaInvoke in the CDK adds a retrier for transient Lambda errors such as Lambda.ServiceException by default (six attempts, starting after two seconds, factor two). So you do not have to write it yourself. I covered how exponential backoff with jitter behaves in detail in this article.

Secondly, compensations may be retried more generously because they are idempotent. For RefundPayment, for example, I catch everything with States.ALL and try five times. If a compensation still fails for good, the execution ends in its own final state, CompensationFailed. That is the one case where a human has to take a look, and that is exactly why it should be distinguishable from a normally declined booking.

Failure case	What happens	Final state
Bike already taken	Catch at ReserveBike, booking becomes FAILED	BookingFailed
Payment declined	Refund (no-op), release bike, booking FAILED	BookingFailed
Confirmation fails	Refund, release bike, booking FAILED	BookingFailed
Refund fails for good	Compensation aborted, alarm needed	CompensationFailed

Semantic lock instead of isolation

A saga has no isolation like a database transaction. Two bookings can want the same bike at the same time. One answer is a semantic lock: the reservedBy field marks the bike as taken, and the condition in the ReserveBike step lets only one booking through. Consequently, the second one runs into a ConditionalCheckFailedException and is cleanly marked as failed. Similarly, the PENDING status on the booking is another such lock: other parts of the system can see that this process has not been decided yet.

The same caution applies in the other direction when releasing. ReleaseBike only removes the reservation if it belongs to its own booking. Otherwise, a failed saga could take the bike away from another, successful booking. If the condition fails, there is nothing to do, so the Catch moves straight on.

Deploying and testing the saga

Step 6: Synthesize and deploy

npx cdk synth --quiet
npx cdk deploy

After deployment, the stack outputs the ARN of the state machine and the table names. Store them in variables; the values below are placeholders:

SM_ARN="arn:aws:states:eu-central-1:123456789012:stateMachine:BikeBookingSaga..."
BIKES_TABLE="BikeBookingSagaStack-Bikes..."
BOOKINGS_TABLE="BikeBookingSagaStack-Bookings..."

# Create a bike that can be booked
aws dynamodb put-item --table-name "$BIKES_TABLE" \
  --item '{"bikeId":{"S":"bike-7"}}'

Step 7: The success case

aws stepfunctions start-execution \
  --state-machine-arn "$SM_ARN" \
  --name b-1001 \
  --input '{"bookingId":"b-1001","bikeId":"bike-7","customerId":"k-42","amountCents":4500}'

aws dynamodb get-item --table-name "$BOOKINGS_TABLE" \
  --key '{"bookingId":{"S":"b-1001"}}' --query "Item.status.S"

Expected result: "CONFIRMED". Also, I deliberately set the execution name to the booking ID. Step Functions does not allow a second start with the same name per state machine while the execution is running or still retained. That adds protection against duplicate bookings if the caller starts again after a timeout.

Step 8: Provoke failures

Now for the interesting part. Start a second booking for the same bike, which is still reserved, and a third one with an amount above the limit for a free bike:

# Bike already taken: ends in BookingFailed
aws stepfunctions start-execution --state-machine-arn "$SM_ARN" \
  --name b-1002 \
  --input '{"bookingId":"b-1002","bikeId":"bike-7","customerId":"k-43","amountCents":3000}'

# Payment declined: compensation runs, bike becomes free again
aws dynamodb put-item --table-name "$BIKES_TABLE" --item '{"bikeId":{"S":"bike-8"}}'
aws stepfunctions start-execution --state-machine-arn "$SM_ARN" \
  --name b-1003 \
  --input '{"bookingId":"b-1003","bikeId":"bike-8","customerId":"k-44","amountCents":50000}'

aws dynamodb get-item --table-name "$BIKES_TABLE" \
  --key '{"bikeId":{"S":"bike-8"}}'

Expected result: both executions show FAILED in the console with the error BookingFailed. In the graph view of b-1003, you can also see how the execution turns into the compensation chain after ChargePayment. Afterwards, bike bike-8 no longer has a reservedBy field, and the booking is set to FAILED.

Common mistakes

Catch without resultPath: Without it, the error output overwrites the entire input. The compensation then no longer knows the bookingId. That is why every Catch attaches the error under $.error.
Compensation with a hard condition: If ReleaseBike fails because the bike was never reserved, the saga gets stuck. Therefore, handle this case deliberately.
Retrying business errors: States.ALL in the Retry of a forward step also retries declined payments. As a result, it costs time and state transitions without changing anything.
Express instead of Standard: Asynchronously started Express workflows run steps at least once, and you only see their history through CloudWatch Logs. For sagas that involve money, I therefore use Standard workflows.

Cost and security

According to the AWS pricing page, Standard workflows cost 0.025 US dollars per 1,000 state transitions (US East), and 4,000 transitions per month are free. A successful booking in this example needs around five to six transitions, while a failed one needs a few more. At 100,000 bookings per month, you end up at roughly 15 US dollars for Step Functions, plus DynamoDB in on-demand mode and two Lambda invocations per booking. Retries count as additional transitions, which is one more reason to retry only where it makes sense. For the Lambda side, you can do the math with my Lambda cost calculator.

The CDK helps with security: the DynamoDB tasks automatically get IAM permissions only for their specific action and table, and the Lambda functions can only access the payments table. Real payment data does not belong in the execution input, since it is visible in the history. Instead, pass only IDs and amounts. And set up a CloudWatch alarm on failed executions with the error CompensationFailed, so stuck compensations do not go unnoticed.

My take: when the saga pattern is worth it

I consider the saga pattern one of the most important patterns in serverless architectures because it exposes an uncomfortable truth: distributed systems have no global undo. If you ignore that, you still end up building the compensations, just at night, by hand and under pressure. Step Functions, on the other hand, turns the flow into something you can read in a review, trace in the console and test on purpose.

Since late October, an AI agent can also start a saga like this and query its progress. How to set that up with clear limits is covered in Step Functions agent tools via AgentCore Gateway.

At the same time, I push back on the reflex to wrap every three-step flow in a saga. Every compensation is code that rarely runs and is therefore rarely tested. So if TransactWriteItems is enough, use it. If you would rather describe a flow in code than in a state machine, take a look at Lambda durable functions as an alternative. And if you are unsure which service fits which flow, the AWS service finder helps you sort it out.

FAQ

What is the difference between a saga and two-phase commit?

With two-phase commit, a coordinator locks all participants until everyone has agreed and then writes atomically. A saga commits every step locally right away and undoes it with a compensation when needed. It needs no shared locks, but in return it is only eventually consistent.

Do I always need Step Functions for a saga?

No. You can also build a saga through choreography with events via EventBridge or SNS, or in code with Lambda durable functions. Step Functions is a good fit when the flow has several steps and you want to see and monitor it in one place.

Standard or Express workflow for a saga?

For sagas with business consequences such as payments, I recommend Standard workflows. They run each step exactly once unless you configure retries, can run for up to a year and keep the execution history. Express workflows are cheaper for very many short flows, but depending on how they are invoked, they run steps at least once or at most once.

What happens if a compensation fails?

Then manual help is needed. Retry compensations generously, and if all attempts fail, end the execution in a dedicated error state with an alarm. That way, no half-rolled-back process goes unnoticed.

Your next step

Pick a flow in your system that currently spans two or more services and write down one line for each step: what is the compensation, and is it idempotent? If you cannot come up with a counter-action for a step, you have already gained the most important insight before writing a single line of code.

This wraps up the series “Serverless patterns in practice” for now. Which pattern are you still missing? Let me know in the comments, and it might become part seven.

More in this series: Overview “Serverless patterns in practice” · Previous part: Strangler Fig Pattern: Migrate to Lambda Step by Step · Next part: DynamoDB TTL: build seat holds that expire

How to check the retry and catch rules of such a saga automatically, without deploying it, is what I show in Test Step Functions with the TestState API.

Sources

AWS Prescriptive Guidance: Saga orchestration pattern
Step Functions Developer Guide: Handling errors (Retry and Catch)
Step Functions Developer Guide: DynamoDB integration
AWS Step Functions: Pricing
Chris Richardson, microservices.io: Pattern Saga
Tested on 10/28/2026 with Node.js 22.23, TypeScript 7.0.2, aws-cdk-lib 2.275.0 and AWS SDK v3.1158: tsc, 5 tests and cdk synth, including a check of the generated Retry and Catch rules. ARNs and table names in the commands are placeholders.

This post was created with AI assistance.

👉 Next step: Working with AWS CDK? Grab my printable AWS CDK cheat sheet with commands, concepts and a stack skeleton.

Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en

Strangler Fig Pattern: Migrate to Lambda Step by Step

Sascha Mirow-Stanitzok — Sat, 31 Oct 2026 14:56:01 +0000

For me, the strangler fig pattern is the most honest answer to the question “How do we get off our legacy system without stopping everything?”. After all, almost every team has a candidate like that: an application that has been running for years, that nobody likes to touch, and that still earns money every single day. However, the reflex is often a big rewrite. In my experience, that rewrite surprisingly often ends with two systems that both need maintenance.

So in this fifth part of the series, I show you the other route: you put a facade in front of the legacy system and move one feature after another to Lambda. Every step is small, and every step can be rolled back. The example is a made-up bike rental called “Radhafen”, built with the Serverless Framework and TypeScript.

In short

With the strangler fig pattern, every request passes through a facade, here an HTTP API in API Gateway. One route after another moves to Lambda, and everything else keeps flowing to the legacy system through the $default route. The biggest stumbling block is rarely the routes. It is the data that old and new both need.

What you will learn in this article

How the strangler fig pattern works and when it pays off
Why an HTTP API with a $default route makes a good facade for a legacy system
Comparing a feature with Lambda safely using a parallel run
Switching over and rolling back with a single parameter
Which mistakes around data, host names and costs are typical

What is the strangler fig pattern?

The name comes from Martin Fowler. He saw strangler figs in the rain forests of Queensland: plants that sprout high up in a tree, send their roots down along the trunk and wrap around their host over the years. Eventually, the fig stands on its own and the old tree is gone. Applied to software, this means you build the new system around the old one and gradually take over its responsibilities until you can switch it off.

Fowler's reasoning is rather sober. A full replacement often fails because nobody can fully describe the existing behavior, because the business won't wait two years for new features, and because part of the old behavior isn't needed anymore anyway. He sums it up in one sentence: “the reduced risk and earlier value from the gradual approach outweigh its costs.” So the gradual route costs extra, but it pays off through lower risk and earlier benefits.

The three building blocks

Technically, you need three things. First, a facade: a proxy that every request passes through and that decides whether the old or the new system answers. Second, a seam: a spot in the legacy system where a feature can be cut out cleanly. Third, transitional architecture, meaning code you only need during the migration and delete afterwards. That includes, for example, a comparison mode or an adapter that keeps data in sync between old and new.

This transitional code feels like waste at first. I see it more like scaffolding: nobody would renovate a house without scaffolding just because it gets taken down at the end.

Four phases: facade, shadow comparison, live switch, retiring the legacy system.

When does the strangler fig pattern pay off?

Not every migration needs this pattern. For a small application with three endpoints, a rewrite over a weekend is often faster than any facade. The AWS guide to cloud design patterns agrees: the bigger and more tangled the legacy system, the more the gradual approach is worth.

In practice, the pattern fits well if at least two of these points apply: the legacy system is large and poorly documented. Also, the business keeps needing new features while you migrate. Outages or a “freeze” lasting several months are not an option. The external interface is HTTP, so you can put a proxy in front of it. It gets hard, on the other hand, when the legacy system mainly talks to other systems through a shared database. Then there is no door to put the facade in front of.

What about Refactor Spaces?

With Migration Hub Refactor Spaces, AWS had a service that built exactly this kind of facade for you. According to the AWS guide, Refactor Spaces has not been open to new customers since November 7, 2025. As an alternative, AWS points to AWS Transform there. The good news: the facade itself is so little infrastructure that you can build it yourself with a few lines in the Serverless Framework. That's exactly what we'll do now.

The example: a bike rental with a legacy system

Radhafen rents out city bikes, e-bikes and cargo bikes. Its application runs as a single Node application on one server. It can list stations, create reservations, write invoices and calculate prices. It is reachable at https://legacy.radhafen.example.

As the first seam, I pick price calculation, so POST /quotes. In fact, it is a textbook candidate: it doesn't read data from other modules, writes nothing, gets called often and is easy to test. Reservations, in contrast, depend on stations, payments and invoices. They come much later, once the facade has proven itself.

My rule of thumb for the order: first features without side effects and with their own data, then new features built directly in the new system, and core processes with shared data at the very end. That way you get to know the facade before a mistake gets expensive.

How does the HTTP API become a facade?

The facade is an HTTP API in API Gateway. For our purpose, two rules from the AWS documentation matter. First, the $default route catches all requests that don't match any other route and passes the full path on to the integration. Second, the most specific route always wins: an exact match like POST /quotes beats a route with {proxy+}, and that one beats $default.

In other words, that's the whole pattern. The $default route points to the legacy system through an HTTP proxy integration. In addition, every feature you move gets its own route to a Lambda function. Anything without its own route automatically ends up at the legacy system. If you delete a route again, traffic falls back just as automatically. That's your built-in emergency exit.

🎬 Strangler fig pattern: routing through the facade – this animated diagram only runs on my site: watch it on sascha-ms.de

In case you're wondering why I use an HTTP API here and not a REST API: for plain forwarding, HTTP APIs are cheaper and simpler. I covered the differences in detail in REST API vs HTTP API: Choosing API Gateway.

Step by step: implementing the strangler fig pattern with Lambda

Prerequisites

Node.js 22 and an AWS account with permissions for Lambda, API Gateway, CloudFormation and CloudWatch Logs
Serverless Framework v4 (npm i -g serverless); if you are still on v3, Migrate to Serverless Framework v4 will help
A legacy system that is publicly reachable over HTTPS (for private systems, see the section on security)
Basic TypeScript knowledge

1. Create the project

The project is deliberately small. It needs no runtime dependencies because Node.js 22 ships with fetch. The tests run on Node's built-in test runner.

mkdir -p radhafen-strangler/src radhafen-strangler/test
cd radhafen-strangler
# create package.json and tsconfig.json as shown below, then:
npm install

{
  "name": "strangler-fig-lambda",
  "version": "1.0.0",
  "private": true,
  "type": "module",
  "scripts": {
    "typecheck": "tsc --noEmit",
    "test": "node --experimental-strip-types --test test/*.test.ts",
    "deploy": "serverless deploy"
  },
  "devDependencies": {
    "@types/aws-lambda": "^8.10.164",
    "@types/node": "^22.20.5",
    "typescript": "^7.0.2"
  }
}

{
  "compilerOptions": {
    "target": "ES2023",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "strict": true,
    "noEmit": true,
    "allowImportingTsExtensions": true,
    "verbatimModuleSyntax": true,
    "erasableSyntaxOnly": true,
    "skipLibCheck": true,
    "types": ["node"]
  },
  "include": ["src", "test"]
}

2. The facade in the Serverless Framework

For every function with an httpApi event, the Serverless Framework automatically creates an HTTP API. Its logical CloudFormation ID is HttpApi. The framework can't create an integration to an arbitrary URL directly; according to the docs, httpApi only offers the proxy integration to Lambda. That's why we add the integration and the $default route as plain CloudFormation resources. We pass the target URL and the mode as parameters, so you can switch later without changing code.

# serverless.yml
service: radhafen-strangler
frameworkVersion: '4'

params:
  default:
    legacyBaseUrl: https://legacy.radhafen.example
    migrationMode: shadow

provider:
  name: aws
  runtime: nodejs22.x
  architecture: arm64
  region: eu-central-1
  memorySize: 256
  timeout: 10
  logRetentionInDays: 14
  environment:
    LEGACY_BASE_URL: ${param:legacyBaseUrl}
    MIGRATION_MODE: ${param:migrationMode}

functions:
  quotes:
    handler: src/quote-handler.handler
    events:
      - httpApi:
          method: POST
          path: /quotes

resources:
  Resources:
    # Everything without its own route goes to the legacy system unchanged.
    LegacyIntegration:
      Type: AWS::ApiGatewayV2::Integration
      Properties:
        ApiId: !Ref HttpApi
        IntegrationType: HTTP_PROXY
        IntegrationMethod: ANY
        IntegrationUri: ${param:legacyBaseUrl}
        PayloadFormatVersion: '1.0'
        TimeoutInMillis: 29000
    LegacyDefaultRoute:
      Type: AWS::ApiGatewayV2::Route
      Properties:
        ApiId: !Ref HttpApi
        RouteKey: $default
        Target: !Join ['/', ['integrations', !Ref LegacyIntegration]]

Two details matter. First, HTTP proxy integrations use payload format 1.0, which is why it is set explicitly. Second, the integration timeout is 29 seconds, because an HTTP API doesn't allow more than 30 seconds anyway. If your legacy system has endpoints that take longer, you'll find out now at the latest. For those cases, take a look at the job pattern with 202 and SQS from part 1.

3. The new pricing logic

The new logic is a pure function without network access. That makes it easy to test and keeps it cleanly separated from the transitional code. I calculate amounts in cents to avoid floating-point rounding errors.

// src/tariff.ts
// New pricing logic for the “Radhafen” bike rental – a pure function without I/O.

export type BikeType = 'city' | 'ebike' | 'cargo';

export interface QuoteRequest {
  bikeType: BikeType;
  hours: number;
  member: boolean;
}

export interface Quote {
  bikeType: BikeType;
  hours: number;
  totalCents: number;
  currency: 'EUR';
}

// Prices in cents to avoid floating-point rounding errors.
const HOURLY_CENTS: Record<BikeType, number> = { city: 300, ebike: 600, cargo: 800 };
const DAY_CAP_CENTS: Record<BikeType, number> = { city: 1800, ebike: 3500, cargo: 4500 };
const MEMBER_DISCOUNT = 0.1;

export function parseQuoteRequest(body: unknown): QuoteRequest | string {
  if (typeof body !== 'object' || body === null) return 'Body must be a JSON object';
  const b = body as Record<string, unknown>;
  if (b.bikeType !== 'city' && b.bikeType !== 'ebike' && b.bikeType !== 'cargo') {
    return 'bikeType must be city, ebike or cargo';
  }
  if (typeof b.hours !== 'number' || !Number.isInteger(b.hours) || b.hours < 1 || b.hours > 72) {
    return 'hours must be an integer from 1 to 72';
  }
  return { bikeType: b.bikeType, hours: b.hours, member: b.member === true };
}

export function computeQuote(req: QuoteRequest): Quote {
  const fullDays = Math.floor(req.hours / 24);
  const restHours = req.hours % 24;
  const restCents = Math.min(restHours * HOURLY_CENTS[req.bikeType], DAY_CAP_CENTS[req.bikeType]);
  let total = fullDays * DAY_CAP_CENTS[req.bikeType] + restCents;
  if (req.member) total = Math.round(total * (1 - MEMBER_DISCOUNT));
  return { bikeType: req.bikeType, hours: req.hours, totalCents: total, currency: 'EUR' };
}

4. Parallel run: comparing old and new

Now comes the part I find most valuable in migrations. Before the new logic serves real answers, it runs along in shadow mode. A parallel run means: the Lambda function still asks the legacy system, also calculates the result itself and compares both. The customer gets the legacy answer, and you get a log entry with every difference.

In this way, you find the rules nobody ever wrote down. Maybe the legacy system rounds differently, maybe the daily cap only applies from the eighth hour. A parallel run shows you this with real traffic, without anyone seeing a wrong price. There is one limitation: it only works for requests without side effects. You must not create a reservation twice just to compare.

// src/compare.ts
// Compares the legacy response with the new calculation (parallel run).

export interface Difference {
  field: string;
  legacy: unknown;
  next: unknown;
}

export function diffResults(legacy: Record<string, unknown>, next: Record<string, unknown>): Difference[] {
  const fields = new Set([...Object.keys(legacy), ...Object.keys(next)]);
  const diffs: Difference[] = [];
  for (const field of fields) {
    if (JSON.stringify(legacy[field]) !== JSON.stringify(next[field])) {
      diffs.push({ field, legacy: legacy[field], next: next[field] });
    }
  }
  return diffs;
}

// src/legacy-client.ts
// Small client for the legacy system. The timeout is deliberately below the Lambda timeout.

export interface LegacyResponse {
  status: number;
  body: unknown;
}

export async function callLegacy(
  baseUrl: string,
  path: string,
  payload: unknown,
  timeoutMs = 3000,
): Promise<LegacyResponse> {
  const res = await fetch(new URL(path, baseUrl), {
    method: 'POST',
    headers: { 'content-type': 'application/json', 'x-strangler-shadow': '1' },
    body: JSON.stringify(payload),
    signal: AbortSignal.timeout(timeoutMs),
  });
  const text = await res.text();
  let body: unknown = text;
  try {
    body = JSON.parse(text);
  } catch {
    // Not JSON – pass the text through unchanged.
  }
  return { status: res.status, body };
}

The client also sets the x-strangler-shadow header. As a result, the legacy system can recognize these calls in its own metrics. The three-second timeout is deliberately well below the Lambda timeout, so the function can respond cleanly instead of being cut off hard. If the legacy system hangs regularly, the circuit breaker from part 4 is the right addition.

5. The handler with shadow and live mode

// src/quote-handler.ts
// POST /quotes – the first feature extracted from the legacy system.
import type { APIGatewayProxyEventV2, APIGatewayProxyStructuredResultV2 } from 'aws-lambda';
import { computeQuote, parseQuoteRequest } from './tariff.ts';
import { diffResults } from './compare.ts';
import { callLegacy } from './legacy-client.ts';

type Mode = 'shadow' | 'live';

const json = (statusCode: number, body: unknown): APIGatewayProxyStructuredResultV2 => ({
  statusCode,
  headers: { 'content-type': 'application/json' },
  body: JSON.stringify(body),
});

export async function handler(event: APIGatewayProxyEventV2): Promise<APIGatewayProxyStructuredResultV2> {
  const mode: Mode = process.env.MIGRATION_MODE === 'live' ? 'live' : 'shadow';
  const legacyBaseUrl = process.env.LEGACY_BASE_URL ?? '';

  let raw: unknown;
  try {
    raw = JSON.parse(event.body ?? '');
  } catch {
    return json(400, { message: 'Invalid JSON' });
  }
  const parsed = parseQuoteRequest(raw);
  if (typeof parsed === 'string') return json(400, { message: parsed });

  const next = computeQuote(parsed);
  if (mode === 'live') {
    console.log(JSON.stringify({ msg: 'quote-served', source: 'lambda', mode }));
    return json(200, next);
  }

  // Shadow mode: the legacy system stays the source of truth, the new logic only runs for comparison.
  try {
    const legacy = await callLegacy(legacyBaseUrl, '/quotes', raw);
    if (legacy.status !== 200 || typeof legacy.body !== 'object' || legacy.body === null) {
      console.log(JSON.stringify({ msg: 'parallel-run', match: false, reason: 'legacy-status', status: legacy.status }));
      return json(legacy.status, legacy.body);
    }
    const diffs = diffResults(legacy.body as Record<string, unknown>, { ...next });
    console.log(JSON.stringify({ msg: 'parallel-run', match: diffs.length === 0, diffs }));
    return json(200, legacy.body);
  } catch (err) {
    console.log(JSON.stringify({ msg: 'parallel-run', match: false, reason: 'legacy-error', error: String(err) }));
    return json(502, { message: 'Legacy system unreachable' });
  }
}

In shadow mode, the legacy system stays the source of truth. In live mode, only Lambda answers and the legacy system is no longer called for this route. You switch via the MIGRATION_MODE environment variable, which comes from the migrationMode parameter.

6. Testing

The tests replace fetch with a stub. That lets you check all three paths without a real legacy system: a difference in shadow mode, live mode without a legacy call, and the behavior when the legacy system doesn't answer.

// test/tariff.test.ts
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { computeQuote, parseQuoteRequest } from '../src/tariff.ts';

test('hourly price for a city bike', () => {
  assert.equal(computeQuote({ bikeType: 'city', hours: 3, member: false }).totalCents, 900);
});

test('daily cap applies from the seventh hour', () => {
  assert.equal(computeQuote({ bikeType: 'city', hours: 7, member: false }).totalCents, 1800);
});

test('members get a 10 percent discount', () => {
  assert.equal(computeQuote({ bikeType: 'ebike', hours: 26, member: true }).totalCents, 4230);
});

test('invalid input is rejected', () => {
  assert.match(String(parseQuoteRequest({ bikeType: 'tandem', hours: 2 })), /bikeType/);
  assert.match(String(parseQuoteRequest({ bikeType: 'city', hours: 0 })), /hours/);
});

// test/quote-handler.test.ts
import { test, beforeEach, mock } from 'node:test';
import assert from 'node:assert/strict';
import type { APIGatewayProxyEventV2 } from 'aws-lambda';
import { handler } from '../src/quote-handler.ts';
import { diffResults } from '../src/compare.ts';

const event = (body: unknown) => ({ body: JSON.stringify(body) }) as APIGatewayProxyEventV2;
const logs: string[] = [];

beforeEach(() => {
  logs.length = 0;
  mock.restoreAll();
  mock.method(console, 'log', (line: string) => logs.push(line));
  process.env.LEGACY_BASE_URL = 'https://legacy.radhafen.example';
});

test('shadow mode returns the legacy response and reports differences', async () => {
  process.env.MIGRATION_MODE = 'shadow';
  mock.method(globalThis, 'fetch', async () =>
    new Response(JSON.stringify({ bikeType: 'city', hours: 7, totalCents: 2100, currency: 'EUR' }), { status: 200 }),
  );
  const res = await handler(event({ bikeType: 'city', hours: 7 }));
  assert.equal(res.statusCode, 200);
  assert.equal(JSON.parse(res.body!).totalCents, 2100);
  const log = JSON.parse(logs[0]);
  assert.equal(log.match, false);
  assert.deepEqual(log.diffs, [{ field: 'totalCents', legacy: 2100, next: 1800 }]);
});

test('live mode no longer calls the legacy system', async () => {
  process.env.MIGRATION_MODE = 'live';
  const fetchMock = mock.method(globalThis, 'fetch', async () => new Response('{}'));
  const res = await handler(event({ bikeType: 'cargo', hours: 2, member: true }));
  assert.equal(res.statusCode, 200);
  assert.equal(JSON.parse(res.body!).totalCents, 1440);
  assert.equal(fetchMock.mock.callCount(), 0);
});

test('unreachable legacy system returns 502', async () => {
  process.env.MIGRATION_MODE = 'shadow';
  mock.method(globalThis, 'fetch', async () => {
    throw new Error('timeout');
  });
  const res = await handler(event({ bikeType: 'ebike', hours: 1 }));
  assert.equal(res.statusCode, 502);
});

test('invalid JSON returns 400', async () => {
  const res = await handler({ body: '{broken' } as APIGatewayProxyEventV2);
  assert.equal(res.statusCode, 400);
});

test('diffResults only finds real differences', () => {
  assert.deepEqual(diffResults({ a: 1, b: [1, 2] }, { a: 1, b: [1, 2] }), []);
  assert.equal(diffResults({ a: 1 }, { a: 1, c: 'new' }).length, 1);
});

npm run typecheck
npm test

Expected result: tsc reports nothing, and the test runner shows # pass 9 and # fail 0.

7. Deploy and evaluate shadow mode

serverless deploy --param="legacyBaseUrl=https://legacy.radhafen.example" --param="migrationMode=shadow"
# Example calls (use the URL from the deploy output)
curl https://abc123.execute-api.eu-central-1.amazonaws.com/stations
curl -X POST https://abc123.execute-api.eu-central-1.amazonaws.com/quotes \
  -H "content-type: application/json" \
  -d '{"bikeType":"ebike","hours":3,"member":true}'

After the deployment, the framework prints the URL of the HTTP API. A call to /stations ends up at the legacy system, a POST /quotes at Lambda, which still returns the old answer in shadow mode. For the comparison, open CloudWatch Logs Insights, pick the function's log group and count matches and differences:

filter @message like /"msg":"parallel-run"/
| parse @message '"match":*,' as match
| stats count(*) as total by match

Let this run for a few days with real traffic, ideally across a weekend and a month-end. Every difference is either a bug in the new logic or a rule in the legacy system that you deliberately want to keep or drop. That decision belongs to the business side, not to the development team alone.

8. Switch over and roll back

Once the differences are explained, you switch. If something goes wrong after all, the way back is the same command with the old value. For emergencies there is a second brake: if you comment out the function's httpApi event and deploy, the route disappears and traffic falls back to the legacy system through $default.

# Switch over: Lambda answers POST /quotes on its own
serverless deploy --param="legacyBaseUrl=https://legacy.radhafen.example" --param="migrationMode=live"
# Roll back: return to shadow mode
serverless deploy --param="legacyBaseUrl=https://legacy.radhafen.example" --param="migrationMode=shadow"

Only when the route has been stable in live mode for a few weeks do you delete the shadow code and the endpoint in the legacy system. That's part of the transitional architecture. If you forget this step, you end up with three variants of the same logic.

The hard half: data in the strangler fig pattern

Price calculation was simple on purpose. As soon as you move features that write data, things get serious. For example, the reservation in the new system may need stations from the old database, and the old invoicing feature may need reservations from the new one. Three tools have proven themselves for this phase.

The first is an anti-corruption layer. It's an adapter that translates the terms and data formats of the legacy system into those of the new one, so old models don't leak into the new code. The AWS guide describes it as a facade inside the legacy system through which existing callers reach the new service without knowing about it. It gets deleted once nobody needs it anymore.

The second is synchronization through events. When the new service changes its data, it publishes an event, and a small synchronizer writes the change into the old database. That means you deliberately accept eventual consistency: for a short moment, old and new see different states. How to produce such events reliably without losing one is shown by the outbox pattern from part 2.

The third is more of a ban: no dual writes. If a function writes to the old and the new database at the same time, sooner or later one write will fail and the other won't. Then you have two truths and no way to merge them again. For every data area, define exactly one owner at any point in time.

Typical mistakes and how to avoid them

Absolute URLs and redirects

Many legacy systems build links and redirects from the host name they see in the request. Behind the facade, that can suddenly be the internal name, and your users land on legacy.radhafen.example after logging in. So before switching over, test logins, redirects, cookies and emails with links on purpose. It's boring, and it saves you exactly the evening when everything would otherwise be on fire.

First steps that are too big

It's tempting to move the entire reservation module right away because it “has to be rebuilt anyway”. However, that throws away the advantage of the pattern. A good first cut goes live within one or two sprints and can be rolled back with a single command.

The facade as the next monolith

A facade should route, not calculate. As soon as business logic, data transformation or special cases for individual customers end up there, you're building your next monolith, this time in YAML. In short, logic belongs in the Lambda functions, and the facade only knows paths.

No end in sight

The underestimated risk is organizational. After the first three quick wins, the migration becomes a never-ending task, and the legacy system lives on for another five years. Besides the technical side, Fowler explicitly says the organization has to change too, so this work is sustained over time. That's why I would set a visible goal from day one, for example the share of requests the legacy system still receives by the end of a given quarter.

Cost, latency and security

Once the facade is active, every request passes through API Gateway, including those the legacy system still answers. So you pay for traffic that used to go only through your load balancer, and every call gets an extra network hop. Therefore, do a rough calculation of both before you start. For the Lambda part, the Lambda cost calculator helps; you'll find API Gateway prices on the AWS pricing page.

In shadow mode, you also double the load for the migrated feature, because the legacy system keeps calculating and Lambda calculates on top. For cheap price calculation, that doesn't matter. For expensive queries, you should only compare a share of requests, say every tenth one.

On security: an HTTP proxy integration needs a publicly reachable target. If your legacy system runs in a private network, use a private integration via a VPC link to an Application Load Balancer, Network Load Balancer or Cloud Map service instead. In both cases, make sure nobody can bypass the facade and call the legacy system directly. Otherwise you end up with two entrances, and only one of them has your new protections, such as the Lambda authorizer for HTTP APIs.

My take

I consider the strangler fig pattern one of the few architecture patterns that is mainly a leadership tool. Above all, it makes progress visible, spreads risk across many small decisions and gives the team its first win after just a few weeks. For motivation, that often matters more than any target architecture on a slide.

Where I disagree is the common idea that you need to know the perfect target architecture up front. In fact, the first route teaches you more about your legacy system than three workshops. What I would clarify beforehand, though, is the data question: which area belongs to whom, and when? Leave that open, and you'll migrate the routes quickly and then get stuck on the data for a year.

FAQ

Where does the name strangler fig pattern come from?

A strangler fig is a plant that grows around a tree and eventually replaces it. The pattern describes replacing a legacy system gradually, with the new system taking over one feature at a time.

Do I need microservices for the strangler fig pattern?

No. The target can also be a modular monolith or a handful of Lambda functions. What matters is the facade that routes between old and new.

What is the difference between strangler fig and big bang?

With a big bang, the new system is built completely and switched over on a fixed date. With the strangler fig pattern, both systems run in parallel, and each part is moved and can be rolled back on its own.

Does this also work with a REST API or a load balancer?

Yes. REST APIs, an Application Load Balancer with path rules or CloudFront with multiple origins can all serve as the facade. The HTTP API is just particularly lean for getting started.

Your next step

Take your own legacy system and look for the one feature that writes no data, gets called often and is easy to test. That's your first seam. Next, build the facade with just this one route in shadow mode and watch the differences for a week. After that, you'll know more about your legacy system than after any architecture workshop.

Over to you: which feature in your legacy system would you extract first, and what has been holding you back so far? Let me know in the comments.

More in this series: Overview “Serverless patterns in practice” · Previous part: Lambda circuit breaker · Next part: Saga pattern with Step Functions

Sources

Martin Fowler: Strangler Fig
AWS Prescriptive Guidance: Strangler fig pattern
AWS docs: Routes for HTTP APIs ($default, priority)
AWS docs: HTTP proxy integrations for HTTP APIs
AWS docs: Quotas for HTTP APIs
Serverless Framework: HTTP API events
Tested on October 28, 2026 with Node.js 22.23, TypeScript 7.0.2 and @types/aws-lambda 8.10.164: tsc and the 9 tests from the article. The CloudFormation resources generated for the facade were checked with serverless package. The URL in the curl examples is a placeholder.

This post was created with AI assistance.

👉 Next step: Curious what your own Lambda functions cost? Try my Lambda cost calculator – it runs right in your browser.

Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en

Secure AWS Automation with GitHub Actions and OpenID Connect (OIDC)

Narendra | SRE — Sat, 31 Oct 2026 14:47:51 +0000

Secure AWS Automation with GitHub Actions and OpenID Connect (OIDC)

Level: Intermediate

Focus: AWS IAM, temporary credentials, OpenID Connect, GitHub Actions, least privilege.

Implementation status: This is a reference implementation. It has not been deployed or validated in a live AWS account. Validate the provider, trust policy, permissions, and workflow in a dedicated test environment before relying on it.

Originally published on AWS Builder Center

Source code, Terraform configuration, and architecture diagrams GitHub Repository

Introduction

CI/CD pipelines often need access to cloud resources: publishing an artifact, deploying an application, or updating infrastructure. A common shortcut is to create an IAM user's access key and store it as a CI/CD secret. These credentials are reusable until rotated or revoked. If exposed through a compromised runner, workflow, or accidental logging, they may be used outside the pipeline.

AWS supports a different pattern: a workload proves its identity using OpenID Connect (OIDC), then requests temporary AWS credentials for an IAM role. This article explains that pattern using GitHub Actions and AWS Security Token Service (AWS STS).

The objective is not just to remove access keys. It is to establish a narrowly scoped trust relationship between one repository/branch and one AWS role, then grant that role only the permissions the workflow needs.

Figure 1. GitHub Actions obtains a short-lived role session through OIDC instead of using a stored long-lived AWS access key.

1. Why long-lived access keys are risky

Static IAM user access keys can remain valid for a long time. They may be copied into workflows, developer machines, or secret stores. Rotation becomes an operational task, and every copy must be updated safely.

Common risks include accidental commits, credentials printed by workflows, compromised runners, keys remaining active after a project ends, and broad permissions increasing the impact of exposure.

Secret storage is important for credentials that must exist, but it does not change the fact that a static access key is reusable. Where workload federation is supported, short-lived credentials are usually a better fit.

2. The building blocks

Component	Responsibility
GitHub Actions	Runs the automation workflow.
GitHub OIDC issuer	Issues a signed identity token for the workflow context.
AWS IAM OIDC provider	Establishes trust in the token issuer and audience.
IAM role trust policy	Defines which federated identities may assume the role.
AWS STS	Validates the token and issues temporary credentials.
IAM permissions policy	Defines what the assumed role can do in AWS.

Two policy types must be understood separately:

Trust policy: Who can assume this role?
Permissions policy: What can the role do after it is assumed?

A restrictive trust policy does not compensate for an overly broad permissions policy, and vice versa.

3. How the OIDC flow works

Figure 2. AWS validates the token issuer and audience, evaluates the role trust conditions, and STS issues a temporary session.

A GitHub Actions job starts.
The workflow is allowed to request an OIDC token by declaring id-token: write.
GitHub issues a signed token describing the workflow identity.
The AWS credentials action presents the token to AWS STS using AssumeRoleWithWebIdentity.
AWS validates the token against the configured GitHub OIDC provider and evaluates the IAM role trust policy.
If authorized, STS returns temporary credentials.
The workflow uses those credentials to call permitted AWS APIs. The session expires according to the role session duration.

id-token: write permits the job to request an OIDC token; it does not itself grant AWS permissions. AWS authorization still depends on the role trust policy and attached permissions.

4. Configure the GitHub OIDC provider in AWS

An IAM OIDC identity provider must exist in the AWS account before a role can trust it.

Provider URL: https://token.actions.githubusercontent.com
Client ID (audience): sts.amazonaws.com

Create the provider in IAM using the AWS Console or your approved infrastructure-as-code process. Confirm current AWS and GitHub documentation for provider setup and certificate/thumbprint handling rather than copying an old thumbprint from an example.

The Terraform example in this project accepts the ARN of an already-created GitHub OIDC provider as an input. This keeps provider lifecycle management separate from the example role. Do not create duplicate providers for the same issuer in one account; reuse an existing provider where appropriate.

5. Restrict the role trust policy

For a workflow running from the main branch of one repository, the expected subject claim has this form:

repo:OWNER/REPOSITORY:ref:refs/heads/main

The Terraform example builds the subject from input variables and requires an exact match. It also checks that the audience is sts.amazonaws.com.

The trust policy should allow the configured GitHub OIDC provider as the federated principal, the sts:AssumeRoleWithWebIdentity action, the expected audience, and the exact repository-and-branch subject.

Avoid broad conditions such as repo:OWNER/* unless every repository in that scope is intentionally trusted. A wildcard can allow an unintended repository to assume the role.

Branches and GitHub Environments

The subject format changes when a workflow uses a GitHub Environment. The subject can identify the environment instead of the branch. If you add an environment such as production, update the trust policy to match the exact subject format and configure environment protection rules. Do not copy a branch-based trust policy unchanged into an environment-based workflow.

6. Grant only the permissions the workflow needs

The example role has a deliberately small read-only S3 policy for one bucket and the reports/ prefix:

s3:ListBucket on the specific bucket, constrained by the s3:prefix condition.
s3:GetObject on objects under reports/ in that bucket.

It does not grant bucket creation, object writes, deletion, IAM administration, or access to every S3 bucket.

For a real pipeline, start with the exact API operations required. Scope resource ARNs wherever the service supports resource-level permissions. Some actions require "Resource": "*"; document why and consider applicable condition keys. Do not attach AdministratorAccess just to make a workflow succeed.

7. Configure the GitHub Actions workflow

The sample workflow is in .github/workflows/oidc-demo.yml. It runs on pushes to main and can also be started manually.

Its important permissions are:

permissions:
  contents: read
  id-token: write

The workflow configures AWS credentials using a role ARN and Region, then runs:

aws sts get-caller-identity

This confirms which AWS identity the job is using. It then lists the approved S3 prefix as a permissions demonstration.

Store the role ARN and bucket name as GitHub Actions repository variables. The role ARN is an identifier, not a secret. The workflow must not contain an AWS access key ID or secret access key.

For production, pin third-party GitHub Actions to a reviewed full commit SHA rather than relying only on a mutable major-version tag. Review action updates through your normal dependency-management process.

8. Security considerations

OIDC reduces the need for long-lived credentials, but it does not make a pipeline automatically secure.

Restrict the subject to the required repository and branch or protected environment.
Protect the trusted branch with reviews and required status checks.
Review workflow changes: someone who can modify a trusted workflow may change what the role does.
Use least privilege for AWS actions and resources.
Grant id-token: write only to jobs that need federation.
Protect sensitive environments with required reviewers and deployment rules.
Pin third-party actions to reviewed immutable references.
Never print tokens, credentials, or sensitive deployment output.
Use CloudTrail to review role-assumption and API activity.
Choose a session duration that is sufficient but not unnecessarily long.

A compromised workflow running in a trusted repository and branch may still obtain the role. OIDC establishes workload identity; repository governance and workflow security remain essential.

9. Troubleshooting

Symptom	Possible cause	What to check
`Not authorized to perform sts:AssumeRoleWithWebIdentity`	Trust policy does not match token claims	Provider ARN, audience, exact `sub` condition.
`No OpenIDConnect provider found`	Provider missing or ARN/issuer incorrect	Confirm provider exists in the same AWS account.
Audience mismatch	Trust policy expects another audience	Confirm `sts.amazonaws.com` is configured.
Subject mismatch	Branch, repository, or environment subject differs	Check workflow context and exact condition.
S3 `AccessDenied`	Role assumed, but permissions are insufficient	Check attached policy, bucket policy, prefix, and organization controls.
Workflow cannot request OIDC token	`id-token: write` missing or overridden	Check workflow/job permissions.

Run aws sts get-caller-identity after configuring credentials. If it succeeds, role assumption worked. An access-denied error on a later S3 command is a separate permissions issue.

10. Cost considerations

OIDC federation does not require a continuously running compute service. However, AWS API usage and the resources your workflow operates can incur charges. GitHub Actions usage is subject to GitHub's plan and runner limits. Check current pricing and usage policies before scaling automation.

11. Validation checklist

This project has not yet been deployed or tested. In a dedicated non-production account:

Confirm the GitHub OIDC provider exists in the intended AWS account.
Review the Terraform plan and confirm the exact repository/branch subject.
Apply only after reviewing the plan and permissions.
Configure AWS_ROLE_ARN and DEMO_BUCKET as GitHub repository variables.
Trigger the workflow from the trusted main branch.
Confirm aws sts get-caller-identity reports the expected assumed role.
Confirm listing the allowed reports/ prefix succeeds.
Confirm reading an object outside the allowed prefix is denied.
Confirm an untrusted branch or repository cannot assume the role.
Review CloudTrail events and remove test resources when finished.

Do not claim the workflow is working until these checks have actually been performed. Use a dedicated test bucket and non-sensitive sample objects.

12. When should you use this pattern?

Use OIDC federation when an external workload such as GitHub Actions needs AWS access and supports issuing a trusted identity token. Examples include CI pipelines that publish artifacts, deployment workflows, and infrastructure automation running Terraform against AWS.

The trust conditions and permissions should differ by workload. A build job that uploads an artifact should not automatically receive the same role as a production deployment job.

Conclusion

Long-lived access keys are not the only way for automation to authenticate to AWS. With OIDC federation, GitHub Actions can present a signed workload identity to AWS STS and receive temporary credentials for a narrowly scoped IAM role.

The strongest design combines a precise trust policy, least-privilege permissions, protected source branches, reviewed workflow changes, and CloudTrail visibility. OIDC reduces credential-management risk, but it does not replace secure CI/CD practices.

The central lesson is to treat the pipeline as an identity with a defined trust boundary: decide exactly which workload may assume the role, what it may do, and how its activity will be reviewed.

References

Author note before publication: Add a short explanation in your own voice about why you chose OIDC federation and what you learned about the difference between a role trust policy and a permissions policy. After completing the validation checklist, replace the implementation-status note with the actual tests performed and their results.

Secure SaaS File Uploads: Signed URLs, Scanning, Isolation

Muhammad Ali Mughal — Sat, 31 Oct 2026 14:30:41 +0000

A file upload endpoint is the one place in a SaaS product where you accept arbitrary bytes from the internet, store them indefinitely and later serve them back to other users, often under your own domain and inside an authenticated session. Most implementations I review treat it as a form field. That design ships stored XSS through SVGs, malware delivered to other tenants, public buckets indexed by search engines and invoices readable by anyone who can guess an ID. Secure uploads are an architecture decision, not a validation rule.

At Loop Origin we build document-heavy B2B portals, from KYC onboarding to dealer ordering, and upload handling is part of every custom web application we design. This is the reference architecture I use, with production configuration for Node and Laravel, the cost trade-offs and the failure modes behind each control.

What is a secure file upload architecture for a SaaS app?

Direct answer: Upload directly to private object storage with short-lived presigned URLs that enforce size and type, land files in a quarantine bucket, scan and validate them asynchronously by content rather than extension, re-encode images, then promote clean files to a private bucket and serve them through short-lived signed URLs from a separate domain with download-safe headers.

The threat model

Threat	Concrete example	Primary control	Layer
Stored XSS	SVG or HTML with script, served inline from `app.example.com`	Separate content domain, `Content-Disposition: attachment`, `nosniff`, CSP `sandbox`	Delivery
Content-type spoofing	`invoice.pdf` that is actually HTML	Magic-byte detection, ignore client MIME type	Validation
Malware distribution	Infected document shared with another tenant's users	Antivirus scan before the file becomes downloadable	Processing
Parser exploits	Malformed image hitting a vulnerable image library	Re-encode in an isolated worker, patched libraries, size limits	Processing
Decompression bombs	Small archive that expands to gigabytes	Reject archives or cap expanded size and file count	Validation
Broken access control	`/files/1043` readable by another tenant	Per-request authorization, unguessable keys, short-lived URLs	Delivery
Path traversal	Filename `../../.env`	Never use client filenames as storage keys	Storage
Metadata leaks	GPS coordinates in a photo	Strip metadata on re-encode	Processing
Storage abuse	Scripted uploads to run up your bill	Per-tenant quotas, rate limits, lifecycle expiry	Storage

The metadata row is easy to underestimate; I showed what a single phone photo leaks in image metadata privacy leaks.

The reference flow

Browser â”€â”€(1) request uploadâ”€â”€â–º API: authorize, create upload record, presign POST
Browser â”€â”€(2) POST fileâ”€â”€â”€â”€â”€â”€â”€â–º quarantine bucket (private, expires after 7 days)
Storage â”€â”€(3) eventâ”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â–º queue â”€â”€â–º scan worker (isolated, no inbound network)
Worker:  magic bytes â–º size/structure â–º antivirus â–º re-encode images â–º strip metadata
Worker â”€â”€(4) cleanâ”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â”€â–º files bucket (private)  + mark record "ready"
Worker â”€â”€(4) rejectedâ”€â”€â”€â”€â”€â”€â”€â”€â”€â–º delete + mark record "rejected" + alert
Browser â”€â”€(5) downloadâ”€â”€â”€â”€â”€â”€â”€â”€â–º API: authorize per request â–º 5-minute signed GET
                                 served from usercontent.example.net

Files never touch the app servers, and nothing is downloadable until the worker marks it ready.

Step 1: Presigned uploads that enforce limits

A presigned PUT URL only proves the client may write one key. A presigned POST policy can also enforce size and content type at the storage layer. In Node with AWS SDK v3:

import { S3Client } from '@aws-sdk/client-s3';
import { createPresignedPost } from '@aws-sdk/s3-presigned-post';
import { randomUUID } from 'node:crypto';

const s3 = new S3Client({ region: process.env.AWS_REGION });
const ALLOWED = new Set(['application/pdf', 'image/jpeg', 'image/png']);

export async function presignUpload({ tenantId, userId, contentType, size }) {
  if (!ALLOWED.has(contentType)) throw new Error('type_not_allowed');
  if (size > 15 * 1024 * 1024) throw new Error('too_large');

  const key = `quarantine/${tenantId}/${randomUUID()}`; // never the client filename
  const { url, fields } = await createPresignedPost(s3, {
    Bucket: process.env.UPLOAD_BUCKET,
    Key: key,
    Conditions: [
      ['content-length-range', 1, 15 * 1024 * 1024],
      ['eq', '$Content-Type', contentType],
    ],
    Fields: { 'Content-Type': contentType },
    Expires: 300,
  });
  await db.uploads.insert({ key, tenantId, userId, status: 'pending', declaredType: contentType });
  return { url, fields, key };
}

In Laravel 10 and later, Storage::disk('s3')->temporaryUploadUrl($key, now()->addMinutes(5)) returns a presigned PUT URL and headers. It is simpler, but enforce size with a bucket-side check in the worker because a PUT URL cannot carry a size range. For uploads that go through the app, validate with mimetypes: (content-based) rather than mimes: alone, and set max: in kilobytes.

Step 2: Validate by content, then scan

The scan worker runs on a separate small instance or container with no inbound network access and read access only to the quarantine prefix. Its pipeline, in order of cost:

# 1. Real type from magic bytes, not the extension
type=$(file --mime-type -b "$f")
case "$type" in application/pdf|image/jpeg|image/png) ;; *) reject "type:$type";; esac

# 2. Antivirus with ClamAV's daemon (exit 0 clean, 1 infected, 2 error)
clamdscan --fdpass --no-summary "$f"; rc=$?
[ $rc -eq 1 ] && reject "malware"
[ $rc -eq 2 ] && retry_later "scanner_error"   # fail closed, never mark ready

Treat a scanner error as "not clean". The most common scanning incident I have seen was a worker that marked files ready whenever the scanner timed out.

For images, re-encode instead of trusting the original: decode, resize to a maximum dimension, and write a fresh JPEG or WebP with metadata stripped. This removes EXIF data, appended polyglot payloads and most malformed structures in one step; run it with resource limits.

Antivirus signatures catch known malware, not targeted payloads, which is why content validation, re-encoding and delivery isolation still matter.

Step 3: Serve files so they cannot attack your app

A clean file is still dangerous if the browser renders it on your app's origin. My delivery defaults:

Content-Disposition: attachment; filename="invoice-2026-10.pdf"
Content-Type: application/pdf            (the validated type, not the uploaded one)
X-Content-Type-Options: nosniff
Content-Security-Policy: sandbox; default-src 'none'
Cache-Control: private, max-age=300

Serve from a separate registrable domain, such as example-usercontent.net, so that a file that does execute script has no access to your app's cookies or same-origin APIs. A subdomain of your main domain is weaker, because cookies scoped to the parent domain still travel; I covered that in cookie security flags.

Every download goes through an API endpoint that checks the tenant and the user's permission on that record, then issues a signed URL valid for a few minutes. Signed URLs are bearer tokens: anyone holding one can use it until it expires, so keep the lifetime short and never log full URLs. In shared-schema multi-tenant systems, scope the lookup by tenant in the query itself, as discussed in multi-tenant SaaS database architecture.

Step 4: Lock down the buckets

Block all public access at the account and bucket level.
Deny requests that are not over TLS with a bucket policy condition on aws:SecureTransport.
Enable default encryption; use a customer-managed key for regulated documents so access is auditable.
Add a lifecycle rule that expires everything under quarantine/ after a few days, so abandoned and rejected uploads do not accumulate.

Cost and trade-offs

Decision	Option A	Option B	Cost drivers	When I choose B
Upload path	Through app servers	Direct presigned to storage	A: app CPU, memory and timeouts; B: almost none	Almost always
Scan timing	Synchronous in request	Asynchronous worker	A: slow uploads and timeouts; B: one queue and worker	Files above a few MB or any volume
Scanner	Self-hosted ClamAV	Managed malware scanning from the cloud provider	A: a worker with enough memory for signatures, plus patching; B: per-GB pricing	Low ops capacity or compliance needs
Delivery domain	App subdomain	Separate registrable domain	B: one extra domain and TLS certificate	Any product where users upload for other users
Image handling	Store original	Re-encode and strip	B: worker CPU per image	User-generated images shown to others

ClamAV keeps signatures in memory; do not co-locate it on a small app server.

Failure modes I have seen

Bucket made public "temporarily" to fix broken image previews, then forgotten.
Download URLs with year-long expiry pasted into emails and support tickets.
Extension allowlist only, bypassed with report.pdf.html or a correct extension on the wrong content.
Scanner fail-open on timeouts, as described above.
Thumbnails generated before scanning, running untrusted files through the image library first.

Implementation checklist

Presigned POST with size and type conditions; random keys under a tenant prefix.
Quarantine prefix with lifecycle expiry; nothing downloadable until marked ready.
Isolated worker: magic-byte check, structure and size limits, antivirus, fail closed.
Re-encode images and strip metadata; reject or tightly cap archives.
Private buckets, TLS-only policy, encryption at rest.
Separate content domain with attachment, nosniff and CSP sandbox.
Per-request authorization and signed GET URLs measured in minutes.
Per-tenant quotas, upload rate limits and alerts on rejection spikes.
Audit log entries for upload, scan result, download and deletion.

Need this reviewed on an existing platform? We cover it in server setup and maintenance engagements and Laravel development work.

Frequently asked questions

Is checking the file extension enough?

No. Extensions and the browser-supplied MIME type are both controlled by the uploader. Detect the type from the file's magic bytes on the server, compare it with your allowlist, and serve it with the validated type and nosniff.

Is ClamAV good enough for scanning uploads?

It catches known malware signatures and is a reasonable baseline for document uploads. It will not catch targeted or novel payloads, so combine it with content validation, re-encoding and safe delivery headers rather than relying on it alone.

How long should signed download URLs last?

Minutes, not days. Generate a fresh URL on each authorized request; long-lived URLs behave like public links once shared or logged.

Why serve user files from a separate domain?

So a file that executes script runs on an origin with no access to your app's cookies, storage or same-origin APIs.

Are SVG uploads safe to allow?

Not as-is. SVG is XML that can contain script. Either convert SVGs to PNG on upload, sanitize them with a strict allowlist, or serve them only as downloads from the isolated content domain.

Follow & Connect

Building Smart Systems: My Journey into AWS, Machine Learning, and Applied AI 🚀

Muntazer Mahdi — Sat, 31 Oct 2026 14:24:56 +0000

Hey everyone! 👋

I’m Mntazer Mahdi, a final-year Computer Science student and full-stack/cloud developer. I’ve spent a lot of time working with Python, JavaScript, Docker, and Microsoft Azure, but lately, I’ve been heavily focusing on expanding my toolkit.

💡 What I'm Learning Right Now:
Machine Learning & AI: Diving deeper into computer vision pipelines, smart automation, and intelligent data handling.

AWS Cloud Infrastructure: Moving workloads to the cloud, setting up EC2 environments, and building scalable backend architectures.

🛠️ What I'm Currently Hacking On:
Right now, my main focus is wrapping up my final-year project: a Smart AI Surveillance & Attendance System using RTSP CCTV streams, OpenCV, Flask, and AWS.

🤝 Let's Connect!
I'm always looking to collaborate on cool hackathon projects, discuss cloud architecture, or chat about backend systems. If you're working on anything related to AI/ML or cloud tech, drop a comment below or say hi! Let’s build something awesome together. ⚡

REST API vs HTTP API: Choosing API Gateway

Sascha Mirow-Stanitzok — Sat, 31 Oct 2026 14:01:05 +0000

REST API vs HTTP API: you face this choice every time you put a Lambda function on the internet through API Gateway in AWS. The console shows both tiles side by side, the names sound almost identical, and many tutorials simply use whatever the author is used to. My position is clear: the HTTP API is my default. There are, however, a handful of reasons that make me reach for the REST API without hesitation, and in 2026 that list has grown rather than shrunk.

In this article you get a decision guide based on the current feature lists and prices from AWS. After that, you build the same small API once as an HTTP API and once as a REST API, with the Serverless Framework and with AWS CDK, and see the differences directly in code.

In short

The HTTP API costs less, is simpler and is enough for most Lambda backends with JWT sign-in. You need the REST API as soon as AWS WAF, API keys with quotas, private endpoints, caching, request validation or response streaming come into play. The biggest pitfall when switching is the different event format your Lambda function receives.

What you will learn in this article

Which features only the REST API has and which only the HTTP API has
What both options cost at 10 or 100 million requests per month
A five-question decision guide for your next project
A Lambda handler that works with payload 1.0 and 2.0
How to set up and test both APIs with serverless.yml and AWS CDK

How do REST API and HTTP API differ?

Amazon API Gateway is the managed front door for your backends: it accepts HTTP requests, checks them if you want it to, forwards them to Lambda, other AWS services or any HTTP endpoint, and returns the response. Under the API Gateway umbrella there are three products. The REST API is the older and more comprehensive option (internally “v1”). The HTTP API arrived later as a leaner, cheaper option (internally “v2”). I will leave the third product, the WebSocket API, aside here. I show how to use this forwarding to arbitrary HTTP endpoints to replace a legacy system route by route in the article on the strangler fig pattern with Lambda.

One common misunderstanding first: the names say nothing about architectural style. With either option you build perfectly normal REST interfaces with resources, verbs and status codes. At AWS, “REST API” is simply a product name. The difference lies in features, price and how a request reaches your function.

The HTTP API deliberately leaves out much of what the REST API has accumulated over the years. In return it costs less, can deploy changes automatically and comes with a JWT authorizer that validates tokens without any code of your own. The REST API, on the other hand, is closer to a complete API management product: keys, quotas, caching, validation, transformations, firewall and developer portal.

Feature comparison: what only the REST API can do

The following table summarizes the feature list from the API Gateway documentation. I kept only the rows where the two options differ and grouped them by topic.

Feature	REST API	HTTP API
Endpoint types	Regional, edge-optimized, private	Regional only
AWS WAF	yes	no
Resource policies	yes	no
JWT authorizer (e.g. Cognito, Auth0, Entra ID)	no (only via Lambda authorizer)	yes, built in
API keys, usage plans, per-client quotas	yes	no
Developer portal	yes	no
Caching	yes	no
Request validation against JSON Schema	yes	no
Body transformation (mapping templates)	yes	no
Canary deployments	yes	no
Automatic deployments	no	yes
Execution logs, X-Ray tracing	yes	no
Mock integrations	yes	no
Private integration via AWS Cloud Map	no	yes
Response streaming	yes	no

Both support IAM authorization, Lambda authorizers, mTLS, custom domains, CORS, access logs, CloudWatch metrics and private integrations through Network or Application Load Balancers. For a typical Lambda backend behind a single-page app, the right-hand column is therefore often all you need.

Response streaming: the new reason for the REST API

One point has shifted my view recently. The REST API can now stream responses. To enable it, you set the integration’s responseTransferMode to STREAM. According to the documentation, a stream may then run for up to 15 minutes; the first 10 MB pass through unthrottled, after which API Gateway limits the rate to 2 MB per second. Streaming works only with proxy integrations, meaning AWS_PROXY for Lambda and HTTP_PROXY, and you lose caching and mapping templates. For AI chats, where the first token should appear on screen quickly, that is a real argument. The HTTP API cannot do this.

The same goes for long-running requests: on the HTTP API every integration ends after 30 seconds at most, and that limit cannot be raised. For Regional and private REST APIs you can request more than 29 seconds through Service Quotas, though you may have to accept a lower throttling limit for your account in return. For long jobs I still prefer the job pattern with 202 and SQS.

What do REST API and HTTP API cost?

The price difference is significant. In the US East (N. Virginia) Region, the HTTP API costs 1.00 US dollar per million requests for the first 300 million per month. The REST API costs 3.50 US dollars per million for the first 333 million. Other Regions have their own prices, so always check the official pricing page for your Region.

Requests per month	HTTP API	REST API	Difference
1 million	$1.00	$3.50	$2.50
10 million	$10.00	$35.00	$25.00
100 million	$100.00	$350.00	$250.00

Two details are easy to miss. First, the HTTP API meters requests in 512 KB increments: a 4.5 MB upload is billed as nine requests. Second, on the REST API the cache costs extra, billed per hour whether it is used or not. The AWS pricing example lists 0.038 US dollars per hour for a 1.6 GB cache, roughly 28 US dollars per month. New accounts get one million calls per month for each option in the Free Tier for twelve months.

My take on the numbers: for small APIs, a few dollars’ difference should not drive the decision. From a few tens of millions of requests per month, however, the gap becomes noticeable, especially when the Lambda function behind it costs only cents. At that point you quickly pay more for the gateway than for the actual compute.

Decision guide: REST API vs HTTP API in five questions

For new projects I go through the following questions in order. As soon as one gets a “yes”, it is the REST API. If all of them get a “no”, I use the HTTP API.

Do you need AWS WAF directly on the gateway or a private endpoint inside your VPC? Both exist only for the REST API. A CloudFront distribution with WAF in front of an HTTP API is an alternative, but the API itself then remains publicly reachable.
Do you sell or hand out access with per-customer quotas? Usage plans with API keys, throttling and a monthly quota are REST only.
Do you want to validate requests against a schema or cache responses at the gateway? That saves Lambda invocations, but only the REST API offers it.
Do you have to stream responses or answer synchronously for longer than 30 seconds? Again, only the REST API helps.
Do you need canary releases, execution logs or X-Ray directly on the gateway? REST once more.

For example, a backend for a React app with Cognito sign-in: five times “no”, so an HTTP API with a JWT authorizer. A partner interface where cafés preorder bread and each café gets its own quota: question 2 is “yes”, so a REST API. If you still want to stay with the HTTP API, you can check keys with a Lambda authorizer for HTTP APIs, but then you have to count quotas yourself.

Five questions: as soon as one is answered with yes, choose the REST API.

If you are not yet sure whether API Gateway is the right entry point at all or whether a Lambda function URL is enough, the AWS service finder helps you decide.

What happens to a request in REST API and HTTP API?

The feature set becomes more tangible when you follow the path of a request. With the REST API, a request can pass several stations before it reaches your function. With the HTTP API, the path is short.

🎬 Request path through REST API and HTTP API – this animated diagram only runs on my site: watch it on sascha-ms.de

Every station in the upper lane is optional. You pay for it with configuration complexity, but you save code in the function. A failed validation or a cache hit never reaches Lambda at all.

Step by step: the same API as HTTP API and REST API

Now it gets practical. We build a small bakery API with two routes: GET /breads/{slug} returns price and stock of a bread, POST /preorders accepts a preorder. A single Lambda function is attached to both API options. You would not do that in a real project, but here it shows very clearly how differently the two gateways package the same request.

Prerequisites

Node.js 22 and npm
An AWS account with credentials in your terminal (for deployment)
For option A the Serverless Framework v4 (npm i -g serverless), for option B the AWS CDK CLI (npm i -D aws-cdk); your account must be prepared for CDK once with cdk bootstrap
Basic Lambda knowledge; if that is new to you, start with building an HTTP API with API Gateway and Lambda in CDK

Step 1: Create the project

mkdir bakery-api && cd bakery-api
npm init -y
npm pkg set type=module
npm i -D typescript tsx @types/node@22 @types/aws-lambda esbuild
mkdir src test

{
  "compilerOptions": {
    "target": "ES2023",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "strict": true,
    "noEmit": true,
    "skipLibCheck": true,
    "types": ["node"]
  },
  "include": ["src", "test", "lib", "bin"]
}

Step 2: Bring payload 1.0 and 2.0 to a common shape

This is the most important technical difference. The REST API sends your function an event in payload format 1.0: method in httpMethod, path in path, client IP under requestContext.identity. The HTTP API uses payload format 2.0 by default: method under requestContext.http.method, path in rawPath, cookies in a separate field, repeated headers combined into one value. CDK and the Serverless Framework both default to 2.0. A handler that knows only one of the formats gets undefined with the other one and, in the worst case, returns a 500.

That is why I translate the event into a small structure of my own right at the start. The rest of the code only knows this structure and is therefore independent of the gateway. This also pays off if you migrate later.

// src/http-event.ts
import type { APIGatewayProxyEvent, APIGatewayProxyEventV2 } from 'aws-lambda';

// REST API sends payload 1.0, HTTP API sends payload 2.0 by default
export type AnyApiEvent = APIGatewayProxyEvent | APIGatewayProxyEventV2;

export interface NormalizedRequest {
  apiType: 'rest' | 'http';
  method: string;
  path: string;
  pathParameters: Record<string, string | undefined>;
  headers: Record<string, string | undefined>;
  body: string | null;
  sourceIp: string;
}

export function isPayloadV2(event: AnyApiEvent): event is APIGatewayProxyEventV2 {
  return (event as APIGatewayProxyEventV2).version === '2.0';
}

function lowerCaseKeys(headers: Record<string, string | undefined>): Record<string, string | undefined> {
  return Object.fromEntries(Object.entries(headers).map(([key, value]) => [key.toLowerCase(), value]));
}

function decodeBody(body: string | null | undefined, isBase64Encoded: boolean): string | null {
  if (body === null || body === undefined) return null;
  return isBase64Encoded ? Buffer.from(body, 'base64').toString('utf8') : body;
}

export function normalize(event: AnyApiEvent): NormalizedRequest {
  if (isPayloadV2(event)) {
    return {
      apiType: 'http',
      method: event.requestContext.http.method,
      path: event.rawPath,
      pathParameters: event.pathParameters ?? {},
      headers: lowerCaseKeys(event.headers ?? {}),
      body: decodeBody(event.body, event.isBase64Encoded),
      sourceIp: event.requestContext.http.sourceIp,
    };
  }
  return {
    apiType: 'rest',
    method: event.httpMethod,
    path: event.path,
    pathParameters: event.pathParameters ?? {},
    headers: lowerCaseKeys(event.headers ?? {}),
    body: decodeBody(event.body, event.isBase64Encoded),
    sourceIp: event.requestContext.identity.sourceIp,
  };
}

Step 3: Write the handler

The handler works only with the normalized request. It validates the preorder body itself, even though the REST API will also do this at the gateway later. The HTTP API does not validate bodies, and tests should run without a gateway. Both options understand the response format with statusCode, headers and body in the same way.

// src/handler.ts
import { normalize, type AnyApiEvent } from './http-event.js';

interface Bread {
  name: string;
  priceCents: number;
  availableToday: number;
}

interface ApiResult {
  statusCode: number;
  headers: Record<string, string>;
  body: string;
}

// Sample data instead of a database, so the focus stays on API Gateway
const BREADS: Record<string, Bread> = {
  rye: { name: 'Rye sourdough', priceCents: 420, availableToday: 12 },
  spelt: { name: 'Wholegrain spelt', priceCents: 480, availableToday: 0 },
};

function json(statusCode: number, payload: unknown, apiType: string): ApiResult {
  return {
    statusCode,
    headers: { 'content-type': 'application/json', 'x-api-type': apiType },
    body: JSON.stringify(payload),
  };
}

export interface Preorder {
  breadSlug: string;
  quantity: number;
  pickupDate: string;
}

// Even behind a REST API with request validation, the function checks the body itself:
// the HTTP API does not validate bodies, and tests should run without a gateway.
export function parsePreorder(body: string | null): Preorder | string {
  if (!body) return 'Body is missing';
  let data: unknown;
  try {
    data = JSON.parse(body);
  } catch {
    return 'Body is not valid JSON';
  }
  const { breadSlug, quantity, pickupDate } = data as Record<string, unknown>;
  if (typeof breadSlug !== 'string' || !(breadSlug in BREADS)) return 'Unknown bread';
  if (!Number.isInteger(quantity) || (quantity as number) < 1 || (quantity as number) > 20) {
    return 'Quantity must be between 1 and 20';
  }
  if (typeof pickupDate !== 'string' || !/^\d{4}-\d{2}-\d{2}$/.test(pickupDate)) {
    return 'Provide the pickup date as YYYY-MM-DD';
  }
  return { breadSlug, quantity: quantity as number, pickupDate };
}

export const handler = async (event: AnyApiEvent): Promise<ApiResult> => {
  const req = normalize(event);

  if (req.method === 'GET' && req.pathParameters.slug) {
    const bread = BREADS[req.pathParameters.slug];
    if (!bread) return json(404, { message: 'Bread not found' }, req.apiType);
    return json(200, { slug: req.pathParameters.slug, ...bread }, req.apiType);
  }

  if (req.method === 'POST' && req.path.endsWith('/preorders')) {
    const result = parsePreorder(req.body);
    if (typeof result === 'string') return json(400, { message: result }, req.apiType);
    return json(201, { message: 'Preorder accepted', ...result }, req.apiType);
  }

  return json(404, { message: 'Route not found' }, req.apiType);
};

Step 4: Cover both formats with tests

The tests build trimmed events in both formats and send them through the same handler. The x-api-type header tells you which option was detected.

// test/handler.test.ts
import { test } from 'node:test';
import assert from 'node:assert/strict';
import type { APIGatewayProxyEvent, APIGatewayProxyEventV2 } from 'aws-lambda';
import { handler, parsePreorder } from '../src/handler.js';

// Trimmed events: only the fields the handler reads
function restEvent(method: string, path: string, slug?: string, body?: string): APIGatewayProxyEvent {
  return {
    httpMethod: method,
    path,
    pathParameters: slug ? { slug } : null,
    headers: { 'Content-Type': 'application/json' },
    body: body ?? null,
    isBase64Encoded: false,
    requestContext: { identity: { sourceIp: '203.0.113.10' } },
  } as unknown as APIGatewayProxyEvent;
}

function httpEvent(method: string, path: string, slug?: string, body?: string): APIGatewayProxyEventV2 {
  return {
    version: '2.0',
    rawPath: path,
    pathParameters: slug ? { slug } : undefined,
    headers: { 'content-type': 'application/json' },
    body,
    isBase64Encoded: false,
    requestContext: { http: { method, sourceIp: '203.0.113.20' } },
  } as unknown as APIGatewayProxyEventV2;
}

test('REST API (payload 1.0) returns a bread', async () => {
  const res = await handler(restEvent('GET', '/breads/rye', 'rye'));
  assert.equal(res.statusCode, 200);
  assert.equal(res.headers['x-api-type'], 'rest');
  assert.equal(JSON.parse(res.body).name, 'Rye sourdough');
});

test('HTTP API (payload 2.0) returns the same bread', async () => {
  const res = await handler(httpEvent('GET', '/breads/rye', 'rye'));
  assert.equal(res.statusCode, 200);
  assert.equal(res.headers['x-api-type'], 'http');
});

test('unknown bread returns 404', async () => {
  const res = await handler(httpEvent('GET', '/breads/baguette', 'baguette'));
  assert.equal(res.statusCode, 404);
});

test('valid preorder returns 201 for both formats', async () => {
  const body = JSON.stringify({ breadSlug: 'rye', quantity: 2, pickupDate: '2026-11-02' });
  assert.equal((await handler(restEvent('POST', '/preorders', undefined, body))).statusCode, 201);
  assert.equal((await handler(httpEvent('POST', '/preorders', undefined, body))).statusCode, 201);
});

test('Base64-encoded body is decoded', async () => {
  const raw = JSON.stringify({ breadSlug: 'spelt', quantity: 1, pickupDate: '2026-11-03' });
  const event = httpEvent('POST', '/preorders', undefined, Buffer.from(raw).toString('base64'));
  event.isBase64Encoded = true;
  assert.equal((await handler(event)).statusCode, 201);
});

test('validation reports clear errors', () => {
  assert.equal(parsePreorder(null), 'Body is missing');
  assert.equal(parsePreorder('{broken'), 'Body is not valid JSON');
  assert.match(String(parsePreorder('{"breadSlug":"rye","quantity":50,"pickupDate":"2026-11-02"}')), /between 1 and 20/);
  assert.match(String(parsePreorder('{"breadSlug":"rye","quantity":1,"pickupDate":"tomorrow"}')), /YYYY-MM-DD/);
});

npx tsc --noEmit
node --import tsx --test test/*.test.ts

Expected result: # pass 6 and # fail 0.

Step 5a: Both APIs with the Serverless Framework

In the Serverless Framework the difference is a single word: httpApi creates routes in an HTTP API, http creates resources in a REST API. REST-specific settings such as API keys, usage plan and schema live under provider.apiGateway, CORS for the HTTP API under provider.httpApi. Version 4 of the framework builds TypeScript itself with esbuild.

# serverless.yml
service: bakery-api
frameworkVersion: '4'

provider:
  name: aws
  runtime: nodejs22.x
  region: eu-central-1
  memorySize: 256
  timeout: 10
  # Settings for the HTTP API
  httpApi:
    cors:
      allowedOrigins:
        - https://example.com
      allowedMethods:
        - GET
        - POST
      allowedHeaders:
        - content-type
  # Settings for the REST API
  apiGateway:
    apiKeys:
      - cafe-am-markt
    usagePlan:
      quota:
        limit: 10000
        period: MONTH
      throttle:
        rateLimit: 10
        burstLimit: 20
    request:
      schemas:
        preorder:
          name: Preorder
          schema: ${file(schemas/preorder.json)}

functions:
  api:
    handler: src/handler.handler
    events:
      # HTTP API (payload 2.0)
      - httpApi: 'GET /breads/{slug}'
      - httpApi: 'POST /preorders'
      # REST API (payload 1.0), API key required
      - http:
          path: breads/{slug}
          method: get
          private: true
      - http:
          path: preorders
          method: post
          private: true
          request:
            schemas:
              application/json: preorder

Store the JSON Schema for the preorder in its own file. API Gateway expects JSON Schema draft 4 here.

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "required": ["breadSlug", "quantity", "pickupDate"],
  "additionalProperties": false,
  "properties": {
    "breadSlug": { "type": "string", "minLength": 1 },
    "quantity": { "type": "integer", "minimum": 1, "maximum": 20 },
    "pickupDate": { "type": "string", "pattern": "^\\d{4}-\\d{2}-\\d{2}$" }
  }
}

serverless deploy

After deployment, the framework lists two base URLs and the value of the API key. The HTTP API has no stage name in its path; the REST API does. The stage name matches the framework stage, which is dev if you do not specify one.

Step 5b: The same APIs with AWS CDK

In CDK, both options have been part of aws-cdk-lib for some time: the REST API in aws-apigateway, the HTTP API in aws-apigatewayv2 with its integrations in aws-apigatewayv2-integrations. Install the libraries first.

npm i aws-cdk-lib constructs
npm i -D aws-cdk

// lib/bakery-api-stack.ts
import { CfnOutput, Duration, Stack, type StackProps } from 'aws-cdk-lib';
import type { Construct } from 'constructs';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import { NodejsFunction } from 'aws-cdk-lib/aws-lambda-nodejs';
import * as apigw from 'aws-cdk-lib/aws-apigateway';
import { CorsHttpMethod, HttpApi, HttpMethod } from 'aws-cdk-lib/aws-apigatewayv2';
import { HttpLambdaIntegration } from 'aws-cdk-lib/aws-apigatewayv2-integrations';

export class BakeryApiStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const fn = new NodejsFunction(this, 'BakeryFn', {
      entry: 'src/handler.ts',
      runtime: lambda.Runtime.NODEJS_22_X,
      memorySize: 256,
      timeout: Duration.seconds(10),
    });

    // Option 1: HTTP API – lean, low cost, payload 2.0
    const httpApi = new HttpApi(this, 'BakeryHttpApi', {
      apiName: 'bakery-http',
      corsPreflight: {
        allowOrigins: ['https://example.com'],
        allowMethods: [CorsHttpMethod.GET, CorsHttpMethod.POST],
        allowHeaders: ['content-type'],
      },
    });
    const httpIntegration = new HttpLambdaIntegration('BakeryHttpIntegration', fn);
    httpApi.addRoutes({ path: '/breads/{slug}', methods: [HttpMethod.GET], integration: httpIntegration });
    httpApi.addRoutes({ path: '/preorders', methods: [HttpMethod.POST], integration: httpIntegration });

    // Option 2: REST API – with validation, API key and usage plan
    const restApi = new apigw.RestApi(this, 'BakeryRestApi', {
      restApiName: 'bakery-rest',
      // Don't create an account-wide CloudWatch role (the setting applies to all REST APIs in the Region)
      cloudWatchRole: false,
      endpointConfiguration: { types: [apigw.EndpointType.REGIONAL] },
      deployOptions: {
        stageName: 'prod',
        throttlingRateLimit: 50,
        throttlingBurstLimit: 100,
        metricsEnabled: true,
      },
    });
    const restIntegration = new apigw.LambdaIntegration(fn);

    restApi.root
      .addResource('breads')
      .addResource('{slug}')
      .addMethod('GET', restIntegration, { apiKeyRequired: true });

    const preorderModel = restApi.addModel('PreorderModel', {
      contentType: 'application/json',
      modelName: 'Preorder',
      schema: {
        schema: apigw.JsonSchemaVersion.DRAFT4,
        type: apigw.JsonSchemaType.OBJECT,
        required: ['breadSlug', 'quantity', 'pickupDate'],
        additionalProperties: false,
        properties: {
          breadSlug: { type: apigw.JsonSchemaType.STRING, minLength: 1 },
          quantity: { type: apigw.JsonSchemaType.INTEGER, minimum: 1, maximum: 20 },
          pickupDate: { type: apigw.JsonSchemaType.STRING, pattern: '^\\d{4}-\\d{2}-\\d{2}$' },
        },
      },
    });
    const bodyValidator = restApi.addRequestValidator('BodyValidator', {
      requestValidatorName: 'validate-body',
      validateRequestBody: true,
    });
    restApi.root.addResource('preorders').addMethod('POST', restIntegration, {
      apiKeyRequired: true,
      requestValidator: bodyValidator,
      requestModels: { 'application/json': preorderModel },
    });

    // Usage plan: throttling and quota per partner (best effort, no protection against abuse)
    const plan = restApi.addUsagePlan('PartnerPlan', {
      name: 'bakery-partner',
      throttle: { rateLimit: 10, burstLimit: 20 },
      quota: { limit: 10_000, period: apigw.Period.MONTH },
    });
    plan.addApiStage({ stage: restApi.deploymentStage });
    const cafeKey = restApi.addApiKey('CafeKey', { apiKeyName: 'cafe-am-markt' });
    plan.addApiKey(cafeKey);

    new CfnOutput(this, 'HttpApiUrl', { value: httpApi.apiEndpoint });
    new CfnOutput(this, 'RestApiUrl', { value: restApi.url });
    new CfnOutput(this, 'CafeKeyId', { value: cafeKey.keyId });
  }
}

// bin/app.ts
import { App } from 'aws-cdk-lib';
import { BakeryApiStack } from '../lib/bakery-api-stack.js';

const app = new App();
new BakeryApiStack(app, 'BakeryApiStack', {
  env: { region: process.env.CDK_DEFAULT_REGION ?? 'eu-central-1' },
});

{
  "app": "npx tsx bin/app.ts"
}

npx cdk synth
npx cdk deploy

The synth output nicely shows how many more resources the REST API needs: resources, methods, model, validator, deployment, stage, usage plan and key. The HTTP API gets by with an API, an integration, two routes and a stage. Setting cloudWatchRole: false prevents CDK from setting an account-wide logging role for API Gateway that affects all REST APIs in the Region.

Step 6: Call both options

After the CDK deployment, you fetch the API key value using the key ID from the outputs. Replace the placeholders with the outputs of your deployment.

aws apigateway get-api-key --api-key <CafeKeyId> --include-value --query value --output text

curl -i https://<http-api-id>.execute-api.eu-central-1.amazonaws.com/breads/rye

curl -i https://<rest-api-id>.execute-api.eu-central-1.amazonaws.com/prod/breads/rye \
  -H "x-api-key: <key-value>"

curl -i -X POST https://<rest-api-id>.execute-api.eu-central-1.amazonaws.com/prod/preorders \
  -H "x-api-key: <key-value>" -H "content-type: application/json" \
  -d '{"breadSlug":"rye","quantity":99,"pickupDate":"2026-11-02"}'

This is what the result looks like (sample output, shortened): the HTTP API responds with 200 and x-api-type: http, the REST API without a key with 403 and {"message":"Forbidden"}, with a key with 200 and x-api-type: rest. The REST API rejects the preorder for 99 loaves with 400 and {"message": "Invalid request body"} without invoking Lambda. Send the same request to the HTTP API and the rejection comes from your handler.

To clean up: serverless remove or npx cdk destroy. As long as nobody calls the APIs, they cost nothing; a cached REST stage would be the exception.

Migrating from REST API to HTTP API

Many older projects run on the REST API because the HTTP API did not exist when they started. Switching can pay off, but it does not happen by itself. I check these points beforehand:

Event format: First switch the handler to a normalization like the one above. Alternatively, you can run an HTTP API with payload 1.0, which keeps the rework smaller.
API keys: If clients send x-api-key, you need a replacement, such as a Lambda authorizer or JWTs.
Mapping templates and validation: Everything that used to happen in the gateway moves into the function. A schema validator like Zod helps.
Stage path: The REST API has /prod/ in the path, the HTTP API with the $default stage does not. With a custom domain and API mappings, URLs stay stable for clients.
Error messages: The HTTP API has no custom gateway responses. Clients that parse specific error texts will notice the difference.
Logging and tracing: Execution logs and X-Ray at the gateway go away; access logs and tracing inside the function remain.

The opposite direction is realistic too: a project starts with the HTTP API, and months later a partner wants quotas or the security team requires WAF. If you already have the normalized handler, you only need to swap the gateway configuration.

Common mistakes when choosing REST API vs HTTP API

Using API keys for access control. AWS states it explicitly in the documentation: “Don’t use API keys for authentication or authorization to control access to your APIs.” Keys identify clients for quotas. For access control you need IAM, Cognito or an authorizer.
Treating usage plans as a cost brake. According to AWS, throttling and quotas are applied on a best-effort basis and can be exceeded. For costs, use AWS Budgets; against abuse, use WAF.
Choosing the REST API “just in case”. You permanently pay the higher price and carry more configuration for features you may never need.
Testing only one payload format. If you test locally with 1.0 events and then deploy behind an HTTP API, you only find the bug in production.
Overlooking the 30-second limit. On the HTTP API it is fixed. Longer work belongs in asynchronous processing.

My take

I start new Lambda backends with the HTTP API. It costs less, is quicker to set up, and the built-in JWT authorizer saves me a lot of work for apps with sign-in. For the team, it also means less gateway configuration that only two people understand.

I would push back on the common claim that the REST API is a legacy product. AWS keeps developing it, and response streaming in particular shows where things are heading: if you deliver AI responses or large downloads, you will find the new features there first. For public interfaces with partners, WAF requirements or private endpoints, the REST API remains my first choice. What matters is making the choice consciously and writing the code so that switching later does not hurt.

FAQ

Is the HTTP API faster than the REST API?

The HTTP API has fewer stations in the request path, but AWS does not give reliable latency figures in its feature overview. For most applications, the Lambda function’s run time dominates anyway. When in doubt, measure with your own load.

Can I use API keys with the HTTP API?

Not with built-in usage plans. You can, however, check keys with a Lambda authorizer and count quotas yourself, for example with a counter per key in DynamoDB.

How long can a request to API Gateway take?

On the HTTP API, 30 seconds at most, fixed. On the REST API, up to 29 seconds by default; for Regional and private APIs the limit can be raised, and response streaming allows up to 15 minutes.

Do I need API Gateway at all, or is a Lambda function URL enough?

For a single endpoint, webhooks or streaming without gateway features, a function URL is often enough. As soon as you need multiple routes, JWT validation, custom domains shared by several services or throttling, API Gateway is the better choice.

Your next step

Pick an API you are running right now and go through the five questions from the decision guide. If every answer is “no” and you are on the REST API, calculate the savings with your real request numbers. As preparation, add the normalization from step 2 to your handler. Then switching later is just a matter of configuration.

And I am curious: which option are you using right now, and was there a concrete reason for it, or was it simply the first tile in the console?

Sources

Tested with: Node.js 22.23, TypeScript 7.0.2, tsx 4.23, aws-cdk-lib 2.275.0, CDK CLI 2.1146.1, @types/aws-lambda 8.10.164 (tsc, 6 tests with node:test, cdk synth including the esbuild bundle). serverless.yml checked for valid YAML only. curl outputs are marked as samples.

This post was created with AI assistance.

👉 Next step: Working with AWS CDK? Grab my printable AWS CDK cheat sheet with commands, concepts and a stack skeleton.

Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en

DEV Community: aws

Desert Hours: Mapping When Transit Disappears With Amazon Location Service

What Is a Transit Desert?

What You Can Explore

At This Point You Should Have

Step 1 — Gather the Data

Step 2 — Count Departures by Hour

Step 3 — Ask Amazon Location for a 10-Minute Walk

🔎 Tip: Place Households Where Housing Is

Step 4 — Build the Hourly Service Area

Step 5 — Ask for Real Journeys

Step 6 — Ship It as Static Files

One Tract, One Day

Compare and Contrast

What Did It Cost?

So, Which One?

Summary

References

AWS FinOps Field Notes (2026-10-31): Automating Orphaned EBS & EC2 Cleanup

Where the Waste Hides

Auditing Unattached Volumes with Python & Boto3

Building `aws-ec2-check`: A Small Open-Source AWS Utility with Node.js

The problem

What aws-ec2-check does

Why build another AWS tool?

How it's built

AWS SDK instead of the Console

Security matters

IAM and least privilege

Keeping the CLI simple

Why TypeScript?

Testing the project

What I learned building it

CLI design

AWS authentication

Error handling

Documentation

What I'd like to add next

Why open source?

Try it

Final takeaway

Honeytokens in Practice: Catching Credential Theft When Malware Uses LLMs to Triage Your Secrets

Honeytoken là gì và vì sao nó hợp với thời LLM

Lớp 1: AWS key mồi với IAM user không có quyền gì

Lớp 2: Canary URL trong file .env và tài liệu nội bộ

Lớp 3: Theo dõi việc đọc file mồi trên chính máy

Kết luận

Test Step Functions with the TestState API

Why teams rarely test Step Functions properly

What the TestState API does

The four statuses at a glance

Mocks instead of real Lambda calls

The example: a bike rental with retry and catch

How to test Step Functions step by step with TypeScript

Prerequisites

Step 1: Create the project

Step 2: Guardrail tests without AWS access

Step 3: A small helper for the TestState API

Step 4: Test retry and catch

Step 5: Run the tests

Step 6: Deploy the same definition with CDK

Typical mistakes when testing with TestState

Cost and security when testing

My take

FAQ

Can I test Step Functions locally?

Does the TestState API need an IAM role?

Can TestState run a whole state machine?

Does TestState work with Map and Parallel states?

Your next step

Sources

Event-Driven Architecture on the Cloud

Contents

When one slow call takes down everything

The mental model: a newsroom, not a phone tree

The picture: one event, many reactions

Queues vs pub/sub vs streams

Publishing and consuming an event

Delivery guarantees: why idempotency is non-negotiable

Common mistakes that cost hours

What `aws-ec2-check` does