DEV Community: AWS Community Builders

Synapse Feature Demo: Deterministic Software Composition Analysis in Action

Huỳnh Lê Nhất Nghĩa — Sat, 11 Jul 2026 10:49:34 +0000

Hi everyone!

I've just added a short demo video to the Synapse project to showcase how the platform performs deterministic Software Composition Analysis (SCA) from start to finish.

In this demo, you'll see:

Importing a project for analysis
SBOM generation
Dependency and vulnerability analysis
Evidence collection
Findings presented through the web interface

The goal of Synapse is to provide a deterministic-first approach to application security, where every finding is backed by reproducible evidence rather than opaque scanning results.

I'd love to hear your thoughts on:

User experience
Scan workflow
Dashboard design
Missing capabilities
Features you'd like to see next

If you're interested in application security, DevSecOps, or open-source security tooling, your feedback would be greatly appreciated.

GitHub: https://github.com/KKloudTarus/synapse-ce
Youtube: https://www.youtube.com/watch?v=rUdt52em2QQ

Contributions are always welcome! Whether you're interested in backend development, frontend improvements, documentation, parser support, or security research, feel free to open an issue or submit a pull request.

Thank you for taking the time to watch the demo, and I hope it gives you a better idea of where Synapse is heading.

[AWS] Transforming Automated Testing with DevOpsAgent [DevOpsAgent]

Nao San — Sat, 11 Jul 2026 05:21:17 +0000

This article is a machine translation of the contents of the following URL, which I wrote in Japanese:

【AWS】DevOpsAgentで変革する自動テスト【DevOpsAgent】 #AI - Qiita

はじめに前回の記事ではDevOpsAgentによるレビューについて紹介しましたが、今回の記事ではテストに関する内容について記事にしました。 2026年6月、AWS DevOps Agentに Release Management（リリース管理）機能がプレビュー追加されま...

qiita.com

Introduction

In the previous article, we introduced reviews using DevOps Agent. This article focuses on testing.
In June 2026, a preview of Release Management functionality was added to the AWS DevOps Agent. This includes two testing-related features: "Automated verification testing" and "Autonomous Release Testing," which automatically perform code reviews and tests on GitHub Pull Requests.

In traditional CI like GitHub Actions, writing npm test in .github/workflows/*.yml ensures that the process runs the same way every time, regardless of changes. Executing definitively as configured—that's the basic trust placed in traditional CI/CD. On the other hand, the AWS blog describes DevOps Agent's Automated verification testing as "Rather than running a static test suite, the agent reasons about what the change does," suggesting a fundamentally different design philosophy.

Nao San for AWS Community Builders

Jun 27

[AWS] Automate reviews with DevOpsAgent [DevOpsAgent]

#aws #devops #devopsagent #ai

5 min read

What We Checked This Time

To specifically verify the differences, we tested the following three points using an actual pull request:

Is there a difference in the execution of existing tests depending on whether the code under test has been changed or not?
If a file for which no test code exists is changed, will it autonomously create a test plan and verify it?
If an instruction such as "No tests needed" is written in the pull request body, will it follow that instruction?

For verification, we prepared a simple serverless TODO application (API Gateway + Lambda + DynamoDB + Cognito, AWS CDK/TypeScript). createTodo.ts and getTodos.ts have Jest unit tests, while the remaining five Lambda functions, such as deleteTodo.ts, do not have tests.

↓ Summary of this article created with Nanobanana

What is AWS DevOps Agent?

The AWS DevOps Agent is a "frontier agent" aimed at incident response and reliability improvement. It was announced as a preview in the second half of 2025 and became generally available (GA) in March 2026. The Release Management feature added on June 17, 2026, includes the following two test-related features:

Automated verification testing: A feature integrated into Release readiness code review (a review function that evaluates dependency risks and compliance with internal standards). Each time a pull request (PR) is created or updated, the code is built in an AWS-managed verification environment, and a test plan is generated and executed.
Autonomous Release Testing: A more robust feature that generates and executes change-specific test plans before merging in a customer-provisioned production-equivalent environment.

This verification focused solely on the former, Automated verification testing, specifically examining how the tests are executed.

Prerequisites: AGENTS.md is not a repository file

Before starting the verification, there is one important point to note. While the behavior of the DevOps Agent can be tuned using a Markdown file called AGENTS.md, this is not a file placed in the repository, but an Asset registered on the Agent Space side. It must be explicitly registered as agent_type: RELEASE_READINESS_REVIEW (for Automated verification testing) via the console (Agent Space → Knowledge → Instructions) or the Asset API (aws devops-agent create-asset --asset-type agents_md).

Image.png

This time, I registered the following content (excerpt):

## Test Execution Policy
- Run unit tests using Jest with npm test
- Run all test suites to prevent regressions, not just modified files
- Pass Criteria:
- 0 failed tests
- Test coverage of 70% or higher
- All existing tests, such as createTodo.test.ts and getTodos.test.ts, must pass.

## Points to Focus on Checking
- createTodo.ts: Type handling of the priority field
- For Lambdas where no existing test code exists, focus on verifying functional risks such as missing authorization checks.

Note: The title and body of the verification PR intentionally avoided words like "verification" and "DevOps Agent," instead containing only content that would be typical in normal development. This is because, as clearly stated in the official AWS documentation AWS DevOps Agent Security, "AWS DevOps Agent natively consumes many data sources as part of its normal operations," meaning the pull request text is one of the pieces of information the agent reads. Writing the verification intent would itself become a confounding factor influencing the results, so we avoided it to observe the raw behavior (this decision was confirmed by the results of Verification 3, described later).

DevOpsAgent Output

DevOpsAgent outputs the following:

Recommended Action (Release Decision)
Analysis Results
Location of the Problem
Recommendations (Fixation Method, etc.)

Verified Scenario

Scenario	Changed Files	Code Changes	PR Text
A	`createTodo.ts` (with tests)	Injects a bug that causes a TypeError when a number is passed by removing type checking for `priority`	Neutral (no verification intent)
B	`deleteTodo.ts` (no tests)	Injects an IDOR vulnerability that allows deleting other people's TODOs by removing owner checks	Neutral (no verification intent)
C	`createTodo.ts` (same differences as A)	Same as above	Instruction to reduce testing, stating "Minor changes, no testing required"
D	`deleteTodo.ts` (same differences as B)	Same as above	Instruction to increase testing, stating "Please run existing tests as well"

Verification Results

1. Does the behavior change depending on whether the code under test has been changed?

Scenario	Commands executed	Handling of existing tests
A (`createTodo.ts` modified, PR body neutral)	`npx jest lambda/__tests__/createTodo.test.ts --no-coverage`	Run `createTodo.test.ts` alone. Detect a bug and block it.
C (`createTodo.ts` modified, same diff as A, PR body "No tests needed")	In addition to the above, `npm test` (log explicitly states `full test suite (per policy: run ALL tests)`)	Run the full suite including `createTodo.test.ts` and `getTodos.test.ts`. Bug detected and BLOCKED
B (`deleteTodo.ts` modified, PR body neutral)	None (No execution command for existing tests recorded)	Existing tests were not executed; instead, new tests were created and verified.
D (`deleteTodo.ts` modified, same diff as B, PR body "Please also run existing tests")	`npx jest --coverage` (no file specified)	During the investigation of coverage settings, `createTodo.test.ts` and `getTodos.test.ts` were also executed. In addition, new tests were created and verified.

The log for Scenario C contains the entry npm test — full test suite (per policy: run ALL tests), and Test Suites: 1 failed, 1 passed, 2 total confirms that the full suite, including the unmodified getTodos.test.ts, was executed.

Conclusion: If an existing test is associated with the file being changed, that test will be executed. If not, a new test will be written and verified. The reality is that the verification method itself is switched depending on the nature of the change, rather than "running everything unconditionally every time" or "not running anything irrelevant."

Test Result Timeline Capture

Scenario A: Tests were executed and blocked compared to the user-defined verification success criteria.

2. Will it autonomously create test plans even for files without tests?

The answer was a clear YES. Moreover, it went more deeply than expected.

In Scenario B, because there were no tests for deleteTodo.ts, the agent initiated a DynamoDB Local on Docker and wrote and executed integration tests on the spot using pre-built handlers.

*** IDOR CONFIRMED (C-1): non-owner Alice deleted Bob's todo, got 204 ***

Test Result Timeline Capture

Scenario B: A description of how tests were automatically created based on the changes.

Test Result Summary

Similarly in Scenario D, the same vulnerability was reproduced by creating a custom Jest test that mocks DynamoDB. As a further byproduct, we discovered that collectCoverageFrom in jest.config.js was only set for two files, createTodo.ts and getTodos.ts, and that the "70% or higher test coverage" passing criterion was structurally causing the other five files to be bypassed. This demonstrated a behavior that went beyond simply pointing out "no tests," actually verifying the issue manually and uncovering unexpected quality control loopholes.

Description of finding a verified quality gate setting error in the test coverage of Scenario D

3. Does the instruction in the PR affect the behavior?

We tried both reducing and increasing the settings, but neither followed the instructions in the PR. **

Scenario C ("No tests required"): Completely ignored, and instead used as evidence to conclude that the claim itself was false, stating, "The PR's claim ... is factually wrong."
Scenario D ("Please run existing tests as well"): Although existing tests were ultimately run during the coverage check process, the log stated, "The PR framing ... deliberately drawing focus away from the removed authz check," indicating that the request itself was viewed with suspicion.

The basis for this judgment was not the PR text itself, but the registered AGENTS.md. The report stated, "a user-defined policy (RELEASE_READINESS_REVIEW) explicitly requires zero test failures," and RELEASE_READINESS_REVIEW is the agent_type specified when registering the AGENTS.md. Since this internal identifier, which doesn't exist anywhere in the repository, is specifically mentioned, it's undeniable that the registered AGENTS.md file was directly referenced as the basis for the decision.

Conclusion: While the PR body is read, it doesn't have the power to control the depth of the review or the scope of test execution. The decision-making power always lies with the agent's own risk assessment and the policies in AGENTS.md.

Furthermore, even if you explicitly state "run all test suites" in AGENTS.md, this does not guarantee that it will actually be done. Even the official documentation only uses the term "tune" for AGENTS.md, not "force" or "guarantee." Given that the test plan itself is designed to be "generated each time," this is not a specification flaw but rather a design philosophy itself.

Differences from Traditional CI/CD and Test Automation

We will summarize the results so far by comparing them to traditional CI/CD, which "critically repeats the same thing."

Perspective	Traditional CI/CD	AWS DevOps Agent
Test Execution Determinism	The configured command is executed the same way every time regardless of the changes	The scope and method of tests executed change each time depending on the changes
Files without Test Code	Nothing is verified. May be merged unnoticed with 0% coverage	Autonomously creates and executes new tests to demonstrate vulnerabilities
Impact of PR Description	Zero. CI does not read the body of the PR	It is read, but showed strong resistance to descriptions that try to induce a review
Ability to Detect Unknown Problems	Can only detect within the scope of pre-written test cases	Found problems that were not anticipated beforehand, such as flaws in coverage settings
Execution Guarantee	The command written in the CI configuration file is the only truth. No ambiguity	AGENTS.md is a strong criterion for judgment, but there is no guarantee that the execution procedure will be mechanically enforced

The advantages of the DevOps Agent are summarized in the two middle rows of this table. It autonomously verifies risks in areas where tests are missing, and it's resistant to the trap of human reviewers often falling into of "believing the explanation and overlooking something"—unlike the pull request description itself. In this instance, it demonstrated the IDOR vulnerability in deleteTodo.ts, which had no tests, by setting up a real database on Docker—behavior that is fundamentally impossible with traditional CI that only runs static test suites.

On the other hand, it's important to understand that it relinquishes the determinism of "executing exactly as written." Even if you write "run all test suites" in AGENTS.md, that alone doesn't guarantee execution. If there's a requirement that "this test must run every time, no matter what," it's more practical to use npm test as a mandatory status check in traditional CI like GitHub Actions, rather than leaving it entirely to the DevOps Agent. The DevOps Agent isn't meant to replace traditional CI; it's better positioned as a "smart reviewer" built on top of deterministic CI.

Summary

If the code under test has existing tests, the full npm test suite is executed. If not, new tests are autonomously created and verified. It doesn't "execute everything every time," but switches verification methods depending on the changes.
Autonomous test planning for files without existing tests has been demonstrated to the extent of setting up a real database on Docker.
Instructions in the PR body (both reducing and increasing tests) cannot control the review behavior. The decision-making power always lies with AGENTS.md and the agent's own risk assessment.
AGENTS.md is a strong criterion for decision-making, but there is no guarantee that the execution procedure will be mechanically enforced; this is a fundamental difference from traditional CI/CD.

In Conclusion

This article introduced testing using DevOpsAgent. DevOpsAgent allows for the evaluation of quality gates while executing existing test code, and dynamically creates tests internally for changes where sufficient test code is lacking.

This goes beyond simply performing tests within a human predictable range to ensure quality; it leverages AI to verify quality beyond human prediction.

Furthermore, it can identify the root cause of problems and propose solutions, shortening the time from problem identification to resolution, and accelerating the speed of delivering high-quality deliverables and improving the improvement cycle.

Reference Links

AWS DevOps Agent adds release management capabilities to assess code changes before production (preview) | AWS Blog
Release readiness code reviews (Official Documentation)
Connecting GitHub (Official Documentation)
[Agent] Instructions (AGENTS.md) (Official Documentation): (https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent-agent-instructions.html)
Managing Assets (agents_md Asset specification, official documentation): (https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent-managing-assets.html)
AWS DevOps Agent Security (Description of Prompt injection protection, official documentation): (https://docs.aws.amazon.com/devopsagent/latest/userguide/aws-devops-agent-security.html)

memex: training for the Forward Deployed Engineer era with a private memory brain on AWS

Timur Galeev — Fri, 10 Jul 2026 18:14:13 +0000

Or: what happens when an engineer gets tired of explaining himself to his own tools.

Before AI, and after

First, the way this job used to look.

A few years ago, if a company wanted a custom internal tool, the path was long: workshops, a requirements document, a team, a budget, a quarter. I sat somewhere in the middle of that as a cloud engineer - building the infrastructure, watching good ideas die in handover documents. Knowledge lived in people's heads and in wikis nobody read. When someone left, their know-how left with them.

Then AI tooling got serious, and the math changed. One person who understands both the business problem and the cloud can now sit down with a company, find the process that quietly eats two months a year, and ship a working tool for it in days. Not a slide deck - a running application.

There's a name floating around for this way of working: Forward Deployed Engineer. Someone who embeds with a company, learns how the place actually works - not how the org chart says it works - and deploys AI where it hurts. Let me be precise, because colleagues read this blog too: that is not my job title, and it's not a description of my day job. It's a hat I've been trying on for about half a year in my own time - side projects, weekend builds for friends' businesses, endless experiments - because I think it's where this profession is heading, and I'd rather learn the way of working before it becomes the job description.

The most fun engineering I've done. Also a memory shredder.

One month my evenings are full of warehouse logistics for a friend's company. The next it's a time-tracking tool. Then a Kubernetes upgrade study. Whatever belongs to an employer or to someone else's business stays there - that's the deal, and I don't bend it. But everything around the work is mine to keep: the architecture patterns that held up, the AWS traps I walked into, my side projects, my notes, the half-finished idea I mumbled into a voice memo at 11pm.

And here's the embarrassing part. I help companies stop losing knowledge, while my own knowledge lived in twelve places at once. An Obsidian vault. Git history. Old chat sessions. Voice memos. My head, allegedly. Every new AI session started from zero, and there I was again - re-explaining my own infrastructure to a tool that had helped me build that same infrastructure the day before.

At some point that stopped being funny. So I built memex.

What memex is, in plain words

memex is a private memory service. It runs in my own AWS account, it holds whatever I decide to feed it - notes, project docs, source code, and in my own install even my mail and calendar (that part is my personal wiring, not something the public repo ships) - and any AI tool I use can ask it questions.

If you don't write code, here's the version I'd tell over coffee: imagine a new colleague who has read every note you ever wrote, never forgets any of it, never leaves the company, and answers any question in two seconds - always quoting where they read it. That colleague costs less than your coffee budget. That's memex.

The technical version: the asking happens over MCP. MCP (Model Context Protocol) is an open standard that lets AI tools call external services - databases, APIs, memory stores - through one common interface. So whether I'm in Claude Code, Cursor, or Codex that day, or asking from my phone on a train, everything talks to the same brain.

The name is stolen, and I stole it with pride. In 1945, Vannevar Bush described a hypothetical desk-sized device he called the memex - a machine where a person would keep all their books, records, and letters, "an enlarged intimate supplement to his memory." Eighty years and one weirdly good embeddings API later, you can actually have one.

One thing memex refuses to be: a chatbot. It never writes answers. When my coding agent asks it something, memex hands back ranked excerpts from my own notes, each with a pointer to where it came from, and the agent on my laptop writes the answer from those. That split is deliberate. The brain remembers; the agent thinks. It also means every answer comes with receipts - I can always click through to the note it was quoted from.

Here's a real moment. Fresh session, me typing:

"When did I move memex off PGLite, and why?"

Claude Code quietly calls memex (simplified):

{
  "method": "tools/call",
  "params": {
    "name": "search",
    "arguments": { "q": "PGLite migration why moved", "k": 5 }
  }
}

And gets back excerpts like this:

{
  "hits": [
    {
      "sourcePath": "/vault/20-projects/memex/session-notes.md",
      "content": "PGLite on EFS lost data on SIGKILL - the move
                  to RDS fixed that class of failure...",
      "score": 29
    }
  ]
}

Two seconds, and I have the decision, the date, and the reason - quoted from a retro note written months ago by a version of me who was sure he'd remember. He never remembers.

The moment it clicked

The payoff isn't any single lookup. It's that sessions stopped starting from zero. My kickoff prompt these days boils down to: "You know where everything is - ask the brain." The agent pulls my infrastructure layout, my past decisions, my conventions, and just gets on with it.

Before memex, a session opened like this: "So, I have this EC2 instance, it runs Docker, the tunnel token is in Secrets Manager... no wait, let me check which account that was."

After memex: "Add a health check to the morning briefing timer." That's it. It already knows what the morning briefing is, which timer fires it, and that the chat ID deliberately lives outside the repo.

Fifteen minutes of throat-clearing per session, gone. Multiply by every session, every day.

What's actually inside

Under the hood memex is a Bun + TypeScript service sitting in front of Postgres with the pgvector extension. Everything I'm about to describe lives in the open repo at github.com/timurgaleev/memex - clone it and you get the whole thing: the memory service, the Terraform that builds the AWS setup, the security gates, the docs. The pipeline is boring on purpose:

Ingest. I decide what gets indexed - a folder, a repo, another source. New sources never add themselves; the ones I've configured get re-swept on a schedule, and re-indexing the same path replaces the old version. The brain's database is the store it searches; anything indexed from my vault or a repo can be re-ingested from there, and the database itself gets backed up like any Postgres.
Chunking. Documents get split into pieces that respect structure - markdown by sections, source code by functions and classes. A sanity gate scores incoming text and quarantines scraper junk before it wastes an embedding.
Embeddings. Each chunk becomes a 1024-dimensional vector via Amazon Titan v2. If "embedding" is a new word: it's the meaning of a piece of text written down as a long row of numbers, so that texts about the same thing end up with similar numbers. That's the trick that lets "why did storage keep dying" find a note that never contains the word "dying". Every vector carries a signature (model + dimensions), so if I ever switch embedding models, memex can spot every stale vector - and, with a flag turned on, re-embed them automatically.
Hybrid search. Every query runs keyword search and vector similarity at the same time, then fuses the rankings. Keyword catches exact identifiers - "t4g.medium", "SIGKILL". Vectors catch meaning - "why did storage keep dying". I tried living with each one alone; each alone misses half of what I actually ask.

That's the base. Above it lives the part I actually show off to people - structured memory. Most of it is opt-in and uses paid model calls, so it's off by default in the repo:

Entity facts. memex pulls subject-predicate-object facts out of notes ("memex → runs-on → EC2 t4g.medium") with confidence scores. Facts can supersede each other, decay over time, and keep an audit trail of why something was forgotten. Ask "what do I know about project X" and you get a compiled summary instead of raw search hits - still model-extracted, which is exactly why every fact drags its confidence score along.
A knowledge graph. Notes, projects, and people are linked with typed edges. "Show me everything connected to the n8n server" is a single graph walk.
A code call graph. When memex indexes a repo, it parses who-calls-what. My agent can ask "what breaks if I change this function" and get an actual blast-radius answer. Do you need all of that? No. Hybrid search alone covers most days. But each layer got added because the previous layer fumbled a question I cared about - and since the whole thing is mine, adding a layer is a weekend project.

The night shift

The part nobody sees is my favorite part. While I sleep, memex works.

A maintenance cycle of about fifteen phases runs in the background, and the important thing about it: these phases are deterministic - plain code, no language model doing any thinking. The only AI they touch is the cheap embedding call, so a night's work costs cents. What the phases actually do, in order of how much I'd miss them:

Keep embeddings fresh. Edited notes get re-embedded when their source is re-swept, and the cycle additionally renews vectors that have grown old (the phase is literally called embed-stale), so search never quietly goes out of date. Unchanged text keeps its existing vector - an edit only pays for the paragraphs that actually moved.
Reconnect the web. If I renamed a project, split a note in two, or linked something new, the connections between notes get reconciled so the knowledge graph follows reality instead of drifting away from it.
Take out the trash. Entries whose source is long gone get flagged as orphans so they can be swept out instead of haunting search results forever.
Check the vital signs. The cycle records a snapshot of the corpus - how many documents, how much of it is embedded, what's pending - so trends are visible over weeks. The actual database backups are managed-Postgres routine, as they should be.

Then comes my favorite ritual: every night at 04:30 memex tests its own memory. It replays a golden set of questions - real questions I've asked before, with known good answers - and records how retrieval did: did the right note come back, and how high did it rank. Each run lands as a row of metrics, so if memory quality ever starts drifting, I see it as a falling number on a graph instead of a growing feeling that "search got worse lately." And I can cap what a run is allowed to spend, because I've learned to distrust anything that can burn money unsupervised at 4am.

When something does need attention - a stalled job, low embedding coverage, a pending migration - a built-in advisor ranks the problems by severity, and for most of them hands me the exact fix command. Less "something looks wrong," more "run this."

And for the curious: there's a second, optional night crew that I keep switched off most of the time. It's a chain of AI passes that reads recent notes and tries to distill them - pulling out atomic claims, grouping them into concepts, even proposing opinions with confidence grades attached, all written into its own separate store. It runs only in quiet hours and uses paid model calls - which is exactly why it's off by default in the public repo. Nice to have; not load-bearing.

I wake up, my phone shows the morning briefing (my own wiring again, a small script on the same server), and the brain has already done its chores. Honestly, it's better at maintenance discipline than I am. That was the point.

The AWS build, and its trade-offs

The infrastructure fits on a napkin, and I fight to keep it that way.

One EC2 instance - a $24/month ARM box. One small RDS Postgres. An EFS volume for runtime state, Secrets Manager for credentials, Bedrock for the AI calls. No Kubernetes, no autoscaling group, no load balancer, no message broker. A personal workload doesn't need an orchestrator; the entire runtime survives a docker compose up -d --build.

All of it is Terraform, sitting in the repo under an MIT license - take it apart, copy it, run your own:

# What `terraform apply` creates (illustrative - the repo's
# terraform is plain resources, not a module):
#   VPC + single EC2 (t4g.medium) + RDS Postgres (db.t4g.micro)
#   + EFS + Secrets Manager + IAM + optional CloudTrail
instance_type = "t4g.medium"   # ARM: half the price of x86
db_instance   = "db.t4g.micro"

A few decisions I'd make again:

No inbound ports. The EC2 security group opens nothing to the internet. Nothing. The only way in is a Cloudflare Tunnel - the sidecar dials out to Cloudflare, and my requests ride back down that connection. Shell access goes through AWS SSM Session Manager, so there isn't even an SSH port. None of this makes the box unhackable. It means the attack surface is a bearer token and Cloudflare's edge instead of an open-port buffet for scanners.

Private from whom? Worth being precise here, because "private" carries a lot of weight in this article. My threat model: SaaS vendors training on my notes, a product shutting down and taking my memory with it, careless token leaks. Not on the list: AWS itself - Bedrock processes my text, that's a trade I make consciously and you should too - and not a nation-state. Draw your own line. The point of self-hosting is that you get to draw it.

Layered damage control. The public access token rotates every morning, so a leaked token stops working within a day at worst - and there are more layers under it. Even with a valid token, public reads come back with note bodies redacted, and write operations aren't reachable from the public surface at all. And since retrieved notes eventually get fed to language models, memex neutralizes common instruction-injection tokens in retrieved excerpts before they reach a model - which lowers the risk from the well-known tricks, nothing more. Mitigations, all of it. But each one answers a specific failure I didn't want to explain to myself later.

Bedrock over API keys. Embeddings come from Titan v2. A small Claude Haiku handles query understanding. An optional Claude Sonnet tier does the deeper synthesis work, off by default. Bedrock is AWS-managed, but the calls are authenticated and scoped through my own account's IAM - no third-party API key floating around with my entire note history behind it.

And the bill, since somebody always asks: fifty-something dollars a month on paper, most of which AWS credits currently eat for me; the steady Bedrock share is coffee money, around thirty cents a day. My one splurge was a $23 afternoon of re-embedding the entire corpus with richer context. Worth it exactly once.

Scar tissue

I'll keep this short, but skipping it entirely would turn the article into marketing. The first version kept its database in an embedded Postgres on EFS - a hard SIGKILL ate my index one evening, and that's how RDS earned its place. I once built a full Cognito login setup before admitting that memex serves programs, not people at login forms - tore it out the same week. And retrieval still misses sometimes, when I ask in words I never wrote down. That's exactly what the nightly self-test is for.

Why build instead of buy

There are real products in this space - memory SaaS, notes apps growing AI features, vector-database startups. I went the other way, for three reasons:

Ownership is the feature. Years of decisions, projects, and half-thoughts is the most personal dataset I have. A memory product that shuts down, changes pricing, or "updates its privacy policy" isn't a tool - it's a landlord.
The interface is a standard. MCP means my memory speaks the same protocol AI tools are adopting across the industry. Betting on a protocol beats betting on any single vendor's plugin ecosystem.
The work is the education. Every hard problem in this project - chunking, ranking fusion, fact supersession, tenant isolation, cost control - keeps showing up in real conversations within months, wherever AI tooling is on the table.

For plenty of people, buying is the right call. If your notes live happily in one app and you trust the vendor, go live your life. But if you're the kind of person who reads Terraform for fun, github.com/timurgaleev/memex is waiting. Expect an afternoon to a day of Terraform, DNS, and reading docs.

How my week actually works now

Back to the Forward Deployed Engineer thing, because memex doesn't work alone. Three tools, three jobs, and my week runs through all of them.

vibekit holds the rules - how my AI agents are allowed to behave, what they must never touch, synced into every tool I use from one repo. vibestack holds the workflows - the repeatable loops I run daily: plan a feature, review a diff, ship a release, investigate a bug. And memex holds the memory. Behavior, workflow, recall. Each would survive the others' deletion, which is my working definition of good boundaries.

A normal day: my phone buzzes at 07:00 with a morning briefing my server composed on its own - calendar, mail, house status. On the train I ask the brain something I half-remember about a side project. Somewhere in a conversation - at work, or around a friend's business - someone describes a process that eats two months a year, and something clicks: I've seen this shape before, and my own stack is where I learned to recognize it. In the evening I open a session on a personal project, the agent asks memex where things stand, and we continue where I left off instead of starting over.

That last part is the real change AI brought to my engineering life. Before, my evenings-and-weekends work was throwaway - too little time, too much re-explaining, projects abandoned at the setup stage. Now the setup is remembered, the workflows are one command, and a one-person R&D department is a realistic thing to run before breakfast. Whatever judgment I have about AI tooling - what's actually worth deploying, where it breaks, what it costs - comes from running all of this myself, watching it fail on a Tuesday, and fixing it before work.

Takeaways

AI tools without memory waste your time at scale. The re-explaining tax is invisible until you remove it; then it's obscene.
Retrieval and reasoning should be separate. A brain that returns cited evidence works with any agent that speaks the protocol. A chatbot works with nothing but itself.
Small infrastructure is a superpower. One ARM box, one managed Postgres, one tunnel. Boring enough to never think about, which is the whole point of infrastructure.
Privacy is a spectrum you should place yourself on deliberately - decide who you're defending against before you decide what to build or buy.

The repo is at github.com/timurgaleev/memex - Terraform, docs, and every scar described above. If you build your own brain, or already run one, I genuinely want to hear how you solved the parts I got wrong. Eighty years after Vannevar Bush sketched the memex, the desk-sized memory machine is real, it costs less than a gym membership, and it fits in a t4g.medium.

What would yours remember?

Code or diffusion? A field guide to programmatic image generation

Alexey Vidanov — Fri, 10 Jul 2026 17:54:43 +0000

A diagram is not a picture. It is a data structure that happens to be visible.

Treat it as pixels and you pay pixel prices. $0.02 to $0.20 per diffusion render. No source file. No source-level way to fix a typo: you need another generation or an image-editing step.

Most teams pay this by reflex, for things that were never really pictures: diagrams, charts, dashboards, labeled boxes.

There is a second path. The LLM writes code. A renderer turns the code into the image without another model call or image-generation API fee. Generation is probabilistic, but the accepted source artifact is not. Once stored, it is repeatable, editable, and diffable.

This is the field guide I wish I'd had. Three questions, asked in order, tell you which path to take and what it will cost. The rest of the series is the evidence behind each one.

Decision 1: is it structure, or is it texture?

This one question routes most cases.

Structure is anything you could describe with coordinates, shapes, labels, and data. Diagrams, flowcharts, architecture, charts, dashboards, icons, logos, most illustrations, 3D geometry. If a developer could draw it from a spec, an LLM can write it as code.

Texture is photoreal skin, brushstrokes, film grain, real faces, real places. For most prompt-driven workflows, code is the wrong abstraction for natural texture. Blender can render photorealism, but you pay for it in scene complexity and iteration time. This is diffusion's home turf.

The test: could you write instructions precise enough that two people would draw nearly the same thing? If yes, it is structure. Generate code. If it needs the "surprise me" of a trained image model, it is texture. Use diffusion.

There is a middle path. Put a stock or owned photo underneath, then add a code-generated overlay for the text and data that must be exact. Generating the overlay runs about $0.003, and the overlay carries validated text instead of asking an image model to reproduce it. That figure covers the overlay only. If the base image itself comes from diffusion, add its $0.02 to $0.20 on top.

The property nobody prices in

Before the next two decisions, one asymmetry that matters more than the per-image price.

Broken code fails loud. Invalid XML does not parse. A malformed SVG renders blank. The pipeline throws, CI goes red, nobody downstream sees it.

Valid code can still be wrong, and that failure is quiet too. An SVG with the label "Amazn" renders fine. So the loud failure only covers syntax. The durable advantage sits one level up: code exposes machine-checkable properties. Assert the SVG contains "eu-central-1". Assert the draw.io XML has exactly twelve nodes. Diff this week's chart against last week's and see the change, line by line. Wrong content becomes a test you can write once and run forever.

A bitmap exposes far fewer native checks. Validating one usually means reconstructing structure through OCR or computer vision, an extra model judging another model's output. In practice the misspelled label, the reversed arrow, and the phantom service render successfully and wait for a human eyeball. Many defects therefore become recurring manual or vision-based review tasks. You cannot grep a PNG.

Code does not guarantee correctness. It makes more of correctness testable. For anything that ends up in a customer deliverable, a compliance document, or an automated pipeline, that is the property to buy.

Decision 2: how much rendering power?

If it is structure, pick the lowest renderer class that covers the job. Every tier runs the same pattern: the LLM writes code, a renderer produces the visual without another image-generation API fee. You only pay for capability you use.

You need	Renderer	LLM writes
Static 2D vector, icons, illustration	SVG	markup
Charts, KPIs, parameterized visuals	SVG (templated)	markup + data
Diagrams with domain icons (AWS, etc.)	draw.io XML	XML + verified stencils
Particles, simulations, interactive 2D	Canvas / p5.js	JavaScript
3D in the browser	Three.js	JS scene
Math / explainer video	Manim	Python scene
Game scenes, levels	Godot (.tscn)	scene text
Photorealistic 3D stills	Blender	bpy Python
AAA-grade real-time frames	Unreal Engine	Python API

Two clarifications keep this honest.

First, the ladder is not a strict capability gradient. Godot is not "more capable" than Manim. Each tier unlocks a capability class the previous one cannot reach: templated data, domain stencils, animation, 3D, physics, photorealism. Pick by class, not by rank. Output tokens grow roughly an order of magnitude across the ladder, from a few hundred for a simple SVG to several thousand for a full 3D scene. Treat those as approximations. Complexity of the scene drives the count more than the renderer does.

Second, "no API fee" is not "free." SVG, draw.io, Canvas, and many Three.js scenes render in a browser with low operational overhead. Manim, Godot, Blender, and Unreal need a runtime and consume real compute: blender --background, godot --headless. That headless step is what turns this into a pipeline. Prompt in, PNG out, no human in the loop. AWS Batch or Amazon ECS is the right home for heavy Blender or video renders. AWS Lambda covers lightweight jobs like SVG-to-PNG rasterization that fit inside its 15-minute limit.

Decision 3: which model tier? Cost buys a ceiling

The law: cost equals output tokens times model rate. What the tier buys is a quality ceiling. I ran the same crayon-illustration prompt across the Amazon Bedrock lineup.

Model	Out tokens	Cost	What you get
Amazon Nova Pro	1,102	$0.004	valid but primitive, bare shapes
Claude Haiku 4.5	3,419	$0.014	structured, gradients, glow, animation
Claude Sonnet 5	6,128	$0.062	good composition, minor filter quirks
Claude Opus 4.8	4,089	$0.105	cleanest, most deliberate
Claude Fable 5	7,531	$0.381	most sophisticated, slowest

Same prompt, a roughly 95x cost range from Nova Pro to Fable. Two test prompts is a small sample, so read this as directional, not as law. Two observations survive even at this size.

Capability drives quality, not token count. Opus produced the cleanest result with fewer output tokens than Sonnet, 4,089 against 6,128. More tokens from a weaker model do not buy composition.

In this illustration test, returns diminished above Opus. Fable's result was the most sophisticated, but not four times better at four times the price. Across these two prompts, the Nova-to-Haiku jump produced the clearest quality gain per dollar, and the Opus-to-Fable jump the least.

The second prompt, a network-topology diagram, held the pattern. Nova Pro drew plain circles and lines for $0.007. Haiku added depth, labels, and animated data-flow dashes for $0.011.

Cheap is not free of consequences. Cheap is primitive.

The rule:

Nova or Haiku ($0.004 to $0.02): structural work where correct beats beautiful. Wireframes, placeholders, internal diagrams, high-volume batches.
Sonnet or Opus ($0.06 to $0.16 at current rates): anything that has to impress. Editorial art, public hero visuals, polished technical figures.
Fable ($0.38): one-off showpieces where the sophistication earns the price and the wait.

Now scale it. At the measured $0.004 inference cost, 1,000 outputs of comparable token length would run about $4 in model spend, excluding retries, validation, and rendering infrastructure. And every output is a diffable text asset in Git. The same volume through diffusion runs $20 to $200 in per-image fees and yields 1,000 opaque bitmaps you can neither edit nor verify.

Methodology. Output tokens and latency are measured; dollar figures are calculated from Amazon Bedrock on-demand rates, July 2026, us-east-1. Two rates need flags. The Sonnet 5 figure uses launch pricing of $2/$10 per million input/output tokens, valid through August 31, 2026; at the standard $3/$15 rate from September 1 the same run costs about $0.093. Fable 5 was measured after access was restored on July 1, 2026. Using it requires opting in to provider_data_share; AWS states that inputs and outputs may be retained for up to 30 days for abuse detection and potential human review. For regulated workloads, treat that as an architectural decision, not an invocation detail.

The skeleton trick

One technique cheats the cost-quality trade-off entirely. Hand a cheap model a well-designed SVG skeleton and let it fill in only the data. You get premium-looking output at Nova-tier cost, because the polish lives in the template you wrote once, not in tokens you buy every call. It follows the same economic principle as prompt caching: amortize the expensive, stable part across many cheaper operations.

This also shrinks the failure surface. The model no longer designs the image. It fills slots in a structure you already verified. Less freedom, fewer ways to be wrong.

The one-page matrix

Measured dollar figures live in the benchmark above. The matrix uses cost classes because model cost tracks scene complexity, not just the path. Anchors from the benchmark: very low is under $0.01, medium is a few cents, high approaches $0.10 per image.

Need	Path	Model tier	Model cost	Editable	Runtime
Architecture / flow diagram	draw.io XML	Haiku to Sonnet	Low to medium	Full XML	Browser
Chart / KPI / dashboard	SVG (templated)	Nova to Haiku	Very low	Full	Browser
Icon / logo mark	SVG	Nova	Very low	Full	Browser
Editorial illustration	SVG + style doc	Sonnet to Opus	Medium to high	Full	Browser
Generative / creative art	Canvas / p5.js	Sonnet	Medium	Full	Browser
3D product / scene	Three.js / Blender	Sonnet to Opus	Highly variable	Full	Browser / CLI
Math explainer video	Manim	Sonnet	Variable	Full	Python + ffmpeg
Game asset / level	Godot .tscn	Sonnet	Variable	Full	Godot
Photoreal person / place	Diffusion	n/a	Per-image, $0.02-0.20	No	API
Photo + exact text overlay	Hybrid (photo + SVG)	Haiku	Very low + base image	Partial	Browser

When diffusion wins

Use diffusion when the value is in the texture. Photoreal people and places. Painterly surfaces, ink bleed, film grain. The happy accident of a trained model.

Use it for exploration. Ten variations of a mood board in thirty seconds is something no renderer gives you.

And use it when the image is a genuine one-off that nobody will ever edit, version, or audit. If machine-checkability is worth nothing to you, you are not losing anything by giving it up.

The point is not purity. It is spending diffusion money only where diffusion earns it, and letting code handle the structured majority for cents.

The evidence

Each article in the series is one branch of this framework, with runnable Amazon Bedrock code:

The cost reality: the floor, and where it breaks.
Editorial illustration: one style document, consistent plates.
AWS architecture diagrams: 270+ verified draw.io stencils.
The renderer spectrum: SVG through Unreal.
The model benchmark: measured cost against polish, Nova to Fable.

Live demo with a cost calculator and gallery: labs.p.awsnavigator.com/code-as-canvas. Source: github.com/vidanov/llm-programmatic-image-gen.

Built with Claude on Amazon Bedrock. Alexey Vidanov · GitHub · dev.to

Building Production AI Agents on AWS Bedrock — Architecture and Code Decisions Worth Keeping in Mind

Jorge Tovar — Fri, 10 Jul 2026 16:13:33 +0000

The Context

Models are stateless. They process one request, produce a text result, and forget. They don't take actions, don't integrate by default with your data, or coordinate complex workflows on their own. Agents solve this by harnessing a model in a runtime system that gives it tools, memory, MCPs, guardrails, hooks and a reasoning loop.

This application is a serverless rental management bot — landlords send natural-language messages and receive notifications through Telegram and email. The best part is that analysis of current payments, debts, and trends is done by an agent powered by Claude. It gives you insights about your data, it remembers your preferences, it's a real collaborator. That's what every agent can do. A chatbot is just text with no deep integrations with your data or systems — by definition, less capable.

Code on GitHub: https://github.com/jorgetovar/whatsapp-rental-manager

Agents are easy to build but not simple. We should keep in mind everything we've always done in backend systems and production applications: security, integrations, cohesion, coupling, data management, scalability, reliability, costs, operational excellence, latency and performance — all the non-functional requirements still apply.

Architecture Learnings

1. Bedrock over calling the model API directly

The operational answer: IAM authentication. There's no API key to rotate, leak, or store in Secrets Manager. The same boto3 session pattern you use for DynamoDB works for Bedrock. Cross-region inference (us.anthropic.claude-sonnet-4-6) routes to the lowest-latency US region automatically and gives higher throughput limits than a single-region endpoint.

def create_agent(model_id: str = None, extra_context: str = "") -> Agent:
    model_id = model_id or os.environ.get("BEDROCK_MODEL_ID", "us.anthropic.claude-sonnet-4-6")
    boto_session = boto3.Session(
        profile_name=os.environ.get("AWS_PROFILE"),
        region_name=os.environ.get("AWS_REGION_NAME", "us-east-1"),
    )
    model = BedrockModel(model_id=model_id, boto_session=boto_session)
    return Agent(model=model, system_prompt=SYSTEM_PROMPT + extra_context, tools=[...])

2. Agent memory

Lambda is stateless. Every invocation starts cold. To give Claude memory of the conversation, history must be persisted externally and rehydrated on each request.

I'm using DynamoDB to keep track of messages, but there are better options — AWS Strands has built-in session handling that can summarize history or keep an exact number of messages, which is what we're doing right now.

_MAX_MESSAGES = 20    # ~10 turns — enough context, safe DynamoDB item size
_TTL_SECONDS = 86400  # 24h — conversations reset overnight

def save_history(chat_id: str, messages: list) -> None:
    trimmed = _sanitize_messages(_strip_metadata(messages)[-_MAX_MESSAGES:])
    get_table("conversations").put_item(Item={
        "chatId": str(chat_id),
        "messages": json.dumps(trimmed),
        "ttl": int(time.time()) + _TTL_SECONDS,
    })

DynamoDB TTL does the cleanup automatically — no cron job, no cost.

3. Dynamic context injection vs. conversation history

There are two ways to give the model current state: inject it into the system prompt per request, or let it accumulate in conversation history.

Conversation history is unreliable for state. It gets trimmed. It contains noise. The model's attention decays over long contexts. If the landlord added a new property three turns ago, you can't guarantee the model remembers it now.

The pattern that works:

def _build_user_context(caller: dict) -> str:
    landlord = get_table("landlords").get_item(Key={"landlordId": landlord_id}).get("Item")
    active = sum(1 for l in leases if l.get("status") == "active")
    return (
        "\n\nESTADO DEL USUARIO (fijado por el sistema):\n"
        f"- Rol: ARRENDADOR — {name}\n"
        f"- Inmuebles activos: {active}\n"
        f"- Ya aceptó política de datos. NO repitas bienvenida."
    )

agent = create_agent(extra_context=user_ctx)
agent.messages = load_history(chat_id)
response = agent(text)

4. Tool design: the docstring is the API contract

With the Strands SDK, tools are Python functions decorated with @tool. The docstring is not documentation — it's the API contract with the model. Be explicit. This steers the agent in the right direction and produces reliable tool calls.

@tool
def log_payment(
    tenant_name_or_phone: str,
    amount: int,
    period: str = "",
    method: str = "other",
) -> str:
    """Registra un pago recibido de un inquilino.
    Ej: tenant_name_or_phone='Juan', amount=800000.
    period en YYYY-MM (vacío = mes actual).
    method: nequi/daviplata/efectivo/other
    """
    ...

When the user says "Juan pagó 800k", Claude infers log_payment(tenant_name_or_phone="Juan", amount=800000) because the docstring is explicit about what each parameter means. Vague docstrings produce wrong tool calls.

5. Security in agents: the model is not the trust boundary

The most dangerous pattern in agent design: letting the model decide who the user is or what they're allowed to do.

In this system, identity comes from the phone number, which Telegram provides on every webhook. Role is resolved from DynamoDB by phone — never from model input or output.

def resolve_caller_sync(phone: str) -> list[CallerContext]:
    contexts: list[CallerContext] = []

    # Check if registered landlord
    resp = landlords_table.get_item(Key={"landlordId": phone})
    if resp.get("Item"):
        contexts.append(CallerContext(role="landlord", landlordId=phone))

    # Check if active renter
    for item in paginated_query(leases_table, IndexName="tenantPhone-index", ...):
        if item.get("status") == "active":
            contexts.append(CallerContext(role="renter", landlordId=item["landlordId"], ...))

    return contexts

It's the same principle as a traditional backend application where a JWT token travels in every request and you extract the claims to get the customer_id — you never accept it from the request body, because that would be a spoofing vulnerability.

Webhook requests are also verified with a timing-safe HMAC check before any processing happens:

def _verify_signature(secret: str, header: str) -> bool:
    return hmac.compare_digest(secret, header or "")

hmac.compare_digest takes constant time regardless of where the strings differ — an attacker can't probe the secret byte-by-byte by measuring response latency.

6. Infrastructure as Code

Individual Lambdas get exactly the permissions they need — the late-tenant reminder reads leases and tenants; it can't write. Deployments are a single command and that's it. Anyone can replicate this application easily.

Policies:
  - DynamoDBReadPolicy:       # read-only for reminder Lambdas
      TableName: !Ref LeasesTable
  - DynamoDBCrudPolicy:       # full CRUD for the webhook Lambda
      TableName: !Ref PaymentsTable
  - !Ref SesPolicy            # shared managed policy across all reminder functions

Secrets Manager dynamic references resolve at deploy time:

TELEGRAM_BOT_TOKEN: !Sub '{{resolve:secretsmanager:miarriendobot/telegram:SecretString:token}}'

Note: the secret value is baked into the Lambda environment variable at deploy time. Rotating the secret in Secrets Manager does not update the Lambda automatically — you must redeploy or update the environment variable directly.

7. Observability

Agents are not deterministic. Now more than ever it's important to understand what's happening in the system — what the integration points are, whether something is failing, how to get notified. Having alarms, tracing, and good logs is always paramount.

One non-obvious gap: the webhook Lambda must always return HTTP 200 — Telegram retries delivery indefinitely on anything else. The side effect is that API Gateway always looks healthy even when every invocation is throwing an exception. Standard uptime monitoring is completely blind here.

WebhookErrorAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    MetricName: Errors
    Namespace: AWS/Lambda
    Dimensions:
      - Name: FunctionName
        Value: !Ref WebhookFunction
    Statistic: Sum
    Period: 300
    EvaluationPeriods: 1
    Threshold: 3
    ComparisonOperator: GreaterThanOrEqualToThreshold
    AlarmActions:
      - !Ref AlertTopic

A CloudWatch alarm on the Lambda Errors metric is the only real signal. Without it, you find out the bot is broken from a user complaint, not a page.

8. Data model design for DynamoDB

The denormalization decision is a real trade-off. Storing canon directly on the Lease even though Property also has it means reminder Lambdas fetch one item and have everything — no second read, no join.

class Lease(BaseModel):
    leaseId: str
    landlordId: str
    propertyId: str
    tenantPhone: str
    canon: int      # duplicated from Property — intentional: reminders need one read
    dueDay: int
    startDate: str
    status: str = "active"

The explicit cost: update_canon must write both tables. DynamoDB is not relational — design for your read patterns, not for normalization.

9. Testability

Any job gated on a calendar date is untestable in CI without a bypass parameter. job_monthly_summary(force=True) runs immediately regardless of the date. Design for testability from day zero.

def job_late_tenant_reminder(force: bool = False):
    today = datetime.now(timezone.utc)
    for lease in _get_all_active_leases():
        reminder_date = _due_date_this_month(due_day) + timedelta(days=5)
        if not force and today.date() != reminder_date.date():
            continue  # skip — not the right day
        # send reminder...

The same function runs locally via APScheduler and in production via EventBridge — no mocking, full parity.

10. AWS credentials — the one-liner that works in both environments

boto3.Session(profile_name=os.environ.get("AWS_PROFILE"))

This single line works in both local development and Lambda without any branching. The key is os.environ.get("AWS_PROFILE") — when AWS_PROFILE is not set in Lambda, .get() returns None, and boto3.Session(profile_name=None) correctly falls back to the IAM execution role via IMDS.

Two mistakes to avoid:

os.environ.get("AWS_PROFILE", "") returns "" instead of None — boto3 then tries to find a profile named "" and fails silently.
Never set AWS_PROFILE as a Lambda environment variable — boto3 will look for a credentials file that doesn't exist in Lambda's runtime.

Locally, your .env sets AWS_PROFILE=default-c1 and the named profile is used. In Lambda, the variable is absent and the execution role takes over automatically. No if local else lambda branching, no hardcoded profile names, no credentials in code.

11. Context is the most important part of any agent

Tools and good context make agents powerful. We pass just enough — but important — signals that allow for a coherent conversation. The model doesn't need everything; it needs the right things at the right time.

- Rol: ARRENDADOR — Jorge Tovar
- Inmuebles activos: 2
- Ya aceptó política de datos. NO repitas bienvenida.

The model knows the user's role, their current state, and what it should not ask for. That's the entire context it needs to behave correctly.

12. Never trim conversation history mid-tool-call

Always cut at complete exchange boundaries — never in the middle of a toolUse/toolResult pair. We shipped without this and Bedrock raised a ValidationException at turn 10. The fix walks the trimmed list and drops any incomplete exchange from the tail:

def _sanitize_messages(messages: list) -> list:
    # Drop leading user messages that only contain toolResult blocks
    # (their corresponding assistant toolUse was trimmed away)
    while start < len(messages):
        msg = messages[start]
        has_text = any(isinstance(b, dict) and "text" in b for b in msg.get("content", []))
        if msg.get("role") != "user" or has_text:
            break
        start += 1

    # Drop any assistant toolUse not matched by the next message's toolResult
    if tool_use_ids and not tool_use_ids.issubset(result_ids):
        break  # missing toolResult — drop from here

The end of a complete reasoning loop is the only safe trim point.

13. Hooks are the preferred and deterministic way to validate

The model shouldn't be responsible for things that can be validated deterministically in code. Request-scoped context is set once at the start of every invocation and propagated automatically to every tool — no threading required, safe because Lambda processes one request at a time per execution environment.

# lambda_handler.py — set before every agent call
CURRENT_CONTEXT.clear()
CURRENT_CONTEXT.update(caller.model_dump())

# any tool — reads from the same dict
def _landlord_id() -> str:
    return CURRENT_CONTEXT.get("landlordId", "")

The model receives that string, understands it's a business rule, and responds to the user naturally — "Sorry, that feature is only available for landlords."

14. Structured error responses matter

Make it explicit whether an error is retryable or a business rule violation. The model handles them differently, and so does the user.

Conclusion

Building agents on AWS Bedrock is not fundamentally different from building any production backend system — the non-functional requirements don't disappear because there's a language model in the middle. Security, observability, testability, and data integrity matter just as much, and in some cases more, because agent behavior is harder to predict than deterministic code.

The code is open source — if you're building something similar, the architecture is yours to take and adapt.

There has never been a better time to be an engineer and create value in society through software.

If you enjoyed the articles, visit my blog at jorgetovar.dev.

Hitting the Iceberg REST Catalog Directly: Understanding the Differences Between Glue Data Catalog and S3 Tables

Aki — Fri, 10 Jul 2026 03:32:10 +0000

Original Japanese article: Iceberg REST Catalogを直接叩いて、Glue Data CatalogとS3 Tablesの違いを理解する

Introduction

I'm Aki, an AWS Community Builder (@jitepengin).

Most of the time, when working with Iceberg tables, we reach for PyIceberg or Spark. I'm no exception, and honestly there were parts of the PyIceberg configuration — rest.sigv4-enabled, rest.signing-name, warehouse — that I understood only vaguely.

Iceberg defines a standard called the Iceberg REST Catalog Open API specification, and AWS implements it through two separate endpoints:

The AWS Glue Iceberg REST endpoint (https://glue.<region>.amazonaws.com/iceberg)
The Amazon S3 Tables Iceberg REST endpoint (https://s3tables.<region>.amazonaws.com/iceberg)

If two implementations follow the same spec, sending the same requests to both and comparing the results should reveal what's actually different between them.

In this article, I'll bypass clients like PyIceberg entirely and hit the REST API directly to explore the differences between the two endpoints.

To state the conclusion up front:

Even though both implement the same Iceberg REST Catalog specification, Glue is designed as an "entry point to multiple catalogs," while S3 Tables is designed as an "entry point to a single table bucket." That difference is visible just by looking at the URL paths.

I previously wrote about the relationship between S3 Tables and Glue Data Catalog in another article — worth a read alongside this one:
Does Amazon S3 Tables Replace AWS Glue Data Catalog? Understanding Their Relationship

What Is the Iceberg REST Catalog?

The Iceberg REST Catalog is a specification that standardizes Iceberg catalog operations as an HTTP API. It's published as an OpenAPI definition (YAML), and any catalog that conforms to it can be accessed the same way from clients such as PyIceberg, Spark, and Trino.

The key points of the spec are:

URL paths follow a pattern like GET /v1/{prefix}/namespaces, where {prefix} is a free-form segment
Clients first call GET /v1/config to retrieve endpoint configuration (the default prefix and other settings)
The table metadata body (schema, snapshots, etc.) is returned as JSON in the LoadTable response

In other words, when you write catalog.load_table("ns.table") in PyIceberg, what's actually happening under the hood is an HTTP request to GET /v1/{prefix}/namespaces/ns/tables/table.

Here's a summary of the two AWS implementations before we dive in:

Item	Glue Iceberg REST endpoint	S3 Tables Iceberg REST endpoint
Endpoint	`https://glue.<region>.amazonaws.com/iceberg`	`https://s3tables.<region>.amazonaws.com/iceberg`
Contents of `{prefix}`	`/catalogs/{catalog}` (catalog hierarchy)	URL-encoded table bucket ARN
Value passed as `warehouse`	Glue catalog ID	Table bucket ARN
SigV4 signing service name	`glue`	`s3tables`
Access control	IAM + Lake Formation	s3tables IAM actions only

Here's the overall picture as a diagram:

              Iceberg REST Catalog spec
           GET /v1/{prefix}/namespaces/...
                        │
        ┌───────────────┴───────────────┐
        │                               │
  Glue endpoint                  S3 Tables endpoint
        │                               │
 prefix = /catalogs/{catalog}    prefix = table bucket ARN
 (multi-catalog hierarchy)       (one bucket = one catalog)
        │                               │
  IAM + Lake Formation           s3tables IAM actions
        │                               │
        └───────────────┬───────────────┘
                        │
              Same Iceberg table
          (points to the same metadata.json)

Note: a catalog doesn't manage the metadata.json file itself — it provides a reference to where the latest metadata-location is. The catalog's essential job is knowing where the current metadata.json lives.

Setting Up the Test Environment

Let's create a test table bucket, namespace, and table via the CLI. Read the region as Tokyo (ap-northeast-1) and the account ID as 123456789012.

# Create a table bucket
aws s3tables create-table-bucket \
  --name penguin-rest-test \
  --region ap-northeast-1

# Create a namespace
aws s3tables create-namespace \
  --table-bucket-arn arn:aws:s3tables:ap-northeast-1:123456789012:bucket/penguin-rest-test \
  --namespace analytics

# Create a table (with schema)
aws s3tables create-table \
  --table-bucket-arn arn:aws:s3tables:ap-northeast-1:123456789012:bucket/penguin-rest-test \
  --namespace analytics \
  --name daily_sales \
  --format ICEBERG \
  --metadata '{
    "iceberg": {
      "schema": {
        "fields": [
          {"name": "sales_date", "type": "date", "required": false},
          {"name": "amount", "type": "long", "required": false}
        ]
      }
    }
  }'

SigV4 Signing

You can't hit these endpoints with plain curl.

The Iceberg REST Catalog spec defines an OAuth2-based authentication flow, but AWS's implementation uses IAM SigV4 signing instead — a standard-spec API with AWS-flavored authentication.

That rest.sigv4-enabled: true setting in PyIceberg is exactly what enables this signing.

Computing SigV4 signatures by hand is painful, so I used awscurl for this exercise.

pip install awscurl

awscurl picks up credentials from environment variables or a profile and sends SigV4-signed requests for you.

The important part is the --service option, which must specify the correct service name for signing: glue for the Glue endpoint, s3tables for the S3 Tables endpoint.

Hitting the S3 Tables Iceberg REST Endpoint

Let's start with the simpler of the two — the S3 Tables endpoint.

GET /v1/config

getConfig is the first API a REST Catalog client calls. The warehouse query parameter takes the table bucket ARN.

Note

When using awscurl, pass the raw, unencoded ARN as the query parameter value. awscurl automatically URL-encodes query parameters before computing the signature, so if you pre-encode the value yourself, it gets double-encoded and you'll get a SignatureDoesNotMatch error.

BUCKET_ARN="arn:aws:s3tables:ap-northeast-1:123456789012:bucket/penguin-rest-test"

awscurl --service s3tables --region ap-northeast-1 \
  "https://s3tables.ap-northeast-1.amazonaws.com/iceberg/v1/config?warehouse=${BUCKET_ARN}"

Result (excerpt):

{
  "defaults": {
    "prefix": "arn%3Aaws%3As3tables%3Aap-northeast-1%3A123456789012%3Abucket%2Fpenguin-rest-test",
    "io-impl": "org.apache.iceberg.aws.s3.S3FileIO",
    "write.object-storage.enabled": "true",
    "write.object-storage.partitioned-paths": "false",
    "s3.delete-enabled": "false",
    "rest-metrics-reporting-enabled": "false"
  },
  "overrides": {}
}

The interesting part is the prefix in the response. For the S3 Tables endpoint, prefix is exactly the URL-encoded table bucket ARN, and it lives under defaults. In other words, this endpoint is designed around a "one table bucket = one catalog" model — if you want to work with multiple table buckets, you register multiple catalogs on the client side.

defaults also included the write-side FileIO implementation (io-impl), object storage layout settings (write.object-storage.*), and a setting that disables S3 delete operations (s3.delete-enabled). It's interesting to see, from actual output, that some of the parameters we're used to configuring individually on the client (PyIceberg) side are actually being pushed down as server-side defaults through this config response.

Incidentally, this getConfig call is authorized under the s3tables:GetTableBucket IAM action. The official documentation includes a mapping table showing which s3tables IAM action corresponds to each REST operation.

Note

The spec's CatalogConfig also defines an endpoints field, which lets the server return a list of supported endpoints (in a format like "GET /v1/{prefix}/namespaces"). When I checked, the S3 Tables endpoint's response did not include an endpoints field.

Listing Namespaces and Tables

Now that we know the prefix, let's list namespaces and tables.

Note

There's a subtlety worth calling out here.

The warehouse query parameter for GET /v1/config needs the raw ARN, since awscurl automatically URL-encodes query parameters before signing — pre-encoding it yourself causes double-encoding and a SignatureDoesNotMatch error.

The {prefix} segment in the path, however, behaves differently. If you put the ARN into the path completely unencoded (colons and slashes as-is), the / characters inside the ARN get interpreted as path separators, and the request no longer matches the modeled URL pattern /v1/{prefix}/namespaces — you get an UnknownOperationException.

The correct approach is to leave the colons raw but percent-encode only the single / inside bucket/<table-bucket-name> as %2F. That makes the request match the correct route (and once it reaches the authorization layer, you'll get a 403 if you lack permissions, rather than a routing error).

Even though it's the same operation — putting an ARN into a URL — the encoding rules differ between query parameters and path segments. That's something you only really notice by actually hitting the endpoint.

BUCKET_ARN_PATH="arn:aws:s3tables:ap-northeast-1:123456789012:bucket%2Fpenguin-rest-test"

# List namespaces
awscurl --service s3tables --region ap-northeast-1 \
  "https://s3tables.ap-northeast-1.amazonaws.com/iceberg/v1/${BUCKET_ARN_PATH}/namespaces"

# List tables
awscurl --service s3tables --region ap-northeast-1 \
  "https://s3tables.ap-northeast-1.amazonaws.com/iceberg/v1/${BUCKET_ARN_PATH}/namespaces/analytics/tables"

Result (excerpt):

{
  "namespaces": [
    ["analytics"]
  ]
}

{
  "identifiers": [
    {
      "name": "daily_sales",
      "namespace": ["analytics"]
    }
  ]
}

There's one difference from the spec worth noting here: the Iceberg REST spec allows multi-level namespaces (like a.b.c), but S3 Tables only supports a single level.

Let's confirm this by trying to create a multi-level namespace:

awscurl --service s3tables --region ap-northeast-1 \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"namespace": ["level1", "level2"]}' \
  "https://s3tables.ap-northeast-1.amazonaws.com/iceberg/v1/${BUCKET_ARN_PATH}/namespaces"

Result (excerpt):

{
  "error": {
    "code": 400,
    "message": "Multipart namespaces are not supported.",
    "type": "bad_request"
  }
}

The error message spells it out directly — "multipart namespaces are not supported" — confirming the single-level restriction empirically.

Retrieving Metadata with LoadTable

This is the API I most wanted to check today.

awscurl --service s3tables --region ap-northeast-1 \
  "https://s3tables.ap-northeast-1.amazonaws.com/iceberg/v1/${BUCKET_ARN_PATH}/namespaces/analytics/tables/daily_sales"

Result (excerpt):

{
  "metadata-location": "s3://34c72c19-610d-4e5c-d8d1qwx3db1p3tbr8g9jch9n6e14sapn1b--table-s3/metadata/00000-163a447a-c64d-44f9-8619-5db9561e549b.metadata.json",
  "metadata": {
    "format-version": 2,
    "table-uuid": "9507ea2b-7a71-40a7-b6a9-1d6e633955e1",
    "location": "s3://34c72c19-610d-4e5c-d8d1qwx3db1p3tbr8g9jch9n6e14sapn1b--table-s3",
    "current-schema-id": 0,
    "schemas": [
      {
        "type": "struct",
        "schema-id": 0,
        "fields": [
          { "id": 1, "name": "sales_date", "required": false, "type": "date" },
          { "id": 2, "name": "amount", "required": false, "type": "long" }
        ]
      }
    ],
    "default-spec-id": 0,
    "partition-specs": [{ "spec-id": 0, "fields": [] }],
    "properties": {
      "write.parquet.compression-codec": "zstd"
    },
    "current-snapshot-id": -1,
    "snapshots": []
  },
  "config": {
    "tableBucketId": "3b2a6702-dd99-44f2-bc03-2f9f6b273104",
    "namespaceId": "20576faa-861e-499f-a30f-4d6c7550dd55",
    "tableId": "34c72c19-610d-4e5c-bee4-6ae9f093c583"
  }
}

This JSON is exactly what's behind the information we normally see through table.schema() or table.snapshots() in PyIceberg. The response directly confirms that a catalog's essential job is "managing and returning the S3 path of the latest metadata.json" (metadata-location).

Incidentally, the config field in the response contained S3 Tables-internal identifiers: tableBucketId, namespaceId, and tableId. Per spec, the Glue endpoint's config is supposed to include temporary credentials (vended credentials), but here on the S3 Tables endpoint there's no credential delegation at all — it just returns internal management IDs.

Note

The S3 Tables endpoint also documents a limit: operations against a table whose metadata.json exceeds 50MB return a 400 error. Worth keeping in mind — if metadata bloats from accumulated snapshots and similar growth, you could hit this API-level ceiling.

Checking What's Happening Under the Hood via CloudTrail

An interesting characteristic of the S3 Tables endpoint is that REST API calls get logged in CloudTrail as their corresponding native S3 Tables actions. Per the official documentation, a single LoadTable call logs both of the following:

GetTableMetadataLocation (a management event)
A data event corresponding to GetTableData

In other words, the audit log itself reveals that this endpoint is implemented as a proxy that translates the Iceberg REST API into native S3 Tables API calls.

Let's check CloudTrail directly:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=GetTableMetadataLocation \
  --region ap-northeast-1 \
| jq '.Events[0] | {time: .EventTime, user: .Username, event: .EventName}'

Result (excerpt):

{
  "time": "2026-07-09T13:46:44+00:00",
  "user": "penguin-test-user",
  "event": "GetTableMetadataLocation"
}

Even though we only called LoadTable over REST, CloudTrail logs it under the native S3 Tables event name s3tables:GetTableMetadataLocation, complete with the exact caller (IAM username).

Hitting the Glue Iceberg REST Endpoint

Now let's look at the Glue endpoint, which behaves quite differently.

GET /v1/config

awscurl --service glue --region ap-northeast-1 \
  "https://glue.ap-northeast-1.amazonaws.com/iceberg/v1/config?warehouse=123456789012"

Result (excerpt):

{
  "defaults": {
    "prefix": "123456789012",
    "header.Content-Type": "application/x-amz-json-1.1",
    "rest.sigv4-enabled": "true",
    "rest.signing-name": "glue",
    "rest.signing-region": "ap-northeast-1",
    "rest-table-scan-enabled": "true",
    "rest-data-commit-enabled": "true",
    "token-refresh-enabled": "false"
  },
  "overrides": {
    "prefix": "catalogs/123456789012"
  }
}

There's a difference from the S3 Tables endpoint worth calling out here: defaults.prefix is "123456789012" (just the account ID), while overrides.prefix is "catalogs/123456789012" (with catalogs/ prepended) — the two values genuinely differ. Per spec, overrides takes precedence over defaults, so the prefix a client should actually use is catalogs/123456789012. It's confirmed, from actual output, that defaults and overrides really can disagree.

The other defaults entries are worth a look too — rest.sigv4-enabled and rest.signing-name map directly onto the PyIceberg configuration items of the same name. rest-table-scan-enabled and rest-data-commit-enabled are presumably feature flags for server-side scan planning and write commits, discussed further below.

Incidentally, this response also didn't include an endpoints field.

What you pass as warehouse for the Glue endpoint is a Glue catalog ID (defaulting to the current account's root catalog if omitted). Where the S3 Tables endpoint's warehouse was "a bucket ARN," here it's "where in the catalog hierarchy to connect."

Prefix Rules: Encoding the Catalog Hierarchy

The Glue endpoint's prefix always takes the form /catalogs/{catalog}. The official documentation defines the following rules for how {catalog} is written:

Target	Prefix notation	Example REST path
Default catalog of the current account	`:`	`GET /v1/catalogs/:/namespaces`
Default catalog of a specific account	Account ID	`GET /v1/catalogs/123456789012/namespaces`
Nested catalog	`catalog1:catalog2`	`GET /v1/catalogs/rmscatalog1:db1/namespaces`
Nested catalog of a specific account	`accountId:catalog1:catalog2`	`GET /v1/catalogs/123456789012:s3tablescatalog:bucket/namespaces`

The : notation for the default catalog is confusing at first glance, but it makes more sense once you think of it as "encoding the catalog hierarchy's separator as : instead of the path separator /." It's essentially using the free-form nature of the Iceberg REST spec's prefix to cram Glue's multi-catalog hierarchy into the URL.

Listing Namespaces (Default Catalog)

awscurl --service glue --region ap-northeast-1 \
  "https://glue.ap-northeast-1.amazonaws.com/iceberg/v1/catalogs/123456789012/namespaces"

Note

The default catalog can be expressed either as a single colon (:) or as the account ID. Here I'm using the catalogs/{accountId} form, which is also what showed up in overrides.prefix in the getConfig response above.

Result (excerpt):

{
  "namespaces": [
    ["hive"],
    ["icebergdb"]
  ]
}

What comes back is the familiar list of Glue databases — the response directly confirms the mapping "Glue database = Iceberg namespace." Notably, S3 Tables-backed databases under s3tablescatalog are not included here. The default catalog only covers Glue databases directly under the account; to see S3 Tables namespaces, you need to explicitly reach them via the nested catalog prefix, described below.

Incidentally, the Glue endpoint has the same single-level namespace restriction as S3 Tables. What looks like a multi-level structure isn't expressed by making namespaces deeper — it's expressed by nesting catalogs. That seems to be a consistent design choice on Glue's part.

Reading an S3 Tables Table Through the Glue Endpoint

This is the heart of today's investigation: reading the same table we read via the S3 Tables endpoint, this time through the Glue endpoint.

S3 Tables with Glue integration enabled is mounted under the s3tablescatalog federated catalog. In PyIceberg, you'd specify warehouse as 123456789012:s3tablescatalog/penguin-rest-test. Applying the earlier prefix-conversion rule, the path becomes:

awscurl --service glue --region ap-northeast-1 \
  "https://glue.ap-northeast-1.amazonaws.com/iceberg/v1/catalogs/123456789012:s3tablescatalog:penguin-rest-test/namespaces/analytics/tables/daily_sales"

Result (excerpt):

{
  "config": {
    "createdBy": "123456789012",
    "s3TableArn": "arn:aws:s3tables:ap-northeast-1:123456789012:bucket/penguin-rest-test/table/34c72c19-610d-4e5c-bee4-6ae9f093c583",
    "ownerAccountId": "123456789012",
    "metadata_location": "s3://34c72c19-610d-4e5c-d8d1qwx3db1p3tbr8g9jch9n6e14sapn1b--table-s3/metadata/00000-163a447a-c64d-44f9-8619-5db9561e549b.metadata.json",
    "format": "ICEBERG",
    "warehouse_location": "s3://34c72c19-610d-4e5c-d8d1qwx3db1p3tbr8g9jch9n6e14sapn1b--table-s3",
    "table_type": "ICEBERG"
  },
  "metadata": {
    "format-version": 2,
    "table-uuid": "9507ea2b-7a71-40a7-b6a9-1d6e633955e1",
    "current-schema-id": 0,
    "schemas": [
      {
        "type": "struct",
        "schema-id": 0,
        "fields": [
          { "id": 1, "name": "sales_date", "required": false, "type": "date" },
          { "id": 2, "name": "amount", "required": false, "type": "long" }
        ]
      }
    ],
    "current-snapshot-id": -1
  },
  "metadata-location": "s3://34c72c19-610d-4e5c-d8d1qwx3db1p3tbr8g9jch9n6e14sapn1b--table-s3/metadata/00000-163a447a-c64d-44f9-8619-5db9561e549b.metadata.json"
}

Note

The nested-catalog prefix notation used above (accountId:s3tablescatalog:bucketName) was derived from the conversion-rule table in the official documentation. Sending an actual request in this format resulted in correct routing, and both authorization and the underlying data checked out.

The metadata-location and table-uuid (9507ea2b-7a71-40a7-b6a9-1d6e633955e1) here match exactly what we retrieved via the S3 Tables endpoint earlier. Different entry point, same underlying entity.

The config contents are worth noting too. In this request — where we didn't set the x-iceberg-access-delegation header — config didn't contain S3 access credentials, only metadata like s3TableArn (the original S3 Tables ARN) and warehouse_location (the actual S3 path). As I'll cover below, one more configuration step was needed before credentials would actually come back.

For the same table: through the S3 Tables endpoint the path is /v1/{bucketArn}/namespaces/analytics/tables/daily_sales, while through the Glue endpoint it's /v1/catalogs/{accountId}:s3tablescatalog:{bucketName}/namespaces/analytics/tables/daily_sales. Physically they point to the same metadata.json, but the path structures are completely different because the two services have entirely different mental models of "what a catalog is." That, I think, is the essential difference between the two endpoints.

Access Control Differences (Lake Formation)

Accessing via the Glue endpoint requires Lake Formation grants in addition to IAM policies (glue:GetCatalog, glue:GetTable, etc.), because S3 Tables tables get registered as Lake Formation resources when Glue integration is enabled.

Further, for an external engine to read the actual data, you need to enable full-table access for external engines in Lake Formation, and allow the IAM role to call lakeformation:GetDataAccess. This mechanism is what issues the temporary credentials known as vended credentials.

In practice, though, this alone wasn't enough. It took explicitly registering the table bucket as a federated Lake Formation resource via aws lakeformation register-resource --with-federation, and setting up a trust policy so the Lake Formation service itself (lakeformation.amazonaws.com) could assume the IAM role used for issuing credentials, before vended credentials actually started being issued. If this registration step is missing, requests with the header described below (even with otherwise-correct IAM policies and Lake Formation grants) just come back with a credential-less config — only s3TableArn and warehouse_location — with no error at all, which makes it an easy thing to miss.

When PyIceberg hits the Glue endpoint, it attaches this header to the request:

x-iceberg-access-delegation: vended-credentials

Per the Iceberg REST spec, this header signals to the server that the client wants credential delegation. However — at least in this environment — as long as the federation registration was complete, calling LoadTable without this header still returned credentials the same way.

In other words, what actually determines whether credentials come back doesn't seem to be the presence of the header, but whether the Lake Formation federation registration is complete. The header is the spec-compliant way to signal intent, but at least in this environment it wasn't the deciding factor in the Glue endpoint's behavior.

Still, to verify things properly per spec, let's call LoadTable with the header attached:

# With the vended-credentials header
awscurl --service glue --region ap-northeast-1 \
  -H "x-iceberg-access-delegation: vended-credentials" \
  "https://glue.ap-northeast-1.amazonaws.com/iceberg/v1/catalogs/123456789012:s3tablescatalog:penguin-rest-test/namespaces/analytics/tables/daily_sales"

Result (excerpt):

{
  "metadata-location": "s3://34c72c19-610d-4e5c-d8d1qwx3db1p3tbr8g9jch9n6e14sapn1b--table-s3/metadata/00000-163a447a-c64d-44f9-8619-5db9561e549b.metadata.json",
  "metadata": { "...": "..." },
  "config": {
    "s3.access-key-id": "(masked)",
    "s3.secret-access-key": "(masked)",
    "s3.session-token": "(masked)",
    "s3.session-token-expires-at-ms": "1783651228000",
    "s3TableArn": "arn:aws:s3tables:ap-northeast-1:123456789012:bucket/penguin-rest-test/table/34c72c19-610d-4e5c-bee4-6ae9f093c583",
    "warehouse_location": "s3://34c72c19-610d-4e5c-d8d1qwx3db1p3tbr8g9jch9n6e14sapn1b--table-s3"
  }
}

s3.access-key-id, s3.secret-access-key, and s3.session-token are actually present here — these are the vended credentials. Along with s3.session-token-expires-at-ms (session expiration in epoch milliseconds), this confirms from the response itself that the catalog is issuing a temporary storage key directly.

The S3 Tables endpoint, meanwhile, has no notion of Lake Formation at all — authorization there is handled entirely via s3tables:* IAM actions, and resource-based policies on the table bucket are also available.

Same underlying table, but the authorization model changes depending on which entry point you go through — something worth keeping in mind during access-design work. Incidentally, exactly how far you can deliberately vary Lake Formation permissions and what happens as a result (does removing a grant actually deny access, etc.) feels like enough material for its own article, so I'll leave that for another time.

Summarizing the Differences Between the Two Endpoints

Here's what actually turned up from hitting both endpoints directly.

Basic Structure

Item	Glue endpoint	S3 Tables endpoint
Endpoint	`glue.<region>.amazonaws.com/iceberg`	`s3tables.<region>.amazonaws.com/iceberg`
Signing service name	`glue`	`s3tables`
`warehouse` value	Glue catalog ID	Table bucket ARN
prefix	`/catalogs/{catalog}` (hierarchy encoding)	URL-encoded bucket ARN
Catalog scope	Account-wide (multi-catalog hierarchy)	Single table bucket
Storage targeted	General-purpose S3 Iceberg + S3 Tables	S3 Tables only
Access control	IAM + Lake Formation (hybrid possible)	s3tables IAM actions only
Credential vending	Vended credentials (requires `register-resource --with-federation`)	None (uses the caller's own credentials)

Where to Use Which, and Gotchas

The official documentation gives guidance on when to use each endpoint:

If you only need basic read/write access to a single table bucket: the S3 Tables endpoint
If you need to integrate multiple catalog sources, or need centralized governance and fine-grained access control via Lake Formation: the Glue endpoint

From actually hitting both directly, here are a few things I'd flag:

Getting the signing service name wrong (glue / s3tables) results in a 403. This is exactly the kind of mistake a wrong rest.signing-name in your client config produces.
Same table, different authorization model depending on which endpoint you go through (whether Lake Formation is involved). Be explicit up front about which path you're connecting through when designing permissions.
CTAS isn't supported on either endpoint. You can work around this by splitting it into CREATE TABLE + INSERT INTO.
dropTable on the S3 Tables endpoint requires purge=true. Depending on your Spark version, DROP TABLE PURGE can end up sending purge=false anyway — in that case, you'll need to delete via the native DeleteTable API instead.

Looking Ahead (Some Personal Thoughts)

Hitting both endpoints directly left me with the impression that each service's design philosophy shows through in how it uses the "free-form" parts of the Iceberg REST Catalog spec.

The S3 Tables endpoint — with its prefix being the bucket ARN, authorization handled through s3tables IAM actions, and CloudTrail logging it as a native API — looks, from this angle, like a thin translation layer over the storage API.

The Glue endpoint, on the other hand, encodes a catalog hierarchy into the prefix, has Lake Formation stepping into authorization, and (when configured correctly) issues temporary storage keys via vended credentials. Both general-purpose S3-based Iceberg and S3 Tables are visible through the same entry point — which lines up with something I speculated in an earlier article, that Glue Data Catalog may be evolving into a metadata plane for AWS as a whole. That same idea seems to show up directly in how the REST API's paths are designed.

The fact that authentication is SigV4 rather than the OAuth the spec assumes is another data point suggesting that how a vendor uses the "free" parts of a standard spec reveals something about its design thinking — probably not unique to the Iceberg REST Catalog.

One more thing I noticed while reading the spec: the latest version defines server-side scan planning endpoints (planTableScan and friends), where the server does the work of building a scan plan. Glue already has a separate extension endpoint (https://glue.<region>.amazonaws.com/extensions) that independently offers server-side scan planning for Redshift Managed Storage — so it looks like a capability that existed as a proprietary extension is now being absorbed into the standard spec. That's an interesting trend — a proprietary extension leading the way before the standard catches up — and I'd like to compare Glue's extension API against the standard plan-related endpoints in a future article.

Snowflake's Catalog-Linked Database (CATALOG_API_TYPE = AWS_GLUE) connects to exactly this Glue endpoint, so seeing how this API gets called from the Snowflake side is another topic I want to dig into going forward.

Conclusion

In this article, I hit the Iceberg REST Catalog directly to sort out the differences between the Glue Data Catalog and S3 Tables endpoints. It took some effort, but I came away with a clearer understanding and a few new insights.

To summarize:

Design philosophy: even though both implement the same Iceberg REST Catalog spec, Glue is an "entry point to multiple catalogs" while S3 Tables is an "entry point to a single table bucket" — a difference visible just from the URL paths.
Prefix design: the spec's free-form {prefix} is used by Glue to encode a catalog hierarchy (/catalogs/{catalog}), and by S3 Tables to encode the table bucket ARN.
Authentication: both use IAM SigV4. Getting the signing service name wrong (glue / s3tables) results in a 403.
Authorization model: for the same table, the Glue endpoint goes through IAM + Lake Formation, while the S3 Tables endpoint uses only s3tables IAM actions — the model changes depending on the path.
Vended credentials: temporary credentials aren't issued just by attaching the x-iceberg-access-delegation: vended-credentials header — you also need to complete resource registration via register-resource --with-federation.
Debugging tips: inspecting the actual CanonicalRequest and headers with awscurl -v, using CloudTrail's errorCode to triage error types, and pre-checking URL assembly with echo were all useful, unglamorous techniques.

Day to day, relying on PyIceberg's or Spark's abstractions is more than enough. But looking at the raw HTTP requests once gives you a clearer mental map between each client configuration item and "that part of that request," which raises the resolution you get when triaging connection errors.

I hope this article is useful to anyone trying to understand how the Iceberg REST Catalog actually works.

Amazon Deprecated My Email, So I Moved It to Switzerland

lordz-md — Thu, 09 Jul 2026 17:06:48 +0000

I am an ex AWS Community Builder. I had the badge, the Slack channel access, and the mild compulsion to explain to strangers that yes, actually, WorkMail was fine.

And it was fine, which is the tragedy, that isn't Stockholm syndrome talking. The pitch was narrow and it landed, at least for me: native Outlook support and Exchange-shaped behavior — calendars, free/busy, mobile sync that just worked — without buying into the entire Microsoft 365 apparatus and the licensing spreadsheet that comes with it. Four dollars a user. Fifty gigs. Data in a region I picked, encrypted with a KMS key I held. If you needed Outlook on the desk but did not need Teams, WorkMail was the only managed email service I ever used that appeared to grasp that a person might want just email.

Then, on the last day of March, AWS put it on the list.

Not a list. The list. Fourteen services and features got sunset or shoved into maintenance mode in a single availability update, and WorkMail went out alongside RDS Custom for Oracle, the WorkSpaces Thin Client, and something called the AWS Service Management Connector, which I had to read three times to satisfy myself it wasn't placeholder text somebody forgot to replace. In the words of Corey Quinn, "getting the Old Yeller treatment in one blog post is a bold move" — his read, in Last Week in AWS issue #466, being that somebody at Amazon had finally gone and checked the usage metrics. No new customers after April 30, 2026, console goes dark March 31, 2027.

The community did get loud about it, which surprised me until I read what people were actually upset about. Nobody was mourning WorkMail. They were doing arithmetic on their own dependencies. App Runner's deprecation had leaked earlier in the year and then been walked back; CodeCommit was deprecated and subsequently resurrected. Fourteen at once reads less like lifecycle management and more like a policy change, and once you have had that thought you cannot stop having it about every service you rely on.

The reaction to WorkMail specifically was closer to anthropological: wait, people were using that?

Reader: I was people. I was, by some estimates, a statistically significant fraction of people.

Stages of grief, abbreviated

I sat on it for a while, in the manner of anyone handed a deadline eighteen months out. Then a weekend arrived with nothing in it and I made the mistake of opening the migration guide being on painkillers from a surgery I had in the weeks before.

AWS's own guidance suggests third-party landing spots like Kopano Cloud and Zoho Mail. These are perfectly reasonable, WorkMail-shaped replacements from vendors who have every incentive to make the move painless. I looked at them for 6-7 minutes and went to Proton Mail instead, which is the kind of decision that in retrospect sounds principled for some people and looked a lot like spite at the time.

To be clear about the principle, since everyone assumes they know it: I did not have a road-to-Damascus moment about surveillance capitalism ( in a similar move I decided to upgrade my iPhone instead of buying a Fairphone ). That's the reason people cite and it's a perfectly good one, but it isn't mine and I'm not going to borrow it to sound better. Mine was narrower and maybe nerdier. I wanted my mail under Swiss jurisdiction rather than in a bucket subject to whatever the current American legal weather happens to be. I also wanted the cryptography to be a library I could actually read ( not that I could, but I can feed the code to Claude using Fable and let it explain it to me ) — Proton maintains OpenPGP.js in the open, third parties audit it, and when I have a question about what is happening to my mail I can answer it by reading source instead of reading a trust page written by the marketing department.

That's the whole thesis. Everything else was a tiebreaker.

The road not taken, and why I did not take it

I bet you wonder why didn't I go self host!? I know. I could have run iRedMail on an EC2 instance. Postfix, Dovecot, Rspamd, a Let's Encrypt cron job, and total sovereignty over every byte. It's good software. I've done it before. The knowledge is still in there somewhere, filed next to my Sendmail trauma.

Here's what that weekend looks like instead. EC2 blocks outbound traffic on port 25 to public addresses by default, so before you can send a single message you get to sign in with your root account credentials and fill in the Request to Remove Email Sending Limitations form, explaining to Amazon in the Use Case Description field why you, personally, should be trusted with SMTP. Approval is discretionary. People get denied. There are re:Post threads about this that read like hostage negotiations. You'll also want an Elastic IP with a reverse DNS record, which is a separate request, because the modern internet has decided that an EC2 IP address sending you mail is guilty until proven otherwise and Microsoft will cheerfully drop your messages into a void with no bounce and no appeal.

Then you own it. Forever. Patching, spam filtering, backups, become your own problem, and at some point in a year or two a disk fills up at 3 a.m. and your mail server stops accepting mail and you find out about it because a client texts you, because you are a one man shop and you don't implement the observability you do for your clients.

And then money wise it doesn't work. A right-sized instance, EBS, snapshots, and an Elastic IP land you in the neighborhood of a few Proton seats before you have spent one minute of your own time, which — and I say this with love for everyone who has ever hand-tuned a Postfix main.cf — is not free. I'm too old for this. I'd rather pay someone in Switzerland.

Self-hosting genuinely wins in exactly two cases. You need total control of the server, or you have data residency requirements tight enough that "Swiss company, Swiss datacenter" doesn't clear the bar and only "this rack, this jurisdiction, this contract" will. Those are real requirements and if you have them, none of the above applies and you should stop reading. Everyone else is choosing a hobby, not an architecture.

Step one: admit that WorkMail was doing three jobs

WorkMail was three products in a trench coat, and only one of them is email.

Human mailboxes. This is the part you migrate.
Email flow rules with a Run Lambda action. This is not email. This is a workflow engine that convinced you it was email.
Transactional sending for some app you shipped in 2019 and have not thought about since.

If you are number three: STOP. Go outside. That should have been going through SES the whole time, and the only reason it wasn't is that you had a working SMTP endpoint and no supervision. Verify the domain in SES, point the app at it, remove it from the migration checklist. Proton is not your application's SMTP relay, and treating it like one is how you end up reading bounce logs at 1 a.m. while questioning your career.

If you are number two: keep SES as well, and take a moment to appreciate that you already were. Inbound mail arrives at Amazon SES first and is handed to WorkMail afterward — if SES blocks a message, your flow rules never see it. SES has been standing in front of your mail server this entire time, quietly, like a bouncer you forgot you hired.

Proton, needless to say, has no equivalent to a Run Lambda rule. You configured those in the WorkMail console, picked your sender and recipient patterns, and your function fetched the message body through the workmailmessageflow API. There is no CLI verb for any of it, which tells you something about how much AWS expected you to be doing this. It was an AWS-flavored appendage grafted onto email, not a feature of email, and nothing outside AWS is going to replace it. Route the automated addresses to SES receiving rules, keep the Lambda, let Proton handle the humans.

What survives this triage is invariably smaller than what you feared. Mine went from eleven mailboxes to seven, and two of the four casualties were aliases nobody had sent to since the Trump administration. The first one.

Get your data out before you touch DNS

WorkMail ships a native export. Use it — not because you'll import from it, but because you want a cold copy in S3 before you start moving load-bearing DNS records around.

aws workmail start-mailbox-export-job \
  --organization-id m-a123b4c5de678fg9h0ij1k2lm234no56 \
  --entity-id S-1-1-11-1111111111-2222222222-3333333333-3333 \
  --role-arn arn:aws:iam::111122223333:role/WorkmailMailboxExportRole \
  --kms-key-arn arn:aws:kms:eu-west-1:111122223333:key/KEY-ID \
  --s3-bucket-name my-workmail-escape-hatch \
  --s3-prefix exports/alice/ \
  --client-token $(uuidgen)

Prerequisites, because AWS would never let you have a nice thing unmediated: an S3 bucket with public access blocked, a symmetric KMS key, and an IAM role that export.workmail.amazonaws.com is permitted to assume. The trust policy scopes to aws:SourceArn for a single organization, though you can drop that condition and reuse the role across several. It is about fifteen minutes of policy-wrangling per organization, not per user, which by AWS standards qualifies as a delightful surprise.

Loop it over your entity IDs, walk away, return to a bucket full of .zip files containing MIME-format messages.

Now, two things before you call this a backup and go to bed.

It exports email and calendar items only. Contacts and tasks do not come along. This is documented, in the sense that it is written down somewhere you were never going to look. Also, the job runs over a period of time rather than capturing the mailbox at an instant, so it is emphatically not a snapshot — mail arriving mid-export lands wherever it lands. Poll describe-mailbox-export-job for progress, and be aware that list-mailbox-export-jobs will only show you the previous seven days, because retention is a feature you pay extra for elsewhere.

Full procedure, trust policy JSON included: Exporting mailbox content and the start-mailbox-export-job CLI reference.

This is your insurance. It is not your migration path. Your migration path is IMAP, like it was before Y2K.

Migration is boring, which is the entire point

Proton's Easy Switch handles generic IMAP sources, not merely the Gmail and Outlook boxes its marketing page is excited about. WorkMail is a generic IMAP source. Feed it:

Host: imap.mail.<region>.awsapps.com
Port: 993, TLS
Username: the full address
Password: the account password

It runs server-side, chews through your folder hierarchy, and re-encrypts everything on arrival. For a 40 GB mailbox with a decade of nested folders named things like Archive/2019/misc/actually-important, budget most of a day and — this is the important part — do not watch it. Nothing good has ever come from watching a mailbox sync.

If you'd rather drive, imapsync speaks fluently to both ends and gives you folder filtering and a log file that tells you the truth. I ran imapsync on the two mailboxes I actually cared about and let Easy Switch handle the rest, on the theory that a tool you can debug is worth more than a tool you can trust. Both landed clean, which was mildly disappointing. I prepared mentally I might need to tell my wife that I will not watch a movie with her that weekend.

DNS: twenty minutes that determine your weekend

Drop your MX TTL to 300 or smaller the day before cutover. Everyone forgets this. You will forget this. Then, in Proton's admin console, collect:

A protonmail-verification= TXT record
MX at mail.protonmail.ch (priority 10) and mailsec.protonmail.ch (20)
SPF: v=spf1 include:_spf.protonmail.ch ~all
Three DKIM CNAMEs
A DMARC record, which you should have had for years, and we are not going to discuss it further

Add verification, SPF, and DKIM first. Leave MX pointed at WorkMail. Let it settle. Then swap MX. Keep the WorkMail organization breathing for a week afterward — it's four dollars a mailbox, the cheapest insurance policy in the history of the discipline — and run a delta sync at the end to sweep up whatever landed on the old side while the internet made up its mind.

Parts that will annoy you, ranked

No IMAP. No SMTP. Not really. Proton is end-to-end encrypted, which means your mail client cannot simply talk to it. You run Proton Bridge locally, which stands up a loopback IMAP/SMTP endpoint for your client to connect to. It works. It is also a daemon you must now think about, and if your team is on Thunderbird or Apple Mail, that conversation happens before the migration, not during it, unless you enjoy conflict.

Server-side search is different. Proton cannot index what it cannot read, which is the whole selling point, so search is client-side against a locally built index you download once. This is fine. It is not Exchange. If your workflow is "search fourteen years of mail from a hotel wifi on a borrowed laptop," recalibrate.

Storage math, honestly. WorkMail: $4/user/month, 50 GB. Proton Mail Essentials: $6.99/user/month billed annually, 15 GB. Proton loses that comparison on paper and I'm not going to pretend otherwise by adding up "value" until the number comes out right. It wins the comparison back only if you were already paying separately for a VPN, a password manager, or file storage, because Workspace Standard bundles the lot at $12.99. If all you want is a mailbox, you are paying a premium for Swiss jurisdiction and zero-access encryption. That is a legitimate thing to buy. Just know that's the transaction.

Calendar interop. Remember those free/busy lookups I was bragging about four hundred words ago? Gone. Proton Calendar is end-to-end encrypted and handles invitations perfectly well, but there is no EWS endpoint for Outlook to interrogate and there is not going to be one. This was the single hardest thing to surrender. If your organization schedules meetings by staring at a grid of everyone's availability, price that in before you commit, and possibly consider scheduling fewer meetings.

Was it worth it!?

I did not leave WorkMail on principle. I left because Amazon deprecated it and I was given a deadline. The principle only decided where I landed, which I suspect is true of most principles.

But the migration forced the triage I'd been avoiding for four years: the app mail that should have been SES, the alias nobody had touched since 2021, the "team" mailbox that was, on closer inspection, one person. The cutover consumed a weekend. The only casualty was an email flow rule I wrote in 2018 and am still not certain ever did anything.

The Community Builder badge expired anyway, but who knows maybe this article will get me back into the program next year?

Note: If you're still using WorkMail, you have until March 2027 and you should not use all of it. DNS propagation is patient. Your users are not. And AWS has now established, conclusively, that "it's been running for a decade" is not a support commitment.

Amazon Transcribe with Hermes Agent - Custom STT Plugin

Piotr Pabis — Thu, 09 Jul 2026 13:10:15 +0000

I run Hermes Agent on an EC2 instance, firstly, because I know it well, and secondly, because I get a lot of AWS Community Builder credits. Although I use OpenRouter as the model provider (as Bedrock's offering is quite limited), there are other services that can be used to complete the agent's harness for abilities such as speech-to-text, text-to-speech, OCR and others. It turns out, building custom plugins for Hermes is surprisingly easy! So today, we are going to build one for speech-to-text to utilize Amazon Transcribe (which I also refer to as AWS Transcribe). I will follow this guide from Hermes docs to build it.

Repository with full code is here

The infra basics

Before we start any code, I want to set up the infrastructure part. First, we need an S3 bucket where we are going to upload the audio files. I will also set an expiration policy on the prefix transcribe-inputs so that in case the cleanup routine doesn't run, the bucket will not fill up with random audio files.

resource "random_string" "bucket_suffix" {
  length  = 8
  special = false
  upper   = false
}

resource "aws_s3_bucket" "bucket" {
  bucket = "hermes-bucket-${random_string.bucket_suffix.result}"
}

resource "aws_s3_bucket_lifecycle_configuration" "lifecycle" {
  bucket = aws_s3_bucket.bucket.id

  rule {
    id     = "delete-transcribe-inputs-after-1-day"
    status = "Enabled"
    filter { prefix = "transcribe-inputs" }
    expiration { days = 1 }
  }
}

Next thing is to set up the required permissions for the agent. I will use the following IAM Policy document that can help you create the actual policy for the role or user of your choice. I am using Hermes on EC2 so I will attach this policy to a role with instance profile. These are the minimal permission that will allow to upload the audio files, process them and clean up afterwards.

data "aws_iam_policy_document" "transcribe_policy" {
  statement {
    sid    = "AllowTranscribeJobs"
    effect = "Allow"
    actions = [
      "transcribe:GetTranscriptionJob",
      "transcribe:DeleteTranscriptionJob",
      "transcribe:StartTranscriptionJob",
      "transcribe:ListTranscriptionJobs",
    ]
    resources = ["*"]
  }

  statement {
    sid = "AllowReadWriteTranscribeInputs"
    effect = "Allow"
    actions = [
      "s3:GetObject",
      "s3:PutObject",
      "s3:DeleteObject"
    ]
    resources = ["${aws_s3_bucket.bucket.arn}/transcribe-inputs/*"]
  }
}

resource "aws_iam_role_policy" "transcribe_policy" {
  name   = "HermesAgentTranscribePolicy"
  role   = aws_iam_role.hermes.id
  policy = data.aws_iam_policy_document.transcribe_policy.json
}

The plugin basics

Based on the docs I mentioned earlier we can define the basic draft of our plugin. Create a new directory: ~/.hermes/plugins/aws-transcribe or wherever you have it installed. In there you need at least two files plugin.yaml and __init__.py, so create both. plugin.yaml is just a metadata file that will keep name, version.

name: aws-transcribe
version: 0.1.0
description: "AWS Transcribe STT backend"

For the actual code, we need to define a new class that will override required functions. Before we start it, let's define how the overarching pipeline will look like. First we will receive a local file path. This we need to upload to S3 and call Amazon Transcribe to start transcription job and, as the job is async, we need to wait for it to finish. Afterwards, we fetch transcribed text and delete S3 object and the Transcribe job. (Theoretically a streaming transcription is also possible and is faster but unavailable in boto3. You can alternatively try JavaScript or Go.)

As we have some idea what to do now, we can write a skeleton of the new plugin. Functions referenced in transcribe are not yet existing.

from __future__ import annotations
from agent.transcription_provider import TranscriptionProvider
from typing import Any, Dict
import logging
import boto3
import os

logger = logging.getLogger(__name__)

class AWSTranscribeError(Exception):
    pass

class AWSTranscribeProvider(TranscriptionProvider):
    @property
    def name(self) -> str:
        return "aws-transcribe"

    @property
    def display_name(self) -> str:
        return "AWS Transcribe"

    def is_available(self) -> bool:
        # Perform test if we have any AWS connectivity and a bucket is set
        try:
            boto3.client("sts").get_caller_identity()
            return bool(os.environ.get("AWS_TRANSCRIBE_BUCKET"))
        except Exception as exc:
            return False

    def transcribe(self, file_path, *, model=None, language=None, **extra):

        try:    
            s3_uri = self._upload_to_s3(file_path)
            job_name = self._start_transcription_job(s3_uri, language or "auto")
            job = self._wait_for_job(job_name)
            text = self._get_transcript(job)
            self._try_cleanup(s3_uri, job_name)

            return {
                "success": True,
                "transcript": text,
                "provider": "aws-transcribe",
            }

        except Exception as exc:
            logger.error(f"{exc}")
            return {
                "success": False,
                "transcript": "",
                "error": f"aws-transcribe failed: {exc}",
                "provider": "aws-transcribe",
            }


def register(ctx):
    ctx.register_transcription_provider(AWSTranscribeProvider())

Let's dissect it step by step. At the top, we import libraries needed for now and create a new logger that will inform us if something fails. Errors will be visible in Hermes' Gateway logs (for example journalctl -xe --user -u hermes-gateway if you use systemd installation). The first two methods in the class are just informative. is_available is sort of a health check - is everything we need configured to run the plugin? I check first if there's any IAM role or user configured that can call AWS and whether the staging bucket is defined in environment variable AWS_TRANSCRIBE_BUCKET (this should be set in ~/.hermes/.env). The method transcribe is called by Hermes when there's a file sent by the user and is_available passes. It expects to return the dict you see there near returns. In the try-catch block we simply call pipeline methods one after another, catching and logging any errors in case they happen. The last method register is done outside the class and is just a loader function for Hermes plugin manager.

Parts of the pipeline

Now we can start defining all the nested functions. First we upload the file to S3 bucket. To make it unique and safe to pass to Transcribe, we will use an UUID as the object key.

import uuid
# ...

class AWSTranscribeProvider(TranscriptionProvider):
    # ...
    def _upload_to_s3(self, local_file_path: str) -> str | None:
        """
        Upload a local file to S3 and return the S3 URI.
        """
        ext = os.path.splitext(local_file_path)[1]
        s3_key = f"transcribe-inputs/{uuid.uuid4().hex}{ext}"
        client = boto3.client("s3")
        client.upload_file(local_file_path, os.environ.get("AWS_TRANSCRIBE_BUCKET"), s3_key)
        return f"s3://{os.environ.get('AWS_TRANSCRIBE_BUCKET')}/{s3_key}"

With the S3 URI that we receive, we can continue by starting the transcription job. There's an option to configure language for STT in Hermes but I decided to support also "auto" which will let Transcribe identify the language by itself. For safety and uniqueness I also use UUID to name the job and need return it to store for later.

class AWSTranscribeProvider(TranscriptionProvider):
    # ...
    def _start_transcription_job(self, s3_uri: str, language: str) -> str | None:
        """
        Start a transcription job and return the job name.
        """
        job_name = f"hermes-{uuid.uuid4()}"
        client = boto3.client("transcribe")
        if language == "auto":
            client.start_transcription_job(
                TranscriptionJobName=job_name,
                Media={"MediaFileUri": s3_uri},
                IdentifyLanguage=True
            )
        else:
            client.start_transcription_job(
                TranscriptionJobName=job_name,
                Media={"MediaFileUri": s3_uri},
                LanguageCode=language
            )
        return job_name

Next up we have to wait for the job to finish. I will poll the API every 5 seconds up to 60 as waiting anything longer can be either bad user (me 😅) experience or that something larger went wrong. I can simply use other service like iOS built-in or Wispr Flow after that time. This method will respond with a dict containing a presigned S3 URI (in AWS'es own S3 bucket) in case the job is successful.

import time
# ...
class AWSTranscribeProvider(TranscriptionProvider):
    # ...
    def _wait_for_job(self, job_name: str, timeout: float = 60.0, poll_interval: float = 5.0) -> str | None:
        """
        Wait for a transcription job to complete and return the transcript.
        """
        client = boto3.client("transcribe")
        deadline = time.monotonic() + timeout

        while True:    
            response = client.get_transcription_job(TranscriptionJobName=job_name)
            job = response["TranscriptionJob"]
            status = job["TranscriptionJobStatus"]

            if status == "COMPLETED":
                return job

            if status == "FAILED":
                raise AWSTranscribeError(f"Transcription job failed: {job.get('FailureReason', 'No failure reason was returned by AWS.')}")

            remaining = deadline - time.monotonic()
            if remaining <= 0:
                raise AWSTranscribeError(f"Timed out waiting for transcription job {job_name} after {timeout:.1f} seconds.")
            time.sleep(min(poll_interval, remaining))

In next step we need to use received presigned URL to download a JSON file that will contain all the information about the transcription job such as language detected and most importantly the actual transcribed text.

import json
from urllib.request import urlopen
# ...
class AWSTranscribeProvider(TranscriptionProvider):
    # ...
    def _get_transcript(self, job: Any) -> str:
        with urlopen(job["Transcript"]["TranscriptFileUri"]) as response:
            transcript_json = json.load(response)
            transcripts = transcript_json.get("results", {}).get("transcripts", [])
            if not transcripts:
                return ""
            return transcripts[0].get("transcript", "")

And the last function that is a good idea and you can implement optionally is cleanup. That way Transcribe won't hold all the text you speak and S3 will not be bloated with unnecessary audio files. In case this fails, we will only log a warning and the transcription itself will be successful.

class AWSTranscribeProvider(TranscriptionProvider):
    # ...
    def _try_cleanup(self, s3_uri: str, job_name: str) -> None:
        """
        Try to clean up the temporary files and job.
        """
        try:
            client = boto3.client("s3")
            client.delete_object(Bucket=os.environ.get("AWS_TRANSCRIBE_BUCKET"), Key=s3_uri.split("/", 3)[-1])
        except Exception as exc:
            logger.warning(f"Failed to clean up S3 file \"{s3_uri}\": {exc}")
        try:
            client = boto3.client("transcribe")
            client.delete_transcription_job(TranscriptionJobName=job_name)
        except Exception as exc:
            logger.warning(f"Failed to clean up transcription job \"{job_name}\": {exc}")

Configuring and enabling

Once you have the files in place you can edit the main configuration file. In ~/.hermes/config.yaml perform the following changes (but first copy the file as a backup). In section plugins add aws-transcribe to the list of enabled and ensure it is not on the list of disabled plugins.

# ...
plugins:
  enabled:
    - aws-transcribe
    # ... other enabled plugins
  disabled:
  # no aws-transcribe on the list

Now add add configuration in speech-to-text section. I will change the default provider to aws-transcribe and create a new section for it. Set the model to anything and language to either "auto" if you want Transcribe to identify the language or a valid language code like en-GB (look on this site for the list of languages). You can also leave config for other providers as is.

stt:
  enabled: true
  provider: aws-transcribe
  local:
    model: base
  openai:
    model: whisper-1
  aws-transcribe:
    model: whatever
    language: auto

It is also possible that you might not have boto3 installed into the packages. In that case run the following:

cd ~/.hermes/hermes-agent
../bin/uv pip install boto3

Restart the gateway using hermes gateway restart or systemctl --user restart hermes-gateway. Look into the logs with journalctl --user -feu hermes-gateway (f option keeps tailing the logs so that you can look for any errors appearing). Perform a test of transcription with a messaging platform of your choice. I for example use Matrix. I told the agent previously to just echo back the message received and do nothing else with it. In CloudTrail I see that both StartTranscriptionJob and DeleteTranscriptionJob happened.

Introducing AWS Sandbox Environment: Learn AWS for Free Without a Credit Card

Sarvar Nadaf — Thu, 09 Jul 2026 10:11:18 +0000

I was scrolling through the AWS What's New feed when one announcement stopped me. AWS Builder Center now lets you spin up a free sandbox environment directly from a workshop page. No AWS account. No credit card. No billing alerts at 3am because you forgot to terminate a NAT Gateway.

I've spent the last decade helping clients onboard teams to AWS. The number one friction point was never the technology. It was the signup. "What if I get charged?" kills more cloud careers before they start than any certification exam ever will.

That friction just disappeared.

What AWS Actually Announced

On July 8, 2026, AWS launched free sandbox environments inside AWS Builder Center. Here are the confirmed details straight from the announcement:

Cost: Zero. Completely free.
Duration: 8 hours from the moment you activate it
Frequency: One sandbox per week, resets every Sunday
Provisioning time: Most environments ready within 15 minutes
Cleanup: Fully automatic after your 8 hours expire
Requirements: No personal AWS account needed, no credit card

The sandbox is a pre-provisioned AWS account that you access through eligible workshops. These are step-by-step guided labs covering topics like building serverless APIs, setting up VPCs, deploying containers, or working with AI services. You're not getting a stripped-down simulator. You're deploying real resources into a real AWS account.

Why This Matters More Than You Think

I've onboarded probably 40+ engineers to AWS across different client engagements. The pattern repeats every single time. Day one of training, someone raises their hand: "What happens if I accidentally leave something running?" That question comes from fear. And fear makes people skip hands-on practice entirely. They read docs. Watch videos. Never deploy a thing.

The AWS Free Tier tried to solve this. It helped, but didn't kill the fear because you still needed a credit card on file. I remember my early days after graduation. No steady income, learning AWS on Free Tier, feeling like I was on borrowed time. When the tier expired and a $4.50 charge showed up for some resource I forgot to terminate, it felt like $45. When you're broke and learning, even a small unexpected bill makes you close the console and not come back for weeks.

AWS recently revamped the Free Tier into a proper "Free Plan" with $200 in credits and auto-closure after 6 months. Big improvement. But Builder Center Sandbox goes further: you don't even need an account. The barrier is zero.

How It Compares to Other Free Options

AWS now has four distinct paths to hands-on practice without paying. Each serves a different need.

Option	Cost	Credit Card	Time Limit	Access Type	Best For
Builder Center Sandbox (NEW)	Free	No	8 hrs/session, 1/week	Workshop-guided	Zero-commitment first experience
Free Tier (Free Plan)	Free ($200 credits)	No	6 months	Full console, 90+ services	Self-directed projects
Skill Builder	$29/mo or $449/yr	Yes	Subscription	Labs + exam prep	Certification prep
Educate	Free	No	Self-paced	Curated courses + labs	Students, career-switchers (13+)

Let me break down when you'd pick each one.

Builder Center Sandbox is for when you want to try AWS without creating anything. No account, no commitment, no trace. You follow a workshop, deploy real resources, learn by doing, and everything vanishes after 8 hours.

Free Tier (Free Plan) is for when you want to build your own thing. A side project, a portfolio piece, an experiment. You get an actual account with $200 in credits and 6 months to use them. It auto-closes, so no surprise bills.

Skill Builder is for professionals investing in certifications. The $29/month gets you immersive labs, exam prep courses, and practice exams. Worth it if you're targeting a specific cert.

Educate is the entry ramp for students and people switching careers. Free, beginner-friendly, and includes a job board for ages 18+.

The Decision Framework

"I want to try AWS for the first time without any commitment" → Builder Center Sandbox

"I want to build my own side project on AWS for free" → Free Tier (Free Plan)

"I'm preparing for an AWS certification exam" → Skill Builder subscription

"I'm a student or career-switcher with no tech background" → AWS Educate

"I'm an experienced dev who wants to test a specific architecture pattern" → Free Tier (Free Plan) or Builder Center Sandbox (if a relevant workshop exists)

What You Can't Do (Limitations)

Based on the announcement, here are the real constraints:

Workshop-specific, not open console. You can't open the AWS Console and freelance. The sandbox is tied to eligible workshops. At launch, only select workshops support it, with more being enabled over time.
8-hour hard limit. If you're building something that takes multiple sessions, this won't work. The environment gets wiped completely after 8 hours. No saving progress.
One per week. You can't chain multiple sandboxes in a day. One activation, resets Sunday. Plan your learning accordingly.

What I'd Like to Know (But Can't Confirm Yet)

A few questions the announcement doesn't answer:

Do you need an AWS Builder ID to access it? (Probably yes, but unconfirmed)
Which specific workshops are sandbox-enabled at launch?
What services are available inside the sandbox? (Likely varies per workshop)
Are there geographic restrictions?
Can you export your work (code, configs) before the 8-hour window closes?

I'll update this article as those details become clear.

Who Should Use This Today

If you fall into any of these categories, go to builder.aws.com/workshops right now:

You've been "meaning to learn AWS" for months but never started
You teach or mentor others and need a zero-friction lab environment for your students
You're evaluating AWS services before committing your company's budget
You want guided, structured practice without managing your own cleanup
You already know AWS but want to explore workshops on topics outside your comfort zone

My Take

As someone with 7 AWS certifications and four years as an AWS Community Builder, I've watched this problem from both sides. Watched clients spend thousands on training environments that collected dust. Watched junior engineers accumulate surprise bills that killed their curiosity. I've seen people give up on cloud because the onboarding felt hostile.

This sandbox removes a massive psychological barrier. It seems like a small feature. It's not.

The best way to learn cloud is to break things in cloud. AWS just made breaking things free, temporary, and consequence-free. That's the right move.

Have you tried the Builder Center Sandbox yet? Which workshops are available in your region? Drop your findings in the comments. I'll compile a list as people report back.

This article is actively updated. AWS Builder Center Sandbox launched July 8, 2026 with select workshops. As AWS enables more workshops, expands features, or changes limits, I'll update this post. Follow me to stay current.

Follow me for more on AWS architecture, DevOps, and AI tooling: sarvarnadaf.com | LinkedIn

Building Isolated Browser Sessions with AWS Lambda MicroVMs

Jaya Ganesh — Thu, 09 Jul 2026 09:20:47 +0000

AWS Lambda is one of my favourite AWS services, which introduced me to the world of serverless. The service started growing, getting packed with a lot of great features. One such feature which was recently launched is Lambda MicroVM. This opens up a different kind of possibility for what we can build with Lambda, not just simple and short-lived functions. We can now think about running isolated per-user workloads on demand. We can deploy developer sandboxes, preview environments, and testing tools without having to deploy an ECS or EKS.

The ability of the microVMs to start instantly and be able to suspend and resume execution up to 8 hours is what made me think of an option to create an isolated browser session with the ability to choose the region to pick where we want that browser to be located.

What I built:

SafeBrowse is a small reference project that launches one Chromium session per user. The user signs in, selects a region, starts a session and gets an embedded browser inside the app.

Behind the scenes, that browser is running inside an AWS Lambda MicroVM.

The goal was not to build a production remote browser isolation product. The goal was to understand what Lambda MicroVMs make possible and how we can build real user-facing workloads around them.

Architecture overview:

At a high level, the architecture has two parts. The first part is the control plane. This is where the user signs in, creates sessions, views active sessions, and terminates them. The second part is the runtime. This is where the actual Chromium browser runs inside a Lambda MicroVM in the selected AWS region.

Why the Proxy Was Needed

One interesting problem came up while building this. The browser cannot directly talk to the MicroVM endpoint the way the backend can. The MicroVM endpoint expects specific headers for authorization and routing. A normal browser iframe cannot attach those headers when loading an external origin. So instead of exposing the MicroVM URL directly to the frontend, I used a CloudFront-based embed proxy. The frontend asks the backend for a short-lived embed token. Then the iframe loads a SafeBrowse-owned CloudFront URL. Lambda@Edge resolves the token, finds the right MicroVM endpoint, and injects the required headers server-side. So the browser only talks to a normal HTTPS URL. The MicroVM details stay behind the proxy.

User Flow:

The user flow is simple. The user signs in with Cognito. Then they select a region. Then they click launch. The backend creates a session record in DynamoDB and starts a Lambda MicroVM running Chromium. Once the session is active, the user opens the browser inside the app. When they are done, they terminate the session.

What Lambda MicroVMs Open Up

This is the part I found most interesting. Lambda MicroVMs make it possible to think about serverless in a new way. We are not limited to simple request-response functions anymore. We can start building isolated, temporary, user-specific compute environments.

For example, this pattern could be used for:

remote browser sessions
safe preview environments
temporary developer sandboxes
per-user automation workers
browser testing tools
short-lived data processing environments
AI agent workspaces
isolated execution environments for internal tools

The important idea is that each user or task can get its own isolated runtime.

You start it when needed. You use it for a short period. You shut it down when done. That is a very useful model.

What I Learned

The biggest learning was that the MicroVM itself was not the hard part.

The harder part was designing the access flow around it. How does the user reach the workload? How do we avoid exposing internal endpoints? How do we keep the frontend simple? How do we terminate sessions cleanly? How do we track ownership, status, region, and TTL?

That is where the control plane becomes important. Lambda MicroVMs give us the runtime. But we still need a clean control plane around it.

SafeBrowse started as a simple question. Can I launch a browser per user using Lambda MicroVMs and make it feel like part of a normal web app? The answer is yes.

The architecture is still simple, but the pattern is powerful. A serverless control plane can manage users, sessions, tokens, regions, and lifecycle. Lambda MicroVMs can run the isolated workload. CloudFront and Lambda@Edge can provide a clean browser-facing entry point. Together, this opens up a new category of serverless applications. Not just APIs. Not just background jobs. But real, temporary, isolated compute experiences that users can interact with directly.

SafeBrowse is not a production Remote Browser Isolation product. It is a reference architecture to explore how Lambda MicroVMs can run isolated, user-facing workloads on demand using a serverless control plane.

GitHub Repo: https://github.com/jayaganeshk/safebrowse

Prompt Caching Is an Architectural Pattern, Not a Cost Optimization

Amit Kayal — Thu, 09 Jul 2026 08:55:30 +0000

Prompt Caching Is an Architectural Pattern, Not a Cost Optimization

One question I ask during almost every AI architecture review is surprisingly simple:

What part of this prompt actually changes between requests?

Most of the time, the answer is "almost nothing."

The system prompt is identical. Tool definitions are identical. Business rules don't change. Product catalogs remain the same. Even in RAG applications, much of the retrieved context is often reused across multiple agent steps.

Yet many AI systems resend all of this information on every model call.

At small scale, nobody notices.

At enterprise scale, you're paying to process the same information thousands—or sometimes millions—of times.

That's an architecture problem.

The Hidden Cost Isn't the Model

Teams spend a lot of time comparing GPT, Claude, Gemini, or open-source models.

Far fewer teams measure how many input tokens they're repeatedly sending to those models.

A typical enterprise agent request might include:

A 10,000-token system prompt
Tool definitions
Security and governance instructions
Product or knowledge documents
Few-shot examples
The user's question

Only the last item changes.

Everything else is effectively static.

If every request reprocesses the entire prompt, inference costs grow linearly with usage—even though most of the context hasn't changed.

What Prompt Caching Actually Solves

Prompt caching recognizes that large portions of a prompt are stable.

Instead of reprocessing those tokens every time, the platform stores the reusable prompt prefix and reuses it on subsequent requests when that prefix is identical. The cached prefix can include system prompts, tool definitions, reference documents, images, conversation history, and other static context.

The important point is that this isn't application-level caching.

Nothing changes in your business logic.

The model simply avoids doing the same work twice.

The Economics Change Quickly

Anthropic's pricing illustrates why prompt caching matters.

For a standard 5-minute cache:

Initial cache creation costs approximately 1.25× the normal input-token price.
Subsequent cache reads cost only 10% of the normal input-token price.

For workloads that need a longer lifetime, a 1-hour cache is also available. The initial write costs more, but cache reads remain heavily discounted.

The implication is straightforward.

The first request pays to build the cache.

After that, every reuse of the same prompt prefix is dramatically cheaper.

The more frequently that shared context is reused, the greater the return.

This Matters Even More for Agentic AI

Traditional chat applications make one model call.

Enterprise agents rarely do.

A single user request may trigger:

Planning
Multiple tool calls
Document retrieval
Validation
Reflection
Final response generation

Each of those stages often carries exactly the same system instructions, governance rules, and tool definitions.

Without prompt caching, every stage pays the full input-token cost.

With prompt caching, those shared instructions become reusable infrastructure rather than repeated overhead.

As agents become more capable, prompt caching becomes more valuable—not less.

Prompt Structure Starts Affecting Cost

Once prompt caching enters the picture, prompt design stops being just a prompt engineering exercise.

It becomes part of system architecture.

Some practical design principles are worth following:

Keep stable instructions at the beginning of the prompt.
Separate static knowledge from dynamic user input.
Avoid changing reusable prompt prefixes unnecessarily.
Cache large reference documents instead of resending them.
Use separate cache breakpoints for content that changes at different frequencies.

Small design decisions can determine whether every request becomes a cache hit or a cache miss.

Design for Reuse

One pattern I've started applying consistently is to think about prompts the same way we think about software components.

Stable organizational knowledge belongs in one layer.

Agent behavior belongs in another.

Dynamic business context is added later.

User input comes last.

That separation doesn't just improve maintainability.

It also maximizes cache reuse, reduces latency, and lowers operating costs.

The Bigger Lesson

Prompt caching isn't simply a billing feature exposed by an API.

It changes how enterprise AI systems should be designed.

As prompts grow to include governance policies, compliance rules, tool schemas, and organizational knowledge, the question is no longer whether those prompts are expensive.

The real question is whether your platform is paying for them once—or paying for them on every single request.

That is an architectural decision, not an implementation detail.

Stop Deploying Blind: Add a Blast Radius Gate to Your CI/CD Pipeline

Scott Burgholzer — Thu, 09 Jul 2026 03:10:20 +0000

Series: Introducing Blast Radius — See What Breaks Before You Deploy

What We Are Building

This article shows you how to use Blast Radius in your own CI/CD pipeline. By the end, you'll have a deployment gate that automatically analyzes infrastructure changes, scores downstream risk, and blocks merges that exceed your threshold.

Blast Radius comments on your PR with an analysis table (Highest Score, Affected Resources, AI Recommendation, Confidence) plus an AI-generated summary.

There are currently two paths to use Blast Radius:

GitHub Action: fastest and auto-comments on PRs
Raw CLI: works in any CI system such as GitLab, Jenkins, CircleCI, or whatever you're running.

Blast Radius is meant for your infrastructure. It works on any stage: dev, testing, production, or any stages you may use. It calls live AWS APIs (Config and Resource Explorer) against your actual account. The results reflect real dependency relationships, not static analysis.

Prerequisites — Deploy the Backend

The following must be enabled in your AWS Account before deploying the backend:

AWS Config must be enabled and recording in the region you deploy to. This is where Blast Radius reads dependency relationships. Here's how to enable AWS Config.
Resource Explorer must be enabled. It's the fallback for resources Config doesn't have relationships for. Here's how to enable Resource Explorer.
CDK Bootstrap must be run via cdk bootstrap in your account and region you are deploying to.
Optional: Bedrock model access if you want AI summaries. You may choose any model via cross-region inference profile. Without enabling Bedrock model access, threshold gates will still work but the AI gate will not.

Deploy Steps:

git clone https://github.com/sburgholzer/BlastRadius.git
cd BlastRadius
npm install
npm run build
cd packages/infra
cdk deploy

This creates an API Gateway, Step Functions state machine, Lambda functions, DynamoDB tables, S3 bucket, and a CloudFront distribution for the frontend.

When the deployment finishes, grab the API URL from the CDK output. That's the only value you need for the rest of this article.

To enable AI summaries, set enableBedrockSummary in the CDK props. You can also configure resultsRetentionDays for S3 lifecycle management. More detail on these options is in the README.

GitHub Action — The Five-Minute Path

Terraform Workflow

Terraform is the simplest path. Generate the plan, convert it to JSON, and pass it to the action:

- run: terraform plan -out=plan.out
- run: terraform show -json plan.out > plan.json

- uses: sburgholzer/BlastRadius@v0.1.0
  with:
    format: terraform-plan
    input: plan.json
    threshold: 75
    ai-gate: true
    api-url: ${{ secrets.BLAST_RADIUS_URL }}

The action reads the JSON plan, runs the analysis, and comments the results on your PR.

CDK Workflow

CDK requires a few more steps. Blast Radius needs to know what CloudFormation will actually do — the changeset actions, not just what the template looks like. A cdk synth alone doesn't give you that; you have to ask CloudFormation directly.

- name: Generate CloudFormation changeset
  run: |
    npx cdk synth 2>/dev/null
    cp cdk.out/MyStack.template.json template.json
    CHANGESET_NAME="blast-radius-$(date +%s)"
    aws cloudformation create-change-set \
      --stack-name MyStack \
      --change-set-name $CHANGESET_NAME \
      --template-body file://template.json \
      --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND \
      --change-set-type UPDATE \
      --output json
    aws cloudformation wait change-set-create-complete \
      --stack-name MyStack \
      --change-set-name $CHANGESET_NAME
    aws cloudformation describe-change-set \
      --stack-name MyStack \
      --change-set-name $CHANGESET_NAME \
      --output json > changeset.json
    aws cloudformation delete-change-set \
      --stack-name MyStack \
      --change-set-name $CHANGESET_NAME

- name: Blast Radius Analysis
  uses: sburgholzer/BlastRadius@v0.1.0
  id: blast-radius
  with:
    format: cloudformation
    input: changeset.json
    threshold: 75
    ai-gate: true
    api-url: ${{ secrets.BLAST_RADIUS_API_URL }}

Each step has a specific job: synth produces the template, create-change-set asks CloudFormation what it would change, wait holds until it's ready, describe-change-set captures the answer as JSON, and delete-change-set cleans up so it doesn't linger between runs.

CloudFormation Workflow

Same as CDK but without the synth step — you already have a template file:

- name: Generate CloudFormation changeset
  run: |
    CHANGESET_NAME="blast-radius-$(date +%s)"
    aws cloudformation create-change-set \
      --stack-name MyStack \
      --change-set-name $CHANGESET_NAME \
      --template-body file://template.json \
      --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \
      --change-set-type UPDATE \
      --output json
    aws cloudformation wait change-set-create-complete \
      --stack-name MyStack \
      --change-set-name $CHANGESET_NAME
    aws cloudformation describe-change-set \
      --stack-name MyStack \
      --change-set-name $CHANGESET_NAME \
      --output json > changeset.json
    aws cloudformation delete-change-set \
      --stack-name MyStack \
      --change-set-name $CHANGESET_NAME

- name: Blast Radius Analysis
  uses: sburgholzer/BlastRadius@v0.1.0
  with:
    format: cloudformation
    input: changeset.json
    threshold: 75
    ai-gate: true
    api-url: ${{ secrets.BLAST_RADIUS_API_URL }}

This is essentially the same as the CDK Workflow, except we don't need to run a synth.

What the GitHub Action Does Under the Hood

Downloads the bundled CLI from the GitHub release
Runs analyze with --ci flag for JSON output
Parses the JSON into GitHub Action outputs
Comments on the PR with a formatted table + AI summary (disable with comment-pr: false)

Action inputs:

Input	Required	Default	Description
format	✅		Input format: cloudformation / terraform-plan / canonical
input	✅		Path to the input file (changeset JSON / terraform plan JSON / canonical manifest)
api-url	✅		Blast Radius API URL
threshold			Risk score threshold (0-100). Fails if any resource exceeds this.
ai-gate		false	Fail if AI recommends against deployment
no-summary		false	Skip AI summary generation
comment-pr		true	Automatically comment results on the PR
version		latest	CLI version to use

Action outputs:

Output	Description
analysis-id	The analysis ID
verdict	pass or fail
highest-score	Highest impact score (0-100)
total-affected	Total affected resources
recommend-deploy	AI deployment recommendation (true/false)
confidence	AI recommendation confidence (high/medium/low)
summary	AI-generated risk summary (markdown)
result-json	Full JSON result (for custom downstream steps)

CLI in Any CI System

The GitHub Action is a convenience wrapper. Underneath, it's the CLI. If you're on GitLab, Jenkins, CircleCI, Bitbucket Pipelines, any other tool, or want greater control than the GitHub Action, you can use the CLI directly.

It's a single bundled JavaScript file. No npm install, no dependencies beyond a Node.js runtime. Download it, run it, read the exit code.

How to get the CLI:

curl -sL https://github.com/sburgholzer/BlastRadius/releases/latest/download/blast-radius.js -o blast-radius.js
sed -i '1{/^#!/d}' blast-radius.js

The sed command strips the shebang line. Node.js can't parse #!/usr/bin/env node when you invoke it with node blast-radius.js directly (needed in CI environments that don't support executing .js files as scripts).

Environment setup:

Set BLAST_RADIUS_API_URL as an environment variable pointing to your deployed API Gateway URL. That's the only required config. Everything else goes in command flags.

GitLab CI example

blast-radius:
  image: node:20
  stage: validate
  script:
    - curl -sL https://github.com/sburgholzer/BlastRadius/releases/latest/download/blast-radius.js -o blast-radius.js
    - sed -i '1{/^#!/d}' blast-radius.js
    - node blast-radius.js analyze --format terraform-plan --input plan.json --threshold 75 --ai-gate --ci
  variables:
    BLAST_RADIUS_API_URL: $BLAST_RADIUS_URL
  artifacts:
    paths:
      - blast-radius-result.json
    when: always

Note: --ci outputs JSON to stdout. Capture it with > blast-radius-result.json and save as an artifact for downstream jobs. Exit code 1 naturally fails the GitLab job, no extra configuration needed.

Jenkins example

stage('Blast Radius') {
  environment {
    BLAST_RADIUS_API_URL = credentials('blast-radius-url')
  }
  steps {
    sh '''
      curl -sL https://github.com/sburgholzer/BlastRadius/releases/latest/download/blast-radius.js -o blast-radius.js
      sed -i '1{/^#!/d}' blast-radius.js
      node blast-radius.js analyze --format cloudformation --input changeset.json --threshold 75 --ai-gate --ci
    '''
  }
}

Store your API URL in Jenkins credentials and reference it as an environment variable. Non-zero exit fails the stage, standard Jenkins behavior.

CircleCI example

- run:
    name: Blast Radius Analysis
    command: |
      curl -sL https://github.com/sburgholzer/BlastRadius/releases/latest/download/blast-radius.js -o blast-radius.js
      sed -i '1{/^#!/d}' blast-radius.js
      node blast-radius.js analyze --format terraform-plan --input plan.json --ai-gate --ci
    environment:
      BLAST_RADIUS_API_URL: ${BLAST_RADIUS_URL}

Parsing the JSON output

With --ci, the CLI outputs a JSON object to stdout:

{
  "analysisId": "abc-123",
  "verdict": "fail",
  "exitCode": 1,
  "reason": "ai-gate",
  "recommendDeploy": false,
  "confidence": "high",
  "riskSummary": {
    "highestScore": 82,
    "totalAffected": 14,
    "critical": 2,
    "high": 5,
    "medium": 4,
    "low": 3
  },
  "naturalLanguageSummary": "## Executive Overview\n\nThis deployment presents..."
}

What you can do with this:

Pipe to jq for specific fields: node blast-radius.js analyze ... --ci | jq '.riskSummary.highestScore'
Send to Slack: extract verdict, highestScore, and naturalLanguageSummary into a webhook payload.
Build a custom PR comment in GitLab/Bitbucket using the JSON fields.
Feed into a dashboard or metrics system; track risk scores over time per team or per service.

Exit Codes

Any CI system's native "fail on non-zero" behavior works out of the box. No special error handling needed.

Code	Meaning
0	Pass - threshold OK and AI approves (or gates not enabled)
1	Fail - threshold exceeded OR AI recommends against deployment
2	Error - invalid input / pipeline failure / timeout / misconfiguration

Choosing Your Gate Strategy

Threshold only (`--threshold N`)

This is deterministic. Same input, same result, every time. It's based on the scoring formula from the previous article: depth (how far the resource is from the changed resource), criticality (how critical the resource is to your infrastructure), and change severity (how much the change affects the resource). Each factor is weighted to produce a score between 0 and 100.

Because it's a formula, there's no AI dependency. If you run Blast Radius without Bedrock, you still get a usable result for your CI/CD pipeline.

This gate is good for compliance-driven teams, environments where you need reproducible gates, or if you want to understand how the scoring system works before enabling AI.

N is your cutoff: if the highest score of any resource exceeds that value, Blast Radius fails your pipeline. Here's a general guideline:

90: very permissive. Only blocks Critical-category changes. Good for dev/staging environments.
75: moderate. Blocks Critical and most High-category hits. Good default for production.
50: conservative. Blocks anything above Medium. Good for sensitive workloads.

Start at 75 to see how it works in your environment, then fine-tune from there.

AI gate only (`--ai-gate`)

This is where judgment-based thinking comes into play. The AI gate looks at the full graph, not just the individual scores. It catches patterns like shared dependencies, fan-out risk, and cascading failure paths. It will return recommendDeploy: true/false with confidence: high/medium/low.

This is good for teams that trust AI tooling or want to experiment with it to catch systemic risks. If you use this gate, I recommend reading through the summary and doing manual investigation to verify the reasoning, and to understand what to fix when it blocks.

One caveat: --ai-gate requires Bedrock enabled on the backend. If Bedrock is disabled, the CLI exits with error code 2.

Using Both Gates Together

This is my recommended configuration. When you use both gates, Blast Radius fails your pipeline if EITHER triggers. The threshold catches obvious high-scoring resources. The AI catches non-obvious patterns below your threshold value.

Running Without AI Summary

If you run Blast Radius with --no-summary, it skips the AI-generated natural language summary but will still run the analysis. This saves a few seconds if you only care about the score.

Note: --ai-gate overrides --no-summary because the gate needs the AI to run.

Reading the Output

You've set up the gate. It ran. Now you're looking at a PR comment (or JSON output) with scores, dependency chains, and an AI recommendation. Here's how to read it all.

The PR Comment Anatomy

The GitHub Action comments on your PR with two parts:

Summary table: four key metrics at a glance:
- Highest Score: the single scariest number in the analysis (0-100)
- Affected Resources: total number of downstream resources discovered
- AI Recommendation: Deploy ✅ or Do Not Deploy ❌
- Confidence: how sure the AI is (high/medium/low)
AI-generated summary (if Bedrock is enabled): structured markdown with:
- Executive overview: one-paragraph assessment
- Key findings: the specific resources and patterns that drove the recommendation
- Cascading risks: what could go wrong if you deploy anyway
- Recommendation: the final call with reasoning

If you're using the CLI with --ci, you get the same information as JSON fields: riskSummary, recommendDeploy, confidence, and naturalLanguageSummary.

Risk Scores — What the Numbers Mean

Score	Category	Meaning
75-100	Critical (red)	Direct or near-direct impact on high-criticality resources (databases, auth systems, core networking). Stop and think.
50-74	High (orange)	Significant downstream impact. Review the dependency chain before deploying.
25-49	Medium (yellow)	Worth noting. Probably fine, but look at what's at the end of the chain.
0-24	Low (green)	Far away, low criticality, or a non-destructive change type. Safe to proceed.

How the Score is Calculated (review)

impactScore = round((depthScore × 0.30) + (criticalityScore × 0.40) + (changeTypeSeverity × 0.30))

Depth (30%) — how many hops away. Depth 1 = 100, Depth 5 = 60, Depth 10 = 10.
Criticality (40%) — how important the resource type is. Database = 100, Lambda = 75, S3 = 50, Log group = 25.
Change severity (30%) — how dangerous the action is. Remove = 100, Replace = 80, Modify = 50, Add = 30.

Criticality is weighted highest because what's affected matters more than how far away it is. Deleting a test log group one hop away is less dangerous than modifying something three hops from a production database.

Dependency Chains — the Breadcrumb Trail

Every scored resource includes a dependencyChain: the ordered path showing exactly how risk flows from your change to that resource.

Example: sg-abc123 → ec2-instance-1 → rds-prod

Read left to right: your changed resource → intermediate resources → the thing at risk.

Longer chains mean lower depth scores (the resource is further away).
But a long chain ending at a Critical-class resource (like a database) can still score high because criticality carries 40% of the weight.
Short chains ending at low-criticality resources score low even though they're close.

In the frontend graph view, the dependency chain is the highlighted path when you click a node. In the PR comment, it appears in the detail expansion for each resource.

AI Confidence Levels

When using the AI gate, the model returns a confidence alongside its recommendation:

High: clear signal. The graph strongly supports the recommendation. Trust it.
Medium: mixed signals. Some risk factors present, some mitigating. Worth human review before overriding.
Low: AI isn't sure. Limited data, ambiguous patterns, or a very small graph. Don't gate on this alone — treat it as advisory.

When the AI says "don't deploy" but scores are below threshold:

This is the AI gate earning its keep. It's seeing a pattern that individual scores don't capture:

Fan-out risk: many resources all depending on one thing you're changing.
Single points of failure: a resource that's the only path between your change and critical infrastructure.
Cascading depth: the graph is deep and converges on something important at the far end.

Check the AI summary for the specific reasoning. It will explain what pattern it detected and why it recommends against deployment.

When the AI says "deploy" but you're nervous:

The AI can be wrong. If your gut says something's off:

Look at the dependency chain for the highest-scored resource. Does the path make sense?
Check whether the graph is missing resources (AWS Config relationships aren't always complete).
Use the interactive frontend to explore the full graph visually.
When in doubt, deploy to a non-production stage first.

Local Development and Testing

You don't have to go straight to CI/CD pipeline. The CLI has commands for generating, inspecting, and exploring results locally. The interactive frontend also gives you a visual way to understand your infrastructure's dependency landscape.

`blast-radius generate` — Create Input Files without Submitting

The generate command produces the same input file that analyze would create and submit, but stops there. No analysis is run, no API is called.

# CDK: synthesizes and creates a changeset, saves to disk
blast-radius generate --format cdk --stack MyStack --output changeset.json

# Terraform: runs terraform plan + show, saves the JSON
blast-radius generate --format terraform-plan --output plan.json

# CloudFormation: creates changeset from template, saves it
blast-radius generate --format cloudformation --stack MyStack --template cfn.json --output changeset.json

Why this is useful:

Verify your CI workflow produces valid input before enabling the gate
Inspect exactly what Blast Radius will see — review the changeset JSON to understand the resources and change types
Build a library of test inputs for different scenarios
Share with teammates: "here's what the analysis would run against"

`--save` flag — save AND submit

If you want to both inspect the input and run the analysis in one shot:

blast-radius analyze --format cdk --stack MyStack --save changeset.json --ai-gate

This saves the generated input to disk, then submits it. When a result comes back surprising ("why did my security group change score 82?"), you can open changeset.json and see exactly what was sent: which resources, which change types, which properties.

The Interactive Frontend

After deploying the backend, the CloudFront URL serves a React SPA where you can explore results visually.

Analysis list (/analyses): shows all past analyses sorted by date. Click any row to drill in.
Dependency graph: Cytoscape.js interactive graph. Your changed resources appear as solid blue nodes. Downstream dependencies fan out, colored by risk category (red/orange/yellow/green). Click any node for a detail panel: resource ID, type, score, risk category, and full dependency chain.
Filters: filter by risk category, resource type, or toggle direct changes on/off. Filters apply as intersection (resource must match all active filters).
Table view: sortable alternative to the graph. Sorted by impact score (highest first) by default. Paginated at 50 rows.
Export: download the full analysis as JSON for archiving or feeding into other tools.
AI Summary: rendered as formatted markdown below the graph (if Bedrock was enabled for the analysis).

The frontend is useful for exploring results visually after CI runs, understanding dependency patterns in your account before they cause problems, demoing Blast Radius to stakeholders, and investigating why the AI gate blocked a deployment.

Running against the demo examples

The repo includes examples/cdk-demo/ with a two-step demo that doesn't require your real production infrastructure:

01-baseline/: deploys a realistic infrastructure: VPC, ECS Fargate service, Aurora PostgreSQL cluster, ALB, Lambda function, S3 bucket.
02-risky-change/: modifies the security group (restricts from public to internal-only) and resizes the database instance.

# Deploy baseline
cd examples/cdk-demo/01-baseline
npm install && npx cdk deploy

# Run Blast Radius against the risky change
cd ../02-risky-change
npm install
BLAST_RADIUS_API_URL=<your-url> blast-radius analyze --format cdk --stack BlastRadiusDemoBaseline --ai-gate

This produces a real analysis showing 7 affected resources cascading from the security group and database changes, ECS service, Lambda, ingress/egress rules, scored and visualized.

Cost note: The demo creates real AWS resources (~$0.45/hr). Run cdk destroy when you're done. Don't do what I did and forget to run the destroy!

Other useful commands

# Check status of a running analysis
blast-radius status --analysis-id abc-123

# Fetch completed results
blast-radius export --analysis-id abc-123 --format json

What's Next?

You now have a working deployment gate. Every infrastructure PR gets analyzed for downstream risk before it can merge. Whether you're using the GitHub Action, the CLI in GitLab, or exploring results in the frontend, the workflow is the same: generate, analyze, gate, decide.

Next in this series, we go under the hood:

Article 3: Serverless IaC Risk Analysis — The Architecture Behind Blast Radius: How the pipeline is designed. Why Step Functions. How the canonical format abstraction makes multi-IaC support possible without touching the analysis engine.
Article 4: 349 Tests, Zero Mocks — Building Blast Radius in TypeScript: Engineering decisions, property-based testing with fast-check, dependency injection over module mocking, and the Lambda runtime gotchas that wasted a day.

These articles are coming in the next few weeks. I'll update this post with links as each one publishes.

DEV Community: AWS Community Builders

Synapse Feature Demo: Deterministic Software Composition Analysis in Action

[AWS] Transforming Automated Testing with DevOpsAgent [DevOpsAgent]

【AWS】DevOpsAgentで変革する自動テスト【DevOpsAgent】 #AI - Qiita

Introduction

Previous Article

[AWS] Automate reviews with DevOpsAgent [DevOpsAgent]

What We Checked This Time

What is AWS DevOps Agent?

Prerequisites: AGENTS.md is not a repository file

DevOpsAgent Output

Verified Scenario

Verification Results

1. Does the behavior change depending on whether the code under test has been changed?

Test Result Timeline Capture

2. Will it autonomously create test plans even for files without tests?

Test Result Timeline Capture

Description of finding a verified quality gate setting error in the test coverage of Scenario D

3. Does the instruction in the PR affect the behavior?

Differences from Traditional CI/CD and Test Automation

Summary

In Conclusion

Reference Links

memex: training for the Forward Deployed Engineer era with a private memory brain on AWS

Before AI, and after

What memex is, in plain words

The moment it clicked

What's actually inside

The night shift

The AWS build, and its trade-offs

Scar tissue

Why build instead of buy

How my week actually works now

Takeaways

Code or diffusion? A field guide to programmatic image generation

Decision 1: is it structure, or is it texture?

The property nobody prices in

Decision 2: how much rendering power?

Decision 3: which model tier? Cost buys a ceiling

The skeleton trick

The one-page matrix

When diffusion wins

The evidence

Building Production AI Agents on AWS Bedrock — Architecture and Code Decisions Worth Keeping in Mind

The Context

Architecture Learnings

1. Bedrock over calling the model API directly

2. Agent memory

3. Dynamic context injection vs. conversation history

4. Tool design: the docstring is the API contract

5. Security in agents: the model is not the trust boundary

6. Infrastructure as Code

7. Observability

8. Data model design for DynamoDB

9. Testability

10. AWS credentials — the one-liner that works in both environments

11. Context is the most important part of any agent

12. Never trim conversation history mid-tool-call

13. Hooks are the preferred and deterministic way to validate

14. Structured error responses matter

Conclusion

Hitting the Iceberg REST Catalog Directly: Understanding the Differences Between Glue Data Catalog and S3 Tables

Introduction

What Is the Iceberg REST Catalog?

Setting Up the Test Environment

SigV4 Signing

Hitting the S3 Tables Iceberg REST Endpoint

GET /v1/config

Listing Namespaces and Tables

Retrieving Metadata with LoadTable

Checking What's Happening Under the Hood via CloudTrail

Hitting the Glue Iceberg REST Endpoint

GET /v1/config

Prefix Rules: Encoding the Catalog Hierarchy

Listing Namespaces (Default Catalog)

Reading an S3 Tables Table Through the Glue Endpoint

Access Control Differences (Lake Formation)

Summarizing the Differences Between the Two Endpoints

Basic Structure

Where to Use Which, and Gotchas

Threshold only (`--threshold N`)

AI gate only (`--ai-gate`)

`blast-radius generate` — Create Input Files without Submitting

`--save` flag — save AND submit