DEV Community: Amit

An Agent That Can Pay for Its Tools

Amit — Sat, 22 Aug 2026 06:29:43 +0000

If agents are going to act on the open internet, they will eventually hit a price tag.

That changes the design problem. Calling an API is easy compared with letting a non-human actor spend money without handing it private keys, unlimited budget, or a prompt-level spending rule it can be talked out of.

The thesis: payment is not a tool call. It is a governed financial action.

That means the payment path needs more than a model deciding "yes" or "no." It needs user consent, credential isolation, session budgets, policy checks, telemetry, and a narrow proof path that the model cannot bypass. The useful primitive is not "agent calls payment API." The useful primitive is "agent can pay for a bounded resource without owning the wallet."

I tested that pattern with Amazon Bedrock AgentCore Payments, Coinbase CDP, x402, Cedar policy enforcement, CloudWatch, and a Strands agent plugin. The result is strong enough to change the mental model, with one caveat: this is testnet evidence, and not every x402 endpoint accepted the same generated proof.

What The Payment Primitive Owns

The clean architecture separates four jobs:

Layer	Job
Paid resource	Returns HTTP `402` with price, token, network, and `payTo` address.
Agent/tool	Encounters `402`, extracts the payment requirement, and asks for proof.
Payment manager	Generates the x402 payment proof without exposing wallet secrets to the model.
Governance	Enforces user consent, session budget, policy allow/deny, and observability.

That split matters. If the model sees private keys, the design is already broken. If the spending limit is prompt text, the design is also broken. The payment boundary has to sit outside the model context.

In the experiment, the AgentCore side owned the payment manager, connector, instrument, session, and proof generation. Coinbase CDP owned the delegated wallet/signing path. x402 provided the HTTP 402 challenge and proof format. Cedar controlled whether the paid action was allowed before the payment path ran.

The key detail: the model never needed the payment credential. It only needed a tool response that said, effectively, "this resource costs X on Y network." The proof came from the payment infrastructure.

The First Proof: HTTP 402 To HTTP 200

The base flow worked.

The test created a payment manager, Coinbase CDP credential provider, connector, active payment instrument, and short-lived payment session. The session had a 0.10 USD budget and a 30 minute TTL. The wallet was funded with Base Sepolia testnet USDC, and WalletHub permission was granted for the app to pay from that wallet.

Then the tool called a paid endpoint. The endpoint returned HTTP 402 with an x402 version 2 payment requirement on Base Sepolia. AgentCore generated a PAYMENT-SIGNATURE proof through the ProcessPayment API. The retry against the AWS-documented endpoint returned HTTP 200.

That proves the core primitive: an agent-side tool can hit a paid resource, receive a price challenge, generate proof through infrastructure, and retry successfully without the model touching the wallet.

I also ran a controlled two-sided merchant test. Instead of paying a sample endpoint, I created a separate merchant wallet and ran a local x402 endpoint that advertised that wallet as payTo. The paid retry returned HTTP 200, the buyer wallet decreased, and the merchant wallet increased by 0.001 testnet USDC.

That second test matters because it proves both sides of the flow: the buyer can generate proof, and the merchant can receive value.

Budget Is Not A Prompt Instruction

The most important test was not whether a payment could succeed. It was whether a payment could be stopped.

I created a session with a 0.02 USD budget and used a local paid endpoint priced at 0.01 testnet USDC. AgentCore generated two payment proofs in the session. After those attempts, the session showed 0 USD available spend. The next same-session proof request failed with InsufficientBudget.

That is the right failure mode.

The agent did not need to remember the budget. The prompt did not need to say "please do not spend more than two cents." The model could be instructed to ignore limits and the infrastructure would still reject the over-budget payment.

This is the deeper point: financial constraints belong in mechanisms, not model behavior. Prompts are useful for intent. They are the wrong place to enforce spend.

Cedar Before Payment

Budgets answer "how much can this session spend?" They do not answer "who is allowed to invoke this paid action with these arguments?"

That is where policy belongs.

I reused a live AgentCore Gateway with a Cedar policy engine in enforce mode. A valid user token called a governed action with two amounts:

Request	Result
`45.0`	Allowed
`600.0`	Denied with JSON-RPC `-32002` before backend execution

During the denied-call window, the Payment Manager logs showed zero new Payment processed events and CloudWatch showed zero new SpendAmount datapoints.

That proves the first control: policy can stop the workflow before the payment path starts.

Then I hardened the shape. A runner first called Gateway/Cedar with a small amount. If Cedar allowed it, the runner invoked the real x402 payment path. If Cedar denied it, the runner did not attempt payment. The allowed branch generated payment proof; the denied branch made no payment attempt.

Finally, I moved the paid-fetch implementation itself behind Gateway as a Lambda target. Cedar allowed amount < 0.01, denied the over-limit branch, and the Lambda entered AgentCore Payments only after policy allow.

The Gateway-hosted run proves the full chain: Cedar allow, proof generation inside the Lambda, denial before Lambda/payment for the over-limit request, and a settled 200 on the allowed branch's retry.

The architecture is still the right one:

Gateway policy allow/deny -> paid tool execution -> AgentCore payment proof -> retry paid resource

The policy decision happens before money can move. The payment session budget still applies after policy allow. Those are separate controls, and they should stay separate.

Observability Is Part Of The Product

Payment systems without audit trails are demos.

AgentCore Payments publishes metrics into AWS/Bedrock-AgentCore — SpendAmount, OperationSuccess, OperationFailure, OperationLatency, ActiveSessions, PaymentTokenFetchSuccess, PaymentTokenFetchFailures — with SpendAmount carrying ProcessPayment datapoints per Coinbase connector and payment manager. Vended log delivery for the payment manager produces lifecycle events for session creation, instrument retrieval, and payment processing, all in your own CloudWatch log group.

Metrics and logs are one delivery pipeline. X-Ray span correlation is a second, entirely separate one, and it is easy to miss because nothing about ProcessPayment failing to appear in X-Ray looks like a missing configuration step — it just looks like empty search results. Log delivery uses logType=APPLICATION_LOGS pointed at a CWL (CloudWatch Logs) destination. Spans require their own delivery source with logType=TRACES, pointed at an XRAY destination:

aws logs put-delivery-source \
  --name "payments-traces-source" \
  --resource-arn "arn:aws:bedrock-agentcore:REGION:ACCOUNT:payment-manager/MANAGER-ID" \
  --log-type "TRACES"

aws logs put-delivery-destination \
  --name "payments-xray" \
  --delivery-destination-type "XRAY"

aws logs create-delivery \
  --delivery-source-name "payments-traces-source" \
  --delivery-destination-arn "arn:aws:logs:REGION:ACCOUNT:delivery-destination:payments-xray"

Configuring only the log pipeline — the natural first step, since it's what the getting-started docs walk you through — leaves X-Ray permanently empty with no error anywhere to point at the missing piece. I discovered this in July and confirmed it was not a platform gap in August: once the trace pipeline is wired up, Bedrock.AgentCore.Payments.ProcessPayment spans appear within a couple of minutes carrying the documented attributes: payments.spend_amount, payments.merchant, payments.payment_session_id, payments.payment_instrument_id, down to aws.request_id.

That last set of attributes is the actual payoff. Metrics tell you that spend happened; a span tells you which spend, tied to which session, which merchant, which request. For a platform where agents pay third parties on a customer's behalf, that's the mechanism for answering "which specific payment failed, for which customer, at which merchant, for how much" during an incident or a billing dispute — one X-Ray trace correlating the Gateway request, the Cedar decision, the Lambda execution, the ProcessPayment call, and the merchant settlement, without grepping across log groups by hand.

Agent-Native Payment Flow

The script path is useful for proof. The agent-native path is what product builders will care about.

I tested the Strands AgentCorePaymentsPlugin with the real http_request tool. The tool returned HTTP 402. The plugin detected it, called AgentCore Payments through PaymentManager.generate_payment_header, injected a PAYMENT-SIGNATURE header, and requested a retry.

The tool retried with the injected header and got back HTTP 200 — the plugin owns the entire handshake, proof generation through settled resource access, without the tool author writing any custom 402 handling.

Two Prerequisites That Will Block You Cold

Using Coinbase as a payment provider requires subscribing to "Coinbase Wallets for AgentCore Payments" in AWS Marketplace. Until that subscription is active, CreatePaymentConnector and every wallet operation reject with SubscriptionRequiredException. It's a real, metered charge — $0.005 per wallet operation, consolidated onto your AWS bill.

Separately, Coinbase CDP wallet permissions are wallet-scoped, time-bound grants: the end user picks 7, 30, 60, or 90 days when authorizing the agent to sign. ProcessPayment returns AccessDeniedException: Delegated signing grant is not active the moment it lapses. If a payment call starts failing with that specific error, check the grant's expiry before assuming anything else is wrong — it fails the exact same way whether the grant expired yesterday or was never issued.

Quick Create's Seatbelt

Quick Create provisions Coinbase credentials for you: authorize once via OAuth in the console, and AgentCore creates the payment credential provider on your behalf, no pasted API keys. Point it at a Coinbase project that already has a manually-created Wallet Secret, though, and it refuses outright: "This Coinbase project already has a Wallet Secret, so a new one can't be created automatically."

That's a safe-failure guardrail, not a bug — silently generating a second secret could orphan wallets already derived from the first. But Quick Create only completes its automatic path for a project that has never had a Wallet Secret generated, and the quick-start guides still document manual CDP key generation first. Follow the docs in order and you permanently lose access to Quick Create for that project; only a fresh Coinbase project gets the one-click path.

Three Bugs That Weren't AWS's

Moving the paid-fetch implementation behind Gateway as a Lambda target kept failing the same way at first: a proof generated, a retry rejected with an empty 402 body. The obvious suspects were a rate limit on the sandbox endpoint or an edge-cache hit. Both testable, both wrong — a plain script from a local machine worked fine against the identical endpoint.

Temporary CloudWatch logging inside the Lambda found the real story: ProcessPayment was returning a fully valid, signed proof, authorization and signature intact, on every call. The Lambda was throwing it away:

# what the Lambda was doing — wrong
payment = client.process_payment(...)
result = payment.get("processPayment", {})   # this key does not exist
proof = result.get("paymentOutput", {})       # -> always {}

# the boto3 response is flat, no wrapper
# {"processPaymentId": "...", "status": "PROOF_GENERATED",
#  "paymentOutput": {"cryptoX402": {"payload": {...signed...}}}}
proof = payment.get("paymentOutput", {})       # correct

The process_payment reference confirms it: processPaymentId, status, and paymentOutput are flat, top-level response fields, matching every AWS SDK example published for the API. The bad .get() call silently returned {}, so the retry always carried payload: null — which any x402-compliant merchant correctly rejects.

Two smaller bugs rode along with it: the deployed Lambda source was missing import os, crashing on every cold start behind a retry loop that swallowed the real traceback; and a policy-engine ID was hardcoded from an earlier public-sanitization pass instead of reading from the environment. None of the three were platform bugs. All three had been quietly filed under "the sandbox is flaky" until logging showed otherwise. "It's the network," "it's rate limiting," and "it's flaky" are all real phenomena — in a system with this many moving parts, any of them was plausible. None of them were the fix. Reading the actual response was.

What Is Still Missing

This experiment ran on testnet and does not prove mainnet charging, customer billing, tax handling, refund workflows, fraud handling, or production compliance. Those are different systems.

The core primitive (payment proof generation, session budgets, policy gating, observability) has been validated against GA and all endpoint retry tests have succeeded end-to-end with settled HTTP 200 responses. CloudWatch traces and X-Ray span correlation were reconfirmed as long as both delivery pipelines are wired — that was a configuration gap, not a platform gap.

Coinbase's x402 Bazaar — a curated MCP server exposing thousands of pay-per-use x402 endpoints — is reachable through Gateway. The open thread: I still haven't proven end-to-end agent discovery of a live Bazaar endpoint with a successful settled payment, only that the pieces wire. The blocks are technical (Gateway-to-Bazaar MCP compatibility) not architectural. The pattern itself is sound.

So What

The interesting part of agent payments is not that an agent can call a payment API.

The interesting part is that spending can be bounded by infrastructure, authorized by policy, observed through logs and metrics, and executed without putting wallet secrets into model context.

That is the pattern worth carrying forward:

Intent lives in the model.
Authority lives in policy.
Budget lives in the payment session.
Credentials live outside the model.
Evidence lives in telemetry.

An agent that can pay for its tools is not a wallet with a chat box. It is a governed actor with a narrow financial action path. That is the difference between a demo and something you can reason about.

MPP vs x402: Two Competing Answers to the Same HTTP 402

Amit — Sat, 22 Aug 2026 06:29:07 +0000

Two protocols answer HTTP's thirty-year-old, never-implemented 402 Payment Required status code, and Amazon Bedrock AgentCore Payments speaks both. x402 (Coinbase) and MPP (Stripe and Tempo) are direct competitors for the same job: an agent hits a paywalled resource and pays for it inline, no API key, no billing account set up in advance. They disagree about what "pay" is allowed to mean.

sequenceDiagram
    participant A as Agent
    participant M as Merchant
    participant AC as AgentCore Payments
    A->>M: GET /resource
    M-->>A: 402 Payment Required + challenge
    A->>AC: ProcessPayment(challenge)
    AC->>AC: check session budget, sign with wallet
    AC-->>A: signed proof / credential
    A->>M: retry + proof (x402: PAYMENT-SIGNATURE, MPP: Authorization)
    M-->>A: 200 OK

Both protocols follow that identical outer loop. Where they diverge is what each one chooses to standardize.

Same shape, different commitments

	x402	MPP
Settlement	On-chain only — stablecoins, primarily USDC on Base	Multi-rail — Tempo stablecoins, Stripe cards/ACH/BNPL via Shared Payment Tokens, Bitcoin via Lightning
Retry header	`PAYMENT-SIGNATURE` (v2) or `X-PAYMENT` (v1)	`Authorization: Payment <base64url-token>`
Pricing model	`exact` (fixed price) and `upto` (metered, settle-for-less-than-authorized)	Charge intent, plus a session primitive for high-frequency metered billing
Facilitator	Required — a third party verifies and settles the on-chain transfer	Framed as a formal HTTP auth scheme; some methods avoid a separate facilitator dependency
Transport	HTTP only	HTTP, plus an MCP transport binding — MCP tool servers can charge per tool call directly
Backing	Coinbase, fully open and permissionless	Stripe + Tempo, with Visa and Lightspark extending it to cards and Lightning

AgentCore's release notes describe x402's exact and upto schemes and MPP as three parallel options behind one ProcessPayment API — you pick paymentType (CRYPTO_X402 or MPP) per call, and the wallet, budget check, and signing path are identical underneath.

x402's `upto` scheme: pay for what you actually used

upto lets a merchant advertise a ceiling instead of a fixed price. The buyer's wallet authorizes a Permit2 allowance up to that ceiling, and the merchant settles for whatever was actually consumed — no larger than the ceiling, no second round trip. That's what makes per-token LLM inference or metered compute billable in a single request/response instead of a price negotiation.

I proved this by extending a local x402 merchant with UptoEvmScheme and a Settlement-Overrides response header carrying the real charge:

// merchant route: authorize up to a ceiling, settle for less
app.get("/metered-recap", (_req, res) => {
  res.setHeader(
    "Settlement-Overrides",
    JSON.stringify({ amount: "1000" }),  // actual metered charge
  );
  res.json({ status: "paid", settledUnits: "1000" });
});

# buyer: authorize a ceiling, let the merchant settle lower
payment = client.process_payment(
    ...,
    paymentType="CRYPTO_X402",
    paymentInput={"cryptoX402": {
        "version": "2",
        "payload": accept,
        "permit2AllowanceLimit": accept["amount"],  # the ceiling
    }},
)

ProcessPayment authorized a ceiling of 15000 base units on Base Sepolia testnet; the merchant declared an actual charge of 2000. Checking the wallet's on-chain balance immediately before and after confirmed it moved by exactly 2000, not 15000 — real settlement tracked real usage, not the authorization.

The gotcha: the ProcessPayment X-Ray span's payments.spend_amount attribute reads the ceiling, not the settled amount, because AgentCore signs the authorization before the merchant declares what it actually consumed. A spend dashboard built on that span attribute alone will overstate real spend for upto transactions — reconcile against actual settlement, not the span, to get the true number.

MPP: one interface, four rails behind it

MPP standardizes the challenge-response interface and lets the merchant advertise whichever payment methods it accepts — evm, tempo, solana, and through Stripe, card and fiat — inside one WWW-Authenticate: Payment header, per the mpp-specs IETF draft co-authored by Tempo and Stripe.

No live MPP-compatible merchant exists anywhere yet to round-trip against — not from AWS, not in Coinbase's Bazaar. To prove AgentCore's side of the handshake, I built the smallest thing that could: a synthetic, spec-compliant evm-method challenge.

WWW-Authenticate: Payment id="<unique-id>", realm="<realm>",
  method="evm", intent="charge",
  request="<base64url JSON: amount, currency, recipient, methodDetails.chainId>"

payment = client.process_payment(
    ...,
    paymentType="MPP",
    paymentInput={"mpp": {
        "version": "1",
        "wwwAuthenticateHeaders": [header_value],  # forwarded verbatim
    }},
)
credential = payment["paymentOutput"]["mpp"]["paymentCredential"]
# -> "Payment <base64url-token>", ready to attach as Authorization

Two checks confirmed AgentCore genuinely parsed the challenge rather than rubber-stamping it: the session's available budget dropped by exactly the amount declared in the request (0.02 → 0.019 USD for a $0.001 charge), and the returned credential was a well-formed, 1176-character token with the documented Payment prefix. That's genuine credential generation, matching the ProcessPayment response schema exactly. What it doesn't prove is settlement — there's nothing real to settle against yet.

When to reach for which

x402 when buyers already hold stablecoins and you want the simplest integration, a full on-chain audit trail, and sub-cent transaction costs — paywalled content, one-off compliance lookups, real-time market data.
MPP when your buyer population doesn't hold crypto, or you need card/ACH reach alongside stablecoins, or you're billing something metered and high-frequency where MPP's session primitive fits better than repeated 402 round trips.

AgentCore Payments doesn't make you choose between them. It abstracts the protocol behind one API, so the decision moves from "which protocol do I integrate" to "which protocol does this merchant accept" — which is the right place for that decision to live.

The open thread

Full settlement against a live MPP merchant hasn't been tested anywhere yet — no public merchant exists that accepts MPP payments through the complete cycle. I've proved that AgentCore can generate valid credentials and that budget tracking works, but settling a real charge against a real merchant and confirming the merchant received payment remains untested. This is a genuine gap, not a missing step in the documentation. Once live MPP merchants exist in production, the full picture — credential generation, settlement success, reconciliation — will be available to verify.

A Browser That Pays

Amit — Sat, 22 Aug 2026 06:28:32 +0000

Amazon Bedrock AgentCore names five official use cases for Payments: research, financial analysis, pay-per-inference, on-demand storage, and — quoted with a named customer — browser agents. Anchor Browser, a cloud browser-automation platform, integrated AgentCore Payments specifically to unlock paywalled web content for its customers' agentic workflows.

That combination — a real, isolated Browser Tool session driving through an actual 402 paywall — surfaces a protocol detail most payment paths never hit: a browser doesn't see what a script does.

The flow

sequenceDiagram
    participant B as AgentCore Browser (Playwright/CDP)
    participant S as Paywalled site
    participant AC as AgentCore Payments
    B->>S: navigate
    S-->>B: 402 + @x402/paywall HTML widget
    B->>B: read data-requirements DOM attribute
    B->>AC: ProcessPayment(challenge)
    AC-->>B: signed proof
    B->>S: re-navigate, proof injected via route interception
    S-->>B: 200 OK

The documented pattern uses Playwright connected to a managed Browser Tool session over Chrome DevTools Protocol, with Playwright's response interception catching a 402 mid-navigation:

from bedrock_agentcore.tools.browser_client import browser_session
from playwright.sync_api import sync_playwright

with browser_session(REGION) as client:
    ws_url, ws_headers = client.generate_ws_headers()
    with sync_playwright() as p:
        browser = p.chromium.connect_over_cdp(ws_url, headers=ws_headers)
        page = browser.contexts[0].new_page()
        response = page.goto(paid_url)   # 402 on first navigation

The wrinkle: two different 402s

Here's the part that isn't written down anywhere. A script's 402 comes back as JSON with a PAYMENT-REQUIRED header holding the base64-encoded challenge. Point a real browser (Chromium's Accept: text/html) at the same URL, and the server returns something else entirely: a human-facing @x402/paywall HTML page, no PAYMENT-REQUIRED header at all, with the identical challenge JSON embedded in a widget instead:

<div id="payment-widget"
     data-requirements='{"x402Version":2,"accepts":[{"scheme":"exact",
       "network":"eip155:84532","amount":"2000","payTo":"0x...","asset":"0x..."}]}'>
  <!-- Install @x402/paywall for full wallet integration -->
</div>

That's the standard @x402/paywall convention — the same challenge, rendered for a human wallet-connect UI, but just as readable by an automated browser:

raw = page.eval_on_selector(
    "#payment-widget", "el => el.getAttribute('data-requirements')",
)
requirement = json.loads(raw)

Once the challenge is read from the DOM instead of a header, everything downstream is identical: ProcessPayment generates a real proof, Playwright's route interception injects it into the retry, and the re-navigation settles to 200:

def add_payment_header(route, request):
    route.continue_(headers={**request.headers, **payment_headers})

page.route(paid_url, add_payment_header)
retry = page.goto(paid_url)   # 200

Why this matters

Every other payment path in AgentCore — a script, a Lambda, a backend service — talks to a merchant as a machine. Browser agents talk to the same merchant the way a human would. The target site was built for human traffic and gates agent access the same way it gates a browser without an active subscription. The paywall doesn't know or care that a script is driving the Chromium under the hood. It serves the human-facing challenge either way, and the agent has to be able to read it.

That's the actual shape of the "browser agents" use case: not a new payment mechanism, but the same ProcessPayment primitive plugged into a client that has to speak the paywall's native language — HTML and DOM, not headers and JSON — before it can pay at all.

For anyone building this: the gap isn't in AgentCore Payments. It's in knowing that the same endpoint returns different response shapes based on what User-Agent and Accept headers the request carries. A script gets the header-based challenge. A browser gets the widget. If you're working with the managed Browser Tool through a paywall, you need to handle both, or at minimum understand that the browser path requires DOM extraction instead of header parsing.

What's missing

Browsers drive the unscripted web — portals, JavaScript-rendered content, interaction-gated resources — but they're only half the story. Most of these paywalled portals still have API endpoints that prefer a script path, and the cleaner play, where available, is usually to find the native API. The real complexity emerges when a portal only surfaces data through the browser UI, forcing the agent to navigate like a user. Paywalls add another layer: they're not guarding the browser itself; they're guarding specific resources behind the browser. Understanding when a paywall applies to the browser path vs. the API path, and whether the managed Browser Tool or a native API fetch is the right reach, is the decision tree most projects face first.

The one thing AgentCore Payments doesn't clarify in the docs (and that teams building this pattern hit immediately) is exactly this: how to detect which challenge shape you're getting, and how the managed browser's isolation plays with credential caching across sessions. That's the open thread for the next builder who runs this live.

AgentCore Registry Is a Governed Catalog for Agent Interfaces

Amit — Sat, 22 Aug 2026 06:07:27 +0000

AgentCore Registry provides discovery for reusable agent interfaces, but the important behavior is not simple name resolution. The tested path was: create a registry, publish an A2A descriptor, move the record through approval, search approved records, resolve the Agent Card, and invoke the discovered agent.

That makes Registry a governed catalog for the agent graph. Hardcoded runtime ARNs are the visible pain. The deeper problem is uncontrolled reuse: which agents exist, which interface they expose, who approved them, and whether a caller is using the right version.

AWS Agent Registry gives that workflow a control plane: create a registry, publish a record, submit it for approval, approve it, then search only approved records. The record can describe an A2A agent, an MCP server, an agent skill, or a custom resource. The shape is intentionally catalog-first.

The thesis: agent discovery becomes a governance problem before it becomes a routing problem.

Hardcoded ARNs Are Only Part of the Problem

Hardcoded ARNs are easy to hate. They make demos brittle. They make account migration annoying. They make blue/green rollouts awkward. They also create the comforting illusion that the problem is only lookup.

It is not.

If a caller has a hardcoded ARN, at least it knows exactly what it is calling. A Registry-backed caller moves one level up: it asks for a capability, receives one or more approved records, resolves the interface, and invokes the selected agent. That adds power, but it also adds a new question:

Who gets to publish something discoverable?

That is where Registry differs from DNS. DNS does not ask whether a service is approved for reuse. Registry does. A record moves through lifecycle states before it becomes searchable. In our test, the A2A record moved from DRAFT to APPROVED before the search path could use it.

That lifecycle is the control point.

The Experiment

The verification ran two parallel paths: one against the preview API and one against the GA surface. Both proved the Registry-to-A2A integration path works end to end.

Test	Result	What It Proved
Preview: Create, approve, search, resolve	PASS	A live A2A Agent Card can be published as an approved Registry record and discovered by search.
Preview: Invoke through discovered card	PASS	A caller can resolve the Agent Card from a Registry record and invoke the discovered agent through A2A without hardcoding the runtime ARN.
GA: Control and discovery APIs	PASS	The GA Registry APIs (`agent-registry-control`, `agent-registry`) create, approve, list, retrieve, and delete records.
GA: Registry-to-A2A invocation	PASS	A caller retrieved an A2A Agent Card from a GA Registry record and invoked the agent through A2A, then cleaned up the record and Registry.

The target was a deployed A2A specialist agent exposed through an Agent Card:

Name: enterprise support escalation specialist
Protocol version: 0.3.0
Function: assess support tickets, return severity, owner, SLA, and next steps

The Registry record was created as an A2A descriptor. The test captured:

Registry status: READY
Record after create: DRAFT
Record after submit: APPROVED
Search by name: returned the approved record

Then the search script selected the A2A record, parsed the Agent Card returned in the descriptor, and invoked the agent through the resolved card URL. The caller did not hardcode the specialist's runtime ARN in the delegation path.

The test case was deliberately operational: an enterprise customer had a double charge and production API access failure during quarter-end close. The discovered specialist returned:

P2 severity, Named TAM owner, 12-hour SLA.

That response matters less than the path. The important part is that the caller found an approved agent record, resolved its interface, and delegated over A2A.

What the Evidence Supports

The preview validation and GA revalidation support a narrower and more precise statement:

Registry is the catalog layer where agent interfaces become approved assets.

That is more than runtime address lookup. It changes what teams design around.

If Registry were only ARN lookup, the natural question would be, "Why not use an environment variable?" For a single caller and a single callee, that is still a good answer. A static env var is simpler, cheaper, and easier to debug.

Registry becomes worth the extra lookup when the organization needs reuse with control:

Multiple callers need to discover the same agent.
A platform team wants to publish approved A2A agents or MCP servers.
Records need owners, versions, status, and audit history.
Callers search by capability instead of importing another team's configuration.
Agent interfaces are validated before they become reusable.

The Registry record is not only a pointer. It is metadata plus interface contract plus lifecycle state.

The Namespace Migration Is Part of the Contract

The migration is no longer future work. The current GA path uses the agent-registry-control client for lifecycle operations and the agent-registry client for discovery. This shift is not merely naming cleanup. It affects every script that creates, searches, or audits Registry resources, along with IAM policies, ARN formats, events, metrics, and quotas.

The two namespaces differ on every surface:

Surface	Preview Namespace	GA Namespace
CLI/API	`bedrock-agentcore`, `bedrock-agentcore-control`	`agent-registry-control`, `agent-registry`
IAM actions	`bedrock-agentcore:*`	`agent-registry:*`
ARNs	`arn:aws:bedrock-agentcore:...:registry/...`	`arn:aws:agent-registry:...`
Events/metrics	Bedrock AgentCore sources	Agent Registry sources

The practical rule: keep Registry client names, endpoint assumptions, and IAM action prefixes isolated in one layer. Do not scatter them through experiment code.

What the First Search Miss Showed

The first capability-style search did not return the record:

tier two escalation agent for enterprise billing tickets

The name search did:

enterprise support escalation specialist

That miss shows that catalog quality matters. Search is only as good as the record metadata, descriptor content, and query vocabulary. A Registry record with thin descriptions creates a weak discovery surface. A record with clear capability language gives the caller more to work with.

For production use, publishing an agent includes the same care as publishing an API:

crisp description
clear capability names
versioning
owner metadata
examples of when to use it
examples of when not to use it

Without that, Registry becomes a list of endpoints. With it, Registry becomes a catalog of approved agent capabilities.

What Registry Does Not Solve

Registry does not eliminate the rest of distributed-agent design.

It does not prove the agent is safe to call. Approval helps, but runtime behavior still needs observability and policy. It does not solve identity propagation across an A2A hop. The caller still needs the right permissions, and the callee still sees the authenticated caller according to the runtime auth model. It does not prevent delegation loops. Multi-agent systems still need hop budgets and loop controls.

It also does not remove the need to understand the interface. In our test, the caller resolved an A2A Agent Card and invoked the agent through that card. That worked because the callee exposed a real A2A surface. A plain HTTP runtime would need a custom record or another descriptor strategy.

This is the right boundary. Registry is not an orchestration layer. Its job is to make approved capabilities discoverable with enough metadata for a caller to decide what to do next.

So What

If you are building a multi-agent system, the question is not "How do agents find each other?"

The better question is: "Which agents are approved for reuse, and how does a caller know?"

For a small system, an environment variable is fine. For an agent graph that spans teams, accounts, or protocols, discovery needs governance. Registry gives that governance a place to live.

The next frontier is not lookup. It is trust: richer metadata, better search vocabulary, signed cards, policy-aware invocation, and clear ownership for every reusable agent capability.

That is where the agent graph starts to look less like a pile of runtimes and more like an operating model.

The open question I am holding: I have tested single-caller discovery paths. The multi-agent pattern — agents that invoke each other over A2A, each with its own identity and Registry record — is where the authorization model gets genuinely harder, and it is where I haven't yet wired the full lifecycle.

Part of a series working through Amazon Bedrock AgentCore by building on it. Start with The AgentCore Map, then Where AgentCore Sits in the Architecture. The mechanics behind Registry: AgentCore Runtime and Two Ways to Authorize an Agent Tool.

What the Trace Knew That the Logs Didn't

Amit — Sat, 22 Aug 2026 06:06:52 +0000

The short version

Most observability posts break something on purpose and show the dashboard light up. I started differently: observability was the lens that caught failures in other experiments for months before I ran dedicated observability tests. That turned out to be the honest way to learn what observability actually buys — the failures weren't designed to be visible, so what showed up mattered.

Four accidental lessons came first (observability catching failures from unrelated experiments), then four deliberate ones (dedicated observability tests that each turned up something the previous tests had missed or misunderstood):

An agent that logged a complete answer while dead.
A tool invisible to evaluation systems because it didn't emit spans.
A two-agent loop only visible from the shared trace, never from either agent's logs.
A delegation where 95% of the trace was protocol noise.
A latency gap that looked like broken tracing but wasn't.
A loop-detection signal that's visible in traces but isn't proof on count alone.
A cross-agent correlator that works on two telemetry backends but hands you the wrong trace first.
An agent running as a plain local process showing up in the same observability pipeline as a Runtime-hosted one.

Each maps to a rule for debugging a misbehaving agent.

How AgentCore observability is wired

When you enable observability on AgentCore Runtime, every invocation emits OpenTelemetry (OTel) spans to CloudWatch — the same span/trace model any instrumented service uses. Agent frameworks emit these automatically: a span per model call, a span per tool execution, nested under a trace, all tagged with a session.id. The spans land in account-wide logs and a per-runtime log group. On top of the spans sit CloudWatch metrics — Invocations, Errors, SystemErrors, Latency — all queryable in the CloudWatch GenAI Observability console.

Two prerequisites: you must enable CloudWatch Transaction Search once per account (the docs don't mention it, but span queries silently return nothing without it), and there's a 2–10 minute ingestion lag between an invocation and its spans being queryable. Both cost me time because neither is documented loudly.

Lesson 1: logs lie, metrics don't

The first real lesson came from a deployment that looked successful and wasn't.

An early version of the deployment had a __main__ block that ran a one-shot test — print(json.dumps(triage(...))) — instead of starting the HTTP server. The agent called the model, escalated, produced a correct answer, and printed it to stdout. The logs showed a complete, correct response. But the process exited without ever starting the HTTP server. /ping had nothing listening, so the platform reported initialization failure. The agent was simultaneously working (logic ran, logged a perfect answer) and broken (nothing served, health check dead).

If I'd debugged from the logs, I'd have wasted hours. The metrics told the truth instantly: Invocations and a failure signature that didn't match the happy log line. Logs capture what the code printed; the platform contract is HTTP. A correct-looking log line is not proof the agent is serving.

Rule 1: Logs capture what the code printed. Metrics capture what the platform observed. When they disagree, the metrics are right. Check Invocations/Errors/SystemErrors and the client's actual HTTP response before you trust a log line.

Lesson 2: if there's no span, it didn't happen (as far as tooling is concerned)

The second lesson came from evaluation work, and nothing "failed" in the obvious sense.

I was scoring an agent with a trajectory evaluator, which grades whether the agent called the expected tools in order. The scenarios kept returning the correct outcome (a refund was approved or blocked as policy required) but showed an empty tool trajectory — [] — as if no tool had been called at all. The refund tool clearly had run; its effect was in the response.

The root cause: the tool was a plain Python function, not a framework-native one. Plain function calls don't emit OTel spans. The trajectory evaluator reads spans; if there are none, the trajectory is empty. The tool was working and completely invisible to the observability layer simultaneously.

Once I promoted it to a framework-native tool, the same scenarios showed a full trajectory. The deeper lesson: an empty trajectory is not proof the tool didn't run. It's proof the tool isn't instrumented. Understanding that made "empty trajectory" a useful signal in its own right — it tells you exactly which parts of your agent are invisible to everything downstream (evaluators, dashboards, audits), which is far more dangerous than a tool that visibly fails.

Rule 2: In an agent, "observable" is a property you opt into. Framework-native tools emit spans; raw function calls don't. Anything not emitting a span is a blind spot — not just for debugging, but for evaluation and audit. Instrument the actions that matter before you need to see them.

Lesson 3: some failures are only visible from above

This one would cost real money in production, and it's the clearest argument for tracing over logging.

Two agents — a frontline and a specialist — can each make locally-correct routing decisions and still form a global loop: frontline escalates to specialist, specialist decides it's out of scope and routes back, forever. There's a documented incident where exactly this ran for four weeks and roughly $47K before anyone noticed. No single agent's logs contain the loop. The frontline's logs show "I escalated." The specialist's logs show "I routed back." Neither shows the cycle.

The only place the loop is visible is a view that spans both agents. With a shared correlation id propagated across every hop — the same session.id on both agents' spans — you can see the entire path in one trace. Without cross-agent tracing, you're debugging a distributed failure from two half-views that each look fine.

Rule 3: Single-agent logs can't see multi-agent failures. Propagate one correlation id (session.id) across every hop so a single trace shows the whole path — that's the substrate that makes loops, dropped context, and delegation failures detectable at all.

Lesson 4: observability has a volume problem too

The fourth lesson is the counterweight, because more spans is not strictly better.

When I put a frontline→specialist delegation under evaluation, the shared session came back with 207 spans — and 197 of them were protocol plumbing (queue operations, fired dozens of times). The eight spans that actually mattered — the model calls and the tool execution — were buried in protocol noise, and the evaluator choked trying to reconstruct a coherent trace from the flood.

The fix was a span filter that dropped the protocol scope before analysis. The general point: an agent's trace is not automatically useful. Frameworks and protocols emit spans at wildly different granularities, and a naive "capture everything" posture produces traces where signal is a rounding error. You end up filtering by instrumentation scope to get back to the spans a human or an evaluator can reason about.

Rule 4: Capture-everything is not a strategy. Know which instrumentation scopes carry the signal for your question and filter to them. The goal is a trace you can read, not the most spans.

Lesson 5: a latency gap isn't proof the trace is lying

This came from a dedicated test, and it's a mistake I made and caught in the same session — worth keeping because it corrected its own first conclusion.

I invoked a deployed agent twice, once with a prompt that triggers a tool call and once with pure reasoning, and compared wall-clock against the trace's duration. Both showed the same pattern: the client measured roughly 6.5–7 seconds more than the trace accounted for. My first conclusion: a real gap in trace attribution, latency the trace simply couldn't see.

It wasn't. Both calls minted a fresh runtimeSessionId. AWS documents plainly that every unique session provisions a new, dedicated microVM. Startup splits into a platform-managed phase (microVM provisioning, before your instrumentation exists) and an application phase (your code). I confirmed directly: two calls on the same session collapsed the gap to about 0.2 seconds (replicated 3x). Cold-call wall-clock stayed at 9–12 seconds, but the attribution changed — the platform overhead was invisible to the trace by structural design, not by accident.

Rule 5: A latency gap between client wall-clock and trace duration is not automatically a tracing defect. Check whether you're minting a new runtimeSessionId per call first. A fresh session is a fresh microVM, and cold-start provisioning happens before your instrumentation exists, so it will never show as a span. Reuse session ids across real calls in a conversation.

Lesson 6: a repeated span isn't provably a loop

Tracing can show you a pattern, but a pattern isn't proof of a cause.

I forced an agent to make three tool calls in one turn (a three-part research prompt) and checked whether the trace showed a distinct signal. It did: three identically named spans, cross-confirmed by the runtime's logs. But a real stuck loop — the same call repeated because the agent isn't making progress — would produce the exact same span count. Neither the trace nor the application log carries the actual tool-call arguments, so span count alone can't tell the two apart.

AWS's own debugging guide resolves this cleanly: the primary infinite-loop signal is a volume anomaly — high token usage combined with a low or zero error rate, plus a span count and session duration far outside the normal range — not argument-level inspection as a first move. I confirmed this by temporarily enabling Bedrock model invocation logging (disposable, account-wide, so I disabled it immediately after) and re-ran the same test. The recovered model completion showed the real, distinct query strings behind each tool call — proof, not inference, that this was legitimate parallel tool use and not a loop.

Rule 6: Repeated identically-named spans are a real, countable loop symptom — but not proof by themselves, because legitimate parallel tool use produces the same shape. Reach for the volume signal first (token usage, span count, session duration vs. normal range). Only drop into argument-level inspection via Bedrock model invocation logging once a session already looks suspicious, and treat that logging as temporary — it has no per-agent scope.

Lesson 7: the correlator is real, but the first trace it hands you is a decoy

This came from a dedicated test run a month after Lesson 3, specifically to check whether that finding still held and on what infrastructure.

Lesson 3 established that a shared session.id correlates a two-agent A2A handoff on CloudWatch logs. The open question: does that correlation also work through AWS X-Ray? I invoked the same kind of delegation and checked X-Ray directly: it returned the session's spans just as reliably as CloudWatch had.

But checking twice surfaced a real gotcha. My first pass came back with zero matching rows — for a few minutes, that looked like the correlation had broken or the platform had moved this telemetry. It hadn't. A wider query window (my first check waited 45 seconds and searched a narrow slice of time; a retry with several more minutes of margin) found the rows, right in line with what I'd measured before. The lesson: "zero results" from an observability query is frequently just a too-tight window, not proof a backend has no data.

The second, more durable finding: the correlated session always contains at least two distinct traces, not one. The first is the A2A discovery handshake — the receiving agent fetching the caller's metadata — and it's essentially empty: near-instant, no model calls. The actual work is the second trace. Any tooling that assumes "the first trace in a session is the interesting one" will silently grab the handshake and miss the substance every time.

Rule 7: Don't trust a single empty query result as proof a telemetry backend has no data — widen the window before you conclude that. When correlating a multi-agent session, don't assume the first trace is the one that matters; the A2A discovery handshake produces a real, separate, empty trace ahead of the actual work.

Lesson 8: the pipeline doesn't care who deployed the agent

Does AgentCore Observability only work for agents AgentCore itself deployed, or does it reach agents running anywhere?

AWS's August 2026 blog answers plainly: agents on-premises, on other clouds, anywhere with internet access and IAM credentials, all land in the same dashboard as Runtime-hosted agents. I tested the plainest version of that claim: a local Python script on this machine, zero AgentCore Runtime involvement, calling Bedrock directly via boto3. Wrapped in opentelemetry-instrument with the documented AWS Distro for OpenTelemetry environment variables, it appeared in both telemetry backends tagged aws.service.type: gen_ai_agent, with the same gen_ai.* semantic-convention spans, the same token-usage metrics, and the same structured logs a Runtime-hosted agent produces — indistinguishable in the telemetry.

It wasn't frictionless. The first run threw a 400: The specified log stream does not exist on its first export batches — the log stream doesn't pre-exist, and ADOT creates it lazily on first write. That's a one-time, self-resolving cost, structurally the same as Lesson 5's cold start: an artifact of first contact with infrastructure that provisions itself lazily, not a persistent defect. A second run, after the stream existed, produced no errors.

Rule 8: AgentCore Observability's reach isn't bounded by AgentCore Runtime. The same ADOT SDK plus IAM credentials plus documented OTEL environment variables that instrument a Runtime-hosted agent will land a plain local process in the identical pipeline, tagged the same way. Expect a one-time, self-resolving provisioning hiccup on first contact with a new log group — the non-Runtime cousin of a cold start.

The operational workflow this leaves you with

Put the eight lessons together and you get a concrete way to debug a misbehaving AgentCore agent:

Start at metrics, not logs. Errors/SystemErrors/Latency on the runtime tell you whether the platform saw a failure. A clean-looking log line proves nothing (Lesson 1).
Pull the trace for the failed invocation by session.id. Wait out the ingestion lag — spans aren't instant.
Filter to the scopes that carry signal before you read it, or the protocol noise will bury the failure (Lesson 4).
Find the first span with an error status — that's your root-cause step. If the step you expect isn't in the trace at all, it isn't instrumented, which is its own bug (Lesson 2).
For multi-agent paths, confirm the correlation id propagated across every hop, or you're only seeing half the failure (Lesson 3) — and skip the first trace in the session, which is usually just A2A's discovery handshake (Lesson 7).
Before treating a latency gap as a tracing bug, rule out cold start. A fresh runtimeSessionId is a fresh microVM (Lesson 5).
Don't call a repeated span "a loop" on count alone. Reach for the token-usage/span-count volume signal first; only turn on model invocation logging, temporarily, if you need to prove it with actual tool arguments (Lesson 6).
Widen the query window before trusting a zero-result answer. An empty result from a telemetry query is frequently just a too-tight window (Lesson 7).
Set a CloudWatch alarm on Errors/SystemErrors so the next one pages you instead of hiding in a log that looks fine.
Don't assume Runtime deployment is a prerequisite for visibility. If part of your system runs outside AgentCore Runtime entirely (on-prem, another cloud, a plain process), the same ADOT-based setup gets it into the same dashboard — expect one lazy-provisioning hiccup on first contact (Lesson 8).

What's still open

Four of these eight lessons came from experiments that broke in ways I didn't plan for, and a trace (or missing one) that explained why. The other four came from finally testing observability directly — and two of those corrected their own first conclusion within the same session.

One thing I didn't test: the sequential "stuck, retrying across turns" pattern specifically. Lessons 2 and 6 tested parallel tool-calling and multi-step tool use, but not the case where an agent makes the same tool call, fails, and retries it turn after turn until the token budget runs out. That pattern is a real, non-redundant gap — AWS's own canonical loop-detection example uses a sequential loop, not a parallel one. The volume signals that work for detecting high token usage with low error rates apply to it, but whether the trace shape itself gives you a distinct signal for sequential retries (beyond volume) is something I haven't confirmed yet. That's the open thread I'm leaving with: the same observability pipeline that caught and distinguished the other seven patterns hasn't been tested against the one loop shape it's hardest to debug.

Multi-Agent Memory: Agents That Actually Share Context

Amit — Sat, 22 Aug 2026 06:06:16 +0000

The shared-context problem

The default way multi-agent systems share context is the prompt. Agent A finishes, you take what it produced, and you paste it into agent B's input. It works for two agents and one hop. It stops working the moment you have a chain: the context grows with every hand-off, you can't query it after the fact, and there is no boundary between what agent B should see and what it shouldn't. The "shared state" is a string that keeps getting longer and that every agent in the chain can read in full.

AgentCore Memory offers a different mechanism, and it's worth being precise about what it is. Agents don't message each other's memory. They write to and read from one shared memory store, and records are organized by namespace. An agent retrieves by naming a namespace prefix; it gets back only what lives under that prefix. The namespace is the sharing boundary and the isolation boundary at the same time — which is the whole idea, and also where the interesting failure mode lives.

I wanted to prove three things on live infrastructure: (1) a writer agent can deposit context that a reader agent recovers without that context being in the reader's prompt, (2) the recovery is by namespace, not by passing a session id or the text around, and (3) a second actor's namespace sees none of it. I built a minimal multi-agent system to test this.

The setup: one store, two agents, three actors

I modelled a real support workflow. A classifier agent (the "writer") stores a ticket determination under a shared actor namespace. A specialist agent (the "reader") queries that namespace and should recover the determination. A third actor — a different tenant — gets its own unrelated ticket, and a query scoped to its namespace must come back empty.

All three run against the same memory resource with the SEMANTIC strategy. The SEMANTIC strategy's namespace template is:

/strategies/{memoryStrategyId}/actors/{actorId}/

That template matters more than it looks, and I'll come back to why.

The classifier stores a short exchange: a customer posing a two-part incident, and the classifier's determination.

USER:      Customer Contoso reports two problems at once: a billing
           double-charge and a production API-key authentication failure.
           Are these one incident or two separate tickets, and who should own it?

ASSISTANT: Ticket classification: TIER-2 ESCALATION. The billing
           double-charge and the production API-key auth failure share a
           root cause: an identity-service outage. Treat as a single
           multi-system incident, not two tickets. Recommended owner: the
           named technical account manager.

The specialist never receives that text. It asks a question — "What is the escalation tier and owner for the Contoso multi-system incident?" — against the shared actor's namespace, and semantic retrieval is left to find the answer.

What happened

The specialist recovered the context on the first poll after extraction. Scoped to the shared actor's namespace, its query came back with a single record — a distilled summary of the incident (Contoso, the billing problem, the API-key auth failure) that the classifier had stored, with the incident text never in the specialist's prompt. The same query scoped to the decoy tenant's namespace returned zero records. Verdict: the specialist recovered the classifier's context by namespace, and it did not leak to the other actor.

Two things to be honest about in that record. First, what came back is the extracted fact — the durable summary of the incident — not the verbatim "TIER-2" label. That is what SEMANTIC does: it distills the conversation into a retrievable fact, and the fact carries the substance a specialist needs (the two simultaneous failures, the customer, the amount) so it doesn't re-derive the situation from scratch. If you need the literal label preserved word-for-word, semantic retrieval is the wrong strategy; a summarization strategy or an exact-record write is. Second, the isolation result is the quiet win: the decoy tenant's namespace returned zero, not "a low-scoring match." The boundary is structural, not a ranking threshold you hope holds.

This lines up with what AWS documents. The namespace-design guidance is explicit that retrieve_memory_records "Returns ONLY records stored at" the given namespace, and that namespaces give you "clean isolation between users, and IAM-based access control." The specification docs go further: you can write IAM policies that restrict retrieval by actor, session, and namespace scope, using those scopes as condition keys. So the isolation I observed at the API level can be enforced at the IAM level too — the application-level scoping and the identity-level policy reinforce each other.

The failure mode that cost me two runs

The first two times I ran this, the specialist got zero records and it looked like a broken feature. It wasn't. The classifier had stored a single lone ASSISTANT message — the determination, by itself. SEMANTIC extraction produced nothing from it, waited out, and I very nearly wrote "namespaced sharing doesn't work."

The fix, and the finding: SEMANTIC extracts facts from a conversation, not from a statement. The working semantic runs earlier always wrote alternating USER/ASSISTANT turns — a real exchange. When I changed the classifier to store the incident as a two-turn exchange (the customer's problem, then the classifier's determination), extraction produced a record within about 60 seconds and the specialist recovered it on the first poll. One message in, nothing out; a turn pair in, a clean fact out.

This is the same shape of trap that single-agent memory hits from a different angle: the feature is fine, but if you feed it the wrong input shape or query the wrong namespace, it hands you zero records and lets you believe it's broken. Two rules fall out of it:

Write conversations, not lines. If you want SEMANTIC to extract something, give it a turn it can extract from. A determination with no surrounding exchange is not something it will summarize into a durable fact.
Build the namespace from the strategy's real template. Querying /actors/{id}/ when the strategy writes to /strategies/{strategyId}/actors/{id}/ returns zero and looks identical to "extraction failed." Read the template off the resource; don't assume it.

The namespace design that makes this composable

The pattern that scales isn't "everyone shares one namespace." It's deliberate separation by actor:

A system/shared actor — for facts that many agents legitimately share (a ticket classification, a research finding, a computed result). Any agent scoped to that actor can read it.
A per-user actor (keyed by, say, the caller's JWT sub) — for context that must never bleed across users.

System facts don't pollute per-user recall, and user facts don't leak across accounts, because they live under different namespaces and retrieval is prefix-scoped. That's the same reasoning AWS's namespace-design guidance gives: treat it like a key schema or an object-storage prefix layout, and design the access patterns up front. This experiment is the minimal proof of the mechanism underneath that design — write under one actor, read it back by namespace, and confirm a different actor sees nothing.

So what

Shared memory across agents is real and it's clean: the specialist recovered the classifier's context by namespace with nothing passed in its prompt, and the isolation boundary returned a hard zero rather than a hopeful low score. That's a genuinely better primitive than prompt-stuffing — it's queryable, it's bounded, and the boundary can be backed by IAM.

The practical finding is the input contract. Semantic sharing works when you write a conversation and query the strategy's real namespace; it silently returns nothing when you write a lone message or guess the namespace prefix. Both mistakes look exactly like a broken feature and neither is. Design for the contract — turn pairs in, correct namespace out — and one memory store becomes the shared context layer a multi-agent system actually needs.

The thing I haven't fully solved: semantic extraction needs a real conversation to extract from. A single message in isolation — even if it's the exact determination the next agent needs — returns zero records and forces you to either feed it a fake exchange or fall back to prompt-stuffing. That's the honest limitation of the approach, and it's the whole reason this post exists.

Part of a series on Amazon Bedrock AgentCore. This post follows AgentCore Memory: What an Agent Remembers When the Session Is Gone — that post covers single-agent memory recovery across sessions; this one extends it to multi-agent sharing with namespace isolation.

Beyond Semantic Retrieval: Metadata Filtering in Agent Memory

Amit — Sat, 22 Aug 2026 06:05:40 +0000

When semantic search isn't enough

Semantic retrieval answers "what's related to this?" You give it a query, it returns records close in meaning. That's the right tool for "what has this customer told us about their architecture" — you want proximity, not exactness.

It's the wrong tool for "the high-priority billing records for this customer" or "events from the last seven days." Those aren't proximity questions; they're predicates. Within a single namespace, a broad semantic search returns everything close in meaning and gives you no way to say only these. That's what metadata filtering is for. AWS's framing is apt: namespaces isolate by primary entity (user, tenant), and metadata filtering narrows within a namespace by attribute — priority, category, department, time range. It's the WHERE clause for agent memory.

I set out to prove the WHERE clause works. It does — with a prerequisite that isn't optional and isn't loud about itself.

The setup: records that differ only by metadata

I wanted records that semantic search couldn't cleanly separate, so the metadata predicate would be doing the real work. Five support memories under one actor namespace, differing mostly in their tags:

Text	category	priority	amount
Disputed duplicate charge; refund issued	billing	high	4200
How to read the monthly billing statement	billing	low	—
Production API 500s during deploy; rolled back	reliability	high	—
Feature request: export usage as CSV	product	low	—
Billing overage of $1,300 flagged for review	billing	high	1300

Rather than wait for extraction — the slow path where an LLM reads events and pulls records — I wrote the records directly with batch_create_memory_records. That API takes custom content and a metadata map with typed values (stringValue, numberValue, stringListValue, dateTimeValue). It's the deterministic path: no extraction lag, records I fully control.

The write returned clean: 5 records written, 0 failed.

The gate: success that isn't

Then I listed the records back and inspected what metadata actually survived:

stored metadata keys observed:
  ['x-amz-agentcore-memory-createdAt',
   'x-amz-agentcore-memory-recordType',
   'x-amz-agentcore-memory-updatedAt']
custom keys (category/priority/amount) present on records? False

My category, priority, and amount were gone. The batch API accepted them, reported success, and dropped them on the floor. Only system-generated keys remained.

And filtering on a custom key doesn't fail soft — it fails hard:

[custom filter] category = billing:
  ERROR: ValidationException — Filter key 'category' is not a valid filter key

Not "zero results." A 400. The service refuses to filter on a key it doesn't know about.

This is the prerequisite the tutorials gloss over. Metadata filtering is gated on the memory having those keys declared as indexed metadata. For an extraction strategy, that's the strategy's memoryRecordSchema.metadataSchema — it tells the LLM which keys to populate. The AWS docs are explicit: "only keys defined in the strategy's metadataSchema are populated on extracted records — event metadata keys not in the schema are ignored." The same gate applies to direct batch writes: an undeclared key is not stored and not filterable. The memory I was writing to had no metadata schema and no indexed custom keys, so the custom tags had nowhere to live and nothing to filter against.

The working path: system-indexed keys

To confirm the mechanism is sound and it was only the index declaration missing, I filtered on the keys that are indexed by default — the system-generated ones:

[system filter] recordType = BASE:              5 record(s)
[system filter] createdAt AFTER 2026-07-01:     5 record(s)

Both work. An exact string match (EQUALS_TO on recordType) and a date-range predicate (AFTER on createdAt) both return correctly, no errors. So metadataFilters is real: the operators fire, the date type works, and the compound call is accepted. The feature isn't broken — it's conditional. Give it an indexed key and it does exactly what the WHERE-clause framing promises. Give it an undeclared key and it rejects the request.

I also ran the same query through retrieve_memory_records (semantic) with and without a metadata filter — "customer billing problem" alone, then the same query plus recordType = BASE. Both returned the set without error, confirming the metadata filter composes with semantic search in one call. (On five records the counts don't diverge; the point being verified here is that semantic search accepts an indexed-key predicate as a pre-filter, which it does.)

One retrieval-shape footnote that cost real time

A smaller trap worth recording: list_memory_records with the namespace parameter (prefix match) returned zero for my batch-written records, while the same call with namespacePath (hierarchical match) returned all five. The records were there the whole time; the prefix-vs-path distinction decided whether I saw them. This is the same class of mistake — query the wrong shape and a working feature looks empty. When batch-written records seem to vanish, try namespacePath before concluding the write failed.

When to use each strategy

The decision table for retrieval still holds — with the gate made explicit:

Use case	Approach	Prerequisite
"What has this user said before?"	SEMANTIC retrieval	none beyond the strategy
"How does this user prefer to be addressed?"	USER_PREFERENCE	none beyond the strategy
"Only high-priority billing records this week"	metadata filter	custom keys declared as indexed metadata
"Filter by when it happened / record type"	metadata filter on system keys	none — indexed by default
"What did the agent learn on similar tickets?"	EPISODIC	episode must be closed

So what

Metadata filtering on AgentCore Memory is a genuine capability, not a stub: string equality and date-range operators filter correctly, and the filter composes with semantic search in a single call. That's the WHERE clause the pitch promises.

But the capability is gated, and the gate is quiet. Custom metadata keys must be declared as indexed metadata on the memory — via a strategy's metadataSchema for extracted records, or the equivalent index configuration for direct writes — before they'll be stored or filtered. Skip that step and batch_create_memory_records still returns success while silently dropping your tags, and the first filter call fails with ValidationException: not a valid filter key.

The honest takeaway for anyone reaching for this: design your indexed metadata keys up front, the same way you'd design a database index, because an un-indexed tag on AgentCore Memory isn't a slow query — it's no query at all. The mechanism works the moment the index exists; it does nothing, loudly and then silently, until it does.

Sources

"Structured metadata for long-term memories" — AgentCore docs on metadata schema, per-strategy configuration, value types, and indexed key declaration
"batch_create_memory_records" — boto3 reference for direct record creation with metadata maps and typed values
"list_memory_records / retrieve_memory_records" — boto3 reference for retrieval with metadataFilters, operators, and namespace scoping

Bedrock Managed Knowledge Bases as a Native Gateway Tool

Amit — Sat, 22 Aug 2026 06:05:05 +0000

The Lambda you no longer have to write

Earlier in this series, giving an agent access to a knowledge base meant writing a Lambda that wraps bedrock:Retrieve, registering that Lambda as an AgentCore Gateway target, and letting the agent call it as an MCP tool. It worked. The Lambda was pure glue — it existed only to translate an MCP tool call into a Bedrock retrieval API call and translate the response back. You owned its code, its IAM role, its deployment, its cold starts, and its bugs, for a function that did nothing but forward a query.

AgentCore now has a native bedrock-knowledge-bases connector. You attach a Bedrock Knowledge Base directly to a Gateway as a managed-connector target, and the Gateway exposes the retrieval tools over MCP with no Lambda in the path. I replaced the wrapper with the connector, probed it over MCP, and found the one constraint that decides whether you can use it at all.

What the connector exposes

I attached a Knowledge Base to an existing Gateway as a bedrock-knowledge-bases connector target, then ran a SigV4-signed MCP tools/list against the Gateway. Two tools came back, auto-discovered, with no schema I had to write:

Retrieve — "Search the managed knowledge base for relevant source passages." A single hybrid search returning the most relevant passages (1–100, defaulting to 10). This is the direct analogue of what my old Lambda wrapped.
AgenticRetrieveStream — the interesting one. Per AWS's docs it "plans a retrieval strategy, runs multiple retrieval steps across your managed knowledge bases, optionally expands to full documents, and streams back both the supporting results and a synthesized, citation-backed answer." It takes a conversation (query + history), not just a string, and does multi-step retrieval with planning rather than a single lookup.

That second tool is a real capability jump. The Lambda wrapper only ever did single-shot retrieval, because writing a planning-and-multi-step-retrieval loop into a glue Lambda would have been a project. Here it arrives as a discovered tool the agent can call.

The Gateway handles schema management, endpoint resolution, and service authentication itself — the docs are explicit that the knowledge base tools are auto-discovered through tools/list and available to the agent without additional configuration. My probe confirmed it: both tools present, correct input schemas, no wrapper code anywhere.

The finding: the Lambda actually goes away

The point wasn't "a KB tool works" — the wrapper already proved that. It was whether the managed connector genuinely removes the custom infrastructure or just hides it.

Comparing the two targets on the same Gateway directly:

	Old path	New path
Target kind	`lambda`	`managed_connector`
Backend	a Lambda function	`connectorId: bedrock-knowledge-bases`
Custom code	a retrieval Lambda you own	none
Tools	one (whatever you coded)	`Retrieve` + `AgenticRetrieveStream`

The old target pointed at a Lambda ARN. The new target points at a connector ID. Lambda removed from new path: verified true. That is the whole value proposition: the retrieval integration moves from code you operate to configuration the platform operates, and you gain the agentic tool you wouldn't have built yourself. Same Gateway, same MCP surface, same auth model — one fewer thing to own.

The caveat that decides everything: Managed only

Here is the constraint that isn't obvious until it stops you. The connector is supported only for Amazon Bedrock Managed Knowledge Bases — not customer-managed ones.

Bedrock has two kinds of KB. A customer-managed KB is the classic one: you set up and operate the vector store (OpenSearch Serverless, Aurora, Neptune) and control ingestion, parsing, and indexing. A managed KB hands all of that to Bedrock — no vector store to provision, no retrieval infrastructure to run.

The connector only talks to the managed kind. So before I could attach anything, I had to create a managed KB and ingest the corpus into it — I couldn't point the connector at a pre-existing customer-managed KB. That's a real migration cost the feature grid doesn't foreground: "native Gateway integration" quietly assumes you're already on (or willing to move to) managed KBs. If your organization standardized on customer-managed KBs for control over the vector store, adopting this connector isn't a config change — it's a re-platforming of the knowledge base itself.

That's the honest trade. You delete a Lambda, but the price of entry is a Managed KB.

Governance stays where you'd want it

The reason this fits the "governed hub" pattern rather than being a pure convenience feature: the governance surface doesn't degrade when the Lambda disappears.

Least-privilege IAM survives. The Gateway service role needs only bedrock:Retrieve (and the agentic action) on the specific KB ARN — the same narrow grant the Lambda's role needed. The knowledgeBaseId is bound in the target's parameter values, not exposed to the model, so the agent can't redirect the query at a different KB by talking its way there.
Observability survives. I enabled Gateway vended-log delivery for the connector and captured lifecycle events for managed-KB calls in CloudWatch, plus Gateway metrics — the same audit trail the rest of the Gateway tools get. A retrieval tool with no audit trail is a liability; this one keeps the trail.
The MCP auth boundary is unchanged. Callers still reach the tool through the Gateway's existing auth (SigV4, aws_service="bedrock-agentcore"), so whatever inbound authorization and Cedar policy you've layered on the Gateway still applies. The connector didn't punch a hole around your controls; it slotted into them.

That's the right shape: less infrastructure to own, no loss of the controls that make a retrieval tool safe to expose to an agent.

So what

The bedrock-knowledge-bases connector is a clean win with a clear boundary. If you're on Managed Knowledge Bases, it deletes the retrieval Lambda outright, hands you an agentic multi-step retrieval tool you wouldn't have built, and keeps IAM, observability, and MCP auth exactly where they were. If you're on customer-managed KBs, the connector isn't for you yet — adopting it means migrating the knowledge base, not just re-pointing a target.

The broader pattern is the one this series keeps hitting: AgentCore is steadily replacing glue you operate with connectors the platform operates — first for APIs (the Gateway itself), then web search, now retrieval. Each one removes a Lambda and keeps the governance. The catch is always in the fine print of what the managed version requires — here, that your KB is the managed kind.

What I haven't tested yet is per-user access control at the document level within the KB. The managed connector works with IAM scoping (the Gateway role can only retrieve from this KB), but per-user authorization inside the KB — whether Cedar at the Gateway is enough, or whether you need managed KB document-level controls — remains an open question.

Harness vs. Runtime: When to Graduate

Amit — Sat, 22 Aug 2026 06:04:29 +0000

The short version

I ran the same support-triage task on both AgentCore run modes. Harness stood up a working agent in 20.4 seconds against 398 lines of Runtime Python, and it passed the IAM-gated and no-tool scenarios cleanly. It could not enforce per-user Cedar policies on the refund tool.

Both modes end up on the same compute. CloudTrail records harness operations under AWS::BedrockAgentCore::Runtime. The difference is who owns orchestration — you write the code (Runtime) or AgentCore handles it (Harness).

The documented graduation triggers are hooks, custom loops, and bidirectional streaming. The barrier I hit first was none of those — it was per-user outbound identity. What follows is what worked, what broke, and where the hard boundary is.

What Harness gives you without writing agent code

For nearly every AgentCore capability, Harness delivers it in configuration: Memory (all four strategies), Gateway tool wiring, Browser and Code Interpreter as built-in tools, outbound Identity, automatic Observability, model switching mid-session, versioning, and VPC networking. All available without customer code.

Runtime offers the same capabilities — with "you write the code required" for each. The session manager, the memory client, the tool orchestration, the span instrumentation — all available, all require code.

The three things the documented feature grid says Harness cannot do:

✗  Hooks (AfterToolCallEvent, BeforeToolCallEvent)
✗  Non-agent-loop patterns (graph, workflow, tree-of-thought)
✗  Bidirectional streaming

Those are the published graduation triggers. A fourth one — per-user outbound identity — isn't documented; I found it by running the config.

H1: Two API calls, no entrypoint code

The Harness create-and-invoke path is exactly as small as advertised. The entire orchestration for a triage agent — model, system prompt, one Gateway tool, memory — is about 25 lines passed to create_harness(). The equivalent Runtime agent is 398 lines. Harness is roughly 6% of the Runtime file.

Two measurements stand out:

Time to READY: 20.4 seconds. From create_harness() to a harness that answers, a little over 20 seconds. The equivalent Runtime path builds a container — around 180 seconds before the first invoke. Harness is ~9× faster to first response because there's no image to build.
The invoke is not SSE. boto3's invoke_harness returns a proper event stream. A JWT-protected harness requires raw HTTPS POST to /harnesses/invoke?harnessArn=... and parsing an application/vnd.amazon.eventstream body with botocore.eventstream.EventStreamBuffer.

Two scenarios passed cleanly:

The no-tool password reset: correct steps, clean end of turn, no tool call. Passed.
The web-search region question: the harness called the WebSearch gateway tool twice and cited live results. Passed.

Both authorize outbound with the harness's own IAM role (SigV4). There is no per-user identity involved. This is Harness at its best: a governed tool, wired by ARN, no code.

Then I pointed it at the Cedar-gated refund tool. It stopped there.

H2: The wall is identity, not hooks

The support agent has a refund tool behind a CUSTOM_JWT gateway with a Cedar policy engine in enforce mode. The policies are per-user:

permit  alice  process_refund  when amount < 500      // standard tier
forbid  bob    process_refund                          // blocked, any amount
permit  carol  process_refund  when amount < 2000     // enterprise tier

Cedar evaluates on the principal — explicitly the invoking user's identity from the JWT sub claim.

The Runtime path is trivial: the agent forwards the caller's own inbound JWT straight through to the gateway. alice's refund is permitted, bob's is forbidden, on the first call, for any user, with no extra machinery.

A Harness has no code, so it can't forward anything. It authorizes outbound calls through its configured outboundAuth, which offers exactly three modes: awsIam, none, and oauth. I tried the two that could carry identity.

awsIam outbound. The harness signs the gateway call with its own execution role. Cedar sees the role, not a user — there is no OAuthUser principal, so no per-user policy matches. The harness simply is not alice. Correct behavior, wrong outcome.

oauth outbound. This is the intended path: the harness mints the invoking user's token from AgentCore's Identity token vault. It's also where the real work was. Making it go took five distinct fixes, each a genuine requirement:

The response is a binary event stream, not SSE — parse with EventStreamBuffer, and headers are plain strings.
The execution role needs vault access — bedrock-agentcore:GetResourceOauth2Token plus secretsmanager:GetSecretValue. Without it the tool fails with AccessDeniedException.
The grant type defaults to the wrong flow. oauth outbound defaults to CLIENT_CREDENTIALS (machine-to-machine). The provider is user-federation (3LO) — it's either M2M or 3LO, never both. The request fails with Error parsing ClientCredentials response. Fix: set grantType=AUTHORIZATION_CODE.
Authorization-code outbound needs a return URL — defaultReturnUrl, registered on the workload identity.
The token is vaulted under the wrong identity. This is the blocker.

The vault is keyed on (workload_identity, user_id). I had pre-consented users and stored their tokens under a standalone workload identity. But a Harness calls the vault under its own workload identity — minted when the harness is created. It looks up alice under that identity, finds nothing, and falls back to starting a fresh browser-consent flow — which a headless server invocation cannot complete. Every case failed with You must provide a ResourceOauth2ReturnUrl to proceed with this flow.

Why re-vaulting doesn't fix it

You could fix issue 5 mechanically: create the harness, read back its generated workload identity, register return URLs on it, run the 3LO consent loop keyed to that identity, then invoke. It would work. It also carries three operational costs:

The harness's workload identity doesn't exist until the harness does, so you can't pre-consent users ahead of deploying.
It changes on every rebuild — a new suffix each time — so re-consenting all users becomes part of your deploy.
3LO requires per-user browser consent the first time regardless, which is inherent to the flow.

The one clean escape would be token passthrough — let the harness forward the inbound user JWT unchanged, the way Runtime code does. The gateway target model has a passthrough concept, but it is not exposed on the harness outboundAuth surface. Token exchange (OBO, RFC 8693) is the other theoretical route, and it's dead because Cognito doesn't implement it.

The honest finding: per-user Cedar through a Harness is architecturally possible and operationally impractical. Not because policy enforcement fails — Cedar at the gateway is identical for both paths — but because the code-less path binds outbound identity to an ephemeral, per-deploy workload identity and a consent step it can't drive headless.

The decision rule

Use Harness until you hit one of these. The first three are documented; the fourth is the one this run added.

Trigger 1 — Hooks. You need to intercept what the model sees after a tool call. The AfterToolCallEvent pattern — sanitizing an authorization failure, normalizing output, grounding the model on a deterministic fact — is Runtime-only. If your tool returns something the model must interpret rather than a clean fact, you need the hook.

Trigger 2 — Custom loop logic. Tree-of-thought, graph routing, supervisor/worker, A2A handoffs — any pattern where the orchestration loop itself carries business logic. Harness runs one managed loop; you can't fork it.

Trigger 3 — Model-level controls. Prompt-cache keys, extended-thinking budgets, per-token routing between a cheap model and an expensive one. Harness bedrockModelConfig exposes modelId, temperature, maxTokens, topP. Anything finer belongs in code.

Trigger 4 — Per-user outbound identity. If a tool enforces authorization on the end user's identity — per-user Cedar, an OAuth API scoped to the user, anything that needs the caller's own token downstream — the Harness path forces you through vault tokens keyed to an ephemeral workload identity plus a consent flow it can't run headless. Runtime forwards the caller's JWT in one line. This is the trigger the feature grid doesn't name, and for identity-aware agents it's the one you hit first.

What this means for the support agent

The support agent needs Runtime for two independent reasons, each tied to specific evidence.

The first is Trigger 1. The AfterToolCallEvent hook intercepts the policy engine's raw output and replaces it with a clean REFUND APPROVED / BLOCKED: fact before the model writes the customer reply. Without it, the model infers from raw authorization data — sometimes correct, sometimes fabrication. Cedar at the gateway works in Harness; the deterministic post-processing of Cedar's decision does not.

The second is Trigger 4. The refund tool is per-user Cedar. Runtime forwards each caller's JWT straight to the gateway, so the standard-tier user is permitted and the forbidden user is blocked on the first call with no vault, no pre-consent, no workload-identity juggling. The Harness equivalent is the five-layer, per-deploy, consent-bound path above.

Both are boundaries, not defects. Harness handled the IAM-gated web search and the no-tool tickets with 25 lines of config and a 20-second cold start — less work than the Runtime equivalent. The identity-bound, hook-dependent refund path is what needs code.

That is the graduation the two-model system is designed for: start on Harness, cross to Runtime when a trigger fires, and pay the code cost only for the parts that require it.

Results summary

Metric	Harness	Runtime
Config lines to working agent	~25 lines	398 lines
Time from create to first invoke	20.4 s	~180 s (container build)
Password reset (no tool)	pass	pass
EU region web search (IAM gateway)	pass	pass
Per-user Cedar refund (JWT gateway)	blocked — outbound identity	pass (JWT forwarded)
Memory wiring	one config block	~40 lines of code
Observability	automatic	ADOT + instrumentation

The pattern is consistent: for governed tools that authorize on the agent's own identity (IAM) or on nothing, Harness is less work and faster. For tools that authorize on the user's identity, or that need deterministic post-processing of the tool result, Runtime is not optional.

So what

Start on Harness. For governed tools that authorize on the agent's identity or on nothing, 25 lines of config and a 20-second cold start beat the Runtime equivalent. Watch for the identity boundary. The first time a tool needs to act as the user rather than as the agent, move to Runtime. That is where the config-only model ends and code begins.

I still haven't fully worked through what happens with token passthrough if it were exposed on the Harness outboundAuth surface — whether the per-user vault keying would become irrelevant if the harness could forward the inbound token verbatim. That's the operational escape hatch nobody's tried yet.

Build a Regression Suite for an Agent

Amit — Sat, 22 Aug 2026 06:03:54 +0000

A support agent I was testing produced a confident, empathetic reply telling a customer it had authority to override a refund policy. A human reviewer would have marked it good. Two quality scores agreed: correct and fully helpful.

The agent never consulted the policy engine. The authority was fabricated. The refund was blocked by design, and the agent wrote reassuring prose anyway.

That gap — a response that reads well, scores well on obvious metrics, and is wrong in ways only a behavioral check can catch — is the reason to build an evaluation suite. Not "does the output sound good," which is what humans and output-quality scores measure, but "did the agent do the right things, and was what it said true."

I built a suite against a live, policy-governed support agent. This is what four evaluation dimensions caught, and three classes of bugs nothing else did.

Four things worth measuring

Output quality — is the answer fluent and helpful — is the baseline. But on its own it rated a fabricated refund as a success. A useful suite measures four distinct things:

Goal success — did the agent accomplish the task behaviorally? Did it call the right tool, respect the policy, route to a human when it should. Phrased as assertions about observable customer outcomes, not tone.
Helpfulness — was the response useful and well-formed. The conventional metric.
Correctness — was what the agent said factually true, checked against ground truth. Fluent and factually wrong is its own failure mode.
Trajectory — did the actual sequence of tool calls match what the task required? This reads the execution trace, not the text.

The first two are easy to over-trust. The lessons below all come from the last two, and from goal-success phrased behaviorally.

The test setup

Eight scenarios, one support agent for a fictional analytics company. A refund tool gated by a real policy engine. Three users with different rules: one may approve refunds under $500, one is forbidden from refunds entirely, one may approve up to $2,000. The policy engine ran live on every tool call; the allows and denies were real, not mocks.

Simple ticket: password reset, no tools needed
Complex ticket: repeat billing, needs escalation
Refund allowed by policy ($45, under $500 limit)
Refund denied by policy ($750, over $500 limit)
Unconditional forbid (any amount, any user)
Enterprise-tier refund allowed ($1,500, under $2,000 limit)
Search-grounded question: EU region availability
Over-limit refund blocked, routed to human ($5,000 over $2,000 limit)

Lesson 1: outcome metrics can score a wrong agent perfectly

The fabricated-authority response was warm, articulate, actionable. Output-quality scoring loved it. The goal-success assertion — "the response tells the customer the refund is blocked by policy" — scored it zero, because the agent never mentioned policy, never said no, and offered to process refunds it had no authority to process.

The root cause was architectural: the refund check ran after the agent had already written its reply, so the policy decision never reached the model. The agent was composing support prose with no idea what the policy would decide.

A human reviewer would not have caught it — the reply read great. Only an assertion about what the agent should behaviorally do separated "sounded good" from "did the right thing."

The lesson: outcome-quality metrics can score a fully wrong agent perfectly. The behavioral goal-success assertion is what catches policy failures in the real world.

Lesson 2: the trajectory check is a probe for observability gaps

After moving the policy check earlier so the decision reached the model, goal success went green on those scenarios. The trajectory check came back empty.

Both were true: the refund tool ran and returned the right answer, and the trajectory evaluator could not see it. The reason: the refund was a plain Python function, not an instrumented tool, so it did not emit execution spans the trajectory evaluator reads.

The code worked. The observability layer was blind. Promoting the function to an instrumented tool made the same call appear in traces. Then trajectory went green.

That is the sharp, non-obvious value of trajectory scoring: an empty trajectory on a scenario where you know the tool ran tells you that part of your agent is invisible to tracing, and therefore invisible to debugging, audit, and production monitoring too.

The lesson: trajectory scoring is a test of your observability as much as your agent. A tool that runs invisibly to the evaluator is invisible to everything else too.

Lesson 3: a fluent answer can still be factually wrong

One scenario asked about regional availability. The agent correctly called web search — the trajectory confirmed it. It then wrote a clean, confident answer listing a UK region as meeting EU data-residency requirements.

Goal success passed: the assertions checked that the agent used search, addressed the compliance question, and cited results — all true. The correctness check, which only judges factual accuracy, flagged it: post-Brexit, a UK region does not satisfy EU data residency. Meaningful factual error, inside a response that followed the right procedure.

This is why goal success and correctness are both in the suite and neither replaces the other. Goal success tells you the agent did the right things; correctness tells you whether what it said was true. A "did the right things, said something false" result is a real failure mode. You only see it if you measure both.

The fix that lasts: turn the caught error into a behavioral assertion, so the next run fails if the agent repeats the claim. The hallucination a judge caught once becomes a deterministic regression test forever.

The property that makes it a regression suite

Each lesson, once caught, became a locked scenario in a fixed dataset. That is the mechanism that turns a pile of evaluations into a regression suite: a failure found today becomes a permanent test on every future deploy. Nobody has to remember what went wrong last week — the scenario remembers it. Reintroduce the bug and the scenario goes red.

Two things make the locked dataset durable rather than brittle:

Assert on observable outcomes, not internal architecture. An assertion like "the agent classifies this as complex tier" breaks the moment you change the architecture. "The response confirms the refund was approved" tests the business requirement and survives any rewrite underneath. When I first wrote assertions against the agent's internal steps, they went stale immediately. Rewriting them as customer-observable outcomes is what made the suite stable.
Run the same evaluators in production. The suite's value multiplies when the exact evaluators used in development also sample live traffic. A policy regression that drops goal success from green to red in the test dataset shows the identical drop in production sampling. Alarm on it directly with no separate pipeline.

What the suite caught

On the first run, all eight scenarios fired. The suite caught:

An ordering bug where the policy decision never reached the model (caught by goal success)
Instrumentation gaps making refund calls invisible to tracing (caught by trajectory)
A policy engine that was not attached to the gateway (caught by goal success reverting to zero)
A factual error about Brexit and UK region classification (caught by correctness)
Missing acknowledgments of billing patterns (caught by goal success with tight assertions)
A missing runtime configuration for policy gateway URLs (caught by trajectory and goal success converging on failure)

The single line of signal that connected all of them: the evaluators ran the same way in development and production, and they all measured behaviors that mattered, not just prose quality.

So what

The reason to evaluate an agent like software — not like a demo — is that the failures that matter are invisible to the checks you reach for first. A human reviewer and an output-quality score both passed a response that fabricated authority and ignored a policy. Only a behavioral goal-success assertion caught it. Only a trajectory check caught a tool that ran but could not be traced. Only a correctness check caught a fluent, procedurally correct, factually false answer.

So the suite worth building measures four things and leans on the three past the obvious one: did it do the right things behaviorally, was what it said true, and did the tool calls actually get traced. Lock every failure into a fixed dataset, phrase the assertions as customer-observable outcomes, and run the same evaluators against production. That is the distance between "the agent looked fine in the demo" and "the agent is still doing the right thing on the ten-thousandth ticket."

What's still open

I proved this on one domain with eight scenarios, which is enough to find the pattern but not enough to say it scales. The real unknown is running the suite on an agent that matters — one with enough agent-driven branching that the locked dataset starts to feel like a bottleneck instead of a check. Eight scenarios caught real bugs. Eight hundred? That is where the cost of maintaining a fixed dataset starts to matter against the value of what it catches. I have not hit that edge yet.

AgentCore Browser: A Managed, Isolated Browser an Agent Can Drive

Amit — Sat, 22 Aug 2026 06:03:18 +0000

Some work only exists on a screen. A carrier portal that won't show a rate until you log in. A price that isn't in the HTML until you scroll. A vendor dashboard that exports data only after three nested clicks. None of it has an API, and that's exactly why it's still manual. Industry surveys put browser-bound, portal-heavy processes as the single biggest place agent pilots stall — reported at 79% of organizations adopting agents but only 11% running them in production, with portals cited as a named blocker.

The reflex, when you hit one of these, is to run Playwright on a box. That works until you ask the operational questions: where does the browser run, who else can see its session, what happens when the agent logs into something as a user, and how do you audit what it did. Those are the questions AgentCore Browser answers. It is a managed, per-session, isolated Chromium that an agent drives — and the "managed and isolated" part is the whole point, not an implementation detail.

This is the browser organ from the AgentCore map. Reach for a browser only after ruling out a native API and an existing connector. This post covers what the managed primitive actually gives you, and where it stops.

What it is: a browser as a session, not a process

The mental shift is the same one Runtime asks for. You don't run a browser; you start a session. Each session is its own containerized (Firecracker) environment, isolated from your system and from every other session, ephemeral, and terminated on a TTL or an explicit stop. You get a CDP (Chrome DevTools Protocol) endpoint and drive it with Playwright — the same Playwright API you'd use locally, pointed at a browser AWS operates instead of one you manage.

I drove it directly to confirm the basic loop. One session, against a scraping-friendly sandbox site: start the session, connect Playwright over the CDP WebSocket, navigate, scroll, read the DOM, close.

managed session started: session_id=01KXP11N...
navigate -> HTTP 200  title='All products | Books to Scrape - Sandbox'
extracted: 20 product listings, 20 price tokens
browser closed

Nothing about that is surprising once you hold "it's a session" in your head. The session id is the handle; the CDP endpoint is how you drive it; the isolation and the teardown are the platform's job, not yours.

How you reach it: two paths

There are two ways to call the managed browser, and picking the right one is most of the practical decision.

The local MCP path — interactive, zero infrastructure. AWS ships an MCP server (awslabs.amazon-bedrock-agentcore-mcp-server) that, scoped to the browser primitive, exposes the full ~21-tool Playwright-style surface — browser_navigate, browser_click, browser_type, browser_snapshot, browser_evaluate, browser_mouse_wheel, and the start/stop/get/list_browser_session lifecycle. You wire it into a client, it authenticates with your AWS profile, and you're driving a managed browser with no code to maintain. This is the daily driver for interactive and exploratory work.

The governed path — IAM-gated, server-side, autonomous. For an agent that runs without a human at the keyboard, the browser sits behind an AgentCore Gateway (IAM-authorized) in front of a Lambda that drives the CDP session. Now the browser is a first-class, governed tool: only principals in your account can call it, it composes with Cedar policy and AgentCore Identity, and it's the reach path an autonomous Runtime agent uses. A SigV4-signed navigate → extract → close through the gateway returns results multi-call, with an application-level SSRF guard rejecting metadata/loopback/private-range URLs. It's the same managed browser underneath; the difference is who's allowed to drive it and what governs the call.

The rule of thumb: local MCP for a human exploring, governed gateway for an agent operating.

What it's for: the use cases, grounded

The managed browser earns its place on the tasks a plain HTTP fetch can't do — dynamic, JavaScript-rendered, or behind interaction. The market taxonomy is consistent: web research and extraction dominates, followed by form-filling and authenticated portal workflows.

Research and extraction on real e-commerce. A consumer task — find the best-value 65" and 55" TVs — run through the managed browser against a live retailer. It surfaced a detail worth keeping because it shows the primitive doing real work: the prices weren't in the initial DOM. Modern retail sites lazy-load prices via an IntersectionObserver, so the elements render only after the viewport scrolls past them. The pattern that works, every time:

navigate(url)            -> DOM loaded, price elements empty
mouse.wheel(0, 800)      -> triggers lazy-load, prices render
evaluate(innerText regex)-> extract the now-present prices

A document.body.innerText.match(/\$[\d,]+/g) scan proved far more durable than CSS selectors, because the site rewrites its class names frequently. The run also caught a genuine pricing anomaly (a newer TV priced below its older sibling) that a systematic scrape surfaces and a human skimming the page would miss — which is the actual argument for automating it.

Autonomous multi-step research. In a parallel experiment, an agent used the managed browser to answer a "find three highly-rated lunch spots within a 5-minute walk, with a signature dish each" prompt end to end — navigate, cross-reference two sites (it fell back from one that blocked scraping to another's data plus a maps site for walking times), synthesize, and save the result to a file, with zero human intervention and zero errors across a ~20-minute run. That's the #1 use case — research and extraction on sites without a clean API — running autonomously on the managed browser.

The differentiator: isolation you can prove

Anyone can run Playwright. What the managed browser adds is the operational envelope, and the first piece of it is isolation. The docs say each session runs in its own isolated environment; I wanted to prove it rather than trust it, so I ran the browser analog of a memory-isolation test.

Two sessions, same origin. In session A, I wrote a distinctive secret to both localStorage and a cookie, then read it back within the session. Then I opened a separate session B and read the same keys:

session A  ->  wrote + read back: SECRET-3d33...   (state persists in-session)
session B  ->  localStorage=None, cookie=None      (sees nothing of A)
VERDICT: PASS — isolation holds

Session A kept its own state; session B, a concurrent separate session, saw none of it. That is the property that matters when two agents — or two tenants' agents — drive browsers at the same time: no shared cookie jar, no leaked login, no cross-contamination. It's structural, not a setting you hope holds.

Two more governance features come from the same "managed" fact, and they're the reason to reach for this over a self-run browser:

Live View — human-in-the-loop takeover. The platform can stream a real-time video feed of the session over DCV (via a time-limited SigV4-presigned URL), and a human can take control mid-session and hand it back. That's the concrete answer to "the agent hit a login it can't solve" or "this refund needs a human to look" — the human-in-the-loop seam made real for the browser.
Session replay — audit to your S3. A recording-enabled browser writes the session's DOM changes, actions, console, and network events to your S3 bucket, replayable in the console or programmatically. For a browser acting on your behalf against a portal, that recording is the audit trail. The configuration is a recording block on the browser ({enabled, s3Location: {bucket, prefix}}) and a least-privilege IAM role that can write to that prefix and log to CloudWatch — nothing else.

None of these three — provable isolation, live human takeover, replay-to-S3 — is something you get for free by running Playwright on an EC2 instance. They're the managed envelope, and they're what make the browser a governed tool rather than a script.

The ceiling: managed is not stealth

The honest limitation, stated plainly because the post is worthless without it: the managed browser is a clean, isolated Chromium — not an anti-detection product. It does not rotate fingerprints, match TLS/JA3 signatures, or solve CAPTCHAs, and well-configured anti-bot systems detect it.

I ran a small matrix to characterize exactly where it stops. Open sites navigated fine; an anti-bot-protected site did not:

books.toscrape.com   -> HTTP 200   PASS
httpbin.org/html     -> HTTP 200   PASS
costco.com/...       -> net::ERR_HTTP2_PROTOCOL_ERROR   BLOCKED (before render)

Costco's protection refused the managed Chromium at the connection layer — an HTTP/2 protocol error before a single byte of page rendered. This isn't a bug to file; it's the design boundary. Modern anti-bot (Cloudflare, DataDome, PerimeterX, and AWS's own WAF Bot Control) fingerprints TLS handshakes, HTTP/2 SETTINGS frames, header ordering, and behavioral signals. A clean cloud browser with datacenter egress and no stealth layer is, correctly, identifiable as automation.

If your task requires getting past aggressive bot protection, AgentCore Browser is not the tool — you're in residential-proxy-and-stealth territory, which is a different product category and a different risk conversation.

Know the ceiling and it stops being a surprise: the managed browser is for the open web, your own authenticated portals, and sites that don't actively fight automation — which is most enterprise back-office work — not for adversarial scraping of sites that have decided they don't want you.

When not to use it

The primitive has a clear negative space, and naming it is part of positioning it honestly:

High-throughput pure scraping — if you need 10,000 static pages fast, raw Playwright or a scraping API is an order of magnitude quicker and cheaper. The managed browser's value is isolation and governance per session, not raw crawl throughput.
Testing your own known preview URL — a build pipeline smoke-testing an app it just deployed wants local @playwright/test against that URL in CI, not a managed cloud browser. There's no untrusted web and no governance need, so the managed envelope buys nothing.
A site that has an API — a browser should follow a native API and an existing connector. Don't drive a UI for data you could fetch.

So what

AgentCore Browser is what you reach for when the task is genuinely browser-bound and you need the operational envelope: a per-session isolated Chromium, drivable with ordinary Playwright over CDP, reachable interactively through a local MCP or as an IAM-gated governed tool for an autonomous agent. The isolation is real — two sessions, zero cross-visibility. It does real research and extraction on live sites. It fails cleanly and predictably against anti-bot protection.

The one-line decision: if you'd otherwise stand up Playwright on a box and then have to solve isolation, human-takeover, and audit yourself, use the managed browser — that's the work it does for you. If you just need to hit an API, or hammer static pages, or test your own URL, don't — the managed envelope is overhead you don't need. Match the tool to whether the task is governed and browser-bound, and the primitive earns its place exactly where the screen is the only interface and someone has to be able to prove what the agent did on it.

What's missing

The managed browser is a first-class primitive for browser-bound work inside your account, with the operational envelope built in. What it doesn't ship: stealth against adversarial anti-bot systems. That's the one thing a managed cloud browser structurally cannot hide — it's a datacenter egress with a clean fingerprint, and sites fighting automation will detect it. If stealth is your problem, the tool you need isn't a managed browser on a native cloud infrastructure — it's a purpose-built browser agent with a forked Chromium underneath, which is a different product category entirely.

Grok 4.6 Is on Amazon Bedrock. Here Is How to Wire Grok Build to It

Amit — Fri, 21 Aug 2026 20:39:52 +0000

On August 19, 2026, AWS made SpaceXAI's Grok 4.6 available on Amazon Bedrock. The launch added US and Global cross-Region inference through Bedrock Runtime, plus OpenAI-compatible API access.

That creates an obvious builder question: can Grok Build, SpaceXAI's coding CLI, run its own model through an AWS account rather than a separate provider key and billing boundary?

Yes. Grok Build can call Grok 4.6 on Bedrock directly, without a local inference proxy or a long-lived provider key. The working setup uses Grok Build's custom-model support, Bedrock's OpenAI-compatible endpoints, and a small credential helper that converts an existing AWS session into a short-lived bearer token.

I wired and tested three Grok 4.6 routes:

Model picker entry	Bedrock path	Model ID	Client Region
Runtime US	US cross-Region inference	`us.xai.grok-4.6`	`us-east-1`
Runtime Global	Global cross-Region inference	`global.xai.grok-4.6`	`us-east-1`
Mantle	In-Region inference	`xai.grok-4.6`	`us-west-2`

All three run from the official grok command. Selecting the Bedrock route happens inside Grok Build with /model.

The comparison is operational: Claude Code can run Claude through Amazon Bedrock, and Codex can run OpenAI models through Amazon Bedrock. This setup gives Grok Build the equivalent path for Grok. The CLI remains the model provider's own coding harness, while AWS supplies authentication, inference access, routing, and billing.

The three Grok Build entries do not select different model families. They select three Bedrock access paths for the same Grok 4.6 model.

TL;DR

Grok 4.6 became available on Amazon Bedrock on August 19, 2026. Grok Build has no native Bedrock provider, but its custom models and named credential helpers cover the integration.
The same AWS profile used by other developer tools remains the source of identity; the helper only turns that session into a regional Bedrock bearer token.
Grok Build reads expires_in from the helper response and reruns the helper before the token expires.
Runtime US, Runtime Global, and Mantle require different model IDs, and Mantle Grok 4.6 is available in us-west-2.
One grok process can list and switch among all three routes for the same Grok 4.6 model. The model picker becomes the routing control.

Why the credential helper exists

AWS credentials and a Bedrock bearer token represent the same underlying IAM identity, but they are different request credentials.

A client with native Bedrock support can read an AWS profile and authenticate the request itself. Grok Build treats a custom model as an OpenAI-compatible service. It expects bearer authentication and does not interpret an AWS profile directly.

The credential helper closes that narrow gap:

AWS profile
    ↓
AWS credential provider chain
    ↓
regional short-lived Bedrock token
    ↓
Grok Build → Bedrock OpenAI-compatible endpoint

The helper does not proxy inference. Grok Build sends the model request directly to Amazon Bedrock. The helper runs only when Grok needs a token.

Amazon Bedrock short-term API keys inherit the permissions of the IAM principal that creates them. Their lifetime cannot exceed the underlying AWS session. Grok Build's per-model authentication providers accept a command that returns an access token and its lifetime, then refresh that token automatically.

Verify the AWS identity first

This setup starts with a working AWS profile that already has permission to invoke the selected Grok model.

aws sts get-caller-identity --profile YOUR_AWS_PROFILE

Read the returned account and principal before continuing. A valid AWS session in the wrong account produces a configuration that looks correct and fails at inference time.

Install or update Grok Build:

curl -fsSL https://x.ai/cli/install.sh | bash
grok update --stable
grok --version

I validated this configuration with Grok Build 1.0.5.

Install the Bedrock token generator

Keep the Python dependency under Grok's own directory rather than changing the system Python environment:

mkdir -p "$HOME/.grok/bedrock-auth" "$HOME/.grok/bin"

python3 -m pip install \
  --upgrade \
  --target "$HOME/.grok/bedrock-auth" \
  aws-bedrock-token-generator

AWS documents the package in its guide to generating Amazon Bedrock API keys.

Create ~/.grok/bin/bedrock-token-helper:

#!/usr/bin/env python3
"""Print a short-lived regional Bedrock token for Grok Build."""

import argparse
import json
import os
import sys
from datetime import timedelta

sys.path.insert(0, os.path.expanduser("~/.grok/bedrock-auth"))

from aws_bedrock_token_generator import provide_token


TOKEN_TTL_SECONDS = 2700


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--profile", required=True)
    parser.add_argument("--region", required=True)
    args = parser.parse_args()

    os.environ["AWS_PROFILE"] = args.profile
    os.environ["AWS_REGION"] = args.region
    os.environ["AWS_DEFAULT_REGION"] = args.region

    try:
        token = provide_token(
            region=args.region,
            expiry=timedelta(seconds=TOKEN_TTL_SECONDS),
        )
    except Exception as error:
        print(f"Unable to generate Bedrock token: {error}", file=sys.stderr)
        return 1

    print(
        json.dumps(
            {
                "access_token": token,
                "expires_in": TOKEN_TTL_SECONDS,
            }
        )
    )
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Then restrict execution to the current user:

chmod 700 "$HOME/.grok/bin/bedrock-token-helper"

The command must write only JSON to standard output. Diagnostic messages belong on standard error because Grok parses standard output as the credential response.

Add the three Bedrock models

Add the following to ~/.grok/config.toml, replacing YOUR_AWS_PROFILE with the profile verified earlier:

[auth_provider.bedrock-us-east-1]
command = '"$HOME/.grok/bin/bedrock-token-helper" --profile YOUR_AWS_PROFILE --region us-east-1'
timeout_secs = 30

[auth_provider.bedrock-us-west-2]
command = '"$HOME/.grok/bin/bedrock-token-helper" --profile YOUR_AWS_PROFILE --region us-west-2'
timeout_secs = 30

[model.bedrock-grok-us]
model = "us.xai.grok-4.6"
base_url = "https://bedrock-runtime.us-east-1.amazonaws.com/openai/v1"
name = "Grok 4.6 - Bedrock Runtime US"
description = "US cross-Region inference through Bedrock Runtime"
api_backend = "chat_completions"
auth_provider = "bedrock-us-east-1"
context_window = 500000

[model.bedrock-grok-global]
model = "global.xai.grok-4.6"
base_url = "https://bedrock-runtime.us-east-1.amazonaws.com/openai/v1"
name = "Grok 4.6 - Bedrock Runtime Global"
description = "Global cross-Region inference through Bedrock Runtime"
api_backend = "chat_completions"
auth_provider = "bedrock-us-east-1"
context_window = 500000

[model.bedrock-grok-mantle]
model = "xai.grok-4.6"
base_url = "https://bedrock-mantle.us-west-2.api.aws/openai/v1"
name = "Grok 4.6 - Bedrock Mantle"
description = "In-Region inference through Bedrock Mantle in us-west-2"
api_backend = "chat_completions"
auth_provider = "bedrock-us-west-2"
context_window = 500000

[models]
default = "bedrock-grok-us"
default_reasoning_effort = "medium"

The model IDs are not interchangeable. Amazon Bedrock's Grok 4.6 model card distinguishes Runtime inference profiles from the Mantle base model:

Runtime US uses us.xai.grok-4.6.
Runtime Global uses global.xai.grok-4.6.
Mantle in us-west-2 uses xai.grok-4.6.

The xai namespace remains part of the public model ID even though the vendor is now SpaceXAI.

Launch once and switch models

Start Grok Build with its normal command:

grok

Use /model inside the CLI to select:

bedrock-grok-us
bedrock-grok-global
bedrock-grok-mantle

The selected entry determines the endpoint, Region, model ID, and credential provider. One Grok session can move among all three Grok 4.6 routes without restarting a proxy or exporting a new API key.

For a non-interactive smoke test:

grok --model bedrock-grok-us \
  -p "Reply with exactly: BEDROCK_GROK_OK"

Repeat with bedrock-grok-global and bedrock-grok-mantle. A successful response proves more than token generation: it confirms the AWS identity, regional token, endpoint, model ID, and Grok request path work together.

Token refresh is part of the design

The helper requests a 45-minute token and returns "expires_in": 2700. Grok Build uses that field to cache the token in memory and rerun the provider before expiry. Its authentication guide states that external credentials refresh proactively and retry after an authentication rejection.

The AWS profile remains the actual source of authority. If it uses IAM Identity Center, credential_process, or another refreshable provider, each helper run resolves the current session through that provider chain.

There is one hard boundary: an expired upstream AWS login cannot refresh itself through Grok. Reauthenticate the AWS profile, then retry the model request. No Bedrock token helper can extend an AWS session that has already expired.

What this setup does not solve

This pattern gives Grok Build direct Bedrock inference and automatic credential rotation. It does not turn Grok Build into a native AWS client.

Native Bedrock integrations can expose AWS-specific configuration directly. Grok Build sees an OpenAI-compatible model endpoint plus bearer credentials. That difference is why the helper exists and why the Runtime and Mantle routes must be declared explicitly.

The setup also depends on current model availability. Grok 4.6 on Mantle is a us-west-2 route in the current AWS model card. A model appearing in one Region does not imply that the same base model ID works in every Mantle Region.

So what

The durable pattern is smaller than a proxy:

one AWS identity
    + one regional token helper
    + explicit model routes
    = one Grok Build CLI with a Bedrock model picker

The coding harness stays Grok Build, and the model stays Grok 4.6. Amazon Bedrock supplies model access, routing, and billing. The developer launches the official grok command and selects the required AWS route.

The open question is whether Grok Build will add a native Bedrock provider. If it does, the helper can disappear and the three model entries can collapse into AWS-aware configuration. Until then, named credential providers are the smallest integration layer that preserves direct inference and short-lived identity.

DEV Community: Amit

An Agent That Can Pay for Its Tools

What The Payment Primitive Owns

The First Proof: HTTP 402 To HTTP 200

Budget Is Not A Prompt Instruction

Cedar Before Payment

Observability Is Part Of The Product

Agent-Native Payment Flow

Two Prerequisites That Will Block You Cold

Quick Create's Seatbelt

Three Bugs That Weren't AWS's

What Is Still Missing

So What

MPP vs x402: Two Competing Answers to the Same HTTP 402

Same shape, different commitments

x402's upto scheme: pay for what you actually used

MPP: one interface, four rails behind it

When to reach for which

The open thread

A Browser That Pays

The flow

The wrinkle: two different 402s

Why this matters

What's missing

AgentCore Registry Is a Governed Catalog for Agent Interfaces

Hardcoded ARNs Are Only Part of the Problem

The Experiment

What the Evidence Supports

The Namespace Migration Is Part of the Contract

What the First Search Miss Showed

What Registry Does Not Solve

So What

What the Trace Knew That the Logs Didn't

The short version

How AgentCore observability is wired

Lesson 1: logs lie, metrics don't

Lesson 2: if there's no span, it didn't happen (as far as tooling is concerned)

Lesson 3: some failures are only visible from above

Lesson 4: observability has a volume problem too

Lesson 5: a latency gap isn't proof the trace is lying

Lesson 6: a repeated span isn't provably a loop

Lesson 7: the correlator is real, but the first trace it hands you is a decoy

Lesson 8: the pipeline doesn't care who deployed the agent

The operational workflow this leaves you with

What's still open

Multi-Agent Memory: Agents That Actually Share Context

The shared-context problem

The setup: one store, two agents, three actors

What happened

The failure mode that cost me two runs

The namespace design that makes this composable

So what

Beyond Semantic Retrieval: Metadata Filtering in Agent Memory

When semantic search isn't enough

The setup: records that differ only by metadata

The gate: success that isn't

The working path: system-indexed keys

One retrieval-shape footnote that cost real time

When to use each strategy

So what

Sources

Bedrock Managed Knowledge Bases as a Native Gateway Tool

The Lambda you no longer have to write

What the connector exposes

The finding: the Lambda actually goes away

The caveat that decides everything: Managed only

Governance stays where you'd want it

So what

Harness vs. Runtime: When to Graduate

The short version

What Harness gives you without writing agent code

H1: Two API calls, no entrypoint code

H2: The wall is identity, not hooks

Why re-vaulting doesn't fix it

The decision rule

What this means for the support agent

Results summary

So what

Build a Regression Suite for an Agent

Four things worth measuring

x402's `upto` scheme: pay for what you actually used