DEV Community: Fernando Azevedo

Amazon Quick on a private network: PrivateLink's two planes and where they leak

Fernando Azevedo — Fri, 23 Oct 2026 15:48:23 +0000

When someone from compliance asks whether Amazon Quick "goes over the internet", the honest answer has two parts, and almost every design I have reviewed answers only one. User traffic into Quick uses PrivateLink interface endpoints. Quick's traffic out to your on-premises database, your private MCP server or your internal API uses something else: ENIs that Quick itself injects into your VPC. DNS, endpoint policy, IP/VPC restriction and IAM sit on top of those two planes, and each layer covers less than its name suggests. The useful question is not "is it private?", it's "which byte still leaves for the internet, and who decided that was fine?".

Why this matters now

While Quick was just BI, the VPC connection existed so Quick Sight could read PostgreSQL, SQL Server or Oracle behind a Direct Connect. The problem was well understood and the blast radius fit inside a dashboard. What changed is who uses that same VPC connection. It now carries Quick Agents calling MCP servers hosted on your network, Quick Flows and Quick Automate invoking action connectors, and Quick Research and Apps in Quick pulling from private sources. The ENI that used to run a SELECT for a chart now carries an agent's tool call.

In financial environments this matters in two places. Regulation: CMN Resolution 4.893 and PCI-DSS expect you to show the data path and control who enters the perimeter. The AWS post lists HIPAA, FedRAMP and PCI-DSS as the usual triggers. Audit: "traffic stays on the AWS backbone" only works as evidence if you can say which domains still leave through NAT, and some do.

The 21 Oct 2026 post is a good walkthrough. What it doesn't do, and isn't meant to, is say where each control stops working. That's what this piece covers: the documented limits of each layer, and the cost of keeping all of it working for years, not the cost of building it once.

The two planes of private Quick access, and what stays public

Ingress (user to Quick) over interface endpoints. Egress (Quick to your data) over VPC connection ENIs. Dashed lines are traffic no endpoint covers.

🏢 On-premises: rede corporativa

DNS corporativo forward de .quicksight.aws.amazon.com (network)
Oracle / SQL Server porta do banco liberada só para o SG do Quick (data)
Servidor MCP privado API interna por IP (compute)

📡 Rede: Direct Connect ou Site-to-Site VPN

Direct Connect rota para o CIDR da VPC (network)

🟧 AWS: VPC do cliente (2 AZs no mínimo)

Route 53 Resolver inbound 2+ ENIs (network)
VPCE quicksight-website policy só com aws:PrincipalAccount / aws:OrgId (security)
VPCE quicksight (API) policy por ação; 13 APIs fora (security)
ENIs da VPC connection SG QuickSight-VPC, VPC ID imutável (network)
VPCE secretsmanager / redshift / glue S3 por gateway (security)

🤖 AWS: Amazon Quick (Enterprise)

Quick Sight, Agents, Flows, Automate Identity Center, RLS/CLS (ai)

🌐 Externo: saída pública que continua obrigatória

*.cloudfront.net, *.execute-api, *.kinesisvideo sign-in do Identity Center (external)

Flows

user -> dns: resolves the site name
dns -> resolver: forwards the private zone
user -> dx: HTTPS 443
dx -> vpce-web: console and dashboards
dx -> vpce-api: API calls
vpce-web -> quick: PrivateLink (ingress)
vpce-api -> quick: PrivateLink (ingress)
quick -> eni: assumes execution role, uses ENIs
eni -> dx: data and tool-call egress
dx -> db: SQL query
dx -> mcp: action connector / MCP
eni -> sm: credentials without leaving the VPC
user -> public: assets, metrics, sign-in via NAT/proxy

Ingress plane: interface endpoints, DNS and what the policy accepts

Ingress uses two interface endpoints with different jobs: com.amazonaws.<region>.quicksight-website for the site and com.amazonaws.<region>.quicksight for the API (with a -fips variant). If administrators sign in as IAM users, add console and signin too. Place each one in at least two AZs, with a security group that accepts only 443 from the corporate CIDR.

DNS is where the first thing breaks. The API endpoint, with private DNS on, takes over quicksight.<region>.amazonaws.com by itself. The website endpoint doesn't: its private name is not the public URL. You create a private hosted zone for <region>.quicksight.aws.amazon.com with an alias A record pointing at the endpoint. But a private hosted zone only answers associated VPCs, so corporate DNS has to forward that zone to a Route 53 Resolver inbound endpoint. Without the forward, the browser resolves the public IP, access still works, and nobody notices PrivateLink is being bypassed. Test with nslookup from inside the corporate network and expect an IP from the VPC CIDR.

The website endpoint policy is almost decorative. Principal, Action and Resource must be "*", and the condition only accepts aws:PrincipalAccount or aws:OrgId. It stops your network from becoming a doorway into another account's Quick, which helps against exfiltration, but it doesn't control what the user does. The API endpoint accepts action-level restriction, and that's where being specific pays off.

Egress plane: the VPC connection is an injected ENI, not PrivateLink

When Quick needs to fetch data, it doesn't go through the interface endpoint. To create a VPC connection (Enterprise only), you supply a VPC ID, at least two subnets in different AZs, at least one security group, and an execution role that trusts quicksight.amazonaws.com. Quick assumes that role and creates ENIs in your subnets with ec2:CreateNetworkInterface. From then on your network decides: route table, NACL, security group and the Direct Connect route.

Three details turn into incidents if you don't know them. The VPC ID can't change after creation: reorganizing the network means recreating the connection and repointing every data source. The PARTIALLY AVAILABLE status means one ENI is down and the connection still works. It's exactly the state nobody watches until the second AZ fails. Connections created before 27 Apr 2023 use a non-stateful security group and need an inbound rule that lets return traffic in on every port (0–65535) from the source's SG. If yours dates from then, recreate it.

For on-premises database host names, enter the Resolver inbound IPs in the DNS resolver endpoints field. Without them Quick resolves through public DNS and can't find the host. Credentials go to Secrets Manager through an interface endpoint in the same VPC as the ENIs, so that even the password lookup doesn't leave through NAT.

One limitation changes the design: the Quick Index knowledge base VPC connection currently supports only Amazon Redshift. Confluence, SharePoint, Box, Google Drive, OneDrive and S3 can't use it.

The ENI security group is now the agent's network allow-list: The same VPC connection serves Quick Sight and Quick Agents calling private MCP. In practice, the egress of the QuickSight-VPC SG and the routes on those subnets define everything an agent can reach on your network. If the SG opens the whole on-premises CIDR because "BI needed it", the agent inherits that reach. Split subnets and security groups by purpose (analytical data on one side, action tools on the other), even if it costs a second VPC connection.

Enforcing endpoint-only access: where the restriction stops

Quick's IP/VPC restriction (update-ip-restriction --enabled --vpc-endpoint-id-restriction-rule-map vpce-...=Name) blocks anything that doesn't come from the approved endpoint. It has four properties you need to know before turning it on in production.

It covers web, embedded and mobile, but not the public API. Anyone with credentials can call any operation from any IP. For the API the control is IAM: an aws:SourceVpce or aws:SourceIp condition in a deny policy, ideally in an SCP. Here's the catch: 13 APIs don't work through the interface endpoint (the 8 action connector ones, such as CreateActionConnector, and 5 Flows ones, such as ListFlows). A blanket aws:SourceVpce deny breaks the automation that manages connectors. You write the exception with NotAction for those operations and run that automation from a controlled path with public egress.

UpdateIpRestriction replaces the whole map. A script that sends only the new rule deletes the others. Run describe-ip-restriction first, or better, keep the map versioned and always apply the full state.

A rule takes up to 10 minutes to apply and is global, enforced in every Region and managed only from the account's Region. The limit is 100 rules combined, with IP and VPC rules OR'ed together.

Lockout is real. Without a rule covering your own access, the console won't let you save. But anyone on a source outside the list loses access the moment you turn it on. Publicly embedded dashboards then work only through the endpoint.

What stays public, and what keeping the rest private costs

The documentation lists the domains the browser reaches, and several have no endpoint: *.cloudfront.net (CSS and JS), *.execute-api.<region>.amazonaws.com (client metrics), *.kinesisvideo.amazonaws.com (automation streaming), <region>.signin.aws and, with IAM Identity Center, browser sign-in, which requires the public sign-in and Identity Center endpoints. No public IP is needed, but NAT or a proxy is. Your compliance report should say "the data and API path is private; static assets, telemetry and authentication leave through an inspected proxy". Writing "zero internet" claims something the documentation itself contradicts.

Now the recurring cost, in us-east-1. An interface endpoint costs US$ 0.01 per hour per ENI, plus US$ 0.01 per GB for the first 1 PB. Website and API in 2 AZs are 4 ENIs, about US$ 29/month. Add console and signin, another US$ 29. Secrets Manager, Redshift and Glue in 2 AZs, roughly US$ 44 more. The most expensive piece is the one nobody budgets: a Resolver inbound endpoint costs US$ 0.125 per ENI per hour, with at least two, so about US$ 182/month, plus US$ 0.40 per million queries. If you already have a shared Resolver inbound in a central network VPC, reuse it: it's the biggest line item and there's nothing Quick-specific about it.

Anti-patterns I find in this design

Website endpoint without on-premises DNS forwarding: PrivateLink exists, the browser resolves the public IP and access works from outside. It's a silent failure that only shows up in nslookup or VPC Flow Logs.
Treating IP/VPC restriction as an API control: it doesn't cover the public API. Without aws:SourceVpce in IAM or an SCP, a leaked credential calls everything from anywhere.
Blanket aws:SourceVpce deny with no exception: the 13 action connector and Flows APIs don't go through the endpoint, and the automation that manages connectors stops working without an obvious error.
A script that calls UpdateIpRestriction with only the new rule: the map is replaced whole, the other rules disappear and an entire branch office loses access within 10 minutes.
One VPC connection for BI and agents with the whole on-premises CIDR open: the agent inherits BI's network reach, and the blast radius of a prompt injection becomes the corporate network.
Keeping a VPC connection from before 27 Apr 2023: the non-stateful SG requires return traffic open on 0–65535, a rule no auditor accepts without questions.

Well-Architected lens

security: Layered defense with clear roles: website endpoint policy scoped by aws:OrgId, API endpoint policy by action, IP/VPC restriction for web and mobile, an SCP with aws:SourceVpce and NotAction for the 13 APIs outside the endpoint, RLS and CLS on the data. Restriction map changes land in CloudTrail; alarm on UpdateIpRestriction.
reliability: Two AZs on every endpoint and on the VPC connection, an alarm whenever status leaves AVAILABLE (including PARTIALLY AVAILABLE), and Direct Connect with a backup VPN, because the VPC connection has no other path to on-premises.

What I would do: I'd start with the report I'll hand the auditor, and only then create the first endpoint: which domains leave through the proxy, which controls cover the API and which cover the browser. Then I'd build two security groups and two VPC connections, one for analytical data and one for agent tools, because I've seen a "temporary" BI security group last four years. The restriction map would go into code from day one, applied as full state. The lesson I carry from financial platforms: the expensive control isn't the one you configure, it's the one you think is active and isn't, and in Quick that control is DNS forwarding.

Verdict

Use the full design (website and API endpoints, VPC connection, Resolver inbound, IP/VPC restriction and an SCP with aws:SourceVpce) when you have regulated on-premises data, agents calling private tools, and an auditor who will ask for the path of every byte. If Quick only reads Redshift and S3 in the same account and sign-in goes through Identity Center, a VPC connection with service endpoints covers most of the risk, and IP restriction on the corporate egress covers the browser for well under US$ 300/month. Either way, document what stays public. That sentence is what separates a defensible design from a promise the documentation contradicts.

Rating: Recomendado com ressalvas / Recommended

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

After login: per-operation authorization, revocation and recovery for agent tools

Fernando Azevedo — Wed, 21 Oct 2026 16:01:25 +0000

A valid login is only the starting point

An agent may continue a conversation after a user loses a permission. The operational question is whether the next tool call can still change a resource. This article proposes an architecture exercise, not a report of an AWS deployment or a completed cloud experiment. The AWS scenarios below are test proposals; the local recovery lab has a narrower purpose.

The starting points are Carlos Biagolini's articles on AgentCore Identity and a serverless MCP server with IAM authentication. They motivate a useful follow-up: once identity is established, which component enforces the right to perform each operation, and what happens if execution is interrupted?

Separate three decisions

Authentication establishes the caller's identity. Authorization decides whether that caller may perform a particular operation on a particular resource and tenant. Approval records consent to a concrete proposed change. A valid token, a visible tool or an old approval does not answer all three questions.

Imagine a remediation tool shared by two tenants. A caller may have permission to invoke the endpoint and still lack permission to change the selected resource. Similarly, approval of an earlier proposal should not authorize a modified proposal. These checks must remain explicit even when the model produces a convincing explanation.

A route is a boundary, not the whole policy

API Gateway HTTP APIs support IAM, JWT authorizers and Lambda authorizers. This exercise compares IAM-based invocation with application-defined Lambda authorization. With AWS_IAM, a signed request and invocation permission control entry. A Lambda authorizer applies application logic: format 1.0 requires an IAM policy; format 2.0 also supports a simple Boolean response when enabled.

If several MCP tools share POST /mcp, the tool name and arguments live in the body. Permission to invoke that route does not establish permission for every operation carried by it. In this design, the executor enforces fine-grained authorization. Another trusted layer could do so if it understands the operation, cannot be bypassed and must run before the effect. Separate routes help entry controls but do not remove resource and tenant checks.

Do not accept a model-supplied tenant as proof of authority. In a JWT flow, use validated claims and their authorized relationship to the resource. In an IAM/SigV4 flow, use the authenticated principal and a service-controlled authorization mapping. Carry that trusted context to the execution boundary. A Lambda role's ability to access DynamoDB does not automatically prove that every caller may read every item the function can access.

Revocation has an observable window

IAM documentation explains that temporary-credential permissions are evaluated for AWS requests, while policy updates may take minutes to propagate. Per-request evaluation therefore does not imply instantaneous revocation.

A cached Lambda-authorizer result introduces another window. Include the cache key, route separation and TTL in the test design. Denying future work does not reverse a completed effect. Sensitive operations also need an explicit policy for in-flight work and a check near the effect boundary.

Approval and completion are different facts

Bind approval to reviewer identity, proposal digest, scope, expiry and expected resource state. After a worker restart, a pending record does not prove whether an effect happened. Look for durable target evidence before replaying. If evidence is insufficient, preserve uncertainty and request review instead of guessing.

The public v0.4.0 lab explores that recovery boundary with SQLite and a synthetic target. Its reproduction guide requires Node.js 22.13+ and creates local files. No AWS account is needed. CLI approval is simulated; the lab does not implement authenticated approval, IAM or revocation. Atomic local target-and-receipt writes do not establish exactly-once effects across a remote API.

In the documented exercise, a deliberate crash occurs after the synthetic target and receipt are committed but before the execution ledger acknowledges completion. Recovery uses that receipt to reconcile the operation. Keeping the original proposal matters: generating a new proposal after the target changes produces a different digest. This is a concrete way to discuss recovery without implying a cloud guarantee the exercise cannot demonstrate.

Proposed test matrix

Valid principal requests another tenant: Deny before protected data access. Evidence: Synthetic identity and decision.
Shared MCP endpoint receives an unauthorized tool: Deny at the operation boundary. Evidence: Tool name, scope and decision.
Permission changes during a session: Measure when new calls stop succeeding. Evidence: Change time and decision timeline.
Authorizer result is cached: Test before and after TTL across routes and tenants. Evidence: Cache settings and observations.
Proposal changes after approval: Reject the changed digest. Evidence: Original and changed synthetic digests.
Resource changes before execution: Reject the stale precondition. Evidence: Expected and observed version.
Process stops after effect, before acknowledgement: Reconcile using evidence. Evidence: Ledger, target version and receipt.

For every scenario, distinguish expected from observed results. These AWS authorization tests are proposed, not completed. Use synthetic inputs and avoid storing tokens, secrets or customer data.

Trade-offs and the next experiment

Fresh authorization checks add work and latency. Caching can reduce both while extending the stale-decision window. Choose the design using operation impact, revocation policy and measurements. A low-risk read and a destructive remediation may justify different controls.

Before adding another agent, ask the team to demonstrate which identity drives the decision, which component enforces it and how interrupted work is recovered. A future isolated experiment can add authenticated approval and revocation, with a cost ceiling and cleanup procedure defined before deployment.

What would you test first: cross-tenant access, revocation during a session, or recovery after the effect? A useful answer includes the evidence needed to prove the outcome.

References

Authorization boundaries

Proposed design; identity and policy must reach the effect boundary.

Trusted application boundary

Authenticated caller (user)
Route entry control (edge)
Tool contract (compute)
Operation + resource + tenant (security)
Proposal approval (security)
Constrained executor (compute)
Durable evidence (storage)

Flows

caller -> entry
entry -> contract
contract -> policy
policy -> approval
approval -> executor
executor -> receipt

C4

A user requests remediation through an agent; the model proposes, the application enforces boundaries.

Containers

API / MCP endpoint (API Gateway + application): Authenticates the caller and controls entry.
Tool executor (Application service): Checks operation, resource, tenant and approval before the effect.
Evidence store (SQLite in local lab): Preserves synthetic proposal, state and receipt for recovery.

Components

Policy check: Uses trusted context independent of model arguments.
Recovery reconciler: Reconciles using evidence; preserves uncertainty without proof.

AWS Well-Architected

security: Deny out-of-scope operations before effects; do not trust model-supplied tenant identity.
reliability: Recover using receipts and preconditions without promising remote exactly-once effects.
operational-excellence: Record expected and observed results with versions and synthetic evidence.
cost: Run the local lab first; define a ceiling and cleanup before an AWS experiment.

Checklist before automation

Bind identity, digest, scope and expiry to approval.: Document the evidence, owner and known limitation.
Test the wrong tenant before data access.: Document the evidence, owner and known limitation.
Measure propagation and caching during revocation.: Document the evidence, owner and known limitation.
Simulate a crash after the effect and inspect the receipt.: Document the evidence, owner and known limitation.

Anti-patterns

Treating authentication or old approval as universal authorization.
Repeating an effect merely because the ledger still says pending.
Promising instant revocation or remote exactly-once execution without measurement.

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

Retro: EMR Serverless on IPv6-only and the subnet left without DNS64

Fernando Azevedo — Tue, 20 Oct 2026 15:42:40 +0000

The dangerous question about IPv6 on EMR Serverless is not "do my jobs run on IPv6?", it is "what does each subnet in the application know how to do when a job asks for an address that only exists in IPv4?". The 10/16/2026 announcement is short: Spark and Hive can now run on dual-stack or IPv6-only subnets, at no additional cost, in every Region and release version of the service. The documentation is more honest than the announcement: the addressing mode is derived from the subnets, and getting IPv6 traffic to your data is your responsibility. This piece is a pre-emptive retro, an incident that did not happen, built from the documentation and the failure modes it lists itself. I would rather write the postmortem before than after.

The scenario: /22 subnets that no longer fit three applications

The starting point is what AWS itself uses as a planning example. Each EMR Serverless worker consumes an IP address in the subnet where it launches, through an ENI. An application with 4 vCPU workers and a 4,000 vCPU ceiling reaches 1,000 workers, meaning 1,000 ENIs. A /22 subnet has 1,024 addresses and 1,019 usable ones, because AWS reserves five. With three applications sharing the same subnets and a 12,000 vCPU account quota, the documentation's recommendation becomes 3,000 free addresses per subnet, in at least two Availability Zones.

In a bank's data platform this usually hits a wall: the VPC IPv4 block was negotiated with corporate networking years ago, does not overlap with on-premises and does not grow without an approval round. That is exactly the case IPv6 solves. A VPC IPv6 subnet is a /64, and the address conversation is over.

In the scenario, the team goes straight to IPv6-only on three subnets, one per AZ, created by an IaC module. Two were built by hand during the pilot and got DNS64. The third came from the module, which did not know DNS64 existed. No integration test caught it, because the test job ran twice and landed on the two good subnets both times.

The numbers that drive the decision

1.000: ENIs per application. 4 vCPU workers with a 4,000 vCPU ceiling: one IP per worker
1.019: Usable IPs in a /22. Barely fits one application; three need 3,000 per subnet
US$ 0,045/GB: NAT gateway processing. Pricing page example (US East Ohio): every byte through NAT64 pays

Timeline of the simulated incident

D-7: subnet swap: The team stops the application, runs update-application with the three IPv6-only subnets in --network-configuration and starts it again. The mode is evaluated at every start, so the change only applies after the restart. Two test jobs pass.
02:00: the nightly batch starts: About a third of the jobs fail with UnknownHostException when calling endpoints that only publish an A record. Each job runs in a single subnet picked from the set, and the ones that landed on the subnet without DNS64 cannot resolve anything that is not IPv6.
02:40: retries hide the pattern: The orchestrator's retry policy resubmits the jobs. Some pass because they land on another subnet. The failure looks intermittent, and intermittent becomes "flaky network" in the on-call's first hypothesis.
04:10: correlation by subnet: Someone opens each job run detail, which shows the subnet used, and crosses it with success and failure. 100% of the failures sit on the same subnet. modify-subnet-attribute --enable-dns64 fixes name resolution.
05:00: the second failure: An ingestion job with an old JDBC driver fails with UnresolvedAddressException. It carried -Djava.net.preferIPv4Stack=true in spark.driver.extraJavaOptions, a workaround from years ago. The documentation forbids forcing the IPv4 stack in any mode.
D+5: the bill arrives: Cost Explorer shows a jump in NAT gateway data processing. S3 reads, which used to leave through the gateway endpoint, now resolve through DNS64 to 64:ff9b::/96 and cross NAT64.

Root cause: EMR Serverless derives the addressing mode from the subnets and does not validate whether the network is ready for it. Each subnet's readiness (IPv6 CIDR, auto-assign, DNS64, a 64:ff9b::/96 route to the NAT gateway, a ::/0 route to the egress-only gateway) was not encoded in the IaC module, and one-subnet-per-job selection turned a deterministic defect into an apparently random failure.

Why the failure looked random: the addressing mode rules

It is worth understanding the rules before blaming the network. An IPv4-only subnet runs as it always did. A dual-stack subnet runs IPv4 until you turn on spark.emr-serverless.networking.ipv6.enabled=true in the application's spark-defaults (for Hive, hive.emr-serverless.networking.ipv6.enabled in hive-site). An IPv6-only subnet runs IPv6 with no property at all.

When the set mixes types, the whole application runs in the one mode every subnet supports. IPv4-only with IPv6-only is rejected at validation. IPv6-only with dual-stack runs everything IPv6-only, including the dual-stack subnets, which must then meet the same prerequisites. IPv4-only with dual-stack runs everything on IPv4.

Three details produce incidents:

The mode applies at start: a running application keeps the mode it started with. You fix the subnet and nothing changes until the next stop/start.
One job, one subnet: a job runs in a single subnet picked from the list. One bad subnet out of three fails a third of the time, and retries mask it.
Dual-stack does not prefer IPv6 by default: even with the opt-in on, the engine uses IPv4 until you set spark.emr-serverless.driverEnv.SPARK_PREFER_IPV6 and spark.executorEnv.SPARK_PREFER_IPV6. Setting those on an IPv4-only subnet kills the job.

The lesson: the addressing mode is derived state, not configured state. The API does not let you declare the mode; it lets you declare subnets and infers the rest. Derived state needs a test that proves the derivation on every subnet, not on average.

Where each byte of a job on an IPv6-only subnet leaves

The path depends on the destination's DNS record: AAAA leaves directly, A becomes 64:ff9b::/96 and pays NAT64.

🟧 AWS: subnet IPv6-only (uma por AZ)

Spark driver SPARK_PREFER_IPV6 automático (compute)
Executors 1 ENI IPv6 por worker (compute)

📡 VPC: resolução e rotas

Route 53 Resolver DNS64 na subnet (network)
NAT gateway rota 64:ff9b::/96 (NAT64) (network)
Egress-only IGW rota ::/0 (network)
Interface endpoints Glue, STS, KMS, Logs (security)

🌐 Destinos

S3 dual-stack s3.dualstack. (storage)
Endpoint só IPv4 JDBC legado, API pública (external)
Endpoint IPv6 AAAA publicado (external)

Flows

drv -> r53: DNS query
exe -> r53: DNS query
r53 -> nat: A record only: synthesizes 64:ff9b::
nat -> v4: translates to IPv4, billed per GB
exe -> eigw: destination with AAAA
eigw -> s3ds: data reads and writes
eigw -> v6: egress without ingress
drv -> vpce: catalog, credentials, keys

Network remediation: turning prerequisites into invariants

The DNS64 fix took one command. The remediation that prevents recurrence took more work, and it is the one that matters.

Subnet module with a contract: the IaC module now requires, for every subnet tagged as an EMR Serverless target, assign_ipv6_address_on_creation = true, enable_dns64 = true, a 64:ff9b::/96 route to the NAT gateway and a ::/0 route to the egress-only internet gateway. Auto-assign only applies to resources created after the change, so turning it on late does not fix an ENI that already exists.

Validation before apply: a pipeline check (policy as code, or a test that reads describe-subnets and describe-route-tables) fails the deploy if any subnet in the set breaks the contract. Validation that runs after the incident is a postmortem, not a control.

Per-subnet canary: since the job picks the subnet and you do not, I create a canary application with a single subnet, one per AZ. A minimal Spark job resolves and connects to the endpoints the batch depends on (catalog, STS, KMS, the data bucket, the source database) and fails loudly. It runs before the batch window and after any network change.

Clean JVM: a sweep across every sparkSubmitParameters and runtimeConfiguration looking for preferIPv4Stack. A 2019 workaround cannot decide the network stack in 2026.

None of this is sophisticated. Its cost is not writing it, it is keeping the contract when someone creates the fourth subnet eighteen months from now without reading this document.

The cost the announcement does not mention: NAT64 and S3

"No additional cost" is true for the feature. It is not true for the path the traffic now takes.

On an IPv6-only subnet, every destination that only publishes an A record gets a synthesized address in 64:ff9b::/96, and that route points to the NAT gateway. The NAT gateway bills per hour and per GB processed; the VPC pricing page uses US$ 0.045 per GB as its example in US East (Ohio). A platform reading 8 TB a night from S3 and sending it through NAT64 pays around US$ 360 a night in processing alone, roughly US$ 10,800 over 30 nights. The S3 gateway endpoint on the workload subnet does not catch this traffic, because the destination is no longer an IPv4 prefix.

S3 only serves IPv6 through its dual-stack endpoints (s3.dualstack.<region>.amazonaws.com), and the client has to ask for them explicitly. Do not trust it before measuring: compare the NAT gateway's BytesOutToDestination before and after the switch.

Other services need the same care. In the AWS IPv6 support table, Glue has dual-stack and a private endpoint with IPv6, but not the public endpoint; Lake Formation also only has IPv6 through its private endpoint. On IPv6-only, that means interface endpoints for catalog and permissions, or paid NAT64 on every call.

And security: the S3 documentation warns that policies with IPv4-only aws:SourceIp make clients incorrectly lose or gain access once they start using IPv6. Review every bucket policy before the switch, not after.

The three modes side by side

Criterion	IPv4-only	Dual-stack with opt-in	IPv6-only
IPv4 consumption	1 per worker	1 per worker, still consumed	Zero
Activation	-	`ipv6.enabled` on the application + `SPARK_PREFER_IPV6` on the job	Automatic from the subnets
IPv4-only destinations	Direct	Direct over IPv4	Only with DNS64 + NAT64, billed per GB
Incident risk	Exhausting the subnet at peak	Low: failure falls back to IPv4	High if any subnet or dependency is not ready
When to use	Address headroom and no IPv6 mandate	Transition and dependency validation	Compliance mandate or real exhaustion, with dependencies mapped

Well-Architected reading

security: An IPv6 address is globally unique: egress only through an egress-only gateway, never a ::/0 route to the internet gateway on worker subnets. EMR Serverless already refuses security groups with ports open to 0.0.0.0/0 or ::/0. Review aws:SourceIp in bucket policies and prefer aws:SourceVpce.
reliability: Spread subnets across at least two AZs, as the documentation recommends, but treat each one as its own failure unit: a canary per subnet and an alarm per subnet. Automatic retries without per-subnet correlation turn a deterministic defect into noise.

Anti-patterns this retro tries to prevent

Validating with two jobs: with three subnets and one picked per job, two green jobs prove nothing about the third.
Going straight to IPv6-only without a dependency inventory: every driver, SDK and endpoint the job touches must support IPv6 or go through NAT64; EMR Serverless does not change your code or third-party endpoints.
Turning on SPARK_PREFER_IPV6 as a global default: on an IPv4-only subnet the preference kills the job.
Fixing the subnet without restarting the application: the mode was fixed at start, and on-call concludes the fix did not work.

What I would do: I would start with dual-stack with the opt-in, on a new application, and keep the IPv6 preference on only for jobs whose dependencies I have already measured. IPv6-only waits for a mandate or real exhaustion, and only after two weeks of green per-subnet canaries and a NAT gateway bytes chart I can explain line by line. The lesson I took from other network migrations is that the defect is almost never in the new service, it is in the subnet someone created with the old module. That is why the contract lives in IaC, not in the wiki.

Lessons and recommendation

Adopt IPv6 on EMR Serverless when the IPv4 space in shared subnets already limits your vCPU ceiling or when there is a formal IPv6 requirement. Use dual-stack with the opt-in while there is any IPv4-only dependency that matters in volume; use IPv6-only when every dependency publishes AAAA or sits behind an interface endpoint, and NAT64 is limited to control calls. Either way, encode DNS64, routes and auto-assign in the subnet module, run a per-subnet canary and measure the NAT gateway before declaring the migration done. The feature is free; an unprepared network bills per GB.

Rating: Adote com dual-stack primeiro / Adopt wi

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

CloudFront dual-stack VPC origins: IPv6 to private origins, with caveats

Fernando Azevedo — Fri, 16 Oct 2026 17:37:34 +0000

For almost two years, anyone who adopted CloudFront VPC origins and had an IPv6 target lived with a bad choice: keep the ALB in a private subnet and stay on IPv4 between edge and origin, or expose the origin to the internet to get end-to-end IPv6. The 10/15/2026 announcement removes that dilemma. The question that matters now is not 'can I get IPv6 to a private origin?', it's 'what in my VPC, my security group and my change process breaks when I turn it on?'. That is the question this analysis answers.

The launch in numbers

US$ 0: Additional cost. Dual-stack VPC origins carry no extra charge; the ENI addresses can be private.
34: Commercial Regions. Every Region with VPC origins, including sa-east-1; four have one excluded AZ.
55/60: Prefix list weight. The CloudFront managed list weighs 55 against the default 60 rules per security group.

What actually changed

VPC origins, launched in November 2024, places an ALB, NLB or EC2 instance in a private subnet and makes CloudFront the single point of entry. The connection arrives through a service-managed ENI created inside your subnet, and access is controlled by a security group AWS creates with the naming pattern CloudFront-VPCOrigins-Service-SG. Until yesterday that ENI spoke IPv4 only.

In September 2025 CloudFront gained IPv6 for custom origins, with three options: IPv4 only (default), IPv6 only and dual-stack. That announcement excluded S3 and VPC origins. What shipped now closes half of that gap: VPC origins accept dual-stack. The other half stays open, because the current CloudFront IPv6 documentation still lists IPv6 only as "excluding Amazon S3 and VPC origins".

In practice, the change is one field: when you create the VPC origin, the console asks for Origin type, and the default is still IPv4. For dual-stack to work, the private subnet needs at least one free IPv6 address in addition to the IPv4 one, because that is where the service ENI is born. Without it, ENI creation fails.

What does not change: gRPC is still out, Lambda@Edge origin request and origin response triggers are still out, a Gateway Load Balancer cannot be an origin, and neither can an NLB with a TLS listener. If you ruled out VPC origins for one of those reasons, you have no new reason to revisit.

End-to-end IPv6 path with a private origin

Every hop where IPv6 can silently fall back to IPv4, and the control that governs that hop.

🌐 DNS e borda

Route 53 alias A + alias AAAA (network)
CloudFront edge IPv6 de viewer ligado (edge)
Origin-facing escolhe IPv4 ou IPv6 (edge)

🟧 AWS: VPC privada (sa-east-1)

ENI do serviço 1 IPv4 + 1 IPv6 livres (network)
ALB interno IpAddressType: dualstack (compute)
Target group IPv4 ou IPv6 (compute)

🔐 Controles

CloudFront-VPCOrigins-Service-SG fonte da regra de entrada (security)
NACL de saída efêmeras 1024–65535 IPv4 e IPv6 (security)

Flows

viewer -> r53: AAAA lookup
viewer -> edge: HTTPS over IPv6
edge -> ofs: cache miss
ofs -> eni: private connection, family picked by the service
eni -> alb: allowed by service SG
alb -> tg: new connection, target group family
svcsg -> alb: inbound rule
nacl -> alb: return path

Where it shines

The trade-off is gone: IPv6 to the origin no longer requires a public subnet, an internet-facing ALB and an IP list to maintain.
Zero license cost: no additional charge, and the ENI IPv6 addresses do not need to be public.
Same security model: the service SG still restricts the origin to your distributions, now across both families.
Brazil included: sa-east-1 is on the list, so a regulated workload that must stay in-country is not left out.

Dual-stack is not guaranteed IPv6

This is the point the announcement does not make explicit, and it matters to whoever has to answer the audit. The CloudFront documentation describes dual-stack mode this way: the service automatically picks IPv4 or IPv6 for the origin connection to prioritize performance and availability. In other words, you don't decide the family of each connection. CloudFront does.

For most sites that is exactly the right behavior. For a team that has to prove to a regulator or a contract that traffic runs over IPv6 all the way to the origin, it is an evidence problem. Turning on dual-stack produces no proof at all; measurement does.

The same holds on the viewer side. The documentation itself warns that CloudFront may answer a viewer over IPv4 when its data says the experience will be better, and that the IPv6 share should remain a minority because not every access network supports the protocol. Two automatic decisions, one at each end, and neither is yours.

What I would do: treat "end-to-end IPv6" as a measured SLO, not a setting. At the edge, the c-ip field in standard logs shows the viewer family. At the origin, ALB access logs show the address that opened the connection, which for VPC origins is the service ENI. The daily ratio of IPv6 connections to total is the number that goes into the report. If the requirement is IPv6-only, VPC origins still doesn't meet it: IPv6 only remains excluded.

Network prerequisites that fail silently

The CloudFront side is one field. The real work is in the VPC, and much of it fails without a clear message.

IPv6 in the VPC and subnets: the ALB only accepts IpAddressType: dualstack if the VPC and its subnets have an associated IPv6 CIDR block, the route tables route IPv6, and the SG and NACL allow IPv6. In an older account built IPv4-only, that is a project, not a click.

An address for the ENI: the subnet where the VPC origin lives needs a free IPv6 address. A /28 that runs out of IPv4 is already a known problem; with dual-stack the same math repeats in the other family.

Origin security group: the documentation gives two options. Managed prefix lists, and for dual-stack that means both, com.amazonaws.global.cloudfront.origin-facing and com.amazonaws.global.ipv6.cloudfront.origin-facing. The VPC documentation assigns the CloudFront list a weight of 55, against the default of 60 rules per security group, which leaves 5 rules for everything else. The other option is to reference CloudFront-VPCOrigins-Service-SG: it takes one rule and restricts the origin to your distributions, not to any CloudFront distribution. That is the one I use.

Outbound NACL: inbound NACL rules are not evaluated for traffic from CloudFront, but outbound rules are, on the return path, on ephemeral ports 1024–65535. A NACL keeps IPv6 rules separate from IPv4 ones. If you only allowed 0.0.0.0/0, you will watch the IPv6 handshake die on the way back.

Limits that still apply: IPv6 only is still unavailable for VPC origins. gRPC and Lambda@Edge origin triggers are not supported. A Gateway Load Balancer and an NLB with a TLS listener cannot be origins, and an NLB must have a security group attached. And IPv6 ends at the ALB: the connection to targets uses the target group's family, so a dual-stack ALB with an IPv4 target group is IPv6 to the ALB, not to the application.

Migrating a VPC origin already in production

Here is the cost the announcement does not show. Per the documented procedure, updating a VPC origin requires first making sure it is not the default origin of any cache behavior, then disassociating it from every distribution that uses it, editing, waiting for the status to return to Deployed (up to 15 minutes), and only then associating it again. In production, that is a scheduled outage window, multiplied by the number of distributions sharing the origin.

I would not edit the existing VPC origin. The safer path is to create a second, dual-stack VPC origin pointing at the same ALB ARN, and swap origins through CloudFront continuous deployment: a staging distribution gets the new origin, you test with a slice of traffic, and you promote the configuration when the signals are clean. Rollback is promoting the previous configuration back, without waiting 15 minutes for a VPC origin to propagate.

The signals I would watch on staging: the distribution's 5xxErrorRate and OriginLatency in CloudWatch, the ALB's TargetConnectionErrorCount and HTTPCode_ELB_5XX_Count, and the share of IPv6 connections in the ALB access logs. If that share stays at zero, the cause is almost always an SG or NACL missing the IPv6 rule, and CloudFront falls back to IPv4 without complaining, which is good for availability and terrible for the target.

If you share VPC origins across accounts through AWS RAM, add coordination with the owner of every consuming distribution.

How to adopt without an outage window

Prepare the VPC: Associate an IPv6 CIDR with the VPC and private subnets, adjust route tables and confirm each subnet has a free IPv6 address for the ENI.
Move the ALB to dual-stack: Run aws elbv2 set-ip-address-type --ip-address-type dualstack on the internal ALB; decide separately whether the target group also moves to IPv6.
Lock down SG and NACL: ALB inbound from CloudFront-VPCOrigins-Service-SG; NACL outbound on ephemeral 1024–65535 for both IPv4 and IPv6.
Create a new VPC origin: Origin type dual-stack, same ALB ARN. Don't edit the existing one; wait for Deployed status.
Test on staging and promote: Staging distribution with the new origin, partial traffic, 5xx signals and IPv6 share; promote only when both are clean.
Finish the edge: Enable viewer IPv6 and add a second AAAA alias in Route 53 for each alternate domain name; review signed URLs using a custom IpAddress policy.

Anti-patterns

Editing the VPC origin in production: the procedure requires disassociating from every distribution, and the wait of up to 15 minutes becomes an outage.
Claiming compliance because the field says dual-stack: CloudFront picks the family per connection; without log measurement, there is no evidence.
Allowing the prefix lists when the origin only serves your own CloudFront: it consumes 55 of the default 60 rules and accepts any distribution, not just yours.
Forgetting the target group: a dual-stack ALB with IPv4 targets delivers IPv6 to the load balancer, not to the application.

Curator's note: In a financial environment I would turn this on first for a low-risk distribution, with staging and an IPv6 share target written down before the change. I have seen an IP family migration pass functional testing and fail the first audit report, because the fallback to IPv4 is silent by design. The lesson: in networking, whatever degrades without an error needs its own metric, otherwise you find out from the wrong person. And I would not edit an existing VPC origin under any circumstances; creating another one and swapping via continuous deployment costs an afternoon and saves a 2 AM window.

Verdict

Adopt it when you have an end-to-end IPv6 requirement, contractual, regulatory or platform-driven, and the origin already is or needs to be in a private subnet: there is no additional cost and the security model does not get worse. Wait if the requirement is IPv6-only, because IPv6 only is still out for VPC origins, or if the VPC has no IPv6 CIDR yet, because then the project is a network project, not a CDN one. Either way, create a new VPC origin and swap via staging; do not edit the one serving traffic.

Rating: 4/5

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

Well-Architected Agent: architecture review becomes a weekly process

Fernando Azevedo — Tue, 13 Oct 2026 18:24:40 +0000

For years, the Well-Architected review was an event: a week-long workshop, a spreadsheet of high and medium risks, and then six months with nobody opening the report. On 10/01/2026 AWS put AWS Well-Architected Agent into preview, which replaces the event with a cycle: scheduled scans, up to 30 recommendations per run, weekly refresh. The question that interests me is not "is the agent right?", it's "who triages 30 findings a week, and what happens to what it cannot see?". After 16 years operating financial platforms on AWS, I've learned that a findings tool without a triage owner becomes just another red dashboard everyone learns to ignore.

The numbers that shape the design

100: accounts per profile. scanning every commercial Region from us-east-1, us-east-2 or us-west-2
30: recommendations per run. with a 7-day cooldown between scheduled runs of the same profile
5/dia: IaC reviews per profile. zip up to 25 MB; S3 folder up to 100 MB, file up to 1 MB

The signal: an aggregator with an opinion

Strip the AI layer and what AWS shipped is an aggregator with an opinion. The related-services documentation is explicit: the agent ingests Trusted Advisor findings as baseline signals, uses Compute Optimizer rightsizing signals for EC2, Lambda and RDS, consumes Security Hub CSPM, Resilience Hub assessment data (with its RTO/RPO) and Cost Optimization Hub. On top of that it adds three things: prioritization against goals you write, cross-pillar trade-off analysis, and ready remediation (console, updated IaC, aws CLI commands and Systems Manager runbooks).

Recommendations come at three levels. Resource: one instance, one function, one database, with dollar impact where applicable. Application: consolidated findings across related resources, a format the documentation itself marks as beta. Architecture: generated from CloudFormation, Terraform or CDK templates sent as a zip or S3 URI, before deployment.

Two operational details change result quality. Without Cost Explorer enabled, the agent estimates cost from public pricing, which ignores Savings Plans, Reserved Instances and contractual discounts; in an account with an EDP, the "dollar impact" can be inflated enough to invert prioritization. And access requires a Business+ Support plan or higher: Business+ gets 2 profiles and 7 applications per profile; Enterprise On-Ramp, Enterprise and Unified Operations get 10 and 30. Developer and Business are excluded.

What changes for architects

Goals became configuration: up to 10 goals per profile, plain text, one pillar each, and they order the queue. A badly written goal mis-prioritizes without raising any error.
Continuous review has fixed throughput: 30 recommendations per run and a 7-day cooldown mean a triage ceiling per profile. The bottleneck becomes the team, not the tool.
Shift-left with a quota: 5 IaC reviews per day per profile cannot back a gate on every pull request. They fit on merge to the release branch.
Criticality is declared, not inferred: application context accepts MISSION_CRITICAL, BUSINESS_CRITICAL, NON_CRITICAL and TEST_DEVELOPMENT. Whoever skips it gets recommendations tuned for the average case.
Silence is not compliance: the unsupported type list includes AWS::WAFv2::WebACL, AWS::Organizations::Policy and AWS::IAM::RolePolicy. No finding there proves nothing.

The blind spot that matters in a bank

The quotas page carries a long table of CloudFormation resource types the agent does not analyze. Most are peripheral (IoT Analytics, DeviceFarm, Evidently). But a few are exactly the controls an auditor checks first in an environment under BACEN 4.893 or PCI-DSS:

AWS::WAFv2::WebACL, WebACLAssociation, IPSet and RuleGroup: the entire WAF edge.
AWS::Organizations::Policy: SCPs, that is, the organization's data perimeter.
AWS::IAM::RolePolicy, UserPolicy and GroupPolicy: inline policies, where the "Action": "*" nobody remembers writing usually lives.
AWS::OpenSearchService::Domain, AWS::Shield::Protection and Security Hub's own configuration.

The failure mode here is silent. The report arrives clean on the security pillar, someone takes the screenshot to the risk committee, and the absence of WAF findings is read as a well-configured WAF. It isn't: it's an unanalyzed WAF.

The mitigation is cheap and has to be explicit. Keep those controls covered by AWS Config conformance packs and Security Hub CSPM directly, outside the agent, and write in the governance document which types the agent does not cover and which tool covers each one. The list is marked preview and should shrink, so review it every quarter instead of assuming it is stable.

Where the agent reads, where it writes

Role chain in a dedicated account, input signals and remediation paths. Only the IaC path goes back through the pipeline.

🔐 AWS: Conta de tooling dedicada

Agent profile us-east-1 · 10 metas (ai)
Execution role trust wellarchitected.amazonaws.com (security)
CloudTrail API wellarchitected:* (security)

🟧 AWS: Contas de workload (até 100)

Access role WellArchitectedAgentResourceScanning (security)
Recursos todas as regiões comerciais (compute)

📡 AWS: Sinais de entrada

Trusted Advisor Security Hub CSPM (data)
Compute Optimizer Cost Optimization Hub (data)
Cost Explorer sem ele: preço público (data)

📤 Saída: Remediação

IaC atualizado PR no repositório (ci)
AWS CLI / SSM runbook só em sandbox (external)

Flows

profile -> exec: assumes
exec -> access: sts:AssumeRole, ExternalId = profile ARN
access -> res: config read
ta -> profile: baseline findings
co -> profile: rightsizing and cost
ce -> profile: actual cost per resource
profile -> ct: audit
profile -> owner: 30 recommendations/week
owner -> iac: accepts and versions
owner -> cli: tests outside prod

IAM: the design is good, the default less so

The access model deserves credit. Instead of a service-linked role, the agent uses customer-managed roles in a chain: an execution role in the profile account, trusting wellarchitected.amazonaws.com and holding only sts:AssumeRole, and an access role in each workload account, with the read-only WellArchitectedAgentResourceScanning managed policy. The profile ARN is used as the external ID, which closes the confused deputy, and there is an iam:PassRole check at creation. The documentation states neither role allows modifying resources. Onboarding is opt-in per account.

That said, three adjustments I'd make before turning it on in production:

Dedicated account: AWS itself recommends hosting the profile in an account with no workloads. In a Control Tower environment, that is the tooling or audit account, never the management account.

Trust by exact ARN: use the execution role ARN, with the service-role/ path the console creates, not the account root. Roll out access roles via StackSets with the same name in every account (the common-name pattern the profile accepts) so that revoking is one operation, not a hundred.

Processing residency: the profile can only live in us-east-1, us-east-2 or us-west-2, even though it scans sa-east-1. Configuration metadata and tags leave the Region for analysis. If your tagging policy carries customer names or CPF numbers in tags (I've seen it), that becomes a conversation with the DPO under LGPD before it becomes one with the auditor.

Four remediation paths, four maintenance costs

Criterion	Console	Updated IaC	AWS CLI	SSM runbook
Creates drift	Yes	No	Yes	Yes, unless the runbook updates the code
Review trail	CloudTrail only	Pull request with approval	CloudTrail only	SSM execution history
Where I use it	Understanding the finding	Production	Sandbox	Repeatable action with no IaC state, e.g. orphaned snapshot

How I'd position it today

The agent doesn't replace a human Well-Architected review, it changes what the human review has to cover. The mechanical work (oversized instance, bucket without versioning, single-AZ RDS tagged MISSION_CRITICAL) leaves the workshop agenda. What remains for people is what the agent admittedly cannot see: SCPs, WAF, inline policies, and architecture decisions that show up in no configuration at all, like why a settlement flow is synchronous.

In practice, three design decisions:

One profile per regulatory domain, not per team: with 10 profiles on Enterprise, separate what has distinct SLOs and regimes (payments, data, internal platform). On Business+, with 2 profiles and 7 applications, the choice is harder: regulated production in one, everything else in the other.

Goals versioned in a repository: plain-text goals are a ranking parameter. Treat them as code: a file in the platform repository, applied through the wellarchitected namespace API, with review. A goal like "reduce cost" is weak; "reduce non-production compute cost by 20% without touching the core's 15-minute RTO" gives the agent a trade-off to respect.

A queue with an owner and an SLA: 30 recommendations a week is about 120 a month per profile. Without an owner who marks items complete or suppresses them with a reason, the backlog grows linearly and the tool loses credibility in two months. I'd turn on cost and resilience first, where the cost of error is low, and security only later, with the unsupported list beside it.

Anti-patterns you can already predict

Running the CLI command in production: it fixes the finding and creates drift; the next terraform apply reverts the fix and nobody notices until the next scan.
Reading a clean security pillar as attestation: with WAF, SCPs and inline policies out of scope, a clean report is partial coverage, not compliance.
Prioritizing by dollars without Cost Explorer: the agent falls back to public pricing and ignores Savings Plans and EDP; the cost queue ends up sorted by a number that isn't yours.
Profile in the management account: it concentrates the role that reaches 100 accounts in the place with the largest blast radius in the organization.
IaC gate on every PR: 5 reviews per day per profile run out before lunch; the gate either fails open or blocks the team.

The goal is the new prompt: The goals text defines the order of the remediation queue across 100 accounts. That is production configuration with effects on money and risk, written in natural language with no validation. Whoever edits goals needs the same change control as whoever edits SCPs.

Curator's note: I'd turn the agent on this week in a dedicated tooling account, cost and resilience only, with Cost Explorer enabled before the first run. Access roles go out through StackSets, trust pinned to the exact execution role ARN, and goals live in a versioned file. The lesson I carry from years of Trusted Advisor and Security Hub: the cost of a findings tool is not the license, it's the weekly triage someone has to sustain for years. If there is no named owner for the queue, I'd rather not turn it on.

Verdict

Adopt now if you already have a Business+ plan or higher, Control Tower with a tooling account, and a platform team able to triage about 120 items a month per profile: the agent removes the mechanical work from reviews and delivers IaC remediation that fits your pipeline. Wait if your main need is security evidence for regulatory audit: while WAF, SCPs and inline policies are out of scope, it complements AWS Config and Security Hub CSPM, it does not replace them. Either way, production remediation goes only through the IaC path.

Rating: Adotar com escopo / Adopt with scope

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

Sharing a GPU cluster on HyperPod: three designs and when each one pays off

Fernando Azevedo — Sat, 10 Oct 2026 16:19:29 +0000

The question that usually opens the shared GPU cluster discussion is "how do I split the machines across teams?". The question that decides whether the design survives is a different one: when the research team's training job gets evicted at 2 AM to make room for an inference endpoint, who finds out, how many GPU-hours were thrown away, and which cost center absorbs them? The reference architecture AWS published on 10/08/2026 for SageMaker HyperPod on EKS answers a good part of that. Here I put it side by side with the two alternatives I see most often and say under which conditions each one is worth its maintenance cost.

The real choice: three ways to split GPUs

Option A, one shared HyperPod EKS cluster with Task Governance: the design in the post. Each team gets a hyperpod-ns-<team> namespace, its own SageMaker domain for Studio users, and a permission set in IAM Identity Center for CLI users. Fairness across teams belongs to Task Governance, built on Kueue: guaranteed quota per team, lending of idle capacity, and priority classes.

Option B, one cluster per team, ideally in separate accounts: the boundary is the AWS account. There is no noisy neighbor because there is no neighbor. In exchange, each team carries its own capacity floor, and GPUs idle in one cluster cannot rescue a full queue in another.

Option C, a shared cluster with namespaces and ResourceQuota only: this is what shows up when the platform starts as a single-team cluster and other teams trickle in. It works until the first week of contention, when the quota becomes a hard ceiling and arrival order decides who gets to train.

The difference between the three is not the cost of standing up the cluster. It is what each one demands in continuous operation: reviewing quotas, debugging Kueue admission, maintaining the UID/GID webhook, reconciling chargeback. That is the cost that stays for years, and it is the one I compare on.

From login to GPU: where each boundary in design A is enforced

Identity decides the namespace, the namespace decides the queue, the queue decides whether the pod runs on its own or borrowed quota, and the namespace becomes the chargeback key.

🔐 IAM Identity Center: identidade

Permission set AWSReservedSSO_TeamA_* (security)
SageMaker domain TeamA-domain + TeamA-role (security)

🟧 AWS: HyperPod EKS control plane

EKS access entries escopo: hyperpod-ns-team-a (security)
Admission webhook UID/GID via sessionName (compute)
DynamoDB mapa POSIX (data)

🤖 Task Governance: Kueue

LocalQueue hyperpod-ns-team-a-localqueue (messaging)
ClusterQueue cota garantida + borrow limit (messaging)
Idle resource sharing IdleResourceSharing: Enabled (messaging)

🟧 AWS: pool de GPU compartilhado

Nós HyperPod health checks + auto-recovery (compute)
FSx for Lustre /fsx/TeamA (storage)

📤 Saída: custo e visibilidade

Kubecost custo por namespace (data)

Flows

idp -> ps: federates groups
user -> ps: aws sso login
user -> dom: Studio
ps -> ae: kubectl
dom -> ae: execution role
ae -> wh: pod created
wh -> ddb: resolves UID/GID
wh -> lq: queue-name label
lq -> cq: admission
cq -> idle: over quota: borrows
cq -> gpu: admits pod
gpu -> fsx: checkpoint
gpu -> cost: usage per namespace

Isolation: a namespace is an organizational boundary, not a security one

The post itself states it plainly: a namespace is an isolation boundary, not a hard security boundary, and the design targets several teams within one company, not tenants who do not trust each other. I agree, and I add what that means in a regulated environment.

Design A stacks four controls: EKS access entries scoped per namespace (one for the domain role, another for the AWSReservedSSO_<permission-set>_<id> role), RBAC that returns Forbidden on cross-namespace calls, a default-deny NetworkPolicy per namespace, and S3 restricted to the team's prefix. All four run on the same kernel, the same NVIDIA drivers and the same control plane.

Silent failure number one: a NetworkPolicy without a CNI that enforces it is just YAML. If the Amazon VPC CNI does not have network policy enabled, kubectl apply succeeds, the object exists, and not a single packet is blocked. Test with a team B pod trying to open a connection to a team A service before you call the boundary done.

My rule is this. If the teams share the same regulatory regime and the same data classification, a namespace is enough. If one team trains on customer data under LGPD and another on synthetic data, or if a BACEN 4.893 auditor is going to ask for demonstrable segregation, the right boundary is the account, and the design goes back to B.

Fairness: quota, borrowing, and who gets evicted

Task Governance splits two policies. Compute prioritization decides how idle capacity is handed out (first-come first-serve or fair-share) and how tasks are queued (arrival order or task ranking with priority classes). Compute allocation sets, per team, the guaranteed quota, a fair-share weight from 0 to 100, and the lending posture: Lend and Borrow, Lend, or Don't Lend. The borrow limit can be a percentage, up to 10,000% of quota, or absolute per instance type.

The documentation example shows the mechanics: team A with a quota of 6 using 2, team B with a quota of 5 using 4, a 4-unit job lands on team B and 3 units are borrowed from A. What the documentation also says, and what defines operations, is that a task running on borrowed quota can be preempted by a higher-priority task when capacity runs out.

Silent failure number two: a job without kueue.x-k8s.io/priority-class gets the lowest priority. An inference endpoint deployed without the label becomes the first eviction candidate in the cluster.

Preemption has a price in GPU-hours. A 16-GPU training run on borrowed capacity with a checkpoint every 30 minutes loses up to 8 GPU-hours per eviction, 4 on average. If the cluster evicts that job twice a day, that is roughly 240 GPU-hours a month thrown away without tripping a single alert. That is why I treat borrowed capacity like spot: only jobs with checkpoints on FSx and idempotent resume run on it.

Chargeback: who pays for idle and borrowed GPUs

The post uses Kubecost grouped by namespace, and since the namespace is already the team, no extra workload tagging is needed. That works for the question "how much did team A consume". On its own it does not answer two questions that show up at the first month-end close.

The borrowed GPU: when team B runs on team A's quota, the pod lives in B's namespace. Charging B is correct, but team A will look at the report and see it paid for a guaranteed quota someone else used. Decide upfront whether the guaranteed quota is a fixed price (team A pays for the reservation, used or not) or usage-based. Both rules are defensible; changing the rule mid-quarter is not.

The idle GPU: capacity not allocated to any quota needs an owner in the report. Split Cost Allocation Data for EKS, which started covering GPU, Trainium and Inferentia on 09/02/2025, splits cost per pod and reports unused capacity separately, with a 9:1 GPU versus CPU and memory weighting in the documentation example. That example is an EC2 instance. HyperPod nodes are ml.* instances billed through SageMaker, so confirm on your bill whether that data shows up before you swap Kubecost for CUR. I would keep Kubecost as the chargeback source and use CUR to reconcile the total.

The three designs side by side

Criterion	A: shared HyperPod + Task Governance	B: one cluster per team	C: namespaces + ResourceQuota
Isolation boundary	Namespace + access entry + NetworkPolicy	AWS account	Namespace + RBAC
One team's idle GPU	Lent, with limits and preemption	Stays idle	Stays idle (quota is a hard cap)
Contention	Fair-share by 0–100 weight + task ranking	None across teams	Arrival order
Chargeback	Per namespace, needs a borrowing rule	Account bill, trivial	Per namespace
Parts to maintain	Quota policies, UID/GID webhook, Kubecost, dashboards	N clusters, N upgrades, N pipelines	Few, until the first dispute
When it fails badly	Job without priority-class or checkpoint	Low utilization becomes fixed cost	Large team monopolizes the cluster

Decision matrix

A: shared HyperPod + Task Governance

Pros

One team's idle GPU becomes another's capacity, with IdleResourceSharing covering even unallocated capacity
Explicit priority: production inference preempts experiments
One driver and EKS upgrade for every team

Cons

A namespace will not pass a strong-segregation audit
The UID/GID webhook and its DynamoDB map become yours to run
Preemption wastes GPU-hours when jobs do not checkpoint

Verdict: Default for teams in one company under one regime

B: one cluster per team

Pros

Segregation provable by account, IAM and bill
A bad upgrade's blast radius stays within one team

Cons

Each team pays its own GPU floor, even when idle
Platform effort multiplied by N

Verdict: When data or regulation differ across teams

C: namespaces + ResourceQuota

Pros

Nothing new to learn

Cons

No borrowing, no cross-team priority, no fair-share
GPU disputes become meetings, not policy

Verdict: Only as a step until the second team arrives

Guaranteed quota is not guaranteed capacity: The documentation says to keep the sum of quotas below cluster capacity: 20 instances, quotas summing to under 20. But HyperPod pulls unhealthy nodes out of rotation, and idle resource sharing already excludes NotReady or unschedulable nodes from its math. Team quotas do not shrink with them. In a 16-node cluster with 2 nodes recovering, quotas summing to 16 promise capacity that does not exist at that moment. I keep at least one node per instance type outside any quota.

What I would do: I would start with design A, but with three rules written down before the first job: any manifest without kueue.x-k8s.io/priority-class is rejected by an admission policy, every job running on borrowed quota writes checkpoints to FSx at a known interval, and the billing rule for borrowed quota is signed off by the cost center owners. I would also make the UID/GID webhook's failurePolicy an explicit decision: Fail so the cluster stops scheduling when DynamoDB does not answer, Ignore only if you accept pods with the wrong POSIX identity writing to /fsx. I have seen shared platforms die not from a technical limit, but because the first unexplained eviction made a team ask for its own cluster. The lesson: in a shared cluster, predictability is worth more than utilization.

Verdict

Use Option A (shared HyperPod with Task Governance) when the teams belong to the same company, handle data with the same classification, and someone owns the quota policies and Kueue observability (kubectl describe workload has to become routine, not archaeology). Use Option B (cluster per account) when one team operates under LGPD with customer data and another does not, or when the audit requires segregation by account; accept paying for idle GPUs as the price of isolation. Treat Option C as a transitional state: as soon as a second team competes for GPUs, move to A. The gain in design A is not having a single cluster, it is lending idle GPUs without negotiating in a meeting, and it only pays off if evictions are predictable and chargeback is agreed upfront.

Rating: A, com regras de admissão e checkpoint /

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

Critical or non-critical: the failure contract of ECS daemons

Fernando Azevedo — Thu, 08 Oct 2026 18:00:58 +0000

Every container cluster carries a layer of software nobody asked for and everybody needs: the log shipper, the EDR agent, the metrics exporter, the mesh proxy. For years that layer was treated as a packaging detail, one more sidecar in the task definition, a DaemonSet on Kubernetes. The launch of non-critical daemons in Amazon ECS Managed Daemons, on September 3, 2026, exposes the question that was hiding underneath the packaging: when that agent dies, what should happen to the machine hosting it? The answer is now a parameter (critical), and parameters get versioned, reviewed, and gotten wrong.

The problem: cross-cutting agents are coupling disguised as convenience

The sidecar-per-task model solves distribution and fails at economics and governance. An instance running 40 tasks runs 40 copies of the log shipper; at 128 MiB each, that is 5 GiB of RAM spent duplicating one function. Worse than the waste is the drift: the agent version becomes the property of each application team, and the security team finds out during an audit that three services still run a seven-month-old image.

ECS already had the DAEMON scheduling strategy for the EC2 launch type, but that is placement only, one daemon per instance, with no guarantee that it is up before the first application task serves traffic. For telemetry that is a hole in the graph. For a runtime security agent, it is a window in which the instance processes transactions without coverage.

Managed Daemons, GA on April 1, 2026 for ECS Managed Instances capacity providers, closed that hole by inverting the relationship: the instance only transitions the application task to RUNNING after the daemon is running. The price of that guarantee was rigid coupling, any daemon failure drained and replaced the instance. Correct for a compliance agent. Absurd for a metrics exporter. The September launch does not add a new capability; it undoes a coupling that should never have been the only option.

Anatomy: three parts, two guarantees, one switch

The pattern has three distinct resources. First, a daemon task definition: its own resource type with its own ARN (daemon-task-definition/...), separate from your application task definition. Second, a daemon, created with CreateDaemon, binding that definition to a cluster and to one or more ECS Managed Instances capacity providers. Third, the EC2 instances provisioned by those capacity providers, on each of which ECS keeps exactly one daemon task.

The first guarantee is ordering: ECS provisions the instance, starts the daemon task, and only then promotes the application task to RUNNING. Independent measurements published in April 2026 showed the daemon reaching RUNNING 18 to 21 seconds ahead of the application, with the instance held in REGISTERING during that window. That ordering applies to every daemon, critical or not. This is the point I most often see misread: marking a daemon non-critical does not cost you the guarantee that the agent comes up before traffic; it only costs you the guarantee that it stays up.

The second guarantee is remediation, and that is what the critical switch governs. Note how ECS remediates: it does not restart the container, it drains and replaces the whole instance. That is deliberately blunt, an agent that died may have left eBPF probes detached, network rules half-applied, buffers lost. Replacing the instance is the only cheap way back to a known state.

Instance lifecycle and the criticality branch

The startup ordering (left) is identical in both modes. The critical parameter only decides what happens after the daemon fails, and the non-critical branch is only safe if the observability arm is wired up.

🟦 AWS: Provisioning & startup order

Managed Instances capacity provider (compute)
EC2 instance state: REGISTERING (compute)
Daemon task exactly 1 per instance (compute)

🔀 Failure contract, critical parameter

critical = true | false per-daemon, default true (security)

🟥 critical=true, fail closed

Registration blocked instance never goes ACTIVE (security)
DRAINING + replace ~5 min per wave, 25% default (compute)

🟩 critical=false, fail open

Instance stays ACTIVE running tasks untouched (compute)
Placement continues registration never blocked (compute)
Coverage gap silent unless observed (security)

🟧 Observability, the other half of the pattern

EventBridge Daemon service action (messaging)
Service action log critical + non-critical (data)
CloudWatch alarm + bake time rollback (data)
Coverage SLO % instances with healthy daemon (data)

Flows

svc -> cp: 1. pending task
cp -> ec2: 2. provisions EC2
ec2 -> daemon: 3. daemon first (~18-21s)
daemon -> gate: 4. failed / stopped / unhealthy
gate -> block: fails to start
gate -> drain: fails after ACTIVE
gate -> active: critical=false
active -> place: new tasks land
place -> gap: no agent coverage
gate -> eb: event in both modes
gate -> log: service action log
eb -> alarm: rule → SNS/Lambda
alarm -> drain: circuit breaker rollback
eb -> slo: feeds the coverage error budget
gap -> slo: only visible if measured

What actually differs between the two modes

Criterion	Critical (default)	Non-critical (critical=false)
Daemon fails to start	Instance never completes registration; no application task is placed on it	Registration is never blocked; application tasks launch immediately
Daemon stops or goes unhealthy later	ECS drains and replaces the whole instance (auto-repair)	Instance stays ACTIVE; existing tasks are untouched
Placement of new tasks	Stops on the affected instance	Continues normally, agent or no agent
Startup ordering	Daemon before the application	Daemon before the application, identical
Signal emitted on failure	EventBridge event + service action log	EventBridge event + service action log, identical
Residual risk you take on	Fleet churn: an unstable agent turns into cascading instance replacement	Blind spot: coverage degrades silently if nobody consumes the event

When to use each mode: the rule I apply

The rule fits in one line: mark as critical the agent whose absence changes what the system is allowed to do; mark as non-critical the agent whose absence only changes what you can see. A runtime EDR, a network policy enforcer, an mTLS sidecar, or a file-integrity agent required by PCI DSS fall in the first group, without them the instance should not be eligible to process a transaction. An OTel metrics collector, a node exporter, a continuous profiler fall in the second.

The nuance lives in the log router. In financial environments the log often is the audit trail: "what you can see" becomes "what you can prove", and Fluent Bit stops being auxiliary. Do not classify by tool category; classify by the regulatory obligation that stream underwrites. If the agent's local buffer survives the outage window without dropping events, non-critical is defensible; if it drops them, it is critical.

The design corollary is what pays off most in practice: because criticality is per daemon, and a capacity provider accepts multiple daemons, do not package agents with different contracts into the same task definition. A task definition bundling the EDR and the metrics collector forces the stricter contract on both, and turns a collector memory leak into fleet replacement. Two daemons, two release cycles, two owners, two criticalities.

"Non-critical" is a control-plane term, not a business term. The parameter describes what ECS does to the instance, not how much that agent matters to you.

What you pay for classifying wrong, in both directions

Too critical. A bad agent revision makes the task flap, and every flap becomes an instance replacement. On a 60-instance fleet with the drain percentage at its default of 25%, that is 15 simultaneous replacements per wave; at the measured ~4m50s per wave, a one-line agent change costs roughly 20 minutes of fleet rotation. The direct bill impact is modest and computable: the Managed Instances management fee for a c6a.2xlarge is US$0.037/hour per instance (Oregon), so 60 instances add up to ~US$1,620/month in fee alone, plus EC2 and the overlap during draining. The expensive bill is elsewhere: application task rescheduling, drained connections, cold caches, cold JIT.

Two defaults conspire here. Bake time defaults to 0 minutes, meaning that once the last wave finishes, the window in which the circuit breaker watches alarms to roll back is nil. A regression that only shows up under load sails right through.

Too non-critical. Fail-open emits no pain: the instance stays ACTIVE, the application SLO does not move, and agent coverage decays in silence. Add the documented operational trap: daemon configuration settings are not persisted between updates, and critical defaults to true. An UpdateDaemon fired by a pipeline without an explicit --no-critical re-arms fleet-wide churn that nobody asked for. It is exactly the kind of change that only surfaces at the next incident.

Anti-patterns this launch makes easy

Marking everything non-critical to "stop the churn", you did not remove the failure, you just muted the alarm it used to pull.
Flipping to non-critical without first wiring the EventBridge rule for daemon events to a destination that wakes somebody up. Fail-open with no event consumer is a compliance blind spot that looks like stability.
Packaging the security agent and the telemetry agent in the same daemon task definition, forcing a single failure contract onto software with opposite risk profiles.
Managing daemons via ad-hoc CLI calls: since UpdateDaemon does not persist settings and critical reverts to the true default, every partial call is a silent contract change.
Keeping bake time at 0 while configuring alarms, paying the cognitive cost of the circuit breaker without buying the observation window that makes it useful.
Treating the critical daemon as a substitute for application health checking. It repairs agent coverage, not business health; a healthy agent on a disk-full instance is still "healthy".

A Well-Architected reading of the decision

security: Fail-closed remains the correct default for enforcement agents: a critical daemon keeps the instance from registering without coverage. Govern who may loosen it with IAM on ecs:CreateDaemon/ecs:UpdateDaemon, scoped by daemon ARN and conditioned on tags, so the security team owns the security daemon and the platform team owns the rest.
reliability: Non-critical removes a real correlated failure mode: an agent regression hitting every instance at once used to produce simultaneous fleet replacement. In exchange, agent recovery stops being automatic, plan explicit repair.

Reference configuration I would take to production: Two daemons per capacity provider: sec-runtime with critical: true and obs-otel with critical: false. Drain percentage at 10 (not 25) so a bad wave hits 10% of the fleet, and bake time between 15 and 30 minutes with at least two CloudWatch alarms wired to the circuit breaker, one on application errors, one on the agent's own health. One EventBridge rule capturing daemon service action and daemon deployment state change events, targeting SNS for the critical daemon and a queue/Lambda feeding the coverage SLO for the non-critical one. Every daemon declared in full in IaC, never by partial CLI call, with drift detection, because UpdateDaemon reverts whatever you omit. And check the instance profile: it needs AmazonECSInstanceRolePolicyForManagedInstances; the legacy AmazonEC2ContainerServiceforEC2Role policy stops the daemon from starting.

Curator's note: I have watched a memory leak in an observability agent turn into node rotation during a settlement window, and the post-mortem did not name the leak as root cause, it named the fact that nobody had ever written down which agents were allowed to die. Since then I require an explicit criticality field in the ADR for any agent before it lands on the platform, with the owner and the regulatory obligation it underwrites next to it. This launch finally gives me a place in the control plane to materialize that decision instead of leaving it implicit in a runbook. What I would not do is use critical: false as a painkiller: if an agent fails often enough to hurt, the agent is the problem, and marking it non-critical merely trades a noisy incident for a silent gap. My default stays critical, and every exception needs a coverage SLO to get approved.

Verdict

Adopt it, and treat critical as an architectural decision, not a tuning flag. Keep the critical default for enforcement agents (EDR, network policy, mTLS, file integrity) and for log routers that underwrite an audit trail. Move to non-critical only metrics, profiling and telemetry whose temporary loss is not a loss of evidence, and only after the EventBridge rule has a consumer and the coverage SLO is published. Split agents with different contracts into different daemons, drop the drain percentage to 10, raise bake time to 15–30 minutes, and declare everything in IaC, because UpdateDaemon silently reverts omissions to critical: true. If your cluster is not on Managed Instances yet, this launch does not change your roadmap; if it is, it closes the last reasonable excuse for keeping an agent sidecar in every task definition.

Rating: Adopt, with the switch written down

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

AMIs with a contract: anatomy of EC2's new instance-type block

Fernando Azevedo — Thu, 08 Oct 2026 18:00:24 +0000

Um AMI sempre foi um contrato verbal. O artefato sabe exatamente de que hardware ele precisa, driver ENA, NVMe, arquitetura do binário, extensão de CPU que a JVM compilou, e nada disso estava escrito em lugar nenhum que o RunInstances pudesse ler. A partir de 4 de setembro de 2026 dá para escrever: o dono do AMI declara SupportedInstanceTypes, UnsupportedInstanceTypes ou os dois, e o EC2 bloqueia o lançamento fora da lista. Sem custo adicional, em todas as regiões. A leitura fácil é "menos launch quebrado". A leitura que interessa é outra, o ponto de enforcement mudou de lugar, e com ele mudou quem paga a conta da manutenção.

The problem: the contract existed, it just wasn't executable

Every platform team that publishes golden AMIs has lived this scene. The hardened image was built for x86_64, with a statically compiled EDR agent and a kernel that only loads its network driver on a Nitro host. Someone creates an Auto Scaling group with attribute-based instance type selection asking for 4 vCPUs and 16 GiB, leaves Spot diversification wide, which is the correct recommendation for filling capacity, and EC2 picks a Graviton family because it matches the attributes and is the cheapest right now.

The instance comes up. That detail is what makes the failure mode hurt. It doesn't die at RunInstances; it dies later, with no console, no network, nothing but a health check running out. The ASG waits the default 300-second grace period, marks the instance unhealthy, terminates it and tries again, and since the allocation decision hasn't changed, it picks the same pool. You get a launch/terminate loop billed at a 60-second minimum per instance created, an alarm firing at 2 AM, and a graph that looks exactly like a capacity shortage.

Every defense available lived on the caller side: an ec2:InstanceType condition in an IAM policy, an SCP at the Organization, ExcludedInstanceTypes inside a fleet's InstanceRequirements. All correct, all sharing one structural defect, they describe what that principal may launch, not what that image can survive. Share the AMI with 40 accounts and you replicate the rule 40 times, in 40 places that drift apart.

The contract travels with the artifact, and fails at the gate, not at the health check

Reference design: the pipeline derives the list from a capability query, writes the specification onto the AMI, and RunInstances becomes the single evaluation point for every account that received the image.

🏗️ Pipeline de imagem, build time

EC2 Image Builder pipeline do golden AMI (ci)
DescribeInstanceTypes filtro por arquitetura + hypervisor (compute)
ReplaceImageInstanceTypeSpecification lista explícita, gerada (ci)

🟧 AWS, plano de controle do EC2

AMI InstanceTypeSpecification (storage)
CopyImage / share especificação preservada (storage)
RunInstances avalia tipo × contrato (security)

🚀 Consumidores, launch time

Auto Scaling group ABIS + Spot diversificado (compute)
Launch template versão do AMI fixada (compute)
Instância lançada tipo dentro do contrato (compute)
InvalidParameterCombination lançamento bloqueado (external)

🔍 Sinais, o que você tem que instrumentar

CloudTrail errorCode em RunInstances (data)
Scaling activity falha registrada no ASG (messaging)
CloudWatch metric filter por imageId (data)

Flows

plat -> builder: publishes the golden AMI
builder -> capq: asks what the hardware does
capq -> setspec: compatible types, enumerated
setspec -> ami: writes both lists
ami -> copy: cross-region / cross-account
ami -> gate: contract read at launch
asg -> lt: scales out
lt -> gate: RunInstances
gate -> ok: type allowed
gate -> blocked: type off the list
blocked -> ct: auditable error
blocked -> sa: capacity not delivered
ct -> alarm: metric filter
sa -> alarm

Anatomy: two lists, one evaluation order, zero state

The attribute is InstanceTypeSpecification and it carries SupportedInstanceTypes and UnsupportedInstanceTypes. The logic is short enough to fit in an ADR: with no specification, everything passes (that's the default, and that's why nothing broke on the 4th); with only the unsupported list, everything passes except what matches; with the supported list set, the type must match the supported list and must not match the unsupported one. Denial always wins.

You write it with aws ec2 replace-image-instance-type-specification --image-id ami-... --instance-type-specification '{"SupportedInstanceTypes": ["t3.*","a2.*"], "UnsupportedInstanceTypes": ["t3.micro"]}'. Omitting the parameter removes the restriction. You read it back through describe-images, which returns the InstanceTypeSpecification field with the patterns as stored. A blocked launch returns InvalidParameterCombination from RunInstances, with a message pointing at DescribeImages as the place to look.

Four properties change the design, and none of them are in the announcement:

Replace, not Modify: the operation swaps the whole specification. There is no add or remove, it's read-modify-write with no conditional version. Two pipelines patching the same image in parallel produce last-writer-wins silently, and the loser only finds out when a legitimate launch gets blocked.

Only the owner writes. An account that received the shared AMI consumes the contract and does not negotiate it.

CopyImage preserves the specification. That's what makes the pattern worth it in a multi-Region topology: you declare once and the sa-east-1 replica is born with the contract.

Marketplace is mutually exclusive. You can't set a specification on an AMI listed in AWS Marketplace, and you can't list an AMI that already has one. If you're an ISV shipping an appliance, this feature does not exist for you.

Three ways to constrain instance type, and what each one actually covers

Criterion	IAM / SCP `ec2:InstanceType` condition	`ExcludedInstanceTypes` in ABIS	`InstanceTypeSpecification` on the AMI
Where the rule lives	In the principal's policy, per account or OU	In the fleet or ASG configuration	On the artifact, once, in the owning account
Applies to launches from another account	No, that policy is theirs, not yours	No, the consumer configures the fleet	Yes: EC2 evaluates it regardless of caller
Survives CopyImage and sharing	Irrelevant, it doesn't follow the image	Irrelevant, it doesn't follow the image	Yes, preserved on copy
When the failure shows up	`UnauthorizedOperation` at launch	Never, the type simply isn't chosen	`InvalidParameterCombination` at launch
Maintenance cost	High: replicate and audit across N accounts	Medium: one edit per fleet that exists	Low per place, high per hardware generation

Where the glob betrays you: a string is not an architecture predicate

Both lists accept *. The docs give the examples: t3.* matches the whole t3 family, g5.* matches g5, *.12xlarge matches any 12xlarge in any family, *xlarge matches any type xlarge or larger. It looks expressive. This is where I'd stop and ask the question that matters: are you describing hardware, or are you describing names?

What the AMI actually requires is a capability: arm64 versus x86_64, Nitro versus Xen hypervisor, presence of a GPU, presence of local NVMe. What the specification evaluates is a pattern over the instance type's string. The two coincide today because AWS names Graviton families with a g and local-storage families with a d. They coincide by convention, and a naming convention is not a contract: c6gn, c7gd, m7i-flex, u7in-32tb and mac2-m2pro already show a scheme full of exceptions, suffixes and infixes that no simple glob covers. A *g.* written to exclude Graviton is a heuristic, and a hand-written heuristic, in a file nobody revisits, is debt that accrues interest.

The second trap is temporal, and it's the expensive one. An explicit allow-list blocks every future generation by construction. New families reach new Regions constantly; the day the next generation shows up in your Spot pool, the fleet you deliberately left wide, because wide diversification is the correct practice for Spot capacity, will skip exactly the cheapest, most available family. You get no error: the ASG simply doesn't choose it, price creeps up, interruption rate creeps up, and nobody connects that to a JSON written eight months ago. It's the same tension behind OnDemandMaxPricePercentageOverLowestPrice, which ships with a 20% default headroom precisely so it doesn't narrow your pool without telling you.

Generate the list, don't write the list: The only way to keep the contract honest is to derive it from a capability query in the same build that produced the image: aws ec2 describe-instance-types --filters Name=processor-info.supported-architectures,Values=x86_64 Name=hypervisor,Values=nitro --query 'InstanceTypes[].InstanceType', feeding the result straight into replace-image-instance-type-specification. For fleets that already think in attributes, GetInstanceTypesFromInstanceRequirements plays the same role and lets you preview the set before writing it. The list ends up explicit, verbose and ugly, which is exactly what you want, because it is regenerated on every build, and your lag behind the AWS catalog becomes your pipeline cadence instead of the memory of whoever wrote the glob.

The hard block on incident day, and the signal nobody instruments

The documentation is explicit: EC2 enforces the specification as a hard block, and launch templates and Auto Scaling groups that reference the AMI can fail if the configured instance type isn't allowed. That isn't an implementation detail, it's the pattern's central trade. You converted a slow, expensive failure (zombie instance, health check, loop) into a fast, cheap one (InvalidParameterCombination at RunInstances). Fast and cheap wins on 95% of days. On the other 5%, you're in the middle of a zonal capacity event, price-capacity-optimized has finally found a pool with room, and your contract says no.

In financial systems that has a name: you traded correctness risk for availability risk, and you did it on purpose. When the AMI genuinely won't boot on that hardware, the trade is trivially good, the instance was never going to serve traffic. When the list is merely stale, you invented an unavailability mode that didn't exist before, in exactly the window where elasticity matters. That's why a generated allow-list and a minimal deny-list aren't aesthetic preferences: one errs toward letting through, the other errs toward blocking.

The hole is observability. The specification doesn't affect running instances, emits no metric of its own, and shows up on no dashboard by default. The auditable signal is the RunInstances event in CloudTrail carrying an errorCode, build a metric filter grouped by requestParameters.imageId and alarm on any block, because in steady state the expected rate is zero. A recurring block isn't the control working; it's a launch template pointing at an AMI whose contract changed underneath it. And add the read side to the runbook: describe-images with its InstanceTypeSpecification field is the first thing to check when a launch fails on a parameter, the error message itself sends you there, and whoever is on call at 2 AM needs to know that before opening a support case.

How to get this wrong

Applying the contract to an already-shared AMI without an inventory of consumers: you can't see the launch templates in the other 40 accounts, and the block only surfaces on their next scale-out, which is probably a traffic spike, not a deploy.
Hand-writing the list: a human-curated glob is a snapshot of the instance catalog on the day it was written, and the catalog changes every week.
Two pipelines writing the same AMI's specification: ReplaceImageInstanceTypeSpecification replaces everything and has no conditional write; the second one erases the first with no error.
Using this as a cost control: blocking *.24xlarge to contain spend is FinOps policy disguised as compatibility, that belongs in an SCP with ec2:InstanceType or in OnDemandMaxPricePercentageOverLowestPrice, where exceptions have an owner and a process.
Mistaking this for pipeline validation: the contract fails at launch, not at build. It does not replace a test that boots the image on each allowed family before promoting the AMI to production.

Reading the pattern through the pillars

operational-excellence: The contract becomes generated code in the same pipeline as the AMI, versioned with it. Add describe-images to the runbook: the error message points there, and it's the first diagnostic step.
security: Constraining the list is neither isolation nor least privilege, only the owner writes it and the rule follows the image, but it does not replace an IAM condition or an SCP for governing who launches what.
reliability: A real gain by killing the launch/terminate loop from incompatible AMIs; a real risk of failing to scale during a capacity event if the list is stale. An alarm with an expected rate of zero is what separates the two.
cost: The feature costs nothing and cuts the waste of instances that booted only to die at the health check, but a narrow allow-list pushes the fleet into pricier pools while emitting no signal at all.

What I'd actually do: I'd turn this on first for the AMIs that carry a genuine physical limit, an arm64 image, an image whose driver only exists on Nitro, an inference image that assumes a GPU, and leave general-purpose AMIs alone, where the permissive default is already right. On every image I restrict, I'd generate the list from describe-instance-types inside the build and fail the pipeline if it comes back empty, because an empty list from a broken query blocks everything silently. And I'd put a zero-expected-rate alarm on the RunInstances errorCode in CloudTrail the same day, not next sprint. The lesson that cost me, on financial-grade platforms: any control that blocks something and doesn't count how often it blocked eventually becomes a root cause nobody can name. The block is the easy part; the counter is what saves you at 2 AM.

References

Verdict

Adopt it, narrowly scoped, with the list generated. Use InstanceTypeSpecification when three conditions hold together: the AMI has a real hardware incompatibility rather than a cost preference, you don't control every account that launches the image, and the pipeline that builds the AMI can derive the list from a capability query on every build. Missing any one of them, an ec2:InstanceType condition in an SCP remains the right tool, it errs toward people who know how to ask for an exception. Don't use this as a FinOps control, don't hand-write globs, and don't enable it on a shared AMI before running describe-images against the launch templates already pointing at it. The feature costs nothing and the permissive default guarantees nothing breaks until you act; the price is entirely maintenance, and it comes due every time AWS ships a new family. If your team has nowhere to hang that maintenance, the pattern is still correct and the answer is still no.

Rating: Adopt with scope

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

3LO in Bedrock Managed KB: 5-minute setup, 1 irreversible decision

Fernando Azevedo — Thu, 08 Oct 2026 18:00:20 +0000

The question that matters in this announcement isn't "how long does it take to connect SharePoint?", it's "who can read what you indexed once it's connected?". The user-managed setup (3LO) AWS shipped on September 4, 2026 for SharePoint, OneDrive and Confluence in Amazon Bedrock Managed Knowledge Base answers the first question very well: you click Sign in, authorize with your corporate account, and Bedrock creates the Secrets Manager secret with the refresh token for you. The second question is where I spent three weeks.

What the five-minute path actually trades away

The SharePoint setup page is explicit and worth reading twice: user-managed setup does not support document-level access control (ACL). All indexed content is accessible to anyone who can query the knowledge base, and individual SharePoint permissions are not enforced. The same sentence shows up on the Confluence page, word for word.

This isn't a bug or a temporary gap, it follows directly from the flow. In 3LO, Bedrock stores a delegated refresh token, issued to you, carrying four permissions: Sites.Read.All and User.Read on Microsoft Graph, offline_access to renew the token, and AllSites.Read on the SharePoint Online API. These are delegated permissions, inherited from what your account can see. The crawler walks into the tenant wearing your identity and copies everything it can reach into an index that has no notion of who is who.

The path that does support ACL is a different one: ENTRA_ID_APP_ONLY for SharePoint, ENTRA_APP_ID for OneDrive, SERVICE_ACCOUNT for Google Drive, BASIC for Confluence. Those are application permissions, with admin consent, and for SharePoint with ACLs they require Sites.FullControl.All, or Sites.Selected with a per-site fullcontrol grant, because Sites.Read.All is not enough to read item-level permissions.

The trade, stated plainly: 3LO swaps a two-week ticket with the identity team for an index with no internal border. For a prototype over a public engineering wiki, that's a good deal. For the credit policy repository, it's an incident waiting for a date.

The journey: from a 3LO prototype to a production knowledge base

Week 0, the prototype is born in the console, asking nobody: An analyst holding bedrock:GetDataSourceAuthorizationUrl, bedrock:CreateDataSourceToken and secretsmanager:CreateSecret completes the flow alone. Bedrock creates the secret at arn:aws:secretsmanager:{region}:{account}:secret:bedrock-managedkb-oauth/{prefix}/{connector}/{uuid} and the data source comes up with authType=MANAGED_OAUTH2. No ticket, no tenant ID, no review.
Week 1, the gate nobody read: The Entra App-Only page states what locks the whole journey: the decision to enable ACLs cannot be changed after the data source is created. And 3LO doesn't offer the option at all. Meaning: there is no in-place upgrade from MANAGED_OAUTH2 to ACL, there is a new data source and a full re-ingestion.
Week 2, inventory of what was already copied: Before building the new path, answer what the old one already indexed. The crawl followed the identity of whoever signed in; if that person is senior, the reach is theirs. List the included sites and spaces, cross them against your data classification, and treat the current index as a copy repository, because that's what it is.
Week 3, app registration and certificate: Here the ticket 3LO had eliminated comes back. A Global Administrator or Privileged Role Administrator registers the single-tenant application with no redirect URI, adds the application permissions and grants admin consent. You generate a pair with openssl req -x509 -newkey rsa:2048, upload the .cer to Entra and the .p12 bundle to S3 with SSE. The .p12 is mandatory when ACLs are on.
Week 4, secret, service role and the new data source: The secret carries clientId and certificatePassword, always set the password explicitly; if you omit it, Bedrock opens the bundle using the client ID as the password. The service role gets s3:GetObject on the certificate object with an aws:ResourceAccount condition, plus kms:Decrypt if you used a CMK. The data source is created with certificateS3Path, which ENTRA_ID_APP_ONLY requires even with ACLs off.
Week 5, the application starts proving who is asking: Retrieve gains userContext.userId with the user's universal email. Without it, ACL-enabled data sources return zero results, excellent as a fail-safe, terrible as a production surprise. Authentication stays yours: Bedrock filters by the identity you assert, it does not verify that the assertion is true.
Week 6, delete the old path, including the secret: Delete the MANAGED_OAUTH2 data source and the orphaned bedrock-managedkb-oauth/* secret, it still holds a valid refresh token. On the Microsoft side, revoke by disabling or deleting the service principal under Enterprise applications; on Atlassian the grant is per site, so every *.atlassian.net needs its own revocation.

The two phases and the immutable gate between them

3LO is fast because it ends in a delegated refresh token; production is slow because it starts from a certificate and admin consent. The gate in the middle isn't a step, it's a decision you can only make once per data source.

🧪 Fase 1: Protótipo (3LO, minutos)

Console Bedrock KB GetDataSourceAuthorizationUrl (frontend)
Secrets Manager bedrock-managedkb-oauth/.../uuid (security)
Data source authType=MANAGED_OAUTH2 (data)

🔐 Entra ID / Atlassian, consentimento

Consentimento delegado Sites.Read.All + offline_access (external)
Admin consent do tenant obrigatório se o tenant restringe apps (external)

🚧 O portão imutável

aclEnabled não muda após CreateDataSource (security)

🟧 AWS: Fase 2: Produção (2LO + ACL)

Entra app registration application perms + Sites.FullControl.All (security)
Amazon S3 certificate.p12 (SSE) (storage)
Data source ENTRA_ID_APP_ONLY + ACL (data)
Managed KB conteúdo + permissões crawladas (ai)
Retrieve userContext.userId = e-mail (ai)

👤 Sua aplicação, autenticação verificada

App corporativa OIDC valida o usuário final (compute)

Flows

analista -> signin: 1. Sign in
signin -> consent: 2. OAuth popup
consent -> adminconsent: blocked without consent
signin -> secret3lo: 3. stores refresh token
secret3lo -> ds3lo: 4. generated secretArn
ds3lo -> gate: 5. no ACL, permanently
gate -> dsprod: 6. new data source + re-ingestion
entraapp -> certs3: .p12 (private key)
certs3 -> dsprod: certificateS3Path
dsprod -> kb: content and ACL crawl
app -> retrieve: already-verified identity
retrieve -> kb: pre-retrieval filter
kb -> retrieve: real-time check at the source

Before and after, in numbers the docs confirm

4: delegated permissions granted on the 3LO path. Sites.Read.All, User.Read, offline_access and AllSites.Read, read access to everything the signing account can see, renewable indefinitely.
0: documents filtered by permission under 3LO. No ACL means no userContext: whoever queries the knowledge base reads the entire index, regardless of their SharePoint access.
365 dias: default certificate validity on the production path. An expired certificate makes every sync fail. Rotation becomes a runbook item with an owner and a date, not a memory.

Identity: email is the key, and there is no translation

Getting to the ACL path solves less than it looks, and it solves it in a very specific way. Bedrock Managed Knowledge Base uses email as the universal identifier: there is no alias resolution and no cross-identity-provider mapping. The userId you pass to Retrieve must match exactly the email associated with that user in each connected data source. If your corporate IdP issues fernando.azevedo@company.com and Confluence stores fazevedo@company.com, the match fails silently and the user simply gets no results from that source. Silence is the worst failure mode a search system can have: nobody files a ticket over an incomplete answer, they file over a wrong one.

Groups run on a different ruler. They're identified however the source represents them, name, ID, whatever, and user→group membership comes from the crawl. The docs are blunt: group memberships are as fresh as the last sync. If a permission changed between syncs, the pre-retrieval filter doesn't know. The saving grace is the second layer: for SharePoint, OneDrive, Google Drive and Confluence there's real-time verification against the source, which catches exactly that window. S3 and Custom have no such net, because their ACLs come from a metadata file you supply.

Two notes that change your application design: third-party IdP credentials are cached for up to 1 hour, and permission changes are eventually consistent, typically within a few minutes. So "I revoked access now" and "the RAG stopped showing it" are not the same instant. In a BACEN- or LGPD-bound environment, that gap belongs in the design document, not in the audit findings.

The two paths, side by side

Criterion	Prototype: `MANAGED_OAUTH2` (3LO)	Production: `ENTRA_ID_APP_ONLY` (2LO)
Who authorizes	The user, in the popup, unless the tenant restricts third-party apps, in which case admin consent is required again	Global Administrator or Privileged Role Administrator, with explicit admin consent
Stored credential	Delegated refresh token, in a secret Bedrock creates itself	A `.p12` certificate in S3 plus `clientId`/`certificatePassword` in a secret you create
Document-level access control	Not supported, and the flag can't change afterwards	Supported, with pre-retrieval filtering and real-time verification
Crawl reach	Whatever the signing account can see, variable, invisible from the AWS console	The whole tenant with `Sites.Read.All`/`Sites.FullControl.All`, or site by site with `Sites.Selected`
What breaks	The offboarding of whoever authorized; a token revoked in Entra; content shared after the sync, which needs a resync	An expired certificate (every sync fails); a forgotten per-site grant for a new site, which is not retroactive

The failure modes that only show up later

The token has an owner, and owners resign. The 3LO refresh token belongs to a person. When they leave and the account is disabled, the sync stops, and the index stays right there, holding whatever content they could reach. The production path has no such coupling: the application authenticates with a certificate, with no human sign-in at crawl time or query time.

Fail-closed is good, but it looks like a bug. ACL-aware retrieval fails closed: a group-resolution error, a real-time verification timeout or an internal error means the document is not returned. The response may carry fewer results than the requested maxResults, and Bedrock does not backfill. Your application must treat partial responses as normal, and tell "zero because of ACL" apart from "zero because of relevance", the API signals resolution failures precisely for that.

A document without an ACL is an invisible document. In an ACL-enabled data source, a document whose permissions weren't extracted goes to nobody. Absence is treated as restricted, not public. After the migration, expect a recall drop and investigate that before blaming the embedding model.

The secret prefix is your only IAM lever, and the default is ambiguous. The 3LO IAM page says that without a prefix Bedrock uses default; the connector pages say the created secret uses the bedrock-managedkb-oauth prefix. Don't build a policy on that assumption: supply an explicit secret name prefix at sign-in and write the policy with Resource scoped to it before anyone runs the flow. That's the difference between secretsmanager:CreateSecret on a predictable path and a Resource: "*" nobody will review later.

The risk isn't 3LO, it's the 3LO nobody planned for: Two Bedrock permissions (GetDataSourceAuthorizationUrl and CreateDataSourceToken) plus secretsmanager:CreateSecret are enough for anyone with console access to index the corporate SharePoint into a knowledge base with no internal border, without touching a single identity process. If your production account already grants those actions by wildcard in some "Bedrock developer" policy, you don't have a new feature, you have a new path to accidentally authorized exfiltration. Deny bedrock:CreateDataSourceToken outside sandbox accounts before you discuss RAG architecture.

Anti-patterns I've watched grow out of this flow

Promoting the prototype by switching authType: it doesn't exist. The ACL flag is immutable per data source, so promoting means creating a new data source and re-ingesting, plan the re-ingestion cost and the duplicate-index window.
Authorizing 3LO with the architect's account: the more senior the authorizer, the wider the crawl. If you're going to use 3LO, use an account with deliberately narrow access to the in-scope sites.
Treating ACL awareness as authorization: the docs themselves say it's filtering, not a security boundary. Bedrock does not authenticate end users, it trusts the userId your application asserts.
Unblocking user-installed apps in Atlassian "just for a minute": while that toggle is open, any user can authorize any OAuth app across the entire site. Have the admin authorize directly instead.
Leaving the 3LO secret behind during cleanup: deleting the data source doesn't delete the secret, and it holds a refresh token that still renews.

Well-Architected lens on the decision

security: Deny bedrock:CreateDataSourceToken in production accounts via SCP; require a secret name prefix and scope Resource to it; prefer Sites.Selected over Sites.FullControl.All whenever the site list is known; store the .p12 with SSE-KMS and kms:Decrypt restricted to the service role.
reliability: A certificate with the default 365-day validity takes every sync down when it expires, alarm 45 days ahead. Per-site grants are not retroactive: a new site without a grant simply isn't crawled, with no visible error on the data source.

Let the prototype exist, in an account where it can't grow up

The obvious temptation after reading all this is to block 3LO and be done. I wouldn't. The friction it removes is real: without it, every RAG experiment over the internal wiki turns into a two-week ticket with the identity team, and the practical outcome is that nobody experiments, or worse, somebody exports the wiki into a bucket and builds RAG on the side, with no crawler, no sync and none of the checks we just discussed.

The design I recommend is topological, not political. One sandbox account where bedrock:GetDataSourceAuthorizationUrl and bedrock:CreateDataSourceToken are allowed, with no classified data and a short lifecycle, the whole index dies in N days by process, not by memory. Production accounts with those two actions denied by SCP, which makes 3LO disappear from the menu without a case-by-case argument.

What crosses that border isn't the data source; it's the learning. The prototype answers the genuinely expensive questions: which Confluence spaces hold content that resolves real doubts, which chunking strategy survives an 80-page policy document, whether agentic retrieval with maxAgentIteration above the default of 5 changes the answer enough to justify the cost. None of that depends on ACLs. When the answer is "worth it", then you open the ticket with the Entra team knowing exactly which sites to request under Sites.Selected, and the ticket is shorter because you arrive with a scope, not an intention.

What I'd do on Monday: Before enabling anything, I'd grep the production account's policies for bedrock:*, the odds that the two new actions are already granted by wildcard are high, and no IAM review done before September 2026 could have accounted for them. Then I'd create the sandbox account with the inverted SCP, because blocking production without offering the right place just pushes the problem outside my field of view. I learned that the expensive way on a financial platform: the architecture decision that looks like security is almost always about ergonomics, people route around inconvenient controls with a creativity no policy anticipates. And I'd keep the rule I apply to every new connector: if the flag can't be changed later, it isn't configuration, it's design, and design goes into an ADR, with the trade-off written and signed before CreateDataSource.

Verdict

Use user-managed setup (3LO) when all three conditions hold together: the content is already readable by the knowledge base's entire audience, the index lives in an account with no classified data, and it has an agreed expiration date. Outside that, go straight to ENTRA_ID_APP_ONLY for SharePoint, ENTRA_APP_ID for OneDrive or BASIC for Confluence, not because 3LO is poorly built, but because the ACL flag can't change after CreateDataSource, and today's five-minute shortcut costs a full re-ingestion plus an Entra-team ticket later. This announcement isn't about authentication; it's about where your organization's access boundary exists, and 3LO answers that it stops existing inside the index. That's a legitimate choice for a prototype and an audit finding for everything else.

Rating: Adote em sandbox, negue em produção

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

RAG on ServiceNow: the connector that erases user criteria

Fernando Azevedo — Thu, 08 Oct 2026 17:59:45 +0000

The documentation for Amazon Bedrock Managed Knowledge Base's ServiceNow connector carries two sentences that look like implementation detail when read apart. The first tells you to grant knowledge_admin to the service account, and explains why: the role bypasses per-knowledge-base user criteria restrictions. The second warns that ServiceNow data sources don't support document-level ACLs, every authenticated user who can query the knowledge base sees every crawled document. Together they describe an enterprise assistant that answers anything for anyone. This is the retro of the pilot where I found that out the cheap way: in staging, four hours after the first sync.

What happened

I stood the connector up the day it shipped, September 4, 2026, against a staging instance loaded with a real export: roughly 12,000 knowledge articles spread across six separate knowledge bases, and 40,000 active catalog items. The goal was to measure sync time and retrieval quality for an internal IT assistant, the exact use case AWS names in the announcement.

I followed the documentation line by line. The glide.oauth.inbound.client.credential.grant_type.enabled property set to true. A service account flagged Web service access only, no interactive sign-in. The OAuth application registered through the interceptor page, not by inserting straight into oauth_entity, as the guide itself warns. An Inbound Authentication Profile wired to a REST API Access Policy over now/table, because without it the token is valid and the Table API still returns 401. Step 3 asks for exactly two roles: knowledge_admin and catalog_admin. I assigned both, and ServiceNow inherited around fourteen contained roles.

The sync finished. Then I asked the question I ask of every enterprise RAG before anything ships: I queried the assistant about a subject I knew was restricted to one group on the source instance. The answer came back whole, citing the article, for an identity that held no role there at all. It wasn't a hallucination, it was correct retrieval of a document ServiceNow would never have rendered for that user.

The pilot stopped there. What follows is the retro, blameless: the configuration was right per the documentation, and that is precisely the point.

Timeline

T+0h, service account created with the roles from the guide: knowledge_admin and catalog_admin assigned per step 3. The documentation note says, in plain letters, that these roles bypass per-KB and per-catalog user criteria restrictions. I read it and complied.
T+0h20: OAuth verified with curl: POST /oauth_token.do with grant_type=client_credentials, then GET /api/now/table/kb_knowledge?sysparm_limit=1. HTTP 200 with data. AWS's verification table reads an empty array as a symptom of an insufficient role, meaning success here is already success with bypass.
T+0h35, data source created with no filter: crawlKnowledgeArticles, crawlServiceCatalogs and both attachment fields set to true; filterConfiguration omitted. With no filter the connector crawls every published article and every active item the account can see, which is now everything.
T+3h50, sync completes slower than planned: The troubleshooting page explains it: without inclusionServiceCatalogSysIds, the crawl sweeps the 100,000-plus active items of an enterprise instance. I had 40,000 and it still took hours.
T+4h05, the control query returns restricted content: A natural-language question, no userContext, about an article restricted to one group. Chunk and citation came back normally. Nothing in the trace flags an anomaly, because nothing anomalous happened.
T+4h15, data source deleted, retro opened: Index destroyed in staging, with no end-user exposure. The real cost of the incident was one day of work and a few dollars of storage, the design, not so much.

Root cause: two controls removed at different points of the path: ServiceNow's user criteria isn't ignored at query time, it is ignored at ingestion, by a role the documentation instructs you to assign, and never rebuilt afterwards, because the ServiceNow connector isn't in Managed Knowledge Base's ACL matrix. SharePoint, OneDrive, Google Drive and Confluence support pre-retrieval filtering plus real-time verification; S3 and Custom support pre-retrieval filtering from metadata you supply; Web Crawler and ServiceNow support neither. The rule that seals the hole is unforgiving: non-ACL data sources in a mixed knowledge base return results to every user, with or without userContext on the Retrieve call. This is not a connector bug, it is its published contract, and designing around it is your job.

Where user criteria dies

The path of a restricted article, from the kb_knowledge table to the assistant's answer. The two red points in the drawing aren't failures: they are documented behaviors that, added together, remove the only access control the source ever had.

🏢 ServiceNow, origem do conteúdo

kb_knowledge ~12k artigos, user criteria por KB (external)
sc_cat_item ~40k itens ativos + anexos (external)
svc account (2LO) knowledge_admin + catalog_admin (security)

🟧 AWS: Ingestão

Secrets Manager clientId / clientSecret / instanceUrl (security)
ServiceNow connector v1 incremental por sys_updated_on (compute)
Managed Knowledge Base chunks + embeddings, sem ACL (ai)

🟧 AWS: Recuperação

Retrieve / AgenticRetrieveStream userContext sem efeito aqui (ai)
Assistente de TI autentica no IdP corporativo (compute)

🔐 Fronteira que sobra depois do retro

KB 'publica' crawlPublicKnowledgeArticlesOnly = true (storage)
KB 'restrita' sys IDs isolados + IAM por grupo (storage)
IAM na chamada Retrieve Condition por tag de audiência (security)

Flows

svc -> kb: 1. knowledge_admin bypasses user criteria
svc -> cat: 2. catalog_admin bypasses catalog restrictions
sm -> conn: secret in the same Region as the KB
conn -> svc: 2LO token + Table API /now/table
conn -> mkb: 3. ingests text and metadata, permissions not
mkb -> ret: hybrid search, top-k over everything
emp -> app: natural-language question
app -> ret: 4. query with no per-document boundary
ret -> app: chunk + citation from any crawled article
conn -> kbpub: fix: public articles only
conn -> kbres: fix: single-audience sys IDs
app -> iam: fix: role per audience, not per app
iam -> kbres: bedrock:Retrieve conditioned by tag

The fix: scope at ingestion, because there is none at query time

When the access boundary doesn't exist at retrieval time, it has to exist earlier, in what enters the index. Four concrete changes.

One knowledge base per audience, not one per system. Retrieve takes a knowledgeBaseId; that is the only unit IAM can see. I split into two bases: one fed with crawlPublicKnowledgeArticlesOnly set to true, another with inclusionKnowledgeBaseSysIds listing only the sys IDs whose audience matches an IdP group. The 10,000-knowledge-bases-per-account-per-Region quota isn't the constraint here; the constraint is operational, which is why granularity stopped at audience rather than group.

Authorization moves back into the application. The bedrock:Retrieve permission is now granted through a distinct role per audience, with a Condition on the knowledge base's resource tag. The service answering the user resolves the group from the IdP token and assumes the matching role. That is more code than I would like, and it is the price of using a source with no ACL.

sys ID filtering stopped being an optimization. inclusionServiceCatalogSysIds and inclusionKnowledgeArticleCategorySysIds are scope control and sync-time control at once: the troubleshooting page attributes hours-long syncs precisely to their absence.

Protected deletion. I enabled deletionProtectionConfiguration with deletionProtectionThreshold at 15. If a sync tries to remove more than 15% of the index, which a user criteria mistake at the source triggers easily, the delete phase is skipped instead of emptying the assistant in production.

ACL support by connector, what it changes in the design

Criterion	Pre-retrieval ACL filter	Real-time verification	What it forces on you
ServiceNow	Not supported	Not supported	Segregate per knowledge base and authorize in IAM/the application; never mix audiences in one index.
SharePoint / OneDrive	Supported	Supported	Requires app-only auth (`ENTRA_ID_APP_ONLY` / `ENTRA_APP_ID`) and an email identical to the source's.
Google Drive	Supported	Supported	Domain-wide delegation with `SERVICE_ACCOUNT`; permission changes between syncs are caught at verification.
Confluence	Supported	Supported	Admin token for the real-time check, with `BASIC` auth, rotation becomes a runbook item.
Amazon S3 / Custom	Supported	Not supported	The ACL is your own file/metadata: permission freshness becomes your pipeline's responsibility.
Web Crawler	Not supported	Not applicable	Public content only; anything else is a leak by construction.

Identity: the detail that decides whether the mitigation works

Even on connectors that do carry ACLs, Managed Knowledge Base's identity model is narrower than most enterprise designs assume, worth knowing before you promise fine-grained segregation to the risk team.

The universal identifier is email. userContext.userId on Retrieve is always the user's email, and it must match exactly the email associated with that user in each connected data source. There is no alias resolution and no cross-identity-provider mapping. If the company carries two domains from an acquisition, half the people get zero results, silently, which is the worst failure mode available.

Groups come from the last sync. User-to-group membership is crawled at ingestion and resolved at query time from that snapshot. Real-time verification covers the window between syncs on the four connectors that support it; third-party IdP credentials are cached for up to one hour, and permission changes are eventually consistent, typically within minutes. For a termination, that is exposure time, treat revocation as an IdP and application event, not a knowledge base one.

Missing ACL means inaccessible, not public. A document with no extracted permissions in an ACL-enabled source is returned to nobody, and evaluation fails closed: a group-resolution error yields fewer results, never more. That is the correct behavior, and it is also the most common explanation for "the assistant got dumber after the deploy".

And the sentence the documentation insists on repeating: ACL-aware is filtering, not authorization. Authentication is yours.

Reading it through the pillars

security: The connector concentrates power in one service account holding knowledge_admin and catalog_admin, the documentation explicitly asks you not to add admin, itil or snc_read_only, and that restriction is the little least-privilege you get. The Secrets Manager secret holds clientId, clientSecret and instanceUrl in the same Region as the knowledge base; rotating means regenerating the client secret in the Application Registry, which is shown only once. Treat knowledgeBaseId as a security boundary and condition bedrock:Retrieve by tag.
reliability: Incremental sync keys on sys_updated_on, so an article restored from backup at the source may never return to the index if the timestamp doesn't move. deletionProtectionThreshold (default 15%) keeps a bad sync from emptying the index. The ceilings that matter on a bad day: 200 data sources per knowledge base, 50 concurrent ingestion jobs, 10 TB of raw data per knowledge base, 600 RPM of Retrieve with a 25 RPS burst, and 300 RPM of AgenticRetrieveStream per account.

Anti-patterns this retro put in writing

One knowledge base for the whole company: mixing ServiceNow (no ACL) with SharePoint (ACL) in the same index lets the weakest source set the access level, non-ACL documents return to everyone, even with userContext on the call.
Treating userContext as access control: it filters by the identity you assert, verifying nothing. Without upstream authentication it is a query parameter, not a boundary.
Crawling with no filterConfiguration on an enterprise instance: it turns into an hours-long sync over 100,000-plus catalog items and fills the index with content nobody will ask about, at US$ 5.00 per GB per month to keep it.
Registering the OAuth application by inserting straight into oauth_entity: the documentation requires the interceptor page; the shortcut yields a record that authenticates and then fails at the Table API, and you debug 401 in the wrong place.
Trusting source-side user criteria as an outbound control: it is evaluated per user session inside ServiceNow. An integration account with knowledge_admin doesn't violate it, it is simply never subjected to it.

What now gets measured before any release

The retro doesn't end at a configuration fix; it ends at a signal. A RAG leak throws no error, shows no latency anomaly and never lands on an availability dashboard, it shows up as a useful answer for the wrong person. Three things joined the release checklist.

A canary query suite. A small set of questions whose correct answer is silence, one per audience, run against each knowledge base after every sync. It is access regression testing, it runs in seconds and it costs US$ 1.00 per thousand calls. The alternative is learning about it from the ombudsman.

Document counts per sync, with an expected band. The statistics object from GetIngestionJob reports how many documents were added, failed and removed. A sharp swing usually means a scope change at the source, a category moved, a catalog reactivated, before it means a platform problem. With deletionProtectionThreshold enabled, the sync that would skip its delete phase is exactly the event you want alarming.

The origin of cited chunks. Because the assistant returns citations, the dataSourceId of each retrieved chunk is free telemetry. If a restricted-audience knowledge base shows up in an open-channel answer, the alarm fires without anyone having to read the answer.

The connector saves weeks of hand-rolled pipeline, what it does not save is the design of who may see what. That part was never outsourceable.

Curator's note: If I were putting this into production in a financial-grade environment tomorrow, I'd start with a single knowledge base fed by crawlPublicKnowledgeArticlesOnly set to true and nothing else, the set of articles already public to every employee, and only then open additional bases per audience, each with its own role and tag. The hard-won lesson this pilot reinforced: when a connector erases the source's permission model, the damage doesn't surface as a failure, it surfaces as quality. The assistant gets better when it leaks, because it has more context, and that is why nobody files a ticket. I learned to read the ACL support matrix before the connector's feature list, and to treat the sentence "all authenticated users who can query the knowledge base can see all crawled content" as an architecture requirement, not a footnote.

References

Verdict

Adopt the connector when the ServiceNow content you want indexed is already visible to the assistant's entire audience, public IT knowledge bases, an open service catalog, unrestricted HR FAQs, and when the cost of maintaining your own Table API pipeline, pagination, attachments and sys_updated_on incremental sync is real for your team. In that slice it is a good deal: weeks less code, native scheduling, and bulk-deletion protection by configuration. Don't adopt it as-is when the instance uses user criteria to separate audiences, when risk, legal or fraud runbook articles live in the same kb_knowledge, or when a regulatory requirement forces you to trace who could see what, because then the boundary isn't in the connector, it's in your knowledge base layout and the IAM around Retrieve. What AWS shipped is not outsourced access control, it is outsourced ingestion, and those two keep charging their maintenance cost to whoever always paid it: you.

Rating: Adotar com escopo explícito / Adopt with

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

ECS Early Success Criteria: where your deployment rollback ends

Fernando Azevedo — Thu, 08 Oct 2026 17:59:42 +0000

The question Amazon ECS just handed back to you is not how long your deployment takes, it is the point past which you accept having no automatic rollback. Since September 4, 2026, earlySuccessCriteria lets you close a rolling deployment once a fraction of tasks is healthy, and the documentation says out loud what that costs: after the deployment completes, the circuit breaker and alarm-based rollback no longer roll the service back.

The three numbers that define the feature

90/100: healthy tasks to complete the deployment. With healthyPercent 90 and a desired count of 100. The remaining ten launch through regular service scaling, outside the deployment lifecycle.
2 semanas / 2 we: cleanup window in DEFERRED mode. ECS tries to drain source revision tasks for up to two weeks after success. A task protected beyond that is never cleaned up.
0: new deployment statuses. The lifecycle stays IN_PROGRESS → SUCCESSFUL. Nothing in EventBridge or CloudTrail tells an early success apart from a full one.

The signal: the deployment stopped being a binary contract

Until now, an ECS rolling deployment only completed when four things were true at once: the target revision reached 100% of the desired count with every task running and healthy, neither the circuit breaker nor a CloudWatch alarm triggered a rollback, the alarm bake time elapsed, and the source revision tasks were cleaned up. That is an all-or-nothing contract, designed for a homogeneous fleet that launches in seconds.

earlySuccessCriteria, available for the ROLLING strategy in all commercial Regions and GovCloud (US), splits that contract in two. You declare a healthyPercent, the fraction of the desired count that must be running and healthy on the target revision, and ECS completes the deployment there. With a desired count of 100 and healthyPercent 90, the deployment turns SUCCESSFUL at task 90; the remaining ten launch afterwards through regular service scaling.

The motivating case is easy to recognize: GPU-accelerated inference. The tail of a fleet on specialized capacity does not launch at the speed of its head, the variable is hardware availability, not your container. The deployment stays open waiting for two tasks that depend on instance supply, the pipeline behind it queues, and the next deploy never starts. The same applies to the other side of the trade: services with long-lived connections, whose source revision is slow to drain and holds the deployment open for a reason that has nothing to do with the health of the new code.

Where protection ends: the deployment timeline

The same deployment, split by the success boundary. Left of it everything is reversible; right of it nothing is.

🟧 AWS ECS, janela protegida (rollback ativo)

earlySuccessCriteria enable=true, healthyPercent=90 (compute)
1ª task saudável / first healthy task pré-condição da avaliação (compute)
Gate: 90 de 100 tasks arredondado p/ cima / rounded up (compute)
Deployment circuit breaker BOUNDED_PERCENT 50 → 3..200 (security)
CloudWatch alarm + bakeTimeInMinutes (security)

🔀 Cleanup da revisão fonte / source revision cleanup

BLOCKING drena antes do SUCCESSFUL (messaging)
DEFERRED SUCCESSFUL antes de drenar (messaging)

🟩 Fora do deployment, sem rollback / no rollback

Service scaling 10 tasks restantes / remaining (compute)
Tasks da revisão fonte scale-in protection ≤ 2880 min (compute)

📡 Sinais / observability

DescribeServiceDeployments snapshot pós-sucesso (data)
DescribeServices contagem viva / live counts (data)
EventBridge IN_PROGRESS → SUCCESSFUL (messaging)

Flows

cicd -> cfg: applies deployment-configuration
cfg -> first: ECS waits for 1 healthy task
first -> gate: counts healthy tasks
cb -> gate: can roll back up to here
alarm -> gate: bake time before completion
gate -> blocking: cleanup = BLOCKING
gate -> deferred: cleanup = DEFERRED
blocking -> scaling: success after draining
deferred -> scaling: success before draining
deferred -> drain: async drain, up to 2 weeks
scaling -> ds: only source of real counts
gate -> dsd: freezes the counters
gate -> eb: same event as always
cicd -> eb: unblocks the next deploy

What changes for whoever designs the service

ROLLING only: the BLUE_GREEN, LINEAR and CANARY strategies have no earlySuccessCriteria, they already solve the same problem through traffic shifting, at a different cost.
minimumHealthyPercent closes the range: healthyPercent is an integer between 0 and 100 and must sit between the service minimumHealthyPercent and 100. With the 100% default for replica services, there is no range left.
It rounds up: a desired count of 3 at 50% completes at 2 tasks, not 1. A desired count of 10 at 80% completes at 8. On a small service the gain is zero.
There is no shortcut to zero: ECS launches at least one task on the target revision and waits for it to become healthy before evaluating the percentage. Broken code does not complete early, it fails.
Once complete, it is final: the circuit breaker and alarm-based rollback stop acting after success, including while the remaining tasks launch. A completed deployment cannot be stopped.

The line you are moving is the rollback line

Reading this as a faster deploy is reading it wrong. healthyPercent is not an accelerator, it is a declaration of where automatic protection ends. The documentation is explicit: after ECS completes the deployment, the circuit breaker and alarm-based rollback no longer roll the service back, including while the remaining tasks launch through regular service scaling.

Translated to on-call terms: with a desired count of 100 and healthyPercent 90, the last ten tasks launch without a net. If the ninety-fifth fails for a reason that only shows under real load, a connection pool running out, a secret that only resolves in one AZ, a per-instance license cap: ECS replaces it with the same target revision, repeatedly, without returning to the previous one. The failure mode is not the deployment failing; it is the deployment passing and the service degrading slowly, with the pipeline already green and the team already on another task.

The circuit breaker still protects what is left of the window, and its sensitivity matters. The default BOUNDED_PERCENT type multiplies the value (default 50) by the desired count and clamps the result between 3 and 200: with 100 desired tasks that is 50 tolerated failures before rollback; with 400, the 200 cap applies. Worse, with resetOnHealthyTask at true, the default, the counter resets on every healthy task, so interleaved failures never accumulate. Shortening the protected window with healthyPercent while leaving the circuit breaker on its permissive default is applying two relaxations at once and calling it speed.

BLOCKING or DEFERRED: same flag, two different risks

Criterion	BLOCKING	DEFERRED
When it turns SUCCESSFUL	After `healthyPercent`, the bake time and the cleanup of source revision tasks.	After `healthyPercent` and the bake time. Cleanup happens outside the deployment.
Source revision tasks	Already gone when the pipeline gets its green light.	May keep running; ECS tries to drain them for up to two weeks.
Capacity cost	Predictable: both revisions coexist only inside the deployment window.	Two revisions billing at once for an indeterminate period, within `maximumPercent`.
Risk you accept	A longer deployment if draining has a long tail; CI/CD tool timeouts.	Traffic served by old code after the deploy is declared successful.
Use when	This is your default. Pick it unless you have a written reason not to.	Long-lived connections, game sessions, async jobs, or task scale-in protection with short expiry.

Choosing the number with arithmetic, not faith

Three constraints close the range before any opinion enters. First: healthyPercent must be greater than or equal to the service minimumHealthyPercent. If you kept the 100% replica-service default, there is no range at all, lowering minimumHealthyPercent is a prerequisite, and it governs how many tasks the scheduler may stop during the swap, which is a capacity decision, not a speed one.

Second: the value is rounded up against the desired count. With 3 tasks at 50%, the deployment completes at 2. With 10 at 80%, at 8. On a small service each percentage point is worth a whole task, and the time saved trends to zero: earlySuccessCriteria only pays on a large fleet or on slow capacity.

Third: ECS launches at least one task on the target revision and waits for it to become healthy before evaluating the percentage. And a task only counts as healthy after passing the health checks you configured, with a load balancer that means the target group health check, and with an essential container that defines a health check, both.

From there it is a risk decision, and a conditional one. Use 90% or more when the service is stateless behind an ALB and the health check exercises a real dependency, not a /healthz that returns 200 without touching anything. Stay at 100%, or leave the feature off, when the tail of tasks carries state, a shard or a per-instance license. And if the motivation is a slow pipeline, measure first: in a rolling deployment the time usually lives in healthCheckGracePeriodSeconds, in the container health check startPeriod and in the target group deregistration delay, none of which the success rule changes.

DEFERRED has an expiry date: In DEFERRED mode, ECS tries to clean up source revision tasks for up to two weeks after success, and it cannot stop tasks with active scale-in protection. Protection caps at 2880 minutes (48 hours) per call, but it is renewable: a worker that renews its own protection in a loop keeps the old revision alive past the window and ECS gives up. Add the 3-hour CloudFormation update-stack timeout and you have two revisions billing at once, serving traffic, with nobody watching, because the deployment that created them is already marked successful. Instrument the age of source revision tasks via DescribeServices; there is no ready-made alarm for it.

What the dashboard starts saying, and what it stops saying

The feature invents no new status. The lifecycle stays IN_PROGRESS → SUCCESSFUL, the EventBridge service deployment state change events and CloudTrail records are the same, and a deployment that completed early shows up normally when you filter ListServiceDeployments by SUCCESSFUL. That is excellent for compatibility and terrible for auditing: no consumer of your event bus can tell rolled out fully apart from rolled out to 90% and handed the rest to autoscaling. If your change process needs that distinction, and under BACEN, PCI-DSS or SOX it does, the source is DescribeServiceDeployments, which returns the early success criteria configuration alongside the deployment.

There is a silent trap in the counters. While the deployment is in progress, DescribeServiceDeployments returns live counts; after it completes, it returns a snapshot from the moment of completion, in both cleanup modes. A dashboard reading task count from there will show 90 forever, even with 100 tasks running and even with the source revision still draining. The live count lives in DescribeServices, and nowhere else.

So the alarm that matters moves. Before, an EventBridge rule with eventName equal to SERVICE_DEPLOYMENT_FAILED covered most of the release risk; now it covers only the window up to healthyPercent. What deserves an alarm instead is the gap between desiredCount and runningCount on the target revision a few minutes after success, the number of task replacements per revision, and the age of source revision tasks in DEFERRED mode. None of those three comes for free.

How this will be used wrong

healthyPercent as a pipeline accelerator: dropping to 50% because the build is slow buys pipeline minutes with half the fleet launching without automatic rollback. The time was in the health check and draining timers, not in the success rule.
One value for every service via a shared Terraform module: the right number depends on desired count, capacity cost and whether the service carries state. A global healthyPercent is the definition of a setting nobody reviews.
DEFERRED combined with scale-in protection renewed in a loop: old tasks become permanent, ECS gives up after two weeks, and you find out from the invoice rather than the dashboard.
Keeping the release dashboard on DescribeServiceDeployments: after success the counters become a snapshot. The panel stays green and frozen while the tail of the fleet does whatever it wants.

What actually changed: What changed is not the duration of the deployment, it is who answers for the tail. With earlySuccessCriteria, the launch of the last tasks moves from the deployment plane, which has rollback, a circuit breaker and a failure event, to the service scaling plane, which has none of the three. It is the same choice you already make when you set an SLO: accepting known error in exchange for speed. The difference is that here the choice becomes an integer in an infrastructure-as-code file, and an integer in an IaC file does not get a risk review, it gets a syntax review.

Curation note: Today I would enable this on exactly one kind of service: inference on specialized capacity, where the tail of tasks depends on hardware availability rather than on my code, and even then with healthyPercent at 90, sourceServiceRevisionCleanup at BLOCKING, and the circuit breaker switched to a COUNT thresholdConfiguration with a low value, precisely because the protected window got shorter. On a payments service I would not enable it: the gain is pipeline minutes and the loss is automatic rollback under real load, which is exactly the control I cannot rebuild at 2 AM. The lesson that cost me dearly on financial platforms: every mechanism that unblocks the pipeline sends its bill on the first failure that only appears with production traffic, and the bill arrives on-call, not in the retro. Before touching the success criterion I measure where the deployment actually spends time, when the answer is grace period and deregistration delay, healthyPercent would have changed nothing and the risk would have been taken for free.

References

Verdict

Enable earlySuccessCriteria when the tail of your deployment depends on capacity you do not control: GPU, specialized instances, long-lived connection draining, and when the service is stateless behind a health check that exercises a real dependency. In that case: healthyPercent between 90 and 95, BLOCKING as the default, DEFERRED only where long-lived connections or short, monitored scale-in protection exist, the circuit breaker switched to COUNT to compensate for the shorter protected window, and a new alarm on the gap between desiredCount and runningCount a few minutes after success. Do not enable it as a cure for a slow pipeline: rolling deployment time lives in the health check and draining timers, and swapping the success rule only moves risk past the rollback line. The enable: false default remains right for most services, this is a well-designed exception feature, with its boundary documented out loud, not a new baseline.

Rating: Ligar por exceção, não por padrão / Enab

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

Fable 5.1 on Bedrock: data retention is now an architecture decision

Fernando Azevedo — Thu, 08 Oct 2026 17:59:06 +0000

The September 7 AWS Weekly Roundup filed Claude Fable 5.1 as one more model launch between EC2 R9g and Amazon Linux 2027. It isn't. For the first time on a frontier model in Bedrock, the key that unlocks access lives neither in IAM nor in the Marketplace, it lives in an account-scoped data retention setting that you have to flip through the API, because there is no console UI for it. I spent the week reading the data-retention and abuse-detection docs line by line, and the conclusion is uncomfortable for anyone operating under regulation: enabling Fable 5.1 is a landing-zone decision, not a product-team decision.

What actually changed, and what didn't

Bedrock has always sold itself on two security models: zero operator access (no service operator can read model input or output) and zero data retention by default. The abuse detection doc still asserts both. What Anthropic introduced with the Covered Model category is a named exception: for Claude Fable 5 and Fable 5.1, all traffic is retained for up to 30 days for automated offline abuse detection, and classifier-flagged traffic is subject to human review performed by AWS.

Two things that did not change, worth repeating because a hurried reading gets both wrong. First: your content does not go to Anthropic. The provider_data_share mode is legacy, and the doc itself states that sharing content with model providers is not supported today, it grants a permission nobody exercises. Second: nothing changes for Claude models released before Fable 5. The doc is explicit, and the mechanism is elegant: each model declares its own allowed_modes, so an account set to aws_review that also calls Opus 4.8 retains nothing from Opus: Opus permits none, and none wins for it.

The friction isn't what AWS does with the data. It's that the authorization to do it lives in a scope most teams don't control.

How the effective mode resolves and where the model gets blocked

Resolution is hierarchical: the first scope whose value is not inherit wins. If the effective mode lands below the model's minimum, the model shows up as status: "unavailable" in the model list, not a permission error, but the model quietly leaving your catalog.

🔐 Governança, o que a organização permite

SCP Deny em bedrock:DataRetentionMode (security)
IAM PutAccountDataRetention · UpdateProject (security)

🟧 AWS, resolução do modo efetivo

Projeto mais específico · pode ser inherit (compute)
Conta PUT /v1/data_retention (compute)
Padrão do modelo somente leitura (data)

🤖 Bedrock, a trava de disponibilidade

allowed_modes none < default < aws_review < provider_data_share (ai)
Claude Fable 5.1 exige aws_review ou superior (ai)
status: unavailable status_reason explica o motivo (external)

📦 Fronteira de retenção: 30 dias dentro da AWS

Prompts e saídas até 30 dias · região de destino do CRIS (storage)
Detecção de abuso offline automatizada · todo o tráfego (security)
Revisão humana pela AWS só tráfego marcado por classificador (security)
Enterprise Frontier Safeguards ZDR até 31/12/2026 (security)

Flows

scp -> acct: caps which modes can be written
iam -> proj: who may change the scope
proj -> acct: if inherit, defer
acct -> mdef: if inherit, defer
acct -> gate: effective mode
app -> gate: inference call
gate -> fable: mode ≥ aws_review
gate -> unavail: mode none or default
fable -> store: writes input and output
store -> auto: scans full volume
auto -> human: escalates classifier hits
efs -> store: suspends retention while eligible

The four modes, and what each one costs in governance

Mode	What happens to the data	Does Fable 5.1 work?
`none`	Zero retention. Nothing is written to durable storage. On the Responses API, `store=true` is rejected.	No, the request is blocked.
`default`	The model's own policy applies. If ZDR applied before, it still does. `store=false` does not guarantee zero retention.	No, it shows up as `unavailable`.
`aws_review`	Retained up to 30 days inside the AWS boundary; AWS may review. The provider receives nothing.	Yes, this is the minimum required.
`provider_data_share` (legacy)	In practice identical handling to `aws_review`: provider sharing is not supported today.	Yes, it sits higher in the ordering and already satisfies it.
`inherit`	No opinion at that scope; defers to a broader one. It is the default for new accounts and projects.	Depends on what sits above.

The account became the isolation boundary, and your SCP already said no

If you work under BACEN 4.893, PCI-DSS or LGPD, odds are your organization already has an SCP denying anything that writes prompts to durable storage. The Bedrock docs even publish that SCP as an example, denying bedrock-mantle:PutAccountDataRetention whenever bedrock-mantle:DataRetentionMode is anything other than none. That SCP is correct, and it makes Fable 5.1 permanently unavailable across the whole OU.

What bothers me isn't the block; it's the granularity. The mode resolves across project → account → model default, and the only practical command today is aws bedrock put-account-data-retention --mode aws_review, which applies to the entire account. Turning Fable 5.1 on in the shared production account authorizes AWS human review over every workload that runs on a model requiring review, indefinitely, until someone remembers to turn it off. Project scope exists precisely to avoid that, and it is the only reason I would consider enabling this without cutting a new account.

The practical recommendation inverts the instinct: don't ask for an SCP exception, ask for an account. A dedicated account, inside its own OU with an SCP that permits aws_review and nothing beyond it, gives you a named blast radius, a separate bill, and a one-line audit answer. An exception carved into a production OU's SCP is debt nobody audits later.

Data residency: outside the US, the Global profile is not a choice

This is the part every piece of coverage I read missed, and it is the most serious one for anyone operating in Brazil or Europe. The Fable 5.1 model card lists the bedrock-runtime endpoint with In-Region unsupported in every Region on earth. The Geo profile (us.anthropic.claude-fable-5-1) exists only in the US, Canada and GovCloud. In sa-east-1, eu-central-1, eu-west-1 and everywhere else, the only open door is the Global profile: global.anthropic.claude-fable-5-1, which routes inference anywhere in the world.

Now pair that with a sentence in the abuse detection doc: if cross-region inference is enabled, retained inputs and outputs are stored in the destination Region, meaning wherever the request was processed. Put the two together and the result is blunt: a bank in São Paulo calling Fable 5.1 is authorizing 30-day retention of prompts in a Region it did not pick and cannot predict per request. That is not a latency footnote, it is a data-residency assertion that belongs in your DPIA before any proof of concept.

For residency-constrained shops the options are short and none are free: run through the US Geo profile and classify the data as transferred, negotiate Enterprise Frontier Safeguards eligibility while it still grants ZDR, or accept that Fable 5.1 stays outside the regulated perimeter and serves workloads with no personal data, code generation over an internal repository, for instance.

The playbook for tomorrow morning

Find out which mode your accounts are already in: Before discussing anything, run GET /v1/data_retention on every account with Bedrock enabled. Older accounts sitting on provider_data_share are already above aws_review in the ordering, they gain Fable 5.1 access without anyone having approved anything. That finding usually changes the meeting.
Pick the scope before the mode: Project beats account, account beats model default. If you need a pilot, set data_retention on the project through POST /v1/organization/projects/{project_id} and leave the account on inherit or none. Never start at the account.
Write the SCP before flipping the mode: The bedrock:DataRetentionMode condition key (and bedrock-mantle:DataRetentionMode) exists for exactly this. An SCP that permits aws_review and denies provider_data_share stops anyone from escalating back to the legacy mode out of habit.
Pin the inference profile in code: There is no In-Region endpoint. Explicitly choose between us.anthropic.claude-fable-5-1 and global.anthropic.claude-fable-5-1 and document the reason next to the data that flows through it. Leaving that to an ownerless environment variable is how global routing reaches production unreviewed.
Fix your sampling parameters before the first deploy: temperature must be 1.0 or unset, top_p 0.99 or unset, the two cannot be sent together, and top_k is unsupported. A wrapper that copies your Sonnet parameters breaks on the first call, and it breaks in production, not in the unit test that mocks the client.
Treat stop_reason: "refusal" as a primary path: The model returns HTTP 200 with stop_reason: "refusal" and a stop_details object carrying the category. The docs warn that refusal rates are materially higher than on previous Claude models. Code that only inspects the HTTP status will hand the user an empty response and log nothing.
Put December 31, 2026 on the calendar: Enterprise Frontier Safeguards makes eligible customers ZDR through December 31, 2026. After that, all traffic returns to being retained for up to 30 days for automated offline abuse detection. If your compliance sign-off depends on ZDR, it has an expiry date.

The cache checkpoint that fails silently: Fable 5.1 prompt caching accepts at most 4 checkpoints per request, with a 5-minute or 1-hour TTL, across the system, messages and tools fields. The expensive detail: the minimum is 512 tokens per checkpoint, and below that caching is disabled with no error at all, the response comes back normal, with cache counters at zero. In an agent that runs for hours, that is the difference between reusing your system prompt and paying for it at every step. Instrument cacheReadInputTokens and cacheWriteInputTokens as first-class CloudWatch metrics, not as log fields: with no alarm, a refactor that shortened the system prompt by 40 tokens disappears into the noise and surfaces only on the invoice.

The real cost: adaptive thinking, billed refusals, and an endpoint without guardrails

Fable 5.1's adaptive reasoning is always on and cannot be disabled; what you control is the effort level: low, medium, high, xhigh, max, defaulting to high. A public Converse test with {"output_config":{"effort":"max"}} showed the jump: 410 thinking tokens against zero on the default call, with latency moving from ~8.3s to ~13.7s. Thinking tokens are billed output tokens. A miscalibrated effort inside an agent loop doesn't raise your bill by 10%, it raises it by multiples, and the only evidence is a usage block nobody aggregates.

The second cost leak is refusal. Prompt-stage refusals, blocked before inference begins, are not billed. Mid-stream refusals, after partial output, are billed for the tokens generated before the block. With a materially higher refusal rate coming from the cybersecurity and life-sciences classifiers, a vulnerability-analysis pipeline can pay for output the user never saw.

And there is an endpoint choice that reshapes the whole architecture: on bedrock-mantle, Fable 5.1 supports neither Guardrails nor Knowledge Bases nor Agents nor the Converse API: only streaming, prompt caching and token counting. Bedrock Guardrails exist only through bedrock-runtime. If you need your own content filter in front of the model, there is no decision to make; bedrock-runtime is the only path, and it's the one the docs recommend for new applications anyway.

Anti-patterns already taking root in week one

Flipping aws_review on the shared production account: the mode is account-scoped, not per request. One CLI line authorizes retention and review over every future workload on any model requiring review, including workloads not yet written.
Treating legacy provider_data_share as harmless because nothing is shared: it is the most permissive mode in the ordering. An account sitting on it since 2024 unlocks Covered Models with nobody having reviewed the decision.
Assuming global. is just a latency optimization: outside the US and Canada it is the only option, and it decides which Region holds your prompts for 30 days. That is a residency decision disguised as a model ID prefix.
Reusing your Sonnet client with temperature: 0.2 and top_k: Fable 5.1 requires temperature at 1.0 or unset and does not support top_k. Determinism through low temperature is not a lever here; use output structure and validation instead.
Counting on Guardrails at the bedrock-mantle endpoint: they do not exist there. Discovering it after the security design has been signed off costs a redesign, not a patch.
Measuring cost without separating thinking tokens: reasoning cannot be turned off and the default is high. Without effort and usage as dimensions on your cost metric, you cannot explain the bill even a day later.

Questions that came up in week one

My account is on `provider_data_share`. Do I need to migrate to use Fable 5.1?

No. Since provider_data_share sits above aws_review in the ordering, it already satisfies the requirement and nothing breaks. It is still worth migrating: aws_review describes the behavior that actually happens, and in an audit report the gap between "we authorized sharing with the provider" and "we authorized review by AWS" is the gap between a question and a meeting.

Can I use Fable 5.1 and keep zero data retention?

Only by exception. The Enterprise Frontier Safeguards program grants eligible customers ZDR through December 31, 2026, and the roadmap mentions customer-managed keys and bring-your-own-bucket storage for abuse detection. Outside it, ZDR eligibility for Claude models is evaluated per account and per model, and Anthropic is the decision maker, talk to their representative, not only to AWS.

Where do I configure this in the console?

Nowhere. The docs state explicitly that at launch there is no console UI for data retention: API or SDK only. In practice that's a feature: it forces the change through infrastructure as code and leaves a CloudTrail trail, instead of becoming one click by whoever has console access.

Does turning on `aws_review` make my other models start retaining data?

No. Each model declares its own allowed_modes, and a model that permits none: Opus 4.8, for instance, retains nothing no matter how permissive the account setting is. The mode states what you allow, not what happens. Even so, isolate by project: the audit will ask what you permitted, not what occurred.

What I would do on Monday: I would not carve an exception into the production OU's SCP: I would open an account. A dedicated account, its own OU, an SCP that permits exactly aws_review and nothing above it, global.anthropic.claude-fable-5-1 pinned in code with a comment saying why, and no customer data crossing into it. The hard-won lesson behind that came from elsewhere: every exception I ever carved into a production OU policy outlived the project that justified it and became an orphan permission nobody could explain two years later. An account is revocable, carries its own bill, and answers the audit in one line. And before any pilot I would run GET /v1/data_retention across every account with Bedrock enabled: I'd bet at least one has been on provider_data_share forever, holding Covered Model access nobody approved.

Verdict

Fable 5.1 delivers on the axis that matters for long-horizon agents: a 1M-token context window, 128K max output, a June 2026 knowledge cutoff, and reasoning effort configurable across five levels. If your workload is code generation and review over an internal repository with no personal data, adopt it, in a dedicated account, with the project set to aws_review, the inference profile pinned, and stop_reason: "refusal" handled as a normal path. If your workload touches customer data under LGPD or BACEN and you sit outside the US, the verdict inverts, and it's a condition rather than an opinion: the only available profile is Global, the 30-day retention lands in whichever Region processed the request, and that belongs in a signed impact assessment before the first call. It isn't the model that is unavailable to regulated environments, it's the retention mode that doesn't yet have the granularity regulated environments require. Enterprise Frontier Safeguards is the bridge, and it has an expiry date: December 31, 2026.

Rating: Adote com escopo isolado, bloqueado por

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.