<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Fernando Azevedo</title>
    <description>The latest articles on DEV Community by Fernando Azevedo (@fernando_azevedo_6844e930).</description>
    <link>https://dev.to/fernando_azevedo_6844e930</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1787949%2F6b76fcdd-ad42-4051-833f-22554f10309f.jpg</url>
      <title>DEV Community: Fernando Azevedo</title>
      <link>https://dev.to/fernando_azevedo_6844e930</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/fernando_azevedo_6844e930"/>
    <language>en</language>
    <item>
      <title>Air-gapped vault meets EBS Archive: pre-mortem of a 72-hour RTO</title>
      <dc:creator>Fernando Azevedo</dc:creator>
      <pubDate>Fri, 13 Nov 2026 18:37:03 +0000</pubDate>
      <link>https://dev.to/fernando_azevedo_6844e930/air-gapped-vault-meets-ebs-archive-pre-mortem-of-a-72-hour-rto-5efj</link>
      <guid>https://dev.to/fernando_azevedo_6844e930/air-gapped-vault-meets-ebs-archive-pre-mortem-of-a-72-hour-rto-5efj</guid>
      <description>&lt;p&gt;On November 12, 2026, AWS started letting EBS backups held in a logically air-gapped vault move down to Amazon EBS Snapshots Archive. The cost argument sells itself: archive costs US$ 0.0125 per GB-month against US$ 0.05 for the standard EBS snapshot tier. The question I care about isn't how much this saves, it's what happens to your RTO on the day the only clean recovery point sits in archive. To answer it, I wrote the postmortem before the incident.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;How to read this:&lt;/strong&gt; This is a pre-mortem: a game day scenario built from limits AWS documents, not a customer or employer incident. The timeline timestamps are exercise assumptions. The limits (72-hour restore, 5 concurrent restores per account, 90-day archive minimum, no copy after the transition) come from the documentation cited at the end.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  What happened in the scenario
&lt;/h2&gt;

&lt;p&gt;The system is a settlement core in &lt;code&gt;us-east-1&lt;/code&gt;: 12 EBS volumes, 18 TB combined, encrypted with a customer managed key. Protection had two layers. &lt;strong&gt;Daily layer:&lt;/strong&gt; an AWS Backup rule to a standard vault, 35-day retention, no Vault Lock. &lt;strong&gt;Monthly layer:&lt;/strong&gt; a rule copying into a logically air-gapped vault, archive enabled after 30 days in warm storage, and 7-year total retention because the regulator asks for long history.&lt;/p&gt;

&lt;p&gt;The platform team did what the announcement suggests and turned on archive to cut cost. The copy to &lt;code&gt;us-west-2&lt;/code&gt;, where the isolated recovery environment (the clean room) lived, wasn't in the rule: it was a quarterly job copying points from the LAG vault into a second LAG vault in the other Region. On the retention spreadsheet it looked equivalent.&lt;/p&gt;

&lt;p&gt;The attacker got in with administrative credentials for the primary organization and stayed quiet for about four months before encrypting. When encryption started, all 35 days of daily points were already tainted, and the standard vault, with no lock, had its points deleted. The LAG vault did what it promises: compliance mode, nobody deletes, not even root. The most recent clean point was the monthly one from four months earlier. And it had been in archive since day 31.&lt;/p&gt;

&lt;p&gt;Protection worked: the data survived. Recovery failed, because the design never asked how long it takes to bring back a point older than 30 days.&lt;/p&gt;

&lt;h2&gt;
  
  
  Exercise timeline
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;T+0h: detection&lt;/strong&gt;: Settlement core volumes encrypted. CloudTrail shows &lt;code&gt;DeleteRecoveryPoint&lt;/code&gt; succeeding on the standard vault and denied on the LAG vault.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;T+0h45: primary organization declared compromised&lt;/strong&gt;: The team stops trusting any identity in the primary organization and moves to the recovery account in a second organization.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;T+2h: restore access vault approved&lt;/strong&gt;: The Multi-party approval team reaches its threshold and grants the restore access backup vault in the recovery account. It worked as designed.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;T+3h: clean point identified&lt;/strong&gt;: Forensics points to the monthly backup from four months ago as the last clean point. Status: archived.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;T+3h10: the copy to the clean room fails&lt;/strong&gt;: An EBS backup that has moved to archive can't be copied to another vault or Region, including from the restore access vault. The clean room in &lt;code&gt;us-west-2&lt;/code&gt; is useless for that point.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;T+4h: archive restore started in us-east-1&lt;/strong&gt;: Only 5 concurrent restores per account. The 12 volumes become 3 waves, and none can be cancelled once started.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;T+24h: RTO breached&lt;/strong&gt;: The agreed RTO was 24 hours. At the documented worst case, up to 72 hours per wave, the last volume comes back near T+220h, almost nine days later.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Root cause:&lt;/strong&gt; The root cause wasn't the attack, it was a retention design that treated archive as a cost tier when it's also an RTO tier. Three decisions compounded: the cross-Region copy was planned from the LAG vault after the transition, when it's no longer possible; nobody mapped RTO to the age of the point; and the 5-concurrent-restores-per-account quota never made it into the runbook.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Why archive is an RTO decision
&lt;/h2&gt;

&lt;p&gt;An archived point doesn't behave like a snapshot. The EBS documentation lists the failure modes, and each one moves recovery time.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Slow, one-way restore:&lt;/strong&gt; bringing a snapshot from archive back to the standard tier takes up to 72 hours depending on size, and the process can't be cancelled once started. Picked the wrong point? You pay for the full restore and wait again.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;No direct use:&lt;/strong&gt; an archived snapshot can't create a volume, can't go into a block device mapping, can't be shared or copied. Fast snapshot restore is disabled on transition and has to be re-enabled by hand afterwards.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Per-account quotas:&lt;/strong&gt; the default is 5 concurrent archive restores and 25 archived snapshots per volume. With 12 volumes, the first number turns 72 hours into up to 216. The second deserves attention in a 7-year monthly plan, which produces 84 points per volume. I couldn't find in the LAG vault documentation whether that quota applies to points held in the AWS Backup service account. Confirm with Support before assuming.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Full snapshot:&lt;/strong&gt; the archived point is always full, not incremental. 18 TB archived means 18 TB restored, billed at US$ 0.03 per GB.&lt;/p&gt;

&lt;p&gt;The lesson is about engineering, not price: every storage tier needs a written RTO next to it. A warm point comes back in hours. An archived point comes back in days, and only in the Region and account where it already lives.&lt;/p&gt;

&lt;h2&gt;
  
  
  The three numbers that change the runbook
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;US$ 0,0125&lt;/strong&gt;: per GB-month in archive. Against US$ 0.05 on the standard EBS tier: up to 75% less, for points kept 90 days or longer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;72 h&lt;/strong&gt;: archive restore. Documented ceiling to return to the standard tier. Can't be cancelled.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;5&lt;/strong&gt;: concurrent restores per account. Default quota. 12 volumes become 3 sequential waves.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Where the copy must happen: before the transition
&lt;/h2&gt;

&lt;p&gt;The diagram answers one question: at which point of the lifecycle the backup can still leave the Region. The cross-Region copy starts from the warm tier; after archive, the point is only restored where it is. Dashed arrows are asynchronous: the lifecycle transition and the slow archive restore. Simplified: it omits the daily standard vault and the KMS key.&lt;/p&gt;

&lt;h3&gt;
  
  
  Workload account
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Amazon EBS 12 volumes, 18 TB (storage)&lt;/li&gt;
&lt;li&gt;AWS Backup plan monthly rule (compute)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Logically air-gapped vaults
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;LAG vault us-east-1 warm storage (security)&lt;/li&gt;
&lt;li&gt;EBS Snapshots Archive up to 72h restore (storage)&lt;/li&gt;
&lt;li&gt;LAG vault us-west-2 warm copy (security)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Recovery organization
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Multi-party approval approval team (user)&lt;/li&gt;
&lt;li&gt;Restore access vault recovery account (security)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Flows
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;plan -&amp;gt; ebs: snapshots&lt;/li&gt;
&lt;li&gt;plan -&amp;gt; warm: copy action&lt;/li&gt;
&lt;li&gt;warm -&amp;gt; dr: cross-Region copy, warm only&lt;/li&gt;
&lt;li&gt;warm -&amp;gt; archive: lifecycle after 30 days&lt;/li&gt;
&lt;li&gt;mpa -&amp;gt; rav: approves access&lt;/li&gt;
&lt;li&gt;rav -&amp;gt; warm: restore in hours&lt;/li&gt;
&lt;li&gt;rav -&amp;gt; archive: restore in up to 72h&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Remediation: retention in RTO bands
&lt;/h2&gt;

&lt;p&gt;The fix isn't turning archive off, it's to stop treating retention as a single line.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Copy before the transition:&lt;/strong&gt; the copy to the &lt;code&gt;us-west-2&lt;/code&gt; LAG vault becomes a copy action in the monthly rule itself, with the copy lifecycle keeping the point warm. The documentation is explicit: to keep a copy in another account or Region, copy while the backup is still in warm storage. A quarterly job from the LAG vault is exactly what doesn't work.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Bands with a written RTO:&lt;/strong&gt; the last 3 monthly points stay warm in both Regions (RTO in hours). From month 4 on they go to archive (RTO in days, only in &lt;code&gt;us-east-1&lt;/code&gt;). That table goes into the continuity document the business signs, not into the platform team's heads.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Lifecycle constraints:&lt;/strong&gt; the rule must run monthly or less often. Total retention must be at least 90 days longer than time in warm and fall within the vault's minimum and maximum retention. A vault created with a 35-day maximum rejects a 7-year copy and nothing on the spreadsheet warns you; check &lt;code&gt;MaxRetentionDays&lt;/code&gt; first.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Key:&lt;/strong&gt; a volume encrypted with the AWS managed key (&lt;code&gt;aws/ebs&lt;/code&gt;) can't be shared across accounts, so it can't go into the LAG vault. Use a customer managed key with a key policy that allows AWS Backup.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Quota and invitation:&lt;/strong&gt; request the concurrent restore quota increase before the incident. RAM sharing accepts individual account IDs only, and the invitation expires in 12 hours: accept it during the game day, not on attack day.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cost: run the whole math, not just storage
&lt;/h2&gt;

&lt;p&gt;Archive saves real money, but the math has four lines, not one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Full versus incremental:&lt;/strong&gt; on the standard tier you pay only for changed blocks; in archive you pay for the whole volume. From the price ratio (0.0125 / 0.05), an archived full breaks even with a standard incremental that's 25% of the volume size. A volume with little monthly change can cost more archived. That's why the documentation advises against archiving daily incrementals.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Referenced blocks stay:&lt;/strong&gt; data that other snapshots in the lineage still use remains on the standard tier and is billed to the next snapshot. Archiving doesn't zero the standard tier bill.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Restore and early exit:&lt;/strong&gt; restoring the scenario's 18 TB (18,432 GB) costs about US$ 553 each time. Deleting or permanently restoring before 90 days triggers a pro-rated charge for the remaining days. A temporary restore doesn't, but you pay for both tiers during the restore period.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;LAG vault pricing:&lt;/strong&gt; LAG vault storage is billed as AWS Backup, not EBS, and has its own rate table. The pricing page example uses US$ 0.0575 per GB-month warm in Ireland. Check your Region in the calculator before reusing EBS numbers.&lt;/p&gt;

&lt;p&gt;To observe it, filter CUR by the usage types &lt;code&gt;SnapshotArchiveStorage&lt;/code&gt;, &lt;code&gt;SnapshotArchiveRetrieval&lt;/code&gt; and &lt;code&gt;SnapshotArchiveEarlyDelete&lt;/code&gt;. The third one above zero almost always points to a misconfigured lifecycle.&lt;/p&gt;

&lt;h2&gt;
  
  
  Anti-patterns the scenario exposed
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Cross-Region copy after archive:&lt;/strong&gt; planning the second Region as a later job from the LAG vault. After the transition the point can't leave the Region, not even through the restore access vault.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One RTO for the whole retention:&lt;/strong&gt; promising 24 hours for any point across 7 years of history. An archived point has a different RTO, and the business has to sign off on both.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Standard vault without Vault Lock as first line:&lt;/strong&gt; the daily layer was deleted by whoever held admin credentials. Only the compliance-mode vault survived.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Archiving daily incrementals:&lt;/strong&gt; every point becomes full in archive. With a low change rate, the bill goes up instead of down.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Quota discovered during the incident:&lt;/strong&gt; 5 concurrent restores is the default. Requesting an increase with the system down adds days to the RTO.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Pillar reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;security&lt;/strong&gt;: LAG vault in compliance mode, customer managed key and Multi-party approval in a second organization. Approval teams only exist in &lt;code&gt;us-east-1&lt;/code&gt;: that's a Region dependency of your recovery plan and has to be written down.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;reliability&lt;/strong&gt;: Write RTO per point-age band (warm in hours, archive in days) and test both with restore testing from the restore access vault. Cross-Region copy always as a copy action in the rule, while the point is warm.&lt;/li&gt;
&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Curator's note:&lt;/strong&gt; I'd turn on archive in the LAG vault, but only after two things: the cross-Region copy inside the monthly rule and an RTO-per-band table signed by the business owner. I've seen DR plans approved because the retention was right while nobody had measured the time to come back. That isn't a tool failure, it's a question failure. The first test I'd run is a real, timed restore of an archived point, with all 5 concurrent restores busy.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Lessons
&lt;/h2&gt;

&lt;p&gt;Use archive in the logically air-gapped vault when: the point is monthly or sparser, kept well beyond 90 days, exists for retention requirements rather than as an operational recovery line, and the business has accepted in writing a days-long RTO for it. Keep warm, and copied to the second Region before the transition, every point that could be the last clean one after an attack with months of dwell time. The feature is good for long regulatory retention; as the only line of defense against ransomware, it trades cost for an RTO nobody probably approved.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Rating:&lt;/strong&gt; Adopt with RTO bands&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/about-aws/whats-new/2026/11/aws-backup-lag-vault-ebs-archive/" rel="noopener noreferrer"&gt;AWS What's New: AWS Backup adds logically air-gapped vault support for Amazon EBS Snapshots Archive (12/11/2026)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/aws-backup/latest/devguide/logicallyairgappedvault.html" rel="noopener noreferrer"&gt;AWS Backup Developer Guide: Logically air-gapped vault&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/aws-backup/latest/devguide/multipartyapproval.html" rel="noopener noreferrer"&gt;AWS Backup Developer Guide: Multi-party approval for logically air-gapped vaults&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/ebs/latest/userguide/snapshot-archive.html" rel="noopener noreferrer"&gt;Amazon EBS User Guide: Archive Amazon EBS snapshots (quotas)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/ebs/latest/userguide/snapshot-archive-considerations.html" rel="noopener noreferrer"&gt;Amazon EBS User Guide: Considerations and limitations for archiving snapshots&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/ebs/latest/userguide/snapshot-archive-pricing.html" rel="noopener noreferrer"&gt;Amazon EBS User Guide: Pricing and billing for archiving snapshots&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/blogs/storage/streamline-ransomware-protection-while-reducing-cost-with-logically-air-gapped-vault-primary-backup/" rel="noopener noreferrer"&gt;AWS Storage Blog: Streamline ransomware protection while reducing cost with logically air-gapped vault primary backup&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/blogs/architecture/cyber-resilience-on-aws-a-reference-approach-for-recovery-from-ransomware-and-destructive-events/" rel="noopener noreferrer"&gt;AWS Architecture Blog: Cyber resilience on AWS, a reference approach for recovery from ransomware and destructive events&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://fernando.moretes.com/blog/cofre-air-gapped-com-ebs-archive-o-pre-mortem-de-um-rto-de-72-horas" rel="noopener noreferrer"&gt;fernando.moretes.com&lt;/a&gt;. By Fernando F. Azevedo: Senior Solutions Architect.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>securityresilience</category>
      <category>awsbackup</category>
      <category>ebs</category>
      <category>ransomware</category>
    </item>
    <item>
      <title>Glue native SaaS connectors vs AppFlow, zero-ETL and custom code</title>
      <dc:creator>Fernando Azevedo</dc:creator>
      <pubDate>Tue, 10 Nov 2026 18:46:04 +0000</pubDate>
      <link>https://dev.to/fernando_azevedo_6844e930/glue-native-saas-connectors-vs-appflow-zero-etl-and-custom-code-5e3k</link>
      <guid>https://dev.to/fernando_azevedo_6844e930/glue-native-saas-connectors-vs-appflow-zero-etl-and-custom-code-5e3k</guid>
      <description>&lt;p&gt;On November 9, 2026, AWS announced 100 new native connectors in AWS Glue for SaaS applications and REST APIs: payments (Adyen, Square, Checkout.com), accounting (Xero, QuickBooks), billing (Chargebee, Recurly, Zuora), operations (PagerDuty, Sentry, Jenkins, Vercel) and even usage and cost data from Anthropic. You create the connection once in the Glue Data Catalog and any job can reuse it under the same Glue permissions model, on Glue 5.0 and later, in every commercial Region. I've written and maintained acquirer API extractors by hand. They never broke at deploy time. They broke the day the vendor changed pagination and nobody noticed for a week. So this piece doesn't ask whether the connector works. It asks which of the four paths available today costs the least to maintain.&lt;/p&gt;

&lt;h2&gt;
  
  
  The real choice: four paths to get data out of a SaaS
&lt;/h2&gt;

&lt;p&gt;When a finance team asks for "Adyen transactions in the lake by tomorrow", there are four answers on AWS today.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Glue native connector:&lt;/strong&gt; a Spark job on Glue 5.0 reads the entity through the SaaS &lt;code&gt;connection_type&lt;/code&gt;, with &lt;code&gt;ENTITY_NAME&lt;/code&gt;, &lt;code&gt;SELECTED_FIELDS&lt;/code&gt;, &lt;code&gt;FILTER_PREDICATE&lt;/code&gt; in Spark SQL, and partitioning through &lt;code&gt;PARTITION_FIELD&lt;/code&gt;, &lt;code&gt;LOWER_BOUND&lt;/code&gt;, &lt;code&gt;UPPER_BOUND&lt;/code&gt; and &lt;code&gt;NUM_PARTITIONS&lt;/code&gt;. The secret lives in Secrets Manager, one per connection.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Glue zero-ETL:&lt;/strong&gt; managed replication, no job, into Redshift, S3 or S3 Tables through the SageMaker lakehouse architecture. The problem is coverage: in the current documentation, the SaaS sources are Salesforce, Salesforce Marketing Cloud Account Engagement, SAP OData, ServiceNow, Zendesk, Zoho CRM, Facebook Ads and Instagram Ads. None of the payment connectors in the announcement are on that list.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Amazon AppFlow:&lt;/strong&gt; a managed flow billed per run and per GB processed, good for scheduled copies with no transformation.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Custom extractor:&lt;/strong&gt; Lambda or Step Functions calling the API and writing to S3. Full control, and full debt.&lt;/p&gt;

&lt;p&gt;The announcement shifts the balance because the first path now covers the long tail that only the fourth used to cover. For sources outside the 100, there is still the Glue REST connector (&lt;code&gt;RegisterConnectionType&lt;/code&gt;), with OAuth2 client credentials, authorization code, JWT, basic or custom auth, and cursor or offset pagination. So even the "my SaaS isn't on the list" case has a declarative way out before you reach for code.&lt;/p&gt;

&lt;h2&gt;
  
  
  The four paths side by side
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Criterion&lt;/th&gt;
&lt;th&gt;Glue native connector&lt;/th&gt;
&lt;th&gt;Glue zero-ETL&lt;/th&gt;
&lt;th&gt;AppFlow&lt;/th&gt;
&lt;th&gt;Custom extractor&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;SaaS coverage&lt;/td&gt;
&lt;td&gt;100 new + earlier ones; REST for the rest&lt;/td&gt;
&lt;td&gt;8 SaaS applications today&lt;/td&gt;
&lt;td&gt;Its own connector catalog&lt;/td&gt;
&lt;td&gt;Any API&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Transformation at ingestion&lt;/td&gt;
&lt;td&gt;Full Spark, &lt;code&gt;QUERY&lt;/code&gt; and &lt;code&gt;FILTER_PREDICATE&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;None, it replicates&lt;/td&gt;
&lt;td&gt;Simple mapping and filtering&lt;/td&gt;
&lt;td&gt;Whatever you write&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cost model&lt;/td&gt;
&lt;td&gt;US$0.44/DPU-hour (Flex: US$0.29)&lt;/td&gt;
&lt;td&gt;By ingested volume, 1 MB minimum per request&lt;/td&gt;
&lt;td&gt;US$0.001/run + US$0.02/GB&lt;/td&gt;
&lt;td&gt;Lambda cents + engineering hours&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Incremental&lt;/td&gt;
&lt;td&gt;You own the watermark via filter&lt;/td&gt;
&lt;td&gt;Managed&lt;/td&gt;
&lt;td&gt;Managed where the connector supports it&lt;/td&gt;
&lt;td&gt;You write and test it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Who absorbs API changes&lt;/td&gt;
&lt;td&gt;AWS (connector); you (schema contract)&lt;/td&gt;
&lt;td&gt;AWS&lt;/td&gt;
&lt;td&gt;AWS&lt;/td&gt;
&lt;td&gt;Your on-call&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  The incremental load is yours, and it's where payments break
&lt;/h2&gt;

&lt;p&gt;The Stripe connector documentation spells out a detail that applies to much of the long tail: only field-based partitioning exists. There's no record-based partitioning (the connector can't retrieve a total count) and no ID-based partitioning, because the primary key is a string. In practice, parallelism and incremental loads depend on a time field you pick, and owning the watermark is your job.&lt;/p&gt;

&lt;p&gt;The design I use is simple. A DynamoDB table with partition key &lt;code&gt;source#entity&lt;/code&gt; stores the last processed &lt;code&gt;created&lt;/code&gt;. The job reads that value, builds &lt;code&gt;FILTER_PREDICATE&lt;/code&gt; with a 15 to 30 minute overlap window backwards, sets &lt;code&gt;LOWER_BOUND&lt;/code&gt; and &lt;code&gt;UPPER_BOUND&lt;/code&gt; on the same field and writes into an Iceberg staging table. A &lt;code&gt;MERGE&lt;/code&gt; on the transaction's natural key (the acquirer's payment ID) makes reruns idempotent. Only after the commit does the job advance the watermark, with a &lt;code&gt;ConditionExpression&lt;/code&gt; on the previous value so two concurrent runs can't both move the pointer.&lt;/p&gt;

&lt;p&gt;The overlap exists because payment APIs aren't append-only. Refunds, chargebacks and status changes rewrite old records, and an &lt;code&gt;updated&lt;/code&gt; field isn't always available as a filter. Without the window, you lose the update. Without the &lt;code&gt;MERGE&lt;/code&gt;, you duplicate the transaction. Both bugs look the same: reconciliation against the bank statement doesn't close, and someone finds out at month-end close, not on the day.&lt;/p&gt;

&lt;h2&gt;
  
  
  Payment SaaS ingestion with a native connector and your own watermark
&lt;/h2&gt;

&lt;p&gt;The connector handles auth and pagination; the watermark, the idempotent MERGE and publishing to Redshift are still yours. The dashed line is the watermark update, which only happens after the Iceberg commit. Simplified: one SaaS, one entity, no Lake Formation in the picture.&lt;/p&gt;

&lt;h3&gt;
  
  
  SaaS
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Adyen API REST, paginated (external)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Control plane
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Secrets Manager one secret per connection (security)&lt;/li&gt;
&lt;li&gt;Glue connection Data Catalog (data)&lt;/li&gt;
&lt;li&gt;Watermark DynamoDB source#entity (data)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Lake
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Glue 5.0 job FILTER_PREDICATE (compute)&lt;/li&gt;
&lt;li&gt;S3 Iceberg MERGE by payment ID (storage)&lt;/li&gt;
&lt;li&gt;Redshift reconciliation (data)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Flows
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;conn -&amp;gt; secret: reads credential&lt;/li&gt;
&lt;li&gt;job -&amp;gt; conn: uses connection&lt;/li&gt;
&lt;li&gt;job -&amp;gt; wm: reads watermark&lt;/li&gt;
&lt;li&gt;job -&amp;gt; saas: HTTPS, time window&lt;/li&gt;
&lt;li&gt;job -&amp;gt; s3: idempotent MERGE&lt;/li&gt;
&lt;li&gt;job -&amp;gt; wm: advances after commit&lt;/li&gt;
&lt;li&gt;s3 -&amp;gt; rs: publishes&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Cost: the DPU is the cheap part
&lt;/h2&gt;

&lt;p&gt;Let me run the math for a payment source pulled hourly. A job with 2 G.1X workers (2 DPU) running 10 minutes consumes about 0.33 DPU-hours: roughly US$0.15 per run at US$0.44, or close to US$105 a month at 24 runs a day. On Flex, at US$0.29, it drops to about US$70, at the cost of a non-guaranteed start. The same volume on AppFlow, with 720 runs and 1 GB per run, lands around US$15: US$0.72 in runs and US$14.40 in data.&lt;/p&gt;

&lt;p&gt;Looking only at the bill, AppFlow wins. But the bill isn't the cost. AppFlow charges for runs that found no new data and bills runs that failed on a misconfiguration as successful, and it doesn't transform: someone still has to deduplicate, type and &lt;code&gt;MERGE&lt;/code&gt; afterwards. If that step is already a Glue job, you pay the DPU anyway and gain a second tool to operate.&lt;/p&gt;

&lt;p&gt;The custom extractor has the lowest bill of all, Lambda cents. Its cost shows up elsewhere: credential rotation, API version changes, a new rate limit, a renamed field. On platforms I've maintained, one acquirer extractor ate something like two to three engineering days per quarter just to keep working, and that number never made it into the business case.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Rule I use:&lt;/strong&gt; if the cadence is hourly or slower, start on Flex and move to standard only when the freshness SLO requires it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Security and governance: what changes on a regulated platform
&lt;/h2&gt;

&lt;p&gt;The most concrete governance gain is the connection as a Data Catalog resource. Before, every extractor had its own way of storing an API key, almost always an environment variable someone copied across accounts. Now there's one secret per connection in Secrets Manager, and access goes through IAM.&lt;/p&gt;

&lt;p&gt;Three adjustments I make before releasing to production:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Role scope:&lt;/strong&gt; the documentation example grants &lt;code&gt;secretsmanager:GetSecretValue&lt;/code&gt;, &lt;code&gt;PutSecretValue&lt;/code&gt; and the ENI actions on &lt;code&gt;"Resource": "*"&lt;/code&gt;. In an environment audited under BACEN 4.893 or PCI-DSS, I narrow the resource to the connection's secret ARN and use &lt;code&gt;aws:ResourceTag/data-domain&lt;/code&gt; so the finance job can't read the Gusto secret, which holds payroll data.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Minimization at the source:&lt;/strong&gt; &lt;code&gt;SELECTED_FIELDS&lt;/code&gt; isn't only an optimization. If the acquirer's transaction report carries cardholder data the analysis doesn't need, don't pull it. What never enters the lake never enters audit scope.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Network:&lt;/strong&gt; the connection accepts its own network options. Running the job in a private subnet with egress through a fixed-IP NAT makes the SaaS-side allowlist easier, and some acquirers require it.&lt;/p&gt;

&lt;p&gt;Finally, observability: an alarm on &lt;code&gt;glue.driver.aggregate.numCompletedTasks&lt;/code&gt; equal to zero isn't enough. The signal that matters is ingested volume against the moving average for the same hour, because the typical failure mode isn't an error, it's zero rows with a success status.&lt;/p&gt;

&lt;h2&gt;
  
  
  Decision matrix
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Glue native connector
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Pros&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Covers the long tail of payments, billing and operations&lt;/li&gt;
&lt;li&gt;Reusable connection, per-connection secret, Glue IAM&lt;/li&gt;
&lt;li&gt;Transformation and MERGE in the same job&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Cons&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Watermark and overlap window are yours&lt;/li&gt;
&lt;li&gt;Field-only partitioning on several connectors&lt;/li&gt;
&lt;li&gt;DPU is expensive for small volume at high frequency&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Verdict:&lt;/strong&gt; Default for finance sources outside zero-ETL&lt;/p&gt;

&lt;h3&gt;
  
  
  Glue zero-ETL
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Pros&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;No job to operate, managed incremental&lt;/li&gt;
&lt;li&gt;Targets Redshift, S3 or S3 Tables&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Cons&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Only 8 SaaS sources, no payment ones&lt;/li&gt;
&lt;li&gt;No transformation on the way in&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Verdict:&lt;/strong&gt; Use it whenever the source is on the list&lt;/p&gt;

&lt;h3&gt;
  
  
  AppFlow
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Pros&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Lowest bill for scheduled copies&lt;/li&gt;
&lt;li&gt;No code&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Cons&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Bills empty runs and misconfiguration failures&lt;/li&gt;
&lt;li&gt;Needs a second step to deduplicate&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Verdict:&lt;/strong&gt; Fine for marketing and CRM with no reconciliation SLO&lt;/p&gt;

&lt;h3&gt;
  
  
  Custom extractor
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Pros&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Full control over rate limits and webhooks&lt;/li&gt;
&lt;li&gt;A bill measured in cents&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Cons&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Every API breaking change lands on your on-call&lt;/li&gt;
&lt;li&gt;Secrets and retries reinvented per source&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Verdict:&lt;/strong&gt; Only for near real-time via webhook or exotic APIs&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;The connector removes pagination, not the contract:&lt;/strong&gt; The native connector hands AWS the part that broke the most: authentication, pagination and endpoint changes. What stays yours is the data contract. If Adyen renames a field, the job doesn't fail, it writes nulls. A Glue Data Quality check on null rates and volume in the staging layer is what turns that silent failure into an alarm.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What I'd do:&lt;/strong&gt; I'd start by inventorying the custom extractors that already exist and mapping each one against the list of 100. The ones on the list become Glue jobs with a DynamoDB watermark and an Iceberg MERGE, and the old code gets deleted in the same sprint, not kept as a fallback. The lesson I paid for: a homegrown acquirer extractor never gets retired by decision, it gets retired by incident. Swapping it while it still works costs far less than swapping it during month-end close.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Recommendation
&lt;/h2&gt;

&lt;p&gt;Use &lt;strong&gt;zero-ETL&lt;/strong&gt; when the source is one of the 8 supported ones and you don't need to transform on the way in. Use the &lt;strong&gt;Glue native connector&lt;/strong&gt; when the source is payments, billing or accounting, reconciliation is involved and you can own the watermark and the idempotent MERGE: today it's the default I'd adopt for Adyen, Square, Xero and Chargebee. Use &lt;strong&gt;AppFlow&lt;/strong&gt; for scheduled marketing and CRM copies, where a duplicated row doesn't break a balance sheet. Keep a &lt;strong&gt;custom extractor&lt;/strong&gt; only for near real-time webhooks or APIs that even the REST connector can't cover. In every case, the alarm that matters is volume against the average, not job status.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Rating:&lt;/strong&gt; Glue native connector for finance source&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/about-aws/whats-new/2026/11/aws-glue-adds-100-new-native-connectors/" rel="noopener noreferrer"&gt;AWS Glue adds 100 new native connectors for popular SaaS applications (AWS What's New, 2026-11-09)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/glue/latest/dg/available-connections.html" rel="noopener noreferrer"&gt;AWS Glue: Available connections&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/glue/latest/dg/rest-api-connections.html" rel="noopener noreferrer"&gt;AWS Glue: REST API connections&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/glue/latest/dg/stripe-connection-options.html" rel="noopener noreferrer"&gt;AWS Glue: Stripe connection options&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/en_us/glue/latest/dg/stripe-limitations.html" rel="noopener noreferrer"&gt;AWS Glue: Stripe connector limitations&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/glue/latest/dg/zero-etl-using.html" rel="noopener noreferrer"&gt;AWS Glue: Zero-ETL integrations&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/glue/pricing/" rel="noopener noreferrer"&gt;AWS Glue pricing&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/appflow/pricing/" rel="noopener noreferrer"&gt;Amazon AppFlow pricing&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://fernando.moretes.com/blog/conectores-saas-nativos-do-glue-contra-appflow-zero-etl-e-codigo" rel="noopener noreferrer"&gt;fernando.moretes.com&lt;/a&gt;. By Fernando F. Azevedo: Senior Solutions Architect.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>dataplatforms</category>
      <category>awsglue</category>
      <category>saasingestion</category>
      <category>dataplatform</category>
    </item>
    <item>
      <title>LLM-generated data quality rules are cheap. Owning them for years is not</title>
      <dc:creator>Fernando Azevedo</dc:creator>
      <pubDate>Fri, 06 Nov 2026 18:08:59 +0000</pubDate>
      <link>https://dev.to/fernando_azevedo_6844e930/llm-generated-data-quality-rules-are-cheap-owning-them-for-years-is-not-fj5</link>
      <guid>https://dev.to/fernando_azevedo_6844e930/llm-generated-data-quality-rules-are-cheap-owning-them-for-years-is-not-fj5</guid>
      <description>&lt;p&gt;After 16 years building financial platforms on AWS, the question I ask about data quality isn't "who writes the rule?", it's "who answers for it at 2 AM, when it blocks a legitimate load?". The Indegene case, published on November 4, 2026, matters because of that second question: an agent proposes rules and code, but only what passes human approval enters the central repository. Combined with the &lt;code&gt;ADVANCED&lt;/code&gt; mode in AWS Glue Data Quality, it shows where the cost is heading: out of authoring and into maintenance.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three numbers to calibrate the decision
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;17&lt;/strong&gt;: Regions with ADVANCED mode. The documented list does not include &lt;code&gt;sa-east-1&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;14d&lt;/strong&gt;: Default sample retention in S3. Lifecycle rule on the Athena query results&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;267h&lt;/strong&gt;: Review in my own hypothetical scenario. 200 tables × 40 columns × 1 rule × 2 minutes&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The signal: generating rules became a commodity
&lt;/h2&gt;

&lt;p&gt;Indegene's Data Quality Agent is part of its NEXT Unified Data Platform. It was built with Strands Agents, runs on Amazon Bedrock AgentCore and serves Amazon Redshift and Snowflake behind AWS WAF, API Gateway and Cognito, with OAuth 2.0 and JWT validation on each request. The flow has 10 steps: source, schema, profiling, anomalies against a historical baseline, existing rule management, rule mode (column, table or custom query), suggestions with severity, review, SQL and Python generation and validation, and storage in the central repository.&lt;/p&gt;

&lt;p&gt;Two knowledge bases provide the context. The internal one combines glossaries, data dictionaries, SOPs, regulatory guidance, approved rules, validated scripts, profiling results, historical anomalies and expert feedback. The external one comes from approved frameworks, web search and controlled scraping. That's what lets the agent suggest the 10-digit format for an NPI column without anyone teaching it.&lt;/p&gt;

&lt;p&gt;In September 2026, Glue Data Quality gained &lt;code&gt;ADVANCED&lt;/code&gt; mode, which uses Bedrock over metadata and a random sample of the table to generate DQDL rules. Once the data plane itself ships generation out of the box, generation stops being a platform differentiator.&lt;/p&gt;

&lt;p&gt;One limit on what I read: the part of the post I could open ends at the architecture description, with no detail on the validation mechanism and no results figure. That's why I don't quote any percentage gain here.&lt;/p&gt;

&lt;h2&gt;
  
  
  What changes for whoever designs the platform
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Rules become design-time artifacts:&lt;/strong&gt; the documentation warns that &lt;code&gt;ADVANCED&lt;/code&gt; can return different rules for the same table, so generation can't sit in the execution path.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The bottleneck moves:&lt;/strong&gt; writing stops being the limit, human review capacity takes its place.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Technical and business rules now share one lifecycle:&lt;/strong&gt; the one living in code and the one living in an SOP enter the same versioned repository.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The sample becomes a data surface:&lt;/strong&gt; generating with context means sending real rows to a model, and that requires a classification decision before the first run.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The knowledge base can propagate errors:&lt;/strong&gt; approved rules return as context, so an error approved today can be suggested again tomorrow.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Where a generated rule fails silently
&lt;/h2&gt;

&lt;p&gt;The risk isn't the rule that breaks, it's the rule that passes. A rule with a syntax error fails on the first run. The ones below stay green for months:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Inverted semantics:&lt;/strong&gt; in row-level &lt;code&gt;CustomSql&lt;/code&gt;, a record returned by the query counts as passed and a record not returned counts as failed. A model that writes the natural audit query, &lt;code&gt;select id from primary where amount &amp;lt; 0&lt;/code&gt;, passes exactly the wrong records.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The approving filter:&lt;/strong&gt; with &lt;code&gt;where&lt;/code&gt;, filtered-out rows are labeled &lt;code&gt;Passed&lt;/code&gt; by default. Use &lt;code&gt;rowLevelConfiguration.filteredRowLabel&lt;/code&gt; = &lt;code&gt;SKIPPED&lt;/code&gt;, or a filter that matches nothing looks like compliance.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Column-scoped composites:&lt;/strong&gt; by default, &lt;code&gt;(A) or (B)&lt;/code&gt; evaluates each rule across the whole table and combines the results afterward. The documentation's example fails in that mode and passes with &lt;code&gt;compositeRuleEvaluation.method&lt;/code&gt; = &lt;code&gt;ROW&lt;/code&gt;. &lt;code&gt;Completeness&lt;/code&gt;, &lt;code&gt;Uniqueness&lt;/code&gt;, &lt;code&gt;DatasetMatch&lt;/code&gt; and &lt;code&gt;ReferentialIntegrity&lt;/code&gt; aren't even supported in &lt;code&gt;ROW&lt;/code&gt; mode.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Empty history:&lt;/strong&gt; with no prior data points, &lt;code&gt;last(k)&lt;/code&gt; returns 0.0, so &lt;code&gt;RowCount &amp;gt; last()&lt;/code&gt; passes on the first run without proving anything.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Frozen domain:&lt;/strong&gt; a rule derived from the sample, such as &lt;code&gt;ColumnValues "territory" in [...]&lt;/code&gt;, treats a new legitimate value as a defect, and can treat an old defect as a valid domain.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The post's business example, market share totaling 100% per territory, carries the same numeric trap as a financial reconciliation: it needs an explicit tolerance. One way to write it in DQDL, with a tolerance you have to calibrate: &lt;code&gt;CustomSql "select count(*) from (select territory_id from primary group by territory_id having abs(sum(market_share) - 100) &amp;gt; 0.5) t" = 0&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where human approval separates generation from execution
&lt;/h2&gt;

&lt;p&gt;The diagram answers one question: what has to happen for a model suggestion to become a production rule. It simplifies the 10 steps described in the post. The dashed arrows are the asynchronous return to the knowledge base, and that's the path by which an approved error can come back as a suggestion.&lt;/p&gt;

&lt;h3&gt;
  
  
  Generation
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Knowledge bases internal + external (data)&lt;/li&gt;
&lt;li&gt;Data Quality Agent Strands + AgentCore (ai)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Governance
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Data steward review and edit (user)&lt;/li&gt;
&lt;li&gt;Code validation SQL + Python (security)&lt;/li&gt;
&lt;li&gt;Rule repository approved rules (storage)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Runtime
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Redshift, Snowflake rule execution (data)&lt;/li&gt;
&lt;li&gt;Monitoring and audit results (compute)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Flows
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;kb -&amp;gt; agent: RAG&lt;/li&gt;
&lt;li&gt;agent -&amp;gt; steward: suggests with severity&lt;/li&gt;
&lt;li&gt;steward -&amp;gt; validate: approves&lt;/li&gt;
&lt;li&gt;validate -&amp;gt; repo: stores&lt;/li&gt;
&lt;li&gt;repo -&amp;gt; engine: runs&lt;/li&gt;
&lt;li&gt;engine -&amp;gt; monitor: publishes results&lt;/li&gt;
&lt;li&gt;repo -&amp;gt; kb: approved rules&lt;/li&gt;
&lt;li&gt;monitor -&amp;gt; kb: historical anomalies&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The sample boundary is an architecture decision
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;ADVANCED&lt;/code&gt; mode is practical, and the data protection documentation is clear about what it does. Athena samples rows through the &lt;code&gt;glue-dataquality-sampling&lt;/code&gt; workgroup, and Glue doesn't exclude any column from the sample. Metadata and rows go to Bedrock, which doesn't retain inputs or outputs after processing and doesn't use them to train models. For many scenarios, that's enough.&lt;/p&gt;

&lt;p&gt;For healthcare professional, patient or bank customer data, three details change the math:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;No masking in the call:&lt;/strong&gt; &lt;code&gt;ADVANCED&lt;/code&gt; doesn't support &lt;code&gt;PreProcessingQuery&lt;/code&gt;, so the &lt;code&gt;SELECT&lt;/code&gt; without the sensitive column, the obvious way out, isn't available. Point the run at an already masked table.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cross-Region inference:&lt;/strong&gt; the run uses geographic cross-Region inference, and sampled content can leave the source Region, within the geographic boundary of the profile. With &lt;code&gt;sa-east-1&lt;/code&gt; off the list, that path has to enter your LGPD transfer inventory and, where applicable, your BACEN 4.893 assessment.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Bedrock IAM isn't the control:&lt;/strong&gt; the recommendation role doesn't need Bedrock permissions, so denying &lt;code&gt;bedrock:InvokeModel&lt;/code&gt; on it doesn't guarantee the send is blocked. The useful control is pinning &lt;code&gt;aws:SourceArn&lt;/code&gt; to the table in the trust policy and not giving the role access to the sensitive table.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;code&gt;BASIC&lt;/code&gt; deserves attention too. In supported Regions, it attempts to enhance the ruleset with Bedrock by sending the ruleset and column metadata, without you enabling anything. A statistical ruleset can carry real values in a &lt;code&gt;ColumnValues ... in [...]&lt;/code&gt; list, so check that against your classification.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three paths for generating rules
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Criterion&lt;/th&gt;
&lt;th&gt;Glue &lt;code&gt;BASIC&lt;/code&gt;
&lt;/th&gt;
&lt;th&gt;Glue &lt;code&gt;ADVANCED&lt;/code&gt;
&lt;/th&gt;
&lt;th&gt;Own agent (Indegene pattern)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Rule context&lt;/td&gt;
&lt;td&gt;Table statistics&lt;/td&gt;
&lt;td&gt;Metadata and sampled rows&lt;/td&gt;
&lt;td&gt;Glossary, SOP, history and expert feedback&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sources&lt;/td&gt;
&lt;td&gt;Data Catalog tables and ETL jobs&lt;/td&gt;
&lt;td&gt;Data Catalog tables, in 17 Regions&lt;/td&gt;
&lt;td&gt;Whatever you connect; in this case, Redshift and Snowflake&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Data sent to the model&lt;/td&gt;
&lt;td&gt;Ruleset and column metadata, when enhanced&lt;/td&gt;
&lt;td&gt;Metadata and rows, with no column exclusion&lt;/td&gt;
&lt;td&gt;Whatever your design allows, and the control is yours&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;The cost that stays&lt;/td&gt;
&lt;td&gt;Reviewing statistical rules&lt;/td&gt;
&lt;td&gt;Reviewing every run and paying for Athena&lt;/td&gt;
&lt;td&gt;Maintaining agent, knowledge bases, evals and all of the above&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Use when&lt;/td&gt;
&lt;td&gt;The dataset is new and the goal is structural coverage&lt;/td&gt;
&lt;td&gt;The data can be sampled and Glue is already the catalog&lt;/td&gt;
&lt;td&gt;The rule depends on knowledge outside the table, across several platforms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Human approval is capacity, not a checkbox
&lt;/h2&gt;

&lt;p&gt;Human-in-the-loop usually appears in a diagram as a person icon. In operations, it's a queue. Run the math for your environment. In my hypothetical scenario, 200 tables with 40 columns and a single rule per column give you 8,000 rules. At 2 minutes per rule, that's 267 hours of review before the first execution. Nobody reviews that carefully: at some point in the queue, approving becomes clicking, and the control stays on paper while it stops existing in practice.&lt;/p&gt;

&lt;p&gt;The per-rule severity that Indegene's agent already suggests is what makes that queue manageable. Review the critical ones one by one, batch-approve low-risk structural rules with audit sampling, and discard without guilt any rule nobody would claim as owner. A rule with no named owner shouldn't enter the repository.&lt;/p&gt;

&lt;p&gt;Then comes the recurring cost. Going by the examples on the Glue pricing page, a 5-DPU evaluation running 20 minutes costs about US$ 0.73, at US$ 0.44 per DPU-hour. Across 200 tables, daily, that's about US$ 4,380 per month, and it's the smallest of the costs. The largest is the false alert: every bad rule triggers an investigation, and a team that chases three false alarms stops looking at the fourth. Measure per rule how many failures turned into a real fix, and retire the ones that never did.&lt;/p&gt;

&lt;h2&gt;
  
  
  How I would position for this
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Treat rules as code:&lt;/strong&gt; the approved ruleset goes to a versioned repository, with hash, owner, severity and reason. Execution reads only that version and never calls the model. This also resolves the nondeterminism: running generation twice doesn't change what's in production.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Test the rule against seeded defects:&lt;/strong&gt; for each rule, one fixture with good records and another with the defect it promises to catch. The rule must pass on the first and fail on the second. This is the test that catches the inverted &lt;code&gt;CustomSql&lt;/code&gt;, and no text review catches it as reliably. Strands evaluation tools help measure the agent, but an LLM judge evaluating LLM output doesn't replace executing the rule.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Don't confuse a valid plan with a safe query:&lt;/strong&gt; Redshift's &lt;code&gt;EXPLAIN&lt;/code&gt; shows the plan without running it, and it also accepts &lt;code&gt;UPDATE&lt;/code&gt; and &lt;code&gt;DELETE&lt;/code&gt;. Passing it proves the query compiles, not that it only reads. Combine it with a &lt;code&gt;SELECT&lt;/code&gt; allowlist in a parser and a read-only role.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Separate the generation identity from the execution identity:&lt;/strong&gt; the November 5 post on AgentCore and Lake Formation shows the right path for exploration: the query runs under the user's mapped role, with LF-Tags and data cell filters, and enforcement doesn't depend on the SQL the agent wrote. Production execution is a different story. Glue's support matrix marks tables with Lake Formation column or data filters as unsupported across several formats, so the quality role tends to have broad access. Treat that role as a privileged principal and audit it as one.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I wouldn't do
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Generating rules at run time:&lt;/strong&gt; the same data can get different rules between runs, and the audit loses the answer to "which rule applied that day?".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Measuring success by rule count:&lt;/strong&gt; per-column coverage is easy to inflate, and every extra rule is one more alert source for someone to maintain.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Feeding web content straight into the rules RAG:&lt;/strong&gt; a scraped page is untrusted input. Require the same curation as the internal base, or a suggested rule can originate from text nobody approved.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Relying on &lt;code&gt;labels&lt;/code&gt; for severity everywhere:&lt;/strong&gt; DQDL labels exist only in Glue ETL, not in Data Catalog-based Data Quality. Whoever standardizes on them finds the gap in the report.&lt;/li&gt;
&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;The lesson in the feedback loop:&lt;/strong&gt; Approved rules and validated scripts enter the internal knowledge base. That improves the next suggestion and also gives a bad approval a reach it didn't have before: the error stops affecting one table and starts guiding the next ones. When you revoke a rule, remove its vector from the knowledge base too. Deleting it only from the repository leaves the error alive in the context.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Curator's note:&lt;/strong&gt; I'd start with Glue &lt;code&gt;BASIC&lt;/code&gt; on a non-sensitive dataset and use &lt;code&gt;ADVANCED&lt;/code&gt; only on a masked table, before writing a line of agent code. I'd write the seeded-defect fixtures in the first week, because it's the one item on this list that gets more expensive with every rule approved without it. I'd only move to my own agent when the rule that matters depended on SOPs and glossaries, across more than one platform, which is the actual condition of the Indegene case. The lesson I take from financial systems: the cost of an automated control isn't the cost of creating it, it's the cost of proving years later why it exists and whether it still works.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  My recommendation
&lt;/h2&gt;

&lt;p&gt;Use the native Glue capability when your catalog already lives there, the rule can be explained by the data and the sample can go to the model. Build an agent when the rule depends on knowledge outside the table, across several platforms, and you can pay for its maintenance. In both cases, don't approve at scale before you have an owner per rule, seeded-defect tests and retirement criteria: without those, generating faster only brings the debt forward.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Rating:&lt;/strong&gt; Adote com governança / Adopt with govern&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/blogs/machine-learning/how-indegene-is-modernizing-and-scaling-enterprise-data-quality-in-life-sciences/" rel="noopener noreferrer"&gt;How Indegene is modernizing and scaling enterprise data quality in life sciences&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/glue/latest/dg/data-quality-getting-started.html" rel="noopener noreferrer"&gt;Getting started with AWS Glue Data Quality: recommendation modes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/glue/latest/dg/data-protection-advanced-dq-recommendations.html" rel="noopener noreferrer"&gt;Data protection for advanced data quality rule recommendations&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/glue/latest/dg/data-quality-authorization.html" rel="noopener noreferrer"&gt;Configure IAM permissions for AWS Glue Data Quality&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/glue/latest/dg/dqdl.html" rel="noopener noreferrer"&gt;Data Quality Definition Language (DQDL) reference&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/glue/latest/dg/dqdl-rule-types-CustomSql.html" rel="noopener noreferrer"&gt;DQDL rule type: CustomSQL&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/redshift/latest/dg/r_EXPLAIN.html" rel="noopener noreferrer"&gt;Amazon Redshift: EXPLAIN&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/blogs/big-data/deliver-multi-tenant-ai-analytics-at-scale-with-agentcore-and-lake-formation/" rel="noopener noreferrer"&gt;Deliver multi-tenant AI analytics at scale with AgentCore and Lake Formation&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://fernando.moretes.com/blog/regra-de-qualidade-gerada-por-llm-custa-pouco-dificil-e-mante-la-por" rel="noopener noreferrer"&gt;fernando.moretes.com&lt;/a&gt;. By Fernando F. Azevedo: Senior Solutions Architect.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>dataplatforms</category>
      <category>dataquality</category>
      <category>agentesdeia</category>
      <category>awsglue</category>
    </item>
    <item>
      <title>Lambda with Kafka queues: parallelism past partitions, and what it costs</title>
      <dc:creator>Fernando Azevedo</dc:creator>
      <pubDate>Tue, 03 Nov 2026 18:31:33 +0000</pubDate>
      <link>https://dev.to/fernando_azevedo_6844e930/lambda-with-kafka-queues-parallelism-past-partitions-and-what-it-costs-217d</link>
      <guid>https://dev.to/fernando_azevedo_6844e930/lambda-with-kafka-queues-parallelism-past-partitions-and-what-it-costs-217d</guid>
      <description>&lt;p&gt;For years the answer to "my Kafka consumer can't keep up with the spike" was the same: more partitions. The problem is that a partition is not a throughput knob, it is a contract with the producer, with rebalancing and with whoever operates the cluster. On November 2, 2026, AWS added a second consumption mode, &lt;code&gt;Queue&lt;/code&gt;, to the self-managed Kafka event source mapping (ESM), built on Apache Kafka 4.2 share groups. The question that matters is not "does this scale further?". It does. The question is: who guarantees that record 4,812 on partition 3 was processed once, and what happens to it when it fails for the fifth time at 2 AM?&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem: the partition became a concurrency ceiling
&lt;/h2&gt;

&lt;p&gt;In &lt;code&gt;Stream&lt;/code&gt; mode (the default), the ESM uses the consumer group protocol: each partition belongs to exactly one consumer. The Lambda documentation is explicit about the consequence: in provisioned mode, to keep ordering within a partition, &lt;code&gt;MaximumPollers&lt;/code&gt; is capped at the topic's partition count, and each poller handles up to 5 MB/s or up to 5 concurrent invocations.&lt;/p&gt;

&lt;p&gt;Do the math for a 12-partition topic: 12 pollers times 5 invocations is 60 simultaneous executions, at most. If the handler calls a fraud model that takes 800 ms, the ceiling is close to 75 records per second with single-record batches, and no &lt;code&gt;ReservedConcurrentExecutions&lt;/code&gt; setting changes that.&lt;/p&gt;

&lt;p&gt;The two traditional ways out carry a maintenance cost, not a build cost:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Over-partitioning:&lt;/strong&gt; 12 becomes 96. The partition key distribution changes, rebalancing gets slower, the broker carries more segment files and replicas, and you almost never go back, because shrinking partitions does not exist in Kafka.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Copying into another queue:&lt;/strong&gt; a consumer that republishes to SQS. Now you operate two queues, two retention policies, two duplication points and a bridge that someone has to keep current.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For work where each event is an independent unit (media encoding, notifications, telemetry ingestion, per-transaction fraud scoring), paying the price of ordering without needing it is pure waste.&lt;/p&gt;

&lt;h2&gt;
  
  
  Anatomy: share group, lock and acknowledgement
&lt;/h2&gt;

&lt;p&gt;KIP-932 swaps partition ownership for record ownership. The broker keeps a window of in-flight records per partition, between the share-partition start offset (SPSO) and end offset (SPEO). Each consumer that fetches data acquires a temporary lock on records, not on the partition. When done, the consumer acknowledges each record with one of three types: &lt;code&gt;ACCEPT&lt;/code&gt; (processed), &lt;code&gt;RELEASE&lt;/code&gt; (return for another attempt) or &lt;code&gt;REJECT&lt;/code&gt; (unprocessable, never comes back).&lt;/p&gt;

&lt;p&gt;Three group parameters govern the behavior, all settable per group with &lt;code&gt;kafka-configs.sh --entity-type groups&lt;/code&gt; and clamped by broker minimums and maximums:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;share.record.lock.duration.ms&lt;/code&gt;: how long the lock lasts. The default is 30 s; once it expires, the record becomes available to another consumer.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;share.delivery.count.limit&lt;/code&gt;: how many deliveries a record gets before it is archived. The default is 5.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;share.partition.max.record.locks&lt;/code&gt;: how many records can be locked at once per partition. This is the real parallelism ceiling within a partition.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Kafka 4.2, released on 02/17/2026, declared share groups production-ready and added the &lt;code&gt;RENEW&lt;/code&gt; type, to extend the lock during long processing.&lt;/p&gt;

&lt;p&gt;In Lambda, the Queue mode ESM uses the Kafka Share Consumer: each event poller reads from all partitions, with no per-partition ordering, and supports up to 10 concurrent invocations, double the Stream mode figure. The event payload stays identical to traditional Kafka, with &lt;code&gt;topic&lt;/code&gt;, &lt;code&gt;partition&lt;/code&gt; and &lt;code&gt;offset&lt;/code&gt;, so the function code does not change.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where parallelism comes from, and where the failing record goes
&lt;/h2&gt;

&lt;p&gt;The diagram answers one question: in Queue mode, who controls concurrency and who controls failure. Concurrency comes from the number of pollers, not partitions. Failure has two paths: with an on-failure destination configured, Lambda sends the record to SQS; without one, the broker redelivers up to &lt;code&gt;share.delivery.count.limit&lt;/code&gt; and then archives it. The dashed arrow is the asynchronous failure path. Simplification: VPC networking and cluster authentication were left out.&lt;/p&gt;

&lt;h3&gt;
  
  
  Kafka 4.2 self-managed
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Producers unchanged (external)&lt;/li&gt;
&lt;li&gt;Topic, 12 partitions share group (messaging)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Lambda ESM Queue mode
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Event pollers 10 invocations each (compute)&lt;/li&gt;
&lt;li&gt;Lambda function idempotent handler (compute)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Side effects
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;DynamoDB idempotency keys (data)&lt;/li&gt;
&lt;li&gt;SQS on-failure destination (messaging)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Flows
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;producers -&amp;gt; topic: publishes&lt;/li&gt;
&lt;li&gt;pollers -&amp;gt; topic: acquires lock / ack&lt;/li&gt;
&lt;li&gt;pollers -&amp;gt; fn: invokes in batches&lt;/li&gt;
&lt;li&gt;fn -&amp;gt; ddb: conditional write&lt;/li&gt;
&lt;li&gt;pollers -&amp;gt; dlq: retries exhausted&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Stream and Queue on the same ESM
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Criterion&lt;/th&gt;
&lt;th&gt;Stream (default)&lt;/th&gt;
&lt;th&gt;Queue&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Ordering&lt;/td&gt;
&lt;td&gt;Guaranteed per partition&lt;/td&gt;
&lt;td&gt;None&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Invocations per poller&lt;/td&gt;
&lt;td&gt;Up to 5&lt;/td&gt;
&lt;td&gt;Up to 10&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Poller ceiling (provisioned)&lt;/td&gt;
&lt;td&gt;Partition count&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;MaximumPollers&lt;/code&gt; up to 2,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;On-demand&lt;/td&gt;
&lt;td&gt;Scales on offset lag&lt;/td&gt;
&lt;td&gt;5 pollers, 5 MB/s or 50 invocations&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Starting position&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;StartingPosition&lt;/code&gt; accepted&lt;/td&gt;
&lt;td&gt;Not supported; uses the share group position&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Batching window&lt;/td&gt;
&lt;td&gt;Supported&lt;/td&gt;
&lt;td&gt;Validation error&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Amazon MSK&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Not at launch&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  The configuration that decides whether it works
&lt;/h2&gt;

&lt;p&gt;Turning it on is one line: &lt;code&gt;SelfManagedKafkaEventSourceConfig.ConsumptionMode: Queue&lt;/code&gt;. Operating it is a different conversation. Four decisions make the difference.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Scale:&lt;/strong&gt; Queue mode on-demand stops at 5 pollers at 1 MB/s each, meaning 5 MB/s or 50 invocations, whichever comes first. For real spikes, use provisioned mode: &lt;code&gt;MinimumPollers&lt;/code&gt; from 1 to 200, &lt;code&gt;MaximumPollers&lt;/code&gt; from 1 to 2,000, each poller with up to 5 MB/s. In practice the function's concurrency is the limit: 40 pollers times 10 is 400 executions, and the account's default concurrency quota has to absorb that without starving everything else.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Lock versus timeout:&lt;/strong&gt; the default lock is 30 s. If a 100-record batch takes 45 s in the function, the broker treats the lock as expired and hands the same records to another poller. The documentation I read does not say whether the poller renews the lock, so I treat &lt;code&gt;share.record.lock.duration.ms&lt;/code&gt; greater than the function timeout as a rule, and &lt;code&gt;BatchSize&lt;/code&gt; small enough to fit inside it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cost:&lt;/strong&gt; provisioned mode bills per EPU-hour, US$ 0.185 in us-east-1 in the documentation's own example, and one EPU holds 10 pollers by default. Queue ESMs only share a &lt;code&gt;PollerGroupName&lt;/code&gt; with other Queue ESMs; mixing with Stream returns a validation error.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Backlog:&lt;/strong&gt; there is no &lt;code&gt;StartingPosition&lt;/code&gt;. A new share group starts at the latest offset, so an ESM created over a topic with an accumulated backlog ignores that backlog until someone adjusts &lt;code&gt;share.auto.offset.reset&lt;/code&gt; on the group.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;The record that disappears without a warning:&lt;/strong&gt; In Queue mode, &lt;code&gt;MaximumRetryAttempts&lt;/code&gt; defaults to 0 and, with no on-failure destination configured, the broker is the one redelivering. After &lt;code&gt;share.delivery.count.limit&lt;/code&gt; attempts (5 by default), the record is archived: it does not come back, it does not land in any queue, it raises no error in the function. For billing notifications or fraud alerts, that is a lost event with no trail. Configure the on-failure destination on SQS, SNS or a Kafka topic before the first deploy, not after the first incident.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  When to use it, and when not to
&lt;/h2&gt;

&lt;p&gt;Use Queue mode when three things are true at once: each event is an independent unit of work, the handler is idempotent on a business key (not on the offset) and the measured bottleneck is function concurrency, not broker throughput. Per-transaction fraud scoring, push and email delivery, thumbnails and transcoding, telemetry enrichment: all of them pass that test.&lt;/p&gt;

&lt;p&gt;Do not use it when state depends on sequence. Per-account ledger postings, order state machines, balance updates, CDC that applies an &lt;code&gt;UPDATE&lt;/code&gt; after an &lt;code&gt;INSERT&lt;/code&gt;: anything where "event 2 before event 1" produces a wrong number. In a financial system that is not a performance bug, it is a ledger divergence that someone reconciles by hand at month-end close.&lt;/p&gt;

&lt;p&gt;Do not use it as a shortcut to avoid understanding your own lag either. If the consumer is behind because the handler makes a synchronous 3 s call to a legacy system with a 20-connection pool, doubling concurrency only moves the queue from Kafka to the legacy system, and now it goes down too. The new parallelism needs an explicit limit at the far end: &lt;code&gt;ReservedConcurrentExecutions&lt;/code&gt; on the function or a &lt;code&gt;MaximumPollers&lt;/code&gt; value derived from the capacity of whatever receives the call.&lt;/p&gt;

&lt;p&gt;And note the launch scope: only self-managed Kafka and Confluent Cloud, on version 4.2 or later with share groups enabled. Amazon MSK waits until it offers that version.&lt;/p&gt;

&lt;h2&gt;
  
  
  Anti-patterns
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Idempotency on the offset:&lt;/strong&gt; &lt;code&gt;topic-partition-offset&lt;/code&gt; only deduplicates broker redelivery. A producer retry creates a new offset for the same payment; the key has to be the &lt;code&gt;transactionId&lt;/code&gt;, with &lt;code&gt;attribute_not_exists(pk)&lt;/code&gt; in DynamoDB and a TTL aligned with topic retention.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Switching an ordered consumer's mode in production:&lt;/strong&gt; moving a balance flow from Stream to Queue because lag went up. Throughput improves the same day and reconciliation breaks at month-end close.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Lock shorter than the timeout:&lt;/strong&gt; a function with a 120 s timeout and &lt;code&gt;share.record.lock.duration.ms&lt;/code&gt; at the 30 s default. Every slow batch is delivered twice, and invocation cost doubles exactly at the peak.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Parallelism with no ceiling at the far end:&lt;/strong&gt; &lt;code&gt;MaximumPollers: 2000&lt;/code&gt; with a downstream that accepts 50 connections. Kafka stops piling up and the database starts refusing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mixed poller group:&lt;/strong&gt; planning shared EPUs across Stream and Queue ESMs. The API refuses it, and the provisioned-mode budget calculated with grouping savings comes out wrong.&lt;/li&gt;
&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Migration without a big bang:&lt;/strong&gt; Share groups and consumer groups are independent and read the same topic without interfering. Create the Queue ESM pointing at a shadow function that only writes the idempotency key and measures &lt;code&gt;Duration&lt;/code&gt; and &lt;code&gt;ConcurrentExecutions&lt;/code&gt;, adjust &lt;code&gt;share.auto.offset.reset&lt;/code&gt; if you need the backlog, compare for a week and only then disable the Stream ESM. No producer needs to know.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Curator's note:&lt;/strong&gt; I have seen a topic go from 24 to 192 partitions because of one slow consumer, and that decision outlived the consumer, the team and the cluster version. What I would do today: Queue mode for the notification and scoring flow, an SQS on-failure destination from day one, a lock longer than the timeout and a &lt;code&gt;MaximumPollers&lt;/code&gt; value derived from downstream capacity, not from the Kafka peak. The balance flow stays on Stream, with the partitions it already has. The hard-won lesson: the cost of over-partitioning was never creating the partitions, it was living with them for years.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Verdict
&lt;/h2&gt;

&lt;p&gt;Queue mode solves a real problem: it removes the partition from its role as concurrency ceiling without touching producers, topics or function code, and it doubles invocations per poller from 5 to 10. That said, it shifts responsibility: ordering goes away, duplication becomes the rule and, with no on-failure destination, the broker silently archives the record after 5 deliveries. &lt;strong&gt;Adopt it&lt;/strong&gt; for independent work that is idempotent on a business key, on self-managed Kafka 4.2 or Confluent Cloud, with the on-failure destination and lock sized before deploy. &lt;strong&gt;Do not adopt it&lt;/strong&gt; for any flow where sequence defines state; there, Stream with a well-chosen partition count is still the right answer.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Rating:&lt;/strong&gt; Adotar com condições / Adopt with condit&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/about-aws/whats-new/2026/11/aws-lambda-queue-kafka-esm/" rel="noopener noreferrer"&gt;AWS What's New: Lambda supports queues for Apache Kafka in self-managed Kafka event sources (02/11/2026)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/lambda/latest/dg/polling-kafka-queue.html" rel="noopener noreferrer"&gt;AWS Lambda Developer Guide: Polling Kafka topics in Queue mode&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/lambda/latest/dg/kafka-scaling-modes.html" rel="noopener noreferrer"&gt;AWS Lambda Developer Guide: Apache Kafka event poller scaling modes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/lambda/latest/dg/with-kafka.html" rel="noopener noreferrer"&gt;AWS Lambda Developer Guide: Using Lambda with self-managed Apache Kafka&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932+Queues+for+Kafka" rel="noopener noreferrer"&gt;KIP-932: Queues for Kafka&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://kafka.apache.org/blog/2026/02/17/apache-kafka-4.2.0-release-announcement/" rel="noopener noreferrer"&gt;Apache Kafka 4.2.0 Release Announcement&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.confluent.io/platform/current/config-manage/kafka-queues.html" rel="noopener noreferrer"&gt;Confluent Platform: Share Groups&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://fernando.moretes.com/blog/lambda-com-filas-do-kafka-paralelismo-acima-das-particoes-e-o-que-ele" rel="noopener noreferrer"&gt;fernando.moretes.com&lt;/a&gt;. By Fernando F. Azevedo: Senior Solutions Architect.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>awscloud</category>
      <category>kafka</category>
      <category>awslambda</category>
      <category>eventdriven</category>
    </item>
    <item>
      <title>One listing, many legal entities: AWS Marketplace subsidiaries in practice</title>
      <dc:creator>Fernando Azevedo</dc:creator>
      <pubDate>Fri, 30 Oct 2026 17:14:07 +0000</pubDate>
      <link>https://dev.to/fernando_azevedo_6844e930/one-listing-many-legal-entities-aws-marketplace-subsidiaries-in-practice-3i3o</link>
      <guid>https://dev.to/fernando_azevedo_6844e930/one-listing-many-legal-entities-aws-marketplace-subsidiaries-in-practice-3i3o</guid>
      <description>&lt;p&gt;Anyone who has operated a group with one entity per country knows the hidden cost of AWS Marketplace: the same listing recreated in every seller account, with descriptions and pricing dimensions that drift silently. On 10/29/2026 AWS made it possible to publish the product once in the primary account and authorize subsidiaries to sell it through direct private offers, each under its own tax identity, currencies and bank account. The announcement sells this as the end of duplication. I read it differently: duplication leaves the catalog and becomes an authorization problem, with states, defaults and a revocation that does not do what the name suggests.&lt;/p&gt;

&lt;h2&gt;
  
  
  The situation: the catalog was never the problem
&lt;/h2&gt;

&lt;p&gt;In a group with a US entity, a Brazilian Ltda and an Indian subsidiary, the reason for three seller accounts is not technical, it is fiscal. Each entity has to be the seller of record in its market: it holds the customer contract, invoices in local currency, pays its own Marketplace fees and gets paid into its own bank account. The documentation is explicit about this, and the new capability does not change that premise. It only removes the obligation to maintain the product inside every account.&lt;/p&gt;

&lt;p&gt;The expensive part was never creating the listing. It was keeping three copies in sync for years: every new pricing dimension, every description tweak, every product version multiplied by the number of entities, and every drift found by a customer before you found it. That is the maintenance cost this capability attacks, and it is real.&lt;/p&gt;

&lt;p&gt;What it puts in its place deserves the same care. The primary account now issues authorizations, one per product and per subsidiary, stating in which countries and with which pricing dimensions that entity can sell. That is access control over revenue. In a regulated environment, anything that decides who sells what, to whom and where belongs in the same bucket as an IAM role: it needs an owner, an audit trail and periodic review.&lt;/p&gt;

&lt;p&gt;One detail that tempers the optimism of anyone who already has duplicate listings: live agreements on a subsidiary's own listing stay there and cannot be moved to the primary account's product. The migration applies to new offers, not to the installed base. You will live with both models until the last legacy contract expires.&lt;/p&gt;

&lt;h2&gt;
  
  
  The model: two layers and nothing shared by default
&lt;/h2&gt;

&lt;p&gt;Authorization has two layers, and mixing them up is the first mistake I would expect to see in production.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Connection:&lt;/strong&gt; done in AWS Partner Central. The primary sends the request, with up to 10 account IDs per submission, and an administrator in the subsidiary accepts it. Acceptance has a side effect that needs legal sign-off: it records the subsidiary's consent to share partner attributed revenue and its Marketplace insights reports with the primary. The connection alone authorizes no sales at all.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Authorization:&lt;/strong&gt; a &lt;code&gt;SubsidiaryAuthorization@1.0&lt;/code&gt; entity in the Catalog API, created by the primary with &lt;code&gt;StartChangeSet&lt;/code&gt; and the &lt;code&gt;CreateSubsidiaryAuthorization&lt;/code&gt; change type. It starts in &lt;code&gt;Draft&lt;/code&gt;, becomes &lt;code&gt;Active&lt;/code&gt; through &lt;code&gt;UpdateAvailability&lt;/code&gt; and ends in &lt;code&gt;Restricted&lt;/code&gt;. It carries the product ARN (immutable), the &lt;code&gt;Dimensions&lt;/code&gt; list, the &lt;code&gt;SubsidiaryAccountId&lt;/code&gt; (also immutable, it becomes the &lt;code&gt;SubsidiaryTargetingRule&lt;/code&gt;) and a &lt;code&gt;BuyerTargetingTerm&lt;/code&gt; with &lt;code&gt;CountryCodes&lt;/code&gt; and &lt;code&gt;OfferTargeting&lt;/code&gt;, which today accepts only &lt;code&gt;PrivateOffer&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Service role:&lt;/strong&gt; both accounts need &lt;code&gt;AWSMarketplaceSubsidiaryAuthorizationServiceRole&lt;/code&gt;, which AWS uses to create the resource share between them. Without it, in the console the authorization is created and never becomes active, and the subsidiary sees nothing. Through the API the error shows up as &lt;code&gt;INCOMPATIBLE_ACCOUNT_STATE&lt;/code&gt; on the primary or &lt;code&gt;INCOMPATIBLE_SUBSIDIARY_ACCOUNT_STATE&lt;/code&gt; on the subsidiary, and only in &lt;code&gt;DescribeChangeSet&lt;/code&gt;, because validation is asynchronous.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Human IAM:&lt;/strong&gt; separate from the service role, and you need both. To authorize, the primary uses &lt;code&gt;AWSMarketplaceSellerFullAccess&lt;/code&gt; or &lt;code&gt;AWSMarketplaceSellerProductsFullAccess&lt;/code&gt;; to create an offer from the Private offers page, the subsidiary uses &lt;code&gt;AWSMarketplaceSellerOfferManagement&lt;/code&gt;. Those managed policies are broad. In production I would write a custom policy restricted to the Catalog API actions each team actually uses.&lt;/p&gt;

&lt;h2&gt;
  
  
  Who controls what: product in the primary, contract and money in the subsidiary
&lt;/h2&gt;

&lt;p&gt;The diagram answers one question: where the control boundary sits between accounts. The primary owns the product and the authorization; the subsidiary is the seller of record, sets the price and gets paid. The dashed line is reporting visibility only and moves no money. Simplification: the Partner Central connection and the service role are implicit in the sharing arrow.&lt;/p&gt;

&lt;h3&gt;
  
  
  Primary account
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Authorization repo Catalog API (ci)&lt;/li&gt;
&lt;li&gt;Product listing SaaSProduct (data)&lt;/li&gt;
&lt;li&gt;SubsidiaryAuthorization Draft, Active, Restricted (security)&lt;/li&gt;
&lt;li&gt;Marketplace insights consolidated (data)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Subsidiary account
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Private offer seller of record (compute)&lt;/li&gt;
&lt;li&gt;Disbursement local bank, currency (storage)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Buyer
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Buyer account tax address (user)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Flows
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;pipe -&amp;gt; authz: StartChangeSet&lt;/li&gt;
&lt;li&gt;authz -&amp;gt; prod: references ARN&lt;/li&gt;
&lt;li&gt;authz -&amp;gt; offer: scopes countries and dimensions&lt;/li&gt;
&lt;li&gt;offer -&amp;gt; buyer: private offer&lt;/li&gt;
&lt;li&gt;buyer -&amp;gt; bank: pays via AWS&lt;/li&gt;
&lt;li&gt;offer -&amp;gt; insights: reports&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Where the risk lives: revoking is not switching off
&lt;/h2&gt;

&lt;p&gt;The announcement says that revoking an authorization immediately stops the subsidiary from creating new offers. Correct, and incomplete for anyone designing a control. The documentation details four behaviors that change the runbook.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Published offers survive:&lt;/strong&gt; revoking does not cancel offers already published. The buyer can still accept an offer the subsidiary published before revocation, and existing agreements continue under their agreed terms, including renewals and disbursements to the subsidiary. If the reason for revoking is the divestiture of the entity, a restriction in a country or a corporate dispute, the block does not reach the exposure already out in the market.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;Restricted&lt;/code&gt; is terminal:&lt;/strong&gt; there is no reactivation. To authorize again, you create a new authorization, which gets a new ID. Any system using the authorization ID as a business key breaks on the first accidental revocation.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The connection takes everything down:&lt;/strong&gt; ending the connection in Partner Central ends every authorization between the two accounts. An analyst who removes a connection to fix revenue attribution takes down the selling capability of the whole entity.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Narrowing breaks drafts:&lt;/strong&gt; removing a country or a dimension makes publishing fail for draft offers that used that scope. The offer stays parked and the local seller only finds out when trying to publish, usually the day before quarter close.&lt;/p&gt;

&lt;p&gt;The practical lesson: revocation is a process, not a click. Revoke the authorization, list the subsidiary's published offers not yet accepted, decide what happens to each one, and only then touch the connection. To shorten the exposure window, use short expiration dates on the private offers subsidiaries publish.&lt;/p&gt;

&lt;h2&gt;
  
  
  Authorization as code: &lt;code&gt;UpdateInformation&lt;/code&gt; replaces everything
&lt;/h2&gt;

&lt;p&gt;With more than two subsidiaries, console-made authorizations drift within weeks. The Catalog API lets you treat each authorization as a versioned resource, and three details make that mandatory, not cosmetic.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Full replacement:&lt;/strong&gt; &lt;code&gt;UpdateInformation&lt;/code&gt; replaces &lt;code&gt;Name&lt;/code&gt;, &lt;code&gt;Description&lt;/code&gt;, &lt;code&gt;Resource&lt;/code&gt; and &lt;code&gt;Terms&lt;/code&gt; wholesale; anything you omit is removed. Synchronous validation returns 422 if &lt;code&gt;Dimensions&lt;/code&gt; is empty, but a &lt;code&gt;BuyerTargetingTerm&lt;/code&gt; without &lt;code&gt;CountryCodes&lt;/code&gt; passes, and an empty list means all countries. A script that only meant to change the description can open the Brazilian entity's sales to the whole world. The only safe approach is to always send the complete state, read from a versioned file.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The default sits on the wrong side:&lt;/strong&gt; buyer eligibility is evaluated against the buyer's tax address. An authorization with no countries contradicts the fiscal premise that justified the local entity in the first place. In the pipeline, I would block the merge of any authorization with an empty &lt;code&gt;CountryCodes&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Products do not propagate:&lt;/strong&gt; adding a dimension to the product does not reach the subsidiary; you have to call &lt;code&gt;UpdateInformation&lt;/code&gt; with the new list. If the product has &lt;code&gt;Metered&lt;/code&gt; or &lt;code&gt;ExternallyMetered&lt;/code&gt; dimensions, all of them must be in the authorization, otherwise the change set fails with &lt;code&gt;INCOMPATIBLE_PRODUCT_DIMENSIONS&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The rest is idempotency and observability. &lt;code&gt;StartChangeSet&lt;/code&gt; is asynchronous: store the &lt;code&gt;ChangeSetId&lt;/code&gt;, poll &lt;code&gt;DescribeChangeSet&lt;/code&gt; until &lt;code&gt;SUCCEEDED&lt;/code&gt; or &lt;code&gt;FAILED&lt;/code&gt;, and read the &lt;code&gt;ErrorDetailList&lt;/code&gt;. Two authorizations for the same product and subsidiary cannot overlap countries and dimensions (&lt;code&gt;DUPLICATE_SUBSIDIARY_AUTHORIZATION&lt;/code&gt;), which gives you a natural key for your state: product, subsidiary and country set, not the ID. Reconcile daily with &lt;code&gt;ListEntities&lt;/code&gt; using &lt;code&gt;EntityType: SubsidiaryAuthorization&lt;/code&gt; and diff against the repository.&lt;/p&gt;

&lt;h2&gt;
  
  
  Playbook for tomorrow
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Map entity, country and currency&lt;/strong&gt;: Build the entity by country by currency table and confirm every subsidiary has completed its tax interview, KYC and payment setup, with a bank account and at least one disbursement currency. Without payment setup the subsidiary cannot create offers, even with an active authorization. In India, the account must be standalone, outside AWS Organizations, disburses only in INR and sells only to buyers in India.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Create the service role on both ends&lt;/strong&gt;: In each account, in Partner Central, Settings, Service roles tab, create &lt;code&gt;AWSMarketplaceSubsidiaryAuthorizationServiceRole&lt;/code&gt; before the first authorization. Without it the authorization never activates and the console gives no warning.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Accept the connection with legal sign-off&lt;/strong&gt;: Acceptance records consent to share attributed revenue and insights reports with the primary. In a group with minority shareholders or a joint venture, that is a corporate decision, not an IT one.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Version every authorization&lt;/strong&gt;: One file per product and subsidiary, with explicit &lt;code&gt;CountryCodes&lt;/code&gt; and the complete &lt;code&gt;Dimensions&lt;/code&gt; list. The pipeline sends the whole state via &lt;code&gt;UpdateInformation&lt;/code&gt;, polls &lt;code&gt;DescribeChangeSet&lt;/code&gt; and fails the build on any non-empty &lt;code&gt;ErrorDetailList&lt;/code&gt;.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Release in two steps&lt;/strong&gt;: Create in &lt;code&gt;Draft&lt;/code&gt;, review &lt;code&gt;DescribeEntity&lt;/code&gt; against the file and only then call &lt;code&gt;UpdateAvailability&lt;/code&gt; with &lt;code&gt;Active&lt;/code&gt;. Activation takes a few minutes; do not promise the local team the offer goes out the same minute.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Write the revocation runbook&lt;/strong&gt;: Revoke the authorization, list published and unaccepted offers, decide what happens to each one, and only then touch the connection. Include who approves, because &lt;code&gt;Restricted&lt;/code&gt; cannot be undone.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Reconcile daily&lt;/strong&gt;: &lt;code&gt;ListEntities&lt;/code&gt; with &lt;code&gt;SubsidiaryAuthorizationFilters&lt;/code&gt; by status and account, diffed against the repository. Use the primary's consolidated Marketplace insights to cross-check offers, agreements and renewals per subsidiary.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;The authorization name is an interface:&lt;/strong&gt; The subsidiary picks the authorization by its &lt;code&gt;Name&lt;/code&gt; (1 to 100 characters) when creating an offer. Put the product and scope in the name, for example &lt;code&gt;Security Suite | BR, AR | users, api&lt;/code&gt;, and generate it from the versioned file. A hand-written name drifts from the real scope on the first change, and the local seller picks the wrong authorization without noticing.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Three ways to sell through more than one entity
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Criterion&lt;/th&gt;
&lt;th&gt;Subsidiary authorization&lt;/th&gt;
&lt;th&gt;Resale authorization (CPPO)&lt;/th&gt;
&lt;th&gt;Duplicate listing&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Who sells&lt;/td&gt;
&lt;td&gt;Another seller account of your own group&lt;/td&gt;
&lt;td&gt;Third-party Channel Partner&lt;/td&gt;
&lt;td&gt;Each account, with its own product&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Price&lt;/td&gt;
&lt;td&gt;Subsidiary sets it; the authorization carries no price&lt;/td&gt;
&lt;td&gt;Markup over the wholesale set by the ISV&lt;/td&gt;
&lt;td&gt;Each account sets it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Currency&lt;/td&gt;
&lt;td&gt;Those configured in the subsidiary itself&lt;/td&gt;
&lt;td&gt;The one the ISV defines in the authorization&lt;/td&gt;
&lt;td&gt;The account's own&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Offer types&lt;/td&gt;
&lt;td&gt;Direct private only&lt;/td&gt;
&lt;td&gt;Private through the partner&lt;/td&gt;
&lt;td&gt;Public and private&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Maintenance cost&lt;/td&gt;
&lt;td&gt;1 listing + N versioned authorizations&lt;/td&gt;
&lt;td&gt;1 listing + one authorization per partner&lt;/td&gt;
&lt;td&gt;N listings synced by hand&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Use when&lt;/td&gt;
&lt;td&gt;Local entity must be seller of record and sells through private offers&lt;/td&gt;
&lt;td&gt;A third party resells and needs margin&lt;/td&gt;
&lt;td&gt;The local entity needs a public offer or its own CPPO&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Anti-patterns I would expect in the first quarter
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Authorization without &lt;code&gt;CountryCodes&lt;/code&gt;&lt;/strong&gt;: an empty list means all countries, and the entity created for one fiscal market can now sell to any tax address.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treating revocation as a kill switch&lt;/strong&gt;: offers already published remain acceptable and agreements keep renewing; the block applies only to new offers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Partial updates by script&lt;/strong&gt;: &lt;code&gt;UpdateInformation&lt;/code&gt; removes everything not sent, so a description patch can wipe countries and dimensions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Removing the connection to fix attribution&lt;/strong&gt;: ending the connection ends every authorization between the two accounts at once.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Authorization ID as a business key&lt;/strong&gt;: &lt;code&gt;Restricted&lt;/code&gt; is terminal and re-authorizing creates a new ID; key on product, subsidiary and countries.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Shipping a new dimension and forgetting the authorization&lt;/strong&gt;: the subsidiary keeps selling the old product version until someone calls &lt;code&gt;UpdateInformation&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Questions that will come up
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Can the subsidiary create a CPPO for a local channel with the authorized product?
&lt;/h3&gt;

&lt;p&gt;No. The Catalog API rejects a resale authorization created from a product the primary authorized. The subsidiary can only create CPPOs for products it owns.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can I migrate agreements from the duplicate listing to the primary's product?
&lt;/h3&gt;

&lt;p&gt;No. They stay on the original listing. Plan for coexistence until expiry and use the authorization at renewal time, as a new offer.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can the primary create offers or complete KYC on the subsidiary's behalf?
&lt;/h3&gt;

&lt;p&gt;No. Each account creates its own offers and completes its own tax interview and KYC. The service role also gives the primary no access to the subsidiary's offers, agreements or financial data.&lt;/p&gt;

&lt;h3&gt;
  
  
  Are Marketplace fees consolidated in the primary?
&lt;/h3&gt;

&lt;p&gt;No. Listing and transaction fees are charged to the transacting subsidiary and are not pooled, netted or rebilled across accounts. Your transfer pricing model still lives outside the Marketplace.&lt;/p&gt;

&lt;h3&gt;
  
  
  What does it cost and where is it available?
&lt;/h3&gt;

&lt;p&gt;No additional cost, in all Regions where AWS Marketplace is available, according to the 10/29/2026 announcement.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Curator's note:&lt;/strong&gt; I would start with the lowest-volume subsidiary, a single product and two countries, and only move to the next one after running the revocation runbook end to end in a test setup. Before the first active authorization, the pipeline would already be blocking empty &lt;code&gt;CountryCodes&lt;/code&gt; and partial updates. The hard-won lesson I bring from financial platforms: the control that causes the most trouble is not the one that blocks, it is the one that looks like it blocks and does not. Here, that control is called revocation, and its name promises more than it delivers.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Verdict
&lt;/h2&gt;

&lt;p&gt;Adopt subsidiary authorization when the group has local entities that must be seller of record, sells mostly through private offers and maintains duplicate listings today: the maintenance gain is direct. Do it with authorizations in a repository, &lt;code&gt;CountryCodes&lt;/code&gt; always explicit and a revocation runbook that handles published offers. Keep the local listing when the entity needs a public offer or its own CPPO, because neither is supported in this release.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Rating:&lt;/strong&gt; Adote com runbook&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/about-aws/whats-new/2026/10/aws-marketplace-subsidiary-account-authorization" rel="noopener noreferrer"&gt;AWS What's New: AWS Marketplace now enables partners to list once and sell through multiple subsidiary accounts (29/10/2&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/marketplace/latest/userguide/subsidiary-offers.html" rel="noopener noreferrer"&gt;AWS Marketplace Seller Guide: Creating private offers as a subsidiary account&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/partner-central/latest/getting-started/manage-subsidiary.html" rel="noopener noreferrer"&gt;AWS Partner Central: Managing AWS subsidiary account connections&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/marketplace/latest/developerguide/work-with-subsidiary-authorizations.html" rel="noopener noreferrer"&gt;AWS Marketplace API: Work with subsidiary authorizations&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/marketplace/latest/userguide/using-roles-for-subsidiary-authorization.html" rel="noopener noreferrer"&gt;AWS Marketplace Seller Guide: Subsidiary authorization service role&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/marketplace/latest/userguide/channel-partner-offers.html" rel="noopener noreferrer"&gt;AWS Marketplace Seller Guide: Creating private offers as an AWS Marketplace Channel Partner&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/marketplace/latest/userguide/getting-started-seller-india.html" rel="noopener noreferrer"&gt;AWS Marketplace Seller Guide: Getting started as a seller in India&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://fernando.moretes.com/blog/uma-listagem-varias-entidades-juridicas-subsidiarias-no-aws" rel="noopener noreferrer"&gt;fernando.moretes.com&lt;/a&gt;. By Fernando F. Azevedo: Senior Solutions Architect.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>awscloud</category>
      <category>awsmarketplace</category>
      <category>governance</category>
      <category>catalogapi</category>
    </item>
    <item>
      <title>AWS Builder Loft São Paulo: a new home for building together</title>
      <dc:creator>Fernando Azevedo</dc:creator>
      <pubDate>Fri, 30 Oct 2026 17:14:04 +0000</pubDate>
      <link>https://dev.to/fernando_azevedo_6844e930/aws-builder-loft-sao-paulo-a-new-home-for-building-together-16eg</link>
      <guid>https://dev.to/fernando_azevedo_6844e930/aws-builder-loft-sao-paulo-a-new-home-for-building-together-16eg</guid>
      <description>&lt;p&gt;São Paulo has a new home for builders. I left thinking about what we could build together. On the evening of October 29, the talks, conversations and community encounters at the AWS Builder Loft São Paulo opening left me with a welcome feeling: many people are ready to learn, share what they know and help an idea become something real.&lt;/p&gt;

&lt;p&gt;&lt;a href="/community-evidence/aws-builder-loft-sao-paulo-2026-10-29.jpg" class="article-body-image-wrapper"&gt;&lt;img src="/community-evidence/aws-builder-loft-sao-paulo-2026-10-29.jpg" alt="My record of the opening: the stage, the people, details of the space and the joy of being part of this beginning. São Paulo, October 29, 2026."&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Fernando Azevedo&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What I took home from that evening
&lt;/h2&gt;

&lt;p&gt;As a solutions architect and an AWS Community Builder in AI Engineering, I spend much of my time connecting real problems with solutions that work. Good architecture takes questions, context and a willingness to reconsider choices. A good community needs those things too.&lt;/p&gt;

&lt;p&gt;That was what excited me most that evening. Being with people who want to create makes room for conversations that are difficult to fit into a presentation or a quick exchange of messages. A question reaches someone who has faced the problem before. One person's experience gives someone else the confidence to begin. An idea gains a second perspective.&lt;/p&gt;

&lt;p&gt;I left happy to have been part of this beginning and keen to keep contributing. The collage accompanying this piece preserves some of that memory: the talks, people gathering, the space and the small details that make an encounter memorable.&lt;/p&gt;

&lt;h2&gt;
  
  
  A place designed for continuity
&lt;/h2&gt;

&lt;p&gt;The idea behind the Loft was already present in the &lt;a href="https://aws.amazon.com/blogs/aws/in-the-works-aws-builder-lofts-in-berlin-hyderabad-and-sao-paulo/" rel="noopener noreferrer"&gt;official AWS announcement of the expansion to Berlin, Hyderabad and São Paulo&lt;/a&gt;: permanent spaces for learning, collaboration and community activities. For São Paulo, the announcement highlights connections with local communities, universities and startup networks.&lt;/p&gt;

&lt;p&gt;That continuity matters. Having somewhere to return gives a conversation a next chapter. Someone can arrive with a question, try an approach and come back with what worked, or what needs to change. Knowledge travels with the people involved.&lt;/p&gt;

&lt;p&gt;I see particular value in bringing beginners together with people who have production experience. Beginners ask questions that make us explain things better. People who have dealt with incidents and difficult decisions can share the precautions that do not always appear in a demonstration.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;A conversation can become a partnership. A question can open the way to a solution.&lt;br&gt;
Fernando Azevedo&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Why this matters for Brazil
&lt;/h2&gt;

&lt;p&gt;In my view, a space like this can connect people working on similar challenges who have not yet met. Developers, architects, students, data professionals, security specialists and founders have much to learn from one another. Bringing those perspectives together makes it easier to understand the problem before choosing the technology.&lt;/p&gt;

&lt;p&gt;I also see an opportunity to create more knowledge grounded in our own circumstances. Costs, operations, accessibility, team development and the needs of Brazilian users belong in technical discussions. Sharing those experiences in Portuguese helps more people participate; publishing them in English as well broadens the exchange with the global community.&lt;/p&gt;

&lt;p&gt;The work can reach beyond people who are able to be physically present in São Paulo. A gathering can lead to a public repository, an accessible guide, a recording or an account that explains a decision. That is how I imagine the Loft's energy reaching other cities too: through useful materials, collaboration between groups and people willing to carry ideas forward.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I would like to bring to the table
&lt;/h2&gt;

&lt;p&gt;I want to contribute what I have been studying and building in architecture, cloud and artificial intelligence, and learn from people with different experiences. In AI Engineering, I see room for sessions where we examine a real solution: the problem it solves, its limits, how to test it, what it costs to operate and what happens when it fails.&lt;/p&gt;

&lt;p&gt;One idea would be a hands-on session about agent tools, using a small example that anyone can reproduce. Another would be an architecture discussion where the group compares alternatives and records the decision. These are ideas I would like to develop with the community, not activities already scheduled at the Loft.&lt;/p&gt;

&lt;p&gt;My &lt;a href="https://fernando.moretes.com/mcp" rel="noopener noreferrer"&gt;open AWS architecture MCP&lt;/a&gt;, &lt;a href="https://fernando.moretes.com/studies" rel="noopener noreferrer"&gt;architecture studies&lt;/a&gt; and &lt;a href="https://fernando.moretes.com/blog" rel="noopener noreferrer"&gt;blog articles&lt;/a&gt; can provide starting points. The best outcome would be someone testing an example, finding a limitation, asking a better question and helping us improve the material.&lt;/p&gt;

&lt;h2&gt;
  
  
  From an encounter to a next step
&lt;/h2&gt;

&lt;p&gt;For someone just starting out, my invitation is simple: bring a real question. You do not need a finished project or knowledge of every service. Explaining what you tried and where you got stuck is already a good start to a conversation.&lt;/p&gt;

&lt;p&gt;For those with more experience, it is worth bringing the difficult choices too: the alternative you rejected, the unexpected cost, the failure that changed the design. These accounts help others move forward with a clearer understanding.&lt;/p&gt;

&lt;p&gt;When a conversation produces an idea, we can choose a small next step: write an example, review a diagram, test a hypothesis or open a repository discussion. Recording what we learn and crediting contributors keeps the encounter going. &lt;a href="https://builder.aws.com/" rel="noopener noreferrer"&gt;AWS Builder Center&lt;/a&gt; and &lt;a href="https://builder.aws.com/community/user-groups" rel="noopener noreferrer"&gt;AWS User Groups&lt;/a&gt; offer ways to continue that exchange. To attend future in-person activities, check the announcements and registration details for the local edition.&lt;/p&gt;

&lt;h2&gt;
  
  
  Thank you for being part of this beginning
&lt;/h2&gt;

&lt;p&gt;My thanks to the AWS team for the invitation and organization, to the speakers, AWS Heroes, community leaders and everyone who took part in that evening. It was a pleasure to be there with you.&lt;/p&gt;

&lt;p&gt;I left AWS Builder Loft São Paulo grateful and eager to keep the conversation going. I want to continue contributing alongside AWS builders, users and customers, sharing what I learn and listening to the problems we still need to solve.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What idea, challenge or project would you bring to the table?&lt;/strong&gt; It could be an unanswered question, a prototype that needs another perspective or a topic you would like to teach. Let's start there.&lt;/p&gt;

&lt;p&gt;Let's learn, share and build together. There is much to look forward to in the AWS community in Brazil.&lt;/p&gt;

&lt;h2&gt;
  
  
  Background and ways to stay involved
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/blogs/aws/in-the-works-aws-builder-lofts-in-berlin-hyderabad-and-sao-paulo/" rel="noopener noreferrer"&gt;AWS News Blog, anúncio da expansão dos Builder Lofts&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://builder.aws.com/" rel="noopener noreferrer"&gt;AWS Builder Center&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://builder.aws.com/community/user-groups" rel="noopener noreferrer"&gt;AWS User Groups&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://fernando.moretes.com/blog/aws-builder-loft-sao-paulo-comunidade-que-constroi" rel="noopener noreferrer"&gt;fernando.moretes.com&lt;/a&gt;. By Fernando F. Azevedo: Senior Solutions Architect.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>awscloud</category>
      <category>awsbuilderloft</category>
      <category>comunidadeaws</category>
      <category>sopaulo</category>
    </item>
    <item>
      <title>Your agent crashed after the write. Should you retry?</title>
      <dc:creator>Fernando Azevedo</dc:creator>
      <pubDate>Tue, 27 Oct 2026 16:46:39 +0000</pubDate>
      <link>https://dev.to/fernando_azevedo_6844e930/your-agent-crashed-after-the-write-should-you-retry-232p</link>
      <guid>https://dev.to/fernando_azevedo_6844e930/your-agent-crashed-after-the-write-should-you-retry-232p</guid>
      <description>&lt;p&gt;The hardest moment in an automation often sits between two acknowledgements: the target accepted the change, but the executor has not recorded completion. Restarting the agent restores process availability; it does not resolve uncertainty about the previous effect. In this lab, I terminate the process in that exact interval and inspect what survives. The next decision depends on persistent evidence, not the model's confidence.&lt;/p&gt;

&lt;h2&gt;
  
  
  What was verified
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Implementation:&lt;/strong&gt; lab-security-remediation, tag v0.4.0; local tests executed on October 27, 2026.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Observed result:&lt;/strong&gt; 22 tests passed. In the receipt exercise: version 2, one receipt, and no second change after recovery and resubmission.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scope:&lt;/strong&gt; Local SQLite, synthetic target, simulated review. No credentials or AWS calls required.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Not demonstrated:&lt;/strong&gt; Does not establish exactly-once execution across remote APIs, power-loss resilience, or production-ready authorization.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Start with the question the log cannot answer
&lt;/h2&gt;

&lt;p&gt;Imagine a tool that fixes a configuration. It receives a reviewed proposal, checks the target version, and applies the change. Before returning success, the process terminates. The client sees an interruption. The target may already have changed. A log line saying the call started cannot distinguish those possibilities.&lt;/p&gt;

&lt;p&gt;In this exercise, the target is only a SQLite row representing a synthetic bucket. The &lt;code&gt;blockPublicAccess&lt;/code&gt; property is a teaching device; this is not an S3 call. The proposal identifies the operation and the expected version. An intent record, called the ledger, stores the reviewed request. A receipt records that this request produced this target version.&lt;/p&gt;

&lt;p&gt;That separation creates a testable question: &lt;strong&gt;what evidence connects this operation to this change?&lt;/strong&gt; Seeing the desired configuration does not tell us who changed it. Finding a restarted process does not tell us whether its predecessor finished. A client error describes the client's observation, not necessarily the target's reality.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where evidence must survive
&lt;/h2&gt;

&lt;p&gt;Local lab architecture. The tables share one SQLite file, but intent, effect, and completion use separate transactions. The target update and its receipt are atomic with each other.&lt;/p&gt;

&lt;h3&gt;
  
  
  Local CLI
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Reviewed executor (compute)&lt;/li&gt;
&lt;li&gt;Recovery reader (compute)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  SQLite · separate commits
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Operation ledger (storage)&lt;/li&gt;
&lt;li&gt;Target + receipt (data)&lt;/li&gt;
&lt;li&gt;Audit events (storage)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Flows
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;executor -&amp;gt; ledger: INTENT&lt;/li&gt;
&lt;li&gt;executor -&amp;gt; target: EFFECT&lt;/li&gt;
&lt;li&gt;target -&amp;gt; recovery: EVIDENCE&lt;/li&gt;
&lt;li&gt;ledger -&amp;gt; recovery: REQUEST&lt;/li&gt;
&lt;li&gt;recovery -&amp;gt; ledger: CLASSIFIES&lt;/li&gt;
&lt;li&gt;recovery -&amp;gt; events: DECISION&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Prepare a run another person can reproduce
&lt;/h2&gt;

&lt;p&gt;Use Node.js 22.13 or later with &lt;code&gt;node:sqlite&lt;/code&gt; available. The tagged source is the reference for this exercise; do not assume the published npm version is identical. Installing dependencies is not required to run this lab. Clone into a new directory, check the tag, and run the tests before changing a scenario.&lt;/p&gt;

&lt;p&gt;Choose fresh database names for each complete repetition. A reused database retains history, and that history changes the expected result. Preserve the original proposal too: generating another after the change can produce a different digest for the same identifier and should cause a conflict, not implicit permission to execute again.&lt;/p&gt;

&lt;p&gt;The intentional failures are separate from inspection commands. In a script using &lt;code&gt;set -e&lt;/code&gt;, explicitly capture exit code 86 so the expected termination does not end the investigation. The uncertainty exercise returns 2, another expected outcome. These codes are evidence; do not erase them with an indiscriminate &lt;code&gt;|| true&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Setup: pinned source and local tests&lt;/em&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/fernando-moretes/lab-security-remediation.git
&lt;span class="nb"&gt;cd&lt;/span&gt; lab-security-remediation
git checkout v0.4.0
node --version
npm &lt;span class="nb"&gt;test&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Experiment 1: the effect exists; the acknowledgement was lost
&lt;/h2&gt;

&lt;p&gt;Create the database and export the proposal. Running with &lt;code&gt;--approve&lt;/code&gt; simulates explicit review and records intent. The &lt;code&gt;after-target&lt;/code&gt; crash point terminates the process after the commit that updates the target and writes its receipt, before marking the operation complete. This is not an exception caught inside one function: the CLI exits, and subsequent commands open the database in other processes.&lt;/p&gt;

&lt;p&gt;After the crash, inspect three things together: operation status, target version, and receipt existence. In the October 27 run, the operation was &lt;code&gt;pending&lt;/code&gt;, the target was at version 2 with &lt;code&gt;blockPublicAccess: true&lt;/code&gt;, and one receipt matched the proposal digest. That combination demonstrates why retrying solely because the ledger is pending would be an uninformed decision.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Intentional crash: run exits 86; continue with inspect&lt;/em&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;node bin/cli.mjs durable init ./receipt.sqlite
node bin/cli.mjs durable plan ./receipt.sqlite &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; receipt-proposal.json
node bin/cli.mjs durable run ./receipt.sqlite receipt-proposal.json --approve --crash after-target
&lt;span class="c"&gt;# Expected exit: 86&lt;/span&gt;
node bin/cli.mjs durable inspect ./receipt.sqlite
node bin/cli.mjs durable recover ./receipt.sqlite synthetic-finding-001
node bin/cli.mjs durable run ./receipt.sqlite receipt-proposal.json --approve
node bin/cli.mjs durable inspect ./receipt.sqlite
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Recovering the record does not require repeating the effect
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;recover&lt;/code&gt; command reads the operation, receipt, and current state. Here it changes the classification to &lt;code&gt;reconciled&lt;/code&gt; and records the decision in the event history. It does not mutate the target. Resubmitting the original proposal also returns &lt;code&gt;reconciled&lt;/code&gt;, with &lt;code&gt;retryable: false&lt;/code&gt;. In the final inspection, the version remains 2 and the receipt count remains 1.&lt;/p&gt;

&lt;p&gt;Those are the invariants worth checking. The test is not simply searching for the word success in the output. Compare the observable effect before and after recovery. Compare it again before and after resubmission. If the version increased again, an apparently correct status would hide a second execution.&lt;/p&gt;

&lt;p&gt;A receipt is historical evidence, not a permanent health certificate. If the target changes later, the earlier operation's history can remain valid while current state diverges. The implementation exposes &lt;code&gt;driftAfterReceipt&lt;/code&gt; to distinguish those questions. Restoring a desired configuration, when necessary, requires a new reviewed decision about current state.&lt;/p&gt;

&lt;h2&gt;
  
  
  Observed results from both experiments
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Inspection&lt;/th&gt;
&lt;th&gt;With receipt&lt;/th&gt;
&lt;th&gt;Without receipt, with drift&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;After interruption / drift&lt;/td&gt;
&lt;td&gt;pending; version 2; 1 receipt&lt;/td&gt;
&lt;td&gt;pending; version 2; 0 receipts&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Recovery&lt;/td&gt;
&lt;td&gt;reconciled; exit 0&lt;/td&gt;
&lt;td&gt;uncertain; exit 2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Original proposal resubmission&lt;/td&gt;
&lt;td&gt;reconciled; version remains 2&lt;/td&gt;
&lt;td&gt;uncertain; version remains 2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cloud calls&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Experiment 2: the state looks right; ownership is unknown
&lt;/h2&gt;

&lt;p&gt;Now use another database. Interrupt execution at &lt;code&gt;after-intent&lt;/code&gt;, before applying the effect. Then use the teaching command &lt;code&gt;drift&lt;/code&gt; to change the target through another path. During recovery, the desired property is true and the version is 2, just as in the first experiment. The decisive difference is the missing receipt.&lt;/p&gt;

&lt;p&gt;The observed result is &lt;code&gt;uncertain&lt;/code&gt;, &lt;code&gt;retryable: false&lt;/code&gt;, with exit code 2. Resubmitting the original proposal preserves that classification and does not mutate the target again. The system does not infer ownership from matching values. This behavior trades automatic progress for a decision that can be investigated.&lt;/p&gt;

&lt;p&gt;Do not delete the operation to unblock a queue. That removes the memory used to recognize duplicates. Preserve the database, proposal, and results; investigate the change and define the next step as a new decision. The lab offers no magic administrative resolution button and should not suggest that manually editing the ledger is a safe operating procedure.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The final configuration matches, but the receipt is missing&lt;/em&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;node bin/cli.mjs durable init ./uncertain.sqlite
node bin/cli.mjs durable plan ./uncertain.sqlite &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; uncertain-proposal.json
node bin/cli.mjs durable run ./uncertain.sqlite uncertain-proposal.json --approve --crash after-intent
&lt;span class="c"&gt;# Expected exit: 86&lt;/span&gt;
node bin/cli.mjs durable drift ./uncertain.sqlite &lt;span class="kc"&gt;true&lt;/span&gt;
node bin/cli.mjs durable inspect ./uncertain.sqlite
node bin/cli.mjs durable recover ./uncertain.sqlite synthetic-finding-001
&lt;span class="c"&gt;# Expected exit: 2, status: uncertain&lt;/span&gt;
node bin/cli.mjs durable run ./uncertain.sqlite uncertain-proposal.json --approve
&lt;span class="c"&gt;# Expected exit: 2; no additional target change&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Three commits explain the failure windows
&lt;/h2&gt;

&lt;p&gt;The implementation persists reviewed intent in one transaction. In another, it compares the expected version and writes the target together with its receipt. In a third, it marks completion in the ledger. The receipt and change are atomic with each other; the whole execution is not one transaction. That choice deliberately creates the window we just tested.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;BEGIN IMMEDIATE&lt;/code&gt; serializes writers to the local file with a bounded contention wait. Storage uses WAL and &lt;code&gt;synchronous=FULL&lt;/code&gt;. These choices give the exercise transactional semantics, but process-termination tests are not power-loss, disk, corruption, or network-filesystem tests. The cost of serialization matters too: one local file does not represent distributed scalability.&lt;/p&gt;

&lt;p&gt;The suite covers other boundaries: crashes before the target commit, before the completion commit, and after completion; concurrent processes; repeated identifiers with different payloads; unreviewed proposals and state changes. The 22 passing tests check these properties in the lab. They are not an availability percentage or a production agent success rate.&lt;/p&gt;

&lt;h2&gt;
  
  
  C4
&lt;/h2&gt;

&lt;p&gt;A person runs a recovery lab and inspects synthetic evidence. The model calls no AWS services and requires no LLM.&lt;/p&gt;

&lt;h3&gt;
  
  
  Containers
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;CLI&lt;/strong&gt; (Node.js): Independent processes for plan, run, inspect, and recover.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Persistent store&lt;/strong&gt; (SQLite local file): Operations, target, receipts, and history persisted in one file.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Components
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Proposal validation&lt;/strong&gt;: Validates the request, simulated review, and expected version.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Conditional mutation&lt;/strong&gt;: Updates target and receipt in the same commit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Recovery classifier&lt;/strong&gt;: Reads evidence and records the classification without executing the mutation.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What changes when the tool calls AWS
&lt;/h2&gt;

&lt;p&gt;In a real integration, recording intent in a database and calling a remote API are operations in different systems. The local commit does not encompass the API. The lab deliberately keeps target and receipt together so evidence is inspectable; moving the design to the cloud requires reassessing that boundary.&lt;/p&gt;

&lt;p&gt;The AWS Builders' Library explains the value of an explicit client-provided identifier for recognizing retry intent. Check the concrete operation's idempotency contract: scope, token lifetime, behavior with changed parameters, and repeated-call responses. Do not infer a universal guarantee because another service accepts a token.&lt;/p&gt;

&lt;p&gt;For each mutable tool, document which remote evidence is available, how long it remains queryable, and who resolves inconclusive outcomes. When the target cannot provide sufficient proof, retain a visible uncertainty state. Adding a queue, an agent framework, or an MCP server does not independently close the gap between effect and acknowledgement.&lt;/p&gt;

&lt;p&gt;Separate current authorization from historical evidence too. In this lab, &lt;code&gt;--approve&lt;/code&gt; is only a local review simulation. A real service must verify identity, scope, and validity before mutation; revoked approval must not become permission for a new execution just because an old record exists. A recovery read and a mutation attempt require different decisions.&lt;/p&gt;

&lt;h2&gt;
  
  
  Decision: uncertainty is an operational outcome
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Status:&lt;/strong&gt; accepted&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Context&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The client may lose the response after a committed effect at the target.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Decision&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Use persistent evidence to reconcile; block resubmission when the effect cannot be attributed to the operation. In this lab, recover never mutates the target.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Consequences&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Requires an owner for investigating uncertain states.&lt;/li&gt;
&lt;li&gt;Receipt and proposal retention is part of the recovery contract.&lt;/li&gt;
&lt;li&gt;May require intervention even when current state looks desirable.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Checklist for reviewing your own tool
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Define intent&lt;/strong&gt;: Choose a stable identifier for the same operation and an explicit rule for conflicting payloads. Save the original proposal and contract version.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Locate the real commit&lt;/strong&gt;: Mark where the target changes and where acknowledgement is persisted. Inject an interruption between them, away from production resources.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Compare effects&lt;/strong&gt;: Observe version, receipts, and history before and after replay. A success status alone does not demonstrate the absence of duplicates.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Exercise ambiguity&lt;/strong&gt;: Have another actor change the target without the operation's receipt. Verify that matching values do not become assumed ownership.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Define retention and operations&lt;/strong&gt;: Document the evidence retention window, investigation owner, and consistent database-copy procedure. Close processes before copying the lab file.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  A Well-Architected reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;operational-excellence&lt;/strong&gt;: Turn uncertain into an investigation queue with an owner and minimum evidence. Recovery needs a runbook and an exercise as well as observability.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;security&lt;/strong&gt;: Local review authenticates nobody. In production, protect the ledger, minimize sensitive data, and revalidate authorization for the next mutation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;reliability&lt;/strong&gt;: Test lost acknowledgement, not only unavailability before a call. Separate duplicate recognition, reconciliation, and a newly authorized attempt.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;performance&lt;/strong&gt;: Measure contention and reconciliation latency in the real architecture. SQLite writer serialization is not evidence of distributed throughput.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;cost&lt;/strong&gt;: The exercises make no AWS calls. For a remote implementation, estimate evidence storage and recovery queries; do not invent savings from this local test.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;sustainability&lt;/strong&gt;: Avoiding redundant effects and attempts reduces unnecessary work, but the lab measures neither energy nor emissions. Do not turn that hypothesis into a number.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Traps the exercise makes visible
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Retrying every interrupted call without consulting effect evidence.&lt;/li&gt;
&lt;li&gt;Treating the desired final value as proof that this operation produced it.&lt;/li&gt;
&lt;li&gt;Generating a new proposal after the change and calling it a retry of the original.&lt;/li&gt;
&lt;li&gt;Deleting receipts or ledger records without a policy that preserves the duplicate-recognition window.&lt;/li&gt;
&lt;li&gt;Presenting a local SQLite transaction as an exactly-once guarantee across remote services.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Diagnostic replay and mutation retry are different operations
&lt;/h2&gt;

&lt;p&gt;To share a reproducible case, include the code version, synthetic initial state, proposal, interruption point, exit codes, and before-and-after inspections. A terminal screenshot helps communicate; the files and commands make verification possible. Remove credentials and real data before opening an issue.&lt;/p&gt;

&lt;p&gt;The public evidence associated with this article contains results from both scenarios. The invitation is to discuss a concrete contract: which observation permits reconciliation, which permits another attempt, and which requires investigation? A small contribution, such as adding a failure case with an expected outcome, can be more useful than adding another tool to a catalog.&lt;/p&gt;

&lt;p&gt;If you want to experiment, run both scenarios in the repository and describe which failure boundary you would test next. To bring this into your AWS architecture, start with one mutable operation and document its idempotency and recovery contract. The test then becomes a property the team can review, beyond a generic agent demonstration.&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://fernando.moretes.com/community-evidence/2026-10-27-recovery-lab.json" rel="noopener noreferrer"&gt;Observed local results: October 27, 2026 (JSON)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/fernando-moretes/lab-security-remediation/blob/v0.4.0/DURABILITY.md" rel="noopener noreferrer"&gt;Source and exercise, lab v0.4.0&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/fernando-moretes/lab-security-remediation/blob/v0.4.0/test/durable.test.mjs" rel="noopener noreferrer"&gt;Executable recovery tests&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/" rel="noopener noreferrer"&gt;AWS Builders' Library: Making retries safe with idempotent APIs&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://sqlite.org/lang_transaction.html" rel="noopener noreferrer"&gt;SQLite transactions&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://nodejs.org/api/sqlite.html" rel="noopener noreferrer"&gt;Node.js SQLite API&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The question to bring to your review
&lt;/h2&gt;

&lt;p&gt;Before authorizing another attempt, ask for evidence that distinguishes a confirmed effect, an absent effect, and an unknown effect. In this lab, a receipt allows reconciliation without another execution; its absence in the presence of drift requires stopping. The value of the exercise is making that decision reproducible and reviewable by someone else.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://fernando.moretes.com/blog/agente-caiu-depois-da-mudanca-retry-recuperacao" rel="noopener noreferrer"&gt;fernando.moretes.com&lt;/a&gt;. By Fernando F. Azevedo: Senior Solutions Architect.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>aiagents</category>
      <category>idempotency</category>
      <category>crashrecovery</category>
      <category>mcp</category>
    </item>
    <item>
      <title>Migrating agents from GPT-5.6 Terra to GPT-6.1 Terra on Bedrock</title>
      <dc:creator>Fernando Azevedo</dc:creator>
      <pubDate>Tue, 27 Oct 2026 16:46:37 +0000</pubDate>
      <link>https://dev.to/fernando_azevedo_6844e930/migrating-agents-from-gpt-56-terra-to-gpt-61-terra-on-bedrock-1jod</link>
      <guid>https://dev.to/fernando_azevedo_6844e930/migrating-agents-from-gpt-56-terra-to-gpt-61-terra-on-bedrock-1jod</guid>
      <description>&lt;p&gt;The GPT-6.1 Terra announcement on Amazon Bedrock, on 10/26/2026, sells the switch as obvious: 15 percentage points above GPT-5.6 Terra on OpenAI's Agents' Last Exam and token prices at least 50% lower. The question I care about is not whether the new model is better, it's what breaks in your AWS account the day you change the model ID. I read the model card line by line against the 5.6 Terra card, and the answer has names: source Region, API choice, caching and quota. This is the path I would follow to migrate a production agent without learning any of that on call.&lt;/p&gt;

&lt;h2&gt;
  
  
  The starting point
&lt;/h2&gt;

&lt;p&gt;The scenario I use as a reference is common in financial environments: an internal agent that reads policies, queries two or three systems through tool-use and returns structured JSON to a Step Functions workflow. It runs on &lt;code&gt;bedrock-runtime&lt;/code&gt; with the &lt;code&gt;us.openai.gpt-5.6-terra&lt;/code&gt; profile, because the risk team does not accept global routing, and the US geographic profile, sourced from N. Virginia, stays within &lt;code&gt;us-east-1&lt;/code&gt;, &lt;code&gt;us-east-2&lt;/code&gt; and &lt;code&gt;us-west-2&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;A typical run has about 40 thousand input tokens, 30 thousand of which are a stable prefix (instructions, tool definitions, policy excerpts), and 3 thousand output tokens. At 5.6 Terra list price on US CRIS (US$ 2.20 per million input, US$ 0.22 per million cache read and US$ 13.20 per million output), that run costs roughly US$ 0.068 in steady state. It looks small until you multiply by 200 thousand runs a month: about US$ 13.6 thousand in inference alone.&lt;/p&gt;

&lt;p&gt;These numbers are arithmetic on published prices, not customer measurements. They exist to size the decision. What actually matters is the calendar: the 5.6 Terra model card guarantees EOL no sooner than 07/13/2027, with a legacy period of at least six months. There is time, but there is no slack for a rushed migration in the last quarter, which is exactly when the team is busy with something else.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the model card says and the announcement doesn't
&lt;/h2&gt;

&lt;p&gt;The announcement talks about capability, speed and price. The model card talks about operations, and that is where the migration risk lives. Reading both cards side by side, four differences change the design.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Region on &lt;code&gt;bedrock-mantle&lt;/code&gt;:&lt;/strong&gt; 5.6 Terra runs in-Region on the &lt;code&gt;bedrock-mantle&lt;/code&gt; endpoint in &lt;code&gt;us-east-1&lt;/code&gt;, &lt;code&gt;us-east-2&lt;/code&gt;, &lt;code&gt;us-west-2&lt;/code&gt; and both GovCloud Regions. 6.1 Terra, at launch, only in &lt;code&gt;us-east-1&lt;/code&gt;. If you point the OpenAI SDK at &lt;code&gt;bedrock-mantle.us-west-2.api.aws&lt;/code&gt;, you don't migrate by changing a string.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Source Region on &lt;code&gt;bedrock-runtime&lt;/code&gt;:&lt;/strong&gt; the 6.1 Terra table does not list &lt;code&gt;sa-east-1&lt;/code&gt; or &lt;code&gt;me-central-1&lt;/code&gt; as a source for the global profile, and it does not include the &lt;code&gt;in.&lt;/code&gt; geographic profile that 5.6 Terra offers for Mumbai and Hyderabad. If you call from São Paulo, this decides where the workload's client has to run.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Invoke API:&lt;/strong&gt; the 6.1 Terra API table marks &lt;code&gt;InvokeModel&lt;/code&gt; as supported on &lt;code&gt;bedrock-runtime&lt;/code&gt;; the 5.6 Terra table marks it as unsupported. That helps teams with legacy Invoke code, but it doesn't change the Guardrails and application inference profile restrictions, which remain tied to Converse.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Service tiers:&lt;/strong&gt; Standard only. No Priority, Flex or Reserved. If your capacity plan depended on reserving throughput, it does not exist for this model today.&lt;/p&gt;

&lt;p&gt;None of this is in the What's New post. An announcement is a trigger; the operational contract lives in the documentation.&lt;/p&gt;

&lt;h2&gt;
  
  
  The journey, in six decisions
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Inventory how the model is called today&lt;/strong&gt;: Before touching code, pull four facts per application from invocation logs and CloudTrail: endpoint (&lt;code&gt;bedrock-runtime&lt;/code&gt; or &lt;code&gt;bedrock-mantle&lt;/code&gt;), API (Converse, Responses, Chat Completions), source Region and input size distribution. CloudTrail's &lt;code&gt;additionalEventData.inferenceRegion&lt;/code&gt; field, logged in the source Region, shows where each request was actually processed. Count how many requests exceed 272 thousand tokens: they move to a different price band.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Prepare IAM and SCPs before the first request&lt;/strong&gt;: The model card requires &lt;code&gt;bedrock:InvokeModel&lt;/code&gt; on the account's default project (&lt;code&gt;arn:aws:bedrock:{region}:{account-id}:project/default&lt;/code&gt;) in addition to the inference profile. Missing it produces an &lt;code&gt;AccessDeniedException&lt;/code&gt; that looks like a credentials bug. For a geographic profile, the SCP must allow every destination Region in the profile; for a global profile, it must allow &lt;code&gt;aws:RequestedRegion&lt;/code&gt; equal to &lt;code&gt;unspecified&lt;/code&gt;, which many landing zones block by default.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Evaluate offline with real traffic&lt;/strong&gt;: Replay 300 to 500 logged, already sanitized runs against &lt;code&gt;us.openai.gpt-6.1-terra&lt;/code&gt;. Compare three signals per task: valid JSON rate against the schema with &lt;code&gt;strict: true&lt;/code&gt;, correctness of tool choice and arguments, and output tokens per task. OpenAI's benchmark is a hypothesis; your workload is the proof.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Reposition the cache&lt;/strong&gt;: On Responses, mark the end of the stable prefix with &lt;code&gt;prompt_cache_breakpoint&lt;/code&gt; and use &lt;code&gt;prompt_cache_options.mode: explicit&lt;/code&gt; so the automatic breakpoint doesn't consume the four writes per request. The minimum prefix is 1,024 tokens and the TTL is 30 minutes. Track &lt;code&gt;cached_tokens&lt;/code&gt; against &lt;code&gt;cache_write_tokens&lt;/code&gt;: writes cost 1.25x uncached input.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Canary with a written promotion criterion&lt;/strong&gt;: Split traffic by flag (5%, 25%, 100%), with the model ID coming from configuration, not code. Write down beforehand what promotes and what rolls back: schema failures no higher than baseline, equal or lower p95 latency, cost per run and &lt;code&gt;InvocationThrottles&lt;/code&gt; within budget. A criterion decided after the result is rationalization.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Retire the old path on purpose&lt;/strong&gt;: Keep the 5.6 Terra profile as rollback for 30 days after reaching 100%. Then remove it from the IAM policy. A forgotten permission becomes an escape hatch: someone falls back to the old model in a hotfix and nobody notices until the invoice.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Converse or Responses: the decision that blocks the migration
&lt;/h2&gt;

&lt;p&gt;On &lt;code&gt;bedrock-runtime&lt;/code&gt;, the 6.1 Terra model card ties features to different APIs, and they don't add up. Guardrails only work through Converse. Application inference profiles, which is how you attribute cost per team with tags, are also Converse only. Explicit caching, on the other hand, only exists on Responses. On Converse you get implicit caching, which the documentation itself calls best effort.&lt;/p&gt;

&lt;p&gt;Converse has two more restrictions: it doesn't accept &lt;code&gt;document&lt;/code&gt; blocks, and it rejects an image inside &lt;code&gt;toolResult&lt;/code&gt; with a &lt;code&gt;ValidationException&lt;/code&gt;. An agent that returns a tool screenshot to the model breaks on the first call.&lt;/p&gt;

&lt;p&gt;The cost of this choice can be calculated. In the reference scenario, the 30-thousand-token prefix costs US$ 0.033 uncached and US$ 0.00165 as a cache read, at US$ 1.10 and US$ 0.055 per million. Every miss costs 20 times more on the prefix. If implicit caching hits 60% and explicit hits 95%, the gap across 200 thousand runs a month is close to US$ 2.2 thousand.&lt;/p&gt;

&lt;p&gt;Use Converse when AWS Guardrails and cost allocation by application profile are audit requirements, and measure the real hit rate through &lt;code&gt;cacheReadInputTokens&lt;/code&gt; before accepting the loss. Use Responses when the prefix is large and stable and content filtering already happens in another layer you control. Running half the fleet on each API without deciding is the worst case: two audit trails, two cost models.&lt;/p&gt;

&lt;h2&gt;
  
  
  Canary migration backed by evidence
&lt;/h2&gt;

&lt;p&gt;The diagram answers one question: where control of the model switch lives. The router reads the model ID from a flag and splits traffic between the two US CRIS profiles. Dashed arrows are asynchronous: invocation logs arrive later, and promotion is a human decision based on the harness, not an automatic trigger. IAM, SCPs and the profile's destination Regions are simplified away.&lt;/p&gt;

&lt;h3&gt;
  
  
  Application
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Agent service ECS / Lambda (compute)&lt;/li&gt;
&lt;li&gt;Model router AppConfig flag (compute)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Amazon Bedrock runtime
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;GPT-5.6 Terra us.openai.gpt-5.6-terra (ai)&lt;/li&gt;
&lt;li&gt;GPT-6.1 Terra us.openai.gpt-6.1-terra (ai)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Evidence
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Invocation logs S3 + CloudWatch (storage)&lt;/li&gt;
&lt;li&gt;Eval harness replay + diff (ci)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Flows
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;agent -&amp;gt; router: asks for model&lt;/li&gt;
&lt;li&gt;router -&amp;gt; old: 95% Converse&lt;/li&gt;
&lt;li&gt;router -&amp;gt; new: 5% canary&lt;/li&gt;
&lt;li&gt;old -&amp;gt; logs: logs&lt;/li&gt;
&lt;li&gt;new -&amp;gt; logs: logs&lt;/li&gt;
&lt;li&gt;logs -&amp;gt; eval: replays&lt;/li&gt;
&lt;li&gt;eval -&amp;gt; router: promotes or rolls back&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Before and after, at US CRIS list price
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;-50%&lt;/strong&gt;: Input per million tokens. US$ 2.20 on 5.6 Terra to US$ 1.10 on 6.1 Terra, context up to 272 thousand tokens&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;-58%&lt;/strong&gt;: Output per million tokens. US$ 13.20 to US$ 5.50; cache reads drop 75%, from US$ 0.22 to US$ 0.055&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;-57%&lt;/strong&gt;: Cost per agent run. From ~US$ 0.068 to ~US$ 0.029 in the 40 thousand input (30 thousand cached) and 3 thousand output scenario, if output volume doesn't grow&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Quota, long context and the cost that shows up later
&lt;/h2&gt;

&lt;p&gt;The price cut is the easy part. Quota is what sinks the migration. On &lt;code&gt;bedrock-runtime&lt;/code&gt;, limits are measured in tokens per minute with a 10x burndown on output: each output token consumes 10 from quota. Tokens read from cache don't count toward the input quota. In the reference scenario, one run consumes 10 thousand uncached input tokens plus 30 thousand output-equivalent tokens. Three quarters of the quota goes to output.&lt;/p&gt;

&lt;p&gt;This creates a silent failure mode. If the new model answers with 20% more output tokens, because it reasons more or writes more, cost still drops substantially and no FinOps alarm fires. Quota, however, rises 15% per run, and the first signal is a &lt;code&gt;ThrottlingException&lt;/code&gt; at the morning peak. That's why the canary watches &lt;code&gt;OutputTokenCount&lt;/code&gt; and &lt;code&gt;InvocationThrottles&lt;/code&gt; in CloudWatch, not just the invoice.&lt;/p&gt;

&lt;p&gt;The second late cost is long context. 6.1 Terra accepts a 1,050,000-token context (OpenAI's page states 922 thousand maximum input), but above 272 thousand input tokens the whole request moves to another band: input at US$ 2.20 and output at US$ 8.25 per million on US CRIS. An agent that accumulates history without summarizing crosses that line around turn twenty and doubles the input cost of every call from then on. Set an explicit history ceiling, around 250 thousand tokens, and summarize above it. Having a 1-million-token context doesn't oblige anyone to use 1 million.&lt;/p&gt;

&lt;h2&gt;
  
  
  Each model's operational contract
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Criterion&lt;/th&gt;
&lt;th&gt;GPT-5.6 Terra&lt;/th&gt;
&lt;th&gt;GPT-6.1 Terra&lt;/th&gt;
&lt;th&gt;GPT-6.1 Sol&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Input / output US CRIS (US$ per million)&lt;/td&gt;
&lt;td&gt;2.20 / 13.20&lt;/td&gt;
&lt;td&gt;1.10 / 5.50&lt;/td&gt;
&lt;td&gt;2.20 / 11.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cache read (US$ per million)&lt;/td&gt;
&lt;td&gt;0.22&lt;/td&gt;
&lt;td&gt;0.055&lt;/td&gt;
&lt;td&gt;0.11&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;In-Region on &lt;code&gt;bedrock-mantle&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;us-east-1, us-east-2, us-west-2, GovCloud&lt;/td&gt;
&lt;td&gt;us-east-1&lt;/td&gt;
&lt;td&gt;us-east-1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;InvokeModel&lt;/code&gt; on &lt;code&gt;bedrock-runtime&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Guardrails on &lt;code&gt;bedrock-runtime&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Converse only&lt;/td&gt;
&lt;td&gt;Converse only&lt;/td&gt;
&lt;td&gt;Chat Completions, InvokeModel and Converse&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tiers beyond Standard&lt;/td&gt;
&lt;td&gt;None&lt;/td&gt;
&lt;td&gt;None&lt;/td&gt;
&lt;td&gt;Ultrafast (6x Standard)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Lifecycle&lt;/td&gt;
&lt;td&gt;EOL no sooner than 07/13/2027&lt;/td&gt;
&lt;td&gt;EOL no sooner than 10/26/2027&lt;/td&gt;
&lt;td&gt;OpenAI terms, at least 6 months' notice&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Risks I would handle before the canary:&lt;/strong&gt; &lt;strong&gt;São Paulo as source:&lt;/strong&gt; the 6.1 Terra table doesn't list &lt;code&gt;sa-east-1&lt;/code&gt; as a global profile source; if you call from there, move the client or wait. &lt;strong&gt;GovCloud and &lt;code&gt;bedrock-mantle&lt;/code&gt; outside &lt;code&gt;us-east-1&lt;/code&gt;:&lt;/strong&gt; no path at launch. &lt;strong&gt;Longer output:&lt;/strong&gt; with a 10x burndown, 20% more output becomes 15% more quota per run in the reference scenario, with no cost alarm. &lt;strong&gt;The 272-thousand-token line:&lt;/strong&gt; cross it and the whole request pays the long-context band. &lt;strong&gt;Vendor benchmark:&lt;/strong&gt; the 15 points and "three times faster" come from OpenAI's Agents' Last Exam, not from your workload.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Anti-patterns in this migration
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Changing the model ID and calling it a migration&lt;/strong&gt;: without an inventory of endpoint, API and source Region, the first failure comes from &lt;code&gt;bedrock-mantle&lt;/code&gt; in &lt;code&gt;us-west-2&lt;/code&gt; or a client in &lt;code&gt;sa-east-1&lt;/code&gt;, not from the model.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Promoting on the vendor benchmark&lt;/strong&gt;: Agents' Last Exam doesn't know your schema or your tools; promotion without replaying real traffic is a bet.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Measuring only the invoice&lt;/strong&gt;: cost drops even if output grows, and a quota with 10x burndown runs out before any FinOps alarm fires.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Switching to the global profile to dodge the Region table&lt;/strong&gt;: it fixes availability and creates a data residency problem, and it requires &lt;code&gt;aws:RequestedRegion&lt;/code&gt; &lt;code&gt;unspecified&lt;/code&gt; in the SCP.&lt;/li&gt;
&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Curator's note:&lt;/strong&gt; I would start this migration in November, not in June 2027. The first week would go entirely to the inventory and to the Converse versus Responses decision, because that decision defines auditing, caching and cost attribution for the next two years. After 16 years operating financial platforms, the lesson that keeps repeating in dependency swaps is this: the cost of a migration is not swapping the component, it's discovering late the contract that changed with it. With language models that doubles, because behavior changes too and no unit test catches it.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/about-aws/whats-new/2026/10/openai-gpt-6-1-terra-on-amazon-bedrock/" rel="noopener noreferrer"&gt;AWS What's New: OpenAI GPT-6.1 Terra is now generally available on Amazon Bedrock&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-6-1-terra.html" rel="noopener noreferrer"&gt;Amazon Bedrock User Guide: GPT-6.1 Terra model card&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-56-terra.html" rel="noopener noreferrer"&gt;Amazon Bedrock User Guide: GPT-5.6 Terra model card&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-6-1-sol.html" rel="noopener noreferrer"&gt;Amazon Bedrock User Guide: GPT-6.1 Sol model card&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html" rel="noopener noreferrer"&gt;Amazon Bedrock User Guide: Prompt caching for faster model inference&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html" rel="noopener noreferrer"&gt;Amazon Bedrock User Guide: Cross-Region inference&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://developers.openai.com/api/docs/models/gpt-6.1-terra" rel="noopener noreferrer"&gt;OpenAI API docs: GPT-6.1 Terra model&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.classmethod.jp/en/articles/bedrock-openai-gpt61-terra/" rel="noopener noreferrer"&gt;Classmethod DevelopersIO: I tried the new GPT-6.1 model Terra on Amazon Bedrock&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Verdict
&lt;/h2&gt;

&lt;p&gt;Migrate, and start now. Use GPT-6.1 Terra when your workload runs on &lt;code&gt;bedrock-runtime&lt;/code&gt; with a &lt;code&gt;us.&lt;/code&gt; or &lt;code&gt;global.&lt;/code&gt; profile from a source Region listed in the model card, when replaying real traffic shows equal or better schema and tool-use results, and when quota can absorb the new output under a 10x burndown. Stay on 5.6 Terra for now if you depend on &lt;code&gt;bedrock-mantle&lt;/code&gt; outside &lt;code&gt;us-east-1&lt;/code&gt;, on GovCloud, or on &lt;code&gt;sa-east-1&lt;/code&gt; as a source with residency requirements; the 07/13/2027 EOL leaves room to wait for the Region table to grow. Either way, decide Converse or Responses before the canary: that is the choice you will maintain, audit and pay for over the next two years.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Rating:&lt;/strong&gt; Migrar com canário / Migrate with a cana&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://fernando.moretes.com/blog/migrando-agentes-do-gpt-5-6-terra-para-o-gpt-6-1-terra-no-bedrock" rel="noopener noreferrer"&gt;fernando.moretes.com&lt;/a&gt;. By Fernando F. Azevedo: Senior Solutions Architect.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>aiagents</category>
      <category>amazonbedrock</category>
      <category>openai</category>
      <category>llmmigration</category>
    </item>
    <item>
      <title>Amazon Quick on a private network: PrivateLink's two planes and where they leak</title>
      <dc:creator>Fernando Azevedo</dc:creator>
      <pubDate>Fri, 23 Oct 2026 15:48:23 +0000</pubDate>
      <link>https://dev.to/fernando_azevedo_6844e930/amazon-quick-on-a-private-network-privatelinks-two-planes-and-where-they-leak-2lk</link>
      <guid>https://dev.to/fernando_azevedo_6844e930/amazon-quick-on-a-private-network-privatelinks-two-planes-and-where-they-leak-2lk</guid>
      <description>&lt;p&gt;When someone from compliance asks whether Amazon Quick "goes over the internet", the honest answer has two parts, and almost every design I have reviewed answers only one. User traffic into Quick uses PrivateLink interface endpoints. Quick's traffic out to your on-premises database, your private MCP server or your internal API uses something else: ENIs that Quick itself injects into your VPC. DNS, endpoint policy, IP/VPC restriction and IAM sit on top of those two planes, and each layer covers less than its name suggests. The useful question is not "is it private?", it's "which byte still leaves for the internet, and who decided that was fine?".&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this matters now
&lt;/h2&gt;

&lt;p&gt;While Quick was just BI, the VPC connection existed so Quick Sight could read PostgreSQL, SQL Server or Oracle behind a Direct Connect. The problem was well understood and the blast radius fit inside a dashboard. What changed is who uses that same VPC connection. It now carries Quick Agents calling MCP servers hosted on your network, Quick Flows and Quick Automate invoking action connectors, and Quick Research and Apps in Quick pulling from private sources. The ENI that used to run a &lt;code&gt;SELECT&lt;/code&gt; for a chart now carries an agent's tool call.&lt;/p&gt;

&lt;p&gt;In financial environments this matters in two places. &lt;strong&gt;Regulation:&lt;/strong&gt; CMN Resolution 4.893 and PCI-DSS expect you to show the data path and control who enters the perimeter. The AWS post lists HIPAA, FedRAMP and PCI-DSS as the usual triggers. &lt;strong&gt;Audit:&lt;/strong&gt; "traffic stays on the AWS backbone" only works as evidence if you can say which domains still leave through NAT, and some do.&lt;/p&gt;

&lt;p&gt;The 21 Oct 2026 post is a good walkthrough. What it doesn't do, and isn't meant to, is say where each control stops working. That's what this piece covers: the documented limits of each layer, and the cost of keeping all of it working for years, not the cost of building it once.&lt;/p&gt;

&lt;h2&gt;
  
  
  The two planes of private Quick access, and what stays public
&lt;/h2&gt;

&lt;p&gt;Ingress (user to Quick) over interface endpoints. Egress (Quick to your data) over VPC connection ENIs. Dashed lines are traffic no endpoint covers.&lt;/p&gt;

&lt;h3&gt;
  
  
  🏢 On-premises: rede corporativa
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;DNS corporativo forward de .quicksight.aws.amazon.com (network)&lt;/li&gt;
&lt;li&gt;Oracle / SQL Server porta do banco liberada só para o SG do Quick (data)&lt;/li&gt;
&lt;li&gt;Servidor MCP privado API interna por IP (compute)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  📡 Rede: Direct Connect ou Site-to-Site VPN
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Direct Connect rota para o CIDR da VPC (network)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  🟧 AWS: VPC do cliente (2 AZs no mínimo)
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Route 53 Resolver inbound 2+ ENIs (network)&lt;/li&gt;
&lt;li&gt;VPCE quicksight-website policy só com aws:PrincipalAccount / aws:OrgId (security)&lt;/li&gt;
&lt;li&gt;VPCE quicksight (API) policy por ação; 13 APIs fora (security)&lt;/li&gt;
&lt;li&gt;ENIs da VPC connection SG QuickSight-VPC, VPC ID imutável (network)&lt;/li&gt;
&lt;li&gt;VPCE secretsmanager / redshift / glue S3 por gateway (security)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  🤖 AWS: Amazon Quick (Enterprise)
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Quick Sight, Agents, Flows, Automate Identity Center, RLS/CLS (ai)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  🌐 Externo: saída pública que continua obrigatória
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;*.cloudfront.net, *.execute-api, *.kinesisvideo sign-in do Identity Center (external)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Flows
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;user -&amp;gt; dns: resolves the site name&lt;/li&gt;
&lt;li&gt;dns -&amp;gt; resolver: forwards the private zone&lt;/li&gt;
&lt;li&gt;user -&amp;gt; dx: HTTPS 443&lt;/li&gt;
&lt;li&gt;dx -&amp;gt; vpce-web: console and dashboards&lt;/li&gt;
&lt;li&gt;dx -&amp;gt; vpce-api: API calls&lt;/li&gt;
&lt;li&gt;vpce-web -&amp;gt; quick: PrivateLink (ingress)&lt;/li&gt;
&lt;li&gt;vpce-api -&amp;gt; quick: PrivateLink (ingress)&lt;/li&gt;
&lt;li&gt;quick -&amp;gt; eni: assumes execution role, uses ENIs&lt;/li&gt;
&lt;li&gt;eni -&amp;gt; dx: data and tool-call egress&lt;/li&gt;
&lt;li&gt;dx -&amp;gt; db: SQL query&lt;/li&gt;
&lt;li&gt;dx -&amp;gt; mcp: action connector / MCP&lt;/li&gt;
&lt;li&gt;eni -&amp;gt; sm: credentials without leaving the VPC&lt;/li&gt;
&lt;li&gt;user -&amp;gt; public: assets, metrics, sign-in via NAT/proxy&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Ingress plane: interface endpoints, DNS and what the policy accepts
&lt;/h2&gt;

&lt;p&gt;Ingress uses two interface endpoints with different jobs: &lt;code&gt;com.amazonaws.&amp;lt;region&amp;gt;.quicksight-website&lt;/code&gt; for the site and &lt;code&gt;com.amazonaws.&amp;lt;region&amp;gt;.quicksight&lt;/code&gt; for the API (with a &lt;code&gt;-fips&lt;/code&gt; variant). If administrators sign in as IAM users, add &lt;code&gt;console&lt;/code&gt; and &lt;code&gt;signin&lt;/code&gt; too. Place each one in at least two AZs, with a security group that accepts only 443 from the corporate CIDR.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;DNS is where the first thing breaks.&lt;/strong&gt; The API endpoint, with private DNS on, takes over &lt;code&gt;quicksight.&amp;lt;region&amp;gt;.amazonaws.com&lt;/code&gt; by itself. The website endpoint doesn't: its private name is not the public URL. You create a private hosted zone for &lt;code&gt;&amp;lt;region&amp;gt;.quicksight.aws.amazon.com&lt;/code&gt; with an alias A record pointing at the endpoint. But a private hosted zone only answers associated VPCs, so corporate DNS has to forward that zone to a Route 53 Resolver inbound endpoint. Without the forward, the browser resolves the public IP, access still works, and nobody notices PrivateLink is being bypassed. Test with &lt;code&gt;nslookup&lt;/code&gt; from inside the corporate network and expect an IP from the VPC CIDR.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The website endpoint policy is almost decorative.&lt;/strong&gt; &lt;code&gt;Principal&lt;/code&gt;, &lt;code&gt;Action&lt;/code&gt; and &lt;code&gt;Resource&lt;/code&gt; must be &lt;code&gt;"*"&lt;/code&gt;, and the condition only accepts &lt;code&gt;aws:PrincipalAccount&lt;/code&gt; or &lt;code&gt;aws:OrgId&lt;/code&gt;. It stops your network from becoming a doorway into another account's Quick, which helps against exfiltration, but it doesn't control what the user does. The API endpoint accepts action-level restriction, and that's where being specific pays off.&lt;/p&gt;

&lt;h2&gt;
  
  
  Egress plane: the VPC connection is an injected ENI, not PrivateLink
&lt;/h2&gt;

&lt;p&gt;When Quick needs to fetch data, it doesn't go through the interface endpoint. To create a VPC connection (Enterprise only), you supply a VPC ID, at least two subnets in different AZs, at least one security group, and an execution role that trusts &lt;code&gt;quicksight.amazonaws.com&lt;/code&gt;. Quick assumes that role and creates ENIs in your subnets with &lt;code&gt;ec2:CreateNetworkInterface&lt;/code&gt;. From then on your network decides: route table, NACL, security group and the Direct Connect route.&lt;/p&gt;

&lt;p&gt;Three details turn into incidents if you don't know them. &lt;strong&gt;The VPC ID can't change after creation:&lt;/strong&gt; reorganizing the network means recreating the connection and repointing every data source. &lt;strong&gt;The &lt;code&gt;PARTIALLY AVAILABLE&lt;/code&gt; status&lt;/strong&gt; means one ENI is down and the connection still works. It's exactly the state nobody watches until the second AZ fails. &lt;strong&gt;Connections created before 27 Apr 2023 use a non-stateful security group&lt;/strong&gt; and need an inbound rule that lets return traffic in on every port (0–65535) from the source's SG. If yours dates from then, recreate it.&lt;/p&gt;

&lt;p&gt;For on-premises database host names, enter the Resolver inbound IPs in the DNS resolver endpoints field. Without them Quick resolves through public DNS and can't find the host. Credentials go to Secrets Manager through an interface endpoint in the same VPC as the ENIs, so that even the password lookup doesn't leave through NAT.&lt;/p&gt;

&lt;p&gt;One limitation changes the design: the Quick Index knowledge base VPC connection currently supports only Amazon Redshift. Confluence, SharePoint, Box, Google Drive, OneDrive and S3 can't use it.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;The ENI security group is now the agent's network allow-list:&lt;/strong&gt; The same VPC connection serves Quick Sight and Quick Agents calling private MCP. In practice, the egress of the &lt;code&gt;QuickSight-VPC&lt;/code&gt; SG and the routes on those subnets define everything an agent can reach on your network. If the SG opens the whole on-premises CIDR because "BI needed it", the agent inherits that reach. Split subnets and security groups by purpose (analytical data on one side, action tools on the other), even if it costs a second VPC connection.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Enforcing endpoint-only access: where the restriction stops
&lt;/h2&gt;

&lt;p&gt;Quick's IP/VPC restriction (&lt;code&gt;update-ip-restriction --enabled --vpc-endpoint-id-restriction-rule-map vpce-...=Name&lt;/code&gt;) blocks anything that doesn't come from the approved endpoint. It has four properties you need to know before turning it on in production.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It covers web, embedded and mobile, but not the public API.&lt;/strong&gt; Anyone with credentials can call any operation from any IP. For the API the control is IAM: an &lt;code&gt;aws:SourceVpce&lt;/code&gt; or &lt;code&gt;aws:SourceIp&lt;/code&gt; condition in a deny policy, ideally in an SCP. Here's the catch: 13 APIs don't work through the interface endpoint (the 8 action connector ones, such as &lt;code&gt;CreateActionConnector&lt;/code&gt;, and 5 Flows ones, such as &lt;code&gt;ListFlows&lt;/code&gt;). A blanket &lt;code&gt;aws:SourceVpce&lt;/code&gt; deny breaks the automation that manages connectors. You write the exception with &lt;code&gt;NotAction&lt;/code&gt; for those operations and run that automation from a controlled path with public egress.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;UpdateIpRestriction&lt;/code&gt; replaces the whole map.&lt;/strong&gt; A script that sends only the new rule deletes the others. Run &lt;code&gt;describe-ip-restriction&lt;/code&gt; first, or better, keep the map versioned and always apply the full state.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A rule takes up to 10 minutes to apply and is global&lt;/strong&gt;, enforced in every Region and managed only from the account's Region. The limit is 100 rules combined, with IP and VPC rules OR'ed together.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Lockout is real.&lt;/strong&gt; Without a rule covering your own access, the console won't let you save. But anyone on a source outside the list loses access the moment you turn it on. Publicly embedded dashboards then work only through the endpoint.&lt;/p&gt;

&lt;h2&gt;
  
  
  What stays public, and what keeping the rest private costs
&lt;/h2&gt;

&lt;p&gt;The documentation lists the domains the browser reaches, and several have no endpoint: &lt;code&gt;*.cloudfront.net&lt;/code&gt; (CSS and JS), &lt;code&gt;*.execute-api.&amp;lt;region&amp;gt;.amazonaws.com&lt;/code&gt; (client metrics), &lt;code&gt;*.kinesisvideo.amazonaws.com&lt;/code&gt; (automation streaming), &lt;code&gt;&amp;lt;region&amp;gt;.signin.aws&lt;/code&gt; and, with IAM Identity Center, browser sign-in, which requires the public sign-in and Identity Center endpoints. No public IP is needed, but NAT or a proxy is. Your compliance report should say "the data and API path is private; static assets, telemetry and authentication leave through an inspected proxy". Writing "zero internet" claims something the documentation itself contradicts.&lt;/p&gt;

&lt;p&gt;Now the recurring cost, in &lt;code&gt;us-east-1&lt;/code&gt;. An interface endpoint costs US$ 0.01 per hour per ENI, plus US$ 0.01 per GB for the first 1 PB. Website and API in 2 AZs are 4 ENIs, about US$ 29/month. Add &lt;code&gt;console&lt;/code&gt; and &lt;code&gt;signin&lt;/code&gt;, another US$ 29. Secrets Manager, Redshift and Glue in 2 AZs, roughly US$ 44 more. The most expensive piece is the one nobody budgets: a Resolver inbound endpoint costs US$ 0.125 per ENI per hour, with at least two, so about US$ 182/month, plus US$ 0.40 per million queries. If you already have a shared Resolver inbound in a central network VPC, reuse it: it's the biggest line item and there's nothing Quick-specific about it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Anti-patterns I find in this design
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Website endpoint without on-premises DNS forwarding:&lt;/strong&gt; PrivateLink exists, the browser resolves the public IP and access works from outside. It's a silent failure that only shows up in &lt;code&gt;nslookup&lt;/code&gt; or VPC Flow Logs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treating IP/VPC restriction as an API control:&lt;/strong&gt; it doesn't cover the public API. Without &lt;code&gt;aws:SourceVpce&lt;/code&gt; in IAM or an SCP, a leaked credential calls everything from anywhere.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Blanket &lt;code&gt;aws:SourceVpce&lt;/code&gt; deny with no exception:&lt;/strong&gt; the 13 action connector and Flows APIs don't go through the endpoint, and the automation that manages connectors stops working without an obvious error.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A script that calls &lt;code&gt;UpdateIpRestriction&lt;/code&gt; with only the new rule:&lt;/strong&gt; the map is replaced whole, the other rules disappear and an entire branch office loses access within 10 minutes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One VPC connection for BI and agents with the whole on-premises CIDR open:&lt;/strong&gt; the agent inherits BI's network reach, and the blast radius of a prompt injection becomes the corporate network.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keeping a VPC connection from before 27 Apr 2023:&lt;/strong&gt; the non-stateful SG requires return traffic open on 0–65535, a rule no auditor accepts without questions.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Well-Architected lens
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;security&lt;/strong&gt;: Layered defense with clear roles: website endpoint policy scoped by &lt;code&gt;aws:OrgId&lt;/code&gt;, API endpoint policy by action, IP/VPC restriction for web and mobile, an SCP with &lt;code&gt;aws:SourceVpce&lt;/code&gt; and &lt;code&gt;NotAction&lt;/code&gt; for the 13 APIs outside the endpoint, RLS and CLS on the data. Restriction map changes land in CloudTrail; alarm on &lt;code&gt;UpdateIpRestriction&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;reliability&lt;/strong&gt;: Two AZs on every endpoint and on the VPC connection, an alarm whenever status leaves &lt;code&gt;AVAILABLE&lt;/code&gt; (including &lt;code&gt;PARTIALLY AVAILABLE&lt;/code&gt;), and Direct Connect with a backup VPN, because the VPC connection has no other path to on-premises.&lt;/li&gt;
&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;What I would do:&lt;/strong&gt; I'd start with the report I'll hand the auditor, and only then create the first endpoint: which domains leave through the proxy, which controls cover the API and which cover the browser. Then I'd build two security groups and two VPC connections, one for analytical data and one for agent tools, because I've seen a "temporary" BI security group last four years. The restriction map would go into code from day one, applied as full state. The lesson I carry from financial platforms: the expensive control isn't the one you configure, it's the one you think is active and isn't, and in Quick that control is DNS forwarding.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Verdict
&lt;/h2&gt;

&lt;p&gt;Use the full design (website and API endpoints, VPC connection, Resolver inbound, IP/VPC restriction and an SCP with &lt;code&gt;aws:SourceVpce&lt;/code&gt;) when you have regulated on-premises data, agents calling private tools, and an auditor who will ask for the path of every byte. If Quick only reads Redshift and S3 in the same account and sign-in goes through Identity Center, a VPC connection with service endpoints covers most of the risk, and IP restriction on the corporate egress covers the browser for well under US$ 300/month. Either way, document what stays public. That sentence is what separates a defensible design from a promise the documentation contradicts.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Rating:&lt;/strong&gt; Recomendado com ressalvas / Recommended &lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/blogs/machine-learning/securing-quick-access-from-corporate-networks-with-aws-privatelink/" rel="noopener noreferrer"&gt;AWS Machine Learning Blog: Securing Quick access from corporate networks with AWS PrivateLink&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/quicksight/latest/developerguide/vpc-interface-endpoints.html" rel="noopener noreferrer"&gt;Quick and interface VPC endpoints (AWS PrivateLink)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/quicksight/latest/user/enabling-ip-restrictions.html" rel="noopener noreferrer"&gt;Managing IP/VPC restrictions in Amazon Quick&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/quick/latest/userguide/working-with-aws-vpc.html" rel="noopener noreferrer"&gt;Configuring VPC connections in Amazon Quick Sight&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/privatelink/pricing/" rel="noopener noreferrer"&gt;AWS PrivateLink pricing&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/route53/pricing/" rel="noopener noreferrer"&gt;Amazon Route 53 pricing (Resolver endpoints)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://fernando.moretes.com/blog/amazon-quick-em-rede-privada-os-dois-planos-do-privatelink-e-onde" rel="noopener noreferrer"&gt;fernando.moretes.com&lt;/a&gt;. By Fernando F. Azevedo: Senior Solutions Architect.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>awscloud</category>
      <category>amazonquick</category>
      <category>privatelink</category>
      <category>vpcendpoints</category>
    </item>
    <item>
      <title>After login: per-operation authorization, revocation and recovery for agent tools</title>
      <dc:creator>Fernando Azevedo</dc:creator>
      <pubDate>Wed, 21 Oct 2026 16:01:25 +0000</pubDate>
      <link>https://dev.to/fernando_azevedo_6844e930/after-login-per-operation-authorization-revocation-and-recovery-for-agent-tools-45i0</link>
      <guid>https://dev.to/fernando_azevedo_6844e930/after-login-per-operation-authorization-revocation-and-recovery-for-agent-tools-45i0</guid>
      <description>&lt;h2&gt;
  
  
  A valid login is only the starting point
&lt;/h2&gt;

&lt;p&gt;An agent may continue a conversation after a user loses a permission. The operational question is whether the next tool call can still change a resource. This article proposes an architecture exercise, not a report of an AWS deployment or a completed cloud experiment. The AWS scenarios below are test proposals; the local recovery lab has a narrower purpose.&lt;/p&gt;

&lt;p&gt;The starting points are Carlos Biagolini's articles on &lt;a href="https://builder.aws.com/content/3Kuk63NZmMt59SD8ONcrmAmf3mE/primeiros-passos-com-amazon-bedrock-agentcore-identity" rel="noopener noreferrer"&gt;AgentCore Identity&lt;/a&gt; and &lt;a href="https://builder.aws.com/content/3Ga6SwtCJCXBaQgy1sxBi5Qw8um/construindo-um-servidor-mcp-serverless-na-aws-com-autenticacao-iam" rel="noopener noreferrer"&gt;a serverless MCP server with IAM authentication&lt;/a&gt;. They motivate a useful follow-up: once identity is established, which component enforces the right to perform each operation, and what happens if execution is interrupted?&lt;/p&gt;

&lt;h2&gt;
  
  
  Separate three decisions
&lt;/h2&gt;

&lt;p&gt;Authentication establishes the caller's identity. Authorization decides whether that caller may perform a particular operation on a particular resource and tenant. Approval records consent to a concrete proposed change. A valid token, a visible tool or an old approval does not answer all three questions.&lt;/p&gt;

&lt;p&gt;Imagine a remediation tool shared by two tenants. A caller may have permission to invoke the endpoint and still lack permission to change the selected resource. Similarly, approval of an earlier proposal should not authorize a modified proposal. These checks must remain explicit even when the model produces a convincing explanation.&lt;/p&gt;

&lt;h2&gt;
  
  
  A route is a boundary, not the whole policy
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-access-control.html" rel="noopener noreferrer"&gt;API Gateway HTTP APIs&lt;/a&gt; support IAM, JWT authorizers and Lambda authorizers. This exercise compares IAM-based invocation with application-defined Lambda authorization. With AWS_IAM, a signed request and invocation permission control entry. A Lambda authorizer applies application logic: format 1.0 requires an IAM policy; format 2.0 also supports a simple Boolean response when enabled.&lt;/p&gt;

&lt;p&gt;If several MCP tools share &lt;code&gt;POST /mcp&lt;/code&gt;, the tool name and arguments live in the body. Permission to invoke that route does not establish permission for every operation carried by it. In this design, the executor enforces fine-grained authorization. Another trusted layer could do so if it understands the operation, cannot be bypassed and must run before the effect. Separate routes help entry controls but do not remove resource and tenant checks.&lt;/p&gt;

&lt;p&gt;Do not accept a model-supplied tenant as proof of authority. In a JWT flow, use validated claims and their authorized relationship to the resource. In an IAM/SigV4 flow, use the authenticated principal and a service-controlled authorization mapping. Carry that trusted context to the execution boundary. A Lambda role's ability to access DynamoDB does not automatically prove that every caller may read every item the function can access.&lt;/p&gt;

&lt;h2&gt;
  
  
  Revocation has an observable window
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_control-access_disable-perms.html" rel="noopener noreferrer"&gt;IAM documentation&lt;/a&gt; explains that temporary-credential permissions are evaluated for AWS requests, while policy updates may take minutes to propagate. Per-request evaluation therefore does not imply instantaneous revocation.&lt;/p&gt;

&lt;p&gt;A &lt;a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-lambda-authorizer.html" rel="noopener noreferrer"&gt;cached Lambda-authorizer result&lt;/a&gt; introduces another window. Include the cache key, route separation and TTL in the test design. Denying future work does not reverse a completed effect. Sensitive operations also need an explicit policy for in-flight work and a check near the effect boundary.&lt;/p&gt;

&lt;h2&gt;
  
  
  Approval and completion are different facts
&lt;/h2&gt;

&lt;p&gt;Bind approval to reviewer identity, proposal digest, scope, expiry and expected resource state. After a worker restart, a &lt;code&gt;pending&lt;/code&gt; record does not prove whether an effect happened. Look for durable target evidence before replaying. If evidence is insufficient, preserve uncertainty and request review instead of guessing.&lt;/p&gt;

&lt;p&gt;The public &lt;a href="https://github.com/fernando-moretes/lab-security-remediation/releases/tag/v0.4.0" rel="noopener noreferrer"&gt;v0.4.0 lab&lt;/a&gt; explores that recovery boundary with SQLite and a synthetic target. Its &lt;a href="https://github.com/fernando-moretes/lab-security-remediation/blob/v0.4.0/DURABILITY.md" rel="noopener noreferrer"&gt;reproduction guide&lt;/a&gt; requires Node.js 22.13+ and creates local files. No AWS account is needed. CLI approval is simulated; the lab does not implement authenticated approval, IAM or revocation. Atomic local target-and-receipt writes do not establish exactly-once effects across a remote API.&lt;/p&gt;

&lt;p&gt;In the documented exercise, a deliberate crash occurs after the synthetic target and receipt are committed but before the execution ledger acknowledges completion. Recovery uses that receipt to reconcile the operation. Keeping the original proposal matters: generating a new proposal after the target changes produces a different digest. This is a concrete way to discuss recovery without implying a cloud guarantee the exercise cannot demonstrate.&lt;/p&gt;

&lt;h2&gt;
  
  
  Proposed test matrix
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Valid principal requests another tenant&lt;/strong&gt;: Deny before protected data access. Evidence: Synthetic identity and decision.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Shared MCP endpoint receives an unauthorized tool&lt;/strong&gt;: Deny at the operation boundary. Evidence: Tool name, scope and decision.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Permission changes during a session&lt;/strong&gt;: Measure when new calls stop succeeding. Evidence: Change time and decision timeline.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Authorizer result is cached&lt;/strong&gt;: Test before and after TTL across routes and tenants. Evidence: Cache settings and observations.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Proposal changes after approval&lt;/strong&gt;: Reject the changed digest. Evidence: Original and changed synthetic digests.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Resource changes before execution&lt;/strong&gt;: Reject the stale precondition. Evidence: Expected and observed version.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Process stops after effect, before acknowledgement&lt;/strong&gt;: Reconcile using evidence. Evidence: Ledger, target version and receipt.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For every scenario, distinguish expected from observed results. These AWS authorization tests are proposed, not completed. Use synthetic inputs and avoid storing tokens, secrets or customer data.&lt;/p&gt;

&lt;h2&gt;
  
  
  Trade-offs and the next experiment
&lt;/h2&gt;

&lt;p&gt;Fresh authorization checks add work and latency. Caching can reduce both while extending the stale-decision window. Choose the design using operation impact, revocation policy and measurements. A low-risk read and a destructive remediation may justify different controls.&lt;/p&gt;

&lt;p&gt;Before adding another agent, ask the team to demonstrate which identity drives the decision, which component enforces it and how interrupted work is recovered. A future isolated experiment can add authenticated approval and revocation, with a cost ceiling and cleanup procedure defined before deployment.&lt;/p&gt;

&lt;p&gt;What would you test first: cross-tenant access, revocation during a session, or recovery after the effect? A useful answer includes the evidence needed to prove the outcome.&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-access-control.html" rel="noopener noreferrer"&gt;HTTP API access controls&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-access-control-iam.html" rel="noopener noreferrer"&gt;IAM authorization for HTTP APIs&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-lambda-authorizer.html" rel="noopener noreferrer"&gt;Lambda authorizers&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_control-access_disable-perms.html" rel="noopener noreferrer"&gt;Temporary-credential permissions&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://builder.aws.com/content/3Kuk63NZmMt59SD8ONcrmAmf3mE/primeiros-passos-com-amazon-bedrock-agentcore-identity" rel="noopener noreferrer"&gt;AgentCore Identity article&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://builder.aws.com/content/3Ga6SwtCJCXBaQgy1sxBi5Qw8um/construindo-um-servidor-mcp-serverless-na-aws-com-autenticacao-iam" rel="noopener noreferrer"&gt;Serverless MCP and IAM article&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/fernando-moretes/lab-security-remediation/blob/v0.4.0/DURABILITY.md" rel="noopener noreferrer"&gt;Local recovery exercise and limitations&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Authorization boundaries
&lt;/h2&gt;

&lt;p&gt;Proposed design; identity and policy must reach the effect boundary.&lt;/p&gt;

&lt;h3&gt;
  
  
  Trusted application boundary
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Authenticated caller (user)&lt;/li&gt;
&lt;li&gt;Route entry control (edge)&lt;/li&gt;
&lt;li&gt;Tool contract (compute)&lt;/li&gt;
&lt;li&gt;Operation + resource + tenant (security)&lt;/li&gt;
&lt;li&gt;Proposal approval (security)&lt;/li&gt;
&lt;li&gt;Constrained executor (compute)&lt;/li&gt;
&lt;li&gt;Durable evidence (storage)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Flows
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;caller -&amp;gt; entry&lt;/li&gt;
&lt;li&gt;entry -&amp;gt; contract&lt;/li&gt;
&lt;li&gt;contract -&amp;gt; policy&lt;/li&gt;
&lt;li&gt;policy -&amp;gt; approval&lt;/li&gt;
&lt;li&gt;approval -&amp;gt; executor&lt;/li&gt;
&lt;li&gt;executor -&amp;gt; receipt&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  C4
&lt;/h2&gt;

&lt;p&gt;A user requests remediation through an agent; the model proposes, the application enforces boundaries.&lt;/p&gt;

&lt;h3&gt;
  
  
  Containers
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;API / MCP endpoint&lt;/strong&gt; (API Gateway + application): Authenticates the caller and controls entry.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tool executor&lt;/strong&gt; (Application service): Checks operation, resource, tenant and approval before the effect.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Evidence store&lt;/strong&gt; (SQLite in local lab): Preserves synthetic proposal, state and receipt for recovery.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Components
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Policy check&lt;/strong&gt;: Uses trusted context independent of model arguments.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Recovery reconciler&lt;/strong&gt;: Reconciles using evidence; preserves uncertainty without proof.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  AWS Well-Architected
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;security&lt;/strong&gt;: Deny out-of-scope operations before effects; do not trust model-supplied tenant identity.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;reliability&lt;/strong&gt;: Recover using receipts and preconditions without promising remote exactly-once effects.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;operational-excellence&lt;/strong&gt;: Record expected and observed results with versions and synthetic evidence.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;cost&lt;/strong&gt;: Run the local lab first; define a ceiling and cleanup before an AWS experiment.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Checklist before automation
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Bind identity, digest, scope and expiry to approval.&lt;/strong&gt;: Document the evidence, owner and known limitation.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Test the wrong tenant before data access.&lt;/strong&gt;: Document the evidence, owner and known limitation.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Measure propagation and caching during revocation.&lt;/strong&gt;: Document the evidence, owner and known limitation.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Simulate a crash after the effect and inspect the receipt.&lt;/strong&gt;: Document the evidence, owner and known limitation.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Anti-patterns
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Treating authentication or old approval as universal authorization.&lt;/li&gt;
&lt;li&gt;Repeating an effect merely because the ledger still says pending.&lt;/li&gt;
&lt;li&gt;Promising instant revocation or remote exactly-once execution without measurement.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://fernando.moretes.com/blog/depois-do-login-autorizacao-revogacao-recuperacao-agentes" rel="noopener noreferrer"&gt;fernando.moretes.com&lt;/a&gt;. By Fernando F. Azevedo: Senior Solutions Architect.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>securityresilience</category>
      <category>aws</category>
      <category>iam</category>
      <category>agenticai</category>
    </item>
    <item>
      <title>Retro: EMR Serverless on IPv6-only and the subnet left without DNS64</title>
      <dc:creator>Fernando Azevedo</dc:creator>
      <pubDate>Tue, 20 Oct 2026 15:42:40 +0000</pubDate>
      <link>https://dev.to/fernando_azevedo_6844e930/retro-emr-serverless-on-ipv6-only-and-the-subnet-left-without-dns64-355a</link>
      <guid>https://dev.to/fernando_azevedo_6844e930/retro-emr-serverless-on-ipv6-only-and-the-subnet-left-without-dns64-355a</guid>
      <description>&lt;p&gt;The dangerous question about IPv6 on EMR Serverless is not "do my jobs run on IPv6?", it is "what does each subnet in the application know how to do when a job asks for an address that only exists in IPv4?". The 10/16/2026 announcement is short: Spark and Hive can now run on dual-stack or IPv6-only subnets, at no additional cost, in every Region and release version of the service. The documentation is more honest than the announcement: the addressing mode is derived from the subnets, and getting IPv6 traffic to your data is your responsibility. This piece is a pre-emptive retro, an incident that did not happen, built from the documentation and the failure modes it lists itself. I would rather write the postmortem before than after.&lt;/p&gt;

&lt;h2&gt;
  
  
  The scenario: /22 subnets that no longer fit three applications
&lt;/h2&gt;

&lt;p&gt;The starting point is what AWS itself uses as a planning example. Each EMR Serverless worker consumes an IP address in the subnet where it launches, through an ENI. An application with 4 vCPU workers and a 4,000 vCPU ceiling reaches 1,000 workers, meaning 1,000 ENIs. A /22 subnet has 1,024 addresses and 1,019 usable ones, because AWS reserves five. With three applications sharing the same subnets and a 12,000 vCPU account quota, the documentation's recommendation becomes 3,000 free addresses per subnet, in at least two Availability Zones.&lt;/p&gt;

&lt;p&gt;In a bank's data platform this usually hits a wall: the VPC IPv4 block was negotiated with corporate networking years ago, does not overlap with on-premises and does not grow without an approval round. That is exactly the case IPv6 solves. A VPC IPv6 subnet is a /64, and the address conversation is over.&lt;/p&gt;

&lt;p&gt;In the scenario, the team goes straight to IPv6-only on three subnets, one per AZ, created by an IaC module. Two were built by hand during the pilot and got DNS64. The third came from the module, which did not know DNS64 existed. No integration test caught it, because the test job ran twice and landed on the two good subnets both times.&lt;/p&gt;

&lt;h2&gt;
  
  
  The numbers that drive the decision
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;1.000&lt;/strong&gt;: ENIs per application. 4 vCPU workers with a 4,000 vCPU ceiling: one IP per worker&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;1.019&lt;/strong&gt;: Usable IPs in a /22. Barely fits one application; three need 3,000 per subnet&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;US$ 0,045/GB&lt;/strong&gt;: NAT gateway processing. Pricing page example (US East Ohio): every byte through NAT64 pays&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Timeline of the simulated incident
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;D-7: subnet swap&lt;/strong&gt;: The team stops the application, runs &lt;code&gt;update-application&lt;/code&gt; with the three IPv6-only subnets in &lt;code&gt;--network-configuration&lt;/code&gt; and starts it again. The mode is evaluated at every start, so the change only applies after the restart. Two test jobs pass.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;02:00: the nightly batch starts&lt;/strong&gt;: About a third of the jobs fail with &lt;code&gt;UnknownHostException&lt;/code&gt; when calling endpoints that only publish an A record. Each job runs in a single subnet picked from the set, and the ones that landed on the subnet without DNS64 cannot resolve anything that is not IPv6.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;02:40: retries hide the pattern&lt;/strong&gt;: The orchestrator's retry policy resubmits the jobs. Some pass because they land on another subnet. The failure looks intermittent, and intermittent becomes "flaky network" in the on-call's first hypothesis.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;04:10: correlation by subnet&lt;/strong&gt;: Someone opens each job run detail, which shows the subnet used, and crosses it with success and failure. 100% of the failures sit on the same subnet. &lt;code&gt;modify-subnet-attribute --enable-dns64&lt;/code&gt; fixes name resolution.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;05:00: the second failure&lt;/strong&gt;: An ingestion job with an old JDBC driver fails with &lt;code&gt;UnresolvedAddressException&lt;/code&gt;. It carried &lt;code&gt;-Djava.net.preferIPv4Stack=true&lt;/code&gt; in &lt;code&gt;spark.driver.extraJavaOptions&lt;/code&gt;, a workaround from years ago. The documentation forbids forcing the IPv4 stack in any mode.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;D+5: the bill arrives&lt;/strong&gt;: Cost Explorer shows a jump in NAT gateway data processing. S3 reads, which used to leave through the gateway endpoint, now resolve through DNS64 to &lt;code&gt;64:ff9b::/96&lt;/code&gt; and cross NAT64.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Root cause:&lt;/strong&gt; EMR Serverless derives the addressing mode from the subnets and does not validate whether the network is ready for it. Each subnet's readiness (IPv6 CIDR, auto-assign, DNS64, a &lt;code&gt;64:ff9b::/96&lt;/code&gt; route to the NAT gateway, a &lt;code&gt;::/0&lt;/code&gt; route to the egress-only gateway) was not encoded in the IaC module, and one-subnet-per-job selection turned a deterministic defect into an apparently random failure.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Why the failure looked random: the addressing mode rules
&lt;/h2&gt;

&lt;p&gt;It is worth understanding the rules before blaming the network. An IPv4-only subnet runs as it always did. A dual-stack subnet runs IPv4 until you turn on &lt;code&gt;spark.emr-serverless.networking.ipv6.enabled=true&lt;/code&gt; in the application's &lt;code&gt;spark-defaults&lt;/code&gt; (for Hive, &lt;code&gt;hive.emr-serverless.networking.ipv6.enabled&lt;/code&gt; in &lt;code&gt;hive-site&lt;/code&gt;). An IPv6-only subnet runs IPv6 with no property at all.&lt;/p&gt;

&lt;p&gt;When the set mixes types, the whole application runs in the one mode every subnet supports. IPv4-only with IPv6-only is rejected at validation. IPv6-only with dual-stack runs everything IPv6-only, including the dual-stack subnets, which must then meet the same prerequisites. IPv4-only with dual-stack runs everything on IPv4.&lt;/p&gt;

&lt;p&gt;Three details produce incidents:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The mode applies at start:&lt;/strong&gt; a running application keeps the mode it started with. You fix the subnet and nothing changes until the next &lt;code&gt;stop&lt;/code&gt;/&lt;code&gt;start&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One job, one subnet:&lt;/strong&gt; a job runs in a single subnet picked from the list. One bad subnet out of three fails a third of the time, and retries mask it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dual-stack does not prefer IPv6 by default:&lt;/strong&gt; even with the opt-in on, the engine uses IPv4 until you set &lt;code&gt;spark.emr-serverless.driverEnv.SPARK_PREFER_IPV6&lt;/code&gt; and &lt;code&gt;spark.executorEnv.SPARK_PREFER_IPV6&lt;/code&gt;. Setting those on an IPv4-only subnet kills the job.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The lesson: the addressing mode is derived state, not configured state. The API does not let you declare the mode; it lets you declare subnets and infers the rest. Derived state needs a test that proves the derivation on every subnet, not on average.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where each byte of a job on an IPv6-only subnet leaves
&lt;/h2&gt;

&lt;p&gt;The path depends on the destination's DNS record: AAAA leaves directly, A becomes &lt;code&gt;64:ff9b::/96&lt;/code&gt; and pays NAT64.&lt;/p&gt;

&lt;h3&gt;
  
  
  🟧 AWS: subnet IPv6-only (uma por AZ)
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Spark driver SPARK_PREFER_IPV6 automático (compute)&lt;/li&gt;
&lt;li&gt;Executors 1 ENI IPv6 por worker (compute)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  📡 VPC: resolução e rotas
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Route 53 Resolver DNS64 na subnet (network)&lt;/li&gt;
&lt;li&gt;NAT gateway rota 64:ff9b::/96 (NAT64) (network)&lt;/li&gt;
&lt;li&gt;Egress-only IGW rota ::/0 (network)&lt;/li&gt;
&lt;li&gt;Interface endpoints Glue, STS, KMS, Logs (security)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  🌐 Destinos
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;S3 dual-stack s3.dualstack. (storage)&lt;/li&gt;
&lt;li&gt;Endpoint só IPv4 JDBC legado, API pública (external)&lt;/li&gt;
&lt;li&gt;Endpoint IPv6 AAAA publicado (external)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Flows
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;drv -&amp;gt; r53: DNS query&lt;/li&gt;
&lt;li&gt;exe -&amp;gt; r53: DNS query&lt;/li&gt;
&lt;li&gt;r53 -&amp;gt; nat: A record only: synthesizes 64:ff9b::&lt;/li&gt;
&lt;li&gt;nat -&amp;gt; v4: translates to IPv4, billed per GB&lt;/li&gt;
&lt;li&gt;exe -&amp;gt; eigw: destination with AAAA&lt;/li&gt;
&lt;li&gt;eigw -&amp;gt; s3ds: data reads and writes&lt;/li&gt;
&lt;li&gt;eigw -&amp;gt; v6: egress without ingress&lt;/li&gt;
&lt;li&gt;drv -&amp;gt; vpce: catalog, credentials, keys&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Network remediation: turning prerequisites into invariants
&lt;/h2&gt;

&lt;p&gt;The DNS64 fix took one command. The remediation that prevents recurrence took more work, and it is the one that matters.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Subnet module with a contract:&lt;/strong&gt; the IaC module now requires, for every subnet tagged as an EMR Serverless target, &lt;code&gt;assign_ipv6_address_on_creation = true&lt;/code&gt;, &lt;code&gt;enable_dns64 = true&lt;/code&gt;, a &lt;code&gt;64:ff9b::/96&lt;/code&gt; route to the NAT gateway and a &lt;code&gt;::/0&lt;/code&gt; route to the egress-only internet gateway. Auto-assign only applies to resources created after the change, so turning it on late does not fix an ENI that already exists.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Validation before apply:&lt;/strong&gt; a pipeline check (policy as code, or a test that reads &lt;code&gt;describe-subnets&lt;/code&gt; and &lt;code&gt;describe-route-tables&lt;/code&gt;) fails the deploy if any subnet in the set breaks the contract. Validation that runs after the incident is a postmortem, not a control.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Per-subnet canary:&lt;/strong&gt; since the job picks the subnet and you do not, I create a canary application with a single subnet, one per AZ. A minimal Spark job resolves and connects to the endpoints the batch depends on (catalog, STS, KMS, the data bucket, the source database) and fails loudly. It runs before the batch window and after any network change.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Clean JVM:&lt;/strong&gt; a sweep across every &lt;code&gt;sparkSubmitParameters&lt;/code&gt; and &lt;code&gt;runtimeConfiguration&lt;/code&gt; looking for &lt;code&gt;preferIPv4Stack&lt;/code&gt;. A 2019 workaround cannot decide the network stack in 2026.&lt;/p&gt;

&lt;p&gt;None of this is sophisticated. Its cost is not writing it, it is keeping the contract when someone creates the fourth subnet eighteen months from now without reading this document.&lt;/p&gt;

&lt;h2&gt;
  
  
  The cost the announcement does not mention: NAT64 and S3
&lt;/h2&gt;

&lt;p&gt;"No additional cost" is true for the feature. It is not true for the path the traffic now takes.&lt;/p&gt;

&lt;p&gt;On an IPv6-only subnet, every destination that only publishes an A record gets a synthesized address in &lt;code&gt;64:ff9b::/96&lt;/code&gt;, and that route points to the NAT gateway. The NAT gateway bills per hour and per GB processed; the VPC pricing page uses US$ 0.045 per GB as its example in US East (Ohio). A platform reading 8 TB a night from S3 and sending it through NAT64 pays around US$ 360 a night in processing alone, roughly US$ 10,800 over 30 nights. The S3 gateway endpoint on the workload subnet does not catch this traffic, because the destination is no longer an IPv4 prefix.&lt;/p&gt;

&lt;p&gt;S3 only serves IPv6 through its dual-stack endpoints (&lt;code&gt;s3.dualstack.&amp;lt;region&amp;gt;.amazonaws.com&lt;/code&gt;), and the client has to ask for them explicitly. Do not trust it before measuring: compare the NAT gateway's &lt;code&gt;BytesOutToDestination&lt;/code&gt; before and after the switch.&lt;/p&gt;

&lt;p&gt;Other services need the same care. In the AWS IPv6 support table, Glue has dual-stack and a private endpoint with IPv6, but not the public endpoint; Lake Formation also only has IPv6 through its private endpoint. On IPv6-only, that means interface endpoints for catalog and permissions, or paid NAT64 on every call.&lt;/p&gt;

&lt;p&gt;And security: the S3 documentation warns that policies with IPv4-only &lt;code&gt;aws:SourceIp&lt;/code&gt; make clients incorrectly lose or gain access once they start using IPv6. Review every bucket policy before the switch, not after.&lt;/p&gt;

&lt;h2&gt;
  
  
  The three modes side by side
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Criterion&lt;/th&gt;
&lt;th&gt;IPv4-only&lt;/th&gt;
&lt;th&gt;Dual-stack with opt-in&lt;/th&gt;
&lt;th&gt;IPv6-only&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;IPv4 consumption&lt;/td&gt;
&lt;td&gt;1 per worker&lt;/td&gt;
&lt;td&gt;1 per worker, still consumed&lt;/td&gt;
&lt;td&gt;Zero&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Activation&lt;/td&gt;
&lt;td&gt;-&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;ipv6.enabled&lt;/code&gt; on the application + &lt;code&gt;SPARK_PREFER_IPV6&lt;/code&gt; on the job&lt;/td&gt;
&lt;td&gt;Automatic from the subnets&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;IPv4-only destinations&lt;/td&gt;
&lt;td&gt;Direct&lt;/td&gt;
&lt;td&gt;Direct over IPv4&lt;/td&gt;
&lt;td&gt;Only with DNS64 + NAT64, billed per GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Incident risk&lt;/td&gt;
&lt;td&gt;Exhausting the subnet at peak&lt;/td&gt;
&lt;td&gt;Low: failure falls back to IPv4&lt;/td&gt;
&lt;td&gt;High if any subnet or dependency is not ready&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;When to use&lt;/td&gt;
&lt;td&gt;Address headroom and no IPv6 mandate&lt;/td&gt;
&lt;td&gt;Transition and dependency validation&lt;/td&gt;
&lt;td&gt;Compliance mandate or real exhaustion, with dependencies mapped&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Well-Architected reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;security&lt;/strong&gt;: An IPv6 address is globally unique: egress only through an egress-only gateway, never a &lt;code&gt;::/0&lt;/code&gt; route to the internet gateway on worker subnets. EMR Serverless already refuses security groups with ports open to &lt;code&gt;0.0.0.0/0&lt;/code&gt; or &lt;code&gt;::/0&lt;/code&gt;. Review &lt;code&gt;aws:SourceIp&lt;/code&gt; in bucket policies and prefer &lt;code&gt;aws:SourceVpce&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;reliability&lt;/strong&gt;: Spread subnets across at least two AZs, as the documentation recommends, but treat each one as its own failure unit: a canary per subnet and an alarm per subnet. Automatic retries without per-subnet correlation turn a deterministic defect into noise.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Anti-patterns this retro tries to prevent
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Validating with two jobs:&lt;/strong&gt; with three subnets and one picked per job, two green jobs prove nothing about the third.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Going straight to IPv6-only without a dependency inventory:&lt;/strong&gt; every driver, SDK and endpoint the job touches must support IPv6 or go through NAT64; EMR Serverless does not change your code or third-party endpoints.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Turning on &lt;code&gt;SPARK_PREFER_IPV6&lt;/code&gt; as a global default:&lt;/strong&gt; on an IPv4-only subnet the preference kills the job.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fixing the subnet without restarting the application:&lt;/strong&gt; the mode was fixed at start, and on-call concludes the fix did not work.&lt;/li&gt;
&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;What I would do:&lt;/strong&gt; I would start with dual-stack with the opt-in, on a new application, and keep the IPv6 preference on only for jobs whose dependencies I have already measured. IPv6-only waits for a mandate or real exhaustion, and only after two weeks of green per-subnet canaries and a NAT gateway bytes chart I can explain line by line. The lesson I took from other network migrations is that the defect is almost never in the new service, it is in the subnet someone created with the old module. That is why the contract lives in IaC, not in the wiki.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Lessons and recommendation
&lt;/h2&gt;

&lt;p&gt;Adopt IPv6 on EMR Serverless when the IPv4 space in shared subnets already limits your vCPU ceiling or when there is a formal IPv6 requirement. Use dual-stack with the opt-in while there is any IPv4-only dependency that matters in volume; use IPv6-only when every dependency publishes AAAA or sits behind an interface endpoint, and NAT64 is limited to control calls. Either way, encode DNS64, routes and auto-assign in the subnet module, run a per-subnet canary and measure the NAT gateway before declaring the migration done. The feature is free; an unprepared network bills per GB.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Rating:&lt;/strong&gt; Adote com dual-stack primeiro / Adopt wi&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/about-aws/whats-new/2026/10/amazon-emr-serverless/" rel="noopener noreferrer"&gt;Amazon EMR Serverless now supports IPv6 and dual-stack connectivity (AWS What's New, 16/10/2026)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/ipv6-subnets.html" rel="noopener noreferrer"&gt;Configure IPv6 subnets for EMR Serverless applications&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/vpc-access.html" rel="noopener noreferrer"&gt;Configuring VPC access for EMR Serverless applications (subnet planning)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-nat64-dns64.html" rel="noopener noreferrer"&gt;DNS64 and NAT64 (Amazon VPC User Guide)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/vpc/latest/userguide/egress-only-internet-gateway.html" rel="noopener noreferrer"&gt;Egress-only internet gateway (Amazon VPC User Guide)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/vpc/latest/userguide/aws-ipv6-support.html" rel="noopener noreferrer"&gt;AWS services that support IPv6&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/AmazonS3/latest/developerguide/ipv6-access.html" rel="noopener noreferrer"&gt;Making requests to Amazon S3 over IPv6&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/vpc/pricing/" rel="noopener noreferrer"&gt;Amazon VPC pricing (NAT gateway)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://fernando.moretes.com/blog/retro-emr-serverless-em-ipv6-only-e-a-subnet-que-ficou-sem-dns64" rel="noopener noreferrer"&gt;fernando.moretes.com&lt;/a&gt;. By Fernando F. Azevedo: Senior Solutions Architect.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>awscloud</category>
      <category>emrserverless</category>
      <category>ipv6</category>
      <category>vpc</category>
    </item>
    <item>
      <title>CloudFront dual-stack VPC origins: IPv6 to private origins, with caveats</title>
      <dc:creator>Fernando Azevedo</dc:creator>
      <pubDate>Fri, 16 Oct 2026 17:37:34 +0000</pubDate>
      <link>https://dev.to/fernando_azevedo_6844e930/cloudfront-dual-stack-vpc-origins-ipv6-to-private-origins-with-caveats-1bj0</link>
      <guid>https://dev.to/fernando_azevedo_6844e930/cloudfront-dual-stack-vpc-origins-ipv6-to-private-origins-with-caveats-1bj0</guid>
      <description>&lt;p&gt;For almost two years, anyone who adopted CloudFront VPC origins and had an IPv6 target lived with a bad choice: keep the ALB in a private subnet and stay on IPv4 between edge and origin, or expose the origin to the internet to get end-to-end IPv6. The 10/15/2026 announcement removes that dilemma. The question that matters now is not 'can I get IPv6 to a private origin?', it's 'what in my VPC, my security group and my change process breaks when I turn it on?'. That is the question this analysis answers.&lt;/p&gt;

&lt;h2&gt;
  
  
  The launch in numbers
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;US$ 0&lt;/strong&gt;: Additional cost. Dual-stack VPC origins carry no extra charge; the ENI addresses can be private.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;34&lt;/strong&gt;: Commercial Regions. Every Region with VPC origins, including sa-east-1; four have one excluded AZ.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;55/60&lt;/strong&gt;: Prefix list weight. The CloudFront managed list weighs 55 against the default 60 rules per security group.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What actually changed
&lt;/h2&gt;

&lt;p&gt;VPC origins, launched in November 2024, places an ALB, NLB or EC2 instance in a private subnet and makes CloudFront the single point of entry. The connection arrives through a service-managed ENI created inside your subnet, and access is controlled by a security group AWS creates with the naming pattern &lt;code&gt;CloudFront-VPCOrigins-Service-SG&lt;/code&gt;. Until yesterday that ENI spoke IPv4 only.&lt;/p&gt;

&lt;p&gt;In September 2025 CloudFront gained IPv6 for custom origins, with three options: IPv4 only (default), IPv6 only and dual-stack. That announcement excluded S3 and VPC origins. What shipped now closes half of that gap: VPC origins accept dual-stack. The other half stays open, because the current CloudFront IPv6 documentation still lists IPv6 only as "excluding Amazon S3 and VPC origins".&lt;/p&gt;

&lt;p&gt;In practice, the change is one field: when you create the VPC origin, the console asks for &lt;strong&gt;Origin type&lt;/strong&gt;, and the default is still IPv4. For dual-stack to work, the private subnet needs at least one free IPv6 address in addition to the IPv4 one, because that is where the service ENI is born. Without it, ENI creation fails.&lt;/p&gt;

&lt;p&gt;What does not change: gRPC is still out, Lambda@Edge origin request and origin response triggers are still out, a Gateway Load Balancer cannot be an origin, and neither can an NLB with a TLS listener. If you ruled out VPC origins for one of those reasons, you have no new reason to revisit.&lt;/p&gt;

&lt;h2&gt;
  
  
  End-to-end IPv6 path with a private origin
&lt;/h2&gt;

&lt;p&gt;Every hop where IPv6 can silently fall back to IPv4, and the control that governs that hop.&lt;/p&gt;

&lt;h3&gt;
  
  
  🌐 DNS e borda
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Route 53 alias A + alias AAAA (network)&lt;/li&gt;
&lt;li&gt;CloudFront edge IPv6 de viewer ligado (edge)&lt;/li&gt;
&lt;li&gt;Origin-facing escolhe IPv4 ou IPv6 (edge)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  🟧 AWS: VPC privada (sa-east-1)
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;ENI do serviço 1 IPv4 + 1 IPv6 livres (network)&lt;/li&gt;
&lt;li&gt;ALB interno IpAddressType: dualstack (compute)&lt;/li&gt;
&lt;li&gt;Target group IPv4 ou IPv6 (compute)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  🔐 Controles
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;CloudFront-VPCOrigins-Service-SG fonte da regra de entrada (security)&lt;/li&gt;
&lt;li&gt;NACL de saída efêmeras 1024–65535 IPv4 e IPv6 (security)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Flows
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;viewer -&amp;gt; r53: AAAA lookup&lt;/li&gt;
&lt;li&gt;viewer -&amp;gt; edge: HTTPS over IPv6&lt;/li&gt;
&lt;li&gt;edge -&amp;gt; ofs: cache miss&lt;/li&gt;
&lt;li&gt;ofs -&amp;gt; eni: private connection, family picked by the service&lt;/li&gt;
&lt;li&gt;eni -&amp;gt; alb: allowed by service SG&lt;/li&gt;
&lt;li&gt;alb -&amp;gt; tg: new connection, target group family&lt;/li&gt;
&lt;li&gt;svcsg -&amp;gt; alb: inbound rule&lt;/li&gt;
&lt;li&gt;nacl -&amp;gt; alb: return path&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Where it shines
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The trade-off is gone:&lt;/strong&gt; IPv6 to the origin no longer requires a public subnet, an internet-facing ALB and an IP list to maintain.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Zero license cost:&lt;/strong&gt; no additional charge, and the ENI IPv6 addresses do not need to be public.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Same security model:&lt;/strong&gt; the service SG still restricts the origin to your distributions, now across both families.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Brazil included:&lt;/strong&gt; &lt;code&gt;sa-east-1&lt;/code&gt; is on the list, so a regulated workload that must stay in-country is not left out.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Dual-stack is not guaranteed IPv6
&lt;/h2&gt;

&lt;p&gt;This is the point the announcement does not make explicit, and it matters to whoever has to answer the audit. The CloudFront documentation describes dual-stack mode this way: the service automatically picks IPv4 or IPv6 for the origin connection to prioritize performance and availability. In other words, you don't decide the family of each connection. CloudFront does.&lt;/p&gt;

&lt;p&gt;For most sites that is exactly the right behavior. For a team that has to prove to a regulator or a contract that traffic runs over IPv6 all the way to the origin, it is an evidence problem. Turning on dual-stack produces no proof at all; measurement does.&lt;/p&gt;

&lt;p&gt;The same holds on the viewer side. The documentation itself warns that CloudFront may answer a viewer over IPv4 when its data says the experience will be better, and that the IPv6 share should remain a minority because not every access network supports the protocol. Two automatic decisions, one at each end, and neither is yours.&lt;/p&gt;

&lt;p&gt;What I would do: treat "end-to-end IPv6" as a measured SLO, not a setting. At the edge, the &lt;code&gt;c-ip&lt;/code&gt; field in standard logs shows the viewer family. At the origin, ALB access logs show the address that opened the connection, which for VPC origins is the service ENI. The daily ratio of IPv6 connections to total is the number that goes into the report. If the requirement is IPv6-only, VPC origins still doesn't meet it: IPv6 only remains excluded.&lt;/p&gt;

&lt;h2&gt;
  
  
  Network prerequisites that fail silently
&lt;/h2&gt;

&lt;p&gt;The CloudFront side is one field. The real work is in the VPC, and much of it fails without a clear message.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;IPv6 in the VPC and subnets:&lt;/strong&gt; the ALB only accepts &lt;code&gt;IpAddressType: dualstack&lt;/code&gt; if the VPC and its subnets have an associated IPv6 CIDR block, the route tables route IPv6, and the SG and NACL allow IPv6. In an older account built IPv4-only, that is a project, not a click.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;An address for the ENI:&lt;/strong&gt; the subnet where the VPC origin lives needs a free IPv6 address. A /28 that runs out of IPv4 is already a known problem; with dual-stack the same math repeats in the other family.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Origin security group:&lt;/strong&gt; the documentation gives two options. Managed prefix lists, and for dual-stack that means both, &lt;code&gt;com.amazonaws.global.cloudfront.origin-facing&lt;/code&gt; and &lt;code&gt;com.amazonaws.global.ipv6.cloudfront.origin-facing&lt;/code&gt;. The VPC documentation assigns the CloudFront list a weight of 55, against the default of 60 rules per security group, which leaves 5 rules for everything else. The other option is to reference &lt;code&gt;CloudFront-VPCOrigins-Service-SG&lt;/code&gt;: it takes one rule and restricts the origin to your distributions, not to any CloudFront distribution. That is the one I use.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Outbound NACL:&lt;/strong&gt; inbound NACL rules are not evaluated for traffic from CloudFront, but outbound rules are, on the return path, on ephemeral ports 1024–65535. A NACL keeps IPv6 rules separate from IPv4 ones. If you only allowed &lt;code&gt;0.0.0.0/0&lt;/code&gt;, you will watch the IPv6 handshake die on the way back.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Limits that still apply:&lt;/strong&gt; IPv6 only is still unavailable for VPC origins. gRPC and Lambda@Edge origin triggers are not supported. A Gateway Load Balancer and an NLB with a TLS listener cannot be origins, and an NLB must have a security group attached. And IPv6 ends at the ALB: the connection to targets uses the target group's family, so a dual-stack ALB with an IPv4 target group is IPv6 to the ALB, not to the application.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Migrating a VPC origin already in production
&lt;/h2&gt;

&lt;p&gt;Here is the cost the announcement does not show. Per the documented procedure, updating a VPC origin requires first making sure it is not the default origin of any cache behavior, then disassociating it from &lt;strong&gt;every&lt;/strong&gt; distribution that uses it, editing, waiting for the status to return to &lt;strong&gt;Deployed&lt;/strong&gt; (up to 15 minutes), and only then associating it again. In production, that is a scheduled outage window, multiplied by the number of distributions sharing the origin.&lt;/p&gt;

&lt;p&gt;I would not edit the existing VPC origin. The safer path is to create a second, dual-stack VPC origin pointing at the same ALB ARN, and swap origins through CloudFront continuous deployment: a staging distribution gets the new origin, you test with a slice of traffic, and you promote the configuration when the signals are clean. Rollback is promoting the previous configuration back, without waiting 15 minutes for a VPC origin to propagate.&lt;/p&gt;

&lt;p&gt;The signals I would watch on staging: the distribution's &lt;code&gt;5xxErrorRate&lt;/code&gt; and &lt;code&gt;OriginLatency&lt;/code&gt; in CloudWatch, the ALB's &lt;code&gt;TargetConnectionErrorCount&lt;/code&gt; and &lt;code&gt;HTTPCode_ELB_5XX_Count&lt;/code&gt;, and the share of IPv6 connections in the ALB access logs. If that share stays at zero, the cause is almost always an SG or NACL missing the IPv6 rule, and CloudFront falls back to IPv4 without complaining, which is good for availability and terrible for the target.&lt;/p&gt;

&lt;p&gt;If you share VPC origins across accounts through AWS RAM, add coordination with the owner of every consuming distribution.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to adopt without an outage window
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Prepare the VPC&lt;/strong&gt;: Associate an IPv6 CIDR with the VPC and private subnets, adjust route tables and confirm each subnet has a free IPv6 address for the ENI.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Move the ALB to dual-stack&lt;/strong&gt;: Run &lt;code&gt;aws elbv2 set-ip-address-type --ip-address-type dualstack&lt;/code&gt; on the internal ALB; decide separately whether the target group also moves to IPv6.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Lock down SG and NACL&lt;/strong&gt;: ALB inbound from &lt;code&gt;CloudFront-VPCOrigins-Service-SG&lt;/code&gt;; NACL outbound on ephemeral 1024–65535 for both IPv4 and IPv6.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Create a new VPC origin&lt;/strong&gt;: Origin type dual-stack, same ALB ARN. Don't edit the existing one; wait for Deployed status.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Test on staging and promote&lt;/strong&gt;: Staging distribution with the new origin, partial traffic, 5xx signals and IPv6 share; promote only when both are clean.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Finish the edge&lt;/strong&gt;: Enable viewer IPv6 and add a second AAAA alias in Route 53 for each alternate domain name; review signed URLs using a custom &lt;code&gt;IpAddress&lt;/code&gt; policy.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Anti-patterns
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Editing the VPC origin in production:&lt;/strong&gt; the procedure requires disassociating from every distribution, and the wait of up to 15 minutes becomes an outage.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Claiming compliance because the field says dual-stack:&lt;/strong&gt; CloudFront picks the family per connection; without log measurement, there is no evidence.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Allowing the prefix lists when the origin only serves your own CloudFront:&lt;/strong&gt; it consumes 55 of the default 60 rules and accepts any distribution, not just yours.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Forgetting the target group:&lt;/strong&gt; a dual-stack ALB with IPv4 targets delivers IPv6 to the load balancer, not to the application.&lt;/li&gt;
&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Curator's note:&lt;/strong&gt; In a financial environment I would turn this on first for a low-risk distribution, with staging and an IPv6 share target written down before the change. I have seen an IP family migration pass functional testing and fail the first audit report, because the fallback to IPv4 is silent by design. The lesson: in networking, whatever degrades without an error needs its own metric, otherwise you find out from the wrong person. And I would not edit an existing VPC origin under any circumstances; creating another one and swapping via continuous deployment costs an afternoon and saves a 2 AM window.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Verdict
&lt;/h2&gt;

&lt;p&gt;Adopt it when you have an end-to-end IPv6 requirement, contractual, regulatory or platform-driven, and the origin already is or needs to be in a private subnet: there is no additional cost and the security model does not get worse. Wait if the requirement is IPv6-only, because IPv6 only is still out for VPC origins, or if the VPC has no IPv6 CIDR yet, because then the project is a network project, not a CDN one. Either way, create a new VPC origin and swap via staging; do not edit the one serving traffic.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Rating:&lt;/strong&gt; 4/5&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/about-aws/whats-new/2026/10/cloudfront-vpc-origin-dualstack/" rel="noopener noreferrer"&gt;AWS What's New: Amazon CloudFront announces dual-stack support for VPC origins (15/10/2026)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-vpc-origins.html" rel="noopener noreferrer"&gt;Amazon CloudFront Developer Guide: Restrict access with VPC origins&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/cloudfront-enable-ipv6.html" rel="noopener noreferrer"&gt;Amazon CloudFront Developer Guide: Enable IPv6 for CloudFront distributions&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/about-aws/whats-new/2025/09/amazon-cloudfront-ipv6-origins/" rel="noopener noreferrer"&gt;AWS What's New: Amazon CloudFront announces support for IPv6 origins (09/2025)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/vpc/latest/userguide/working-with-aws-managed-prefix-lists.html" rel="noopener noreferrer"&gt;Amazon VPC User Guide: AWS-managed prefix lists and weights&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-ip-address-type.html" rel="noopener noreferrer"&gt;Elastic Load Balancing: Update the IP address types for your Application Load Balancer&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/blogs/networking-and-content-delivery/optimizing-application-performance-the-strategic-benefits-of-combining-amazon-cloudfront-with-application-load-balancers" rel="noopener noreferrer"&gt;AWS Networking &amp;amp; Content Delivery Blog: Combining Amazon CloudFront with Application Load Balancers&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://fernando.moretes.com/blog/cloudfront-vpc-origins-dual-stack-ipv6-ate-a-origem-privada-com" rel="noopener noreferrer"&gt;fernando.moretes.com&lt;/a&gt;. By Fernando F. Azevedo: Senior Solutions Architect.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>awscloud</category>
      <category>cloudfront</category>
      <category>vpcorigins</category>
      <category>ipv6</category>
    </item>
  </channel>
</rss>
