DEV Community

How Solid Queue Became the Rails 8 default, and More on Open Source Maintainership

Carla Urrea Stabile — Tue, 23 Jun 2026 22:00:00 +0000

Seven gems to run background jobs. That's what 37signals was running before they said "this can't be right."

In Episode 6 of Making Software, I talked to Rosa Gutiérrez, Principal Programmer at 37signals and board member of the Rails Foundation. She built Solid Queue, the database-backed job queue that ships with Rails 8 by default, and we got into how it was built, what open source maintainership actually looks like, and why Ruby is having a moment again.

Seven gems, one problem. 37signals was running seven separate gems just to cover the edge cases Resque didn't handle natively. That became the brief for Solid Queue.
Disks got fast. Redis got optional. Solid Queue is built on the same insight as Solid Cache: modern database storage is cheap and fast enough that you don't need a separate in-memory service for a lot of production workloads.
What happens when millions of developers start using your gem. Going from internal 37signals tooling to bundled-in-Rails means your edge cases multiply overnight. Rosa walks through what that transition felt like, the good contributions, the bad ones, and the more recent arrival of agent-written PRs.
Readable code is a design choice. Rosa spent a lot of time debugging Resque's internals and adjacent gems that had no naming conventions in common. She decided Solid Queue would never make someone feel that way. She also points out she wrote it pre-AI, when you couldn't just ask Claude to explain it to you.
Ruby is well-suited for AI agents, and the community knows it. Convention-over-configuration turns out to be great for agents too. Rosa explains why the Ruby ecosystem is seeing a quiet comeback, and what the Ruby Triathlon says about the community holding it together.

Things that stuck with me

Seven gems to manage background jobs. The team looked at what they were running and said "this can't be right." That became the brief for Solid Queue. Rosa got picked for the project, built it in production at Hey first, iterated on it for months, and shipped it into Rails 8. She keeps calling it luck. I don't think it's luck.

She mentioned almost as a side note that it's mostly agents opening issues and PRs now. "Agents are polite, so that's nice." She'd just been describing contributors who didn't read the README, opened sparse issues, were occasionally rude. That contrast was funny, and also a bit telling.

As Rosa said it, being nice is free.

What's a gem or library you've used where reading the source code took forever? What made it hard? Let me know in the comments!

Listen to the full episode

Available on YouTube, Apple Podcasts, and Spotify.

Thanks for reading! 👋

TPU Developer Hub: A Technical Review of a High-Performance AI Platform

Fernando Azevedo — Tue, 23 Jun 2026 21:52:06 +0000

When Google launched the TPU Developer Hub, the technical signal was clear: the company wants to reduce friction between ML practitioners and specialized acceleration hardware. As an architect who spends a significant portion of time designing inference and training pipelines for financial systems — where every millisecond of latency and every dollar of compute cost must be justified — I read that announcement with productive skepticism. TPUs are not new; what changes is the developer experience layer and the proposition of making this hardware accessible beyond Google's own research labs. In this article, I analyze what the TPU Developer Hub actually delivers, where it differentiates from alternatives like GPU instances on AWS, where it imposes hard trade-offs, and how I would structure an adoption decision in a regulated financial environment.

Numbers that define the context

~4.6x — TPU v5e throughput gain vs. A100 in LLM training (public JAX/MaxText benchmarks). For dense models above 7B parameters in bfloat16; results vary with network topology and batch size
$2.20/h — Cost per TPU v5e chip on-demand (us-central1, 1 chip). Compared to ~$3.06/h per A100 GPU on equivalent p4d.xlarge on AWS us-east-1; cost parity depends heavily on utilization efficiency
256 chips — Practical minimum scale of a TPU v5p pod for training models >70B parameters. Below this threshold, inter-chip communication overhead reduces hardware efficiency to below 60% MFU (Model FLOP Utilization)

What the TPU Developer Hub is and what it actually changes

The TPU Developer Hub is not a new hardware generation — it is a reorganization of the development experience around existing TPUs. The hub centralizes documentation, interactive notebooks, PyTorch/JAX migration guides, fine-tuning examples with models like Gemma and PaLM 2, and access to pre-configured development environments. The stated goal is to reduce the time from "I have a model" to "I am training efficiently on TPU" from weeks to hours.

From an architectural standpoint, what interests me most is the abstraction layer the hub proposes. Historically, working with TPUs required deep mastery of XLA (Accelerated Linear Algebra), the compiler that transforms high-level operations into hardware-optimized instructions. This created a significant entry barrier — teams accustomed to CUDA and PyTorch needed to relearn static compilation paradigms, static tensor shapes, and explicit sharding strategies.

The hub attempts to address this with three layers: (1) MaxText and MaxDiffusion as high-performance reference implementations already optimized for TPU; (2) Pathways as a distributed runtime that abstracts the physical pod topology; and (3) native integration with Vertex AI for job orchestration. For teams already operating in the Google Cloud ecosystem, this vertical integration is genuinely valuable. For teams with hybrid or multi-cloud workloads — which is the reality of most financial environments I know — the story is more complicated.

Where TPUs shine: the use case that justifies the complexity

There is a workload profile where TPUs deliver clear and measurable competitive advantage: training large-scale dense models with large batches and static tensor shapes. Language models above 7B parameters, diffusion models for financial image generation (documents, reports), and embedding models trained on proprietary financial data corpora — all of these fit well within the TPU efficiency profile.

The technical reason is the systolic array architecture of TPUs: they are optimized for matrix multiplication operations in bfloat16, which is exactly what dominates the forward and backward pass of transformers. The XLA compiler, when fed with static shapes, can plan the entire execution of a training step as a single compiled program, eliminating dispatch overhead and maximizing hardware utilization. In public benchmarks from the MaxText project, TPU v5e achieves Model FLOP Utilization (MFU) above 55-60% on models like LLaMA-2 70B — a number that A100 GPUs rarely exceed 45-50% in comparable configurations.

For a bank or fintech that is continuously pre-training or fine-tuning fraud detection, credit scoring, or financial news sentiment analysis models, this efficiency gain translates directly into lower training costs and faster experimentation cycles. A fine-tuning cycle that takes 18 hours on 8x A100s can drop to 6-8 hours on an equivalent TPU v5e slice — and the cost-per-hour difference favors TPUs when utilization is high and consistent.

TPU Developer Hub strengths

Vertical integration with Vertex AI: training jobs, ML pipelines, and model registry in a single control surface, reducing operational overhead for Google Cloud-native teams
MaxText as a high-performance reference: JAX transformer reference implementation already optimized for TPU, with documented and reproducible MFU — eliminates weeks of manual tuning
Pathways runtime: pod topology abstraction that allows scaling from 1 chip to thousands without rewriting sharding code — critical for iterative experimentation
Competitive cost per FLOP at high utilization: when the workload is appropriate (static shapes, large batches, continuous training), the cost per effective TFLOP is 20-35% lower than equivalent GPUs
Curated notebooks and migration guides: real reduction of the entry barrier for PyTorch-first teams that need to migrate to JAX/XLA

Training and Inference Pipeline with TPU Developer Hub in a Hybrid Financial Environment

Typical flow for a financial ML team using TPUs for training and AWS for inference and data governance — a multi-cloud pattern that maximizes cost efficiency without compromising compliance

📦 Data Layer — AWS S3 + Glue

S3 Raw financial data (storage)
AWS Glue ETL + schema validation (compute)
S3 Curated bfloat16 tensors (storage)

🔵 Google Cloud — TPU Training

GCS Bucket mirrored training data (storage)
Vertex AI Training Job (ai)
TPU v5e Pod MaxText / JAX (compute)
Vertex Model Registry (ai)

🟧 AWS — Inference + Governance

AWS Bedrock custom model import (ai)
Lambda inference wrapper (compute)
API Gateway WAF + throttling (security)
CloudWatch SLO dashboards (compute)

🔐 Security & Compliance

AWS KMS CMK encryption (security)
IAM Permission Boundary (security)

Flows

s3-raw -> glue-etl: ingestion
glue-etl -> s3-curated: validated data
s3-curated -> gcs-mirror: cross-cloud replication
gcs-mirror -> vertex-job: loads tensors
vertex-job -> tpu-v5e: dispatches job
tpu-v5e -> model-registry: saves checkpoint
model-registry -> bedrock: exports GGUF/ONNX model
bedrock -> lambda-infer: invokes inference
lambda-infer -> apigw: response
apigw -> cloudwatch: SLO metrics
kms -> s3-curated: CMK at-rest
iam-boundary -> vertex-job: access control

Where it hurts: the real frictions the hub does not resolve

The TPU Developer Hub improves the development experience, but it does not solve the structural problems that make TPUs difficult in regulated financial environments. I will be direct about each of them.

Ecosystem lock-in: JAX is the first-class citizen in the TPU world. PyTorch/XLA exists, but it is a second-class citizen — dynamic operations, conditional control flow, and variable shapes frequently force XLA recompilations that destroy the performance gain. In financial environments where ML models are frequently developed by data science teams with a PyTorch background, migrating to JAX is not trivial. I am not talking about syntax — I am talking about rethinking how you write training loops, how you do debugging (no eager mode by default), and how you integrate with third-party libraries that lack JAX support.

Limited observability outside the Google ecosystem: The hub integrates well with Cloud Monitoring and Cloud Trace, but if your observability stack is OpenTelemetry + Datadog (as most financial environments I operate in), you will need to instrument manually. There is no native OTLP exporter for TPU chip utilization metrics — you depend on Cloud Monitoring and then export via Pub/Sub to your observability backend.

Compliance and data residency: For banks operating under LGPD, BACEN, or equivalent regulations, the question of where training data resides is critical. Replicating curated financial data to GCS in us-central1 to feed a TPU job requires legal analysis and additional technical controls — DLP, tokenization, data processing agreements. The hub does not address this.

Critical pitfalls before committing to TPUs in production: Dynamic shapes are enemy number one: any operation that produces tensors with shapes that vary between training steps forces an XLA recompilation. In models with variable-length attention (financial documents of different sizes), this can increase step time by 10-50x. Always use static padding or sequence bucketing before migrating to TPU. TPU pod preemption: unlike EC2 instances with Savings Plans, TPU pods do not have availability guarantees at all sizes — especially v5p above 512 chips. Plan checkpointing every 15-30 minutes with GCS as the checkpoint backend, and use Orbax for state management. Cross-cloud egress cost: replicating data from S3 to GCS has AWS egress cost (~$0.09/GB) plus GCS ingress cost. For training datasets above 10TB, this can add $900+ to the experiment cost before running a single step.

TPU v5e vs. AWS p4d.24xlarge (8x A100) — Trade-offs for Financial Workloads

Criterion	Dimension	TPU v5e (8 chips)	AWS p4d.24xlarge (8x A100)
On-demand cost/hour	~$17.60 (8 chips × $2.20)	~$32.77	—
MFU on LLM 7B (bfloat16)	55-62% (static shapes)	42-50% (PyTorch FSDP)	—
Native PyTorch support	Partial (PyTorch/XLA, limitations on dynamic ops)	Full (native CUDA, no restrictions)	—
Integration with AWS ecosystem	Requires cross-cloud (S3→GCS, federated IAM)	Native (SageMaker, S3, CloudWatch, KMS)	—
Compliance/data residency	Requires additional legal analysis for financial data	Controllable via AWS regions + KMS CMK + SCPs	—
Low-latency inference (<50ms)	Not recommended (TPUs optimized for batch)	Adequate with TensorRT + Triton	—

The multi-cloud pattern I would use: TPU for training, AWS for serving

After analyzing the TPU Developer Hub in the context of real financial workloads, the architectural pattern I would recommend is neither "migrate everything to TPUs" nor "ignore TPUs and stay on GPU". It is a pattern of separation of responsibilities by model lifecycle phase.

Training and fine-tuning on TPU v5e: For models above 7B parameters with static and well-structured training data (historical transactions, financial time series, regulatory document corpora), the cost-efficiency profile of TPUs is superior. The key is preparing data on the AWS side — schema validation with Glue, tokenization, static padding, serialization in TFRecord or ArrayRecord — before replicating to GCS. This keeps sensitive financial data in the AWS environment for as long as possible and reduces the volume transferred.

Inference on AWS Bedrock or SageMaker: After training, the model is exported in ONNX format or via conversion to GGUF and imported into AWS Bedrock (custom model import) or deployed on SageMaker with ml.g5.xlarge instances for low-latency inference. This keeps the serving layer within the AWS compliance perimeter, with KMS CMK for encryption of models at rest, VPC endpoints for network isolation, and CloudWatch for p99 latency SLOs.

Orchestration with Step Functions: The complete pipeline — data preparation, cross-cloud replication, Vertex AI job trigger, convergence monitoring, model export, Bedrock deployment — can be orchestrated with AWS Step Functions using external activities for the Google Cloud steps. This keeps the control plane in AWS, where you have audit visibility via CloudTrail and can integrate with your existing change management processes.

How to responsibly adopt the TPU Developer Hub in financial environments

1. Validate the workload profile before any migration — Run a tensor shape profiling on your current training pipeline. If more than 20% of steps produce variable shapes or if you use data-dependent conditional control flow, the XLA recompilation cost will negate the hardware gain. Use jax.jit with static_argnums and donate_argnums on a small data subset before committing to full migration.
2. Establish the data perimeter before replicating to GCS — Classify training data with AWS Macie, apply tokenization or pseudonymization of PII/financial fields with AWS Glue + KMS, and document the Data Processing Agreement with Google Cloud before any transfer. Configure VPC Service Controls on Google Cloud to restrict access to the training GCS bucket exclusively to the Vertex AI job service account.
3. Configure checkpointing with Orbax + GCS from day 1 — Use orbax.checkpoint.CheckpointManager with save_interval_steps=500 and max_to_keep=3. Configure the GCS bucket with versioning and Pub/Sub notifications for checkpoint events — this feeds an AWS Lambda that updates the job status in Step Functions and enables automatic resumption in case of pod preemption.
4. Instrument MFU and chip utilization in your observability backend — Cloud Monitoring exposes TPU metrics via compute.googleapis.com/tpu/container/accelerator/duty_cycle. Configure a Pub/Sub sink to export these metrics in real time to an AWS Lambda that publishes them to CloudWatch as custom metrics. Define an alarm if duty cycle drops below 70% for more than 5 minutes — this indicates a data pipeline problem or excessive XLA recompilation.
5. Export the model in a neutral format and validate before deploying on AWS — Use jax.export + ONNX conversion via jax2tf + tf2onnx to produce a portable model artifact. Numerically validate the equivalence between the model output on TPU and GPU using a reference input set with 1e-3 tolerance in bfloat16. Store the ONNX artifact in S3 with versioning enabled and KMS CMK, and use AWS Signer to sign the artifact before deploying on Bedrock.

What the TPU Developer Hub reveals about the industry's direction

The launch of the TPU Developer Hub is symptomatic of a broader shift: the commoditization of ML acceleration hardware is forcing providers to compete at the developer experience layer, not just in raw FLOPS. AWS did the same with Trainium2 and Neuron SDK; Meta with PyTorch 2.0 and torch.compile; NVIDIA with TensorRT-LLM and Triton Inference Server. The battle is no longer for the fastest chip — it is for which ecosystem captures the developer workflow.

For solutions architects in financial environments, this has a direct implication: the choice of ML hardware is increasingly an ecosystem and governance decision, not a performance one. If your organization already has compliance controls, data pipelines, and audit processes built around AWS, the cost of moving training workloads to Google Cloud TPU is not just the compute cost — it is the cost of replicating or federating the entire governance layer.

This does not mean TPUs are the wrong choice. It means the decision needs to be made with eyes open to the total costs: data egress, cross-cloud compliance overhead, engineer requalification for JAX, and the risk of lock-in to a runtime (XLA/Pathways) that has no equivalent on other providers. The TPU Developer Hub reduces the technical friction of adoption, but it does not eliminate the structural costs. For teams with training workloads above 70B parameters and data that can be prepared and anonymized before transfer, the business case is solid. For everyone else, AWS SageMaker with Trn2 or p4d instances remains the path of least operational resistance.

Anti-patterns to avoid when adopting TPUs in financial environments

Migrating PyTorch models without shape profiling: assuming torch_xla will work transparently is the most common mistake — dynamic ops cause cascading recompilations that make training slower than on CPU
Using TPUs for low-latency inference: TPUs are optimized for batch throughput, not single-request latency — p99 inference latency on TPU can be 3-5x higher than on GPU with TensorRT
Transferring non-anonymized financial data to GCS: LGPD/BACEN violation with severe regulatory risk — always apply pseudonymization and tokenization before any cross-cloud transfer
Ignoring egress cost in TCO: calculating only TPU vs. GPU compute cost without including AWS egress + GCS ingress can underestimate the total experiment cost by 15-25%
Not planning checkpointing before starting long jobs: TPU pods can be preempted without warning — 24h jobs without checkpointing every 30min result in total progress loss

My curation note: In practice, I would use TPUs exclusively for the training phase of large models where the data profile is static and well-controlled — and keep the entire serving, governance, and observability layer on AWS. The hardest lesson I learned in financial environments is that ML infrastructure decisions are rarely technical: they are governance decisions disguised as performance decisions. The TPU Developer Hub is genuinely good at reducing technical friction, but the friction it does not resolve — cross-cloud compliance, JAX ecosystem lock-in, fragmented observability — is exactly the friction that matters in regulated production. If you are evaluating TPUs, start with a fine-tuning experiment on synthetic or already-anonymized data, measure real MFU (not theoretical), and calculate the full TCO including egress before any long-term commitment.

Verdict: Powerful in the right niche, costly outside it

The TPU Developer Hub is a real advancement in the developer experience with specialized ML hardware. It genuinely reduces the entry barrier for JAX/XLA, offers high-quality reference implementations with MaxText, and the integration with Vertex AI creates a cohesive training pipeline for Google Cloud-native teams. For organizations already operating in the Google Cloud ecosystem that need to train models above 7B parameters with static data, the value proposition is clear and the ROI is measurable.

However, for most regulated financial environments I know — where AWS is the primary provider, compliance is non-negotiable, and ML teams have a PyTorch background — the TPU Developer Hub solves problems you do not have while creating problems you do not want. The JAX ecosystem lock-in, the cross-cloud compliance friction, and the inadequacy for low-latency inference are structural limitations that no DX improvement will resolve.

My recommendation: use TPUs for training large models when you have data that can be prepared and anonymized on the AWS side before transfer, and when the training cycle is long enough to amortize the setup overhead.

Rating: 7.5/10

References and further reading

Originally published at fernando.moretes.com. By Fernando F. Azevedo — Senior Solutions Architect.

Pixels to Planning: Geospatial Data Platforms on AWS

Fernando Azevedo — Tue, 23 Jun 2026 21:51:23 +0000

When Google Research publishes on 'pixels to planning' — turning satellite imagery into sustainable planning decisions — the real technical signal is not in the computer vision model. It is in the data platform that must exist before any pixel is processed: petabyte-scale raster ingestion, geospatial partitioning that does not break under analytical load, lineage traceability that satisfies carbon credit auditors, and inference with predictable latency for operational decisions. I have designed data pipelines for financial-grade environments where a single wrong location attribute cost millions in incorrect hedging. The discipline I learned in those environments applies directly here — and that is what I will document.

The Real Problem: Geospatial Data Is Not Just Large Files

Most multispectral satellite image files arrive as Cloud-Optimized GeoTIFF (COG) or HDF5, ranging from 500 MB to 5 GB per scene depending on resolution and band count. Sentinel-2 alone produces roughly 1.6 TB per day globally. When you start ingesting multiple constellations — Sentinel, Landsat, Planet, SAR radar data — the volume grows to tens of terabytes daily before you even compute derived indices like NDVI, NDWI, or land surface temperature.

The mistake I see most often is treating this data like log files: dump everything into S3 with date-based prefixes and expect Athena to handle it. It does not. The problem is that geospatial queries have two conflicting partitioning axes: time (when the image was captured) and space (which tile/bounding box it covers). A typical analytical query — 'show me forest cover change in the Amazon between 2022 and 2024' — needs to cross hundreds of tiles over two years. With naive date-only partitioning, Athena scans time partitions without spatial filtering, generating S3 scan costs that can reach tens of dollars per query.

The correct solution is to use a table format with geospatial predicate pushdown — Apache Iceberg with GeoParquet extension, stored on S3, with the catalog managed via AWS Glue Data Catalog. GeoParquet encodes geometries as WKB binary columns with bounding box statistics per row group, allowing Athena (via Trino) to skip row groups that do not intersect the query polygon. In internal benchmarks I have run, this reduced scanned data volume by 60–80% for typical spatial window queries.

Financial-Grade Geospatial Pipeline on AWS

Full flow from satellite image ingestion to sustainable planning decisions, with lineage traceability and governance.

📥 Ingestão & Landing

S3 Raw Zone COG / HDF5 (storage)
SQS FIFO S3 Event Notifications (messaging)
Lambda Validação & Tag (compute)

⚙️ Processamento & Curadoria

Glue Spark Job COG → GeoParquet/Iceberg (data)
S3 Curated Zone Iceberg + GeoParquet (storage)
Glue Data Catalog Iceberg Metastore (data)

🤖 ML & Inferência

SageMaker Training Segmentação / NDVI (ai)
SageMaker Endpoint Inferência em Tempo Real (ai)
SageMaker Batch Transform em Escala (ai)

📊 Consumo & Governança

Athena Query Espacial (data)
Lake Formation RBAC + Column-level (security)
SageMaker Lineage + OpenLineage (data)
API Gateway Planning API (edge)

Flows

sat -> s3raw: COG Upload
s3raw -> sqs: S3 Event
sqs -> lambda_ingest: Trigger
lambda_ingest -> glue: Start Job
glue -> s3curated: Write Iceberg
glue -> glue_catalog: Register Schema
s3curated -> sm_train: Dataset
sm_train -> sm_ep: Deploy Model
s3curated -> batch: Batch Scoring
s3curated -> athena: Spatial Query
athena -> lf: Access Control
sm_ep -> apigw: Prediction
batch -> lineage: Lineage
glue -> lineage: Lineage

Geospatial Partitioning: The Decision That Defines Cost and Latency

After resolving the storage format, the second critical decision is the partitioning strategy in S3 and Iceberg. For geospatial data, I use a three-level hierarchy: year/month as the first-level partition (for temporal pruning), H3 grid resolution 4 as the second-level partition (covering approximately 1,770 km² per cell, generating ~5,000 cells for global coverage), and sensor as the third level.

H3 (Uber's hierarchical hexagonal grid library) has a critical property for geospatial data: cells at the same resolution level have approximately equal area, meaning partition file sizes are predictable — something rectangular bounding box partitioning does not guarantee. By indexing each scene's geometries with h3.polyfill() in the Glue Job, you can join tables from different sensors using the H3 index as a key, without expensive geometric intersection operations at query time.

An important operational detail: the Glue Job converting COG to GeoParquet should be configured with --conf spark.sql.shuffle.partitions=200 and G.2X workers (8 vCPU, 32 GB RAM) to process multispectral bands without disk spill. For Sentinel-2 10m resolution scenes (13 bands, ~800 MB per scene), processing a full day of global ingestion takes approximately 45 minutes with 20 G.2X workers — a cost of roughly USD 12 per execution. That number matters when you are planning a continuous operations budget.

The Iceberg table should be configured with write.target-file-size-bytes=268435456 (256 MB) and compaction scheduled via Glue Workflow every 6 hours to avoid the small files problem that degrades Athena performance.

Playbook: Building the Geospatial Pipeline on AWS

1. Define the Landing Zone with Lifecycle Policy — Create a dedicated S3 bucket for raw data with S3 Intelligent-Tiering enabled from day one. Configure Object Lock in COMPLIANCE mode for reference data (baseline images) that require immutability for carbon credit auditing. Enable S3 Event Notifications to SQS FIFO with deduplication ID based on the object ETag — this prevents duplicate reprocessing when the same file is re-uploaded by a data provider.
2. Build the COG → GeoParquet/Iceberg Glue Transformation Job — Use Glue 4.0 with native Iceberg support. Add geoparquet, h3-py, and rasterio dependencies via Glue Python Shell or as an additional JAR. The job must: (a) read the COG with rasterio using windowed reads to avoid OOM, (b) compute the H3 index for each tile, (c) write to partitioned GeoParquet, (d) execute MERGE INTO on the Iceberg table to support idempotent re-ingestion using the composite key (scene_id, acquisition_date, sensor).
3. Configure Fine-Grained Access Control with Lake Formation — Register the S3 location in Lake Formation and use Cell-Level Security to restrict access by geometry — for example, climate credit analysts for Brazil can only query H3 cells within the national territory polygon. This is implemented via Row Filter Expressions in Lake Formation with the condition h3_index IN (SELECT h3_index FROM geo_reference WHERE country = 'BRA'). Combine with Column-Level Security to protect sensitive land ownership metadata.
4. Train and Register Models with Full Traceability — Use SageMaker Experiments to track each training run with the Iceberg dataset version metadata (snapshot ID). Register the model in SageMaker Model Registry with manual approval for models that feed financial decisions (natural asset valuation, climate credit scoring). Configure SageMaker Lineage Tracking and export to OpenLineage/Marquez for integration with external data governance tools. The Iceberg snapshot ID as a training parameter is what allows exact reproduction of the dataset used in any audited model.
5. Expose Inference with Controlled Latency via API Gateway — For real-time planning queries (e.g., 'what is the deforestation risk in this polygon over the next 12 months?'), use SageMaker Real-Time Endpoint with ml.g4dn.xlarge instances (1 T4 GPU) and configure Autoscaling with a 70% GPU utilization target. Place REST API Gateway in front with Usage Plans and API Keys per tenant. Configure the API Gateway timeout to 29 seconds (maximum limit) and implement a circuit breaker in the integration Lambda to fall back to cached results in DynamoDB when the endpoint is under pressure.
6. Instrument Observability with OpenTelemetry — Instrument the full pipeline with OpenTelemetry: trace spans from SQS to the inference endpoint, ingestion throughput metrics (scenes/hour), Athena query latency by spatial query type, and model drift (prediction distribution vs. baseline). Send to CloudWatch with EMF (Embedded Metric Format) for custom metrics and configure alarms on P99 inference latency > 8 seconds and Glue Job error rate > 1%. These two SLOs are the minimum to operate with confidence.

GeoParquet + Iceberg: The Game-Changing Combo: If you can only make one architectural change today: migrate your geospatial data from raw GeoTIFF/Shapefile in S3 to GeoParquet on Iceberg tables with a Glue catalog. The migration cost is 1–2 engineering sprints. The return is a 60–80% reduction in Athena scan costs and elimination of geospatial joins at query time. This is not premature optimization — it is a prerequisite for any geospatial analysis at scale.

Data Governance for Financial Decisions: Lineage Is Not Optional

When Earth AI platform outputs feed financial decisions — carbon credit valuation, climate risk scoring for credit portfolios, parametric insurance pricing based on vegetation indices — lineage traceability stops being a best practice and becomes a regulatory requirement. In Brazil, BACEN already requires financial institutions to demonstrate the full methodology behind climate risk models (CMN Resolution 4.945/2021). In Europe, the EU AI Act classifies environmental risk scoring systems as 'high-risk', requiring training data documentation.

In practice, this means each model prediction must be traceable to: (1) the exact Iceberg dataset snapshot used in training, (2) the version of the Glue Job code that processed the raw data, (3) the model version in SageMaker Model Registry, and (4) the feature engineering repository commit. This is what I call quadruple traceability — and most implementations I see cover only (3).

The technical implementation uses SageMaker Lineage Tracking to capture the model → dataset → processing chain, with the Iceberg snapshot ID as the immutable anchor. For integration with external governance tools (Collibra, Alation, DataHub), export lineage events via EventBridge to a Lambda that converts them to OpenLineage format and sends them to the Marquez API. This pattern allows external auditors to query the full lineage without needing direct access to the AWS account.

A critical detail: Lake Formation must have audit logging enabled for all data access operations, with logs sent to a separate S3 bucket with Object Lock. In carbon credit audits I have participated in, the absence of data access audit logs was the primary blocker for certification.

Reference Numbers for Sizing

~USD 12 — Cost per Glue Job run (20 G.2X workers, 45 min). To process 1 day of global Sentinel-2 ingestion (~800 scenes)
60–80% — Reduction in Athena scan volume with GeoParquet + H3. Compared to naive date-only partitioning on raw GeoTIFF
<8s P99 — Latency SLO for geospatial risk inference. ml.g4dn.xlarge with ~200M parameter segmentation models

Defense in Depth: Location Data Is Sensitive Data

There is a dangerous tendency to treat geospatial data as 'just map data' — public, non-sensitive, requiring no rigorous controls. This is a serious mistake. High-resolution location data combined with satellite image time series can reveal: troop movements, industrial facility production capacity, crop conditions before public reports (material non-public information for insider trading purposes), and land occupation patterns with legal implications.

The security architecture I implement for these systems follows Zero Trust with four layers: network (VPC with private endpoints for S3, SageMaker, and Glue — no traffic leaving to the public internet), identity (IAM roles with aws:RequestedRegion and aws:SourceVpc conditions to ensure only services within the VPC access the data), data (dedicated KMS CMK per data classification — one CMK for raw data, another for processed data, another for model outputs — with key policies requiring kms:ViaService for access only via specific AWS services), and application (Lake Formation with Row/Column-level security).

A specific pattern I use to protect highly sensitive data: the processed data S3 bucket has a bucket policy with aws:PrincipalOrgPaths that restricts access only to specific OUs within the AWS Organization. This ensures that even if an IAM role is compromised in another account in the organization, it will not have access to the sensitive geospatial data.

For LGPD and GDPR compliance with data that may contain personal information (urban area imagery at sub-meter resolution), implement a face/license plate detection and anonymization step as part of the Glue transformation Job, before any data reaches the curated zone.

Anti-Patterns I Have Seen in the Field

Storing raw GeoTIFF as the analytical source of truth: GeoTIFF has no predicate pushdown. Every query scans the entire file. For analysis, convert to GeoParquet/Iceberg. Keep the original GeoTIFF only for reproducibility and reprocessing.
Using SageMaker Real-Time Endpoint for batch scoring of millions of polygons: The cost and latency of individual calls make this prohibitive. Use SageMaker Batch Transform with S3 as source/sink — processes 1M polygons in minutes at a fraction of the cost.
Ignoring the small files problem after compaction: Iceberg with many incremental writes generates thousands of small files. Without scheduled compaction, Athena pays the overhead of listing and opening each file. Configure OPTIMIZE via Glue Workflow every 6 hours.
Training models without pinning the dataset snapshot ID: Without the Iceberg snapshot ID as a training parameter, you cannot reproduce the exact dataset of any model in production. This is unacceptable in regulated environments.
Exposing geospatial inference endpoints without per-tenant rate limiting: Large bounding box queries can consume disproportionate resources. Implement Usage Plans in API Gateway with limits per tier (e.g., 100 req/min for basic tier, 1,000 req/min for enterprise).
Assuming public satellite data (Sentinel, Landsat) requires no access control: The raw data may be public, but derived indices, trained models, and calculated risk scores are proprietary assets. Treat them as such from day one.

MLOps for Geospatial Models: Drift Is Not Just Statistical

Computer vision models for geospatial data have a type of drift that does not appear in standard statistical monitors: sensor drift. When a satellite provider updates radiometric calibration processing, or when you add a new constellation to the pipeline, the pixel value distribution changes even though the physical world has not. A model trained on Sentinel-2 L2A can silently degrade when it starts receiving data from a new sensor without retraining.

The solution is to implement drift monitoring at two levels: (1) feature drift using SageMaker Model Monitor with a baseline calculated separately by sensor and by season (summer vs. winter vegetation has completely different distributions), and (2) concept drift by monitoring the output prediction distribution against ground truth collected periodically via crowdsourcing or field validation.

For retraining, I use a continuous training with human approval pattern: a Step Functions workflow triggered when the drift score exceeds a threshold (e.g., PSI > 0.2 for any input feature), executes retraining with the last 90 days of Iceberg data, registers the new model in Model Registry with 'PendingApproval' status, and notifies the data science team via SNS. Manual approval is mandatory before production deployment — not for bureaucratic reasons, but because in financial decisions, a silently degrading model can cause systemic damage before detection.

A concrete capacity number: for land cover segmentation models (U-Net with ResNet-50 backbone, ~25M parameters), full retraining on 90 days of Sentinel-2 data for Brazil (~180K scenes) takes approximately 8 hours on an ml.p3.8xlarge instance (4 V100 GPUs) — a cost of ~USD 90. This is acceptable for monthly or drift-triggered retraining.

Frequently Asked Questions in Geospatial Platform Architecture

Should I use Amazon Location Service or build my own geospatial stack?

Amazon Location Service is excellent for geocoding, routing, and real-time asset tracking use cases. For satellite image analysis, spectral indices, and geospatial ML, you need the full stack (S3 + Glue + GeoParquet + SageMaker). The two are not competitors — use Location Service for the operational location layer and the analytical stack for image processing.

What is the estimated monthly cost for a national-scale Earth AI platform (Brazil)?

For national coverage of Brazil with Sentinel-2 (10m resolution, 5-day revisit): ~USD 800/month in S3 (2 years of processed data storage ~50 TB), ~USD 360/month in Glue Jobs (30 runs/month), ~USD 200/month in Athena (analytical queries), ~USD 400/month in SageMaker Endpoint (ml.g4dn.xlarge, 24/7). Total: ~USD 1,760/month before retraining and auxiliary services. Scales linearly with additional sensors.

How to handle clouds and missing data in satellite image time series?

Store the cloud mask (SCL band in Sentinel-2) as a separate column in GeoParquet. For time series analysis, use temporal interpolation with STAC (SpatioTemporal Asset Catalog) to identify the most recent cloud-free image for each pixel. ML models should be trained with synthetic cloud augmentation for robustness. For critical decisions, always include cloud cover percentage as a confidence metadata field in the inference output.

How to ensure the pipeline is resilient to satellite data ingestion failures?

Use SQS FIFO with a Dead Letter Queue (DLQ) configured for 3 attempts before moving to DLQ. Configure a monitoring Lambda that reads the DLQ every hour and sends alerts via SNS with the scene_id and error. For Glue Job failures, use the native retry mechanism (max 3 retries with exponential backoff) and configure EventBridge notification for persistent failures. Iceberg MERGE INTO guarantees idempotency — reprocessing an already-processed scene does not create duplicates.

AWS Well-Architected Lenses for Earth AI Platforms

security: KMS CMK per data classification, Lake Formation with Cell-Level Security by geometry, private VPC endpoints for all data services, Object Lock in COMPLIANCE mode for auditable reference data, and full audit logging with 7-year retention for regulatory compliance.
reliability: SQS FIFO with DLQ for resilient ingestion, Iceberg MERGE INTO for reprocessing idempotency, Multi-AZ for critical SageMaker Endpoints, and Step Functions with compensation for retraining workflows.
performance: GeoParquet with H3 for geospatial predicate pushdown, Glue G.2X workers for multispectral band processing without spill, Iceberg compaction every 6 hours to avoid small files, and SageMaker Batch Transform for at-scale scoring.

Architect's Note: What strikes me about the Google Research signal is not the model — it is the implication that sustainable planning decisions now depend on data pipelines that most organizations still do not know how to build. If I were starting this project today, I would invest the first two weeks exclusively in getting geospatial partitioning and data lineage right — not in the model. In my experience, the model is the easy part; the platform that feeds it with traceable, correctly partitioned, and governed data is where projects fail. The most expensive lesson I have learned: location data without access auditing is a regulatory liability waiting to be triggered — and in financial environments, that cost is always greater than the cost of doing it right from the start.

Verdict: Earth AI Is a Data Platform, Not an ML Project

The transition from 'pixels to planning' — from raw satellite imagery to sustainable operational decisions — is technically feasible on AWS today, at an accessible operational cost for mid-sized organizations. But success depends on architectural decisions that must be made before the first model is trained: storage format (GeoParquet/Iceberg, not raw GeoTIFF), partitioning strategy (hierarchical H3, not just by date), lineage governance (quadruple traceability), and defense in depth (KMS + Lake Formation + private VPC). For organizations operating in regulated environments — financial, insurance, carbon credit — traceability and audit logging are not optional. Start with the data platform. The model comes after.

Technical References

Originally published at fernando.moretes.com. By Fernando F. Azevedo — Senior Solutions Architect.

Accounts are not just storage. They're one of the reasons Solana is fast.

Tanya Prajapati — Tue, 23 Jun 2026 21:50:49 +0000

This day-27 of learning Solana

For the past few days, I have explored accounts and how data is stored in those accounts.
This the summary of all those important concepts about account every Solana beginner should know.

Solana does not uses any mempool to store transactions when they are being processed. Instead, transaction are stored in a block and marked as successfully executed while they are being parallely processed. This helps in handling large number of transactions in lightening speed.

Accounts are used to store data while programs are used to store executable code (Like smart contract in EVM).

In Solana, there are more than one accoutns each with different purposes and significance.
The System Program (Owner account) : It uses owner account address which is a string of 1's. For every system , the owner address is same. However this does not means that anyone can exploit the transaction or the account.

In Solana , the address is of two types-

The public key: It needs a private key which ensures the account is secure. Only the one who has private key can use that account and make transactions through it.
Program Derived Address: The address is deterministically generated by a program using program ID and a set of seeds. These addresses are owned entirely by a program (smart contract). The PDA act as a specialized smart locker while the user's public key act as the account owner.

Solana uses dynamic parallel account locking -
Transactions must list every single account they intend to interact with before they run, the validator engine uses an advanced system of Read-Locks and Write-Locks. This prevents the "double-spend" problem while allowing entirely unrelated accounts to process in parallel without bottlenecking the network.

GPT-5 vs Claude vs Nova on Bedrock: A Production Governance Bake-off

Fernando Azevedo — Tue, 23 Jun 2026 21:50:48 +0000

The arrival of GPT-5.5, GPT-5.4, and Codex on Amazon Bedrock is not just a product event — it is a signal that Bedrock is consolidating as the unified control plane for frontier models in enterprise environments. For teams operating in regulated sectors, the question has shifted from 'which model to use?' to 'how do we govern multiple frontier models with the same security, traceability, and cost controls we already apply to the rest of our AWS infrastructure?' This analysis does exactly that bake-off: GPT-5.5 vs Claude 3.7 Sonnet vs Amazon Nova Pro, focused on production, not benchmarks.

What changed when GPT-5 landed on Bedrock

Before OpenAI models arrived on Bedrock, choosing GPT-4 or GPT-4o meant leaving the AWS perimeter: direct calls to the OpenAI API, secrets managed outside Secrets Manager, logs that bypassed CloudTrail, and data potentially leaving your residency region. For teams requiring LGPD, PCI-DSS, or SOC 2 compliance, that was a real governance cost, not a theoretical one.

With GPT-5.5 and Codex available via bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream, the model becomes just another ARN resource. That means the IAM policies you already have — including conditions like aws:RequestedRegion, bedrock:modelId, and aws:PrincipalTag — apply directly. CloudTrail records every invocation. Amazon Bedrock Guardrails, with its content filters, PII detection, and grounding checks, covers GPT-5.5 the same way it covers Claude or Nova.

What this does not solve: network latency to regions where the model is still served via cross-region endpoints, and the fact that GPT-5.5 weights do not reside in your account — you are consuming a hosted model, not a deployed one. For use cases requiring full inference isolation, such as document analysis with classified customer data, this remains a threat model item that needs explicit documentation.

The dimension benchmarks miss: operational behavior

Academic benchmarks measure capability under controlled conditions. In financial production, what matters is behavior under load, p99 latency consistency, and the real cost of a response — not the average cost, but the cost of an 8k-token prompt with 2k output at peak hours.

Claude 3.7 Sonnet has a characteristic that matters greatly for agentic workflows: extended thinking mode produces chained reasoning that is auditable. In compliance contexts, being able to show the intermediate reasoning of a credit decision or fraud triage has direct regulatory value. GPT-5.5 also supports chain-of-thought, but the level of control over reasoning verbosity and the separation between scratchpad and final output is still less granular via the Bedrock API than what Anthropic exposes natively.

Amazon Nova Pro, on the other hand, is the only one of the three where you have full visibility into the model lifecycle within AWS. It supports fine-tuning via Bedrock Custom Model Jobs, meaning you can adapt the model to domain-specific vocabulary — derivatives terminology, for example — without relying on prompt engineering. Nova Pro's cost per token is significantly lower, which matters when you are processing millions of documents in batch with Bedrock Batch Inference.

The most common failure mode I see in production is not the model being wrong — it is the system lacking sufficient observability to know when the model was wrong. That leads directly to the instrumentation question.

Unified Control Plane: Model Governance on Bedrock

Inference request flow through Bedrock governance layers, showing how GPT-5.5, Claude, and Nova share the same security and observability controls

🔐 AWS — Segurança e Identidade

IAM Policy bedrock:modelId condition (security)
KMS Encrypt at rest / in transit (security)

🟧 Amazon Bedrock — Plano de Controle

Bedrock Guardrails PII, content, grounding (security)
Bedrock API Gateway InvokeModel / Stream (compute)
Model Invocation Logging S3 + CloudTrail (data)

🤖 Modelos Frontier

GPT-5.5 / Codex OpenAI via Bedrock (ai)
Claude 3.7 Sonnet Extended Thinking (ai)
Amazon Nova Pro Fine-tune + Batch (ai)

📊 Observabilidade

CloudWatch Latency P99 / Tokens (data)
OpenTelemetry Trace ID por invocação (data)

Flows

client -> iam: authentication
iam -> guardrails: policy enforced
guardrails -> gateway: filtered prompt
gateway -> gpt5: InvokeModel
gateway -> claude: InvokeModel
gateway -> nova: InvokeModel
gateway -> logging: async log
kms -> logging: encryption
logging -> cw: metrics
gateway -> otel: trace span

Instrumentation: where most teams get it wrong

Bedrock emits native metrics to CloudWatch: InvocationLatency, InputTokenCount, OutputTokenCount, InvocationClientErrors, InvocationThrottles. But these metrics alone are insufficient to operate an AI system in financial production. What is missing is correlation between the model invocation and business context — which user, which product, which decision was influenced by that response.

The approach that works is instrumenting with OpenTelemetry at the application level, propagating a trace ID that crosses the Bedrock call and is included in the Model Invocation Logging payload. When you enable Model Invocation Logging with S3 + CloudWatch Logs as destination, each record includes the Bedrock requestId. If you inject that requestId as an attribute in your OTel span, you can correlate a customer complaint with the exact prompt and response that generated that decision — that is real auditability.

For GPT-5.5 specifically, one watch point: the model supports response_format: json_object and structured outputs, but schema validation happens on the model side, not in Guardrails. If you need to guarantee that the response respects a specific schema before persisting to DynamoDB, add a validation step in the Lambda that processes the response — do not assume the model will always return valid JSON under load or with adversarial prompts.

Claude 3.7 with extended thinking exposes the reasoning block as a separate field in the response. Store that field in S3 with a 7-year retention policy if you are in a regulated environment — it is decision-making evidence, not just a technical log.

Real cost: beyond price per token

Frontier model cost comparisons frequently stop at input/output token price. That is the smallest component of total cost in production systems. The components that dominate cost are: (1) tokens wasted by poorly structured prompts, (2) retries due to throttling, and (3) the cost of operating the system around the model.

GPT-5.5 has a higher price per token than Claude 3.7 Sonnet and significantly higher than Nova Pro. For a document analysis workload processing 10 million pages per month with an average context of 4k tokens per page, the cost difference between GPT-5.5 and Nova Pro can be on the order of 5-8x. This is not an argument against using GPT-5.5 — it is an argument for using it selectively, in cases where its differentiated reasoning capability justifies the cost.

Bedrock Batch Inference changes the calculation for async workloads. With batch, you get up to 50% discount on token price for Claude and Nova. GPT-5.5 on Bedrock does not yet support batch inference at the time of this analysis — meaning that for large-scale processing, you need to manage your own queue (SQS + Lambda with reserved concurrency) and handle the account-level TPM (tokens per minute) limits.

Bedrock TPM limits for third-party models like GPT-5.5 are managed via service quota, and increases require an AWS Support request. In multi-tenant environments where multiple products share the same AWS account, this can become a bottleneck. The solution is to use AWS Organizations with separate accounts per product and independent quotas — do not share TPM limits between critical and experimental workloads.

GPT-5.5 vs Claude 3.7 Sonnet vs Amazon Nova Pro — Technical Comparison

Criterion	Dimension	GPT-5.5 (OpenAI via Bedrock)	Claude 3.7 Sonnet (Anthropic)	Amazon Nova Pro
Relative cost per token (input)	High (baseline ~$3/MTok)	Medium (~$3/MTok)	Low (~$0.8/MTok)	—
Batch Inference support (Bedrock)	No (at time of analysis)	Yes — up to 50% discount	Yes — up to 50% discount	—
Fine-tuning via Bedrock	Not available	Not available	Yes — Custom Model Jobs	—
Auditable reasoning (structured CoT)	Partial — via structured outputs	Yes — separate extended thinking block	Partial — via prompt engineering	—
Bedrock Guardrails coverage	Yes — same controls	Yes — same controls	Yes — same controls	—
P50 latency (2k token prompt)	~1.8s (estimate; varies by region)	~1.5s (without extended thinking)	~1.2s	—
Model weight residency in AWS account	No — hosted by OpenAI	No — hosted by Anthropic	Yes — Amazon-native	—
Codex / specialized code generation	Yes — Codex available on Bedrock	Strong — Claude 3.7 is top-tier for code	Competent — best with fine-tuning	—

Decision Matrix: Which Model for Which Workload?

GPT-5.5 via Bedrock

Pros

Top-tier reasoning capability for complex, ambiguous tasks
Codex for code generation in AI-assisted DevOps pipelines
Unified governance via IAM, CloudTrail, and Guardrails — no AWS perimeter exit
Structured outputs with JSON schema for direct downstream system integration

Cons

Higher cost per token; no batch inference support on Bedrock
Weights do not reside in AWS account — implications for sensitive data threat models
TPM limits managed via service quota; increases require AWS Support
No fine-tuning available via Bedrock

Verdict: Best for: high-complexity reasoning tasks (legal analysis, due diligence), code generation in CI/CD pipelines, and cases where response quality justifies the premium cost.

Claude 3.7 Sonnet

Pros

Extended thinking with separate, auditable reasoning block — direct regulatory value
Batch inference support with up to 50% discount for async workloads
Excellent at code and technical analysis; consistent in long contexts
Competitive pricing with GPT-5.5 for comparable quality on many tasks

Cons

Extended thinking significantly increases latency — not suitable for real-time inference
No fine-tuning via Bedrock; adaptation relies on prompt engineering and RAG
Weights also do not reside in AWS account

Verdict: Best for: agentic workflows requiring auditable reasoning, regulatory document analysis, fraud triage with explainability, and high-quality batch processing.

Amazon Nova Pro

Pros

Lowest cost per token — 5-8x cheaper than GPT-5.5 for high-volume workloads
Fine-tuning via Bedrock Custom Model Jobs — domain adaptation without prompt engineering
Amazon-native weights; best posture for ultra-sensitive data threat models
Batch inference support; lowest P50 latency among the three

Cons

Reasoning capability below GPT-5.5 and Claude 3.7 on high-complexity tasks
Fine-tuning requires quality dataset and MLOps pipeline — operational overhead
Smaller third-party tool and integration ecosystem

Verdict: Best for: large-scale processing (millions of documents), classification and extraction tasks where fine-tuning pays off, and environments with stricter data sovereignty requirements.

The routing pattern that resolves the dilemma: The answer to 'which model to use?' in mature production systems is not a single choice — it is a router. Implement an AI Gateway in Lambda or ECS that classifies each request by complexity, data sensitivity, and latency requirement, and routes to the appropriate model. Low-complexity, high-volume requests go to Nova Pro. Analyses requiring auditable reasoning go to Claude 3.7 with extended thinking. Code generation in CI/CD pipelines goes to Codex. Same Guardrails, same CloudTrail, same trace ID — unified governance with workload-optimized cost. This pattern reduces total cost by 40-60% compared to using GPT-5.5 for everything, without sacrificing quality where it matters.

Numbers that guide the routing decision

5-8x — Cost per token difference: GPT-5.5 vs Nova Pro. For high-volume workloads, intelligent routing is the largest cost lever available on Bedrock
50% — Maximum discount with Batch Inference (Claude and Nova). Async workloads — document analysis, data enrichment — should use batch by default
7 anos — Recommended retention for reasoning logs in regulated environments. Claude 3.7's extended thinking block is decision-process evidence for regulatory audits

Anti-patterns I encounter in production

Using GPT-5.5 for all workloads because it is 'the most capable' — ignores that 70% of tasks do not need frontier reasoning and pays 5-8x more for it
Not enabling Model Invocation Logging — without prompt/response logs, regulatory audit is impossible and quality regression debugging is blind
Assuming Bedrock Guardrails replaces schema validation in code — Guardrails filters content, not data structure; invalid JSON still passes through
Sharing TPM limits between critical and experimental workloads in the same AWS account — a token-burst experiment can throttle a production feature
Not propagating trace IDs in Bedrock calls — loses the correlation between business decision and model invocation, making incident investigations much slower

My curation note: In practice, what I would do: start with Claude 3.7 Sonnet as the default model for any new workload in a financial environment — the auditable extended thinking is worth more than the cost difference versus Nova Pro when you are in a regulated sector. Introduce GPT-5.5 and Codex specifically for the AI-assisted DevOps pipeline, where code generation quality justifies the premium cost. Nova Pro would enter as a routing destination for large-scale classification and extraction, with fine-tuning trained on domain vocabulary. The lesson I learned the hard way: the biggest risk is not choosing the wrong model — it is not having sufficient observability to know when any model is wrong. Invest in trace IDs, Model Invocation Logging, and quality drift alerts before optimizing which model to use.

Recommendation: do not choose a model, build a router

The arrival of GPT-5.5 and Codex on Bedrock does not make Claude or Nova obsolete — it completes the portfolio. The recommendation is clear: implement an AI Gateway that routes by workload, not by model preference. Use Claude 3.7 Sonnet as the default for tasks requiring auditable reasoning in regulated environments. Use GPT-5.5 and Codex for code generation and high-complexity reasoning tasks where the premium cost is justified by value. Use Nova Pro for large-scale processing and cases where domain fine-tuning is viable. In all cases: enable Model Invocation Logging on day one, propagate trace IDs, and treat TPM limits as an infrastructure quota requiring capacity planning — not a configuration detail. Unified governance on Bedrock is the real asset here; the models are commodities that will evolve. Build your platform around the controls, not around a specific model.

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo — Senior Solutions Architect.

Modern KYC: Serverless, AI and Audit Trails in Financial Services

Fernando Azevedo — Tue, 23 Jun 2026 21:50:43 +0000

For years, KYC was treated as a workflow problem — forms, manual review queues and batch jobs running at 2 a.m. Regulators tolerated day-long latencies because everyone operated at the same pace. That tacit contract is unraveling: open finance, instant payments and regulatory pressure for real-time auditable decisions are forcing an architectural rupture. The signal I analyze here is not about replacing analysts with AI — it is about redesigning the KYC pipeline as a first-class system: event-driven, serverless where it makes sense, with AI as a co-pilot and auditability as a first-class architectural citizen, not a compliance afterthought.

The Cost of Legacy KYC — and What Changes

~$50 — Average cost per manual KYC onboarding in traditional banks. Source: Thomson Reuters 2023 market estimates; includes analysts, rework and tooling
<2 min — Target onboarding latency in fintechs with serverless + AI pipelines. Includes document extraction, identity verification and risk scoring — no human intervention for low-risk cases
60-80% — Reduction in cases escalated to manual review with well-calibrated AI triage. Critically depends on training data quality and confidence thresholds configured per product

The Signal: Why Serverless KYC Is Emerging Now

The movement toward serverless KYC pipelines is not a technological novelty — it is a convergence of pressures that have finally made the architecture both viable and necessary at the same time.

First, tooling maturity. AWS Step Functions Express Workflows has reached a point where orchestrating a 15-step pipeline with conditional branching, exponential retries and compensations is operationally sustainable. The 5-minute per-execution limit of Express Workflows is irrelevant for real-time KYC — if your verification flow takes longer than that, the problem is not the orchestrator, it is the design.

Second, Amazon Textract and Bedrock changed the document extraction equation. Previously, you needed a custom ML pipeline to extract data from driver's licenses, passports or income statements with acceptable accuracy. Today, a combination of Textract with AnalyzeDocument (FORMS + QUERIES mode) and a Bedrock model for semantic validation delivers accuracy comparable to a human analyst on clean documents — with 3-8 second latency per document and cost in the order of fractions of a cent per page.

Third, and perhaps most important for regulated financial environments: AWS's shared responsibility model has evolved with sector-specific certifications (PCI DSS Level 1, SOC 2 Type II, ISO 27001, BACEN Resolution 4.893 alignment). This does not eliminate compliance work, but drastically reduces audit scope when you use managed services with documented controls.

Modern KYC Pipeline — Event-Driven Decision Flow

Complete KYC onboarding flow: from customer submission to auditable decision, with AI as co-pilot and immutable trail in S3.

🌐 AWS — Edge & Ingestion

API Gateway REST + WAF + mTLS (edge)
S3 — Raw Docs SSE-KMS, Object Lock (storage)

⚙️ AWS — Orchestration

Step Functions Express Workflow (compute)
Lambda — Extract Textract AnalyzeDoc (compute)
Lambda — Risk Score Bedrock Claude / Nova (ai)
Lambda — Sanctions Ofac + PEP API (compute)

🧠 AWS — AI & Decisão

Amazon Bedrock Claude 3 Sonnet / Nova (ai)
Lambda — Decision Rules Engine + AI (compute)

🗄️ AWS — State & Audit

DynamoDB KYC State Table (data)
DynamoDB Streams → Audit Fanout (messaging)
S3 — Audit Log Object Lock WORM (storage)
CloudWatch SLO Dashboards + Alarms (security)

Flows

client -> apigw: POST /kyc multipart
apigw -> s3raw: encrypted doc upload
apigw -> sfn: start execution
sfn -> lambda_extract: Step 1: extraction
lambda_extract -> bedrock: semantic validation
sfn -> lambda_sanction: Step 2: sanctions
sfn -> lambda_risk: Step 3: risk
lambda_risk -> bedrock: LLM scoring
sfn -> lambda_decision: Step 4: decision
lambda_decision -> dynamo: persist KYC state
dynamo -> dynamo_streams: change stream
dynamo_streams -> s3audit: WORM audit record
dynamo_streams -> cloudwatch: SLO metrics

Auditability as Architecture, Not as Logging

The most common mistake I see in KYC architectures — even in experienced teams — is treating the audit trail as a side effect: you save the decision result somewhere and call it an audit log. Regulators like BACEN, CVM and COAF do not accept this. They want to know why the decision was made, what data was available at the time of the decision and who or what executed each step.

The architecture I propose inverts this logic: the audit trail is a first-class output of the pipeline, not an application log. Each Step Functions execution generates a unique execution ARN that serves as the traceability primary key. Each Lambda participating in the flow persists its input, output and metadata (Bedrock model version, Lambda function version, millisecond-precision timestamp) in DynamoDB with a partition key kyc#customerId#executionId. DynamoDB Streams then propagates each mutation to an S3 bucket with Object Lock in COMPLIANCE mode — meaning not even the root account can delete the record before the configured retention period (minimum 5 years for KYC in Brazil).

A critical detail: when you use Bedrock as a decision co-pilot, the prompt sent to the model, the full response and the model used (including version) must be part of the audit record. This is not optional — it is what allows reconstructing the decision months later during a regulatory audit. Use explicit modelId in Bedrock calls (never latest) and persist usage.inputTokens + usage.outputTokens for cost traceability and reproducibility.

What Changes for Architects with Modern KYC

Orchestration replaces point-to-point integration: Step Functions Express Workflows with per-state retry configuration (maxAttempts: 3, backoffRate: 2.0, intervalSeconds: 1) eliminates retry logic scattered across multiple services and centralizes failure handling — including compensations (partial state rollback).
AI as co-pilot, not arbiter: Bedrock should augment human decision-making in medium-confidence cases, not replace it. Define explicit thresholds: score < 0.3 = auto-approve, 0.3-0.7 = human review queue, > 0.7 = auto-reject. These thresholds are business parameters, not code constants.
Idempotency is a requirement, not an optimization: Each Lambda in the pipeline must be idempotent using the Step Functions executionId as the idempotency key. DynamoDB with ConditionExpression: attribute_not_exists(pk) ensures retries do not create duplicate records or trigger repeated sanctions checks.
Separation of data and control planes: The control plane (Step Functions, Lambda, Bedrock) and the data plane (DynamoDB, S3) must have separate IAM roles with strict least-privilege. The extraction Lambda role must not have write access to the audit bucket — only the Streams fanout Lambda has that permission.
KYC SLOs are business SLOs: Define explicit SLOs: p99 decision latency < 90s for automatic cases, extraction error rate < 0.5%, sanctions false positive rate < 0.1%. These numbers must be in CloudWatch Dashboards with alarms linked to runbooks, not just in architecture presentations.
Contextual encryption, not universal: Use KMS with customer-managed keys (CMK) and key policies that restrict usage by aws:PrincipalTag/Environment and aws:RequestedRegion. PII data in DynamoDB should use attribute-level encryption with AWS Encryption SDK — not just table encryption — so a compromised key does not expose the entire table.

Real Trade-offs: Serverless KYC Is Not a Silver Bullet

Before recommending this architecture to any client, I need to be honest about where it fails or requires extra care.

Cold starts in critical flows: Lambda with Java or .NET runtime can have cold starts of 800ms-2s. For real-time KYC, this is unacceptable if it occurs in the critical path. The solution is not to blindly migrate to Go or Python — it is to use Provisioned Concurrency for functions in the critical path (extraction and decision), with Application Auto Scaling configured to scale based on ProvisionedConcurrencyUtilization. The additional cost is real (~$0.015/GB-hour for provisioned concurrency vs $0.0000166667/GB-second for on-demand) — you need to model the load profile before deciding.

Textract limitations on low-quality documents: Textract AnalyzeDocument has degraded accuracy on scanned documents with resolution < 150 DPI, uneven lighting (document photo taken with a phone in a dark environment) or laminated documents with glare. In production, you need image pre-processing (Lambda with OpenCV or Amazon Rekognition DetectText as fallback) and a minimum confidence threshold per extracted field — if Confidence < 85 on required fields like name or tax ID, the document must be rejected for resubmission, not processed with uncertain data.

Bedrock cost at scale: A Claude 3 Sonnet call for risk scoring with 2000-token context costs approximately $0.003-0.006 per call. At 100,000 onboardings/month, that is $300-600/month in inference alone — manageable. But if you use Bedrock for every validation step without criteria, cost scales quickly. The rule I apply: Bedrock enters only when deterministic rules cannot resolve — not as the first processing line.

Throttling of external sanctions APIs: OFAC, PEP and CSNU lists are queried via third-party APIs with aggressive rate limits (typically 10-50 req/s per account). During onboarding spikes, this creates a bottleneck. The solution is a 24h TTL cache in ElastiCache Redis for already-verified entities, with forced invalidation when lists are updated — reduces external calls by 70-80% in flows with periodic re-verification.

Architectural Positioning: How to Prepare Your Organization

The most critical gap I observe is not technological — it is organizational. Teams operating legacy KYC have compliance analysts who understand the rules but not the technical pipeline, and engineers who understand the pipeline but not the regulatory implications of each design decision. Serverless + AI architecture amplifies this gap if not managed.

The first change I recommend is creating a KYC Design Authority — a small group (3-5 people) with representation from engineering, compliance and product that reviews and approves changes to the decision pipeline. This is not bureaucracy: it is the mechanism that ensures a change in the Bedrock prompt does not inadvertently violate a credit policy or create unintentional discriminatory bias.

Second, invest in decision observability, not just infrastructure observability. CloudWatch Metrics for Lambda latency is necessary but insufficient. You need business metrics: approval rate by customer segment, risk score distribution over time, disagreement rate between AI and human analyst in review cases. These metrics are the signal that the model is drifting or that business rules changed without pipeline updates.

Third, treat Bedrock prompts as infrastructure code: versioned in Git, reviewed via PR, tested with a curated set of test cases (including regulatory edge cases) before any deployment. A prompt that changes credit approval criteria without going through compliance review is equivalent to a code deploy that changes pricing logic without approval — unacceptable in a regulated financial environment.

Finally, plan for multi-region from the start if you operate in markets with data residency requirements. In Brazil, KYC data with PII must reside in sa-east-1 (São Paulo). If you need active-active DR, DynamoDB Global Tables replication works, but you need KMS key policies that restrict decryption to the primary region — the replica can store but must not decrypt without explicit approval.

The Auditability Paradox with Generative AI: LLMs are inherently non-deterministic: the same prompt with temperature > 0 can produce different responses. This creates a regulatory paradox — how do you audit a decision that may not be reproducible? The architectural answer is: you do not audit reproducibility, you audit traceability. Persist the exact prompt, exact response, exact modelId and timestamp. If a regulator questions the decision, you show the reasoning that existed at that moment — not that the system would make the same decision today. For high-consequence cases (credit rejection, fraud suspicion), use temperature: 0 and top_p: 1 to maximize determinism, and document this as an AI governance policy.

Critical Anti-Patterns in Serverless KYC

Monolithic KYC Lambda: A single Lambda function that does extraction, sanctions check, risk scoring and persists the result. Impossible to unit test, impossible to retry granularly, impossible to audit which step failed.
Using latest as Bedrock model version: Guarantees a model update changes production decision behavior without any review. Always pin to a specific version: anthropic.claude-3-sonnet-20240229-v1:0.
Audit log in CloudWatch Logs as primary source: CloudWatch Logs has configurable retention but lacks immutability guarantees equivalent to S3 Object Lock. For regulatory purposes, CloudWatch is operational observability — S3 with Object Lock COMPLIANCE is the official record.
Sharing KMS key across environments: Using the same CMK for dev, staging and production means a developer with dev access can potentially decrypt production data. Separate keys per environment with SCPs that block cross-account usage are mandatory.
Step Functions Standard Workflow for real-time KYC: Standard Workflows have ~1s state transition latency and cost per state transition ($0.025/1000 transitions). For a 15-state pipeline with 100k executions/day, cost is ~$37/day in transitions alone. Express Workflows are appropriate for KYC: execution < 5 min, duration-based cost, and support up to 100,000 executions/second.

Modern KYC Through the AWS Well-Architected Lens

security: Zero Trust in the pipeline: each Lambda assumes a role with least-privilege, no shared role across functions. KMS CMK with key policy restricted by aws:PrincipalTag. PII encrypted at attribute level in DynamoDB with AWS Encryption SDK. WAF with AWS managed rules + custom rules for rate limiting by tax ID at API Gateway.
reliability: Step Functions Express with per-state retry and catch to Dead Letter Queue (SQS FIFO with deduplication). DynamoDB with on-demand capacity to absorb onboarding spikes without throttling. S3 Object Lock for audit durability. Circuit breaker for external sanctions APIs via Lambda with Redis cache.
performance: Provisioned Concurrency for Lambdas in the critical path. Asynchronous Textract with SNS notification for documents > 1 page. Bedrock with streaming response for progressive user feedback. DynamoDB with partition key kyc#customerId + sort key timestamp#executionId for efficient per-customer queries.
cost: Express Workflows vs Standard: 60-80% savings in orchestration cost for short-duration pipelines. Bedrock only for medium-confidence cases (deterministic rules first). ElastiCache Redis for sanctions cache reduces external API cost by 70%. S3 Intelligent-Tiering for audit logs with decreasing access over time.

Curator's Note: What I Would Do Differently the First Time: In KYC projects I have been involved with, the most consistent regret is not having defined AI confidence thresholds as business parameters in Systems Manager Parameter Store from the start — they end up hardcoded in Lambda code and changing a threshold becomes a full CI/CD pipeline deploy, when it should be a compliance-approved configuration change in minutes. The second regret is not including the internal audit team in the design review before the first production deploy — they identify traceability requirements that engineers do not anticipate, such as the need to record which version of the OFAC list was in effect at the time of verification. I learned that in regulated financial environments, the audit architecture must be designed with auditors, not for auditors.

Verdict: Adopt, but with Explicit AI Governance

The serverless KYC architecture with AI assistance is mature enough for production in regulated financial environments — the pieces are available, the use cases are documented and the costs are justifiable. The risk is not in the technology; it is in governance. Teams that adopt Bedrock for KYC decisions without an explicit AI governance framework — documented thresholds, versioned prompts, drift metrics, mandatory human review for high-consequence cases — are creating regulatory liability that will surface in the next audit. My recommendation: start with a pilot in a low-risk product segment, measure the disagreement rate between AI and human analyst for 90 days, calibrate thresholds with real data and only then expand. The rush to automate KYC is understandable — the cost of getting it wrong in a regulated environment is far greater than the cost of doing it slowly and correctly.

References and Further Reading

Originally published at fernando.moretes.com. By Fernando F. Azevedo — Senior Solutions Architect.

One of the most confusing errors you can face while deploying a Node.js or Docker-based application

FOLASAYO SAMUEL OLAYEMI — Tue, 23 Jun 2026 21:50:34 +0000

FOLASAYO SAMUEL OLAYEMI

Jun 23

Fixing “Git Divergent Branches” on a Production Server (Real DevOps Debugging Walkthrough)

#ai #devops #tutorial #programming

2 min read

One Org or Many? The Postmortem Nobody Wants to Write

Fernando Azevedo — Tue, 23 Jun 2026 21:50:06 +0000

Sometime in 2023, a mid-sized financial institution I worked with consolidated all its business units under a single AWS Organization to simplify cost governance. Eighteen months later, a Service Control Policy mistake applied at the root node silenced the production payments pipeline for 47 minutes. The postmortem revealed something most architects know intuitively but rarely document rigorously: Organizations topology is not a FinOps decision — it is a blast radius decision.

What Happened: Context and Organizational Pressure

The pressure came from above. The CFO wanted consolidated cost visibility, the CISO wanted a single policy enforcement point, and the platform team — already stretched thin — wanted to reduce the number of landing zone automation pipelines. The obvious solution seemed to be: one Organization, multiple OUs, hierarchical SCPs. The reasoning was defensible on a whiteboard.

The problem started when the security team needed to block access to certain AWS regions for LGPD/GDPR compliance. The SCP was drafted with an aws:RequestedRegion condition denying all regions outside sa-east-1 and us-east-1. The test was run against a sandbox OU. Approval was granted. The deploy went to the root node — not the production OU, the root.

What nobody had explicitly mapped: the payments account used us-east-2 for a DynamoDB Global Tables endpoint serving as a low-latency read replica. The SCP blocked dynamodb:GetItem and dynamodb:Query calls originating from Lambda functions in that account to that region. The Python SDK (boto3) with exponential retry masked the error for roughly 8 minutes before the circuit breaker in the payment authorization service tripped. The alert arrived via a CloudWatch Alarm with a 5xx error threshold on API Gateway — but the runbook pointed to the authorization service, not the data layer.

Incident Timeline

T+00:00 — SCP applied at root node — Security engineer runs aws organizations attach-policy pointing to Root ID instead of the sandbox OU ID. No scope validation in the IaC pipeline (Terraform) blocked the operation — the policy already existed, only the target changed.
T+00:08 — First silent SDK errors — boto3 with max_attempts=5 and exponential backoff begins absorbing AccessDeniedException from DynamoDB calls in us-east-2. The 15s Lambda timeout has not yet been reached. No active alarm.
T+00:11 — Circuit breaker trips in authorization service — The payment authorization service, deployed on EKS with Resilience4j, opens the circuit after 10 consecutive failures in 30s. Transactions begin returning HTTP 503 to API Gateway.
T+00:14 — CloudWatch Alarm fires — Alarm PaymentAPI-5xxRate > 1% sends notification to SNS → PagerDuty. On-call engineer receives the alert. Initial runbook points to the EKS authorization service.
T+00:22 — Initial misdiagnosis — Engineer checks EKS pods, Resilience4j logs, CPU/memory metrics. All normal. Escalates to tech lead. No CloudTrail correlation yet.
T+00:31 — CloudTrail and Organizations correlation — Tech lead queries CloudTrail Lake with an Athena query filtering errorCode = AccessDeniedException in the last 60 minutes. Identifies pattern in DynamoDB calls from us-east-2. Cross-references Organizations API and finds the root-level attach-policy.
T+00:38 — SCP detached from root node — Senior security engineer runs aws organizations detach-policy. Change propagation takes approximately 4 minutes across all affected accounts.
T+00:47 — Service restored — Circuit breaker closes after successful health checks. Error rate returns to < 0.1%. Incident closed. Total duration: 47 minutes of partial degradation, 36 minutes of complete unavailability for new payments.

Root Cause: Unrestricted Blast Radius by Design: The root cause was not the human error of targeting the root node — human errors are inevitable. The root cause was architectural: a single AWS Organization with no isolation boundary between regulated workloads (payments, PCI-DSS) and operational workloads (security, tooling). Any SCP applied at the Root affects all accounts simultaneously, with no staging, no canary, no automatic rollback. The design turned a routine security policy change into a change with organization-level blast radius. In financial-grade environments, this is unacceptable.

Topology: Single-Org Blast Radius vs. Multi-Org Isolation

The diagram compares the pattern that caused the incident (left) with the remediated architecture (right). Red edges indicate unrestricted SCP propagation; green edges indicate isolation boundary with controlled promotion.

🔴 Padrão Problemático — Single Org / Problematic Pattern — Single Org

Root Management Account (security)
SCP: DenyRegion applied at Root (security)
OU: Security Tooling Accounts (security)
OU: Payments PCI-DSS Accounts (compute)
DynamoDB us-east-2 replica (data)

🟢 Padrão Remediado — Multi-Org com Isolamento / Remediated Pattern — Multi-Org

Org: Operations Security & Tooling (security)
Org: Payments PCI-DSS Boundary (compute)
SCP: DenyRegion scoped to Ops Org (security)
SCP: PCI Controls independent lifecycle (security)
RAM / PrivateLink cross-org sharing (network)
CloudTrail Lake centralized (delegated) (data)

Flows

scp-bad -> root: attached at Root
root -> ou-sec: unrestricted inheritance
root -> ou-pay: full blast radius
ou-pay -> ddb-replica: blocked by SCP
scp-ops -> org-ops: isolated scope
scp-pay -> org-pay: independent lifecycle
org-ops -> ram-share: controlled sharing
org-pay -> ram-share: access via PrivateLink
org-ops -> ct-lake: delegated logs
org-pay -> ct-lake: delegated logs

Why the Single-Org vs. Multi-Org Decision Is Fundamentally an Isolation Decision

There is a dominant narrative that multiple Organizations increase operational complexity — and it is partially true. But it obscures what is actually being traded. In a single Organization, the root node and top-level OUs are attack surfaces for policy changes with instant propagation and no native progressive rollback mechanism. AWS Organizations has no concept of a "canary SCP deploy". When you apply an SCP at the Root, it takes effect immediately across all ~100, ~500, or ~2000 accounts under that root.

In financial-grade environments with multiple regulatory regimes — PCI-DSS for payments, SOC 2 for data operations, BACEN 4.893 for cyber resilience in Brazil — the temptation to use a single Organization with specialized OUs is understandable. The operational reality is that PCI-DSS compliance controls require network and policy isolation that is cleaner with a real Organization boundary, not just an OU boundary.

The Organization boundary provides: (1) SCP policies with completely independent lifecycles; (2) separate Management Account credentials, reducing the blast radius of high-privilege credential compromise; (3) consolidated billing still available via AWS Organizations trusted access and cross-account Cost and Usage Reports; (4) CloudTrail Lake with a delegated administrator that can aggregate logs from multiple Organizations into a single S3 data store with Athena, maintaining centralized visibility without policy coupling.

The real cost of multiple Organizations is duplication of landing zone automation — and that cost is addressable with an Account Vending Machine based on Control Tower customizations and shared IaC pipelines via cross-account CodePipeline.

Technical Remediation: What We Changed After the Incident

The remediation was not simply "move payments to a new Organization". That would have taken weeks and required re-onboarding dozens of accounts. The remediation was layered, with immediate impact first and structural refactoring afterward.

Immediate (week 1): We implemented a guardrail SCP at the root node that explicitly denies organizations:AttachPolicy for any target that is the Root ID (r-xxxx) or any tier-1 OU containing production workloads. The condition uses aws:ResourceTag combined with an Environment=Production tag applied to OUs via Organizations tag policy. This does not resolve the structural problem, but adds a protection layer against the specific error that occurred. The Terraform pipeline was updated with a precondition that validates the target type before any attach-policy.

Medium term (months 2-3): We created a second AWS Organization for PCI-DSS workloads. The Management Account of the new org uses MFA with a hardware token and access restricted to two senior engineers. SCPs in the new org are managed by a separate pipeline with mandatory two-reviewer approval via pull request. CloudTrail Lake was configured with a cross-organization Event Data Store using the organizationEnabled + delegated administrator feature, aggregating events from both orgs into a single Athena repository.

Observability: We added an EventBridge rule in each org's Management Account that captures organizations.amazonaws.com events of type AttachPolicy and DetachPolicy and publishes to an SNS topic with subscriptions to the security Slack channel and PagerDuty. MTTD for this type of change dropped from ~31 minutes (the time it took to correlate during the incident) to under 2 minutes in post-implementation validation tests.

Single-Org vs. Multi-Org: Real Trade-offs

Criterion	Dimension	Single Organization	Multiple Organizations
SCP blast radius	Root affects 100% of accounts instantly	Isolated by Organization boundary; changes are independent	—
Operational complexity	Lower: single landing zone pipeline, single Control Tower	Higher: multiple pipelines, multiple Control Tower enrollments	—
Cost visibility	Native via Consolidated Billing	Requires cross-account CUR + Athena or AWS Cost Explorer linked accounts	—
Regulatory isolation (PCI, SOC2)	Possible via OUs, but policy boundary is logical, not physical	Physical boundary between orgs; auditors accept more readily	—
Management Account compromise	One compromised account = potential access to entire organization	Blast radius limited to specific org; other orgs unaffected	—
SCP propagation latency	Seconds to a few minutes across all accounts	Same behavior within each org; orgs are independent	—

The Real Problem with SCPs: No Staging, No Automatic Rollback

One of the most important findings from the postmortem was that the team had treated SCPs like ordinary infrastructure code — with the same deployment pipeline as a security group or IAM role. This is a mental model error.

SCPs are access control policies with immediate propagation and no native progressive rollback mechanism. There is no aws organizations deploy-policy --canary 10%. When you attach-policy to an OU with 200 accounts, all 200 accounts are affected simultaneously. AWS Organizations has no concept of deployment rings for policies.

The practical implication is that the SCP change process must be treated like a production database-level change — with a maintenance window, dual approval, and a tested rollback plan. The rollback plan for an SCP is simple: detach-policy. But if you do not know what policy was in place before, or if the change was composed of multiple operations, rollback may not be trivial.

What we implemented: an immutable state registry of SCPs per OU/Root, stored in S3 with versioning enabled and Object Lock in COMPLIANCE mode for 90 days. Before any attach-policy, the pipeline saves the current state. Automated rollback is a Lambda that reads the previous state from S3 and executes the inverse operations. Automated rollback time in tests was 45 seconds — compared to the 7 minutes it took to identify and manually execute during the actual incident.

A frequently overlooked detail: SCPs with explicit Deny take precedence over any Allow in identity policies, including IAM Role policies with AdministratorAccess. This means that not even the account root user (unless explicitly excluded via aws:PrincipalType: Root) can execute actions blocked by an SCP. In our incident, this is what made the situation so severe — there was no escape hatch in the payments account.

FinOps in Multi-Org: The Argument That Overcomes Resistance

The most common argument against multiple Organizations is the loss of consolidated cost visibility. That argument was valid in 2018. In 2024, it is a solved problem — with some important caveats.

AWS Cost and Usage Report (CUR 2.0) can be configured for delivery to a centralized S3 bucket in a dedicated billing account, even across multiple Organizations, using a cross-account S3 bucket policy pattern with s3:PutObject allowed for the billingreports.amazonaws.com service principal from multiple Management Account IDs. Athena + AWS Glue Crawler over this data produces a unified cost view that the CFO can consume via QuickSight with row-level security per business unit.

What is not natively solved: Reserved Instances and Savings Plans are not shared across Organizations. This is a real cost. In our analysis, the payments account used approximately $18k/month in Compute Savings Plans that, upon moving to a new Organization, could no longer be shared with tooling accounts in the original org. The solution was to consolidate Savings Plans in the new payments org and use On-Demand for tooling workloads with more variable usage — the cost delta was approximately $1.2k/month, which was accepted as the cost of regulatory isolation.

A pattern I recommend: use AWS Cost Categories with rules based on CostCenter and BusinessUnit tags applied via tag policies in both Organizations. This allows financial reporting to be agnostic to Organizations topology — the CFO sees by cost center, not by org.

AWS Well-Architected: Affected Pillars

security: SCPs must have a change management lifecycle equivalent to production database changes. Use aws:PrincipalType: Root as an explicit escape hatch in critical SCPs. Implement EventBridge + SNS for immediate detection of policy attach/detach in the Management Account. Consider Organization boundary as physical isolation for PCI-DSS and BACEN 4.893 regimes.
reliability: Organizations design must minimize the blast radius of operational changes. Circuit breakers in downstream services (EKS/Resilience4j, Lambda with Dead Letter Queue) are necessary but insufficient — they mask the error without resolving the cause. Add specific health checks for AccessDeniedException with low-latency alarms (< 5 minutes MTTD). Implement automated SCP rollback with immutable state in S3 Object Lock.

Anti-Patterns That Lead to the Incident

Treating SCPs as ordinary infrastructure in the CI/CD pipeline without differentiated approval by target level (Root, production OU, sandbox OU)
Using a single Organization to consolidate cost governance without evaluating the blast radius of security policies on regulated workloads
Configuring 5xx alarms on API Gateway as the only detection signal without specific alarms for AccessDeniedException in CloudTrail
Assuming that circuit breakers in downstream services substitute for architectural isolation — they are complementary, not equivalent
Failing to map cross-region dependencies of accounts before applying SCPs with aws:RequestedRegion conditions
Relying on OUs as regulatory isolation boundaries for PCI-DSS auditors without explicitly documenting that the boundary is logical, not physical

Architect's Note: After this incident, I started recommending a simple rule: if you have workloads with distinct regulatory regimes (PCI-DSS, SOC 2, BACEN) or with availability SLOs above 99.9%, they belong in separate Organizations — not separate OUs. The additional operational cost of multiple Organizations is real, but it is a predictable and manageable engineering cost; the cost of a 47-minute incident on a payments pipeline is not. The hardest lesson was realizing we treated SCPs as code when we should have treated them like production database schema changes: with staging, dual approval, tested rollback, and a maintenance window. That is not bureaucracy — it is reliability engineering applied to the control plane.

Verdict: When to Use One or Multiple Organizations

Use a single AWS Organization when: all your workloads share the same regulatory regime, the same availability SLO, and the platform team has capacity to implement rigorous guardrails in the SCP change pipeline. Use multiple Organizations when: you have distinct regulatory regimes (especially PCI-DSS or BACEN 4.893), SLOs above 99.9% on critical workloads, or when auditors require evidence of physical policy isolation. The cost of non-shared Savings Plans is quantifiable and generally lower than the cost of a single incident caused by unrestricted blast radius. The decision is not about operational simplicity — it is about where you accept that inevitable human error has consequences.

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo — Senior Solutions Architect.

ADR: Adopting Amazon Bedrock AgentCore in Production

Fernando Azevedo — Tue, 23 Jun 2026 21:50:01 +0000

After 16 years building financial platforms on AWS, I've learned that the most dangerous question in architecture isn't 'does this work?' — it's 'who operates this at 2 AM when it breaks?' Bedrock AgentCore is AWS's answer to the problem of operationalizing AI agents beyond the notebook: managed runtime, memory, tool-use, guardrails, and traceability in a single control plane. This ADR documents how I arrived at the decision to adopt it — or not — in a regulated financial environment, and the consequences you need to internalize before doing the same.

Context and Forces

The scenario that motivated this decision is recurring in financial institutions: a product team wants to expose an AI agent to internal analysts — capable of querying market data via API, running risk calculations in Lambda, retrieving context from regulatory documents via RAG, and recording every action in an immutable audit trail. The MVP worked in two sprints with LangChain + Claude via Bedrock. The problem surfaced the following week.

Five forces made the decision urgent: (1) Cross-turn state management — financial agent sessions last minutes, not seconds; reliably maintaining context in stateless Lambda is brittle. (2) Regulatory traceability — every tool call, every model decision, every response must be auditable with timestamp, identity, and full payload, without relying on ad-hoc logging. (3) Guardrails as contract — in finance, the agent cannot leak PII, cannot recommend products without disclaimers, cannot execute irreversible actions without human confirmation. Implementing this manually in every agent is guaranteed technical debt. (4) Unpredictable token cost — without per-session budget control, a faulty agent loop can consume tens of dollars in minutes. (5) Runtime portability — the platform team doesn't want to maintain a custom agent scheduler; they want an SLA contract with AWS.

Options Considered

Option A: Self-hosted LangChain/LangGraph on EKS

Pros

Full control over execution graph and retry logic
Model portability — swap LLM without platform change
Mature ecosystem of community integrations and tools

Cons

Full operational responsibility: scaling, HA, patching, observability
Guardrails and audit trail must be built and maintained by the team
Session memory management requires custom DynamoDB or Redis
High engineering cost to reach parity with managed features

Verdict: Suitable for teams with mature AI platform; high operational risk for smaller teams

Option B: Bedrock Agents (prior generation, without AgentCore)

Pros

AWS-managed, no runtime infrastructure to operate
Native integration with Knowledge Bases and Action Groups

Cons

Limited observability: partial traces, no native span-level detail
No native per-session budget control
Agent loop customization restricted to what AWS exposes

Verdict: Good for simple cases; observability limitations are blockers in finance

Option C: Amazon Bedrock AgentCore

Pros

Managed runtime with native persistent session memory (AgentCore Memory)
Configurable guardrails as declarative policy, not inline code
Native traceability via CloudTrail + X-Ray with tool-call spans
AgentCore Gateway for tool-use with OAuth2/OIDC and per-tool throttling
Configurable per-session token budget control

Cons

Platform lock-in to AWS for the agent runtime
Execution graph customization more restricted than LangGraph
New service: API surface still evolving, conservative quotas
AgentCore Memory and Gateway costs added on top of inference cost

Verdict: Recommended decision for regulated financial environments with a lean platform team

Option D: Step Functions + Lambda as agent orchestrator

Pros

Native audit via Step Functions execution history
Declarative and testable retry, timeout, and error handling
No new service to learn — team already knows the pattern

Cons

Not an agent runtime: each 'turn' requires a new execution or .waitForTaskToken
Session memory and model context must be managed externally
Cold-start and state transition latency can be noticeable in dialogues

Verdict: Excellent for deterministic workflows; inadequate as a conversational agent runtime

The Decision and the Reasoning Behind It

The decision was to adopt Bedrock AgentCore as the primary agent runtime, with Step Functions as the orchestrator for adjacent deterministic workflows (approvals, reconciliations, notifications). This is not an all-or-nothing decision: AgentCore solves the non-deterministic agent loop problem, while Step Functions remains the right choice for the deterministic business process that wraps the agent.

The decisive argument was the AgentCore Gateway with per-tool OAuth2/OIDC support. In a financial environment, every tool-call is an action with identity: who authorized it, what scope, with which token. Implementing this manually in LangChain would mean building and maintaining an authorization proxy — exactly the kind of infrastructure that generates no business value but generates security incidents when neglected. The Gateway delivers this as declarative configuration, with per-tool throttling (e.g., maximum 10 calls/session for the order execution API) and a native circuit breaker.

The second argument was session memory with configurable TTL. AgentCore Memory persists conversation context in a managed store, with per-session configurable TTL and KMS customer-managed key (CMK) encryption. For LGPD/GDPR compliance, this means I can configure a 24h TTL for analyst sessions and guarantee that no session data persists beyond what's necessary — without building a custom expiration pipeline.

The lock-in trade-off was consciously accepted: the tool-use layer (the Lambda functions that execute the actual actions) remains completely portable. If we need to migrate the runtime in the future, the tools keep working.

Financial Agent Architecture with Bedrock AgentCore

Execution flow of a financial analysis agent: from analyst to AgentCore runtime, through guardrails, tool-use via Gateway, session memory, and observability

🔐 AWS — Segurança & Entrada

API Gateway REST + Cognito JWT (edge)
Bedrock Guardrails PII filter + topic deny (security)

🤖 AWS — AgentCore Runtime

AgentCore Runtime Claude 3.5 Sonnet (ai)
AgentCore Memory TTL=24h, KMS CMK (storage)
AgentCore Gateway OAuth2/OIDC, throttle (security)

⚙️ AWS — Ferramentas (Tool-use)

Lambda: Market Data Bloomberg API proxy (compute)
Lambda: Risk Calc VaR engine (compute)
Knowledge Base OpenSearch + S3 (data)

📊 AWS — Observabilidade & Auditoria

X-Ray span por tool-call (external)
CloudTrail API audit log (storage)
CloudWatch SLO dashboards (external)

Flows

analyst -> apigw: HTTPS + JWT
apigw -> guardrails: input validation
guardrails -> agentcore: sanitized prompt
agentcore -> memory: read/write context
agentcore -> gateway: tool invocation
gateway -> lambda_market: OAuth2 token
gateway -> lambda_risk: OAuth2 token
gateway -> kb: RAG query
agentcore -> guardrails: output filter
agentcore -> xray: traces
apigw -> cloudtrail: API events
xray -> cw: SLO metrics

Concrete Configuration: What Actually Matters

Adopting AgentCore without properly configuring operational controls is worse than not adopting it — you gain a false sense of security without active guardrails. Here are the configurations that make a real difference:

Guardrails as first line: Configure contentPolicyConfig with HATE, INSULTS, SEXUAL, VIOLENCE all set to BLOCK, and sensitiveInformationPolicyConfig with PII filters for CREDIT_DEBIT_CARD_NUMBER, AWS_ACCESS_KEY, NAME, and EMAIL in ANONYMIZE mode. In a financial environment, add topicPolicyConfig with explicitly denied topics: "investment advice without disclaimer", "guaranteed returns". This isn't paranoia — it's the minimum to pass a compliance review.

AgentCore Memory with correct partitioning: The memory partition key must be userId + sessionId, never just sessionId. In multi-tenant environments, sessions from different users with the same sessionId collided in testing — a silent bug that leaks context between users. Configure memoryConfiguration.enabledMemoryTypes with SESSION_SUMMARY for long sessions, reducing context token consumption by up to 40% in sessions exceeding 20 turns.

Gateway with per-tool throttling: Define separate rateLimit for each Action Group. The order execution API should have maxRequestsPerSession: 5 and requireConfirmation: ENABLED. The market data query API can have maxRequestsPerSession: 50. Without this granularity, a faulty agent loop can execute dozens of orders before being detected — a scenario I've seen happen in production with frameworks lacking tool-use controls.

Per-session token budget: Configure sessionConfiguration.maxTokens with a conservative initial value — I recommend 50,000 tokens for typical analysis sessions. Monitor the p95 token consumption per session in CloudWatch and adjust. An agent entering a reasoning loop can consume 200k+ tokens in a single session without this control.

Observability: What to Measure and How

AI agents have a different observability profile from traditional APIs. p99 latency is less useful than turns-per-session distribution and tool-call failure rate per tool. Here is the observability model I implemented:

Agent business metrics (via CloudWatch custom metrics with namespace FinancialAgent):

TurnsPerSession — histogram; alert if p95 > 15 turns (indicates loop or poorly calibrated prompt)
TokensPerSession — histogram; alert if p95 > 40k tokens
ToolCallFailureRate per ToolName — counter; SLO of < 1% failure for critical tools
GuardrailInterventionRate — counter; spike indicates jailbreak attempt or prompt injection

Traces with X-Ray: AgentCore emits spans for each tool invocation with attributes bedrock.agent.toolName, bedrock.agent.sessionId, and bedrock.agent.turnCount. Configure a trace group with filter annotation.bedrock.agent.toolName = "ExecuteOrder" and alert on latency > 2s — order execution above that indicates a downstream API issue.

CloudTrail for regulatory audit: Each InvokeAgent API call is recorded with the caller ARN, sessionId, and inputText (truncated). For compliance, configure an S3 bucket with Object Lock in COMPLIANCE mode and 7-year retention for AgentCore CloudTrail logs. This is the minimum to meet Banco Central do Brasil and SEC audit requirements.

Cost anomaly alarm: Configure an AWS Budget with an alert at 80% of the monthly Bedrock budget, with an SNS action. Add a second CloudWatch alarm on bedrock:InvokeModel with model-id=anthropic.claude-3-5-sonnet and a threshold of 1,000 invocations/hour — above that, something is wrong.

Consequences and Risks You Need to Accept: Runtime lock-in is real: If AWS deprecates or significantly changes the AgentCore API, migration requires rewriting the orchestration logic — not just the tools. Mitigate by keeping tools (Lambda) completely runtime-agnostic and documenting the interface contract in a separate ADR.

Conservative quotas on a new service: AgentCore has concurrent agent sessions quotas that, at launch, were significantly lower than traditional Bedrock Agents quotas. Request quota increases before go-live, not after. A peak event without adequate quota results in ThrottlingException that the end client sees as a timeout.

Guardrails have latency: Each pass through Guardrails adds 100-300ms of latency. In an agent with 10 turns, that's up to 3 additional seconds of accumulated latency. For use cases where latency is critical, consider disabling output guardrails on internal tools (not exposed to the end user) and applying them only on the final output.

Memory is not free: AgentCore Memory charges for storage and per read/write operation. In long sessions with SESSION_SUMMARY active, memory cost can exceed inference cost for short sessions. Monitor MemoryReadLatency and MemoryWriteLatency — above 200ms indicates pressure on the managed store.

Human-in-the-loop is not automatic: requireConfirmation: ENABLED on the Gateway pauses execution and waits for confirmation via callback. If the client doesn't respond within confirmationTimeout (default: 300s), the session expires. Design the UX to make this clear to the user — timeout-expired sessions are the leading cause of complaints in financial agents.

Real Reference Numbers

~40% — Token reduction with SESSION_SUMMARY. In sessions with more than 20 turns, SESSION_SUMMARY reduces context sent to the model by ~40% vs. full history
100-300ms — Latency added by Guardrails per turn. Each Guardrail evaluation (input + output) adds 100-300ms; across 10 turns, up to 3s accumulated
7 anos — Minimum CloudTrail retention for financial compliance. S3 Object Lock in COMPLIANCE mode with 7 years meets Banco Central do Brasil and SEC requirements for agent audit trails

Well-Architected Assessment

security: Declarative guardrails with PII filter and denied topics; AgentCore Gateway with per-tool OAuth2/OIDC; KMS CMK for session memory; CloudTrail with S3 Object Lock for immutable audit. IAM with bedrock:AgentArnLike condition to restrict which agents can invoke which tools.
reliability: Automatic retry with jitter in the Bedrock SDK (max_attempts=3, mode=adaptive); native circuit breaker in AgentCore Gateway per tool; configurable session timeout prevents zombie sessions; concurrent session quotas must be requested before go-live.
performance: SESSION_SUMMARY reduces context tokens by ~40% for long sessions; disabling output guardrails on internal tools reduces accumulated latency; Knowledge Base with OpenSearch k-NN with HNSW and ef_search=512 for low-latency RAG.
cost: AWS Budget with alert at 80% of monthly limit; CloudWatch alarm on invocations/hour per model-id; SESSION_SUMMARY reduces inference cost in long sessions; monitor AgentCore Memory cost separately from inference cost.

What the AWS Blog Doesn't Tell You

AWS service launch blogs are excellent at showing the happy path. What they rarely cover are the edge cases you only discover in production. Here are the three that cost me the most time:

Tool-call idempotency is not guaranteed by the runtime. If AgentCore attempts to invoke a tool and receives a timeout, it may retry — and your Lambda may be invoked twice for the same action. For idempotent tools (queries), this is harmless. For tools with side effects (order execution, email sending), you need to implement idempotency in the Lambda using an idempotencyToken derived from sessionId + turnId + toolName. Without this, order duplication is a matter of when, not if.

The model can ignore requireConfirmation in certain prompt formulations. I tested this: if the system prompt instructs the agent to "be proactive and execute actions without asking for unnecessary confirmation," the model may rationalize that a specific action doesn't need confirmation even with the flag active. The correct defense is dual: the flag on the Gateway and an explicit instruction in the system prompt about when confirmation is mandatory. Never rely on a single layer.

AgentCore doesn't have native multi-agent support yet. If your architecture requires a supervisor agent delegating to specialized agents (multi-agent orchestration pattern), you'll need to implement the delegation logic manually — typically with an agent that calls other agents via tool-use, where each "tool" is actually an invocation of another AgentCore. It works, but cross-session traceability requires manual sessionId correlation via X-Ray.

Anti-Patterns I've Seen in Architecture Reviews

Using AgentCore without configuring Guardrails because "it's an internal environment" — insiders are the primary source of compliance incidents in finance
Storing full session history in memory without SESSION_SUMMARY — token cost grows linearly with number of turns
Implementing critical business logic inside the agent system prompt instead of in testable tools — prompts don't have unit tests
Not requesting concurrent session quota increase before go-live — ThrottlingException during peak usage is predictable and preventable
Assuming AgentCore Gateway replaces a business authorization layer — the Gateway controls access to the tool, not the authorization logic inside the tool
Not implementing idempotency in tool Lambdas with side effects — runtime retries can duplicate irreversible actions

Curator's Note: In practice, what convinced me to adopt AgentCore was not any individual feature — it was the fact that the AgentCore Gateway with per-tool OAuth2/OIDC solves the tool-call identity problem I was about to build manually, which would have taken two sprints and generated permanent technical debt. The hard-won lesson behind this: in financial environments, the cost of building custom security controls is not the initial development cost — it's the cost of maintaining, auditing, and fixing those controls over years. When a managed service delivers the control as declarative configuration, the decision to adopt it is rarely about feature parity; it's about where you want to allocate your team's engineering attention. My recommendation: adopt AgentCore for new production agents, keep tools portable, and invest the saved time in observability and adversarial prompting tests.

Verdict: Adopt with Explicit Controls

Bedrock AgentCore is the right choice for financial teams that need to put AI agents into production without building and maintaining a custom orchestration runtime. The decision is not binary — it's about recognizing that AgentCore's value lies in the operational controls (Gateway, Guardrails, Memory with CMK), not just the execution runtime. The condition for adoption is clear: configure Guardrails before any testing with real data, implement idempotency in all tools with side effects, request concurrent session quota increases before go-live, and monitor TurnsPerSession and TokensPerSession as first-class SLO metrics. Lock-in is real but manageable if tools are kept portable. For teams that lack the capacity to build and operate a custom agent runtime — which is most teams — AgentCore is the correct architectural decision in 2025.

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo — Senior Solutions Architect.

CloudWatch to OTel: Tearing Down the Observability Bridge Pattern

Fernando Azevedo — Tue, 23 Jun 2026 21:49:18 +0000

In financial environments where platform teams need to consolidate signals from dozens of AWS workloads into a unified observability backend — whether Datadog, Grafana Cloud, Honeycomb, or an internal stack — the CloudWatch → OpenTelemetry bridge pattern appears as the obvious solution. But 'obvious' and 'correct' rarely coincide in architecture. This pattern has a specific anatomy, a narrow validity envelope, and failure modes that only surface in production under real load. I'm going to dissect every layer.

The Real Problem: Observability Fragmentation in AWS-Native Environments

Every organization that grows beyond two or three engineering teams faces the same tension: AWS services emit metrics natively to CloudWatch — Lambda, RDS, EKS, API Gateway, MSK — but the corporate observability backend speaks OTLP. The result is a split world: SREs need to open two consoles to correlate an incident, alerts live in different namespaces, and business dashboards become impossible to build without manual ETL.

The bridge pattern exists to solve exactly this. The core idea is simple: a Lambda function (or an OTel collector running on ECS/EKS) subscribes to CloudWatch metric streams via CloudWatch Metric Streams or polls the GetMetricData API, transforms the payload to OTLP format, and forwards it to a collector endpoint. In theory, you get a single observability control plane. In practice, the complexity hides in the details.

What makes this problem especially treacherous in financial environments is the combination of three factors: (1) metric volume — a mid-size AWS account with EKS, Multi-AZ RDS, and Lambda can easily generate 50,000+ metric series per minute; (2) business latency — anomaly detection SLOs require freshness of 60 seconds or less; (3) API cost — each GetMetricData call has a direct cost and a quota of 50 metrics per request, meaning naive polling at scale breaks both the budget and AWS rate limits.

Anatomy of the CloudWatch → OTel Bridge Pattern

Complete flow from metric emission by AWS services to the external observability backend, through the two ingestion paths (Metric Streams and polling) and the security and cost guardrails.

🟧 AWS — Metric Sources

Lambda Invocations/Errors/Duration (compute)
EKS / EC2 CPU, Memory, Network (compute)
RDS / Aurora DBConnections, Latency (data)
API Gateway 4xx/5xx, Latency (network)

🟦 AWS — Ingestion Layer

CloudWatch Namespaces (storage)
CW Metric Streams ~2-3s latency, Firehose (messaging)
Kinesis Firehose JSON/OTel0.7 format (messaging)

🟨 AWS — Transform & Forward

Bridge Lambda OTLP transform + retry (compute)
SQS DLQ failed batches (messaging)
KMS CMK payload encryption (security)

🔵 External — Observability Backend

OTel Collector OTLP/gRPC :4317 (external)
Datadog / Grafana / Honeycomb (external)

Flows

lambda-src -> cw-ns: emits metrics
eks-src -> cw-ns: emits metrics
rds-src -> cw-ns: emits metrics
apigw-src -> cw-ns: emits metrics
cw-ns -> metric-stream: continuous stream
metric-stream -> firehose: OpenTelemetry 0.7
firehose -> bridge-lambda: batch trigger
bridge-lambda -> kms-key: decrypt/encrypt
bridge-lambda -> dlq: fail after 3 retries
bridge-lambda -> otel-collector: OTLP/gRPC export
otel-collector -> obs-backend: processed pipeline

Pattern Anatomy: Two Paths, One Fundamental Trade-off

The pattern has two ingestion flavors, and the choice between them defines everything that follows.

CloudWatch Metric Streams + Kinesis Firehose is the low-latency path. Streams deliver data in OpenTelemetry 0.7 (protobuf) or JSON format with 2–3 second latency. The cost is predictable: $0.003 per 1,000 metric updates. For an account with 100k active series, that's roughly $300/month before Firehose costs. The critical architectural advantage is that you're not polling — data flows, and the Lambda at the Firehose destination receives batches, not individual calls.

Polling via GetMetricData is the fine-grained control path. You choose exactly which metrics, at what resolution, and what lookback period. But the API has a quota of 50 metrics per request and 500 requests per second per account (soft limit). In a large account, hitting that limit is a matter of minutes if the polling code doesn't implement exponential backoff with jitter and doesn't batch correctly. I've seen financial production systems break critical alerts because the poller entered throttling at 09:00 on a Monday morning — exactly when the market opens and transaction volume explodes.

The transformation Lambda needs three non-negotiable capabilities: (1) idempotency — Firehose can re-deliver batches on failure; the Lambda must detect duplicates via payload hash; (2) circuit breaker for the external OTel endpoint — if the collector is down, the Lambda cannot loop consuming concurrency; (3) DLQ with alarm — batches that fail after 3 attempts go to SQS DLQ and a CloudWatch alarm must fire within 5 minutes.

When This Pattern Makes Sense

You have an external observability backend (Datadog, Grafana Cloud, Honeycomb) that speaks OTLP and needs metrics from managed AWS services that have no installable agent.
Metric volume justifies Streams (>10k active series) — below that, the fixed stream cost rarely pays off versus selective polling.
Your metric freshness SLO is ≤60 seconds — Metric Streams delivers in 2–3s; 1-minute polling has effective latency of up to 90s.
The platform team wants to decouple the observability backend from AWS without rewriting application instrumentation — the bridge is transparent to product teams.
You need trace and metric correlation in a single backend: OTel allows enriching metrics with resource attributes (account ID, cluster, service) that native CloudWatch doesn't propagate.

Security and Guardrails: What the Tutorials Don't Tell You

In financial environments, the observability bridge is an underestimated data exfiltration vector. Business metrics — transaction volume, payment latency, authentication error rates — are sensitive information. A misconfigured OTel endpoint can leak this data out of the AWS account without any alarm.

The three control layers I implement in every deployment of this pattern:

IAM with resource conditions: The Lambda role must have cloudwatch:GetMetricData and cloudwatch:ListMetrics permission with condition aws:ResourceTag/Environment: production — never a wildcard. For Metric Streams, the Firehose role needs cloudwatch:PutMetricStream and firehose:PutRecord, but the destination Lambda role must be separate and have only s3:GetObject on the buffer bucket (if using S3 as fallback) and invocation permission.

KMS CMK for payload in transit: Firehose must be configured with ServerSideEncryption using a CMK managed by the security team. The transformation Lambda must decrypt with the same key. This ensures that even unauthorized access to the Firehose stream doesn't expose readable data.

VPC Endpoint for the OTel collector: If the collector runs on ECS inside the VPC, the Lambda must be in the same VPC and use private DNS. If the collector is external (SaaS), traffic must exit through a NAT Gateway with a fixed IP whitelisted in the vendor's firewall — never through an Internet Gateway without egress control. Add a WAF rule on API Gateway if the external collector exposes an HTTP endpoint.

A detail that burned one of my clients: the bridge Lambda running with a 15-minute timeout (maximum) and no reserved concurrency can consume the entire account concurrency pool during an ingestion spike, bringing down critical business functions. Reserve explicit concurrency — typically 10–20 instances are sufficient for a Firehose with a 5MB batch.

Anti-Patterns: When This Bridge Will Explode in Production

Naive polling without backoff: Calling GetMetricData in a loop with a fixed 60s interval for hundreds of namespaces. In accounts with many services, you hit the rate limit in minutes and lose observability data exactly when you need it most — during incidents.
Lambda without DLQ and without error alarm: Silent transformation or export failures mean metrics disappear without any signal. In financial environments, this can mask service degradation for hours.
Forwarding all metrics from all namespaces: CloudWatch has over 200 namespaces. Forwarding everything to the external backend multiplies SaaS ingestion cost by 5–10x without proportional value. Filter by namespace and dimension in the Metric Stream itself.
No idempotency in the transformation Lambda: Firehose guarantees at-least-once delivery. Without payload hash deduplication, you inject duplicate series into the OTel backend, corrupting aggregations and sum/count-based alerts.
Using this pattern for traces and logs: The CloudWatch → OTel bridge is designed for metrics. Trying to forward CloudWatch Logs Insights or X-Ray traces through the same channel creates a fragile multi-purpose system. Use the OTel Collector directly in applications for traces and logs.
No reserved concurrency on the Lambda: During ingestion spikes (market open, nightly batch), the bridge Lambda can exhaust the account concurrency pool and throttle critical business functions.

Reference Design: What Actually Works in Financial Production

After implementing and debugging this pattern in three different financial environments, the design that works has these concrete characteristics:

Ingestion via Metric Streams with namespace filter: Configure the stream to include only relevant namespaces — AWS/Lambda, AWS/EKS, AWS/RDS, AWS/ApiGateway, AWS/MSK — and explicitly exclude high-cost, low-value namespaces like AWS/Billing and AWS/CloudFront (unless you monitor CDN). The format should be opentelemetry0.7 (protobuf), not JSON, to reduce payload size by ~40%.

Firehose with 60s/5MB buffer and S3 as fallback: Configure BufferingHints with IntervalInSeconds: 60 and SizeInMBs: 5. The fallback S3 bucket should have a lifecycle policy to expire objects after 7 days — it exists only for manual replay in case of external collector failure, not as permanent storage.

Transformation Lambda in Python/Rust with OTel SDK: Use opentelemetry-sdk to build the OTLP payload. Add fixed resource attributes at transformation time: cloud.account.id, cloud.region, deployment.environment. These attributes enable cross-account correlation in the backend. Timeout should be 3 minutes (not 15) — if transforming a batch takes more than 3 minutes, there's a volume problem that needs to be solved in the stream filter, not in the timeout.

Observability of the bridge itself: Instrument the Lambda with custom metrics: bridge.metrics.transformed.count, bridge.export.latency.p99, bridge.export.errors.count. Create a CloudWatch dashboard for the bridge itself — it's ironic but necessary: you need to observe the observability system. Define an SLO of bridge.export.errors.count < 0.1% with a 5-minute burn rate alarm.

Real Production Numbers

2–3s — Metric Streams Latency. From AWS service emission to Firehose. 1-minute interval polling has effective latency of 60–90s.
~$0.30 — Cost per 100k series/month. $0.003/1k updates × ~100k series × ~1 update/min × 43,200 min/month ≈ $13k/month — filter aggressively.
50 — Metrics per request in GetMetricData API. Hard quota. With 500 req/s soft limit, polling 25k metrics requires 500 requests — hits the limit in 1 second.

Filter at the Stream, Not in Lambda: CloudWatch Metric Stream supports filters by namespace and by dimension. Use them. Every metric you don't forward saves Firehose cost, Lambda cost, and ingestion cost in the external backend. The transformation Lambda should be dumb and fast — transform format, add resource attributes, and export. Filtering logic in Lambda is an anti-pattern: you've already paid for the data when it arrives at Firehose.

Well-Architected Lenses for This Pattern

security: IAM with resource tag conditions, KMS CMK on Firehose, Lambda in private VPC, controlled egress via NAT with fixed IP. The Lambda role must never have cloudwatch:* — minimum scope per namespace.
reliability: DLQ at Firehose destination, circuit breaker for the external OTel endpoint, burn rate alarm on the bridge SLO. External collector failure testing must be part of the DR runbook.

My Curation Note: I implemented this pattern for the first time in 2022 in a payments environment and learned the hard way that the observability bridge needs its own observability — it sounds obvious, but in practice it's always the last item on the list. What concerns me most about this pattern is not the technical complexity, but the false sense of security it creates: teams assume that because metrics are 'flowing,' observability is working. It's not — not until you have freshness SLOs, error alarms on the bridge, and a runbook for external collector failure. In financial environments, I always require the team to demonstrate what happens when the external OTel endpoint is unavailable for 10 minutes before going to production. The answer to that question reveals whether the design has real guardrails or just good intentions.

Verdict: Use with Surgery, Not Enthusiasm

The CloudWatch → OTel Bridge pattern via Lambda and Metric Streams is technically sound and solves a real observability fragmentation problem. But it has a non-trivial operational cost and a narrow validity envelope. Use it when: (a) you have >10k active metric series from managed AWS services that need to be in an external OTLP backend; (b) your freshness SLO is ≤60 seconds; (c) the platform team has the capacity to operate and observe the bridge itself. Don't use it when: you're trying to consolidate traces and logs through the same channel; when volume doesn't justify the fixed Metric Stream cost; or when there isn't operational maturity to maintain a system that observes other systems. The highest risk is not technical — it's organizational: teams that deploy this pattern and then don't monitor it create an observability blind spot disguised as an observability solution.

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo — Senior Solutions Architect.

Agentic RAG with OpenSearch Serverless: Anatomy of a Pattern

Fernando Azevedo — Tue, 23 Jun 2026 21:48:41 +0000

When AWS announces a new generation of OpenSearch Serverless aimed at agentic AI, the technical signal that matters is not in the press release — it's in the design implications most architects will discover too late: cold starts that destroy latency SLOs, OCU costs that explode with batch embedding workloads, and the illusion that 'serverless' eliminates the need for partition modeling and concurrency control. I have 16 years building financial systems on AWS infrastructure and I know what happens when a promising architectural pattern meets the reality of a regulated environment. This article tears down the agentic RAG pattern from the ground up: the problem it solves, its internal anatomy, the numbers that matter, and — most importantly — when you should not use it.

The Real Problem: Why Classical RAG Breaks in Agentic Workflows

Classical RAG is a two-phase pattern: you retrieve k relevant documents via vector search and inject them into an LLM's context for generation. It works well for static Q&A over a stable knowledge base. The problem surfaces when you add agency — that is, when the LLM iteratively decides which tools to call, which queries to reformulate, and how to compose the final answer from multiple heterogeneous sources.

In a real agentic workflow, the retrieval chain is not linear. A financial agent answering 'what is the consolidated credit risk for this counterparty across all exposures in the last 90 days?' may issue 4 to 8 retrieval calls in sequence or in parallel, each with a different query vector, crossing indices of contracts, market news, rating history, and regulatory data. The vector index becomes a hot-path component with P99 latency requirements below 200ms per call and throughput of tens of queries per second per agent session.

Classical OpenSearch Serverless had a well-documented problem here: the OCU (OpenSearch Compute Units) model scales per collection, not per query, and the cold start of an idle collection can reach 2-3 minutes — unacceptable for any interactive agent. The new generation promises more granular scaling and lower provisioning latency, but the architect still needs to understand the capacity model to avoid a surprise at month-end billing.

Anatomy of the Agentic RAG Pattern with OpenSearch Serverless

Full flow: document ingestion, vector indexing, agentic retrieval-generation cycle, with control plane and observability

📥 Ingestão & Embedding

S3 Documentos brutos (storage)
AWS Glue Chunking + ETL (compute)
Bedrock Titan Embedding v2 (ai)

🔍 OpenSearch Serverless

OpenSearch Serverless Vector Collection (data)
k-NN Index HNSW / FAISS (data)
Reranker Cross-encoder (ai)

🤖 Camada Agêntica

Bedrock Agent Orchestrator (ai)
Tool Registry Lambda Actions (compute)
Session Memory DynamoDB (storage)

🔐 Segurança & Controle

IAM + ABAC Data Access Policy (security)
KMS CMK Encryption at rest (security)

📊 Observabilidade

CloudWatch SLO / Alarms (compute)
OpenTelemetry Trace propagation (compute)

Flows

s3raw -> glue: S3 event trigger
glue -> embed: text chunks
embed -> oss: vectors + metadata
oss -> knn: HNSW index
agent -> tools: tool call
tools -> knn: vector query
knn -> rerank: top-k candidates
rerank -> agent: reranked context
agent -> mem: session / history
iam -> oss: data access policy
kms -> oss: CMK encryption
agent -> otel: trace span
otel -> cw: metrics / logs

Technical Anatomy: Each Component and Its Critical Configurations

Vector Indexing in OpenSearch Serverless

The heart of the pattern is the k-NN index. In OpenSearch Serverless, you choose between HNSW (Hierarchical Navigable Small World) and IVF (Inverted File Index). For agentic workloads with high read throughput and low latency, HNSW with ef_construction=512 and m=16 offers the best recall-latency trade-off — expect P50 of 15-30ms and P99 of 80-150ms for collections up to 10M vectors of 1536 dimensions (Titan Embedding v2). Beyond that, the memory cost per OCU starts pressuring the budget: each OCU supports approximately 8GB of index in memory.

Chunking and Embedding Strategy

Retrieval quality starts in the ingestion pipeline. For financial documents — contracts, prospectuses, regulatory reports — I use hierarchical chunking: 512-token chunks with 64-token overlap, preserving structural metadata (section, page, effective date) as filter fields in the index. This enables hybrid search: vector search + metadata filter, reducing the search space and improving precision without increasing k.

Reranking: The Most Underestimated Component

Retrieving the top-20 candidates via k-NN and reranking them with a cross-encoder (Cohere Rerank or Bedrock Rerank) before injecting into the agent's context is the difference between a system that hallucinates and one that cites correct sources. Reranking cost is low (< $0.002 per 1000 documents on Cohere) and the precision gain in specialized domains is consistently 15-25% in MRR@10 on benchmarks I've run internally.

Numbers That Matter for Sizing

~8 GB — Index per OCU. HNSW index capacity in memory per OpenSearch Compute Unit; plan for 70% maximum utilization
< 150ms — Vector search P99. Realistic P99 latency for collections up to 10M vectors of 1536 dims with HNSW ef_search=256 on warm OCUs
2–3 min — Collection cold start. Provisioning time for an idle collection; the new generation reduces this but does not eliminate it — keep collections warm via heartbeat queries

When to Use This Pattern: The Honest Criteria

The agentic RAG pattern with OpenSearch Serverless fits well under a specific set of conditions. First, workloads with high traffic variability and low predictability: if you have daytime usage peaks with overnight valleys, the serverless model amortizes the cost of idle capacity you would pay for with a provisioned OpenSearch cluster. For a financial analyst support system peaking from 9am to 5pm with residual overnight traffic, savings can be 40-60% compared to a 3-node m6g.2xlarge cluster.

Second, knowledge bases that grow non-uniformly: regulatory documents, meeting minutes, risk reports — corpora that grow in sprints (end of quarter, market events) and stay stable for weeks. The async ingestion model of OpenSearch Serverless, with Glue or Lambda processing SQS queues of new documents, adapts naturally to that rhythm.

Third, multi-tenancy with data isolation: OpenSearch Serverless supports data access policies per collection with IAM conditions based on tags (aws:ResourceTag/tenant). For a financial SaaS platform with multiple clients, you can have one collection per tenant or use index namespaces with granular access policies — without operating separate clusters.

The most important negative criterion: do not use this pattern if you have end-to-end latency SLOs below 500ms for the complete agent response. An agentic cycle with 3-4 rounds of retrieval, reranking, and LLM generation will rarely stay below 3-8 seconds at P50. That is acceptable for async analysis, unacceptable for trading or real-time credit decisions.

Anti-Patterns: What Will Break in Production

Single index for all tenants without metadata filter: Placing documents from multiple clients in the same index without a tenant_id field as a mandatory filter on all queries is a data isolation failure. In regulated financial environments, this is an audit finding, not just a product bug.
Embeddings generated at query time without cache: Calling Bedrock Titan to generate the user query embedding on every request without caching embeddings of frequent queries adds unnecessary 50-100ms and API cost. Use ElastiCache with a 5-minute TTL for recurring queries in agent sessions.
k too high without reranking: Retrieving top-50 or top-100 documents and injecting everything into the LLM context without reranking fills the context window with noise, increases token cost, and degrades response quality. The correct pattern is k=20 with reranking down to top-5.
Ignoring the OCU cost model for batch ingestion workloads: Indexing OCUs and search OCUs are billed separately. A batch ingestion job processing 1M documents can provision 10+ indexing OCUs for hours and generate an unexpected bill. Always throttle the ingestion pipeline and monitor IndexingOCUs in CloudWatch.
No idempotency in the ingestion pipeline: Reprocessing documents without content hash verification creates duplicates in the vector index, increases storage cost, and degrades search precision with redundant chunks. Use a SHA-256 hash of the content as document_id and perform upsert, not insert.
Relying on SDK auto-retry without agent-level idempotency: Agentic workflows with Step Functions or Bedrock Agent that retry retrieval tool calls without idempotency guarantees can emit duplicate queries to the index and accumulate inconsistent context in the session. Each tool call must have a traceable call_id.

Security and Governance in Regulated Financial Environments

In financial systems, the vector index is not just a performance component — it is a repository of potentially sensitive data. Embeddings of confidential documents can, in theory, be partially reversed with embedding inversion attacks. This is not science fiction: recent research has demonstrated partial text reconstruction from vectors of popular models.

Mandatory controls I implement in production:

KMS CMK with annual rotation: Every OpenSearch Serverless collection must use a customer-managed CMK (aws/opensearchserverless is not sufficient for regulated environments). Configure the kms:ViaService condition in the key policy to restrict usage exclusively to the service.
Data Access Policies with least privilege: Separate IAM roles for ingestion (aoss:CreateIndex, aoss:WriteDocument) and for search (aoss:ReadDocument). Never give the agent write permission on the index.
VPC Endpoint for OpenSearch Serverless: In financial environments, all traffic to the index must pass through vpce-opensearchserverless with an endpoint policy that rejects requests from outside the VPC. This eliminates the exfiltration vector via the public internet.
Query auditing via CloudTrail: Enable aoss:APICall in CloudTrail to log all queries to the index. In LGPD/GDPR environments, this is necessary to demonstrate that personal data is not being accessed outside the authorized context.
Data classification in chunk metadata: Add data_classification (PUBLIC, INTERNAL, CONFIDENTIAL, RESTRICTED) and retention_date fields to each indexed document. Use these fields as mandatory filters in data access policies to ensure the agent never retrieves documents above its authorization level.

Observability: What to Monitor and Why

An agentic RAG system without adequate observability is a black box that fails silently. Quality degradation — less precise responses, increasing hallucinations — does not show up in infrastructure metrics. You need a three-layer observability strategy.

Layer 1 — Infrastructure (CloudWatch):

SearchOCUs and IndexingOCUs: alerts when > 80% of configured maximum capacity
SearchLatency P99: SLO of 150ms; alarm at 120ms to allow reaction time
IndexingRate (documents/second): sudden drop indicates a problem in the ingestion pipeline
SearchRequestRate vs SearchErrorRate: error rate > 0.1% triggers investigation

Layer 2 — Application (OpenTelemetry + CloudWatch Logs Insights):
Instrument each agent cycle with a trace span that includes: number of retrieval rounds, effective k per round, reranker latency, context tokens injected into the LLM, and response confidence score. This allows correlating quality degradation with traffic or data changes.

Layer 3 — Retrieval Quality (offline):
I implement an async evaluation pipeline with RAGAS (Retrieval Augmented Generation Assessment) that runs daily over a 5% sample of production queries. The context_precision, context_recall, and faithfulness metrics are published to CloudWatch as custom metrics and integrated into the product's SLO dashboard. A 10% drop in faithfulness over 7 days is an indicator of knowledge base drift that requires reindexing.

Keep Collections Warm with Heartbeat Queries: The cold start of idle OpenSearch Serverless collections is the biggest operational risk of this pattern. Configure an EventBridge Scheduler to fire a lightweight heartbeat query (GET /_cat/indices) every 10 minutes on critical collections. The cost is negligible (< $0.50/month in OCUs) and eliminates the risk of a 2-3 minute cold start on the first access of the day — which in financial systems may be exactly when an analyst needs an urgent answer before market open.

OpenSearch Serverless vs Alternatives for Agentic RAG

Criterion	Criterion	OpenSearch Serverless	Provisioned OpenSearch	pgvector (Aurora)
P99 Latency (10M vectors)	80-150ms (warm)	20-60ms	200-500ms	—
Cold Start	2-3 min (mitigable)	None	None	—
Base monthly cost (idle)	~$700 (2 OCU minimum)	~$400 (3x r6g.large)	~$200 (Aurora Serverless v2)	—
Auto-scaling	Yes, per collection	Manual / UltraWarm	Yes (ACU)	—
Native multi-tenancy	Data Access Policies via IAM	Index-level RBAC	Row-level security (RLS)	—
Hybrid search (vector + BM25)	Yes, native	Yes, native	Not native (workaround)	—

My Curation Note: After implementing variations of this pattern in three distinct financial environments — asset manager, digital bank, and insurer — the hardest lesson I learned is that RAG quality is determined 70% by the ingestion pipeline and 30% by index configuration. Architects who spend weeks tuning HNSW parameters while having poorly structured chunks and missing metadata are optimizing the wrong thing. My practical recommendation: before any index tuning, invest in a golden dataset of 200-300 domain-specific (query, relevant document) pairs and use it to measure context_recall — if it's below 0.75, the problem is in chunking or embedding, not in k-NN. OpenSearch Serverless is a good choice for this pattern when the minimum cost of ~$700/month is justifiable and traffic is genuinely variable; outside of that, a small provisioned cluster with UltraWarm delivers better TCO.

Verdict: Use with Criteria, Configure with Rigor

The agentic RAG pattern with Amazon OpenSearch Serverless is technically mature and operationally viable for financial-grade systems — as long as you accept its constraints clearly. The minimum cost of ~$700/month in OCUs is non-negotiable and cold start is still a real operational risk that requires active mitigation. On the other hand, the IAM-based data access policy model is genuinely superior for regulated multi-tenancy, native hybrid search support (vector + BM25) is a concrete advantage over pgvector, and the absence of cluster management frees engineering capacity for what truly matters: ingestion pipeline quality and continuous retrieval evaluation. My recommendation: adopt this pattern if you have agentic workloads with variable traffic, a specialized domain knowledge base, and multi-tenancy requirements with data isolation. Avoid it if you need sub-100ms P99 latency, have predictable and constant traffic, or if the base cost is not justifiable by usage volume. In any case, invest first in the evaluation golden dataset — without it, you are flying blind.

Rating: Recommended with conditions

Technical References

Originally published at fernando.moretes.com. By Fernando F. Azevedo — Senior Solutions Architect.

EC2 G7e: Architecture Decision for Generative Video Inference

Fernando Azevedo — Tue, 23 Jun 2026 21:48:36 +0000

When AWS released EC2 G7e instances with NVIDIA L40S GPUs, the first question I asked myself was not 'how fast are they?' — it was 'at which layer of my video inference pipeline do they actually belong, and what is the cost of getting that decision wrong in production?' After 16 years building data and AI platforms in financial-grade environments, I have learned that GPU instance selection is an architecture decision with cost, latency, and operability consequences that propagate for months. This ADR documents the reasoning I would apply today.

Context and Forces: Why Generative Video Inference Is Different

Generative video inference is not image inference multiplied by 24 frames per second. The problem is fundamentally different across three dimensions: temporal activation memory, GPU memory bandwidth, and business-acceptable end-to-end latency.

Models like Stable Video Diffusion, Sora-class architectures, and video models available via Amazon Bedrock (Runway, Pika, native models) maintain temporal state across frames. This means the VRAM footprint is not static — it grows with clip duration and resolution. A 10-second generation at 720p can consume 20–30 GB of VRAM on a typical video diffusion model, while 1080p with motion guidance pushes that to 40+ GB. The NVIDIA L40S GPU in G7e instances offers 48 GB GDDR6 VRAM per GPU, with memory throughput of ~864 GB/s — this changes what is possible without offloading to CPU or NVMe swap, which is where latency collapses.

In the financial context I operate in — media platforms for banks, insurers, and fintechs generating report videos, portfolio simulations, and personalized compliance content — the latency SLO is typically < 90 seconds for a 30-second clip at 720p. This is not gaming; it is regulatory content automation. The dominant force here is not raw throughput, but latency predictability under concurrent load, which is where instance selection, batching strategy, and tenant isolation become first-class architecture decisions.

Architectural Forces That Make the Decision Non-Trivial

Before reaching the decision matrix, it is important to name the forces that make this choice genuinely difficult. First, cost per hour vs. cost per video token: G7e instances carry a higher hourly price than G5 or G6, but if the L40S completes a job 40% faster with zero CPU offloading, the cost per generated clip may be lower. I always model this with real benchmark data before any GPU instance decision — the number that matters is (hourly_cost / clips_per_hour_throughput), not the list price.

Second, regional availability and reserved capacity: latest-generation GPUs have limited availability in specific regions. In financial environments with data residency requirements (LGPD, GDPR, local regulations), I cannot simply choose the region with the best G7e availability. This often forces a trade-off between optimal instance and regulatory compliance, and the correct answer may be Savings Plans + On-Demand fallback rather than Reserved Instances for inference workloads with unpredictable spikes.

Third, tenant isolation in multi-tenant environments: in financial platforms, different clients have different data classifications. Running video inference for two clients on the same GPU instance — even with separate containers — may be unacceptable from a compliance standpoint. NVIDIA MIG (Multi-Instance GPU) offers hardware isolation, but the L40S does not support MIG — this is a critical architectural force that shifts the multi-tenancy strategy to per-instance or per-EKS-node isolation.

Fourth, cold start and model warm-up: large video models (5–15 GB of weights) have cold start times of 30–90 seconds on GPU. For on-demand inference workloads, this is unacceptable. The warm pool strategy with dedicated EKS node groups and model pre-loading via init containers is the pattern I use, but it carries a fixed cost that must be justified by request volume.

GPU Instance Options for Generative Video Inference

EC2 G5 (A10G, 24 GB VRAM)

Pros

Wide regional availability, including regions with LGPD/GDPR requirements
Lower hourly price; mature Savings Plans available
Well-tested container and CUDA driver ecosystem in production

Cons

24 GB VRAM insufficient for 1080p video models without aggressive quantization
CPU offloading required for clips > 8s at 720p, predictable latency collapse
Memory throughput (~600 GB/s) creates bottleneck in temporal diffusion models

Verdict: Suitable for prototypes and short clips < 720p; not recommended for financial-grade production SLOs

EC2 G6 (L4, 24 GB VRAM)

Pros

Superior energy efficiency; better cost/watt for low-latency inference
Native FP8 support improves throughput on quantized models
Good option for image inference and short videos with INT8 quantization

Cons

Same 24 GB VRAM limitation as G5 for full-precision video models
Even more limited availability than G5 in some regions

Verdict: Better than G5 for quantized inference; still insufficient for high-resolution generative video without quality trade-offs

EC2 G7e (L40S, 48 GB VRAM)

Pros

48 GB VRAM eliminates CPU offloading for video models up to 1080p full-precision
864 GB/s memory bandwidth reduces denoising time by ~35% vs L4
Ada Lovelace with FP8 + 4th-gen Tensor Cores; best cost/clip for high-resolution loads
Suitable for next-generation video models without infrastructure re-architecture

Cons

No MIG support: multi-tenancy requires per-instance isolation, increasing base cost
Regional availability still limited in 2025; may conflict with data residency requirements
Higher hourly price requires minimum request volume to justify vs G5

Verdict: Correct choice for production with latency SLOs < 90s for 720p–1080p video and sufficient request volume

Amazon Bedrock (managed video models)

Pros

Zero GPU infrastructure management; no operational cold start
Per-request cost model eliminates idle capacity risk
Native integration with IAM, KMS, CloudTrail — simplified compliance

Cons

No control over model version or inference configuration (temperature, guidance scale)
Non-deterministic latency; service quota throttling can break SLOs
Cost per clip can be 3–5x higher than G7e at high volume (>500 clips/day)

Verdict: Ideal for low volume or MVP; does not scale economically for high-volume content platforms

The Decision: G7e as Inference Layer with Bedrock as Managed Fallback

After evaluating the forces and options, the decision I would make for a financial-grade video generation platform in production is: EC2 G7e as the primary inference layer, orchestrated via EKS with Karpenter, with Amazon Bedrock as a managed fallback for spikes and regions without G7e availability.

The core rationale is as follows: for 720p–1080p video workloads with temporal diffusion models of 5–12 GB of parameters, the L40S with 48 GB of VRAM is the first instance in the G family that completely eliminates CPU offloading. This is not an incremental improvement — it is a regime change. In internal benchmarks with Stable Video Diffusion XL (quantized to FP16), the G7e completes a 10-second clip at 768p in approximately 45–55 seconds, while the G5 with partial CPU offloading takes 110–130 seconds. That is the difference between meeting and violating a 90-second SLO.

The EKS orchestration strategy uses Karpenter with a dedicated NodePool for G7e, with karpenter.k8s.aws/instance-gpu-name: l40s as the node selector. The inference pod is configured with resources.limits.nvidia.com/gpu: 1 and a PodDisruptionBudget ensuring at least N-1 replicas are available during node updates. The warm pool is maintained with a minimum 2-replica Deployment always active, with models pre-loaded via init container that downloads from S3 to /dev/shm (shared RAM) at pod startup — this reduces model cold start from 60s to < 5s.

For the Bedrock fallback, I use an AWS Step Functions state machine that detects throttling (HTTP 429 or latency > 120s) and redirects the request to the corresponding Bedrock endpoint, with payload transformation via Lambda. This is transparent to the API client and maintains the SLO even during unexpected spikes.

Generative Video Inference Pipeline: G7e + EKS + Bedrock Fallback

Flow of a video generation request from the client API to artifact delivery, with primary G7e route and managed fallback via Bedrock

🔐 AWS — API & Orchestration

API Gateway WAF + mTLS (security)
Step Functions Inference Router (compute)
SQS FIFO Job Queue (messaging)

🟧 AWS EKS — Primary Inference (G7e)

Karpenter NodePool: L40S (compute)
Inference Pod nvidia.com/gpu: 1 (ai)
Model Cache /dev/shm (48 GB) (storage)

🤖 AWS Bedrock — Managed Fallback

Bedrock Video Model Endpoint (ai)
Lambda Payload Transform (compute)

🗄️ AWS Storage & Observability

S3 Input Encrypted (KMS) (storage)
S3 Output Signed URL (15min) (storage)
CloudWatch GPU Util + SLO (data)

Flows

client -> apigw: POST /generate
apigw -> sfn: start execution
sfn -> sqs: enqueue job
sqs -> inference_pod: consume job
karpenter -> inference_pod: provision G7e node
inference_pod -> model_cache: load model
inference_pod -> s3_input: read prompt/assets
inference_pod -> s3_output: write generated video
sfn -> lambda_xform: fallback: 429 or latency > 120s
lambda_xform -> bedrock: transform payload
bedrock -> s3_output: write via callback
inference_pod -> cw: GPU metrics + latency
s3_output -> client: signed URL

Security and Compliance Configuration for Financial-Grade Video Inference

In financial environments, generative video inference has an attack surface that goes beyond the typical ML workload. The generated artifacts — report videos, portfolio simulations, onboarding content — may contain implicit client data in the prompts or input assets. This requires a security posture that starts at design time.

Encryption in transit and at rest: all input and output S3 buckets use SSE-KMS with customer-managed keys (CMK) per tenant. The key policy includes a kms:ViaService: s3.amazonaws.com condition combined with aws:PrincipalTag/TenantId to ensure that an inference pod for one tenant cannot decrypt assets from another — even if the EKS node IAM Role is shared. This is a security control that is frequently overlooked when using node instance profiles on EKS.

Pod IAM with IRSA: each inference Deployment uses IAM Roles for Service Accounts (IRSA) with a dedicated role per tenant. The role has s3:GetObject and s3:PutObject permissions restricted to the prefix s3://bucket/tenant-id/* via the s3:prefix condition. The OIDC token is automatically rotated by EKS, and the role has a trust policy that restricts sts:AssumeRoleWithWebIdentity to the specific service account in the correct namespace.

Prompt injection protection: generative video models accept text prompts that can be manipulated by malicious users to extract unintended behaviors. I use a Lambda authorizer on API Gateway that passes the prompt through a lightweight classifier (Amazon Comprehend + custom rules) before enqueuing the job. This is not perfect, but it reduces the most obvious attack vector without adding significant latency (< 200ms).

Audit and traceability: each inference job generates a trace ID that is propagated via X-Amzn-Trace-Id through API Gateway, Step Functions, SQS, and into the inference pod via environment variable. This allows correlating a generated video artifact with the original prompt, the user, the model used, and the GPU instance — essential for regulatory audits.

Numbers That Matter: G7e in Production

48 GB — GDDR6 VRAM per L40S GPU. Eliminates CPU offloading for video models up to 1080p FP16 — regime change, not incremental improvement
~50s — p50 latency for 10s clip at 768p (SVD-XL FP16). vs. ~120s on G5 with partial CPU offloading — difference between meeting and violating a 90s SLO
3–5x — Cost per clip Bedrock vs. G7e at volume > 500 clips/day. Inflection point where G7e becomes more economical than managed Bedrock — calculate with your real data

Decision Consequences and Risks: No MIG on L40S: the absence of Multi-Instance GPU on the L40S is the most important architectural consequence of this decision. In multi-tenant workloads, you cannot share a GPU between two tenants with hardware isolation. This means each concurrently running tenant requires a dedicated G7e instance. For platforms with many low-volume tenants, this may make G7e economically unviable and Bedrock the correct choice. Model your concurrency pattern before deciding.

Regional availability: G7e may not be available in the region your compliance requires. Have a documented Plan B — whether G5 with aggressive quantization, Bedrock, or a multi-region architecture with model replication. Discovering this in production is expensive.

Idle capacity cost: the 2-replica minimum warm pool costs ~$X/hour 24/7. For workloads with low-usage windows (e.g., report generation only on business days), consider scheduled scaling via Karpenter or shutting down the warm pool outside peak hours, accepting model cold start as a trade-off.

Driver and container compatibility: the L40S requires NVIDIA drivers >= 525.x and CUDA >= 12.0. Verify compatibility with the EKS NVIDIA Device Plugin and your inference framework version (TensorRT, vLLM, Diffusers) before going to production. Silent driver incompatibilities are a real failure mode.

Observability: What to Monitor in GPU Video Inference

Observability in GPU inference workloads has an additional layer that most platform teams underestimate: the GPU metrics themselves. The CloudWatch Agent with the NVIDIA DCGM plugin exports metrics such as DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_FB_USED (framebuffer/VRAM used), and DCGM_FI_DEV_MEM_COPY_UTIL directly to CloudWatch or to your EKS cluster's Prometheus. I configure alerts at three levels:

VRAM utilization > 85%: indicates the model is approaching memory limits. If this happens consistently, it means the model or batch size needs adjustment, or the instance is being under-specified for the real workload.

GPU utilization < 30% for > 5 minutes during peak hours: indicates the bottleneck is elsewhere — frequently in prompt pre-processing, S3 asset download, or the SQS queue. This is a signal that the architecture has a non-GPU bottleneck that is wasting expensive capacity.

Job p99 latency > 2x p50: indicates long-tail latency, often caused by jobs that exceed available VRAM and partially swap to CPU. In production, I use a job latency SLO with burn rate alert in CloudWatch: if the SLO violation rate (jobs > 90s) exceeds 5% in a 1-hour window, an alarm fires and Karpenter is authorized to provision additional nodes.

For end-to-end traceability, I use OpenTelemetry with the AWS Distro in the inference pod, emitting spans for each step: model download, prompt tokenization, forward pass, frame decode, output upload. This allows identifying exactly where time is being spent in a slow job — information that is essential for optimization and for responding to SLO incidents.

Anti-Patterns I Have Seen in Production with GPU Inference

Using a single large G7e instance (e.g., g7e.48xlarge with 8 GPUs) instead of multiple smaller instances (g7e.xlarge with 1 GPU): for video inference, 8 GPUs on one node do not help if each job uses 1 GPU — you only increase the blast radius of a node failure and reduce resilience.
Not configuring PodDisruptionBudget: EKS node updates without PDB can interrupt in-progress inference jobs, resulting in partially completed jobs that need reprocessing — double cost and SLO violation.
Storing model weights on EBS instead of /dev/shm or EFS with local cache: model cold start via EBS gp3 can be 3–5x slower than via /dev/shm for models > 5 GB, especially with multiple pods initializing simultaneously.
Using Reserved Instances for GPU inference capacity: video inference workloads have unpredictable spikes. Savings Plans with On-Demand fallback via Karpenter offer a better balance between cost and flexibility than 1- or 3-year RIs.
Ignoring the absence of MIG on L40S and attempting multi-tenancy via Kubernetes namespaces: namespaces do not provide GPU isolation — two pods in different namespaces on the same node share the GPU without hardware isolation, which is unacceptable in regulated financial environments.

My Curation Note: If I were implementing generative video inference today in a financial environment, I would start with Bedrock to validate the business model and SLOs, and only migrate to G7e when daily volume justified the permanent warm pool — typically above 300–500 clips/day. The hardest lesson I have learned in GPU workloads is that the real cost is not in the instance hourly price, but in the idle capacity cost of an oversized warm pool and the reprocessing cost of jobs interrupted by missing PDBs. The absence of MIG on the L40S is an NVIDIA design decision with direct architectural consequences for financial multi-tenancy — it is not a configuration detail, it is a force that must appear in your ADR.

Verdict: When to Adopt G7e and When Not To

EC2 G7e with L40S is the correct choice for production generative video inference when three conditions are simultaneously true: (1) the workload requires video models with > 24 GB of VRAM or latency SLOs < 90s for 720p+ clips, (2) daily volume is sufficient to justify a permanent warm pool (> 300 clips/day as an initial heuristic), and (3) the multi-tenancy model is per-instance, not per-shared-GPU. If any of these conditions is not met, managed Bedrock or G5/G6 with aggressive quantization are more defensible choices. The absence of MIG is the most underestimated limiting factor of this instance for financial environments — put it explicitly in your ADR and model the per-instance isolation cost before committing to the architecture.

Rating: Adopt with conditions / Adotar com condi

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo — Senior Solutions Architect.