DEV Community: AWS Community Builders

Every Rule I Added Made It Worse: How Prompt Bloat Killed My Voice

Chidozie Uzoegwu — Fri, 14 Aug 2026 14:11:21 +0000

I measured the system prompt from my old setup last week. It came to 224,833 characters, roughly 56,000 tokens, on every single call.

Ninety percent of that was one block of accumulated rules about how the output should sound.

I did not set out to build that. Nobody does. It grew one reasonable decision at a time, and by the end it was actively making the output worse, which I spent months blaming on the model.

(That prompt-engineering path is no longer what I run in production. It survives in my codebase as a rollback target, which is exactly why I could still assemble it and measure it.)

How a prompt gets to 56,000 tokens

The system generates short-form written content in a specific voice. I started the way most people do: a good frontier model, a carefully written prompt, and iteration.

The loop that got me here is one every prompt engineer will recognise:

Output comes back with a cliché I hate.
Add a line to the prompt banning that cliché.
Output improves. Genuinely, for a while.
New failure appears. Add another rule.
Repeat for six months.

Each individual addition was justified. Each one was a real observed failure with an obvious textual fix. The categories that accumulated:

Banned phrases, specific clichés that kept reappearing
Banned structures, sentence shapes that read as machine-written
Opener bans, a list of ways it was not allowed to start
Register rules, tone and length and formality per content type
Anti-repetition context, the openers and closers used in recent output, injected so it wouldn't repeat itself
Worked examples, retrieved samples showing the target voice
Meta-rules, instructions about which of the above took priority when they conflicted

That last category is the tell. When your prompt needs rules about how to resolve conflicts between its own rules, the prompt has become a program, and nobody is testing it.

Then came the validators

Rules in a prompt are requests, not guarantees. So when a rule failed to hold, I did the obvious thing and enforced it in code afterwards.

That grew into a stack of post-generation checks: regex to catch a banned sentence shape, a stripper for a punctuation habit, a rejection pass for output that restated its input, a vocabulary blocklist, a fallback for when everything got rejected. At one point I shipped seven of these as a single bundle.

Each was a patch for something the prompt couldn't reliably enforce. And here's the part worth sitting with: I was compounding validators to compensate for a weakness in how the model was being steered. Every new validator was evidence the prompt wasn't working, and my response was another validator.

The symptom I misread

Quality got worse as the rules accumulated.

Not dramatically. Gradually. Output got flatter and more cautious, and it started sounding like something written by a committee avoiding mistakes rather than something with a point of view. Which, functionally, is exactly what it was.

My diagnosis at the time was that the model wasn't good enough. I planned a move to a more expensive tier and put a cost estimate together.

That would not have fixed anything. A stronger model given 56,000 tokens of conflicting constraints produces more expensive committee output.

The actual diagnosis

The rules were competing, with each other and with the task.

A language model attends across the whole context. A prompt is not a config file where each line executes independently. It is context the model weighs all at once. Two hundred lines of prohibitions against a two-line description of the actual job means the overwhelming signal is avoid things. The model optimises for whatever you have given it the most evidence you care about, and I had spent six months providing evidence that what I cared about was not breaking rules.

That produces output which breaks no rules and says nothing.

Worse, it doesn't converge. Adding a rule to fix flatness makes the ratio worse. The tool I was using to fix the problem was the thing causing it.

Why prompting couldn't solve this one

Here's the part I had underweighted for months.

You can instruct a model toward a style. You cannot make a style native to it.

Prompting is instruction. It operates on top of what the model already is. Ask for a voice and you get an impression of that voice, held in place by the instruction, degrading the moment the instruction competes with anything else. The rules were load-bearing scaffolding. Remove them and the voice collapsed. Keep them and they crowded out the task.

That is the signature of using the wrong technique. If a behaviour has to be re-specified in full on every single call, it does not belong in the prompt. It belongs in the weights.

What replaced it

I fine-tuned a model on the voice instead. Later articles in this series cover the how, the cost and the evaluation discipline, but the number relevant here is what happened to the prompt.

Approach	Assembled system prompt
Prompt-engineering	224,833 chars, roughly 56,000 tokens
Fine-tuned model	6,805 chars, roughly 1,700 tokens
Difference	About 33x smaller

You can measure your own the same way. Assemble the prompt exactly as your production path does and count it, rather than eyeballing the template file:

prompt = build_system_prompt(sample_input)   # your real assembly path
print(len(prompt), "chars |", len(prompt) // 4, "approx tokens")

Break it down by section too. Mine was 90% one block, which is what told me the problem was concentrated rather than spread evenly:

import re
sections = [s.strip() for s in re.split(r"={40,}", prompt) if s.strip()]
for n, head in sorted(((len(s), s.split("\n")[0][:50]) for s in sections), reverse=True)[:10]:
    print(f"{n:7d}  {head}")

The voice rules didn't move into a shorter prompt. They moved into the model. What is left in the prompt is what a prompt is actually good at: what to write about right now, how long, which mode. Instruction, not identity.

The lesson I'd give my past self

Strip before you add.

When output is wrong, the reflex is to append a rule, because appending is easy and feels like progress. But every rule is a permanent tax, paid on every call, forever, competing with every other rule, and never reviewed. I never once removed a rule. I only ever added.

Three questions I now ask before adding anything to a prompt:

Is this instruction, or identity? Instruction belongs in the prompt. Identity, meaning a voice or a consistent style or a way of reasoning that must hold every time, belongs in the weights.
Is this knowledge? Facts the model needs but cannot be expected to hold belong in retrieval, not permanently in the prompt.
What am I removing to make room? If the answer is nothing, you are not editing. You are accreting.

The rough division that came out of it, and that I would now design around from day one:

Fine-tuning owns identity. Prompting owns instruction. RAG owns knowledge.

Put a job in the wrong one and you will spend months adding rules that make it worse, which is a fairly expensive way to learn where the boundaries are.

Why Did AWS Release Yet Another Vector Store?

Andres Moreno — Fri, 14 Aug 2026 13:32:45 +0000

When AWS announced DynamoDB Vector Search last week, my first thought was... didn't AWS just launch S3 Vectors?

I mean, S3 Vectors went GA back in December. It's been less than a year, and now we have another service that stores vectors and does similarity search. I'll be honest, I was a bit puzzled. So I decided to dig in and figure out what's going on.

The AWS vector landscape

Before we get into DynamoDB Vector Search specifically, let's acknowledge the elephant in the room. AWS already has vector capabilities in a lot of places:

Amazon OpenSearch
Aurora PostgreSQL
Amazon DocumentDB
Amazon MemoryDB
Amazon Neptune
Amazon S3 Vectors
Amazon DynamoDB Vector Search

That's seven services! While listing these out the question became even louder in my head: why is AWS giving us another one?

My initial assumption

My first instinct was to compare DynamoDB Vector Search and S3 Vectors head-to-head. They both store vectors, perform similarity search, support AI workloads, and are serverless. On paper, they sound like the same thing.

I started going through the documentation for both services trying to figure out which one was "better." I was looking at dimensions supported, distance functions, query limits, the typical comparison stuff.

And then I realized I was asking the wrong question. The question isn't which vector store is better?, the question is: what kind of data do these vectors represent?

That distinction changes everything, it's not about the vectors themselves, it's about what the vectors are attached to.

Operational vectors

Think about the data that lives in your application's hot path:

Users: profile embeddings for personalization
Products: catalog embeddings for recommendations
Session memory: conversation context for AI agents
Recommendations: real-time suggestions based on behavior
Fraud detection: transaction pattern matching

These are all operational, data that changes frequently, is queried with low latency requirements, lives alongside other application data that your code already reads and writes, and represents your application's state.

This is where DynamoDB Vector Search makes sense. If your product catalog already lives in DynamoDB, why would you copy those embeddings to a separate vector database? You'd have to build a synchronization pipeline, handle eventual consistency, manage another service, and pay for data movement. With native vector search, you store the embedding right next to the item and query it in the same place with a single write within the same service.

Knowledge vectors

Now think about a different category of data:

PDFs: company documents, research papers
Documentation: internal wikis, product manuals
Policies: compliance docs, HR handbooks
Support articles: knowledge base content
Code: repository embeddings for search

These are knowledge assets, they don't change very frequently, they are typically large collections that get ingested in bulk, and power RAG pipelines where an AI retrieves context before generating a response. They don't belong in your application's transactional database because they're used as reference material instead of for the application state.

This is where S3 Vectors shines. It's purpose-built for storing massive collections of embeddings at the lowest possible cost. It's optimized for the pattern where you embed a ton of documents and then query against them. AWS claims up to 90% cost savings compared to specialized vector databases for these workloads, and for large knowledge bases that makes a big difference.

Why this matters more than pricing

The first thing most people do when two services overlap is compare pricing. And yes, the cost profiles are different. But I think the architectural decision is more important than the price per query.

Data locality: Your operational vectors should live where your operational data lives. If a user's embedding is stored in DynamoDB alongside their profile, you can update both in a single write avoiding stale data from a syncing pipeline.

Data ownership: Knowledge vectors often come from content that lives in S3 already. Keeping the embeddings close to the source material in S3 Vectors means one fewer system to reason about.

Simplicity: Every time you add a synchronization pipeline between two services, you add a failure mode, latency, cost, and operational burden. Choosing the right vector store based on what the data represents lets you avoid that complexity entirely.

Latency: DynamoDB Vector Search delivers single-digit millisecond latency. This is really impactful when you're doing real-time recommendations or fraud detection in the hot path of a user request. S3 Vectors optimizes for throughput and cost on large collections, where a few extra milliseconds won't hurt.

Why DynamoDB Vector Search exists

Let me be specific about what makes this valuable for operational workloads:

Single logical write: Store the embedding alongside the item with a regular PutItem call, without the need for a separate ingestion pipeline.
Native indexing: Create a vector index on an attribute and DynamoDB handles the rest, the same way you would create a new global secondary index. With the added benefit that it automatically scales horizontally as your data grows.
Metadata filters: Narrow search results using your existing DynamoDB attributes. For example, you can search products by similarity but filter to a specific category or marketplace.
Operational workloads: Built for the same access patterns DynamoDB already excels at, that provide high throughput, low latency and predictable performance.
No infrastructure: Same serverless model DynamoDB users already know.

If you're already using DynamoDB and want to add semantic search to your application, you no longer need to maintain a separate vector database and the pipeline that feeds it.

Where S3 Vectors shines

On the other side, S3 Vectors is built for a different set of problems:

Massive collections: Billions of vectors at scale without worrying about provisioning
Enterprise RAG: Power your retrieval-augmented generation pipelines with large knowledge bases
Cost optimization: When you have millions of documents embedded and sitting mostly idle, the cost model of S3 Vectors is hard to beat
Long-lived embeddings: Documents that get embedded once and queried many times over months or years

If you're building a knowledge base for your agents or a documentation search system, S3 Vectors is the natural fit.

The mental model

Here's how I think about it now:

If your vectors represent...	Consider
Users	DynamoDB Vector Search
Products	DynamoDB Vector Search
Recommendations	DynamoDB Vector Search
Session memory	DynamoDB Vector Search
Fraud signals	DynamoDB Vector Search
PDFs	S3 Vectors
Documentation	S3 Vectors
Knowledge bases	S3 Vectors
Support articles	S3 Vectors

The simplest way I can put it: if the data is something your application actively mutates and queries in real-time, it's operational and DynamoDB vector search is the perfect fit. If the data is something you embed once and retrieve many times as context, it's knowledge and you should use S3 Vectors.

Wrap up

Going back to the question I started with: why did AWS release another vector store?

The answer is that AWS isn't just adding more vector databases to the catalog, they're bringing vector capabilities to the storage engines developers already use. DynamoDB users shouldn't have to leave DynamoDB to do similarity search on their operational data, and S3 users shouldn't have to spin up a separate database to search their document embeddings.

It's not about which service is better, it's about which data model matches your vectors.

Let me know what you think about this distinction! I'd love to hear how others are thinking about where to put their vectors.

Andres Moreno

AWS Managed Microsoft AD - integração com FortiGate

Railander Marques — Fri, 14 Aug 2026 13:32:14 +0000

1. Objetivo

Migrar a autenticação LDAP do FortiGate do cliente (hoje em texto claro na porta 389, via túnel VPN) para LDAPS na porta 636, com criptografia TLS ponta a ponta, mantendo o mesmo diretório AWS Managed Microsoft AD.

Resultado esperado: os DCs gerenciados passam a escutar em 636 com um certificado válido, e o FortiGate valida esse certificado contra a CA que confiamos nele.

2. Conceitos — o que confunde todo mundo

2.1 Server-side vs client-side LDAPS

O console do AWS Directory Service tem uma aba de LDAPS com opção de registrar certificado. Ela não serve para este cenário.

	Client-side LDAPS	Server-side LDAPS (o nosso caso)
Quem é o servidor LDAP	Um AD self-managed / outro diretório	O AWS Managed AD
Quem é o cliente LDAP	O AWS Managed AD e apps AWS	O FortiGate
O que se faz no console	Importa o certificado da CA do outro lado e habilita	Nada
Onde está o trabalho	Console	Dentro da EC2 de gerência

Ou seja: não há nada para habilitar, importar ou emitir pelo console do Directory Service. A porta 636 sobe sozinha no momento em que os DCs recebem um certificado válido. Todo o trabalho é PKI dentro do Windows.

2.2 Por que não dá para subir um `.pfx` qualquer

O AWS Managed AD é serviço gerenciado: você não tem RDP nem administrador local nos DCs. A única forma de instalar um certificado neles é o autoenrollment do Active Directory — o próprio DC solicita o certificado a uma CA da floresta e instala sozinho.

Consequência: a CA precisa ser uma Microsoft Enterprise CA ingressada no domínio. Certificado de CA standalone, de CA pública (DigiCert, Let's Encrypt) ou arquivo avulso não funcionam — não é limitação da AWS, é como o autoenrollment do AD funciona.

2.3 A cadeia de confiança

Três certificados diferentes, papéis diferentes:

Certificado	Onde vive	Quem usa	Sai da AWS?
Chave privada da CA	EC2 de gerência	Assina os demais	Nunca
Certificado do DC	Store `LocalMachine\My` de cada DC	Apresentado no handshake TLS da 636	Não (você nem toca nele)
Certificado público da CA	Exportado em `.cer` Base64	Importado no FortiGate	Sim — é o único que se envia

O FortiGate não envia certificado nenhum para o AD. Ele só precisa confiar em quem assinou o certificado do DC.

O certificado de VPN que o cliente eventualmente tenha enviado não tem relação com isso. VPN e LDAPS são camadas independentes: a VPN entrega o pacote na VPC, o LDAPS criptografa a sessão LDAP dentro dela.

3. Arquitetura

3.1 Topologia

   REDE DO CLIENTE                    │             AWS — VPC (sua conta)
                                      │
  ┌────────────────────┐              │    ┌──────────── Subnet privada AZ-a ────────────┐
  │     FortiGate      │              │    │  ┌───────────────────────────────────────┐  │
  │                    │              │    │  │ DC1 — AWS Managed Microsoft AD        │  │
  │ Trusted CA store:  │              │    │  │ ENI 10.0.1.10  ·  Windows Server 2019 │  │
  │  └ CORP-ROOT-CA    │              │    │  │ SG: d-xxxxxxxxxx_controllers          │  │
  └─────────┬──────────┘              │    │  └───────────────────────────────────────┘  │
            │                         │    └─────────────────────────────────────────────┘
            │ TCP 636 (LDAPS)         │
            │                         │    ┌──────────── Subnet privada AZ-b ────────────┐
  ┌─────────┴──────────┐   IPsec   ┌──┴──┐ │  ┌───────────────────────────────────────┐  │
  │  Túnel VPN / DX    │◄─────────►│ VGW │ │  │ DC2 — AWS Managed Microsoft AD        │  │
  └────────────────────┘           └──┬──┘ │  │ ENI 10.0.2.10                         │  │
                                      │    │  └───────────────────────────────────────┘  │
                                      │    └─────────────────────────────────────────────┘
                                      │
                                      │    ┌──────────── Subnet privada (gerência) ──────┐
                                      │    │  ┌───────────────────────────────────────┐  │
                                      │    │  │ EC2 Windows de gerência               │  │
                                      │    │  │  · ingressada no domínio              │  │
                                      │    │  │  · RSAT / AD Tools                    │  │
                                      │    │  │  · AD CS — Enterprise Root CA  ◄── chave privada  │
                                      │    │  └───────────────────────────────────────┘  │
                                      │    └─────────────────────────────────────────────┘

3.2 Fluxo 1 — Emissão do certificado (uma vez, automático)

  EC2 gerência                          DC1 / DC2 (gerenciados)
       │                                        │
       │  1. AD CS instalado como Enterprise CA │
       │──── publica na floresta ──────────────►│
       │     (CN=Configuration → Public Key     │
       │      Services → Certification          │
       │      Authorities / NTAuthCertificates) │
       │                                        │
       │                                        │ 2. DC detecta CA + template
       │                                        │    com permissão Autoenroll
       │                                        │
       │  3. Solicitação via RPC/DCOM (MS-ICPR) │
       │◄─────── TCP 135 + portas dinâmicas ────│
       │                                        │
       │  4. CA assina e devolve o certificado  │
       │──────────────────────────────────────► │
       │                                        │
       │                                        │ 5. Instala em LocalMachine\My
       │                                        │    → serviço LDAP abre a 636

O passo 3 é o motivo de a AWS pedir liberação ampla entre o SG do diretório e o SG da CA: o enrollment usa RPC com portas dinâmicas, não uma porta fixa.

3.3 Fluxo 2 — Autenticação do FortiGate (a cada consulta)

  FortiGate                                          DC (10.0.1.10)
      │                                                    │
      │ 1. TCP SYN :636 ─────────────────────────────────► │
      │ 2. ClientHello ──────────────────────────────────► │
      │ 3. ◄──────── ServerHello + certificado do DC       │
      │                                                    │
      │ 4. Valida: assinado pela CA que confio?            │
      │            CN/SAN bate com o nome que consultei?   │
      │            está dentro da validade?                │
      │                                                    │
      │ 5. Túnel TLS estabelecido ◄──────────────────────► │
      │ 6. LDAP bind (usuário de serviço) ───────────────► │
      │ 7. Search: (sAMAccountName=usuario) ─────────────► │
      │ 8. ◄──────── DN + grupos do usuário                │

O passo 4 é onde 90% das integrações quebram — ver seção 9.

3.4 Portas

Origem → Destino	Porta	Para quê	Obrigatória
FortiGate → DCs	TCP 636	LDAPS	Sim
FortiGate → DCs	TCP 389	LDAP em claro (legado)	Só até a migração concluir
DCs → EC2 da CA	TCP 135 + dinâmicas (49152-65535)	Autoenrollment RPC	Sim
DCs → EC2 da CA	TCP 445	CDP/AIA em file share	Se usar publicação SMB
FortiGate → DCs	TCP 3269	LDAPS no Global Catalog	Só se buscar em múltiplos domínios

4. Pré-requisitos

[ ] EC2 Windows ingressada no domínio do AWS Managed AD (a instância de gerência já existente)
[ ] Credencial no grupo Admins ou AWS Delegated Enterprise Certificate Authority Administrators do diretório (a conta Admin padrão atende)
[ ] Conectividade FortiGate → VPC já funcionando (se o 389 funciona hoje, está atendido)
[ ] Usuário de serviço para bind já existente no domínio
[ ] Janela de ~1h, sendo até 30 min só de espera pela emissão

Atenção ao DN dos objetos. No AWS Managed AD você não cria objetos em CN=Users,DC=.... Tudo fica sob a sua OU delegada: OU=Users,OU=corp,DC=corp,DC=empresa,DC=com. Errar isso é a causa mais comum de bind falhando depois que o TLS já subiu.

5. Passo 1 — Security Groups (console AWS)

Duas liberações distintas, com finalidades diferentes:

a) FortiGate → DCs (o acesso final)

EC2 → Security Groups → selecione d-xxxxxxxxxx_controllers
Inbound rules → Edit → Add rule
- Type: Custom TCP
- Port: 636
- Source: o mesmo CIDR/IP privado que hoje já libera o 389

Nunca use IP público como origem. Os DCs ficam em subnets privadas e não têm IP público — o tráfego chega exclusivamente pelo túnel.

b) DCs ↔ EC2 da CA (o enrollment)

No SG da EC2 de gerência: Inbound → All traffic → Source = SG do diretório
No SG do diretório: Outbound → All traffic → Destination = SG da EC2 de gerência

Sem o item (b) o DC nunca consegue pedir o certificado e o LDAPS jamais sobe, mesmo com o AD CS instalado corretamente.

6. Passo 2 — Instalar a CA

RDP na instância de gerência com a conta Admin do diretório. PowerShell como administrador:

Install-WindowsFeature ADCS-Cert-Authority -IncludeManagementTools

Install-AdcsCertificationAuthority -CAType EnterpriseRootCA `
  -CACommonName "CORP-ROOT-CA" `
  -CryptoProviderName "RSA#Microsoft Software Key Storage Provider" `
  -KeyLength 2048 -HashAlgorithmName SHA256 `
  -ValidityPeriod Years -ValidityPeriodUnits 10 -Force

O que cada escolha significa:

EnterpriseRootCA — é o que registra a CA na floresta e habilita autoenrollment. StandaloneRootCA não funciona para este fim.
CACommonName — nome livre. Não precisa ser o domínio; o AD CS descobre o domínio sozinho porque a instância está ingressada. Escolha algo identificável, porque esse nome vai aparecer no FortiGate do cliente.
ValidityPeriod 10 anos — validade da CA raiz. Os certificados dos DCs terão 1 ano e renovam automaticamente.

Se o cliente já possui PKI corporativa, o correto é instalar como EnterpriseSubordinateCA e submeter o request à raiz dele — nesse caso o pacote enviado precisa conter a cadeia completa, não só um certificado.

7. Passo 3 — Template de certificado

Uma Enterprise CA nova já publica por padrão os templates Domain Controller Authentication e Kerberos Authentication, e os DCs têm permissão de Autoenroll neles. Na maioria dos casos a 636 sobe sem nenhuma configuração adicional — pule para o Passo 4 e só volte aqui se não subir.

Caminho oficial da AWS, caso necessário:

Server Manager → Tools → Certification Authority
Botão direito em Certificate Templates → Manage
Botão direito em Kerberos Authentication → Duplicate Template
Aba Compatibility: Certification recipient → Windows 10 / Windows Server 2016 (os DCs do AWS Managed AD rodam Windows Server 2019)
Aba General: Template display name → LDAPOverSSL
Aba Security: selecione Domain Controllers e confirme Read, Enroll e Autoenroll marcados
OK, feche o console de templates
Botão direito em Certificate Templates → New → Certificate Template to Issue → selecione LDAPOverSSL

O template Kerberos Authentication é o escolhido porque já traz os EKUs certos (Server Authentication, Client Authentication, Smart Card Logon, KDC Authentication) e preenche o SAN com o FQDN do DC e o FQDN do domínio — é isso que permite ao FortiGate consultar por corp.empresa.com sem erro de identidade.

8. Passo 4 — Validar

Aguarde até 30 minutos após o Passo 2/3 e rode na instância de gerência:

$fqdn = (Get-WmiObject Win32_ComputerSystem).Domain
Test-NetConnection $fqdn -Port 636
certutil -dcinfo verify

Critérios de aceite:

TcpTestSucceeded : True
certutil -dcinfo verify lista os DCs com certificado válido e sem erro de cadeia

Teste funcional de bind com o ldp.exe (vem com as AD Tools):

ldp.exe → Connection → Connect
Server: FQDN do domínio · Port: 636 · marcar SSL
A janela deve retornar os atributos do RootDSE (se aparecer só erro 81 = servidor indisponível, o certificado ainda não foi emitido)
Connection → Bind com o usuário de serviço, para confirmar que a conta de bind funciona

Só avance quando os dois passarem. Se falhar aqui, o problema é seu — não adianta enviar nada ao cliente.

9. Passo 5 — Exportar o material e montar o pacote

certutil -ca.cert C:\ca-der.cer
certutil -encode C:\ca-der.cer C:\ca-publica-base64.cer
certutil -backupkey C:\backup-ca

ca-publica-base64.cer → é o arquivo que vai para o cliente. Base64/PEM é o formato que o FortiGate importa.
C:\backup-ca → backup da chave privada. Guarde em local seguro fora da instância (S3 com KMS, cofre de senhas). Nunca envie ao cliente.

Pacote a entregar:

Item	Valor
Certificado da CA	`ca-publica-base64.cer`
Servidor LDAP	FQDN do domínio, ex. `corp.empresa.com` (não IP)
IPs dos DCs	10.0.1.10 e 10.0.2.10 — apenas para DNS/rota, não para configurar como servidor
Porta	636
Base DN	`DC=corp,DC=empresa,DC=com`
Common Name Identifier	`sAMAccountName`
Usuário de bind	o mesmo já em uso no 389

10. Passo 6 — Lado do cliente (FortiGate)

GUI

System → Certificates → Import → CA Certificate → upload do .cer
User & Authentication → LDAP Servers → editar o objeto existente
Server Port → 636 · Secure Connection → marcar · Protocol → LDAPS · Certificate → a CA importada
Test Connectivity e Test User Credentials

CLI equivalente

config user ldap
    edit "AWS-Managed-AD"
        set server "corp.empresa.com"
        set cnid "sAMAccountName"
        set dn "DC=corp,DC=empresa,DC=com"
        set type regular
        set username "CN=svc_fortigate,OU=Users,OU=corp,DC=corp,DC=empresa,DC=com"
        set password ********
        set port 636
        set secure ldaps
        set ca-cert "CORP-ROOT-CA"
        set server-identity-check enable
    next
end

Validação no FortiGate:

diagnose test authserver ldap AWS-Managed-AD <usuario> <senha>

Pré-requisito no lado dele: o FortiGate precisa resolver o FQDN do domínio pelos DCs do AD. Se o DNS dele não aponta para os DCs, configure set dns-primary para o IP de um DC ou negocie o uso do FQDN do DC específico.

11. Troubleshooting

Sintoma	Causa provável	Ação
`Test-NetConnection :636` = False na instância de gerência	Certificado ainda não emitido	Aguardar 30 min; conferir SG DCs ↔ CA; conferir se a CA é Enterprise, não Standalone
Idem, após 1h	Template sem Autoenroll para Domain Controllers ou não publicado	Executar o Passo 3
`ldp.exe` erro 81	LDAP não está escutando na 636	Mesmo diagnóstico acima
636 OK interno, FortiGate não conecta	SG inbound 636 ou rota do túnel	Conferir origem da regra (IP privado, não público)
`Failed to establish SSL connection` no FortiGate	CA não importada, ou cadeia incompleta (caso subordinada)	Reenviar cadeia completa
Erro de certificado apesar da CA importada	FortiGate apontando para IP; `server-identity-check` compara com o CN/SAN	Usar FQDN, ou `set server-identity-check disable`
TLS OK mas bind falha	DN do usuário de serviço errado	Usar a OU delegada, não `CN=Users`
Funcionou e parou de funcionar ~1 ano depois	Certificado do DC expirou sem renovar	Ver seção 12

Comando de diagnóstico no FortiGate:

diagnose debug application fnbamd -1
diagnose debug enable

12. Operação contínua — o que não pode ser esquecido

Ao instalar a CA na instância de gerência, ela vira um trust anchor de longo prazo. Isso cria obrigações:

Item	Risco	Mitigação
Certificado do DC expira em 1 ano	Renovação automática só ocorre se a CA estiver online	Manter a instância viva e monitorada
Instância descartada / recriada	Perda da chave privada da CA → sem renovação e sem CRL	`certutil -backupkey` guardado fora da instância; documentar restore
CRL expira (padrão: 1 semana)	Validações de certificado passam a falhar	Aumentar o intervalo de publicação ou garantir que a CA fica online
Instância desligada por economia	Enrollment e CRL param	Não incluir essa instância em schedulers de stop

Comandos úteis de operação:

certutil -CRL                                   # publica CRL manualmente
certutil -getreg CA\CRLPeriod*                  # consulta período da CRL
certutil -getreg CA\ValidityPeriod*             # validade dos certs emitidos

Adicione ao monitoramento: expiração do certificado dos DCs, expiração da CRL e status do serviço CertSvc.

13. Anexo — por que não usar o AWS Private CA

O AWS Private CA Connector for AD faz o mesmo trabalho de forma gerenciada, sem EC2 e sem chave privada sob sua custódia. É tecnicamente superior, mas custa USD 400/mês por CA em modo general-purpose (o connector em si é gratuito; paga-se a CA e os certificados).

Para um caso de uso restrito a habilitar LDAPS, o AD CS na instância de gerência já existente resolve com custo marginal zero. O Private CA passa a valer a pena quando há emissão de certificados em escala (usuários, máquinas, mTLS) ou quando a custódia da chave privada em EC2 é inaceitável por compliance.

Diferenças operacionais relevantes:

	AD CS na EC2	AWS Private CA + Connector
Custo	Só a EC2	~USD 400/mês por CA
Chave privada	Sob sua responsabilidade	Gerenciada pela AWS
Habilitar cert nos DCs	Automático via autoenrollment	Explícito: Actions → Enable domain controller certificates
Patching / disponibilidade	Sua responsabilidade	AWS
Tempo até emissão	Até 30 min	Até 8 horas

Referências

Connecting Amazon S3 Tables and Snowflake (via the S3 Tables Iceberg REST Endpoint)

Aki — Fri, 14 Aug 2026 12:26:11 +0000

Original Japanese article: Amazon S3 TablesとSnowflakeを連携する（S3 Tables Iceberg RESTエンドポイント経由）

Introduction

I'm Aki, an AWS Community Builder (@jitepengin).

In an earlier article, I verified how to read from and write to an Iceberg table on S3 Tables using Snowflake's Catalog Integration via the AWS Glue Iceberg REST endpoint.

Connecting Amazon S3 Tables with Snowflake

That setup routed through Snowflake → the Glue Iceberg REST endpoint → Lake Formation → S3 Tables, and required setting up an IAM policy, a trust policy, registering a Lake Formation data location, and granting permissions across all three levels (Catalog / Database / Table).

On 2026/8/10, Snowflake's Amazon S3 Tables Iceberg REST catalog integration went GA! This is a new route that connects with SigV4 authentication directly to S3 Tables' native Iceberg REST endpoint, without going through Glue at all.

https://docs.snowflake.com/en/release-notes/2026/other/2026-08-10-amazon-s3-tables-iceberg-rest-catalog-integration-ga

This time, I wanted to put this new route to the test.

Test Environment

I reused the same resources as the previous article.

Table bucket: penguin-rest-test
Namespace: analytics
Table: daily_sales (columns sales_date, amount)
Region: ap-northeast-1

Architecture

This time's architecture connects the S3 Tables gold layer to Snowflake. The key point is that it uses the Amazon S3 Tables Iceberg REST catalog integration instead of Glue Data Catalog.

Differences from the Glue Route (Configuration)

The CATALOG INTEGRATION parameters differ, as shown below, configured specifically for S3 Tables.

Item	Via Glue (previous article)	Direct to S3 Tables (this article)
`CATALOG_API_TYPE`	`AWS_GLUE`	`AWS_S3TABLES`
`CATALOG_URI`	`glue.<region>.amazonaws.com/iceberg`	`s3tables.<region>.amazonaws.com/iceberg`
`CATALOG_NAME`	`<account_id>:s3tablescatalog/<bucket>`	`arn:aws:s3tables:<region>:<account_id>:bucket/<bucket>`

Differences from the Glue Route (Setup Steps)

Of the steps required in the previous article, the following Lake Formation-related steps simply don't exist here:

Registering the Lake Formation data location (register-resource --with-federation --with-privileged-access)
Granting Lake Formation permissions across all three levels (Catalog / Database / Table)
Adding sts:SetSourceIdentity / sts:SetContext to the data access role's trust policy

I wanted to make sure "Lake Formation setup really isn't required" is actually true, so I later confirmed this via CloudTrail.

Creating the IAM Role

Starting from the previous article's IAM policy, I removed all Lake Formation- and Glue-related actions, and tested with a minimal-privilege policy scoped down to just s3tables:* actions.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowS3TablesDirectAccess",
            "Effect": "Allow",
            "Action": [
                "s3tables:GetTableBucket",
                "s3tables:ListNamespaces",
                "s3tables:GetNamespace",
                "s3tables:ListTables",
                "s3tables:GetTable",
                "s3tables:GetTableMetadataLocation",
                "s3tables:UpdateTableMetadataLocation",
                "s3tables:GetTableData",
                "s3tables:PutTableData"
            ],
            "Resource": [
                "arn:aws:s3tables:ap-northeast-1:123456789012:bucket/penguin-rest-test",
                "arn:aws:s3tables:ap-northeast-1:123456789012:bucket/penguin-rest-test/table/*"
            ]
        }
    ]
}

The key point is that there's no need to include lakeformation:GetDataAccess or any glue:* actions.

Creating the Catalog Integration

On the Snowflake side, I created a catalog integration that connects directly to the S3 Tables Iceberg REST endpoint.

CREATE OR REPLACE CATALOG INTEGRATION penguin_s3tables_direct_int
  CATALOG_SOURCE = ICEBERG_REST
  TABLE_FORMAT = ICEBERG
  REST_CONFIG = (
    CATALOG_URI = 'https://s3tables.ap-northeast-1.amazonaws.com/iceberg'
    CATALOG_API_TYPE = AWS_S3TABLES
    ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS
    CATALOG_NAME = 'arn:aws:s3tables:ap-northeast-1:123456789012:bucket/penguin-rest-test'
  )
  REST_AUTHENTICATION = (
    TYPE = SIGV4
    SIGV4_IAM_ROLE = 'arn:aws:iam::123456789012:role/penguin-s3tables-direct-role'
    SIGV4_SIGNING_REGION = 'ap-northeast-1'
  )
  ENABLED = TRUE;

Compared side by side with the Glue-based article, CATALOG_API_TYPE changes from AWS_GLUE to AWS_S3TABLES, and CATALOG_URI changes from the Glue Iceberg REST endpoint to S3 Tables' native endpoint. CATALOG_NAME also now takes the S3 Tables bucket ARN directly, so the previous article's distinctive <account_id>:s3tablescatalog/<bucket> prefix notation is not used here.

Setting Up the Trust Policy

The steps for retrieving Snowflake's IAM user ARN and external ID via DESC CATALOG INTEGRATION, then setting them on the trust policy of the role specified in SIGV4_IAM_ROLE, are the same as with the Glue route.

DESC CATALOG INTEGRATION penguin_s3tables_direct_int;

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "<AWS_IAM_USER_ARN>"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "<AWS_EXTERNAL_ID>"
                }
            }
        }
    ]
}

Everything up to this point is unchanged from the previous article. The difference shows up from here on: we move straight to table creation with no Lake Formation setup at all.

Creating the Table

CREATE OR REPLACE ICEBERG TABLE GOLD.GOLD.s3tables_direct_table
  CATALOG = 'penguin_s3tables_direct_int'
  CATALOG_NAMESPACE = 'analytics'
  CATALOG_TABLE_NAME = 'daily_sales'
  AUTO_REFRESH = TRUE;

The table was created without any issues, with no Lake Formation data location registration or grants in place.

Verifying It Works (Read)

ALTER ICEBERG TABLE GOLD.GOLD.s3tables_direct_table REFRESH;
SELECT * FROM GOLD.GOLD.s3tables_direct_table;

Data that had already been loaded via Athena was readable from Snowflake without any problem.

Confirming the Lake Formation Calls via CloudTrail

This is the heart of this article. I traced through CloudTrail to see exactly which APIs penguin-s3tables-direct-role was actually calling.

Looking at the logs right after running the read operation, this role only calls GetTableBucket, GetNamespace, ListNamespaces, GetTable, ListTables, and GetTableMetadataLocation against s3tables.amazonaws.com, all of them readOnly: true events.

Three things sealed it:

No invokedBy field
Events from the Glue-based Snowflake_Catalog_Integration role always carry "invokedBy": "glue.amazonaws.com", proof that the resource is being accessed via Glue's service role. Events from the direct role have no such field at all, showing that Snowflake is hitting s3tables.amazonaws.com directly.
sourceIPAddress is a real IP
Events from the direct role record Snowflake's actual source IP in sourceIPAddress, whereas Glue-route events show sourceIPAddress: "glue.amazonaws.com", the notation for an internal service-to-service call.
Zero lakeformation.amazonaws.com events
Across the entire test window (roughly 30 minutes, several hundred CloudTrail events including both reads and writes), there were zero lakeformation.amazonaws.com events attributable to penguin-s3tables-direct-role. During that same window, Snowflake_Catalog_Integration (the Glue route) generated a large volume of lakeformation:GetDataAccess calls, so this isn't a case of events happening to not show up in the log; it's a structural fact that the direct route never goes through Lake Formation.

Verifying It Works (Write)

Beyond reads, let's confirm writes work too. Since the IAM policy already includes s3tables:PutTableData, this can be tested with no additional permission changes.

INSERT INTO GOLD.GOLD.s3tables_direct_table (sales_date, amount)
VALUES
    ('2026-08-14', 99999),
    ('2026-08-14', 88888);

Checking from the Athena side, the data inserted from Snowflake was visible without issue.

SELECT * FROM "analytics"."daily_sales" ORDER BY sales_date DESC LIMIT 10;

I also checked CloudTrail for the write operation, and just as with the read, found no Lake Formation events attributable to penguin-s3tables-direct-role.

Checking Catalog-Linked Database

In the previous article, I verified automatic table discovery via a Glue-based Catalog-Linked Database (CLD), so I wanted to confirm the same works on the direct route.

CREATE OR REPLACE DATABASE penguin_s3tables_direct_linked_db
  LINKED_CATALOG = (
    CATALOG = 'penguin_s3tables_direct_int'
  );

As soon as this was created, daily_sales and daily_sales2 under the analytics namespace showed up automatically in the table list, with no DDL required.

Confirming Auto-Discovery

I created a new daily_sales3 table from the Athena side, and it was automatically detected on the SnowSight side as well.

-- On the Athena side
CREATE TABLE `analytics`.daily_sales3 (
  sale_date date,
  product_category string,
  sales_amount double)
PARTITIONED BY (month(sale_date))
TBLPROPERTIES ('table_type' = 'iceberg')

Confirming Bidirectional Writes

I inserted into the auto-discovered daily_sales3 from Snowflake, then checked whether it was visible from Athena.

INSERT INTO PENGUIN_S3TABLES_DIRECT_LINKED_DB.analytics.daily_sales3 (sale_date, product_category, sales_amount)
VALUES
    ('2026-08-14', 'Electronics', 15000.00),
    ('2026-08-14', 'Clothing', 8500.50),
    ('2026-08-14', 'Food', 3200.75);

Confirmed without issue on the Athena side. Just like the CREATE ICEBERG TABLE approach, the CLD approach also works correctly on the direct route.

How the Two Routes Compare

Putting the previous article's findings together with this one's:

Item	Via Glue (previous article)	Direct to S3 Tables (this article)
IAM policy	`s3tables:` + `lakeformation:GetDataAccess` + `glue:` actions	`s3tables:*` actions only
Lake Formation data location registration	Required (`register-resource --with-federation`)	Not required
Lake Formation grants (3 levels)	Required	Not required
Trust policy `SetSourceIdentity` / `SetContext`	Required	Not required
LF calls on read (CloudTrail)	Occur	Zero
LF calls on write (CloudTrail)	Occur	Zero
CLD auto-discovery and bidirectional writes	Succeeded (confirmed in previous article)	Succeeded (confirmed here)
Read/write operations	Succeeded (confirmed in previous article)	Succeeded (confirmed here)

The biggest difference is a clear tradeoff: do you want unified governance through Lake Formation, or a simpler permission design?

If you want to keep things simple, go with the direct S3 Tables route. If you need unified table management, centralized governance controls, and fine-grained access control, the Glue route is the way to go.

Conclusion

This time, I used Snowflake's Amazon S3 Tables Iceberg REST catalog integration, which went GA on 2026/8/10, to verify the new Glue-free direct route.

To summarize:

The new direct route (CATALOG_API_TYPE = AWS_S3TABLES) has no Lake Formation-related setup steps in the official documentation.
Even after completely removing all Lake Formation and Glue actions from the IAM policy, both reads and writes worked without any issues.
Checking CloudTrail confirmed that lakeformation:GetDataAccess calls from the direct role were zero throughout the entire test window.
Events from the direct role carry no invokedBy: glue.amazonaws.com field, and sourceIPAddress shows a real IP, further confirming that Snowflake accesses the S3 Tables endpoint directly, without going through Glue.
Catalog-Linked Database auto-discovery and bidirectional reads/writes also worked correctly on the direct route, and no Lake Formation calls occurred during the CLD's background sync either.

With the integration verified in this article, using the S3 Tables Iceberg REST endpoint, no Lake Formation-related permissions are needed at all. If you're willing to forgo unified governance through Lake Formation in exchange for a simpler permission design, this looks like a strong option.

I hope this article is helpful to anyone considering connecting S3 Tables and Snowflake.

How I build a free VPN over AWS CloudShell

Jérôme GUYON — Fri, 14 Aug 2026 09:51:10 +0000

From time to time when I'm travelling, mainly in Malaysia which has become kind of a second home 🇲🇾, I face geo-restrictions from CDNs or applications that prevent me from accessing what I need. A badly configured car rental website CDN, French government websites that block non-European IPs. This is very occasional usage, and I don't want to pay for yet another subscription and I also don't trust most VPN vendors with my traffic. As an IT professional, the question became: can I build or host my own for nearly $0?

That's when I started looking at AWS CloudShell. A free Linux shell in any AWS region, with outbound internet access. I recently published a blog article about the hidden CloudShell API, and the next logical step was obvious: what if I could route my laptop's traffic through it?

I'm calling it cloudshell-vpn and it's available on macOS. It routes all your traffic through any AWS region using CloudShell as an exit node. The total cost is $0 within the free tier's 100 GB/month outbound.

The CloudShell problem

CloudShell gives you a free Linux shell in any AWS region with outbound internet access. It can run OpenVPN. But it lives behind NAT with no inbound ports. You can't SSH into it, you can't expose a port, and you can't attach an Elastic IP. You can't run SSM StartPortForwardingSession either: CloudShell isn't a registered SSM managed instance. It only supports interactive shell sessions, not SSM documents like port forwarding.

So how do you establish a tunnel to something you can't reach?

The NAT hole punching solution

The approach borrows a technique from peer-to-peer networking. Both sides (your laptop and the CloudShell environment) discover their public IP and port using STUN, then simultaneously send UDP packets to each other. The NAT routers on both sides see outbound traffic and open pinholes. Once both sides have punched through, you have a bidirectional UDP path with no port forwarding required.

Inspired by Dan V.'s cloudshell-store project, I realized I could bypass SSM entirely and establish a direct UDP connection between the laptop and CloudShell using NAT hole punching.

Phase 1: Discovering your public face with STUN

Neither side knows its own public IP address and port. Your laptop knows it's 192.168.1.42:12345, but the outside world sees something like 85.123.45.67:54321, whatever your router's NAT assigned. You can't figure this out by looking at your own network interfaces.

STUN (Session Traversal Utilities for NAT) solves this. It's a simple protocol where you send a UDP packet to a public STUN server and it replies telling you the source IP and port it observed.

  Your laptop (192.168.1.42:12345)
       │
       │  UDP packet ──────────►  stun.l.google.com:19302
       │                                │
       │                                │ "I see you as 85.123.45.67:54321"
       │                                │
       │  ◄── STUN Binding Response ────┘
       │
  Now you know: your public endpoint is 85.123.45.67:54321

Google runs free public STUN servers and the whole exchange is a single UDP round-trip. What is important: the UDP socket you used to contact the STUN server now has an open NAT mapping. Your router remembers "packets from 192.168.1.42:12345 go out as 85.123.45.67:54321" and that mapping is what we'll exploit next.

Both sides do this independently: the laptop discovers its public endpoint, and the agent running inside CloudShell discovers its own.

Phase 2: Exchanging endpoints through the SSM shell

The laptop knows its own public endpoint and the agent knows its own. But they need to know each other's, so we use the existing SSM shell session as the signaling channel.

The laptop already has a terminal session open to CloudShell (that's how we uploaded and started the agent). When the agent starts, it prints its public endpoint to stdout:

AGENT_READY:34.245.12.89:4433

The laptop reads this from the SSM session output. The laptop's public endpoint was passed as a command-line argument when starting the agent. Both sides now know where to punch.

Phase 3: The hole punch, simultaneous UDP packets

NAT mappings are bidirectional once established. When your laptop sends a UDP packet to 34.245.12.89:4433, your router creates a mapping that says "if a packet comes back from 34.245.12.89:4433 to my port 54321, forward it to 192.168.1.42:12345". The same thing happens on CloudShell's side.

The problem is timing. If the laptop sends first, the packet arrives at CloudShell's NAT before CloudShell has sent anything, so CloudShell's NAT drops it (no mapping exists yet). The solution: both sides send packets simultaneously. Both NATs create their mappings at roughly the same time, and within a few hundred milliseconds, packets start flowing in both directions.

  ┌──────────┐                                      ┌──────────┐
  │  Laptop  │                                      │CloudShell│
  │  NAT     │                                      │  NAT     │
  └────┬─────┘                                      └────┬─────┘
       │                                                 │
       │──── UDP "PUNCH" ───────────────────────────────►│ (dropped initially)
       │                                                 │
       │ (dropped initially) ◄──────────── UDP "PUNCH" ──│
       │                                                 │
       │──── UDP "PUNCH" ───────────────────────────────►│ ✓ NAT mapping exists!
       │                                                 │
       │ ✓ NAT mapping exists! ◄────────── UDP "PUNCH" ──│
       │                                                 │
       │◄═══════════════ UDP hole is open ══════════════►│

In practice, both sides send a burst of PUNCH packets for a few seconds. After a few hundred milliseconds, both NATs have seen outbound traffic and created their mappings. The hole is punched.

Phase 4: Local UDP relay

The OpenVPN server starts on port 1194 inside CloudShell. A local UDP relay binds 127.0.0.1:1194 on your laptop and bridges packets to the punched hole. On the CloudShell side, an identical relay bridges between the external socket and the local port 1194.

The OpenVPN choice

I considered several options for the VPN tunnel, including macOS native APIs and WireGuard, but both require either a signed Network Extension or root privileges to create tun interfaces and modify routes. I chose OpenVPN because the free OpenVPN Connect client handles all privileged operations internally: it already has the required entitlements to create the tunnel interface and manage routes, so my tool doesn't need root. It just generates a .ovpn profile and imports it.

Quick start

The easiest way to run is with the launcher script. It checks prerequisites, manages a virtualenv, and starts the VPN:

export AWS_PROFILE=my-profile
./run.sh

Et voilà! Your traffic is now routed through the AWS region of your choice.

Things to know

NAT compatibility - UDP hole punching works with most residential and cloud NATs. It fails with symmetric NATs, which are common in strict corporate firewalls. If your office blocks it, try from home or a mobile hotspot.
CloudShell quota - 200 hours per region per month per account, enough for occasional use. You can request an increase through Service Quotas.
Session timeout - CloudShell auto-terminates after 12 hours of continuous use. The tool sends heartbeats every 5 minutes to prevent idle sleep (20 min timeout), but the 12-hour hard limit still applies.
Data transfer cost - Outbound bytes are billed at standard EC2 rates (~$0.09/GB after the first 100 GB/month free tier).
macOS only - For now.

If you need a real VPN, pay for a real VPN. If you need a free, disposable, region-selectable tunnel for 30 minutes of testing or to bypass a broken CDN geo-block while travelling, this is genuinely useful.

The source is at github.com/guyon-it-consulting/cloudshell-vpn. Give it a try and let me know what you think.

Credits

Inspired by Dan V.'s cloudshell-store project that demonstrated the CloudShell API use
Brice Dauzats: Thanks for the reviews and improvements
Kiro CLI and Kiro Crew: Assisted me in research and validating hypotheses throughout the project 👻👻👻

— Jerome

AWS WAF Challenge: Blocking Bots Before They Reach the Application

Paul SANTUS — Fri, 14 Aug 2026 09:17:29 +0000

When I was called in, the attack had been going on for about a week.

It was targeting the login page of a legacy application that generated its HTML server-side.

Requests numbered in the millions and came from a very large number of IP addresses, which made IP-based blocking largely ineffective. Unlike other campaigns I had encountered, the attacker was also rotating JA3 and JA4 fingerprints. Even rate limiting aggregated on those signals could therefore provide only a partial response.

A few days later, a similar attack targeted a second application belonging to my client. This time, it was no longer the legacy server-side application, but a SPA (single-page application) calling a JSON API to create accounts.

These two incidents gave me the opportunity to use both integration modes offered by AWS WAF Challenge:

the challenge returned directly by WAF for an HTML page;
the challenge solved beforehand by challenge.js, then passed to an API called with fetch.

Why Put the Challenge at the Infrastructure Layer?

If my client called me after seven days, it was because the team had first tried to deal with the attack at the application layer by integrating Cloudflare Turnstile.

The integration was solid. A PrestaShop module managed the keys, global activation, and separate configurations for each tenant. On submission, the application retrieved the token from the form and validated it directly with Cloudflare:

$turnstileToken = Tools::getValue('cf-turnstile-response');

if (empty($turnstileToken)) {
    $turnstileValid = false;
} elseif (!$this->verifyTurnstileToken($turnstileToken)) {
    $turnstileValid = false;
}

Validation then required a server-side call to siteverify:

$response = Tools::file_get_contents(
    'https://challenges.cloudflare.com/turnstile/v0/siteverify',
    false,
    stream_context_create([
        'http' => [
            'method' => 'POST',
            'header' => 'Content-Type: application/x-www-form-urlencoded',
            'content' => http_build_query([
                'secret' => $secretKey,
                'response' => $token,
                'remoteip' => Tools::getRemoteAddr(),
            ]),
            'timeout' => 5,
        ],
    ])
);

This approach worked, but only partially: a significant proportion of the traffic still managed to obtain valid tokens. In particular, we observed tokens generated from a group of Dutch IP addresses and then presented to our infrastructure from Spain. The attacker had therefore industrialized token acquisition and circulation. A token proved that a challenge had been solved, but it was neither a persistent identity nor, in this flow, strictly bound to the IP address that obtained it.

More importantly, this protection had a major drawback: before rejecting a request, the entire chain had to be engaged: the CDN, load balancer, web server, PHP, framework, and our validation code. Even when authentication failed, a significant part of the technical cost had already been incurred. With millions of calls, this affected application performance, although autoscaling limited the impact, as well as the bill.

With a Challenge rule placed in AWS WAF, which the application was already using elsewhere, a request without a valid token is stopped at the edge. It consumes no PHP worker, no database connection, and no application compute capacity. That was exactly what we needed when facing several million requests.

Moving the control also brings a maintenance benefit. Any validation managed by the application necessarily involves code in the form, the controller, secret management, the remote call, and error messages. With a challenge managed by WAF, the backend is not even aware of its existence. It receives only requests that have already passed the check.

The provider of the application-level challenge, Cloudflare versus AWS, is not the point of this comparison. The same distinction would exist with any mechanism validated in the backend. What I want to highlight here, even though I admit I was tired of seeing the developer give in to the “obvious Cloudflare choice” because of its dominant position instead of calling me on the first day of the attack ^^, is the architectural difference between a control managed by the application and one placed at the edge, upstream from the origin.

First Implementation: An Application That Generates Its HTML Server-Side

On the legacy application, authentication uses a native HTML form submitted with POST.

This detail matters. When a user opens an HTML page, the HTTP response becomes a document that the browser must display. AWS WAF can therefore intercept this navigation and respond with its Challenge request. The browser executes the JavaScript provided by AWS WAF, performs the silent proof of work, obtains a token, and the interstitial script then transparently retries the backend request, which WAF now allows through.

In this case, implementing the challenge with AWS WAF is extremely simple.

I started by assigning a label to requests matching the login flow. The challenge rule takes just 15 lines of declarative code, including observability!

rule {
  name     = "ChallengeLoginNoToken"
  priority = 13

  action {
    challenge {}
  }

  statement {
    label_match_statement {
      key   = "login"
      scope = "LABEL"
    }
  }

  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "ChallengeLoginNoToken"
    sampled_requests_enabled   = true
  }
}

The login label was assigned by an earlier rule to POST requests targeting /login. This separation between detection and action makes it possible to reuse the same scope for other protections, such as rate limiting by IP address or JA4 fingerprint.

For a server-side site, there is nothing to add to the PHP code. There is no SDK to integrate and no token to validate in the application controller.

This is the simplest configuration: the browser requests a document, WAF can respond with an executable document, and the application is called only after the challenge has been solved.

Implementation 2: AWS WAF Challenge on a SPA

The second application used a SPA, built with React, for its interface and an API hosted on another subdomain.

Account creation conceptually looked like this:

await fetch("https://api.example.com/account-registration", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
  },
  body: JSON.stringify(payload),
});

If AWS WAF returns its challenge directly to this request, the response arrives inside fetch. The browser does not interpret it as a new document and does not automatically execute the JavaScript contained in the response. The SPA therefore receives a 202 status, potentially with a body it does not know how to process, and the registration flow fails.

This is a property of the browser execution model, not a limitation specific to AWS. JavaScript returned as data by an XHR or fetch request is not executed spontaneously.

The solution is to reverse the order of operations:

the SPA loads the AWS WAF challenge.js SDK;
the SDK silently solves the challenge within the page;
the SPA retrieves a token;
it attaches that token to the API request;
WAF verifies the token before allowing the request through.

Configuring WAF for a SPA

The Web ACL must first know the domains on which its tokens are valid. In my case, the frontend and API used two subdomains of the same root domain:

resource "aws_wafv2_web_acl" "acl_cloudfront" {
  provider = aws.us-east-1

  name  = "cloudfront-api-acl"
  scope = "CLOUDFRONT"

  token_domains = ["example.com"]

  default_action {
    allow {}
  }

  # ...
}

I then assign a label to POST requests targeting the sensitive endpoint and verify the token with a Challenge action:

rule {
  name     = "ChallengeRegister"
  priority = 15

  action {
    challenge {}
  }

  statement {
    label_match_statement {
      key   = "register"
      scope = "LABEL"
    }
  }

  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "ChallengeRegister"
    sampled_requests_enabled   = true
  }
}

Up to this point, the implementation is therefore the same as in V1. However, we need to send a valid token “on the first try,” since we cannot process the challenge synchronously.

The JavaScript SDK URL and secret key to inject into the SPA are provided in the integration section of the AWS WAF console. I inject them into the SPA using an environment variable.

A Small Implementation Detail That Can Become a Blocker

The first version of my loader simply waited for the script's load event. That was not enough. The initial jsapi.js script loads challenge.js in turn, then initializes window.AwsWafIntegration asynchronously.

The first script may therefore have finished loading while getToken() is not yet available. The symptom was particularly misleading: the first submission was sent without a token, which was obtained just a few milliseconds later.

I solved this problem with a bounded wait:

function waitForIntegration(timeoutMs = 3000): Promise<boolean> {
  return new Promise((resolve) => {
    const start = Date.now();

    const tick = () => {
      if (typeof window.AwsWafIntegration?.getToken === "function") {
        resolve(true);
        return;
      }

      if (Date.now() - start >= timeoutMs) {
        resolve(false);
        return;
      }

      window.setTimeout(tick, 50);
    };

    tick();
  });
}

export async function getWafToken(): Promise<string | undefined> {
  await injectWafScript();

  if (!(await waitForIntegration())) {
    return undefined;
  }

  return window.AwsWafIntegration!.getToken();
}

The token is then placed in the header specified by AWS:

export async function register(payload: RegisterPayload) {
  const wafToken = await getWafToken();

  return publicClient.post("/account-registration", payload, {
    headers: {
      "x-aws-waf-token": wafToken,
    },
  });
}

A Second Detail That Can Become a Blocker

In an architecture involving multiple subdomains, CORS must be taken into account, particularly the OPTIONS preflight request, since adding the token makes the request “complex.”

The header must be permitted by the response to the OPTIONS request:

Access-Control-Allow-Headers:
  Authorization,
  Content-Type,
  x-aws-waf-token

This therefore requires a small addition to the CloudFront configuration.

How Much Does This Protection Cost?

Based on the public AWS pricing available in August 2026, the AWS WAF base service is billed at:

$5 per Web ACL per month;
$1 per rule per month;
$0.60 per million requests processed, within the standard WCU allocation.

The Allow, Block, and Count actions do not add a per-action charge. Challenge responses are billed separately at $0.40 per million Challenge responses.

This additional cost remains low, especially if the rule is limited to a few high-value entry points: login, registration, password recovery, order validation, or a sensitive business operation.

The difference compared with the specialized Fraud Control rules is considerable. AWS WAF Account Takeover Prevention and Account Creation Fraud Prevention add a monthly subscription, followed by per-request charges with declining but still high rates. In AWS's public example, 15 million requests analyzed by ATP generate more than $8,000 in Fraud Control charges.

ATP provides much more advanced capabilities. It simply does not address the same economic need. When a targeted Challenge action, rate limiting, and a few network signals are enough, the cost-effectiveness is hard to beat.

In Summary: One Solution, Two Implementations

The first implementation takes 15 lines of Terraform because the browser navigates to an HTML document. AWS WAF can respond directly with the challenge JavaScript.

The second requires a small frontend service because the protected call is made with fetch. The challenge must be executed before the request, and its token must then be passed in a header or cookie.

In both cases, the backend implements no challenge validation. It stores no additional secret key, calls no external service, and consumes no resources for rejected requests.

AWS WAF Challenge is not a replacement for a complete anti-abuse strategy. It does, however, make it possible to move an expensive operation to the right place: before the application.

AWS WAF Challenge : bloquer les bots avant qu’ils n’atteignent l’application

Paul SANTUS — Fri, 14 Aug 2026 09:10:35 +0000

Quand on m’a appelé, l’attaque durait depuis environ une semaine.

Elle visait la page de connexion d’une application historique qui générait son HTML côté serveur.

Les requêtes se comptaient en millions et provenaient d’un très grand nombre d’adresses IP, ce qui rendait un blocage par IP peu efficace. Contrairement à d’autres campagnes que j’avais rencontrées, l’assaillant faisait également tourner ses empreintes JA3 et JA4. Même un rate limiting agrégé sur ces signaux ne pouvait donc apporter qu’une réponse partielle.

Quelques jours plus tard, une attaque similaire a visé une deuxième application chez mon client. Cette fois, il ne s’agissait plus de l’ancienne application server-side, mais d’une SPA (single page app) qui appelait une API JSON pour créer les comptes.

Ces deux incidents m’ont permis d’utiliser les deux modes d’intégration de la fonctionnalité Challenge d’AWS WAF :

le challenge directement renvoyé par le WAF pour une page HTML ;
le challenge résolu en amont par challenge.js, puis transmis à une API appelée avec fetch.

Pourquoi placer le challenge côté infrastructure ?

Si mon client m'a appelé au bout de 7 jours, c'est que l'équipe a d'abord essayé de traiter l'attaque au niveau applicatif, via l'intégration de Cloudflare Turnstile.

L’intégration était sérieuse. Un module PrestaShop gérait les clés, l’activation globale et des configurations distinctes selon le tenant. Lors de la soumission, l’application récupérait le jeton du formulaire et le validait elle-même auprès de Cloudflare :

$turnstileToken = Tools::getValue('cf-turnstile-response');

if (empty($turnstileToken)) {
    $turnstileValid = false;
} elseif (!$this->verifyTurnstileToken($turnstileToken)) {
    $turnstileValid = false;
}

La validation nécessitait ensuite un appel serveur vers siteverify :

$response = Tools::file_get_contents(
    'https://challenges.cloudflare.com/turnstile/v0/siteverify',
    false,
    stream_context_create([
        'http' => [
            'method' => 'POST',
            'header' => 'Content-Type: application/x-www-form-urlencoded',
            'content' => http_build_query([
                'secret' => $secretKey,
                'response' => $token,
                'remoteip' => Tools::getRemoteAddr(),
            ]),
            'timeout' => 5,
        ],
    ])
);

Cette approche fonctionnait, mais seulement partiellement : une proportion significative du trafic obtenait malgré tout des jetons valides. Nous avons notamment observé des jetons générés depuis un groupe d’adresses IP néerlandaises, puis présentés à notre infrastructure depuis l'Espagne : l’attaquant avait donc industrialisé l’acquisition et la circulation des jetons. Le jeton prouvait qu’un challenge avait été résolu, mais il ne constituait ni une identité durable ni, dans ce flux, une liaison stricte avec l’adresse IP qui l’avait obtenu.

Mais surtout, cette protection avait un défaut important : avant de rejeter une requête, il fallait mobiliser tout la chaine : le CDN, le load balancer, le serveur web, PHP, le framework, et notre code de validation : même quand l’authentification échouait, une partie significative du coût technique avait déjà été payée. Et avec des millions d'appels, ça avait un impact sur les performances de l'application (impact limité par l'autoscaling) et sur la facture.

Avec une règle Challenge placée dans AWS WAF (qui était déjà utilisé par ailleurs par l'application), la requête sans jeton valide est arrêtée à la périphérie. Elle ne consomme ni worker PHP, ni connexion à la base de données, ni capacité de calcul applicative. C’était exactement ce dont nous avions besoin face à plusieurs millions de requêtes.

Il y a aussi un bénéfice de maintenance propre au déplacement du contrôle. Toute validation gérée par l’application implique nécessairement du code dans le formulaire, le contrôleur, la gestion des secrets, l’appel distant et les messages d’erreur. Avec un challenge géré par le WAF, le backend ne connaît même pas son existence. Il ne reçoit que les requêtes qui ont déjà passé le contrôle.

Le fournisseur du challenge applicatif (Cloudflare vs AWS) n’est pas le sujet de cette comparaison. La même distinction existerait avec n’importe quel mécanisme validé dans le backend. Ce que je veux relever ici (même si j'avoue avoir été lassé que le dev succombe à l'« évidence Cloudflare » liée à sa position dominante, plutôt que de m'appeler au premier de l'attaque ^^), c'est la différence architecturale entre un contrôle géré par l’application et un contrôle placé à la périphérie, en amont de l’origine.

Première implémentation : une application qui génère son HTML côté serveur

Sur l’application historique, l'authentification se fait par un formulaire HTML natif en POST.

Ce détail est important. Lorsqu’un utilisateur ouvre une page HTML, la réponse HTTP devient un document que le navigateur doit afficher. AWS WAF peut donc intercepter cette navigation et envoyer en réponse sa demande de Challenge. Le navigateur exécute le JavaScript fourni par AWS WAF, réalise le proof of work silencieux, obtient un jeton, puis reprend le script interstitiel relance de façon transparente l'appel au backend, appel que WAF laisse donc passer.

Dans ce cas-là, l'implémentation du challenge par AWS WAF est ultra simple.

J’ai commencé par poser un label sur les requêtes qui correspondaient au parcours de connexion. La règle de challenge tient en 15 lignes de code déclaratif, observabilité comprise !

rule {
  name     = "ChallengeLoginNoToken"
  priority = 13

  action {
    challenge {}
  }

  statement {
    label_match_statement {
      key   = "login"
      scope = "LABEL"
    }
  }

  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "ChallengeLoginNoToken"
    sampled_requests_enabled   = true
  }
}

Le label login était attribué par une règle précédente aux requêtes POST portant sur /login (cette séparation entre détection et action permet de réutiliser le même périmètre pour d’autres protections, par exemple une rate limit par IP ou par empreinte JA4).

Dans le cas d’un site server-side, il n’y a rien à ajouter dans le code PHP. Il n’y a pas de SDK à intégrer, pas de jeton à valider dans le contrôleur applicatif.

C’est la configuration la plus simple : le navigateur demande un document, le WAF peut répondre par un document exécutable, et l’application n’est appelée qu’après résolution du challenge.

Implémentation 2 : Challenge AWS WAF sur une SPA

La deuxième application utilisait une SPA (appli React) pour son interface et une API sur un autre sous-domaine.

La création de compte ressemble conceptuellement à ceci :

await fetch("https://api.example.com/account-registration", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
  },
  body: JSON.stringify(payload),
});

Si AWS WAF renvoie directement son challenge à cette requête, la réponse arrive dans fetch. Le navigateur ne l’interprète pas comme un nouveau document et n’exécute pas automatiquement le JavaScript contenu dans la réponse. La SPA reçoit donc un statut 202, éventuellement avec un corps qu’elle ne sait pas traiter, et le parcours d’inscription échoue.

C’est une propriété du modèle d’exécution du navigateur, pas une limitation spécifique à AWS. Du JavaScript renvoyé comme donnée par une requête XHR ou fetch n’est pas exécuté spontanément.

La solution consiste à inverser l’ordre des opérations :

la SPA charge le SDK challenge.js d’AWS WAF ;
le SDK résout silencieusement le challenge dans la page ;
la SPA récupère un jeton ;
elle joint ce jeton à la requête API ;
le WAF vérifie le jeton avant de laisser passer la requête.

Configurer le WAF pour une SPA

La Web ACL doit d’abord connaître les domaines sur lesquels ses jetons sont valables. Dans mon cas, le frontend et l’API utilisaient deux sous-domaines du même domaine racine :

resource "aws_wafv2_web_acl" "acl_cloudfront" {
  provider = aws.us-east-1

  name  = "cloudfront-api-acl"
  scope = "CLOUDFRONT"

  token_domains = ["example.com"]

  default_action {
    allow {}
  }

  # ...
}

Je pose ensuite un label sur les requêtes POST vers l’endpoint sensible, puis je vérifie le jeton avec une action Challenge :

rule {
  name     = "ChallengeRegister"
  priority = 15

  action {
    challenge {}
  }

  statement {
    label_match_statement {
      key   = "register"
      scope = "LABEL"
    }
  }

  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "ChallengeRegister"
    sampled_requests_enabled   = true
  }
}

On retrouve donc jusqu'ici la même implémentation qu'en v1.. par contre, il nous faut envoyer un token valide "du premier coup" vu qu'on ne saurait pas traiter le challenge de façon synchrone.

L’URL du SDK js et la secret key à injecter dans notre SPA sont fournies dans la section d’intégration de la console AWS WAF. Je l’injecte dans la SPA à partir d’une variable d’environnement.

Un petit détail d'implémentation qui peut être bloquant

La première version de mon chargeur attendait simplement l’événement load du script. Ce n’était pas suffisant. Le script initial jsapi.js charge à son tour challenge.js, puis initialise window.AwsWafIntegration de manière asynchrone.

Le chargement du premier script peut donc être terminé alors que getToken() n’est pas encore disponible. Le symptôme était particulièrement trompeur : le premier submit partait sans jeton, celui-ci étant obtenu quelques milli-secondes plus tard.

J’ai résolu ce problème avec une attente bornée :

function waitForIntegration(timeoutMs = 3000): Promise<boolean> {
  return new Promise((resolve) => {
    const start = Date.now();

    const tick = () => {
      if (typeof window.AwsWafIntegration?.getToken === "function") {
        resolve(true);
        return;
      }

      if (Date.now() - start >= timeoutMs) {
        resolve(false);
        return;
      }

      window.setTimeout(tick, 50);
    };

    tick();
  });
}

export async function getWafToken(): Promise<string | undefined> {
  await injectWafScript();

  if (!(await waitForIntegration())) {
    return undefined;
  }

  return window.AwsWafIntegration!.getToken();
}

Le jeton est ensuite placé dans le header prévu par AWS :

export async function register(payload: RegisterPayload) {
  const wafToken = await getWafToken();

  return publicClient.post("/account-registration", payload, {
    headers: {
      "x-aws-waf-token": wafToken,
    },
  });
}

Deuxième détail qui peut bloquer

Dans une architecture avec plusieurs sous-domaines, il faut penser au CORS (et notamment au preflight OPTIONS puisque l'ajout du token rend la requête 'complexe').

Le header doit être autorisé par la réponse à la requête OPTIONS :

Access-Control-Allow-Headers:
  Authorization,
  Content-Type,
  x-aws-waf-token

Un petit ajout à faire au niveau de la configuration Cloudfront, donc.

Combien coûte cette protection ?

Aux tarifs publics AWS consultés en août 2026, le socle AWS WAF est facturé :

5 dollars par Web ACL et par mois ;
1 dollar par règle et par mois ;
0,60 dollar par million de requêtes traitées, dans l’allocation WCU standard.

Les actions Allow, Block et Count n’ajoutent pas de coût par action. Les réponses Challenge sont facturées séparément, à hauteur de 0,40 dollar par million de réponses Challenge.

Ce coût additionnel reste faible, surtout si la règle est limitée à quelques points d’entrée à forte valeur : connexion, inscription, récupération de mot de passe, validation d’une commande ou opération métier sensible.

La différence avec les règles spécialisées Fraud Control est considérable. AWS WAF Account Takeover Prevention et Account Creation Fraud Prevention ajoutent un abonnement mensuel, puis une facturation par requête analysée avec des tarifs dégressifs mais élevés. Dans l’exemple public d’AWS, 15 millions de requêtes analysées par ATP produisent plus de 8 000 dollars de frais Fraud Control.

ATP apporte des capacités beaucoup plus avancées. Il ne répond simplement pas au même besoin économique. Quand une action Challenge ciblée, un rate limiting et quelques signaux réseau suffisent, le rapport efficacité-prix est difficile à battre.

En résumé : une même solution avec deux implémentations

La première implémentation tient dans 15 liines Terraform parce que le navigateur navigue vers un document HTML. AWS WAF peut lui répondre directement avec le JavaScript du challenge.

La seconde demande un petit service frontend parce que l’appel protégé est réalisé avec fetch. Le challenge doit être exécuté avant la requête, puis son jeton doit être transmis dans un header ou un cookie.

Dans les deux cas, le backend n’implémente aucune validation de challenge. Il ne stocke pas de clé secrète supplémentaire, n’appelle pas un service externe et ne consomme pas de ressources pour les requêtes rejetées.

La fonctionnalité AWS WAF Challenge ne remplace pas une stratégie anti-abus complète. Elle permet en revanche de déplacer une opération coûteuse au bon endroit : avant l’application.

Engineering a Multi-Agent AI Platform — Part 1: Adaptive Depth

MUHAMMAD ABIODUN SULAIMAN — Thu, 13 Aug 2026 19:32:27 +0000

This is Part 1 of a series on engineering AI systems that learn in production. This part covers the classification and strategy dispatch layer.

The problem nobody talks about

Here's a question that keeps coming up in production AI systems: should your model think before it answers?

Not philosophically. Mechanically. When a user asks "What's my account balance?", firing up a chain-of-thought reasoning engine with 16,000 thinking tokens is like hiring a PhD mathematician to count change. You burn compute, you add latency, and the answer isn't any better for it. But when someone asks "Compare the ROI of three deployment architectures under peak load, factoring in failure modes and regional latency differences," a zero-reasoning direct call produces garbage.

Most platforms pick one approach and apply it everywhere. The expensive ones over-reason on simple queries, bleeding money on thinking tokens that produce nothing useful. The cheap ones under-reason on hard problems, producing responses that sound confident but miss the point entirely.

Consider a platform where dozens of autonomous agents serve thousands of tenants across different industries. A scheduling agent handles "move my 3pm to Thursday" alongside an analytics agent wrestling with "why did churn spike in Q3 among enterprise accounts." These requests hit the same infrastructure, but they need radically different cognitive investment.

The system we're building doesn't pick a single depth. It scores each request and routes it to the appropriate reasoning tier. And then, as we'll see in Parts 2 and 3, it learns from outcomes and human feedback to get better at that scoring over time.

Three tiers of thinking

Think of reasoning depth as a spectrum with three practical zones.

Tier 1: No intermediate reasoning (NONE)

The model receives the prompt and generates a response directly. No thinking tokens, no structured intermediate steps. Input goes in, output comes out. This is your workhorse for factual lookups, simple status checks, greetings, and FAQ-style questions. Latency sits around 200ms. Cost is minimal.

When does this fail? When the question requires the model to hold multiple constraints in working memory simultaneously, or when the answer depends on comparing several options against each other. The model will produce something plausible-sounding, but it hasn't actually reasoned through the tradeoffs.

Tier 2: Lightweight structured reasoning (DRAFT)

Here, the system injects a structured instruction into the prompt: "Before answering, write your reasoning inside <draft> tags, limited to 512 tokens. Then provide your final answer inside <answer> tags."

The model gets a small scratchpad. It can organize its thoughts, check a few constraints, maybe outline a comparison. But the budget is tight. 512 tokens of reasoning is enough to decompose a moderately complex question into sub-parts and address each one, but not enough to explore rabbit holes or consider five alternative framings.

The system parses the response, extracts what's between the <answer> tags, discards the draft. The user never sees the intermediate reasoning unless the operator wants to inspect it for debugging.

Tier 3: Full chain-of-thought (FULL_COT)

This tier hands the problem to a model with native thinking-token support. Models like o3 or DeepSeek-R1 have a separate "thinking" channel where they can reason at length before producing a visible response. The system sets a reasoning_effort parameter and allocates a thinking budget (more on budgets in a moment).

The model can now spend thousands of tokens working through a problem. It can consider alternatives, backtrack when it hits a dead end, verify its own logic. For genuinely hard analytical questions, this produces qualitatively different answers than the other two tiers.

The tradeoff is obvious: latency climbs to 2-5 seconds and token costs multiply. You don't want this running on "What time do you close?"

Scoring complexity

So how does the system decide which tier to use? It runs the incoming request through a multi-signal classifier that produces a single score between 0 and 1.

Five signals feed the score:

Task type (weight: 0.30). Is this a factual lookup? A comparison? An analysis? A debugging request? A creative task? The classifier maintains a taxonomy of task types with associated difficulty priors. A status check scores low. A multi-criteria comparison scores high.

Ambiguity (weight: 0.20). How many ways could this request be interpreted? "Tell me about performance" is ambiguous (performance of what? the system? the team? a specific metric?). "What was the p95 latency for the /checkout endpoint last Tuesday" is not. Ambiguity markers include vague pronouns, missing context, and underspecified scope.

Domain vocabulary density (weight: 0.20). Requests loaded with specialized terminology signal that the answer needs domain expertise and careful reasoning. A message full of financial modeling terms or medical terminology likely needs deeper thinking than a conversational greeting.

Lexical complexity (weight: 0.15). Sentence length, subordinate clause depth, vocabulary sophistication. A proxy for how much cognitive load the request itself carries. Not a perfect signal (a short request can demand complex reasoning), but a useful one in combination with the others.

Context dependency (weight: 0.15). How much does the correct answer depend on prior conversation history, tenant-specific data, or cross-referencing multiple information sources? A standalone question scores lower than one that requires synthesizing three previous messages and two documents.

The composite score:

C(x) = 0.30 · S_task(x) + 0.20 · S_amb(x) + 0.20 · S_dom(x) + 0.15 · S_lex(x) + 0.15 · S_ctx(x)

Each signal function S(x) outputs a value in [0, 1]. The weighted sum produces the final complexity score C(x), also in [0, 1].

The routing thresholds:

C(x) < 0.30 → NONE (direct call, no reasoning)
0.30 ≤ C(x) < 0.65 → DRAFT (lightweight scratchpad, 512-token cap)
C(x) ≥ 0.65 → FULL_COT (native thinking tokens with budget)

A concrete example makes this tangible. "What is my account balance?" scores roughly: task_type = 0.1 (factual lookup), ambiguity = 0.1 (clear intent), domain = 0.05 (no specialized vocabulary), lexical = 0.1 (short, simple), context = 0.2 (needs account data but straightforward). Composite: 0.30(0.1) + 0.20(0.1) + 0.20(0.05) + 0.15(0.1) + 0.15(0.2) = 0.03 + 0.02 + 0.01 + 0.015 + 0.03 = 0.105. NONE tier. Just answer it.

Now try "Compare the ROI of three deployment architectures considering latency, cost, and failure modes under peak load." Task_type = 0.9 (multi-criteria analysis), ambiguity = 0.4 (what counts as "ROI" needs interpretation), domain = 0.8 (infrastructure terminology), lexical = 0.7 (complex sentence, multiple constraints), context = 0.6 (needs architecture details from prior context). Composite: 0.30(0.9) + 0.20(0.4) + 0.20(0.8) + 0.15(0.7) + 0.15(0.6) = 0.27 + 0.08 + 0.16 + 0.105 + 0.09 = 0.705. FULL_COT. This one needs to think.

Why this matters in multi-agent systems

A single-agent chatbot can get away with a fixed reasoning depth. But multi-agent platforms have a topology that makes static approaches break down.

Picture a supervisor agent that receives every inbound message. It classifies intent and routes to specialist agents: a scheduling agent, a billing agent, a technical support agent, an analytics agent. Each specialist handles a different slice of the problem space.

Here's what's interesting: the same specialist agent might need different reasoning depths depending on the specific request. The billing agent handles "what's my next payment date?" (NONE) alongside "explain why my invoice increased 40% this month considering the three pricing tiers, the mid-cycle plan change, and the prorated adjustments" (FULL_COT).

The reasoning strategy is selected per-request, not per-agent. The billing agent doesn't have a fixed reasoning depth. It gets classified fresh on every interaction based on what the user actually asked.

It gets more layered when you have agent teams. A lead agent might decompose a complex request into subtasks, delegating each to a different specialist. One subtask might be simple data retrieval (NONE), another might require cross-referencing multiple sources (DRAFT), and a third might involve genuine analytical reasoning (FULL_COT). The system scores each subtask independently.

This per-request granularity is what makes the whole thing tractable economically. If you set every agent to FULL_COT because some of their tasks are hard, you're paying ten times more for the easy ones. If you set them all to NONE because most tasks are easy, the hard ones produce bad answers and you lose user trust. Scoring at the request level means you pay for reasoning exactly where it creates value.

The thinking budget

Even within FULL_COT, not every hard problem deserves the maximum thinking allocation. A moderately complex question might need 4,096 thinking tokens to work through. A genuinely difficult analytical task might need 16,384. Allocating 32,000 thinking tokens to a question that only needs 4,000 wastes compute and sometimes degrades quality (the model can over-think, going in circles or second-guessing correct intermediate steps).

The system maps complexity sub-ranges to budget tiers:

Simple (below threshold): 0 tokens (NONE/DRAFT handle it)
Moderate (0.65-0.80): 4,096 tokens
Complex (0.80-1.0): 16,384 tokens

Budget enforcement works as a state machine. When the model starts thinking, the state is ACTIVE. If the model reaches the budget limit, the state transitions to EXHAUSTED, and the system signals the model to wrap up its reasoning. A grace window of 512 tokens allows the model to reach a natural stopping point rather than being cut off mid-thought. After the grace window, the state transitions to TERMINATED and the model must produce its final answer from whatever reasoning it completed.

This prevents runaway thinking. Without budget enforcement, a model given native thinking tokens can sometimes spend 20,000+ tokens reasoning in circles on a problem that doesn't benefit from it. The budget creates a ceiling without preventing deep reasoning when it's needed.

The diagram

Here's the full request lifecycle through strategy selection:

A request enters the complexity classifier, which evaluates five signal dimensions and produces a weighted score. The score crosses one of two thresholds (0.30 and 0.65), routing the request into one of three strategy lanes. Each lane uses a different mechanical approach to produce the final response: a simple pass-through call, a structured draft envelope with tag parsing, or a full thinking-token API with budget enforcement. All three converge on a response delivered to the user.

What's missing

The classifier works. It routes most requests to reasonable tiers. But it's a heuristic, and heuristics don't learn.

If the system consistently under-reasons on a particular type of request for a specific agent, the classifier keeps making the same mistake. If tenant patterns shift (maybe a business changes industries and their support queries become more technical), the static weights don't adapt.

What we actually want is a system that observes outcomes. Did the DRAFT strategy produce a good result for this context, or did it fall short? Was the FULL_COT response worth the extra latency and cost, or would DRAFT have been equally good?

That's the reinforcement learning router, and it's the subject of Part 2. The core idea: maintain a probabilistic belief about how well each (model, strategy) pair performs in each context, sample from that belief to select an action, observe the outcome, and update the belief. A contextual bandit that learns the terrain.

The classifier gives us a reasonable starting position. The bandit learns to do better.

Next in the series: Part 2 explores how Thompson Sampling learns optimal model+strategy routing from production outcomes.

You can follow me on LinkedIn and Twitter for more updates.

Running Gemma 4 on EC2 G5g: Graviton2 AMD with NVIDIA GPU

xbill — Thu, 13 Aug 2026 18:45:41 +0000

A field report on serving Google's Gemma 4 E2B on AWS EC2 **G5g* — a Graviton2 (aarch64)
host with an NVIDIA T4G (Turing, SM 7.5) GPU. Three obstacles: an arch list nobody
publishes for this combination, a version floor that only the newest vLLM clears, and
64 KiB of shared memory that stops the model dead. Plus the seven things I documented
wrong before I had a box.*


Model	`google/gemma-4-E2B-it` (reference bf16 release)
Hardware	AWS EC2 `g5g.4xlarge` — Graviton2 + 1x NVIDIA T4G, compute capability 7.5, 15,360 MiB
Base image	Deep Learning ARM64 AMI OSS Nvidia Driver GPU PyTorch 2.12 (Ubuntu 24.04)
Software	torch 2.12.0+cu132 · CUDA 13.2 · vLLM v0.27.2rc0 built from source for `sm_75`
Result	43.1 tok/s single-stream greedy, 329,579-token KV cache — after one patch to vLLM

G5g is the only instance AWS has ever shipped that puts an NVIDIA GPU behind a Graviton
host. It launched in 2020, it never got a successor, and Graviton is now on its fifth
generation without one.

That matters more than it sounds. The Arm-plus-CUDA world moved on to NVIDIA's own Arm CPU
— Grace, paired with SM 9.0 and 10.0 parts. Turing stayed well supported, on x86. G5g is
the only hardware that is aarch64 and compute capability 7.5, and almost nobody publishes
a build for that combination.

I put a rig on one anyway. The packaging problem was the quick part. Everything after it
— a compiler that was not there, a version floor I did not expect, and 32 KiB of shared
memory — took far longer, because none of it fails where you are looking.

No published build covers aarch64 and SM 7.5 together

Start with the obvious candidate. vllm/vllm-openai:v0.27.1 publishes both platforms under
one tag, and you can read the arch lists straight out of the image config without pulling a
layer:

docker buildx imagetools inspect vllm/vllm-openai:v0.27.1 --format '{{json .Image}}'

linux/amd64   7.5 8.0 8.6 8.9 9.0 10.0 12.0
linux/arm64       8.0 8.7 8.9 9.0 10.0 11.0 12.0

The one architecture this hardware needs is the only entry the two images disagree on. The
arm64 list is Ampere and up, because that is what ships as an Arm-plus-NVIDIA system: A100,
Jetson Orin, GH200, Blackwell. Turing is not on that list and never will be.

Normally a missing target degrades to JIT from embedded PTX. Not here. The Dockerfile says
so, with a comment:

# Do not add +PTX here: vLLM filters torch's top-level PTX flag when it
# converts global gencode flags into per-kernel arch lists.

So it does not run slowly. It fails outright, with no kernel image is available for execution on the device.

The rest of the ecosystem splits the same way. Check before you plan anything:

Artifact	7.5 on arm64	State
`vllm/vllm-openai` arm64	no	Current. Never had it.
`nvcr.io/nvidia/pytorch` arm64	through 24.10	Dropped by 24.12.
`drikster80/vllm-aarch64`	yes	Abandoned Sept 2024. vLLM 0.6.1, far too old for Gemma 4.
PyPI torch aarch64	no	Built for 9.0 / 10.0 / 12.0.
AWS ARM64 GPU DLAMI	yes	Maintained. PyTorch 2.2 through 2.12.

AWS ships the one PyTorch that still has Turing

This is the finding that saves the whole exercise, and I nearly wrote it off. I had assumed
PyTorch's aarch64 CUDA wheels lacked sm_75 and that a from-source PyTorch build was
coming. That is true of the PyPI wheels. It is not true of AWS.

Read on two different DLAMIs, on the box:

torch 2.7.0+cu128    ['sm_75', 'sm_90', 'sm_100', 'sm_120']
torch 2.12.0+cu132   ['sm_75', 'sm_80', 'sm_90', 'sm_100', 'sm_110', 'sm_120']

AWS sells G5g, so AWS keeps Turing in the build — right through PyTorch 2.12 on CUDA 13.2,
an image cut three months ago. PyTorch never needs building. Only vLLM's own kernels do,
and CMake takes the arch list without argument:

-- CUDA target architectures: 7.5
CMake Warning: Pytorch version 2.11.0 expected for CUDA build, saw 2.12.0 instead.

That warning is worth reading twice, and I come back to it below.

The PyTorch DLAMI has no compiler

Two things the DLAMI does not give you, neither of them documented anywhere I could find.

There is no nvcc. The image ships the driver and a torch built against CUDA, not the
toolkit. You need the keyring and cuda-toolkit-13-2 from NVIDIA's sbsa repo — not the
x86 one, which is an easy reflex to get wrong on an Arm box.

And vLLM now wants Rust. Its vllm-rs frontend needs setuptools_rust plus a toolchain,
and the failure is a bare ModuleNotFoundError: No module named 'setuptools_rust' thrown
from metadata generation, several minutes in.

The newest vLLM was the only one that worked

No vLLM tag pins torch 2.12. They go 2.11, then jump to 2.13. I reasoned that building older
code against a newer runtime was the safer direction, took v0.26.0, and spent an hour being
wrong about it.

It builds fine. It then dies on model load:

transformers.integrations.heterogeneity.configuration_utils.AmbiguousGlobalPerLayerAttributeError:
'head_dim' is a per-layer attribute and may vary across layers.

Gemma 4's head_dim is not one number, and current transformers refuses to hand out a
global value for it. vLLM's config converter was still doing a flat
getattr(config, "head_dim", 0). The per_layer_config handling that copes with it landed
in v0.27.2rc0 — not v0.27.1, which I also checked. The newest tag was the only one that
worked.

If you take one process lesson from this: reach for the latest release first, and make the
constraint say out loud what stopped you when you fall back.

Gemma 4's attention heads are not one size

With the build working the server still would not start, and this failure has nothing to do
with Arm or packaging. It is this model against this chip.

Gemma4 model has heterogeneous head dimensions
{'sliding_attention': 256, 'full_attention': 512}.
FA4 not available, forcing TRITON_ATTN backend.

Read that as a chain, because every link is load-bearing:

Gemma 4's sliding layers are 256 wide. Its global layers are 512.
Only FA4 or Triton support heterogeneous head dims at all.
FA4 is not available, so vLLM forces TRITON_ATTN.
That choice is not yours to make. VLLM_ATTENTION_BACKEND is not a recognised variable in v0.27 — it logs Unknown vLLM environment variable detected and carries on. I set it twice before I read the warning.
Triton's unified attention kernel at head_size=512 wants about 96 KiB of shared memory per block.

64 KiB is the whole problem

Turing's shared memory is two numbers, and both are real. The default static limit per block
is 48 KiB — that is what torch.cuda.get_device_properties().shared_memory_per_block reports,
49,152 bytes. A kernel that needs more has to opt in through the dynamic shared-memory
attribute, and even then it tops out at 64 KiB. Ampere and later have 164 KiB and up.

Triton opts in, so it is measuring against the 64 KiB ceiling. It still does not fit:

triton.runtime.errors.OutOfResources: out of resource: shared memory,
Required: 98304, Hardware limit: 65536

Refused outright. Not slow, not degraded — the kernel will not launch, and it takes the
engine down during CUDA graph capture, which is late enough that you have already watched
the weights load and the KV cache get sized.

The fix is small. Shrink the KV tile until the query block and the K/V tiles fit inside the
budget, and drop the software pipeline to one stage. Gate it on pre-Ampere so it is a no-op
on every other card:

if current_platform.get_device_capability()[0] < 8:
    _smem_budget = 60000
    _esz = q.element_size()
    def _fits(t): return (BLOCK_M + 2 * t) * head_size * _esz <= _smem_budget
    while TILE_SIZE_PREFILL > 16 and not _fits(TILE_SIZE_PREFILL): TILE_SIZE_PREFILL //= 2
    while TILE_SIZE_DECODE  > 16 and not _fits(TILE_SIZE_DECODE):  TILE_SIZE_DECODE  //= 2
    launch_num_stages = 1

With that in vllm/v1/attention/ops/triton_unified_attention.py, graphs capture, the engine
comes up in 76 seconds, and the model serves. This is not upstream. It lives on my
instance and has to be reapplied on any vLLM upgrade, which makes it the obvious thing to
send back.

Most of the build is kernels that can never load

67 minutes on a g5g.4xlarge at MAX_JOBS=12, and the majority of it is FlashAttention.
vLLM compiles FA2 and FA3 regardless of TORCH_CUDA_ARCH_LIST — I watched it grind
through hundreds of sm90 Hopper instantiations on a build targeting 7.5 only. FA2 needs
sm80, FA3 needs sm90. Neither can ever load on this card.

Constraining VLLM_FA_CMAKE_GPU_ARCHES should cut that dramatically. I did not try it,
because by the time I understood what I was looking at the build was 45 minutes in and
interrupting it would have cost more than finishing.

What I got wrong before I had hardware

I wrote the rig's documentation before provisioning anything. Seven claims in it were wrong,
and every correction came off the machine rather than out of an argument. This is the part I
would keep if I kept nothing else.

What I wrote	What the box said
PyTorch aarch64 lacks `sm_75`	AWS DLAMI has it, on both versions I checked
bfloat16 is a hard failure here	Torch upconverts; vLLM logs `Casting torch.bfloat16 to torch.float16` and proceeds
The backend is XFORMERS	`TRITON_ATTN`, forced, not selectable
`VLLM_ATTENTION_BACKEND` picks it	Not a recognised variable. I had shipped dead config.
w4a16 needs sm80+ Marlin	The build compiled `sm75_kernel_float16_u4b8_float16.cu.o`
The GPU has 16 GB	15,360 MiB
`/v1/completions` returns an empty body	It returns `': ok: ok: ok: ok'` — garbage, not silence

That last one has teeth. If you health-check by testing for an empty response, this endpoint
passes while producing nonsense. Use /v1/chat/completions and read the text.

One claim is still standing only because I never tested it: whether g5g.xlarge's 8 GiB of
host RAM can stage 9.5 GiB of weights. Safetensors loading is mmap-backed, so I suspect it
can. It is labelled untested rather than stated as fact, which is where it should have been
all along.

What it does once it runs

content: 'Site Reliability Engineering (SRE) is a discipline that applies
          software engineering principles to infrastructure and operations
          problems to create highly reliable, scalable, and efficient systems.'
finish_reason: stop      usage: 19 prompt / 32 completion / 51 total

Measure	Value
Throughput, single stream greedy	42.9 tok/s @ 64, 43.1 @ 256
KV cache	2.95 GiB, 329,579 tokens
Concurrency at 16k context	20.12x
GPU memory while serving	13,501 / 15,360 MiB
Engine init	76.4 s, graph capture 17 s
Memory bandwidth, measured	277.0 GB/s read · 234.3 GB/s copy (320.1 theoretical)

Before reading too much into 43 tok/s, note what the memory does. The T4G has GDDR6, not
HBM — 256-bit bus at 5,001 MHz, so 320 GB/s theoretical. I measured 277 GB/s on a
streaming read (87% of peak) and 234 GB/s on a read-modify-write. Decode is bandwidth-bound,
so 277 is the real ceiling. For scale, a TPU v5e is about 859 GB/s normalized and a v6e about
1,638 — this part has roughly a third of one and a sixth of the other. It is a bandwidth-limited
card behaving like a bandwidth-limited card.

Single run, single stream, no repeats and no variance figure. One sample per cell, and taken
with the clamped tiles, so it is a floor rather than a characterisation. My Inferentia port
measured about 44 tok/s for E2B on one core, which is the same neighbourhood — but that is a
different harness on different silicon and I would not put the two in one table.

Troubleshooting quick reference

Symptom	Cause
`no kernel image is available`	Stock arm64 image. No 7.5, no PTX. Build from source.
`OutOfResources: shared memory`	Turing's 64 KiB against a 512-wide head. Clamp the tiles.
`AmbiguousGlobalPerLayerAttributeError`	vLLM older than v0.27.2rc0.
`No module named 'setuptools_rust'`	Missing Rust toolchain for `vllm-rs`.
`nvcc: not found`	PyTorch DLAMI has no toolkit. Install `cuda-toolkit-13-2` (sbsa).
`Unknown vLLM environment variable`	You set `VLLM_ATTENTION_BACKEND`. It does nothing.
Healthy endpoint, nonsense output	You checked `/v1/completions`. Use chat completions.

The short version

Take the AWS ARM64 GPU PyTorch DLAMI — it is the only maintained aarch64 stack that still
carries sm_75. Add cuda-toolkit-13-2 from the sbsa repo and a Rust toolchain, because the
image ships neither. Build vLLM v0.27.2rc0 or newer from source with
TORCH_CUDA_ARCH_LIST=7.5 and use_existing_torch.py, and patch the Triton attention kernel
to fit Turing's shared memory before you try to start it. Serve with --dtype float16 and
--kv-cache-dtype auto.

Nothing here failed loudly, and nothing failed where I was looking. The packaging gap I built
the rig around was already solved by AWS; the thing that actually stopped me was 32 KiB of
shared memory and a model whose global attention heads are twice as wide as its sliding ones.
Hardware this far off the mainstream will keep producing that shape of surprise — the fix is
not to reason harder about it, but to get to a box sooner and let it tell you.

Measured on EC2 g5g.4xlarge spot, us-east-1a. NVIDIA T4G, compute capability 7.5,
15,360 MiB, driver 595.71.05. Deep Learning ARM64 AMI OSS Nvidia Driver GPU PyTorch 2.12
(Ubuntu 24.04). torch 2.12.0+cu132, CUDA 13.2. vLLM 0.27.2rc1.dev0+g7f7a32cfe built from
v0.27.2rc0.

0.3% In, 36% Out: Your Fine-Tuned Model Is Copying Your Prompt Examples

Chidozie Uzoegwu — Thu, 13 Aug 2026 17:08:00 +0000

I run a fine-tuned Llama 3.3 70B on Amazon Bedrock. It generates short first-person narrative posts: a setup, a few lines of story, and a closing line that lands the point.

Last week I noticed the closing lines had collapsed into a single template. Not similar in spirit, but literally the same grammatical shape, over and over:

and that's how [someone] [learns/teaches] [something].

Roughly a third of everything the model produced ended that way. The obvious diagnosis was overfitting: the training data must be saturated with that pattern, so the model learned it as the way to end a story. The obvious fix was equally clear. Scrub the training set, retrain, redeploy.

In my setup that runs to about $30 of training compute and a five-hour training job, plus the evaluation pass afterwards before I'd know whether it had worked. Not ruinous, but not free either, and this would have been the third retrain cycle.

Before spending it, I did something I should have done first. I counted.

Step one: count the pattern in your training data

My training file is JSONL, one example per line. The check took about ten seconds:

grep -o -i "that's how" train_v6_2.jsonl | wc -l
# 5

wc -l train_v6_2.jsonl
# 1610

Five occurrences in 1,610 training examples. 0.3%.

Then I counted the same pattern in what the model was actually producing. Every generated post gets persisted, so this was one SQL query over the last 25 rows:

select right(content, 200) as ending
from generated_stories
order by created_at desc
limit 25;

Nine of twenty-five. 36%.

Those two numbers are irreconcilable. Training data at 0.3% cannot produce output at 36%. Fine-tuning shifts a model's distribution, but it does not amplify a pattern by a factor of a hundred. Whatever was driving this, it was not the weights.

Which meant a retrain would have changed nothing, and I'd have concluded my fine-tune was broken.

Step two: read your own prompt

The generation prompt included a line I had written months earlier and never revisited. Structurally it looked like this:

THE CLOSER: lead with a thesis line that names what the story proves, e.g. "and that's how [X] teaches [Y] to expect [Z]."

There it is. I had handed the model one complete, well-formed example of exactly the thing I wanted, and it did the most reasonable thing available to it. It copied the example, template and all.

This is not the model misbehaving. Give a language model a single concrete instance of a pattern and ask it to produce that pattern, and the example becomes the strongest signal in the context window. It will out-compete a 0.3% tendency in the weights every time. I had effectively hardcoded my output and then blamed the training data.

Step three: fix the injection layer, not the model

The fix cost nothing and shipped in an afternoon.

Instead of one hardcoded example, I built a small pool of seven closing lines. The key property is that they are structurally dissimilar: a flat verdict, a two-beat reveal, a consequence line, a prescriptive line, an ironic callback. Critically, none of them uses the shape that was being copied.

My production lines come from a private archive, so here is an illustrative pool in a neutral domain that shows the same variety:

_CLOSER_EXEMPLARS = [
    "The process was the problem.",
    "Nobody had tested it. Everybody had approved it.",
    "We saved an hour that afternoon and spent the next year paying for it.",
    "Write the runbook before you need it. You will not be calm enough later.",
    "So much for the quick fix.",
    "We called it a deadline for four months before anyone admitted it was a guess.",
    "What would you have checked first?",
]

examples = "\n".join(
    f"- {c}" for c in random.sample(_CLOSER_EXEMPLARS, 3)
)

Three are sampled at random per call, plus one explicit instruction: study the function these serve, then write a new one for this story, and do not reuse their wording.

The reasoning is that one example teaches a template, while several dissimilar examples teach a function. If every example looks different but they all achieve the same effect, the only thing left to imitate is the effect.

The result

Measured on live inference against the same deployed model, same theme, no retraining:

Run	Template rate
Before the fix	36% (9 of 25)
After the fix	8% (1 of 12)

Sample sizes are small and I'd want more data before treating 8% as precise. But the direction is unambiguous, and it cost one prompt edit rather than a retrain cycle.

The remaining 8% is, I think, genuinely the weights. That 0.3% is still in there, and a prompt fix can't reach it. That residue is a legitimate item for the next training-data scrub. The difference is that it's now a small, well-understood cleanup rather than a retrain I'd have run on a wrong diagnosis.

The generalizable part

When a fine-tuned model produces something repetitive, there are three possible causes and they have very different price tags:

The prompt is handing it the pattern. Free to fix. Check this first.
The training data is saturated with the pattern. Needs a scrub and a retrain.
The base model has a strong prior. Needs data specifically counteracting it.

The diagnostic that separates them is one line of grep and one SQL query. Compare the rate in your training data to the rate in your output. If the output rate is dramatically higher, the cause sits in your prompt or your retrieval layer, not your weights. If the two rates are similar, it really is a data problem and a retrain is justified.

Two habits came out of this that I'd suggest to anyone running a fine-tune on Bedrock:

Persist your generations. I could only run the "after" comparison because every output was already in a table with a timestamp. Without that history there is no measurement, only impressions, and impressions are what sent me toward a retrain in the first place.
Treat prompt examples as training data. They are, functionally. Anything concrete you put in a prompt as an illustration will get copied, so either give none, or give enough variety that only the underlying function is imitable. One example is the worst of both worlds.

The uncomfortable version of this lesson: the model was fine. I nearly spent money and two days proving that the hard way, and the evidence that it was fine was sitting in a file I could have counted at any point.

Creating DNS with AWS Managed Microsoft AD and Managing Users in AWS Directory Service

GargeeBhatnagar — Thu, 13 Aug 2026 10:56:09 +0000

“ I have checked the documents of AWS for creating dns with aws managed microsoft ad and managing users in aws directory service. AWS Directory Service makes it easy for users and groups to be managed with dns creation. In terms of cost, the solution is cheaper and secure.”

AWS Directory service provides multiple ways to use Microsoft Active Directory with other AWS services. Directories store information about users, groups and devices and administrators use them to manage access to information and resources. AWS Directory Service provides multiple directory choices for customers who want to use existing Microsoft AD or Lightweight Directory Access Protocol aware applications in the cloud. It also offers those same choices to developers who need a directory to manage users, groups, devices and access.

In this post, you will experience creating dns with aws managed microsoft ad and managing users in aws directory service. Here I have created a directory with user.

Architecture Overview

The architecture diagram shows the overall deployment architecture with data flow, aws directory service, users, domain controller, virtual private cloud.

Solution Overview

The blog post consists of the following phases:

Create of AWS Directory Service with AWS Managed Microsoft AD
Create of User in the Directory
Output of Share Directory to Other AWS Account

Phase 1: Create of AWS Directory Service with AWS Managed Microsoft AD

Open the AWS Directory Service console, Create a directory with AWS Managed Microsoft AD. Select standard edition with input of required directory information. In addition to it with vpc and subnet info. The directory will be created successfully.

Phase 2: Create of User in the Directory

Phase 3: Output of Share Directory to Other AWS Account

Clean-up

Directory.

Pricing

I review the pricing and estimated cost of this example.

AWS Directory Service = $0.00 per hour for 1500 hours for Microsoft AD domain controllers usage under the global free trial = 1.519 Hrs x 0.00 = $0.00

Total Cost = $0.00

Summary

In this post, I showed “creating dns with aws managed microsoft ad and managing users in aw directory service”.

For more details on AWS Directory Service, Checkout Get started AWS Directory Service, open the AWS Directory Service console. To learn more, read the AWS Directory Service documentation.

Thanks for reading!

Connect with me: Linkedin

AWS Advanced Certification Knowledge Framework

Anuvindh — Thu, 13 Aug 2026 10:03:21 +0000

Who Is ACKF For?

AWS certification candidates
Cloud engineers and Solutions Architects who want a structured way to understand AWS services
DevOps, DevSecOps, SRE, and Cloud Operations professionals
Students and professionals moving into AWS/cloud careers
AWS trainers, mentors, and study groups looking for a repeatable learning structure
Experienced AWS professionals preparing for recertification who need a faster way to review and recall concepts

Introducing ACKF Powered by the CLEAR™ Methodology

Official site link

AWS certification preparation can quickly become overwhelming.

There are hundreds of services, architectural patterns, limitations, integrations, pricing models, security considerations, and exam scenarios to understand.

Most learners do not struggle because information is unavailable.

They struggle because the information is fragmented.

You may study AWS documentation, watch training videos, complete labs, read whitepapers, create notes, and practise exam questions. However, without a consistent structure, it can become difficult to connect everything together and recall the right information when it matters.

This challenge led me to create:

ACKF - AWS Advanced Certification Knowledge Framework

ACKF is powered by the CLEAR™ Methodology:

C — Compare similar AWS services
L — Limitations that affect design choices
E — Exam Factors hidden within scenarios
A — Architecture placement and integrations
R — Review and Recall for stronger retention

The goal is simple:

Help learners move beyond memorising AWS services and start understanding how to compare, select, and apply them in real architectures.

This framework supported my own AWS certification and recertification journey. Version 1.0 is now public for learners, cloud engineers, architects, trainers, and the wider AWS community to explore and improve.

Example :

ACKF is an independent educational project and is not an official AWS training programme.

I would value feedback from AWS learners, educators, cloud engineers, and solutions architects.

Explore the project:

https://github.com/anuvindhs/ACKF-Powerd-by-CLEAR-Methodology

Explore content for AWS Solutions Architect Professional