DEV Community: AWS Community Builders

Kiro Crew: qué es y cómo empezar a usarlo

Alvaro Llamojha — Sun, 06 Sep 2026 18:55:54 +0000

Kiro Crew: qué es y cómo empezar a usarlo

Kiro Crew es un workspace de desarrollo de código abierto y persistente que se ejecuta en tu propia máquina o en un entorno remoto. Mantiene contexto entre sesiones, permite ejecutar trabajo durante largos periodos, programar tareas recurrentes, coordinar varios agentes y utilizar el mismo entorno desde su dashboard, la aplicación de escritorio, el CLI o canales como Slack y Discord.

Kiro Crew utiliza Kiro CLI como runtime y se comunica con él mediante ACP, Agent Client Protocol. Personalmente, esta es una evolución especialmente interesante porque ya estaba utilizando varias sesiones de Kiro CLI al mismo tiempo. Crew añade un entorno persistente alrededor de ese trabajo y agrupa capacidades que antes estaban repartidas entre terminales, scripts y otras herramientas.

La parte más importante desde mi punto de vista es que Kiro Crew es Código abierto y está disponible en GitHub, por lo que puedes revisar directamente su implementación, documentación y evolución. Puedes incluso crear tu propio fork de Kiro Crew a uno que se adapte más a tus necesidades.

En este artículo vamos a empezar desde cero: qué es Kiro Crew, cómo ponerlo en marcha y cuáles son las principales capacidades que merece la pena probar primero.

1. ¿Qué es Kiro Crew?

El componente principal de Kiro Crew es el Gateway. El Gateway es el proceso que conecta las diferentes interfaces de Crew con las sesiones de los agentes. Dashboard, desktop, CLI y los distintos canales de mensajería pueden utilizar el mismo Gateway, mientras que las sesiones ejecutan Kiro CLI mediante ACP y pueden acceder a herramientas adicionales mediante MCP.

La arquitectura de Kiro Crew se puede representar de una forma bastante sencilla:

Dashboard / Desktop / CLI / Slack / Discord / ...
                         |
                         v
                      Gateway
                         |
                         v
                   Agent sessions
                         |
                         v
                 ACP + Kiro CLI + MCP

Kiro Crew mantiene el estado de estas sesiones y permite ejecutar varias de forma concurrente. La documentación oficial también especifica que el contexto de proyectos puede sobrevivir entre sesiones y reinicios, que los trabajos largos utilizan checkpoints y que las tareas recurrentes pueden ejecutarse mediante schedules.

La información persistente se almacena en el host donde ejecutas Crew. La instalación mantiene bajo ~/.kiro/crew/ elementos como configuración, conversaciones, memoria, Knowledge, tareas programadas y agentes generados.

Esto permite volver a sesiones anteriores y reutilizar información que ya existe (contexto de la IA) dentro del workspace, en lugar de tratar cada interacción como una conversación completamente independiente.

2. Instalar Kiro Crew

Hay varias maneras de instalar Kiro Crew. Dispone de una aplicación de escritorio para macOS, Windows y Linux. También puede instalarse como paquete de Python, ejecutarse mediante Docker o construirse directamente desde el código fuente.

La forma más sencilla de empezar es seguir la guía oficial de instalación de Kiro Crew. La aplicación de escritorio incluye el Gateway y puede ayudarte también con la instalación y autenticación de Kiro CLI durante el primer arranque.

Si utilizas una instalación mediante paquete, código fuente o Docker, el flujo inicial incluye estos comandos:

kirocrew setup
kirocrew doctor
kirocrew gateway

kirocrew setup realiza la configuración inicial. kirocrew doctor comprueba el estado de elementos como Kiro CLI, autenticación, embeddings, configuración y servidores MCP. kirocrew gateway inicia el servidor, cuyo dashboard utiliza por defecto el puerto 5476.

Una vez iniciado el gateway, abre el dashboard; te pedirá si quieres importar la configuración de otros providers y ya estaría todo listo para lanzar una sesión. Cada conversación utiliza una sesión gestionada por Crew.

Mi recomendación es empezar así: una sesión normal desde el dashboard, una tarea sencilla y comprobar cómo se conserva esa sesión antes de añadir automatizaciones o integraciones. Kiro Crew también tiene introspección, por lo que puedes hacer cualquier pregunta si te sientes perdido o no sabes qué más puedes configurar.

3. Memory, Scheduled Tasks, Artifacts y Apps

Una de las diferencias principales de Kiro Crew es que muchas de sus capacidades no están limitadas a una única conversación. Memory y Knowledge, Scheduled Tasks, Artifacts y Apps forman parte del mismo entorno persistente.

Memory y Knowledge

Kiro Crew conserva contexto entre sesiones y reinicios. Las correcciones pueden almacenarse como lessons y los workflows repetidos pueden convertirse en Skills en formato Markdown. También dispone de una Knowledge Library respaldada por búsqueda de texto y vector embeddings para recuperar información relevante.

El sistema de embeddings se ejecuta en el propio Gateway. Durante el primer arranque, descarga el modelo de embeddings y, mientras no esté disponible, las búsquedas utilizan coincidencias por palabras clave. Cuando el modelo termina de descargarse, la búsqueda semántica pasa a utilizar embeddings sin necesidad de reiniciar Crew.

Knowledge es una de las funcionalidades que más utilizaba ya en Kiro CLI. Poder conservar información de sesiones o proyectos y recuperarla después evita tener que volver a proporcionar el mismo contexto en cada nueva sesión.

Knowledge puede además incorporar Artifacts y documentos para hacerlos recuperables posteriormente mediante búsqueda.

Scheduled Tasks

Kiro Crew permite ejecutar trabajo de forma programada utilizando cron jobs. En la arquitectura MCP de Kiro Crew aparecen herramientas específicas para crear, consultar, actualizar, pausar, reanudar y ejecutar tareas programadas.

En mi caso, uso el schedule task para recopilar artículos sobre Kiro para mi Kiro Weekly Reading List. Cada 2 días ejecuta un script que encuentra artículos y los pone en una lista que luego al final de la semana lo proceso.

Ese workflow con su script es mío y ya lo usaba antes de manera manual por Kiro Web. Lo que aporta Crew es la infraestructura para programar ese trabajo y ejecutarlo como una sesión gestionada sin que yo tenga que iniciar manualmente la tarea cada vez.

Artifacts

Kiro Crew dispone de Artifacts persistentes para conservar resultados producidos durante las sesiones. Sus herramientas permiten guardar, consultar, modificar y mantener versiones de esos resultados.

Esto puede utilizarse para documentación, especificaciones, informes, investigaciones u otros archivos que quieras conservar más allá de la conversación que los generó.

Soy un defensor de cómo manejar efectivamente el contexto de la IA; en el caso de Kiro, son Steerings, Skills, Powers y, con Kiro Crew ahora también son los Artifacts. En mi caso, esto es especialmente útil porque genero reportes y documentación en Markdown o HTML que se pueden guardar como artifacts en Kiro Crew.

Artifacts también puede integrarse con Knowledge, haciendo que determinados documentos producidos durante una sesión puedan incorporarse posteriormente a la base de conocimiento de Crew.

Apps: mucho más que un chat

Una de las partes que más me ha llamado la atención de Kiro Crew son las Apps.

Una App puede combinar agentes, Skills, servidores MCP, cron jobs, páginas personalizadas dentro del dashboard y procesos backend. Esto permite crear experiencias específicas para una tarea sin que toda la interacción tenga que ocurrir dentro de una conversación.

Por ejemplo, una app puede proporcionar una interfaz propia mientras utiliza agentes y automatizaciones por debajo. Entre los ejemplos mostrados por Kiro aparecen aplicaciones orientadas a investigación, revisión de código y otras tareas especializadas.

Kiro Crew también incluye un App SDK. Las interfaces del dashboard pueden construirse con TypeScript y React, mientras que otros servicios pueden interactuar con Crew mediante el cliente Python o las APIs REST y WebSocket del Gateway.

Las apps pueden ir desde algo pequeño basado en una Skill hasta aplicaciones con agentes, backend y una interfaz completa. También pueden distribuirse mediante registros o repositorios en Github.

4. Seguridad, Channels y ejecución remota

Permitir que un agente continúe trabajando cuando no estamos delante requiere controlar qué acciones puede ejecutar. Kiro Crew incorpora varias capas de seguridad que incluyen aprobación de herramientas, protección de paths sensibles, comandos denegados, sandboxing del proceso, validación de inputs MCP y redacción de credenciales.

La documentación de seguridad de Kiro Crew describe además una separación entre el Gateway, Kiro CLI y los procesos utilizados por las herramientas.

Las peticiones de herramientas pasan por un PreToolUse gate gestionado por Crew antes de ejecutarse. Esto permite aplicar las políticas de ejecución fuera del proceso controlado por el agente.

Con Kiro CLI ya tenía determinadas operaciones que permitía ejecutar y otras que prefería aprobar manualmente, aún no confiaba en una sesión totalmente automática (el famoso modo YOLO). Kiro Crew y sus capas de seguridad aumentan mi confianza en ejecutarlo en modo permisivo (YOLO). Mi recomendación es probar este modo con un par de tareas y monitorizándolo, una vez ganes confianza en este sistema, activar el modo auto-approve solo por unas horas.

Channels

Kiro Crew no obliga a interactuar únicamente desde el dashboard. Soporta integraciones con distintos sistemas de mensajería, incluyendo Slack, Discord, Telegram y Microsoft Teams, entre otros.

Los mensajes recibidos desde estos canales se conectan con sesiones gestionadas por Kiro Crew y las integraciones pueden incluir también el flujo de aprobaciones.

Ejecución remota

Ahora que puedes usar Kiro desde distintos canales, el siguiente paso es tener Kiro Crew corriendo en remoto. Kiro Crew permite ejecutarse en un host remoto (como en AWS) o mediante Docker. Esto permite mantener el Gateway disponible sin depender de que tu máquina principal permanezca encendida y tenerla disponible desde tus canales.

El proyecto incluye además soporte específico para AWS mediante Cloud Launcher. Kiro Crew puede crear una instancia EC2 utilizando CloudFormation, configurar el entorno y permitir posteriormente el acceso utilizando AWS Systems Manager.

La documentación del Cloud Launcher incluye comandos como:

kirocrew cloud doctor --region eu-central-1
kirocrew cloud launch --region eu-central-1
kirocrew cloud status
kirocrew cloud connect

También existen operaciones para detener, arrancar y destruir el entorno remoto. Muy útil para casos donde no estés usando Kiro Crew a cada hora del día.

5. ¿Qué probar primero?

Kiro Crew tiene muchas capacidades y es fácil sobrecargarse con tantas cosas que configurar desde el primer día. Empezaría con algo mucho más sencillo.

Primero, crearía una conversación desde el dashboard, organizaría carpetas y workspaces, me familiarizaría con los modos de aprobación y empezaría con un par de sesiones. Después, en una sesión en el mismo Kiro Crew, le pediría que llevara tu knowledge existente, agents y skills a Kiro Crew. Si no tienes, le pediría que creara nuevas basándose en tus necesidades.

También probaría alguna de las apps disponibles. Ahora mismo estoy probando la nueva app de AWS Control donde conectar mis cuentas de AWS y ver sus gastos.

La mejor forma de entender Kiro Crew es utilizarlo. Puedes empezar directamente desde la página oficial de Kiro Crew, consultar su documentación o revisar y ejecutar el código desde el repositorio oficial de Kiro Crew en GitHub.

Agentic AI Development with Kiro: The Hidden DevSecOps Layer — Closing the Loop

Alejandro Velez — Sun, 06 Sep 2026 00:56:33 +0000

Level 300

Some time ago we created a blog showing the capabilities of AI DLC and SDD to create quick and efficient prototypes as a MVP, the results were amazing, however, we omitted something: DevSecOps best practices and CICD for the workload.

The prototype was built with a serverless framework and modern cloud-native application patterns. However, moving from a working MVP to a production-ready solution requires stronger alignment with DevSecOps, CI/CD, and operational excellence. That transition leaves several important questions open:

Security posture
• What security vulnerabilities exist in the solution?
• Is the platform ready to withstand common web attacks?
Code quality and production readiness
• What is the overall quality of the code?
• Is this truly an example of an enterprise-ready solution?
Cloud compliance and misconfiguration risk
• Which cloud security compliance gaps still need to be addressed?
• Are there misconfigurations that could create operational or security risk?

The other side of this challenge is organizational readiness. Many companies are enabling development teams with assistants such as Kiro, Claude, Cursor, and similar tools. However, without a mature process to review, scan, govern, and manage code at scale, these tools can introduce high costs, expand security risks, and growing technical debt.

For the Agentic AI era, DevSecOps maturity is no longer optional. A secure software development lifecycle, policy-driven development, and zero-trust principles must become core operating requirements rather than afterthoughts.

Open loop – Starter point a classical DevSecOps CICD system

Suppose that the company already has a continuous integration and delivery framework and tools for that kind of workload with classical tools, in terms of maturity level and L3 Defined and Managed here we have Security gates integrated in CI/CD; SAST/SCA/secrets/IaC; centralized findings; quality gates.

The maturity model we use

We measure against a five-level DevSecOps maturity model. The levels are
cumulative — each assumes the previous.

Level	Name	Characteristic
L1	Initial	Manual builds/deploys; security is a late, manual gate (if any).
L2	Repeatable	Scripted CD; some automated tests; ad-hoc scanning.
L3	Defined / Managed	Security gates integrated in CI/CD; SAST/SCA/secrets/IaC; centralized findings; quality gates.
L4	Agent-Integrated	Pipelines dispatch agents to remediate; findings become PRs; developers review instead of investigate; feedback loops close.
L5	Autonomous / Self-Healing	Agents act on production/runtime signals with policy-bounded autonomy; the system proposes and validates its own improvements.

Solid DevSecOps Level 3
Here is what is genuinely in place. Most organizations never achieve this.

Delivery

• Two independent, path-filtered CI/CD pipelines — backend (CDK/Lambda) and frontend (React/S3/CloudFront), separated because their lifecycles differ.
o Backend: backend-{development,staging,production}.yml
o Frontend: frontend-{development,staging,production}.yml
• Reusable workflows for security scanning and deployment per component.
• Environment promotion: dev → staging (on success) → production (on release).
• OIDC hub/spoke AWS auth — no long-lived cloud credentials in CI.

Security gates (the DevSecOps core)

Category	Tool	Where
Secret scanning	gitleaks + git-secrets	every pipeline
SAST	Semgrep + SonarQube	`sast` job, per component
SCA + SBOM	Trivy → CycloneDX → Dependency-Track	`sca` job, per component
IaC / config	thothctl (checkov, trivy)	backend `iac` job
Findings aggregation	SecObserve	`gate` job
Quality gate	SonarQube "Athleon Industry Standard" (Clean-as-You-Code)	`sast` job

Per-component SonarQube projects: athleon-backend, athleon-frontend, both on a Clean-as-You-Code quality gate (0 new issues, ≥80% new coverage, ≤3% new duplication, A ratings, 100% new hotspots reviewed).
SBOMs published to Dependency-Track per environment (athleon-backend-<env>, athleon-frontend-<env>), CycloneDX normalized to the version DT accepts.
All scanner findings imported into SecObserve for centralized triage.

Kiro adoption (the differentiator)

The company enables the AI adoption for devs centric on Kiro adoption (the differentiator)

Spec-driven development is an established muscle: six specs under .kiro/specs/ with requirements / design / tasks.
Steering docs (.kiro/steering/product.md, structure.md, tech.md) encode product and architecture context for the agent.
One remediation sub-agent (.kiro/agents/sonar-fixer.json) wired to the official SonarQube MCP server.
MCP configured for Kiro CLI and IDE (.kiro/settings/mcp.json).

At this point the hard part is done: the gates, the aggregation, the quality bar, and — crucially — a team fluent in Kiro's spec-driven workflow. The next level is not more tools. It's connecting the tools we have to the agents we have, so the pipeline stops reporting problems to humans and starts handing humans review-ready fixes.

The number of findings was significant:

Also, github security and quality report 344 issues

The devs could manually interact with agent and sonar to improve the code and fix the issues, however, is a manual approach. Running the sonarfixer agent that uses the MCP tools from shell interactive:

kiro-cli chat --agent sonar-fixer "Fix the top criticals on athleon-backend"

Using the MCP and agent:

{
  "name": "sonar-fixer",
  "description": "SonarQube remediation specialist for Athleon - queries SonarQube via MCP and fixes code quality, reliability, and security issues following Clean-as-You-Code.",
  "prompt": "You are the SonarQube remediation specialist for the Athleon platform (multi-tenant calisthenics competition management: AWS CDK, Node.js Lambda, React/Vite frontend). Your job is to reduce SonarQube findings safely and systematically.\n\nProject key: athleon. SonarQube Server: http://sonar.devsecops.lan.\n\nCurrent baseline (improve, do not regress): Security rating D (271 issues), Reliability E (~1.2k), Maintainability A (~2k), Coverage 0-4%, Duplications ~21.7%. The active quality gate 'Athleon Industry Standard' enforces Clean-as-You-Code on NEW code (0 new issues, >=80% new coverage, <=3% new duplication, A ratings, 100% new hotspots reviewed).\n\nWORKING PRINCIPLES:\n1. Use the SonarQube MCP tools to fetch real issues - never guess. Prioritize: (a) BLOCKER/CRITICAL security & reliability first, (b) then high-impact maintainability, (c) then duplication.\n2. Fix in small, reviewable batches grouped by rule or file. One rule family per change set.\n3. Before editing, read the surrounding code and the project's steering docs (.kiro/steering/) to match conventions. Follow existing patterns (shared layer utils, DDD bounded contexts, EventBridge cross-domain comms - never direct cross-domain imports).\n4. For every fix: explain the rule, why it fired, and the fix. Preserve behavior. Do NOT introduce new dependencies without asking.\n5. Add or update tests for any behavior you touch (vitest). Raising coverage on changed files helps the new-code gate.\n6. Security issues: apply secure patterns (input validation, parameterized queries, no hardcoded secrets, least privilege). Never weaken auth/RBAC.\n7. Verify after each batch: run the relevant vitest suite and, if possible, re-check the issue in SonarQube.\n8. NEVER mark issues as false-positive/won't-fix in SonarQube without explicit human approval.\n9. Respect the safety guardrails: no destructive git ops, no production changes, confirm high-impact actions.\n\nWORKFLOW per session:\n- Ask which scope to work on (a domain, a file, a rule, or 'top criticals') if not specified.\n- Query SonarQube for that scope. Present a short prioritized plan.\n- Implement the highest-value batch. Run tests. Summarize what changed and the expected SonarQube impact.\n- Repeat or hand back for review.",
  "mcpServers": {
    "sonarqube": {
      "command": "docker",
      "args": [
        "run",
        "--init",
        "--pull=always",
        "-i",
        "--rm",
        "--network=host",
        "-e",
        "SONARQUBE_TOKEN",
        "-e",
        "SONARQUBE_URL",
        "-e",
        "SONARQUBE_PROJECT_KEY",
        "-v",
        "${workspaceFolder}/.kiro/.sonar-mcp-storage:/app/storage",
        "-v",
        "${workspaceFolder}:/app/mcp-workspace:ro",
        "sonarsource/sonarqube-mcp"
      ],
      "env": {
        "SONARQUBE_TOKEN": "REPLACE_WITH_SONARQUBE_USER_TOKEN",
        "SONARQUBE_URL": "http://sonar.devsecops.lan",
        "SONARQUBE_PROJECT_KEY": "athleon-backend"
      },
      "autoApprove": [],
      "disabled": false
    }
  },
  "tools": [
    "fs_read",
    "fs_write",
    "execute_bash",
    "@sonarqube"
  ],
  "allowedTools": [
    "fs_read",
    "@sonarqube"
  ],
  "toolsSettings": {
    "execute_bash": {
      "allowedCommands": [
        "npx vitest*",
        "npm test*",
        "npm run test*"
      ]
    }
  }
}

For example:

At this stage, the key question becomes: how can DevSecOps teams evolve the existing platform into an agent-integrated DevSecOps model?

The Level 4 Vision: Agent-Integrated DevSecOps

Instead of simply alerting humans to problems, the pipeline should deliver review-ready fixes.
Today, the developer workflow for security or quality finding typically looks like this:

> pipeline fails gate
  → developer gets notified
  → opens SonarQube / SecObserve / Dependency-Track to understand it
  → context-switches into the IDE
  → reproduces, investigates, fixes
  → writes tests
  → opens PR

That's minutes-to-hours of human context switching per finding, and it scales linearly with the number of findings. It is the single biggest drag on time-to-market for a team that already has good gates.

The Level 4 loop:

pipeline finding issue
  → dispatches the right agent (headless Kiro CLI, using an existing sub-agent)
  → agent reads the finding via MCP, fixes it, writes/updates tests, runs them
  → agent opens a PR with an explanation
  → developer REVIEWS the PR (approve / request changes)

Humans are still in the loop — but at the review step, which is where human judgment is most valuable and least repetitive. This is the core TTM (Time to Marker) lever.

What "closing the loop" requires

Four connections must exist that don’t exist today:

Trigger → Agent

A pipeline event (failed quality gate, new critical vuln, new SecObserve finding) must be able to dispatch an agent without a human. Kiro CLI supports this directly:

• kiro-cli chat --no-interactive --agent sonar-fixer "..." runs headless.
• KIRO_API_KEY authenticates in CI with no browser flow.
• Standard exit codes (0/1/2) make it scriptable.

Agent → Context (the MCP mesh)

An agent is only as good as what it can see. Today the sonar-fixer sees SonarQube. A Level-4 agent should see the whole security picture: SonarQube (have), Dependency-Track and SecObserve (custom MCP, APIs already mapped), GitHub (official MCP), and AWS deployed state (awslabs MCP servers).

Agent → Change (PR, not commit)

Agents propose via pull requests, never direct pushes to protected branches. The Clean-as-You-Code quality gate then runs on the agent's PR just like a human's — the same bar applies. This is the safety mechanism that makes autonomy acceptable.

Outcome → Memory (feedback loop)

In an advanced Level 4 setup, each human correction should become durable, shared agent memory—not just a one-time review comment. Approved patterns, rejected approaches, architectural decisions, and remediation lessons should be captured in steering docs and connected to the enterprise knowledge base, so every agent can reuse the same organizational context. Over time, this creates a feedback loop where agents learn from reviews, align with company standards, and reduce repeated mistakes across teams and pipelines.

Principles that keep this safe

Agent integration fails when it's ungoverned. Athleon's Level 4 is bounded by:

• Human-in-the-loop by default. Agents open PRs; humans merge. Autonomy is earned per workflow after it proves reliable, never granted wholesale.

• The same quality gate for agents and humans. An agent PR that lowers coverage or adds issues fails the gate. No special path.

• Least-privilege tooling. Each agent's allowedTools and MCP scope is the minimum for its job (e.g., sonar-fixer can read/write code and run tests, but not touch production).

• No secret sprawl. KIRO_API_KEY and tool tokens live in GitHub secrets, never in agent JSON committed to the repo.

• Reversibility. Every agent action is a reviewable, revertable Git change.

Why “company” is ready for Level 4

Most teams that attempt agent integration are missing the foundation required to make it safe and repeatable. Company is different because the core DevSecOps substrate is already in place:

• Existing quality gates. Agents already have a clear definition of done: pass the quality gate and pass the tests.
• Centralized findings. SonarQube, Dependency-Track, and SecObserve provide structured, queryable inputs that agents can act on.
• Spec-driven execution. The team already works through .kiro/specs/, which means intent is captured precisely enough for agents to follow.
• A working sub-agent. The existing sonar-fixer agent is the seed for a broader remediation fleet.

> In other words, the gap is not capability. The gap is connection.

The flagship Level 4 initiative

If the Company takes one practical step toward Level 4, it should start here:

Convert sonar-fixer from a manually invoked assistant into a pipeline-triggered agent that opens review-ready pull requests.

This approach reuses the gates, findings, specs, and sub-agent that already exist. It requires only one additional secret, KIRO_API_KEY, and immediately shifts the developer workflow from manual investigation to human review. It is also the cleanest proof of value for the broader agent-integrated DevSecOps roadmap.

Conclusion: Closing the Loop

The second part of this journey makes one point clear: building fast with agentic AI is only half of the story. The hidden side is what happens after the prototype works—how the organization governs code quality, security, compliance, and production readiness at scale.

Kiro and similar agents can accelerate development, but they become truly valuable when they are connected to mature DevSecOps foundations: quality gates, centralized findings, secure CI/CD, MCP-based context, and human review through pull requests. This is the shift from an open loop—where tools report issues and humans manually chase them—to a closed loop, where agents transform findings into review-ready fixes.

For companies adopting agentic AI, the goal should not be unchecked autonomy. The goal is governed acceleration: agents operating within clear policies, limited permissions, shared enterprise context, and the same quality standards expected from human developers. In that model, humans stay in control, but their effort moves from repetitive investigation to higher-value review and decision-making.

The practical next step is simple and powerful: evolve the existing sonar-fixer assistant into a pipeline-triggered agent that opens pull requests with tested, explainable remediations. That single move proves the Level 4 model while preserving safety, traceability, and human accountability.
Closing the loop is not about replacing developers. It is about giving development teams a smarter operating model—one where DevSecOps signals become action, agent memory improves with each review, and enterprise knowledge turns isolated fixes into repeatable engineering practice.

✨ Alejandro Velez, Platform Engineering Latam Lead @ GFT | AWS Ambassador

FAIR Data for Agentic AI

N Chandra Prakash Reddy — Sat, 05 Sep 2026 14:41:28 +0000

I got the wonderful opportunity to attend AWS Community Day Chennai on 7th March 2026. There were a ton of great sessions throughout the day, but one specific session absolutely transformed the way I thought about the future of artificial intelligence. The topic was “FAIR Data for Agentic AI” and the speaker was Naveena Ravi.

Whether you’re an experienced data engineer or just beginning your cloud adventure, knowing how to prepare your data for the next generation of AI is key.

The AI Transformation: From Predictions to Actions

Let's be honest, keeping up with AI has felt a bit like trying to sip from a firehose the past few months. To help us understand exactly where we are headed, Naveena opened her session by breaking down the growth of artificial intelligence into three separate phases.

Traditional AI was mostly about using current data to generate predictions and drive decisions. Think of it like your phone’s weather app, trying to figure out if it’s going to rain tomorrow based on previous data. Then came Generative AI that unleashed the power to generate and create brand new material from our text prompts. It’s like asking a chef to create a unique recipe for you based on your favorite ingredients.

And this is when it gets interesting. We are nearing the age of Agentic AI. Agentic AI is about performing action, rather than predicting an outcome or generating a block of text. These complex algorithms can think for themselves, strategize and really do things on your behalf.

Decoding AI Agents

You may be asking yourself, what does it mean for an AI to be “agentic”? "A clear definition was given by Naveena: AI Agents are semi or fully independent pieces of software. They have the unique ability to reason, plan, and act to achieve certain goals. And they can also work smoothly in digital and physical situations.

Think of your company’s database as a large disorganized library. A typical Generative AI model is like a speed reading assistance, if you give it a book, it will summarize the book for you. But an AI Agent is a proactive researcher. You tell it you need a report, and it scans the shelves, pulls the five most relevant-looking books, extracts the best quotes, turns it into a polished paper, and emails it to your employer.

The Building Blocks of an Agent

To enable Agentic AI to do this independent magic, four main components function in perfect balance:

LLM (Large Language Model): This is the main brain of the operation , allowing the system to understand human language and process complex logic .
AI Agent: The orchestrator that takes the user’s aim and turns it into steps, doing the thinking and planning.
RAG (Retrieval-Augmented Generation): The memory system draws in relevant business facts related to the scenario so the AI does not just guess the responses.
MCP (Model Context Protocol): The bridge or communication layer that allows the agent to connect securely to outside tools, applications and environments.

The Next Generation of Amazon SageMaker

These complex, interconnected systems need a very robust base for developers to build on. This is where Amazon SageMaker comes in. “The next generation of SageMaker is built to be the central hub for all of your data, analytics and AI needs.

If you’re developing a startup, stitching together a dozen different tools might be an operational headache. SageMaker fulfills this need with a Unified Studio architecture.

A Unified Architecture

The architectural Naveena gave was a great roadmap for today’s data teams. At the core level you have Open Lakehouse, sitting just underneath a vital layer for Data & AI Governance and on top of this safe base stands the Unified Studio, which divides your workflow into specialized toolsets:

SQL Analytics: Powered by tools such as Amazon Redshift and Amazon Athena to query your data.
Data Processing: Using Amazon EMR and AWS Glue to clean and process raw data.
Model Development: Build and train your algorithms powered by Amazon SageMaker AI.
Gen AI App Development: Built securely using Amazon Bedrock.
Future Capabilities: The ecosystem is also expanding with Streaming (Amazon MSK, Kinesis), Business Intelligence (Amazon QuickSight) and Search Analytics (Amazon OpenSearch Service) products coming soon.

Transforming Raw Data into AI Wisdom

Here’s the thing: An AI agent is only as smart as the data you feed it. But to really enable these agents to make the right choices, your business data needs to be run through a complete, step-by-step process to be transformed into actionable information.

The Data Pipeline Steps

Data Ingestion: The process of getting raw, unstructured data into your system.
Taxonomy / Ontology: Classifying and organizing the data in a way that clearly establishes linkages and hierarchies.
Data Modelling – Schema Creation: Building the actual blueprints/formats of how the data is saved.
Data Quality Management: Cleaning the data such that the information is correct, clean and free of errors.
Data Catalog: Index everything to make the data easy to find for your teams.

The Knowledge Pyramid

To give a sense of this transformation, Naveena presented a stunning pyramid that showed the journey from raw inputs to complex AI.

At the very bottom base is raw Data, the first Data Ingestion phase. One rung higher, Data Processing turns those raw inputs into useful Information. Then at the Knowledge level, we have the Data Catalog which organizes the data for easy discovery. And last Wisdom where the AI Agents are. These AI Agents use all the underlying layers to act smartly.

The Core: F.A.I.R. Data Principles

To ensure your data can safely and securely reach that peak “Wisdom” stage, it must strictly follow the F.A.I.R. Data Principles. Is that you? F.A.I.R. stands for Findable, Accessible, Interoperable and Reusable.

Let's go over what this means for your data infrastructure.

1. Findable

For data to be useful, it needs to be easily discoverable by human workers and computer systems.

Organizations need to have the right governance to tightly control metadata.
The metadata must be significant, very detailed and continuously persistent across the whole business.

Consider this as an online e-commerce store. If a new pair of shoes is not labeled with the correct category, color and size description, no customer (or search engine algorithm) will be able to find it.

2. Accessible

When data is found it must be accessible in a safe manner, with no extra barriers.

To keep security, the actual data should be strictly accessible to authorized individuals only.
But the descriptive metadata needs to be available to relevant people and AI agents so they know what’s there.
The protocol for accessing this information must be open or easily recognized for conventional authentication and authorization procedures.
Crucially, metadata must be available even if the underlying data is removed or is no longer available.

3. Interoperable

Data should not be separate, disconnected silos.

Metadata shall be in standardized, FAIR certified formats.
This makes it easy to identify the data across numerous distinct systems and different AIs Agents.
The data itself must be able to easily cooperate with apps and workflows, allowing easy storage, processing and analysis.

Simply said, your data should have a common language. If your marketing software speaks French and your sales software speaks Japanese, your AI Agent won’t be able to help you. Compatibility means everyone speaks the same technological language.

4. Reusable

And finally, solid data is an asset that should continue to provide value over time.

Metadata must be extremely reusable so engineering teams may design complex, interconnected systems.
It should describe both the metadata and the actual data in a thorough and good way.
At this level of description, the information is easily copied or merged for fresh new use cases down the line.

Key Takeaways

The secret to building mind-blowingly intelligent, autonomous AI agents isn't picking the latest language model, it's really a data organization challenge.

At the end of the session, Naveena raised a vital question to the audience, "Is your DATA Agent Ready?" To be able to answer yes, with confidence, organizations need to heavily focus on these essential pillars:

Data Quality is non-negotiable: unavoidably bad data leads to unavoidably faulty, and even dangerous, agent decisions.
Strict Data Governance: You want to make sure security, compliance and adequate control of access are all locked down entirely.
Clear Data Lineage: You must know precisely where your data started and how it has evolved throughout its existence.
Embrace F.A.I.R. Principles: Only Findable, Accessible, Interoperable, and Reusable data can bridge the gap between fundamental information and actual AI wisdom.

Conclusion

After all, transitioning from standard AI to Agentic AI is a tremendous technological leap. We’re not simply having computers predict the future or write words anymore; we’re trusting them to develop plans and perform real-world activities for us.

To summarize, before unleashing autonomous AI to tackle your complicated business difficulties, you need to make sure your data house is clean, structured, and properly managed. If you’re leading a data team today, I highly recommend checking out the unified design of Amazon SageMaker, and start evaluating your own pipelines to see just how F.A.I.R. your data really is.

About the Author

As an AWS Community Builder, I enjoy sharing the things I've learned through my own experiences and events, and I like to help others on their path. If you found this helpful or have any questions, don't hesitate to get in touch! 🚀

🔗 Connect with me on LinkedIn

References

Event: AWS Community Day Chennai

Topic: FAIR Data for Agentic AI

Date: March 7, 2026

Also Published On

AWS Builder Center

Hashnode

From Data Lake to Lakehouse: Building Modern Analytics Platforms on AWS with S3 Tables & AWS Glue

N Chandra Prakash Reddy — Sat, 05 Sep 2026 12:05:49 +0000

Okay, we will be honest – data architecture is usually one of those topics that makes your head spin. I was there at AWS Community Day Chennai on 07-Mar-2026. There were a lot of good talks on many kinds of cloud technologies, but one speaker in particular caught my eye.

Then we had Vishali Sakthivel and Vikneshwara RB take the stage to discuss about transitioning from Data Lakes to Lakehouses with Amazon S3 Tables and AWS Glue. If you've ever wrestled with messy data pipelines, this workshop felt like a breath of fresh air. I want to break out exactly what I learnt.

The Headache of Traditional Data Architectures

Sound familiar? We did data analytics with on-prem data warehouses for the longest period. This old system has some significant baggage.

We had to work with strict concepts that were really hard to alter.
We were trapped with ETL heavy pipelines that had to be babysat constantly.
Scaling was brutally expensive and vendor lock-in was a significant risk.

Even as we migrated to the cloud there were still serious engineering problems. Data engineers have always struggled with the “small files problem" the presence of thousands of little files would significantly impact query performance.

Imagine reading a book where every sentence is printed on a separate piece of paper and spread out throughout a room. That’s how your database feels dealing with the little files problem. We also suffered schema evolution issues, slow analytical queries, difficult pipeline maintenance, late arriving data and duplicate records.

Enter the Lakehouse and Apache Iceberg

Here is when things become interesting. The industry responded to these problems by moving to the “Lakehouse” architecture. A Lakehouse can be thought of as the best of both worlds. It combines the vast storage scale of a Data Lake and the organized reliability of a Data Warehouse.

Apache Iceberg is the core of this transition. Iceberg is an open table format for data streaming and Lakehouses. You might think of it as a very efficient index for your data. It lets different compute engines like Apache Flink, Spark, Snowflake and Athena read the exact same data without duplicating it.

The Maintenance Trap

But running Apache Iceberg yourself on AWS isn’t a picnic. The speakers raised several serious challenges:

Operational Overhead: You need to keep a close watch on the system.
Metadata Explosion: Tracking data changes creates huge metadata files which bog things down.
Manual Maintenance Trap: Engineers run compaction jobs for hours at a time to maintain things healthy.
Catalog Consistency: It is hard to keep your metadata catalog fully in sync.

S3 Tables to the Rescue

The issue is: Amazon S3 Tables takes care of all that messy operational overhead. S3 Tables offers fully managed Iceberg tables with automatic table maintenance. Simply put, AWS conducts the heavy lifting behind the scenes, such as cleaning up small files and optimizing metadata, so you don't have to.

The architecture consists of few building blocks:

Amazon S3: Storage Buckets Table
AWS Glue: For data transformations.
Amazon Athena: For SQL-based analytics.
QuickSight (BI Reports): For visual dashboards.

Structuring Data: The Medallion Architecture

The workshop was designed to function seamlessly for that hence the talk was on the Medallion Architecture, best practice in arranging data into three layers:

Bronze Layer: This is for raw ingestion and history . It accepts CSV, JSON, and TXT files as they are. This was the raw orders and raw customers tables in their retail use case.
Silver Layer: Here the data is filtered, cleansed, and supplemented. The dataset gets cleaned up and silver_orders and silver_customers are considerably more readable.
Gold Layer: This is the aggregate layer of the business level. It has facts and dimensions like gold_fact_orders and gold_customer_metrics, which are suited for BI reporting and Machine Learning.

They showed a retail analytics use case where data was moving from an S3 bucket to ingestion jobs to Bronze S3 table, transformed through Glue to Silver and transformed again to Gold, and then pushed to QuickSight and SageMaker. Inside an S3 Table bucket in the AWS dashboard these managed tables were plainly visible, grouped neatly.

Getting Data In: Choose Your Weapon

You may be asking how we actually get data into these S3 tables? The speakers broke down three good intake approaches using user profile and data size. The best thing? They showed us the code itself.

1. Python (PyIceberg & PyArrow) for Small/Medium Data

If you are running lightweight data pipelines or single-threaded python programs, PyIceberg + PyArrow is your best friend.

First, you’ll need to connect to the S3 Tables catalog. Think of the catalog as the master index that points Python directly to where your data exists. Notice the usage of AWS SigV4 in the settings, which ensures that your connection is safely authenticated with your normal AWS credentials.

from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "s3tables_catalog",
    **{
        "type": "rest",
        "uri": f"https://s3tables.{region}.amazonaws.com/iceberg",
        "warehouse": table_bucket_arn,
        "rest.sigv4-enabled": "true",
        "rest.signing-name": "s3tables",
        "rest.signing-region": region,
    }
)

Once we connect we are able to construct a namespace (think of it as a folder for our tables) and specify our schema using PyArrow . Imagine a schema as the column headers of an Excel spreadsheet, informing the database what kind of data to expect.

import pyarrow as pa

# Create a namespace
catalog.create_namespace("demo_ns")

# Define exactly what our data looks like
schema = pa.schema([
    pa.field("order_id", pa.int32()),
    pa.field("customer_name", pa.string()),
    pa.field("product", pa.string()),
    pa.field("quantity", pa.int32()),
    pa.field("price", pa.float64()),
    pa.field("status", pa.string()),
])

# Create the empty table
table = catalog.create_table("demo_ns.orders", schema=schema)

Now let’s insert some real data into our table and get it back. The very fascinating thing here is the "predictive pushdown” of the filtered scan. In layman's terms, this means that we are saying to the database, 'just give me the rows where the status is "shipped"' and so we avoid having to download large quantities of irrelevant data.

# Create some sample data
rows = [
    {"order_id": 1, "customer_name": "Alice", "product": "Widget A", "quantity": 10, "price": 29.99, "status": "shipped"},
    {"order_id": 2, "customer_name": "Bob", "product": "Widget B", "quantity": 5, "price": 49.99, "status": "pending"},
    {"order_id": 3, "customer_name": "Charlie", "product": "Widget C", "quantity": 2, "price": 99.99, "status": "shipped"},
]

# Write data to the table
arrow_table = pa.Table.from_pylist(rows, schema=schema)
table.append(arrow_table)

# Filtered scan: Only pull data where status is "shipped"
from pyiceberg.expressions import EqualTo
df_shipped = table.scan(
    row_filter=EqualTo("status", "shipped")
).to_pandas()

2. Spark (AWS Glue) for Big Data & ETL

If you want to do large scale heavy duty data transformation then AWS Glue running on Spark is your go-to service. You just inject the Iceberg extensions into your Spark Session and no hard workarounds are needed to make Spark connect to S3 Tables.

This is what that Glue Job configuration looks like:

# ----------------------------------------------------
# Spark Session Configuration for S3 Tables
# ----------------------------------------------------
spark = (
    SparkSession.builder
    .appName(job_name)
    .config(
        "spark.sql.extensions",
        "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",
    )
    .config(
        f"spark.sql.catalog.{catalog_name}",
        "org.apache.iceberg.spark.SparkCatalog",
    )
    .config(
        f"spark.sql.catalog.{catalog_name}.catalog-impl",
        "software.amazon.s3tables.iceberg.S3TablesCatalog",
    )
    .config(
        f"spark.sql.catalog.{catalog_name}.warehouse",
        bucket_arn,
    )
    .getOrCreate()
)

3. SQL (Amazon Athena) for Analysts

If you are an analyst who prefers SQL, you can work with S3 Tables directly from the Athena console.

For example, running the famous “Time Travel” functionality we talked about above is as simple as adding FOR VERSION AS OF at the end of a normal SQL query. Here’s a sample from the session to show how you may query a historical snapshot of the data:

SELECT * FROM "daily_sales" FOR VERSION AS OF 2246846639951314761;

The "Wow" Moments: ACID, Time Travel, and Schema Evolution

To be fair there are a lot of tools that can shift data around. But S3 Tables adds real database like features to object storage.

ACID Transactions: Now you can execute UPDATE and DELETE operations straight in Athena. If a consumer deletes their account, you perform a normal SQL delete command on your data lake.
Time Travel & Snapshot Isolation: Just think about finding out you've erased the wrong records by mistake. S3 Tables lets you query data as it was in the past, literally, with a single SQL command: SELECT * FROM table FOR VERSION AS OF [snapshot_id].
Schema Evolution: Business needs are always changing. To add a new column “City” to your database, just run an ALTER TABLE command. It updates instantly without having to re-write the full history dataset.

Turning Data into Business Insights

After all, data is only valuable if it answers business questions. The speakers presented a highly optimized query layer on top of the Gold layer with Amazon Athena. Athena uses partition pruning and is highly optimized for Iceberg, so you only pay for the exact data you scan.

They ran some interesting analytical queries from the actual world:

Monthly Revenue Trend: Aggregating total orders and revenue by month.
Customer Lifetime Value & Segmentation: Classifying Customers to Platinum, Gold, Silver and Bronze levels depending on their total spend.
Repeat vs. One-Time Buyers: Behavior analysis to determine how many customers return vs. buy once.

These SQL queries were then wonderfully presented using Amazon QuickSight dashboards to generate pie charts and bar graphs that business stakeholders could truly utilize.

The Future is Here: Claude + MCP Server

Long story short, writing SQL can still be a bottleneck for non-technical people. The speakers thrilled the audience by displaying an integration with Anthropic’s Claude AI with an MCP (Model Context Protocol) server.

They set up a local MCP server called s3tablesagent which enabled Claude to securely access the S3 Tables data warehouse. In the chat they just wrote: "What are the top 10 customers by total revenue and what is their average order value?".

Claude got the intent, queried the Gold tables in the background, and sent back a neatly prepared markdown table with significant business insights, such as “Customer 92 combines both strategies well - fewer orders but a strong average order value”. It was like having a senior data analyst sitting inside the chat window!

Key Takeaways

Here’s a quick summary of what you need to remember based on the summary given at the end of the session:

The Best of Both Worlds: The Lakehouse architecture (Amazon S3 Tables and Apache Iceberg) combines the huge volume of a data lake with the dependability and performance of a traditional data warehouse.
Zero Maintenance: Iceberg administration is simple, because Amazon S3 Tables takes care of metadata, compaction and table optimization behind the scenes.
Massive Scale without Servers: AWS Glue lets you develop distributed, large-scale ETL pipelines without ever handling the underlying infrastructure.
Serverless SQL: Amazon Athena enables data analysts to query big datasets immediately using conventional SQL without having to manage clusters.
Better Organization: By organizing your data, the Medallion Architecture (Bronze, Silver, and Gold layers) will increase your data reliability, management, and ability to prepare for analytics.
Faster Decisions: You can also build Amazon Athena and QuickSight on top of handpicked datasets, to get powerful business insights and dashboards, much faster.

Conclusion

The conclusion is as follows: The new data architecture is evolving rapidly and AWS makes it very accessible. This session was a masterclass on updating your analytics platforms.

Vishali and Vikneshwara performed an amazing job solving complicated, headache producing data engineering difficulties and giving a clean, automated and highly scalable approach. Are you exhausted of wrestling with complicated data pipelines, strict schemas, and small file problems? Amazon S3 Tables might be the tool you have been waiting for.

About the Author

🔗 Connect with me on LinkedIn

References

Event: AWS Community Day Chennai

Topic: From Data Lake to Lakehouse: Building Modern Analytics Platforms on AWS with S3 Tables & AWS Glue

Date: March 7, 2026

Also Published On

AWS Builder Center

Hashnode

I Tried Nx Plugin for AWS, Here's Why I'm Sold

Steven Leung — Sat, 05 Sep 2026 09:40:51 +0000

Who hasn't built a full-stack app on AWS before, we all know the drill. You need an API (usually Lambda with API Gateway), a frontend, some authentication (Cognito) wired up, and IaC (CDK) to help deploy the app. On their own, none of that is hard, but wiring it up all together, especially in a team where every developer has their own style, always takes an amount of time before you even get to write a single line for the business logic.

Nx Plugin for AWS (@aws/nx-plugin), an AWS Labs open source project, tackles that problem with code generators built on top of Nx. Instead of once again writing out boilerplate for every new service or website, you can just run a generator (CLI), answer a few questions, and get production-ready application code plus the CDK or Terraform to deploy it.

This article introduces what the plugin does, the core concepts, and how to scaffold a complete full-stack app. What sets Nx Plugin apart from just another scaffolding tool?

A note on versioning: at time of this article the plugin was pre-1.0, currently working through a 1.0.0-rc.x release candidate series with regular updates. Commands and generator names below were accurate at time of writing, check npm view @aws/nx-plugin version before you start, given the pace, it's likely to have moved on since this article published.

A quick primer on Nx

For the uninitiated Nx is a toolkit specifically for monorepos. Two things define Nx:

A dependency graph across projects. Nx knows how the individual parts of your projects relate to each other (which website depends on which API, which library is shared where), and it uses that graph to only build, test, or lint the things actually affected by a change, with results cached so recurring CI/CD runs should be fast.
Generators. Nx has a plugin system where a package can register generators, scripts that scaffold or modify code in your workspace, invoked via the nx g (or nx generate) command. This is the bit @aws/nx-plugin builds on.

Nx is not AWS-specific, rather AWS adopted and built their plugin on top of Nx. @aws/nx-plugin doesn't replace any of that, it's a collection of generators in your toolbag that plug into your codebase.
Ideally suited for greenfield projects, with a bit of uplift can work with existing codebases.

So when you run pnpm nx g @aws/nx-plugin:ts#api, you're using Nx's own generator mechanism, just pointed at generators AWS Labs has written and opinionated toward AWS services. That also means everything else Nx gives you for free, the dependency graph, caching, task running, applies to the projects the plugin generates without any extra setup.

What it actually gives you

@aws/nx-plugin is a collection of generators for Agentic, API, Lambda, Data Storage, Website (inc Auth), IaC on AWS.

End-to-end by default. Generators produce both application code and the accompanying infrastructure (CDK or Terraform), so you're not left in the lurch wiring up deployment separately.
Component-based. You add pieces as you need them, an API here, a website there, rather than committing to one overwhelming opinionated template.
Your code, not a black box. Everything generated is regular source in your repo. If you don't like something, edit it. There's no hidden framework layer to fight later.
Type-safe by design. Where it makes sense, type safety flows through from backend to frontend.
Local-first development. One of the strengths of Nx is you can develop and iterate against pure local dev and testing, alternatively your local code can also be pointed towards your deployed AWS resources.

Worthy callouts

Everything sits inside your monorepo, thanks to Nx Plugin for AWS, you can use it without knowing about Nx which means you get all the benefits of Nx's dependency graph, caching, and task running without the learning curve for your project.

The @aws/nx-plugin is a build-time tool, not a runtime dependency. Once a generator has performed its job, the plugin is no longer relevant.

You can read every file it produced, commit it, and change it however you want afterwards.

This means:

There’s no proprietary learning curve or configuration to wrestle with when you need to do something the generator didn’t anticipate. You take what was given to you and edit the generated code directly, the same way you would edit any code you wrote yourself.
No new layers of abstraction. Generators emit plain code. There’s no opinionated wrapper API or unknown black box getting between you and frontend, backend, CDK or any of the other tools; what you see is exactly what you get.
The plugin isn’t a dependency. It generates code and steps out of the way. Your application doesn’t import or rely on @aws/nx-plugin at runtime, so you are never locked in, and there's no unsightly evidence of it in your code repository either. If you don’t like something in the generated code, you are free to modify it. Generators exist to get you started, not to be a prison with a lock and key.

The generators

Here's the current set of generators available:

Generator	What it does
`ts#project`	TypeScript library
`ts#api`	TypeScript API (tRPC or Smithy) with API Gateway, Lambda, and Powertools
`ts#rdb`	Relational database support via Aurora RDS
`ts#website`	React app (Vite)
`ts#website#auth`	Adds Cognito auth to a website
`ts#infra`	AWS CDK infrastructure project
`ts#lambda-function`	TypeScript Lambda with type-safe event sources
`ts#mcp-server`	MCP server (TypeScript)
`ts#agent`	Strands Agent (TypeScript)
`py#project`	Python project (managed with uv)
`py#api`	Python API (FastAPI) with API Gateway, Lambda, and Powertools
`py#lambda-function`	Python Lambda with type-safe event sources
`py#mcp-server`	MCP server (Python)
`py#agent`	Strands Agent (Python)
`connection`	Wires projects together, e.g. connecting a frontend to a backend API
`terraform#project`	Terraform project, for teams that prefer Terraform over CDK
`license`	Manages LICENSE files and source headers across the workspace

This is a slice, not the full list, the plugin also ships generators for DynamoDB, relational databases in Python, docs sites, and agent-specific infra (AgentCore gateway/harness) among others. See the full generators list in the docs sidebar for everything currently available.

The connection generator is the real workhorse

Sure adding individual components is great, however getting these individual components to play nicely with each other is another.
So my favourite generator: connection is just magic.

It transforms your project in meaningful ways that make sense, i.e connecting a React site to a API actually generates the wiring between the two, rather than leaving you to do it by hand.

A few of the pairings it supports:

React website to an API backend. It generates a type-safe client, sets up the provider your React app needs, and configures auth (IAM or Cognito) so calls are signed correctly. If your API's types change later, your frontend calls stay in sync without you touching the wiring again.
Frontend to an Agent. Same idea as the API case, your website gets a typed way to talk to a deployed agent.
Agent to an MCP server. Agents often need tools, and MCP servers are how the plugin exposes them. connection handles the IAM-authenticated link between an agent and an MCP server so the agent can call its tools without you writing the boilerplate auth and transport code by hand.
Agent to Agent. Connect two agents together using the A2A protocol.
API (or Agents) to Data. Data meaning Relational Database or Dynamo DB.
Service to service, more generally. Anywhere you've got a source project that needs to call a target project's API, connection is the generator that sets up the client, the auth, and the local dev configuration in one go.

For sake of brevity, too many permutations so better to check out list of various supported connections.

Prerequisites

Before you start, you'll need:

Git
Node 22 or later
A package manager: pnpm 10+, yarn 4+, bun 1+, or npm 10+
uv 0.5.29 or later, plus Python 3.12 (uv python install 3.12.0)
AWS CLI configured, profile/SSO credentials for the AWS account you're deploying to
Docker (Docker Desktop on Mac/Windows, the Engine directly on Linux), or Finch 1.6.0+, required for some generators

Optional IDE extensions

You can get a UI over the generators for the major IDEs, which is useful for those devs like me who may not prefer the terminal experience. Both editors' extensions are thin wrappers around the same underlying Nx Language Server, so functionality is on par.

JetBrains official Nx Console plugin for JetBrains IDEs (WebStorm, IntelliJ IDEA Ultimate, and others), built and maintained by the Nx team rather than a community fork, with the same generator UI and project graph view.
VS Code, install the Nx Console extension too.

Scaffolding a project

Here's a typical flow: an API, a AI agent (Strands), a React website with Cognito auth, connected together, with CDK infrastructure to deploy it.

# Create a new workspace
pnpm create @aws/nx-workspace my-project
cd my-project

# Add an API (tRPC)
pnpm nx g @aws/nx-plugin:ts#api --framework=trpc

# Add a Strands AI agent (Python)
pnpm nx g @aws/nx-plugin:py#agent

# Add a React website
pnpm nx g @aws/nx-plugin:ts#website --framework=react

# Add authentication to your website
pnpm nx g @aws/nx-plugin:ts#website#auth

# Connect your website to your API and agent
pnpm nx g @aws/nx-plugin:connection

# Add CDK infrastructure to deploy it all (or choose Terraform)
pnpm nx g @aws/nx-plugin:ts#infra

Once that's done, you can run everything locally with hot reload:

pnpm nx run-many --target=serve --all

Nifty that both your website and API launch with one CLI command, the website will be available at http://localhost:4200, and changes to either the website or the API will hot-reload as you work. When you're ready to deploy, the generated CDK (or Terraform) project handles pushing it all to AWS.

Build and deploy

Once you've defined your cloud resources (e.g. instantiating the generated constructs in CDK, or wiring up the equivalent Terraform modules), building and deploying is a handful of commands:

# Build everything
pnpm build

# Fix any lint errors automatically
pnpm lint

# If the build reports "The workspace is out of sync", sync it
pnpm nx sync

Then bootstrap and deploy your infrastructure project (named infra by default, swap in whatever you actually called yours):

# Bootstrap (one-time per account/region)
pnpm nx bootstrap infra

# Deploy, CDK
pnpm nx deploy-sandbox infra

# Deploy, Terraform (runs plan then apply)
pnpm nx apply infra

Once deployed, pull down the runtime config so your local website points at the real, deployed API and auth resources instead of local ones (again, swap demo-website for your website project's actual name):

pnpm nx load-runtime-config demo-website
pnpm nx serve demo-website

Better together with AI assistants

In this day and age, a tech article would be incomplete if I didn't work AI into the conversation. AI assistants and Nx Plugin are individually great on their own, but together they will give the developer a whole new way of working.

Nx Plugin also has an MCP server available. Workspaces created with the plugin's preset already include automatic project-level MCP config for six common agents: Claude Code, Cursor, Kiro, Gemini CLI, GitHub Copilot, and Codex. Any of those work inside the workspace with no extra setup, and if you're using another MCP-compatible assistant, you can still skip typing generator commands entirely with a bit of manual config.

Then just describe what you want:

"Use the Nx Plugin for AWS to build a full-stack app with a React website, an API, Cognito auth, and CDK infrastructure."

The assistant calls the generators for you through MCP, which is a great way to learn the available options if you have not already familiarised yourself with the documentation, i.e. generator flags available.

Configure your AI assistant

Add it with whichever of these matches your setup:

Claude Code installs it as a plugin: /plugin marketplace add awslabs/nx-plugin-for-aws then /plugin install nx-plugin-for-aws@nx-plugin-for-aws .
Kiro gets a dedicated Power (bundling the MCP server with steering docs and workflow guides) via the Kiro Powers panel, or the Kiro CLI via kiro-cli mcp add.

Most MCP-compatible assistants use this JSON in a configuration file. Add the following:

{
  "mcpServers": {
    "nx-plugin-for-aws": {
      "command": "npx",
      "args": ["-y", "@aws/nx-plugin-mcp"]
    }
  }
}

Cursor and Codex add the same MCP server entry to their respective config files (~/.cursor/mcp.json or ~/.codex/config.toml).
Cline, Roo Code, and other MCP-compatible assistants can support the above json.

Where to from here

I tried out the Quick Start guide from start to finish, it covers the concepts in this article in more depth, and the Dungeon Adventure tutorial is one of the best guided walkthroughs you'll see, with most of the generators used together in a full working example.

Final thoughts

Pre-1.0 status (see the note near the top) hasn't put me off, honestly it is already a solid product and provides a great developer experience.

If you're already building on AWS, I definitely say it is worth a try. Even just using only one generator earns its keep, say connection to wire up an existing frontend and API, can save you a fair amount of repetitive boilerplate.

If you get stuck or want to chat with other people using it, the maintainers hang out in the #nx-plugin-for-aws channel on the CDK Slack.

EKS Auto Mode: correr Kubernetes sin andar peleando con los nodos

Bianca Torres — Fri, 04 Sep 2026 18:54:26 +0000

Si algo aprendí gestionando EKS en producción, es que mantener los nodos se lleva más tiempo del que uno imagina. Managed node groups, Karpenter, el EBS CSI driver, actualizar AMIs, y el clásico "¿por qué este pod lleva 10 minutos en Pending?" a la hora menos pensada. No es lo más entretenido del rol, pero es parte de lo que nos toca hacer para mantener un cluster sano en producción.

Por eso, cuando probé EKS Auto Mode, me hice esta pregunta: "si dejo que AWS haga todo esto por mí, ¿qué gano y qué pierdo?". Este artículo es esa respuesta, con un laboratorio que puedes seguir tú mismo en menos de 30 minutos.

Mantener nodos es casi un segundo trabajo

Antes de vender la solución, hablemos del problema de verdad. En un EKS "clásico", tú te encargas de un montón de cosas:

Crear y dimensionar los node groups. ¿m5.large? ¿c6i.xlarge? ¿Spot u On-Demand? Cada decisión te la cobra el futuro.
Instalar y afinar un autoscaler. Cluster Autoscaler o Karpenter. Los dos piden configuración, permisos IAM y mantenimiento.
Cuidar los add-ons. CoreDNS, kube-proxy, EBS CSI, VPC CNI... instalarlos, versionarlos, actualizarlos.
Parchear las AMIs. Cada vez que aparece una vulnerabilidad de seguridad en el sistema operativo del nodo (CVE), es nuestro trabajo subsanarlo.
Debuggear el scheduling. Ese 0/3 nodes are available: Insufficient memory que aparece justo cuando ya te ibas a desconectar.

Lo irónico es que nada de eso hace tu producto mejor. Es puro "impuesto de plataforma". Trabajo que haces para que Kubernetes exista, no para que tu app sea mejor.

Entonces, ¿Qué es EKS Auto Mode?

AWS se hace cargo del cómputo, el almacenamiento y el networking del cluster por ti. Por dentro usa Karpenter y nodos con Bottlerocket, pero todo eso queda tras bambalinas. Tú ya no administras node groups ni instalas add-ons uno por uno.

Tú declaras tus apps, AWS decide y levanta la infraestructura para correrlas. Los nodos aparecen cuando tus pods los necesitan, y se van cuando ya no.

Lab: de cero a app corriendo, con nodos que aparecen solos

Lo que necesitas:

AWS CLI v2 configurado (aws configure)
eksctl v0.195.0 o más
kubectl
Permisos IAM sobre EC2, EKS, IAM y VPC

Paso 1: Crear el cluster con Auto Mode

eksctl create cluster \
  --name demo-automode \
  --region us-east-1 \
  --enable-auto-mode

Ese único comando te crea el cluster, la VPC, las subredes, activa Auto Mode y deja los add-ons esenciales gestionados por AWS. Tarda aproximadamente unos 15 minutos.

Paso 2: Mirar el cluster recién creado

kubectl get nodes

kubectl get nodepools

Auto Mode crea dos node pools automáticamente:

system arranca con 1 nodo desde el inicio. Ese nodo corre los componentes internos del propio cluster (CoreDNS y los add-ons gestionados). Por eso kubectl get nodes sí te muestra un nodo apenas creas el cluster.
general-purpose arranca con 0 nodos. Este es el pool donde van a caer tus aplicaciones, y ahí es donde se ve la magia: se queda vacío hasta que despliegas algo que necesite cómputo.

Así que el cambio de chip es este: ver el pool de tus cargas en cero no significa que algo se rompió. Significa que todavía no hay nada tuyo que correr, y AWS no está cobrándote nodos de más.

Paso 3: Desplegar una app

# nginx-demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-demo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx-demo
  template:
    metadata:
      labels:
        app: nginx-demo
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        resources:
          requests:
            cpu: "250m"
            memory: "128Mi"
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-demo
spec:
  type: LoadBalancer
  selector:
    app: nginx-demo
  ports:
  - port: 80
    targetPort: 80

kubectl apply -f nginx-demo.yaml

Paso 4: Ver Auto Mode en acción

kubectl get nodes

En unos 30 segundos vas a ver aparecer un nodo nuevo en el pool general-purpose (ese que estaba en cero) sin que tú lo hayas pedido. AWS se dio cuenta de que tus pods necesitan cómputo y levantó una instancia EC2 solo.

Paso 5: Escalar y mirar qué pasa

kubectl scale deployment nginx-demo --replicas=10
kubectl get nodes

Al escalar en numero de replicas de nuestra app, tener un solo nodo será insuficiente. Si la capacidad de ahora no alcanza, Auto Mode levanta más. No tocaste ningún autoscaler, no editaste ningún node group.

Paso 6: Destruir

eksctl delete cluster --name demo-automode --region us-east-1

Es super importante que al finalizar nuestro laboratorio destruymos todos los recursos creado, para evitar costos innecesarios. Un cluster olvidado es una sorpresa fea en la factura.

Clásico vs Auto Mode

Qué	EKS clásico	EKS Auto Mode
Gestión de nodos	Manual / tu propio Karpenter	AWS
Add-ons (CoreDNS, CSI, etc.)	Los instalas y versionas tú	Gestionados
Actualizar AMIs	Lo gestionas tú	AWS
Autoscaling	Configuras CA o Karpenter	Automático
Tu tiempo operando	Mucho	Poco
Personalización	Total	Acotada
Curva de arranque	Empinada	Suave

Cuándo usarlo y cuándo no (Según mi opinión)

Úsalo si:

Arrancas un cluster nuevo y quieres velocidad sin construir toda una plataforma.
Tu equipo es chico y no quieres tener a alguien dedicado a cuidar nodos.
Tus workloads son bastante estándar: APIs, workers, web.

Piénsalo dos veces si:

Dependes de configuraciones muy específicas de nodo (kernels custom, DaemonSets exigentes, GPUs con drivers a medida).
Ya tienes un Karpenter afinado que te ahorra dinero y no lo quieres soltar.
Necesitas control total del ciclo de vida de la AMI por temas de compliance.

Auto Mode te cambia control por simplicidad. Ganas horas de operación, cedes algunos grados de personalización. Para la mayoría, ese cambio vale muchísimo la pena. Para casos muy puntuales, todavía no.

¿Y cuánto me va a costar?

Auto Mode suma un pequeño cargo de gestión encima del costo del EC2 que corre por debajo. Pero no te quedes solo con "cuánto cuesta el add-on". La cuenta real es: cuánto cuesta el add-on menos las horas de ingeniería que dejas de gastar cuidando nodos, autoscalers y parches. Cuando sumas ese tiempo, el número se ve muy distinto.

En conclusión

EKS Auto Mode no es magia. Es AWS haciendo el trabajo de plataforma que muchos veníamos haciendo a mano. Si gestionas clusters en producción, pruébalo con un workload que no sea crítico y mide cuánto tiempo dejas de invertir en cosas que no hacen mejor tu producto.

AWS Has Two Iceberg REST Catalogs: What Each One Actually Serves

xbill — Fri, 04 Sep 2026 18:29:18 +0000

This article provides a step by step comparison of the two Apache Iceberg REST catalog implementations AWS ships — AWS Glue and Amazon S3 Tables — measuring what each one actually serves. A Python probe harness issues one identical request suite to both and stores the raw response as evidence.

https://github.com/xbill9/lakehouse-iceberg-2026

AWS ships two Iceberg REST catalogs. Both are managed, both are SigV4-signed, and both implement the same published specification. If you are choosing between them, or writing a client that has to work against either, the interesting question is where they diverge.

They score identically and behave differently in thirteen places.

All results below were measured on 2026-09-03.

What Is the Iceberg REST Catalog?

An Iceberg table is a directory of Parquet files plus a chain of JSON metadata files recording which files belong to the table right now. Something has to hold the pointer to the current metadata file, and make commits atomic by swapping it. That is the catalog.

The REST catalog is one HTTP API for that job, so an engine needs one driver rather than one per catalog. The specification lives in the Iceberg repository as open-api/rest-catalog-open-api.yaml and defines 35 operations.

$ curl -sL https://raw.githubusercontent.com/apache/iceberg/main/open-api/rest-catalog-open-api.yaml -o irc.yaml
$ grep -cE '^    (get|post|delete|head|put):' irc.yaml
35

This harness probes 25 of those 35, or 71%.

At This Point You Should Have

An AWS account, and credentials with enough privilege to create a Glue database, an S3 bucket and an S3 Tables table bucket
Python 3.13 with requests, pyiceberg and botocore
botocore available for SigV4 signing

$ aws sts get-caller-identity --query Arn --output text
arn:aws:iam::AWS_ACCOUNT_ID:root
$ python3 -c "
import sys, pyiceberg, requests, botocore
print('python    ', sys.version.split()[0])
print('pyiceberg ', pyiceberg.__version__)
print('requests  ', requests.__version__)
print('botocore  ', botocore.__version__)"
python     3.13.13
pyiceberg  0.12.0
requests   2.34.2
botocore   1.43.34

Everything below was measured with account root, so no result here is a permissions artefact.

Where Are the Two Endpoints?

They are different services with different signing names.

	AWS Glue	Amazon S3 Tables
Endpoint	`https://glue.us-east-1.amazonaws.com/iceberg`	`https://s3tables.us-east-1.amazonaws.com/iceberg`
SigV4 signing name	`glue`	`s3tables`
Warehouse	the account id	the table bucket ARN

$ # GET /v1/config against Glue
{"defaults":{"header.Content-Type":"application/x-amz-json-1.1","rest.sigv4-enabled":"true",
 "rest-table-scan-enabled":"true","prefix":"AWS_ACCOUNT_ID","rest.signing-region":"us-east-1",
 "rest.signing-name":"glue", ...}}

Signing Is Where the First Hour Goes

SigV4 signs the exact query string, so the URL you sign must be the URL you send. Building the URL and then letting an HTTP client re-encode the parameters separately produces a different canonical string:

$ # signing one URL, sending another
HTTP 403
{"message":"The request signature we calculated does not match the signature you
provided. Check your AWS Secret Access Key and signing method."}

Build it once and send it whole. urlencode's default quote_plus is also wrong here, because SigV4 wants %20 rather than +. With both fixed, the same request reaches the service and returns a real answer:

$ # same probe, correct canonical query string
HTTP 404
{"error":{"code":404,"message":"The specified bucket does not exist.","type":"no_such_bucket"}}

A 403 signature error and a 404 no-such-bucket look equally like failure in a log. Only one of them is about the catalog.

Bringing Up AWS Glue

$ aws s3api create-bucket --bucket $BUCKET --region us-east-1
$ aws glue create-database --region us-east-1 \
    --database-input "{\"Name\":\"probe_ns\",\"LocationUri\":\"s3://$BUCKET/probe_ns/\"}"
$ aws glue get-databases --region us-east-1 --query 'DatabaseList[].Name' --output text
probe_ns

Glue rejects createTable without an explicit table location:

$ # createTable with no location field
InvalidInputException: Location information cannot be null while creating an iceberg table

Every other catalog in this suite infers it from the warehouse. Glue does not.

Bringing Up Amazon S3 Tables

$ ARN=$(aws s3tables create-table-bucket --name iceberg-probe \
    --region us-east-1 --query arn --output text)
$ echo $ARN
arn:aws:s3tables:us-east-1:AWS_ACCOUNT_ID:bucket/iceberg-probe
$ aws s3tables create-namespace --table-bucket-arn "$ARN" \
    --namespace probe_ns --region us-east-1
{"tableBucketARN": "arn:aws:s3tables:us-east-1:AWS_ACCOUNT_ID:bucket/iceberg-probe",
 "namespace": ["probe_ns"]}

Two constraints surface only when you hit them. Namespace names reject uppercase, which a timestamped scratch namespace will contain:

$ # createNamespace named irc_probe_20260903T1626
HTTP 400
The specified namespace name isn't valid. Specify a different namespace name, and
then try again.

And createTable requires stage-create in the body:

$ # createTable without it
HTTP 400  stage-create is a required field and cannot be null

Seeding a table also failed against the managed bucket with pyiceberg's default writer:

$ # appending with the default PyArrow FileIO
AWS Error [code 134] during CreateMultipartUpload operation: The authorization
mechanism you have provided is not supported. Please use Signature Version 4.

Switching to pyiceberg.io.fsspec.FsspecFileIO worked. Note the bucket name in that error is not the one you created — S3 Tables stores data in a managed bucket of its own.

The Prefixes Do Not Look Alike

Every client reads the routing prefix from /v1/config and puts it in every later URL. The two services return very different shapes:

Catalog	Prefix as returned
Glue	`catalogs/AWS_ACCOUNT_ID`
S3 Tables	`arn%3Aaws%3As3tables%3Aus-east-1%3A...%3Abucket%2Ficeberg-probe`

Glue returns two path segments. S3 Tables returns a percent-encoded ARN. A client that assumes one segment, or that re-encodes what it was handed, produces URLs neither service routes.

The Scores Are Identical

	🥈 AWS Glue	🥈 Amazon S3 Tables
Read probes served	9/15	9/15
Write probes served	10/17	10/17
Not tested	1	1
`loadTable` fields present	26/30	27/30
Endpoints declared in `/v1/config`	none	none

Read and write surfaces are scored separately rather than summed, and probes whose prerequisite failed are excluded rather than counted as failures.

That table is the least interesting thing in this article. The two implementations arrive at the same totals by different routes.

Neither One Tells You What It Supports

The specification lets a server advertise its own surface:

endpoints: A list of endpoints that the server supports.

— rest-catalog-open-api.yaml, CatalogConfig

Five of the seven catalogs in the wider comparison publish that array. Neither AWS catalog does:

$ python3 -c "
import json
for c in ('aws-glue','aws-s3tables'):
    d = json.load(open('evidence/%s.json' % c))
    print('%-14s declares %s endpoints' % (c, len(d.get('declared_endpoints') or []) or 'no'))
"
aws-glue       declares no endpoints
aws-s3tables   declares no endpoints

The field is optional, so this is not a specification violation. It does mean capability discovery is unavailable on both, and a client has no way to learn what either serves short of probing it — which is what this harness does.

Thirteen Probes Behave Differently

Of 33 probes, 20 return the same verdict on both and 13 do not.

Probe	AWS Glue	Amazon S3 Tables
`list_views`	❌ 406 not supported	❌ 404 unknown operation
`create_view`	❌ 406 not supported	❌ 404 unknown operation
`load_view`	❌ 406 not supported	❌ 404 unknown operation
`replace_view`	❌ 406 not supported	❌ 404 unknown operation
`rename_view`	❌ 406 not supported	❌ 404 unknown operation
`drop_view`	❌ 406 not supported	❌ 404 unknown operation
`rename_table`	❌ 406 not supported	✅ 204
`update_namespace_props`	✅ 200	❌ 404 unknown operation
`plan_table_scan`	⚠️ 200 with exception	❌ 404 unknown operation
`report_metrics`	⚠️ 200 with exception	❌ 400 not supported
`commit_transaction`	⚠️ 200 with exception	❌ 404 unknown operation
`drop_table_purge`	❌ 400 purge forbidden	✅ 204
`drop_table`	✅ 204	❌ 400 purge required

Neither implements views, but they refuse differently, and one of them renames tables while the other does not.

Glue Names the Operation It Is Refusing

Glue's refusals are specific, and they name the endpoint:

$ # POST .../views against Glue
HTTP 406  CreateView endpoint is not supported for Glue Catalog.
$ # POST .../tables/rename against Glue
HTTP 406  RenameTable endpoint is not supported for Glue Catalog.

S3 Tables returns a bare XML document with no operation name and no JSON error body:

$ # POST .../views against S3 Tables
HTTP 404  <UnknownOperationException/>

Both mean the same thing to a user and not to a program. Glue's 406 with a named operation is machine-readable enough to log usefully; the bare exception is not.

The exception to S3 Tables' silence is report_metrics, which is the one place it says what it means:

$ # POST .../tables/{table}/metrics against S3 Tables
HTTP 400  ReportMetrics is currently not supported.

Glue Answers Three Unrouted Operations With HTTP 200

This is the finding a client author should care about most.

$ # POST .../tables/{table}/plan against Glue
HTTP 200
{"Output": {"__type": "com.amazon.coral.service#UnknownOperationException"}, "Version": "1.0"}

The same body comes back from report_metrics and commit_transaction. The mechanism is the AWS protocol layer answering an operation its front door does not route, rather than Glue returning a broken success for an implemented endpoint.

The consequence does not depend on the mechanism. Code that branches on the status code sees three endpoints that work, and only code that parses the body finds out otherwise. The harness gives these their own verdict rather than scoring them OK.

Glue's /v1/config also advertises rest-table-scan-enabled: true, and scan planning is one of the three.

The Drop Requirements Are Opposite

Glue refuses a purge drop. S3 Tables refuses a plain one.

$ # DELETE .../tables/{table}?purgeRequested=true against Glue
HTTP 400  PurgeRequested cannot be true for Glue iceberg tables.

$ # DELETE .../tables/{table} against S3 Tables
HTTP 400  DropTable operation failed. S3 Tables only supports dropping tables with
purge enabled.

The two products have different storage models, so this is defensible rather than a defect — Glue points at a bucket you own, and S3 Tables owns the storage it drops. It is still the sharpest example of why "AWS supports the Iceberg REST catalog" is not a sentence a client can act on.

Neither Supports Multi-Level Namespaces

Both reject the parent query parameter on listNamespaces, and both say so plainly:

$ # GET .../namespaces?parent=probe_ns against Glue
HTTP 400  Glue dataCatalog does not support multipart namespace.
$ # the same against S3 Tables
HTTP 400  Multipart namespaces are not supported.

Where They Agree

Twenty of the 33 probes return the same verdict, and the agreement is the core of the specification. Both serve config, namespace listing and loading, table listing and loading, loadTable with full snapshot history, and both accept all five updateTable actions probed — set-properties, remove-properties, add-schema, set-current-schema and upgrade-format-version.

Both also return nearly identical loadTable documents: 26 of 30 checked specification field paths on Glue, 27 on S3 Tables. The single extra on S3 Tables is a config block. Whatever separates these two, it is not the fidelity of the metadata they return.

Summary

The goal of this article was to measure what AWS's two Iceberg REST catalog implementations actually serve, rather than what "supports the REST catalog" implies. The key to the solution was issuing one identical request suite to both and storing the raw response for every probe. The comparison results were:

Identical totals, thirteen behavioural differences. Both serve 9 of 15 read probes and 10 of 17 write probes, and 20 of 33 probes agree.
Neither publishes an endpoints declaration, so capability discovery is unavailable on both and a client must probe.
Glue answers three unrouted operations with HTTP 200 carrying an UnknownOperationException, one of which is the scan planning its own config advertises as enabled.
Neither implements views, and they refuse differently — Glue with a 406 naming the operation, S3 Tables with a bare <UnknownOperationException/>.
The drop requirements are opposite. Glue forbids purgeRequested=true, S3 Tables requires it.
rename_table works on S3 Tables and not on Glue.

Scope: both catalogs probed once in us-east-1 on 2026-09-03 with account root, against one table shape seeded through pyiceberg 0.12.0, covering 25 of the specification's 35 operations, 5 of its 25 update actions and 1 of its 8 table requirements; neither service exposes a version, so no result here can be tied to a release; and the field tier records that a value is present, never that it is correct.

The strategy for comparing two managed catalogs against one specification was validated with an incremental step by step approach.

Build a Custom AWS Cognito Login Page with Passkeys

Rishi — Fri, 04 Sep 2026 15:30:45 +0000

Passkeys are replacing passwords. Cognito’s Hosted UI supports them out of the box. But Cognito also supports building a custom login page. I spent a weekend figuring out how to bring that same passkey experience to a custom login page.

In this blog post, we will understand how passkeys work and how to create a custom passkey-supported login page using React and AWS Cognito. Along the way, we will also explore how things work under the hood.

What are Passkeys?

Passwords are both difficult to remember and easy to lose. Passkeys bring a passwordless authentication paradigm based on public-key cryptography.

You can think of a website’s login screen as a lock. Passkeys allow your device (or pass manager) to become the key for that lock.
But… but… but…
The interesting part is that the device can prove it has the key without ever handing the key over.

During registration, a public-private key pair is created by your device (or passkey manager). The private key is kept protected on your device, and the public key is passed to the website.

At the time of login, the website sends a fresh challenge to your device. After you verify yourself, your device uses the private key to sign the challenge. The website then uses the public key to verify the signature. If the signature is valid, you’re logged in.
ie. Challenge → Sign with private key → Signature → Verify with public key → ✅

When we use Cognito's Hosted UI, much of this flow is handled for us. But when we build our own login page, we need to connect our UI to the browser's passkey APIs and Cognito ourselves. And that's what we are going to do in the next step!

Setting up Amazon Cognito for Passkeys

First of all, let us create a Cognito user pool with a public client. Then make the necessary changes to enable passkeys. If you are stuck at any step, please refer to the video for more details.

Step1: Go to client settings and make sure Choice-based sign-in: ALLOW_USER_AUTH is enabled. It makes sure entity logging in can select the sign-in method.

Step2: Go to User pool >> Authentication >>Authentication methods >> Passkey and modify the relying party ID. This should be the same as the domain of the login page. For me, it is “localhost” but if you are using a domain name, it will be like “example.com”.

Step3: Navigate to User pool >> Authentication >> Sign-in >> Options for choice-based sign-in. Then enable passkeys as an available sign-in choice.

Also, make sure MFA is either optional or disabled. And you are good to move on to the next section.

Building the Custom Login Page

Frontend code is available here: https://github.com/TrickSumo/cognito-custom-login-passkey-support
Make sure to create a .env file with the details of the user pool created in the last step. Then install dependencies and run the code.

npm i
npm run dev

The frontend uses @aws-sdk/client-cognito-identity-provider for Cognito API calls and @simplewebauthn/browser for handling the WebAuthn browser API calls for passkey registration and authentication.

Registering a Passkey

Create a Cognito user with the help of the signup form. After successful login, register a passkey using the “Add a passkey” button. The browser will prompt you to save the passkey on the device.

This registration process involves two API calls (you can inspect them in the network tab).

The first call is made to Cognito with the header “x-amz-target: AWSCognitoIdentityProviderService.StartWebAuthnRegistration” and it initiates registration of a passkey. If you notice payload, it contains the access token that you received after successful login.

The response of the first call contains the random challenge and other details:

Then the browser’s navigator.credentials.create() API is called to generate a new key pair on the device. The private key is stored securely on the device and never leaves it. The device then uses that private key to sign the challenge received from the first Cognito API call, proving a real authenticator generated this response. The new public key and the signed proof are then packaged together for the next step.

Then the second API call goes to Cognito with the header “x-amz-target:AWSCognitoIdentityProviderService.CompleteWebAuthnRegistration” and attestationObject/publicKey as the payload. This attestationObject is an encoded value that contains the public key, metadata about the authenticator, and the signature (computed over the clientDataJSON, which itself repeats the original challenge along with the page’s origin).

clientDataJSON after decoding from base64:

If all is good, the response from the second Cognito API is… Guess what? Empty JSON object {}.

Signing in with a Passkey

Logging in with a passkey also involves two API calls to Cognito.

The first one is an initiate auth API call with “PREFERRED_CHALLENGE: WEB_AUTHN”. It returns the challenge:

Then the browser navigator.credentials.get() API is called to produce a signed assertion using the private key.

After that, the second API call to Cognito happens with the RespondToAuthChallenge header and signature as payload. Cognito verifies the payload using the public key that we stored during passkey registration. After successful login, accessToken, idToken, and refreshToken are issued.

Managing Passkeys

The same user can have multiple passkeys, and the good news is that Cognito makes it easy to manage passkeys.

You can call Cognito with the ListWebAuthnCredentials and DeleteWebAuthnCredential header/command to list and delete existing passkeys.

These functionalities are already available in the demo code (using the Cognito SDK). Feel free to explore!

Gotchas!

Make sure passkeys are enabled in the Cognito user pool. Also, the Relying Party ID must match your app’s domain.

If MFA is set to “Required” on the user pool, passkeys quietly stop showing up as a sign-in option even if everything else is configured correctly. Keep MFA optional or off if you want passkeys to work.

A passkey can only be registered by a user who’s already signed in ie. signup can’t create one directly

Conclusion

In our implementation, we used Cognito APIs together with the browser’s WebAuthn APIs to register and authenticate users with passkeys.

The nice part is that we don’t have to implement the cryptography ourselves. The browser and passkey provider handle the sensitive operations, while Cognito handles authentication and token issuance.

Hope you enjoyed the tutorial 😊

I Took the Wheel Off My RAG: Managed KB vs. Manual Chunking

Gerardo Arroyo — Fri, 04 Sep 2026 13:39:17 +0000

In April I published a benchmark comparing 5 chunking strategies in Amazon Bedrock Knowledge Bases. The practical conclusion was simple: FIXED_SIZE with S3 Vectors as the backend, and that's it, unless your data justifies something more complex.

Three months later, AWS took the question away from the problem entirely. Amazon Bedrock Managed Knowledge Base doesn't ask you to pick a chunking strategy. Smart Parsing decides parsing by document type, AWS manages the vector store, and if your questions are complex, AgenticRetrieveStream plans a retrieval strategy instead of running a single similarity search.

Sounds like April's discussion is obsolete. So I asked the question that felt honest to me: how much does it cost in quality to have the wheel taken away from you, and under what conditions do you get that loss back?

I didn't have a thesis before running this. I have one now, and it's not the one I expected.

🎯 Spoiler: Simple managed retrieval practically ties my manual April configuration. What didn't work is the agentic planner —the piece that justifies the "agentic" in the name— with the model AWS gives you by default: it didn't decompose a single query across 80 evaluated questions.

This article covers the full methodology: five retrieval configurations over the same corpus, two question sets —the original 25 single-hop questions from April untouched, and 15 new multi-hop questions written against the actual document text—, three different planners, and the 6 infrastructure gotchas I had to resolve before I could measure anything. Because the planner finding has a twist: it depends entirely on which model does the planning, and I didn't know that until I stopped using the one that ships by default.

📌 TL;DR — Key data before you keep reading

Managed KB with simple Retrieve ≈ my manual April configuration in Correctness (0.88 vs. 0.84, a difference within the expected range).

Smart Parsing ingests without failing the two PDFs that broke SEMANTIC and NONE in April — and does it faster than my manual pipeline (183s vs. 407s). It even ingests and correctly retrieves a PDF with no extractable text layer.

AgenticRetrieveStream with the default planner (MANAGED) never generated a single sub-query across 80/80 evaluated questions, including ones a simple retrieval demonstrably couldn't solve.

On multi-hop questions, that made the "agentic" retriever land below simple retrieval (0.50 vs. 0.567 Correctness).

With a small CUSTOM planner (Claude Haiku 4.5), real decomposition does show up — but on questions that no longer needed it, not on the 5 that did.

With a large CUSTOM planner (Claude Sonnet 4.6, the same model as my generator), multi-hop Correctness jumped from 0.50 to 1.00, sustained across two independent runs — but the planner and generator share the same model in that test, and I'm stating that explicitly as a caveat, not as fine print.

The corpus isn't byte-identical to April's: my original bucket was destroyed and I had to re-download the documents. I'm saying that explicitly, not hiding it.

Why This Matters to Me

This isn't academic curiosity. If you're evaluating whether to migrate a production RAG to Managed Knowledge Base, the real question isn't "is it better?" — it's "what do I lose by giving up control of chunking, and what do I gain if I also let the service decide when and how to search more than once?"

That second part is what almost nobody measures. AWS presents AgenticRetrieveStream with numbers from an academic benchmark (MuSiQue) showing gains of up to +37 recall points on 4-hop questions. Those are real numbers, published on their official agentic retrieval blog post, and I break them down further down. But an academic benchmark doesn't tell you what happens when the planner you get by default, without touching a single parameter, meets your own corpus.

That's what I ran.

The Design: Five Configurations, Two Question Sets

I kept the generator (Claude Sonnet 4.6) and the judge (Nova Pro, cross-family) constant across four of the five configurations —the only exception is D, where the service itself generates— to isolate the retrieval layer as the sole variable:

Config	Retrieval	Generation	Question it answers
A	S3 Vectors + `FIXED_SIZE` (April)	Sonnet 4.6	Baseline
B	Managed KB, `Retrieve`	Sonnet 4.6	Did I lose quality by handing over the wheel?
C	Managed KB, `AgenticRetrieveStream` (`generateResponse=False`, planner `MANAGED`)	Sonnet 4.6	Does the default planner work?
D	Managed KB, `AgenticRetrieveStream` (`generateResponse=True`, planner `MANAGED`)	The service itself	What if I let AWS generate too?
E	Managed KB, `AgenticRetrieveStream` (`generateResponse=False`, planner `CUSTOM` = Sonnet 4.6)	Sonnet 4.6	Does a large planner help, not just a different one from the managed default?

The first four ran together. I added E a day later, as a direct response to what I found with C — I cover it in detail further down, in its own section, so as not to mix an exploratory run with the original pre-registered design.

And two question sets over the same corpus of 3 technical documents (the Well-Architected Framework, the AgentCore developer guide, and a RAG evaluation blog post):

1-hop set: the 25 original single-hop questions from April. Untouched, for comparability.
Multi-hop set: 15 new multi-hop and comparative questions, written against the actual document text, not against summaries.

🔍 ProTip #1: If you're going to compare an "agentic" retriever against a simple one, you need two question sets, not one. AWS itself reports that its gain on single-hop questions is under 5 recall points. Running only single-hop questions against a retriever that plans is measuring something you already know in advance won't show a difference — that's not a benchmark, it's a confirmation.

A fairness control that cost me time but was non-negotiable: I re-ran configuration A in full, I didn't reuse the scores published in April. Those came from the native retrieveAndGenerate path; B, C, and D can only be evaluated through the bring your own inference responses (BYOI) path. Comparing scores produced by two different evaluation mechanisms would have been exactly the kind of methodological trap I criticize other benchmarks for.

First Gotcha, Before Writing a Single Line of Retrieval Code

My original plan was to reuse April's Knowledge Base as-is. I couldn't.

April's KB and its corpus bucket no longer existed. That wasn't an oversight — I'd intentionally destroyed them after publishing the article, as I do with almost all my benchmark infrastructure. Verified against the real account before assuming anything: list-knowledge-bases returned zero results across five regions, the S3 Vectors bucket had no buckets, and the corpus bucket returned NoSuchBucket.

I had to recreate just the FIXED_SIZE module from April's repo (Titan v2, 1024 dimensions, 512-token chunks, 20% overlap) — without the other 4 chunking modules, which weren't needed for this benchmark.

And here's the uncomfortable part I do need to say: the corpus couldn't be byte-identical to April's. The data folder was in the .gitignore of the original repo — I never versioned the actual content, so there was no hash or manifest to compare against. Verified by HTTP HEAD:

Document	April (per README)	Today	Conclusion
`bedrock-agentcore-dg.pdf`	~17 MB	30,420,374 bytes	Nearly double. Confirmed NOT identical.
`wellarchitected-framework.pdf`	~14 MB	14,189,927 bytes	Similar size, but without a hash I can't claim exact identity.
`blog-rag-evaluation.html`	—	Modified August 18	Touched after April.

The AgentCore developer guide practically doubled in size over four months, which makes sense — it's the service that's evolved the most in that period. I'm reporting this as an explicit limitation of the re-run. I'm not hiding it, not minimizing it, and I also don't think it invalidates the comparison: the documents are still the same class of content (dense AWS technical documentation), which is what April's benchmark needed to be representative.

⚠️ ProTip #2: If you're going to publish a benchmark you plan to revisit months later, version the corpus with a public hash (even if you don't upload the full files). I didn't do that in April, and it cost me the ability to claim "byte-identical" with evidence, not just intent.

Second Batch of Gotchas: Creating and Querying a Managed KB Isn't Like a Regular One

Six real infrastructure problems, none of them documented together anywhere I could find.

1. Retrieve against a Managed KB rejects vectorSearchConfiguration. My April code used that parameter with no issue against the S3 Vectors KB. Against the Managed KB, the service responded:

ValidationException: Incompatible configuration: vectorSearchConfiguration
is not supported for managed knowledge bases. Use managedSearchConfiguration
instead.

Same internal shape, different container key. Config A uses one, config B needs the other.

2. Creating an S3 data source with type=S3 fails on a Managed KB. The error:

ValidationException: Unsupported data source type for MANAGED knowledge
base type.

The correct form —which I only found against a real AWS example, not against the shape's reference documentation— is type=MANAGED_KNOWLEDGE_BASE_CONNECTOR, with the connector's configuration nested one level deeper than feels intuitive.

3. Creation is asynchronous in a way I didn't expect. With Terraform, config A accepts CreateKnowledgeBase and CreateDataSource back-to-back with no waiting. Against a Managed KB, calling CreateDataSource while the KB is still CREATING fails with ConflictException. You have to poll until AVAILABLE before continuing — typically 2-5 minutes.

4. ragSourceIdentifier in the eval job isn't a free-form label. I tried giving it a descriptive name ("D-setA") and the service rejected it: it has to match exactly the knowledgeBaseIdentifier carried by each line of the BYOI dataset, or it fails with ValidationException.

5. The eval job's output schema is different from the one documented for "automated MODEL evaluation". That documented schema uses automatedEvaluationResult.scores at the root. The real one, confirmed against raw output in S3, is conversationTurns[].results[], with a single JSONL per job at a path the service generates.

6. The judge sometimes can't extract a score. A real example, verified against the raw JSONL: for the question "Summarize the design principles of the Operational Excellence pillar" (config A, 1-hop set), Nova Pro answered in natural language with a perfectly coherent explanation, but Builtin.Faithfulness came back result: null with the message Unable to parse score from the LLM judge response. It's not that the judge failed to reason — the service simply couldn't parse a number out of that specific response. I'm reporting it and counting it in the final tally, not averaging it away by hiding it.

Figure 1: The judge's response is coherent and correctly lists the principles, but the service failed to extract a numeric score from that text. The error is documented right in the console.

With that resolved, I was able to confirm something that broke in my favor: Smart Parsing ingests without complaint the two PDFs that broke SEMANTIC (1 MB limit) and NONE (50,000-character limit) in April. The job finished COMPLETE, 3 of 3 documents indexed, 0 failed, in roughly 183 seconds — faster than my manual FIXED_SIZE + S3 Vectors pipeline over the same corpus (~407 seconds).

I went a step further to really stress-test Smart Parsing: I generated a synthetic 2-page PDF with no extractable text layer at all —confirmed with pypdf that extract_text() returns an empty string on both pages, only content rasterized to image— and added it to the corpus. The ingestion job finished COMPLETE all the same, no failures, no skipped documents. And it didn't stop at "accepted it in name only": a direct Retrieve with a query about that PDF's content returned the correct text as the top result, score 0.981, pointing exactly at the file. The Knowledge Bases documentation does list scanned documents among the types Smart Parsing automatically selects for, but it doesn't name the mechanism: that it runs OCR internally is something I'm inferring from the result, not from an explicit promise.

That's already a data point with real weight: if your corpus has large or scanned files that break manual chunking, Managed KB solves a real problem for you without you having to think about it.

The Results

Nova Pro judge and BYOI across all configurations, with the same scripts for A, B, C, and D. A single run per cell — not averaged over multiple executions, so take it as signal, not statistical certainty.

Figure 2: All 8 combinations of the original design (configurations A-D × 2 sets) finished Completed in the RAG evaluations console. None got stuck halfway or failed silently.

Config	Set	Correctness	Completeness	Faithfulness	Helpfulness
A	single-hop (n=25)	0.84	0.66	0.875	0.847
B	single-hop (n=25)	0.88	0.68	0.95	0.880
C	single-hop (n=25)	0.88	0.67	0.94	0.867
D	single-hop (n=25)	0.94	0.92	0.86	0.940
A	multi-hop (n=15)	0.733	0.783	0.783	0.822
B	multi-hop (n=15)	0.567	0.517	0.60	0.778
C	multi-hop (n=15)	0.50	0.50	0.55	0.778
D	multi-hop (n=15)	0.70	0.70	0.55	0.878
E	single-hop (n=25)	0.90	0.70	0.91	0.873
E	multi-hop (n=15)	1.00	0.883	0.917	0.911

Row E ran a day after A-D, as a direct follow-up to the planner finding — the full detail, including the caveat that the planner and generator share the same model there, is in its own section further down.

And the verdict on the five hypotheses I pre-registered before running anything:

#	Hypothesis	Result	Verdict
H1	B ≈ A in Correctness (single-hop), ±0.05	0.88 − 0.84 = 0.04	Consistent
H2	C ≈ B on single-hop (the planner shouldn't add anything)	Identical Correctness: 0.88 = 0.88	Consistent
H3	C > B by a wide margin on multi-hop	0.50 < 0.567 — C lands below B	Refuted (with the `MANAGED` planner)
H4	C's latency ≥ 3× B's	~2.7s vs. ~0.6-0.9s on single-hop; 3.07s vs. 0.67s on multi-hop (4.6×)	Consistent
H5	D ≤ C in Faithfulness	0.86 ≤ 0.94 on single-hop; tied at 0.55 on multi-hop	Consistent

Figure 3: RAG evaluations' native comparison view, config B (simple Retrieve) against config C (AgenticRetrieveStream) on the multi-hop set. The "agentic" retriever loses on 3 of 4 metrics and ties on the fourth.

Four of five hypotheses held. The one that fell is the interesting one, and it didn't fall by a little.

With a scope caveat on H3. What that hypothesis evaluates is the MANAGED planner, the default. Further down you'll see that with a large CUSTOM planner, the pattern H3 predicted does show up — and that doesn't rescue it, because it was refuted exactly as I wrote it, for the planner it measured. What it does leave is the question I chase through the rest of the article: is what matters the planner's size, or the fact that a large planner ends up sharing a model with the generator?

The Main Finding: The Planner Never Planned

H3 isn't a "didn't help much." It's a measured regression: on the question set where the agentic retriever should shine, it ended up worse than single-step simple retrieval.

And it has an identified mechanical cause, not a statistical mystery. I reviewed the full trace of all 80 agentic questions run (configurations C and D, both full sets). In 100% of cases, the structure was identical: a single Retrieval step —the speculative pass with the raw query— followed by a Planning step that returned "actions": []. A second Retrieval step never appeared. FullDocumentExpansion never appeared. The planner never generated a single sub-query, not even on the multi-hop set questions where I already knew —from a separate check I did before running the benchmark— that single-step retrieval didn't reach both relevant documents.

A concrete example. For the question "which AgentCore component applies logic similar to the Security pillar's 'Keep people away from data' for code execution?", the initial speculative Retrieval brought back only content from the AgentCore developer guide —Code Interpreter, session isolation, security controls— and nothing from the Well-Architected Framework. The subsequent Planning step decided that was already enough.

Figure 4: Raw trace for this same question. After the single Retrieval step, the Planning event with "message": "Agent planning completed" returns "actions": [] — the planner considered the work done without generating a single sub-query.

🎓 ProTip #3: If you're going to measure "iterations" of an agentic retriever, don't count trace steps — count real sub-queries. A Planning step that returns actions: [] still counts as an iteration in the iterations field the SDK exposes, but it doesn't represent any decomposition work. It's the difference between "the planner ran" and "the planner planned".

Is This a Bug? I Went to Verify It Against the Source

Before publishing this as a finding, I checked it against the official AWS post dedicated entirely to this capability, Agentic retrieval for Amazon Bedrock Managed Knowledge Base — the same one that's the source of the MuSiQue numbers I cite above. Three things get confirmed, one gets nuanced.

It confirms this isn't a problem with my data capture. The documentation describes the Retrieval/FullDocumentExpansion step as "one event per executed sub-query." My code logs any step that arrives in the stream, with no name filtering. If the planner had decomposed even once across 80 questions, it would have shown up in the trace. It never did.

It confirms that exiting early is documented behavior, not a bug. The docs explicitly say the planner "may exit early after its evaluation step determines the evidence is sufficient." 100% of my cases are exactly that — a normal service state, not a failure.

It confirms the MuSiQue numbers I already cited, with their exact values: +22.8, +31.9, and +37.3 recall points on 2-, 3-, and 4-hop questions, with gains under 5 points on single-hop questions.

And it nuances something important: the post's two official code examples —single-KB and multi-KB— use foundationModelType: "CUSTOM" with an explicit model. Neither uses the default value (MANAGED). The same post recommends as best practice "starting with a small, fast planner," which implies deliberately choosing a model, not trusting the default.

That leads me to the most honest conclusion I can give: I can't claim "AgenticRetrieveStream doesn't decompose" in general. I can only claim, with evidence from 80 of 80 questions, that the default managed planner —the one anyone who hasn't read this AWS blog post gets— never exercised its decomposition capability on my corpus, not even when the retrieved evidence was insufficient. That all the official reference documentation uses CUSTOM is, in itself, a data point: it suggests AWS doesn't expect the managed model to reliably decompose either, without the user choosing their own planner.

I Tried the Obvious Fix: My Own Planner

I configured foundationModelType=CUSTOM with Claude Haiku 4.5 as the planner —a "small, fast" model, exactly what AWS recommends, and different from both the generator (Sonnet 4.6) and the judge (Nova Pro), so as not to contaminate either role.

I kept this run separate; it doesn't replace configuration C's official results, which remain with the managed planner, because that's what the benchmark's design deliberately measures.

Something did change. With Haiku 4.5 as the planner, 3 of the 15 multi-hop set questions showed planning_actions > 0 — the planner generated real sub-queries, something that never happened even once across the 80 questions with the managed planner. On the single-hop set, 0 of 25 decomposed, which is consistent: it isn't needed there.

But it didn't change what actually mattered. The 5 cross-document questions where simple Retrieve demonstrably failed to bring back evidence from both documents —I separately confirmed the missing content was properly indexed, this wasn't an ingestion problem— still came back with a single document, exactly like with the managed planner. The CUSTOM planner decomposed questions that already had sufficient evidence in a single pass, and didn't touch the ones that genuinely needed both sources.

💡 ProTip #4: Switching from the managed planner to your own isn't a magic button that fixes cross-document multi-hop, at least not with a "small, fast" model like AWS recommends. The publishable result, with what I have so far, is: changing the planner helps the planner work, it doesn't guarantee it works on the right questions.

One obvious question remained unanswered: what if the planner isn't just different, but large? I tested it, and the result made me rewrite this section twice.

I Went Up a Weight Class: A Large Planner (And a Role Collision I Need to Name)

AWS's official blog mentions two candidates as a "large" planner: Claude Sonnet 4.6 and Amazon Nova Premier. Before running anything, I checked which one was still viable against the real account, not against the docs — and found a problem.

Figure 5: Nova Premier shows up marked Legacy in the model catalog. Verified in more detail: startOfLifeTime=2025-04-30, legacyTime=2026-03-13, endOfLifeTime=2026-09-14 — twelve days after this session.

Nova Premier gets ruled out not by design preference, but because anyone trying to reproduce this benchmark after September 14 couldn't use it even if they wanted to. That left me with just one "large" alternative cited by AWS: Sonnet 4.6, the same model I already use as the constant generator in A, B, C, and D.

I need to be explicit here about what this breaks: my own design rule for the Haiku run was to keep the planner isolated from the generator and judge, precisely so as not to contaminate any role. Config E breaks that rule on purpose, documented as a deviation, not an oversight. The result that follows can't be read as "pure large planner" — it has to be read as "large planner that's also the same model that generates the response afterward".

With that caveat already on the table, the numbers:

	`MANAGED` (C)	Haiku 4.5 (CUSTOM, exploratory)	Sonnet 4.6 (E)
Single-hop set, questions with real sub-queries	0/25	0/25	4/25
Multi-hop set, questions with real sub-queries	0/15	3/15	8/15

Sonnet 4.6 decomposed almost three times as often as Haiku. Of the 5 cross-document questions that neither MANAGED nor Haiku solved (b04, b08, b09, b11, b14), verified by searching for Well-Architected Framework-specific vocabulary in the retrieved chunks:

b04 and b14 get fixed — they now do bring back WAF content.
b08 partially improves — some WAF, not as much as the previous two.
b09 and b11 still miss the side that was missing — b09 even generated a sub-query, but not toward the right document; b11 didn't even decompose.

Figure 6: b14 solved with the Sonnet 4.6 planner. The question crosses the Well-Architected Framework's sustainability principle Maximize utilization with AgentCore Runtime's Instances mode — two different documents. The answer cites retrieved passages 11 and 13, matches the ground truth, and the judge gives it Correctness 1.

In other words: a bigger planner decomposes more, but doesn't guarantee it decomposes well. Two of five problem questions are still broken even with the most capable model I could test.

And the eval job with Nova Pro, same 4 metrics:

Figure 7: Native comparison of the two eval jobs on the multi-hop set. E's area dominates C's on all four measured metrics, with the widest gap in Correctness. The other radar axes sit at 0 because those evaluators weren't part of the job.

On the single-hop set the difference is small (0.90 vs. 0.88 Correctness), consistent with H2: a planner, whatever its size, shouldn't add much when a single step is already enough. On the multi-hop set the jump is large: Correctness goes from 0.50 (C) to 1.00 (E) — above even B (simple Retrieve, 0.567) and D (MANAGED planner + service generation, 0.70).

Is It Real, or a Lucky Single Roll?

Before publishing a jump of that magnitude, I repeated the full C and E run on the multi-hop set the next day — same 15 questions, same corpus, same models, retrieval/generation/evaluation pipeline run from scratch, with its own independent evaluation job.

	Correctness	Completeness	Faithfulness	Helpfulness
C, run 1	0.500	0.500	0.550	0.778
C, run 2	0.467	0.500	0.583	0.767
E, run 1	1.000	0.883	0.917	0.911
E, run 2	1.000	0.867	0.867	0.922

The result holds. E's Correctness came out exactly 1.00 across both independent runs — it's not an artifact of a lucky roll. C stays low in both, with a 0.033 difference between runs that's normal judge noise, not a trend. The gap between C and E in Correctness is stable; the other three metrics vary a bit more run-to-run (up to 0.05) but never cross the gap between configurations.

⚠️ ProTip #5: A jump from 0.50 to 1.00 in a single run of 15 questions is exactly the kind of number that invites suspicion of an artifact — an average inflated by discarded records, a miscounted null_result. Before publishing any jump that large, repeat the full run from scratch. If it doesn't hold, it was noise. If it holds, as it does here, you have real evidence worth reporting with confidence.

The honest reading, with all its caveats:

N=15 is still a small set. The pattern repeats across 2 runs, but that doesn't replace a larger N. The direction of the effect is well-supported; the exact values for Completeness, Faithfulness, and Helpfulness carry more run-to-run noise than Correctness.
The planner and generator share a model, the deviation already declared above. I can't rule out that part of the gain comes from Sonnet 4.6, as planner, "knowing better what it's going to be asked next" — a self-consistency effect, not necessarily better decomposition reasoning in the abstract.
It doesn't close the problem entirely. b09 and b11 remain unresolved even with the most capable planner I could test — the gain is real, but partial: 2 of 5 questions fixed, 1 partial, 2 unchanged.
The clean comparison (Nova Premier) isn't available due to its EOL date. This run can't isolate "larger planner" from "planner that's also the generator" — that separation would have required a third large model without that role conflict.

The conclusion the evidence does support: a CUSTOM planner larger than Haiku decomposes more often (8/15 vs. 3/15 on the multi-hop set) and, when it decomposes toward the right side, moves the metrics substantially — something Haiku never managed to demonstrate. What I can't cleanly separate, with this design, is how much of that gain is "more capable model" and how much is "same model that already generates the final answer".

What Changed Since April (And What Didn't)

One claim from April is now outdated, and two assumptions I started this benchmark with turned out to be wrong. All three get corrected here.

Claude Sonnet 4.6 is now on the judge allowlist for RAG/KB evaluation specifically. In April I had to use Nova Pro because of that restriction, so I left the corresponding update note in the original article instead of leaving that fact hanging there. Even so, I kept Nova Pro as the judge in this benchmark: the design calls for a cross-family judge as an independent fairness control, to avoid the self-preference bias known in LLM-as-judge setups when the judge shares a family with the generator (which is still Sonnet 4.6 in A, B, and C). The new availability doesn't change that design logic.

The other two are corrections to my own prior research, not to what was published in April.

foundationModelType accepts CUSTOM and MANAGED, with MANAGED as the default. This wasn't clear in the first pass over the public documentation I did before running anything — only CUSTOM examples were documented. Confirmed both against the updated API reference and against the real botocore shape in the account.

The first boto3/botocore version that supports AgenticRetrieveStream is 1.43.32 — that release shipped the operation and "Managed Knowledge Bases" together, in the same API commit. My machine's global Python had an older version; I had to isolate the project in its own virtualenv.

Decision Table

With the data in hand, here's what I'd tell someone evaluating this migration today:

Your situation	Recommendation
Production RAG with mostly single-hop questions	Migrate to Managed KB with simple `Retrieve`. Quality ties a well-tuned manual configuration, and you save yourself the vector store operations burden.
Your corpus has large files that break manual chunking (`SEMANTIC`, `NONE`)	Managed KB solves this directly via Smart Parsing. It's the clearest, least debatable win in this benchmark.
Frequent multi-hop or comparative questions	Don't turn on `AgenticRetrieveStream` with the default planner without testing it against your own corpus first. In my case it was worse than not using it.
You're going to use `AgenticRetrieveStream` regardless	Set `foundationModelType=CUSTOM` with an explicit model from day one — never leave the default in production without having validated it.
You're going to pick a CUSTOM planner	Try a large model before settling for a small one. In my benchmark, Haiku 4.5 decomposed rarely and poorly; Sonnet 4.6 decomposed almost three times as often and fixed 2 of 5 problem questions, with 1 more partially. The added latency and cost of a larger planner can be justified if your use case has genuinely multi-hop questions.
You're going to use the same model as planner and generator	It's valid, and it worked well in my benchmark — but don't assume the result isolates "large planner" from "planner that knows what it's about to be asked." If you have a way to test a large model different from your generator, do it before generalizing the recommendation to your architecture.

What's Still Open

This benchmark, like April's, has a deliberately bounded scope:

Isolating planner size from role collision. Config E uses Sonnet 4.6 as both planner and generator. The clean separation would have been a large model different from both — Nova Premier was the only option AWS cited, and it fell out of scope due to its end-of-life date (September 14, 2026).
Larger N on the multi-hop set. 15 questions, even replicated across two runs, is still too few for serious confidence intervals.
Multi-KB routing, AgenticRetrieveStream's other core capability that I didn't touch here — this benchmark used a single Managed KB across all configurations.
Cost per query compared across planners. This benchmark measures quality and latency, not cost. And the question left open is fairly concrete: if a large planner decomposes almost three times as often, how much does the per-query bill go up against the gain in Correctness? That gets answered by measuring planning tokens per query, not by looking at aggregate account spend.
Spanish-language corpus, out of scope just like in April.
Solving b09 and b11, the two cross-document questions that neither the managed planner nor any CUSTOM planner I tested fully solved.

If you replicate this in your own account and the managed planner does decompose on your corpus, I want to know — it would be evidence that the behavior depends more on content type than my data suggests.

Conclusion

Managed Knowledge Base isn't a quality regression. On simple retrieval, it does the same as my manual April configuration, and it effortlessly solves the ingestion problem that broke two of the five chunking strategies in April. If your only question is "do I lose quality by giving up chunking control?", the answer with this data is no.

But the "agentic" in AgenticRetrieveStream's name promises something the default planner doesn't deliver, and a poorly chosen planner doesn't guarantee either. It's not misleading marketing —AWS documents early-exit behavior as normal, and its own code examples never use the default— but it is an easy trap to fall into if you turn the feature on expecting it to "be smarter" without explicitly choosing who does the reasoning. With the managed planner, it was a measured regression. With a small planner, it was real but poorly aimed decomposition. With a large planner —even though it shared a model with my generator, a cleanup I couldn't fully isolate due to Nova Premier's retirement— Correctness on multi-hop questions doubled, sustained across two independent runs.

The real advice, if you're working with this in production, isn't "turn on the agentic retriever." It's "turn on the agentic retriever, pick your own planner —and probably a bigger one than you feel like trying first—, and measure on your own corpus whether it decomposes the questions that genuinely need it." Because the range I found on the same corpus, with the same multi-hop question, went from "worse than not using it" to "twice as good as using it without thinking," depending on who was planning.

The full benchmark code —Terraform, numbered scripts, the 15 multi-hop questions with their cross-verification, and the BYOI datasets— is at github.com/codecr/mkb-vs-chunking.

🚀 Final Pro Tip: Before turning on any "agentic" feature in production, ask yourself which model is making the planning decisions and whether that model is the one you chose or the one the service chose for you. And if you're the one choosing, don't settle for the first "small, fast" model the documentation suggests — in my benchmark, the difference between a small planner and a large one was bigger than the difference between having a planner and not having one at all.

If you want the full context, the original chunking benchmark is still the reference for manual configurations. And if what you want is to expose a Managed KB as an MCP tool for your agents, that path goes through AgentCore Gateway as a Knowledge Base target —the connector exposes AgenticRetrieveStream and Retrieve as MCP tools— and my article on Amazon Bedrock + MCP covers the protocol context.

See you in the next article! If you replicated this or tried a different planner, tell me about it in the comments — with this particular finding, the more eyes that verify it, the better. 🚀

Official Resources 📚

How to build a tiny 1.5B text-to-SQL model that beats a 7B

Pradhyumna Holla — Fri, 04 Sep 2026 07:10:20 +0000

I wanted to build something with an LLM using my own hands. Not wire an API into a wrapper, but take a base model, train it, measure it, break it, and serve it over HTTP. Why? Because I wanted to get my hands dirty working with an LLM, making and breaking things.

Text to SQL fits exactly. You ask a question in plain English, the system writes SQL, and you can run the query to find out whether it was right. Correctness is whether the rows match the reference query's rows against the real database, which is a fact rather than a judgement. It also has a mature benchmark in Spider, 10,000 human written questions over 200 real SQLite databases, split so the test databases never appear in training.

So I built it. Qwen2.5-0.5B as the base, LoRA adapters, one g5.xlarge with an NVIDIA A10G, about a dollar an hour. Continued pretraining on SQL text, then supervised fine-tuning on question and query pairs, then reinforcement learning with GRPO on top. The dashboards looked great. Reward climbing to 1.0, loss dropping cleanly through every stage.

It scored 6.4%. Comically bad levels of accuracy.

I didn't find that out for a while, because there was no held out evaluation. Every number I had came from data the model had trained on. The untrained base model scored 17.4%, so three stages of training had made things worse than no training at all.

The fix wasn't a better model. It was building the thing that could tell me I was wrong, an evaluation harness that runs both queries against the real database and compares the rows that come back.

Then I rebuilt. 6.4% to 44.6% on real schemas, then 49.7% with a proper RL reward, all still at 0.5B. Only then did I switch to Qwen2.5-Coder-1.5B, which landed at 68.1%. Sampling eight answers per question and returning whichever result most of them agreed on took it to 71.5%, against 71.2% for Qwen2.5-Coder-7B-Instruct.

The 7B is still better at one attempt, 71.2% to 68.1%. The 1.5B gets there by answering eight times and voting, trading compute for the gap.

That trade is the point though, and the point of this entire blog, and it's the one a lot of people are making now. Take a small model, aim it at one task, and build enough system around it that it beats something far larger at that one thing.

What went wrong

The first version was three training stages run back to back on Qwen2.5-0.5B, a half billion param open model from Alibaba, all of them using LoRA(Low-Rank Adaptation), which freezes the model's real weights and trains a tiny fraction of new, lightweight parameters.

Continued PreTraining: keep training the base model on raw SQL text so it gets used to the shape of the language.

Supervised FineTuning: show it thousands of question and correct-query pairs and have it imitate them.

Reinforcement Learning with GRPO: Group Relative Policy Optimization. The model writes several answers to the same question, each is scored, and it gets pushed toward whichever ones scored above that group's average. It learns from its own attempts rather than from copied answers.

Here is what those runs reported:

Stage	Steps	Loss	Data actually seen
Continued PreTraining	500	2.46 to 0.81	~2.7 passes over the corpus
Supervised FineTuning	500	3.26 to 0.51	0.10 of one pass, about 8,000 of 78,577 examples
Reinforcement Learning	300	reward 0.8 to 1.0	5,000 rows of the same training set

You have to understand that this was a learning project and my approach was flawed, so a few things went wrong. Every curve went the right way, and none of them meant anything, for three reasons.

The data taught the wrong task: sql-create-context hands the model a schema already trimmed to exactly the columns the answer needs, so it never has to work out which table matters. Working that out is the entire job. It also saw only a tenth of that data, because the run was configured by step count rather than by passes over it.

The reward could not teach anything: a formatting reward worth 1.0 fired for essentially every answer, and an exact string match reward worth 2.0 fired for almost none, so a group of four answers usually scored [1.0, 1.0, 1.0, 1.0]. GRPO scores each answer against its group's average, so when they all score the same the update is zero. The metric that tracks this hit 1.00 by step 160, meaning a large share of those 300 steps did nothing at all. And exact string matching is the wrong test anyway. It scores age>56 as zero against age > 56.

There was no held out evaluation: the RL stage ran on a slice of the same data the model had been finetuned on, so every number I had measured how well it remembered its own training set.

That last one is the real defect and the other two follow from it. Bad data and a broken reward are ordinary mistakes. What made them expensive is that nothing in the system could report them. The dashboards were green throughout.

Rebuilding, still at 0.5B

I stayed on the half billion param model for all of this. Not because it was good, but because a full fine tune plus a full evaluation there is about two GPU hours. When you are wrong five times in a row, that matters more than the ceiling does.

Fix the data first

The old training set handed the model schemas already trimmed to the columns the answer needed. The rebuild uses Spider with the real, complete database schemas, so the model has to find the right table among all the wrong ones. Three other things changed at the same time.

Loss is computed on the answer only: A training example is the schema, then the question, then the correct query. With full schemas the schema part is roughly 75% of the tokens. If you score the model on reproducing the whole example, most of the training signal is teaching it to generate database schemas, which nobody asked for. Masking the prompt means every bit of the signal lands on the query.

Two full passes over the data instead of a tenth of one: An epoch is one complete pass through the training set. The old run had covered 0.10 of one.

A learning rate suited to what was actually being trained: The learning rate controls how big each update is. The old value was appropriate for nudging a model that already knew the task. LoRA starts its extra weights from scratch, so it needs a larger one.

That is the single biggest change in the project. 6.4% to 44.6%.

One thing I did not expect. Validation loss picked the wrong checkpoint. Epoch 2 looked slightly worse than epoch 1 on both validation loss and token accuracy, and it was 3.9 points better on real execution accuracy. Loss is a proxy for a proxy. Run the metric you actually care about, on held out data, at every checkpoint.

In hindsight the old setup is a bit embarrassing, but every one of these fixes came from the failures, not from knowing better. Something that's true for most things that we do, not just machine learning.

Then fix the reward

Now that the harness existed, the reward could use it. Instead of a black and white, all or nothing test, the reward is a ladder with partial credit.

Outcome	Reward	What it buys
No SQL, or it will not parse	0.0	nothing
It parses as SQL	0.2	syntax is a solved sub-problem
The database accepted and ran it	0.5	the tables and columns actually exist
The rows match the reference	2.0	correct

The point of the middle rungs is the zero gradient problem from before. A group of four answers where none is fully correct now scores something like [0.2, 0.5, 0.5, 0.2] instead of [1.0, 1.0, 1.0, 1.0]. There is real disagreement inside the group, so there is a real update.

The 0.5 rung is aimed at one specific failure. Going into this, 88% of the model's remaining mistakes were invented column and table names. That rung pays for nothing except the columns existing.

44.6% to 49.7%. The mechanism moved in exactly the way the ladder was designed to move it.

unknown_column   647 → 458   (down 29%)
unknown_table     66 →   7   (down 89%)

How I know that gain is real

Two models answering the same 2,147 questions share most of their answers, and the shared ones tell you nothing. Only the questions where they disagree carry information. McNemar's test looks at exactly those. If the two models were equally good, the disagreements should split roughly evenly between them.

They did not. RL fixed 251 questions and broke 142. The test says a split that lopsided happens by chance with probability below 0.001.

The fixed and broken counts matter as much as the test. The headline is plus 5.1 points, but that is a net. It is not a clean sweep, and reporting it as one would hide 142 regressions.

Three library defaults that broke my reward

None of these throw an error. They just quietly change what the reward means, and nothing tells you it happened.

Reward scaling flattens the ladder: the library divides each group's scores by how spread out that group was. The order of the rungs survives, the spacing does not, so a group whose best answer was genuinely correct ends up pushing exactly as hard as one whose best answer merely parsed. The ladder becomes a ranking. Switching that off keeps the 7.5 times gap between correct and parsed intact.

The reference model is not the one you think: RL usually penalizes the model for drifting too far from a reference version of itself, measured as KL divergence. Under LoRA that reference is your adapter switched off, which is the raw base model, not your finetuned checkpoint. So the penalty was measuring how far supervised finetuning had already moved, not what RL was doing. I set it to zero rather than pull the model back toward something I had deliberately trained it away from.

Row caps must not be able to change a verdict: the reward stops reading rows once it has one more than the correct answer has. Anything longer could not have matched anyway, so the cap can never turn a right answer into a wrong one. It only stops a hallucinated cross join from materializing a million rows inside the training loop.

Reward Hacking

Reward hacking is when the model finds a way to score well without doing the task. It is usually discovered from a reward curve that climbs while the benchmark stays flat, which is to say after the GPU hours are gone.

So before spending the GPU hours, I attacked my own reward. I wrote eleven policies that never look at the question at all, and measured how much reward each one could collect. SELECT 1 always parses and never matches. SELECT * FROM <first table> always runs. WHERE 1 = 0 goes after the empty result loophole. Degenerate cross joins test the row cap and the parser guards.

The best of them collected 26.8% of what a real answer gets, and got 1.8% of questions outright, which is the empty answer slice and exactly where I expected it to score. So the reward is not easy to cheat, and that is a number now rather than an assumption.

This probe is also what caught the reward scaling default above.

Teaching it to reason, and why that made it worse

The original training data told the model to think before answering. The problem was that the "thinking" was one hardcoded sentence repeated across all 5,378 examples. As scratch paper it is blank. It carries no information about the specific question, so all the model learns is to recite a preamble before answering exactly as it would have anyway.

The proper version of this is rejection sampling, sometimes called STaR(Self-Taught Reasoner). Take a larger model that can actually reason, have it solve each of your training questions eight times while showing its work, then run every attempt against the database and keep only the ones whose SQL was correct.

That filter is the entire technique. A large model is wrong confidently and fluently, and training a small model on fluent wrong reasoning is worse than not training it at all. The harness from earlier is what makes the filter possible, which is the third job that one component ended up doing.

It worked as a data pipeline. 4,823 verified traces, covering 90.8% of the training questions, with a median of 114 words of reasoning that named real tables and real columns.

And it made the model worse. 44.6% down to 41.8%.

The reasoning crowded out the SQL

Training data	Tokens the model writes	Of which are SQL	Share of the signal landing on the SQL
Canned sentence	62.6	33.6	53.7%
Real traces	195.1	31.1	15.9%

Training only teaches the model on the tokens it writes. With the canned sentence, it wrote about 63 tokens per answer and 34 of them were the SQL, so more than half its practice was on the query itself. With real traces it wrote about 195 tokens and the SQL was still only 31 of them. The query had not got any longer. It was just buried under 114 words of reasoning.

So the model went from spending half its practice on the query to spending a sixth of it. Same number of passes over the data, same learning rate, 3.4 times less practice on the only part that gets marked. Running it for twice as many passes gave that practice back, and it came out at 47.1%, above where it started.

The gains didn't really stack

Running reinforcement learning on top of the trace trained model was a wash. Plus 0.1 points, 179 questions fixed and 177 broken. Statistically that is nothing.

The reason is that both interventions attack the same failure. Traces help the model name real tables and columns because the reasoning it copied named real tables and columns. The execution reward helps with exactly the same thing, and it is better at it. So the traces had already done a partial version of RL's job and left it nothing to work on.

The best 0.5B model in the project stayed the straightforward one, supervised finetuning plus RL, at 49.7%.

Why this is in the post

It cost real time and it did not produce the headline number, so the temptation is to leave it out. Two reasons not to.

First, a negative result with a measured cause is more useful than a positive one without. "Reasoning traces did not help" is folklore. "Reasoning traces diluted the training signal on the answer by 3.4 times, and doubling the passes over the data fixed it" is something you can act on.

Second, it is a clean example of the thing this whole project is about. The pipeline ran, the data was verified, the traces were genuinely good, and the model got worse. Nothing about that was visible from the training curves. The only reason I know any of it is that there was a held out number to check.

Did I just need a bigger model?

At 49.7% the 0.5B model had run out of road with this training setup. The failure breakdown said the remaining errors were not the kind a better reward fixes, so I changed the base model to Qwen2.5-Coder-1.5B. Three times the parameters, and pretrained on code rather than general text, which should help with a task whose output is code.

Same pipeline, nothing else changed. 67.9%, and 68.1% after RL.

That is a large jump and it invites the obvious objection. Maybe none of the previous work mattered and I just needed a bigger model. That is an empirical question, so I answered it with an experiment instead of an opinion. Run the old pipeline and the new one on both base models. Four runs, every combination.

Pipeline	0.5B	1.5B	What scale bought
Old	4.6%	46.9%	+42.4
New	44.6%	67.9%	+23.2
What technique bought	+40.1	+20.9

Both effects are large and both hold up statistically. Technique at 1.5B fixed 559 questions and broke 110. Scale under the new pipeline fixed 577 and broke 78.

Both matter, and they eat into each other: Technique is worth 20.9 points even on a bigger, code pretrained base, so the work is not just compensating for a weak model. But that is half what it was worth at 0.5B, because the bigger model had already solved some of what the technique was fixing.

The corners of the table show the same thing. Measured on its own, technique is worth 40.1 points and the bigger model is worth 42.4, so you would expect 82 from doing both. Going from the worst combination to the best actually gets you 63.3, from 4.6% to 67.9%. The missing 19 points are questions both changes would have fixed on their own, and you only get paid for those once.

Two things that qualify that table

The old pipeline at 0.5B also carried a tokenizer bug that stopped generation from terminating properly, while the 1.5B version of it used a working stop token. So the +42.4 in the top row is partly "a bug was also fixed". The two numbers I actually rely on are the ones underneath it, where everything else is held fixed.

And the scale column is not purely scale. The new base is both larger and code pretrained, so those two things are tangled together in the +23.2 and I cannot separate them without a fourth base model I did not train. The technique column is clean, since that is the same base with two pipelines. The scale column should be read as "bigger and code pretrained", not "bigger".

Worth saying that the old pipeline was set up to lose as gracefully as possible. Its destructive pretraining stage was left out, the library that corrupted the vocabulary was left out, the working stop token was used. It is the strongest fair version of the thing it loses to.

Reinforcement learning stopped helping

Same reward, same configuration, run on the new base.

RL on the 0.5B base:  +5.1 points   251 fixed, 142 broken   p < 0.001
RL on the 1.5B base:  +0.2 points   122 fixed, 117 broken   not significant

"Not significant" here means the fixed and broken counts are close enough that the difference is indistinguishable from chance. 122 against 117 is a coin flip.

The strange part is that the mechanism still worked perfectly. Hallucinated columns fell 23%. The share of queries the database would actually run rose from 88.2% to 90.6%. Syntax errors dropped from 5 to 2. Everything improved except the score.

The reason is that the reward pays for executability, and executability had stopped being the bottleneck. At 0.5B, invented columns dominated the failures, so fixing them moved the number. At 1.5B the model already runs 90% of its queries, and what remains are queries that execute perfectly and answer the wrong question. This reward cannot see that. A query that runs and returns wrong rows scores 0.5, exactly the same as one that runs and returns wrong rows for a completely different reason.

Two diagnostics from that run are worth carrying to any RL project:

56% of training groups produced no gradient at all: That is the original v0 failure arriving for the opposite reason. Back then the rollouts were uniformly wrong, so nothing separated them. Now they are uniformly right. Reinforcement learning gets harder as your policy gets better, because it runs on disagreement inside a group, and a good model disagrees with itself less.

Training reward rose 24% while held out accuracy moved 0.2 points: With entropy falling steadily the whole time. Entropy here is how spread out the model's choices are, and it falling means the model is getting more confident and less varied. That is precisely the shape of the dashboards that started this whole project. The only difference is that this time there was a held out number sitting next to it saying the gain was not real.

Getting more out of the weights we already have

Everything so far changes the model's weights. At 68.1% it is worth asking what the remaining 32% actually looks like, because the answer decides what you can do about it without training anything.

1,462  correct                                        68.1%
  483  runs perfectly, answers the wrong question     22.5%   no error exists
  202  the database rejects it outright                9.4%   an error message exists

Two completely different populations. The second one has a signal attached, because when SQLite refuses a query it tells you why, in words, like no such column: customer_name. The first one has nothing. The query runs, rows come back, and everything looks fine.

Letting it retry, which barely works

The obvious move for the 202 is to show the model its own error and let it try again, up to three rounds.

Two things make that a measurement rather than a demo. The loop never sees the correct answer. Its only stopping signal is whether the database accepted the query, which is all that exists at inference time in the real world. And the rounds are batched rather than per question. Round one generates all 2,147, and only the roughly 200 rejections go into round two. Looping question by question is about 50 times slower for an identical answer.

It bought 1.1 points. I had predicted 4 and written that down beforehand.

The reason is the useful part.

202  rejected on the first attempt
 63  became something the database would run    31% of rejections
 23  were actually correct                      37% of those repairs
 40  moved from "database refused it" to "runs fine, wrong rows"

There are two multiplications there, not one. An error message tells you that you are wrong. It never tells you what is right. So the model fixes the complaint rather than the answer, and two thirds of its repairs migrate from the visible failure bucket into the invisible one.

At 0.5B it does essentially nothing. 500 rejections, 12 repaired. A weak model told "no such column: X" still has no idea which column does exist.

I do not report a significance test for retry, on purpose. The test works by weighing how many questions got fixed against how many got broken, and retry cannot break any. A query the database rejected was already wrong, so replacing it either helps or changes nothing. The broken count was zero in all four runs, which I checked rather than assumed. With nothing on the losing side, the test would call any gain significant at all, even a gain of one question. What actually means something here is the size of the gain against what it costs.

Voting, which works, and works where retry cannot

The 483 queries that run cleanly and return wrong rows produce no error, so retry is structurally blind to them. But they do disagree with each other, and disagreement is an indicator/signal.

So instead of taking the model's single most likely answer, sample eight different ones, run all eight, group them by the rows they return rather than by the text of the query, and answer with the largest group. Two queries written completely differently that return identical rows are probably both right. A hallucinated one usually returns something nobody else got.

The grouping has to go through the same comparison function the benchmark uses, not a hash of the rows. That comparison ignores column order, so SELECT age, name and SELECT name, age count as one answer. Hashing would be faster and would split exactly the groups voting exists to merge.

Two things are built in as checks rather than assumed. The first of the eight candidates is always the model's ordinary single answer, the one it would have given without voting, so setting k to 1 has to reproduce the plain score exactly. If it ever doesn't, something is wired wrong and I find out straight away. And voting at any k only looks at the first k candidates, so generating 16 once lets me score every budget from 1 to 16 without generating anything again.

The trap

Every query that returns nothing looks identical to every other query that returns nothing. A broken filter, a made up condition matching no rows, and a genuinely empty answer all land in the same group and vote together. In every other group, agreement means several differently written queries arrived at the same rows, which is real evidence. In the empty group it only means several queries failed to return anything, which is not. Three people shrugging is not a consensus.

So an empty group loses to any group that came back with actual rows. Both halves of that trade are measured. It is worth 1.3 points, and it costs 6 questions where the correct answer really was empty.

What the curve says

k	vote@k	pass@k, the ceiling	Gap	Gain from doubling
1	68.1%	68.1%	0.0
2	69.2%	70.1%	0.9	+1.2
4	70.6%	72.8%	2.2	+1.4
8	71.7%	75.4%	3.7	+1.1
16	72.1%	76.9%	4.8	+0.4

vote@k is what the system actually answers with. pass@k is whether any of the k candidates was correct, which you can only know by checking the answer key afterwards. It is the ceiling a perfect chooser could reach.

Voting saturates and the ceiling does not. Each doubling buys 1.2, then 1.4, then 1.1, then 0.4. The last doubling returns a third of the one before it. But pass@k keeps climbing, so the gap between them widens from 0.0 to 4.8 points.

That gap is right answers the model generated and threw away. At 16 samples the bottleneck is no longer producing a correct query. It is recognising the one already in hand. Measured at k = 8 alone this looks like a technique with room left in it, and it is not.

Where the gain comes from

greedy result  →  vote result
    correct    →  correct      1452
  ran, wrong   →  ran, wrong    453
   rejected    →  rejected      106
   rejected    →  correct        45
   rejected    →  ran, wrong     45
  ran, wrong   →  correct        38   ← invisible to retry
    correct    →  ran, wrong      8

fixed 83, broke 8    p < 0.001

Significance testing is legitimate here, because voting can and did break things. Eight regressions makes the comparison two directional again.

There are two separate results in that table. Voting reaches the bucket retry cannot see at all, 38 questions. And it beat retry on retry's own ground, repairing 45 rejected queries against retry's 23. Seven more samples turns out to be a better repair mechanism than one error message, which is worth knowing before building anything cleverer than sampling.

Why voting survives on a weak model and retry doesn't

              retry    voting
1.5B          +1.1      +3.5
0.5B          +0.2      +3.0

Retry needs a model good enough to act on feedback. Voting only needs one that is right sometimes, and then fishes that answer out. The second is a far weaker requirement, which is why voting survives at a model size where retry collapses entirely.

That asymmetry is only visible because both techniques were measured at both sizes. Told as a 1.5B story, it would have been thrown away.

A reliability check that costs nothing

Voting requires running eight or sixteen candidates anyway. Once you have done that, you already have something else for free, and I think it is the most useful output of the whole project. How much did the candidates agree with each other?

Agreement across 16 samples	Questions	Share	How often correct
All 16 agree	1,405	65.4%	85.9%
12 to 15 agree	274	12.8%	60.6%
1 to 11 agree	377	17.6%	46.2%
Nothing ran at all	91	4.2%	0.0%

When the model agrees with itself it is right 86% of the time, and that covers two thirds of all questions. When it does not, it is close to a coin flip(46.2%).

In a product that distinction is worth more than the 3.5 points voting adds to the score. It is the difference between a system that silently returns a wrong number and one that can say "I am not confident about this one, check it". A wrong SQL answer does not look wrong. It looks like a number in a cell.

The group that only appears at sixteen samples

91 questions where not one of sixteen attempts produced a query the database would run. Not a wrong answer among them, because there was never an answer at all. Accuracy there is 0.0%, and it is 0.0% by definition rather than by bad luck.

At eight samples these hide inside the low agreement band, indistinguishable from questions the model merely found hard. At sixteen they separate cleanly, and that separation is the whole value of them. The service can recognize these before it answers rather than after. Saying nothing on 4.2% of questions costs a great deal less than being confidently wrong on them.

Three things that keep it honest

Each band is scored on its own members: An earlier version of this table lumped 7-of-8 agreement in with 6-of-8 and reported the pair as 62.8%. Scored separately, 7-of-8 was 62.8% and 6-of-8 was 47.9%. Lumping them made the weaker half look 15 points better than it was, and anyone trusting that number would have trusted answers they should not have.

The bands are deliberately wide: The middle of the chart is a mess. 5-of-16 scores 33% while 1-of-16 scores 55%, but those are only 27 and 33 questions each, so the gap between them is chance rather than anything real. The order is wrong too, with less agreement sometimes scoring better than more. Splitting that middle into finer bands would be reporting a precision that is not there.

One answer gets no confidence at all: If the model only answered once, there is nothing to compare it against, so the service returns "unmeasured" instead of a number. A confidence figure nobody has checked is worse than no figure.

The same pattern shows up on the 0.5B model, just lower, with unanimous answers correct 80.7% of the time instead of 85.9%. That is what makes this a property of the approach rather than a fluke of one particular model.

What is left is a choosing problem

The vote vs ceiling gap from the last section has a specific shape. Voting picks a wrong answer while holding a correct one 3.6% of the time. That looked like the cheapest remaining win in the project. No training, no GPU, the candidates are already sitting on disk.

So I looked at the failures before building anything. 87% of them are cases where the wrong answer won six votes to two. The model is confidently and consistently wrong, and no rule based on counting votes can override a 6-2 majority.

I built two smarter selectors anyway and measured them at +0.1 and −0.1 points. Dead end, and worth the hour it took to prove rather than the week it would have taken to build. Closing that gap needs a model trained to score candidates against the question, not a tiebreak rule.

Why this is the part I would keep

If I had to throw away everything in this project except one idea, it would be this one. Accuracy tells you how often a system is right on average. It tells a user nothing about the answer currently on their screen.

Self agreement tells you about this answer, not the average one, it costs nothing once you are sampling, it needs no extra model, and it works well enough to act on. The more valuable output of a system like this is not being right more often. It is knowing when to shut up.

The number that would matter in production

Every score so far has handed the model the complete, correct schema for the exact database the question is about. Ask about students, get the student database. Nothing else on screen.

No real deployment works like that. A company's warehouse has hundreds or thousands of tables, nobody knows in advance which database a question belongs to, and the whole thing does not remotely fit in a model's context window. Something has to search the schema first, pick out the handful of tables that look relevant, and pass only those to the model. If that search picks wrong, the model never had a chance.

This section takes the gift away and measures what it was worth.

Building a test that can measure anything

My first design was wrong, and the reason is worth a paragraph.

Spider's databases are tiny. The median one has 4 tables and 19 columns, and writing it into a prompt as CREATE TABLE statements costs about 102 tokens. Searching for the right table among four is not a search problem. Any method scores near perfect and you learn nothing.

The obvious fix is to throw all 206 databases into one big pile and search that. This fails badly.

125 table names appear in more than one database
436 of 1,053 tables affected  (41%)

customers  → 22 databases      student → 12
addresses  → 15                staff   → 11

There are 22 different tables called customers in that pile, belonging to 22 unrelated databases. A question asking about customers genuinely does not say which one it means. No search method can resolve that, however good, because the information needed to resolve it is not in the question. A test built this way would measure an impossible task and then blame the retriever for failing it.

So I built a collision free pool instead. Take databases one at a time, in order of how many benchmark questions they carry, and skip any database whose table names clash with one already taken. That leaves:

81 databases, 300 tables, 1,427 columns
1,457 of the 2,147 test questions
No two tables anywhere in the pool sharing a name

Of those 300 tables, only 107 belong to databases that carry any questions. The other 193 are pure noise, sitting there to be wrong answers.

And the pool genuinely does not fit. Written out as CREATE TABLE statements it comes to 8,262 tokens, against the 3,072 token prompt limit these models were evaluated under. Nearly three times over. Search is not an optimization here, it is the only way the prompt fits at all.

One thing to carry through the rest of this section: every number below is over 1,457 questions, not 2,147. Putting a retrieval number next to a number from earlier in this post is comparing two different tests.

Two ways to search

Each table becomes one short searchable document, just its name and its column names:

student: id, name, age, dept_id

The search query is the question text and nothing else. Never the database name. That is deliberate and important: if the retriever knew which database a question came from, it could just look up that database's tables directly, and every number here would be measuring an answer key rather than a search.

BM25 is keyword matching, the classic search engine approach. A table scores higher when more of the question's words appear in it, and rare words count for more than common ones. Matching the word "hangar" tells you a lot, matching "the" tells you nothing.

One detail carries a surprising amount of weight. Schema names are written like city_code or singerID, and people ask questions using words like "city" and "singer". A normal tokenizer treats city_code as one indivisible thing, so a question about cities can never match it. Mine splits on both snake_case and camelCase and keeps the joined form as well, so city_code becomes city_code, city and code. Bridging that gap between English and identifiers is most of what this job actually is.

Dense retrieval matches on meaning instead of words. A small embedding model turns each table description into a list of numbers, positioned so that texts meaning similar things sit near each other. Do the same to the question, and the nearest tables win. It can match "how many people are enrolled" to a student table without either phrase sharing a single word with the other.

Both methods return the top k tables, where k is just how many you decide to hand the model. Here is how often each finds what the question needs.

k	BM25 recall	BM25 coverage	Dense recall	Dense coverage
1	33.9%	25.1%	58.3%	44.0%
5	63.9%	53.8%	88.1%	79.2%
10	73.0%	64.1%	92.6%	86.3%
20	78.6%	70.7%	95.1%	90.3%

The column that matters is coverage, not recall, and the difference between them is not a technicality.

Recall is the share of needed tables that were found. Coverage is the share of questions where every needed table was found. Say a question needs to join two tables and the search finds one of them. Recall calls that 50%, a decent partial score. But the question is exactly as unanswerable as if the search had found nothing at all, because you cannot write half a join. Coverage scores it zero, which is the truth.

Dense wins clearly, which is not surprising given that questions are English and schemas are identifiers. BM25 is in here as the honest baseline, since it is free and needs no GPU, and if it had come close that would have been worth knowing.

What it costs end to end

Now run the actual model on what each search method hands it. Four conditions, identical in every respect except which schema text goes into the prompt, all over the same 1,457 questions.

Oracle is a cheat. It shows the model exactly the tables the correct answer uses and nothing else. Not achievable in reality, since knowing which tables the answer needs is most of the problem. It is here as a ceiling.
Gold is the whole correct database, which is what every number earlier in this post used. Re-measured here on the smaller question set so it compares fairly.
Dense and BM25 are the real conditions, picking 10 tables out of 300.

Condition	Schema shown	Accuracy	Queries that run
Oracle	exactly the tables the answer uses	68.4%	92.1%
Gold	the whole correct database	63.5%	89.7%
Dense, top 10	10 tables out of 300	45.2%	66.8%
BM25, top 10	10 tables out of 300	37.1%	57.5%

Two findings, and I predicted neither.

Showing less schema helps, if it is the right schema: Oracle beats gold by 4.9 points, and gold is only showing tables from the correct database. Every irrelevant table costs something, even a related one.

That is the exact opposite of what was true in training, where trimmed schemas were actively harmful because the model never learned to pick a table out of a crowd. Training on clutter is necessary. Having no clutter at question time is pure upside.

Most of the loss is distraction, not absence: This is the one that surprised me. Split the dense results by whether the search actually found everything the question needed.

all needed tables present   1,257 questions (86.3%)   accuracy 51.9%
at least one missing          200 questions (13.7%)   accuracy  3.0%

The missing group behaves exactly as you would expect. 3.0% is effectively zero, and those questions were unanswerable the moment the search missed. Fine.

But look at the other group. Every table the question needed was sitting right there in the prompt, and accuracy was still only 51.9% against oracle's 68.4%. Sixteen and a half points destroyed purely by the nine irrelevant tables sitting next to the right ones. The share of queries the database will even run drops from 92.1% to 66.8%, because the model keeps reaching for a plausible looking table that belongs to some completely unrelated database.

Stacking those two costs together:

oracle                             68.4%
  minus distraction  16.5 points →  51.9%   right tables present, plus nine more
  minus absence       6.7 points →  45.2%   13.7% of questions miss a table

Distraction costs two and a half times what absence does.

I had predicted 58 to 61% before running this, by reasoning that accuracy would be roughly coverage multiplied by the baseline. In other words, I assumed that finding the tables was the whole problem, and that once found, the model would perform as it always had. The gap between that prediction and the real 45.2% is exactly the distraction cost, which my mental model of the system had no room for at all.

Retrieving fewer tables does not help

If distraction is the dominant cost, then handing the model fewer tables should reduce it. I wrote that prediction down before testing it. Six more runs, both search methods at three different values of k, say it is wrong.

Method	k	Coverage	Accuracy when covered	Accuracy when missing	Accuracy overall
Dense	5	79.2%	55.6%	1.0%	44.3%
Dense	10	86.3%	51.9%	3.0%	45.2%
Dense	20	90.3%	49.4%	2.1%	44.8%
BM25	5	53.8%	63.6%	2.2%	35.3%
BM25	10	64.1%	57.1%	1.3%	37.1%
BM25	20	70.7%	54.1%	2.6%	39.0%

Look at the dense rows first. Overall accuracy reads 44.3, 45.2, 44.8 across a fourfold change in how many tables get retrieved. Flat enough that you would reasonably conclude k does not matter much.

That conclusion would be wrong, and the last two columns show why. Two opposing forces are cancelling out. Coverage climbs from 79% to 90%, because searching wider finds the right tables more often. Accuracy on questions that were already covered falls from 56% to 49%, because every extra table is another distraction. One goes up, the other goes down, and the total sits still.

BM25 starts so starved of recall, only 54% coverage at k=5, that the coverage gain outweighs everything else and its overall number climbs from 35.3 to 39.0. Its covered accuracy falls on exactly the same schedule though, 64% to 57% to 54%.

Whether raising k appears to help depends entirely on whether your search method still had recall left to gain. It never depends on the distraction going away, which got worse in every single condition tested.

So the honest reading is narrower than "retrieve less". The number of tables you retrieve is not a dial worth tuning. It trades coverage against distraction at roughly one for one, and the 16.5 point distraction penalty survives every setting I tried. What would actually move this is making the model itself robust to irrelevant tables, by training it on cluttered schemas instead of clean ones. That is a training change, not a search change.

What it costs to run

A project can report 68.1% accuracy and say nothing at all about whether an answer takes 200 milliseconds or 30 seconds. So the last piece is an HTTP service, and a benchmark of it.

The service reuses the voting and retry code directly rather than reimplementing it. That is the point. A second copy of the clustering or selection logic could drift away from the numbers those techniques were measured at, and nothing would tell you. A request returns the SQL, the rows, a confidence band from the agreement table, and a timing breakdown.

Measured on one A10G handling one request at a time, which is what a service does, unlike the evaluation harness that batches sixteen together. Over the full test split, so the accuracy column is real evidence rather than a spot check.

Mode	Accuracy	p50	p95	p99	Mean	Queries/sec
Greedy	67.9%	2,652 ms	5,100 ms	7,162 ms	3,088 ms	0.32
Vote, 8 samples	71.3%	5,650 ms	10,789 ms	15,272 ms	6,522 ms	0.15
Retry, 3 rounds	68.9%	2,714 ms	9,739 ms	16,042 ms	3,777 ms	0.26

p50 is the median request. p95 is the slowest one in twenty. p99 is the slowest one in a hundred.

That accuracy column is doing quiet work. It reproduces the harness from a completely different code path with different batching, giving +3.4 for voting and +1.0 for retry against the harness's +3.5 and +1.1.

The tail is the finding

Retry's median is indistinguishable from plain greedy, 2,714 ms against 2,652, because most requests never enter the loop at all. Every bit of its cost sits in the tail.

Its mean of 3,777 ms hides that from both directions simultaneously. It is 40% above a median that describes most requests, and 76% below a p99 that describes the requests a user actually complains about. It describes neither.

Voting is the opposite shape. It costs 2.1 times more at the median, but its p99 is only 2.7 times its own median, so what you measure in staging is what you get in production.

The mean says retry is cheaper. The tail says voting is more predictable. Users experience the tail.

That last one is the whole system running. Three questions against a test database, answered by the service. The first two are correct and unanimous. The third asks for distinct types of planes owned by pilots, and returns four plausible plane names from the wrong table. Nothing in the SQL or the output reveals that. The only sign is one sample out of eight disagreeing, which drops the confidence badge from high to medium.

What it ran on

All of this is one g5.xlarge in us-east-1. A single NVIDIA A10G with 24GB, four vCPUs, $1.006 an hour on demand. The three serving benchmarks above took ~1.8, ~4 and ~2.5 hours respectively, which is most of a working day of GPU time spent purely on measuring something I had already built.

That ratio is the honest summary of the project. Far more compute went into finding out whether things worked than into making them work.

Four pieces of infrastructure earned their place:

Automatic shutdown when idle: A cron job checks GPU utilization every minute and shuts the box down after 30 minutes below 5%. A forgotten g5.xlarge is about $170 a week, and this is the single highest value cost control in the project.

But the first version measured the wrong thing. GPU utilization alone is not "is this machine in use". An evaluation run spends real minutes loading weights, building prompts, executing SQL against SQLite and scoring results, all at 0% GPU. That script shut down a live run twice before I fixed it. Idleness is now graded. If a process is still holding GPU memory, or a human is still logged in, the limit becomes 180 minutes instead of 30. Not immunity, because "forgot to log out" must not cost $170 a week either.

Rescuing work from spot instances: Spot instances are 50 to 60% cheaper, and AWS can reclaim them with a two minute warning delivered through the instance metadata service. Reclaiming means terminate, not stop, so the disk goes with it. A service polls for that warning and syncs checkpoints to S3 when it arrives. Two minutes is enough to do one thing quickly and not enough to do anything clever.

It is not the primary mechanism though. A missed notice or a kernel panic gives no warning at all, so checkpoints also go to S3 on every save. Resuming is not restarting the same machine, it is a fresh machine pulling the last checkpoint from S3 and carrying on from that step, which I tested by killing a run and deleting its disk.

One thing to note is that I never actually ran on spot. AWS took forever to process my quota request and by the time it came through, most of the project was already done. So this is built and tested but not battle worn. It is a genuinely good feature and half the price is half the price, so if you use it, let me know how it holds up.

Never mirror a checkpoint directory to S3. The tidy way to stop S3 filling up with old checkpoints is aws s3 sync --delete, which mirrors the local directory. It is also one fresh instance away from deleting the only copy of a run, because a new machine's checkpoint directory is empty and mirroring an empty directory deletes everything. Old checkpoints are about 110MB each. Letting them pile up is far cheaper than the failure they prevent.

Also, I originally set everything up in us-east-2, because GPU instances ran about 30% cheaper there. Then I found out I could not launch an instance at all.

AWS caps how many of each instance type you can run, per region, and those caps start at zero for GPU instances until you ask for an increase. Mine in us-east-2 was zero and stayed zero, so the cheaper price was for something I was not allowed to buy. Everything moved to us-east-1, where I already had approval for a few. Check your limit in a region before you plan around its pricing.

Also worth knowing: a stopped instance still charges you for its disk. Mine was about $16 a month for a 200GB volume doing nothing at all. If you are finished with a machine, terminate it rather than leaving it stopped.

What I would build next

Something trained to pick the answer: At sixteen tries the model writes a correct query for 77% of questions and picks it only 72% of the time. Those 4.8 points are right answers it already produced and threw away. I tried two hand written rules for choosing better and they scored +0.1 and −0.1, so this needs a model trained to judge candidates against the question, not another rule.

Training on messy schemas: The model learned on tidy, correct database descriptions, then gets handed cluttered ones full of irrelevant tables and loses 16.5 points to the clutter. That is fixable in training, by showing it messy schemas while it learns, rather than in the search step.

Faster serving: 2.7 seconds for a typical question is slow, and the service currently handles one request at a time. Processing several at once on the GPU is the standard fix and the single biggest weakness in these numbers.

Not multi turn reinforcement learning, which was my original plan. The idea was to train the model on the whole back and forth of the retry loop rather than on single answers. Two things killed it. Only 37% of the queries it repairs end up correct, so there is not much there to learn from. And at 1.5B, over half of training groups already teach the model nothing, because all its attempts agree with each other and it needs disagreement to learn. Training on longer conversations makes that worse.

Conclusion

The headline is that a 1.5B model matched a 7B one. That is true, and it is also the least interesting thing here.

The model did not get good because it got bigger. It went from 6.4% to 49.7% at half a billion parameters, before I touched the model size at all, and every one of those gains came from something outside the weights. A way to check answers by running them. A reward that had a gradient in it. Training data that did not hide the hard part. Then, once the model was as good as that size allowed, the same measuring tools were pointed at a bigger base and the gains transferred intact.

And the last 3.4 points, the ones that actually drew level with the 7B, came from no training whatsoever. Sample eight answers, run all eight, keep whichever result most of them agreed on. That is a system decision, not a model decision.

Look at what the pieces actually are, stripped of SQL.

A mechanical check for whether an answer is right
A reward built out of that same check, so evaluation and training agree by construction
Sampling several answers and using agreement between them to choose
Using that same agreement to know when to keep quiet
Retrieval, when the context needed does not fit
Held out measurement on every claim, including the ones that came back negative

Only the first is SQL specific, and only barely. Anywhere you can mechanically decide whether an output is correct, all six apply. Code that has to compile and pass tests. Structured extraction validated against a schema. Maths with a checker. An API call that either succeeds or does not. In each case you can build the same loop, and in each case a small model wrapped in it will go further than its parameter count suggests.

This is why the small model + good system approach works, and why so many people are converging on it right now. You are not trying to beat a frontier model at everything. You are picking one task, building the infrastructure that knows what correct looks like, and letting a cheap model take as many attempts as it needs.

The expensive part was never the model. It was building the thing that could tell me when I was wrong.

Thank you for reading my blog, this was fun to build, and I learnt a lot of things too. Happy to receive your thoughts in the comments! Connect with me on LinkedIn and X.

What I left off the file portal on AWS — how far FSx for ONTAP features reach, and the work handed to a schedule (Part 6)

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Fri, 04 Sep 2026 00:59:30 +0000

Introduction

This is the last part of the series. Part 1 built a file portal on Amazon FSx for NetApp ONTAP (FSx for ONTAP) S3 access points, Part 2 added ransomware response and WORM retention, Part 3 put AI agents on it, Part 4 delegated 182 storage operations, and Part 5 collected what the running cluster refused.

This part is what was left off the screen. That is the difference from last time: where Parts 4 and 5 were about operations you run from a screen, this one has two subjects. How much of ONTAP stays usable without going through the portal at all, and the operational work that was handed to a schedule instead of a button.

What this part does not cover: performance figures (this measures whether things are possible and under what condition, not throughput), running SnapLock or Object Lock (irreversible, so not enabled without an instruction naming the retention), and actual tiering into a capacity pool (needs a different aggregate configuration, which was not on this path).

The reach of ONTAP features on this path

When you read and write files over the S3 API through an FSx for ONTAP S3 access point, how much of ONTAP stays usable alongside it? You want qtrees to carve up directories, quotas to stop a tenant running away with the space, FlexClone to spin a verification copy. Does any of that sound like your environment?

Start looking into it and NetApp's documentation gives you pause. ONTAP S3 interoperability records qtrees, quotas, FlexClone, and a volume clone of a FlexGroup containing ONTAP S3 buckets as not supported — for the ONTAP S3 server.

That is where it stalls. The FSx for ONTAP S3 access point is an AWS mechanism, and what it attaches to is a volume, not a bucket. Attaching one does stand up an ONTAP S3 server on the SVM, and I/O through the access point goes through ONTAP's S3 protocol stack. So the two are not unrelated. Does the same restriction appear?

I measured it. Up front: everything I measured worked.

ONTAP feature	Result on the S3 access point path
Qtree	Can be created. Appears as an S3 prefix, and an object PUT into it lands inside the qtree
Quota	A tree quota refuses an S3 PUT
FlexClone (volume granularity)	The clone can be made, and the clone itself can take an S3 access point
FlexClone (file granularity)	A file written over S3 can be cloned, and the clone is visible as an object through the access point
FlexGroup volume	An S3 access point can be attached. PUT / GET / LIST and multipart all work
Clone of a FlexGroup	The clone can be created

The four items that table lists did not appear as restrictions on this path. Reading that table as "not supported there, therefore not supported here" is wrong, and so is the reverse: a "supported" row is not evidence about this path either. The subjects differ, so nothing is known until it is measured.

What actually tripped me up was not in the table at all. Three things.

An NTFS volume with a UNIX identity on an SVM with no CIFS server attaches, reaches AVAILABLE, and then refuses every data operation with AccessDenied. AVAILABLE says nothing about the health of the file system layer
A volume created on the ONTAP side takes ten to twenty minutes to appear in the AWS API. The documentation says "several minutes"
A volume that has once carried an S3 access point cannot be deleted from the ONTAP side, even after every access point is detached. Only aws fsx delete-volume removes it

Below is the walkthrough, and how the controls were set up.

Test environment

Item	Value
Date	2026-08-26 (UTC)
Region	ap-northeast-1
ONTAP version	9.18.1P3D1
Deployment type / throughput	SINGLE_AZ_1 / 128 MBps
Test volumes	UNIX security style 1 GiB × 2 (one with an access point, one as the control), NTFS 1 GiB × 1, FlexGroup 400 GiB × 1
Access point identity	UNIX `root`, internet origin
Cleanup	Throwaway volumes only. All deleted

A note on the identity: UNIX root was used to keep the measurement simple. Every request through an access point is authorized as that one file system identity, so specifying root removes any narrowing by file permissions. This is not a recommended configuration. Use a dedicated user with only the permissions the writes need, and split access points per purpose (FileSystemIdentity cannot be changed after creation).

The distinction that matters most — different subjects

The results below have to be read with these two kept apart.

Mechanism	What it is	Whose
ONTAP S3	The S3 server with which ONTAP serves buckets. This is what NetApp's table is about	NetApp
FSx for ONTAP S3 access point	An AWS access point attached to an FSx for ONTAP volume	AWS

They are not unrelated. Attaching one stands up an ONTAP S3 server on the SVM, and the I/O goes through ONTAP's S3 protocol stack (measured). That is exactly why NetApp's table is the right place to start.

Being the right place to start is not the same as being the conclusion. The target is a volume rather than a bucket, and the namespace and the authorization path both differ. "Not supported there, therefore not supported here" turns something unchecked into something checked. The reverse reading is the same error.

The walkthrough

Qtree — usable

Step	Result
Create a qtree in the volume with an access point	Succeeded
Control: the same qtree in the volume with no access point	Succeeded. No difference
`list-objects-v2 --delimiter /`	The qtree appears in CommonPrefixes
PUT into the qtree prefix	Succeeded
Inspect the qtree from ONTAP	The written file is inside the qtree directory

NetApp's "Qtrees: not supported" does not manifest as a refusal on this path. The qtree is created, it is visible as a prefix in the S3 namespace, and an object written into that prefix lands inside it. Together with the quota result below, it behaves as a qtree.

Incidentally, a volume with an access point attached carries an internal ____NTAP_S3_MAPPING directory at its root. It is visible over NFS and SMB, so anyone who inspects a collect-layer volume by hand will see it.

Quota — enforced against an S3 PUT

Step	Result
Set a tree quota on the qtree (space 1 MiB, files 10) and enable quotas on the volume	Succeeded
`quota report`	The rule is active, and the file written over S3 is counted in files used
PUT 15 small objects	8 succeeded, 7 refused. files used stopped at exactly 10/10
What the S3 client received on refusal	HTTP 507 `InsufficientCapacity` / `Maximum storage capacity of file system has been reached.`
Control: raise the files limit 10 → 50 and re-PUT the refused key with the same body	Succeeded

The control carries the conclusion. Raising the limit alone made the identical PUT succeed, so the refusal was the quota, not a capacity shortage and not a permissions artefact.

That the response mis-describes its cause matters to the design. The file system was not full; a qtree file-count quota was reached. An operator who sees a 507 and "maximum storage capacity of file system" will consider growing the file system. If you use quotas on this path, write down what that response actually means in the runbook.

FlexClone — both volume and file granularity

There are two granularities, and they behave differently.

	File granularity	Volume granularity
Where it lands	The same volume. It appears as a new key on the same access point	A separate volume
Authorization boundary	Same as the parent	Its own access point, its own policy, its own identity
Time before it is usable	None	599 to 1,177 s measured, for a volume created through the ONTAP API to appear on the AWS side
Visible over S3	Yes. `StorageClass=FSX_ONTAP`, sha256 matches the source	Yes. LIST / GET / PUT through the clone's own access point

At either granularity the clone shares data blocks with its parent. Cloning a 256 MiB file four times:

Point	Logical	Physical
Just after one 256 MiB PUT	—	253,534,208 B
After four file-granularity clones	1,350,942,720 B	277,200,896 B

Logical grew to five copies, physical by about 23 MB. FSx for ONTAP bills provisioned SSD capacity, so there is no capacity to provision per duplicate.

NetApp states that copies consume no storage except what is required for metadata until changes are written to the copy (FlexClone volumes, files, and LUNs). The operation that ends the sharing is a split (volume clone split start), which I did not measure. From ONTAP 9.4, on AFF systems where the volume guarantee is none, the documentation states that the split shares the physical blocks rather than copying data (splitting procedure).

A file-granularity clone gives no observable failure, however. POST /api/storage/file/clone returns 202 and a job UUID, but the UUID resolved to 404 entry doesn't exist and appeared in no job listing. The same fsxadmin retrieves volume-create and volume-clone jobs as state=success, so this is not a permissions problem. A call naming a destination directory that does not exist also returned 202 and created nothing. Judge the outcome by inspecting the destination file.

FlexGroup — access point attachment

Creation has conditions, learned by failing in order.

Step	Result
Create with default parameters	Failed. `Volumes of this type must be at least 50GB`
Retry at 50 GiB	Failed. `Aggregates not matching FabricPool requirements: aggr1`
Retry with the aggregate named explicitly	Failed. `Minimum size is "400GB"`
400 GiB, explicit aggregate, `tiering.policy=none`, thin	Succeeded

FlexGroup volumes can be created from the AWS Management Console, the AWS CLI and the Amazon FSx API (Managing volumes). Naming the aggregate explicitly was required because this one was created through the ONTAP REST API; that constraint does not appear on the AWS-side path.

What follows is the point: whether an access point on the AWS side attaches to a FlexGroup created on the ONTAP side.

Step	Result
Appears on the FSx for ONTAP side as `VolumeStyle=FLEXGROUP`	It does, with an `fsvol-` identifier
Attach an S3 access point to the FlexGroup	Succeeded. `Lifecycle=AVAILABLE`
PUT / GET / LIST	Succeeded. The GET content matched
12 MiB multipart upload	Succeeded. `StorageClass=FSX_ONTAP`
Snapshot the FlexGroup	Succeeded
Clone that FlexGroup	Succeeded. `style=flexgroup`, `is_flexclone=true`, online

That last row is the item NetApp's table records as "volume clone of the FlexGroup volume containing ONTAP S3 buckets: not supported".

What actually failed on this path

The constraints were not on NetApp's side of the table. Within what I measured, these three are the real traps.

A security-style and identity mismatch — failure after a successful attachment

Step	Result
Attach with an NTFS-security-style volume + UNIX identity (`root`)	Succeeded. `Lifecycle=AVAILABLE`
PUT through that access point	Refused. `AccessDenied`, body only `Access Denied`
Control: same identity, same caller, UNIX volume	PUT / GET / LIST all succeeded
CIFS server on this SVM	None. The UNIX-to-Windows mapping cannot resolve

AVAILABLE does not mean the file system layer is healthy. Attachable and usable are different properties: IAM and the access point policy are both passed, and the file system layer refuses after that. The body is only Access Denied and does not name the layer. A refusal at the identity layer says no identity-based policy allows ... instead, so the difference in the body is the signal.

Trying WINDOWS identity on the same SVM does not even complete the attachment.

Step	Result
Attach with WINDOWS identity on an SVM with no CIFS server	Failed. `did not stabilize` (`NotStabilized`), stack rolled back
Whether an attachment survives the rollback	It does not. Nothing was orphaned

The wait for something created in ONTAP to appear on the AWS side

A volume created through the ONTAP API is not immediately in the AWS-side describe-volumes. Without an fsvol- identifier, neither AWS::FSx::S3AccessPointAttachment nor create-and-attach-s3-access-point can reference it.

Measurement	Result
FlexGroup, polled every 20 s with no gaps	Appeared at 599 s (about 10 min), with an `fsvol-` identifier
A FlexClone volume, polled every 20 s with no gaps	Appeared at 1,177 s (about 19.6 min)
A separate run	Still absent at 1,258 s (about 21 min)

The three observations disagree, so this is not an upper bound. The order of magnitude is tens of minutes, which is not enough to bake a fixed wait into a design. AWS documents the following, and the measurement is longer than its "several minutes".

Amazon FSx periodically syncs with ONTAP to ensure consistency. If you create or modify volumes using NetApp applications, it may take up to several minutes for these changes to be reflected in the AWS Management Console, AWS CLI, API and SDKs.

— Managing FSx for ONTAP resources using NetApp applications

This delay has to be kept apart from absence. In an observation of about 2.5 minutes, a volume created through the ONTAP API shows no fsvol- identifier, and it would be easy to conclude from that that neither a clone nor a FlexGroup can take an access point. Wait, and it appears, and it attaches. 2.5 minutes sits inside the "a few minutes" AWS documents, so what is observable in that window is "not yet appeared".

It lags in the other direction too. Setting a junction path on the ONTAP side and attaching immediately fails with Amazon FSx is unable to attach S3access point because the volume is not mounted. The AWS-side JunctionPath was None at that moment and still was two minutes later. Setting it with aws fsx update-volume made it visible on the AWS side in about 40 seconds, and the attachment then succeeded. The error message was telling the truth. What was wrong was my assumption that setting it in ONTAP made it visible to AWS.

A residual NAS bucket refusing volume deletion on the ONTAP side

First, the terms. An S3 access point is an AWS-side mechanism and an object store NAS bucket is an ONTAP-side object, and the two are different things. Attaching an access point creates a corresponding NAS bucket on the ONTAP side.

What happened here is residue of the latter. After every access point was detached, the NAS bucket remained on the ONTAP side, and ONTAP refused to delete the volume because of it.

Cannot delete volume "..." in SVM "..." because it is associated with the following
object store NAS buckets: "amazon-fsx-<volume-id>"

What was checked	Result
Where the bucket name comes from	An exact match for the volume ID. It is created per volume, not per access point
After deleting every access point	Still refused, and still refused hours later
Bringing the volume back online and mounted	Still refused
Re-attaching an access point and detaching it in the correct order	Still refused. It is independent of the access point lifecycle
`aws fsx delete-volume`	Succeeded. Both the volume and the bucket went

NetApp's documented procedure for removing a NAS bucket configuration uses vserver object-store-server bucket delete, which could not be applied here because the target could not be listed.

Whether this residue is by design or a defect is unconfirmed. No AWS documentation I could find describes a NAS bucket outliving its access points and blocking a volume delete; the documented removal path is detach-and-delete-s3-access-point. If you hit the same state, delete-volume on the AWS side will complete the teardown, but ask support to establish the cause.

Not being listed is not the same as not existing. Laid out by reader:

Reader	How it shows this bucket / S3 server
ONTAP REST `/protocols/s3/buckets`	Does not list it
ONTAP CLI `vserver object-store-server bucket show`	Does not list it
ONTAP REST `/svm/svms` (the `s3` field)	Reports the S3 server as `enabled`
ONTAP REST `/protocols/s3/services`	Does not list the same SVM

Objects that AWS manages are hidden from the standard ONTAP S3 views. I confirmed the bucket's absence across two readers, and both shared the same blind spot.

If you have made clones, the teardown order needs one more step of care. delete-volume returned DELETING and then silently went back to CREATED (observed twice). The reason appears only in LifecycleTransitionReason on describe-volumes: Failed to delete volume because it has one or more clones. The clones had been deleted, and were sitting in ONTAP's volume recovery queue. The parent kept clone.has_flexclone at true, and the AWS-side delete reads that flag and refuses. Purging the recovery queue cleared the flag and the same delete-volume worked. That queue appears in neither the console nor the FSx for ONTAP API.

Putting it in a runbook

Translated into operational terms, the measurements come to five points.

#	What to write in the runbook
1	Match the security style to the identity. UNIX identity for UNIX volumes, WINDOWS identity for NTFS ones. Avoid a UNIX identity on NTFS even where the SVM does have a CIFS server
2	Do not use `AVAILABLE` as a connectivity check. Make the acceptance criterion a PUT and a read-back after attaching
3	Budget tens of minutes into automation that creates a resource in ONTAP and then handles it through the AWS API. Poll for the `fsvol-` identifier rather than sleeping a fixed interval
4	Write through the AWS management plane. Where a setting can be made from either side, as with the junction path, the AWS side reflects faster (about 40 seconds against more than two minutes)
5	Tear down through the AWS-side API. A volume that has carried an access point can keep a NAS bucket on the ONTAP side, so a procedure built on ONTAP's `volume delete` gets stuck there (`aws fsx delete-volume` completes)

What the schedule took over

Everything so far was about whether an operation is possible. Operations has another kind of work: checking capacity, taking stock of snapshots, revisiting tiering policy, confirming throughput is still right-sized. All of it is daily or weekly, and a button for it does not get pressed. That work suits a schedule better than a screen, so six patterns live in operations/.

ID	Pattern	What it does
OPS1	capacity-rightsizing	Capacity and throughput monitoring, tier recommendation, what-if cost
OPS2	storage-efficiency	Tracks deduplication and compression efficiency, recommends enabling them
OPS3	tiering-optimizer	Cold-data analysis, tiering policy recommendation and savings estimate
OPS4	snapshot-lifecycle	Retention policy compliance check and drift detection
OPS5	cost-optimization	Cost breakdown, unit economics ($/GB), growth projection
OPS6	qos-monitoring	QoS policy adherence, bandwidth contention risk

The shape is the same in all six. An EventBridge schedule starts a Step Functions workflow, a VPC Lambda collects from the ONTAP REST API and CloudWatch, another analyses, and a third writes the report. Unit tests total 74 across the six (capacity-rightsizing 27, snapshot-lifecycle 19, tiering-optimizer 11, storage-efficiency 6, cost-optimization 6, qos-monitoring 5).

The report has two destinations. With OutputDestination=FSXN_S3AP it is written to an FSx for ONTAP volume through an S3 access point, which means anyone with the NFS or SMB mount reads it in the file explorer they already use. On the real cluster the same 4,240-byte JSON appeared on the ONTAP volume as in the S3 bucket, the Step Functions execution took around 15 seconds, and four metrics were published to the FSxOps namespace.

A recommendation comes out like this.

{
  "recommendation_type": "downsize",
  "target": "vol_backup_temp",
  "current_value": "10.0% (1024 GB allocated, 102 GB used)",
  "recommended_value": "Shrink to 205 GB or enable autosize (grow_shrink)",
  "monthly_cost_delta_usd": -102.4
}

DemoMode=true runs the whole workflow on mock data with no file system, so the report layout and the shape of the recommendations can be reviewed first. Running cost itself — Lambda, Step Functions and S3 storage — came to roughly $1-5 a month for a daily run over a handful of file systems.

Progressive automation — four levels by design, two in the code

Honestly, here. The template's AutomationLevel accepts 0 through 3.

Level	Intended behaviour	Implemented
0	Report only	Yes (default)
1	Notify over SNS	Yes
2	Execute after human review	No
3	Auto-execute within guardrails	No

The code branches in exactly one place, automation_level >= 1, deciding whether to send SNS. So 2 or 3 deploys successfully and behaves identically to Level 1. The values the parameter accepts and the values the code distinguishes do not match.

The values listed in AllowedValues are not a declaration of what is implemented — accepting a value and honouring it are different things. It is the same point Part 4 made about not shipping buttons that cannot be pressed, showing up on the parameter side instead. Until the approval flow exists, narrowing AllowedValues to 0 and 1 would be the accurate move.

None of this replaces existing tooling. The vendor's CloudWatch dashboards and auto-resize solutions, and the SSD capacity auto-expansion AWS documents, differ in immediacy and in scope. The six here suit a flow where analysis and a recommendation come first and a person decides. Choose on the use case.

On how recommendations are treated: OPS4's snapshot deletion recommendations exclude anything below the minimum retention. A recommendation is an assistive signal, not a final decision. Which retention period satisfies your regulatory requirements is a legal and compliance judgement, and neither this article nor the pattern stands in for it.

What I did not measure

"Unverified" is not "cannot be done". Every result above turned out that way, so on this path nothing is known until it is measured.

Item	State
FlexClone at LUN granularity	Unverified. Creating a LUN needs an iSCSI configuration, which is not on this path
The FlexClone split	Unverified. Documentation only
An upper bound on the ONTAP-to-AWS reflection time	Unverified. The three observations disagree
FabricPool tiering	Unverified. Needs a different aggregate configuration
QoS / deduplication and compression / SnapMirror / Vscan	Unverified
SnapLock / Object Lock	Unverified. Irreversible, so not enabled without an instruction naming the retention value
Attaching on a FlexCache cache volume	Unverified. Requires a cluster peer and an SVM peer
OPS Levels 2 and 3	Not implemented. The approval flow and auto-execution are not in the code
Whether the OPS recommendations are sound	Not evaluated. They are produced; none was applied and measured
OPS across many file systems	Unmeasured. Only a handful were run

Conclusion

Two things were left off the screen, and both were written up here.

On the ONTAP side I took NetApp's table as the starting point and measured on the S3 access point path. The four items it lists as unsupported did not appear as restrictions here. Qtrees work, a quota refuses an S3 PUT, FlexClone works at both volume and file granularity, a FlexGroup takes an access point, and a clone of that FlexGroup can be created. The constraints were elsewhere: AVAILABLE says nothing about the file system layer, something created in ONTAP takes tens of minutes to appear on the AWS side, and a volume that has carried an access point can only be deleted from the AWS side.

On the operations side, the daily work went to a schedule rather than a screen. There is one mismatch between the declaration and the code here: AutomationLevel accepts 0 through 3 while the code distinguishes only up to 1.

Every failure in these six parts looked like a correct conclusion on its own. A conclusion is not settled until one more control, or a second reader, is added.

Thank you for reading all six parts. It started with showing files in a browser and ended outside the screen entirely. I hope the series is useful to anyone operating FSx for ONTAP, or building this kind of admin screen.

All test resources were deleted. The figures come from one specific environment and configuration and will differ with workload and setup. This article does not stand in for governance or regulatory judgement.

What I learned driving FSx for ONTAP from a file portal on AWS — FlexGroup creation, capacity rebalance, and deletes that refuse (Part 5)

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Fri, 04 Sep 2026 00:59:05 +0000

Introduction

Part 4 described putting 182 ONTAP operations on a file portal: the design of delegation and record, and the treatment of buttons that cannot be pressed and buttons that were never built.

This part is what the running cluster refused while those 182 were being built. None of it was visible while reading the documentation; each one arrived as an error code. That is the difference from last time: Part 4 was about design, this one is about measurement.

The readers I have in mind are people writing code against the FSx for ONTAP ONTAP REST API. You do not need to be building a portal — if you call the same API, you stop in the same places.

What this part covers: FlexGroup creation, FlexGroup capacity rebalance, how to read the capacity numbers, deletes that refuse, and four pitfalls in the S3-compatible API. Each with the response the system actually returned.

What this part does not cover:

ONTAP internals
Absolute throughput figures (the subject here is whether something is possible and under what condition — but the defaults that decide whether a measured figure can be trusted are covered in a section below)
Running SnapLock compliance (irreversible, and it would make a test file system undeletable for months, so it was deliberately not done)

Up front:

Creating a FlexGroup through the ONTAP REST API requires naming the target aggregate. FSx for ONTAP aggregates have tiering enabled, so ONTAP's automatic placement skips them. Creating one from the AWS console or the Amazon FSx API does not need this step
The same root cause is documented on the FlexCache side as a flag (use_tiered_aggregate), and it had not reached the volume-creation side
The capacity rebalance max_runtime has a lower and an upper bound that are not in the API reference. With ONTAP's defaults it never starts
Starting a rebalance enables an irreversible setting. Stopping it does not undo it
space.used means a different quantity depending on the volume. Snapshots inside the reserve are not counted
A deleted FlexClone child volume stays invisible to the API while still blocking the parent volume's delete. There are two ways out, and both were measured
Two places the documentation and the running system disagreed, and both changed the screen's wording
Six combinations of defaults return a plausible figure while measuring something other than what you intended. Three of them come from reads being served out of cache

Test environment

Item	Value
Dates	mid-August 2026 to 2026-09-02 (JST)
Region	ap-northeast-1
ONTAP version	9.18.1P3D1
Deployment type	SINGLE_AZ_1
Caller	ONTAP REST API from a VPC Lambda (Python 3.13)
Privilege	`fsxadmin`
Cleanup	All test volumes deleted

The distinction that matters most — ONTAP's words and AWS's words

Mix these up and searching for the error text will not find the answer.

Term	Where it appears
FabricPool, `use_tiered_aggregate`	ONTAP error text and REST field names
tiering, capacity pool tier	AWS documentation and console

They refer to the same mechanism. But the error arrives in ONTAP's vocabulary while the documentation explains it in AWS's, so searching the AWS documentation for the error string returns nothing. The portal's wording follows the AWS side; only API strings are quoted as-is.

What to watch when creating a FlexGroup

A FlexGroup presents several "constituents" — internally split real volumes — as one namespace.

To set the scope: FlexGroup volumes can be created from the AWS console, the AWS CLI and the Amazon FSx API (Managing volumes). What follows is the ONTAP REST API path, which is the one the portal uses.

ONTAP normally picks the aggregates if you do not name them. On FSx for ONTAP that automatic selection does not resolve.

Aggregates not matching FabricPool requirements: aggr1

An FSx for ONTAP file system has a primary storage tier (SSD) and a capacity pool storage tier, and infrequently accessed data is tiered to the latter (AWS documentation). Aggregates in that configuration fall outside ONTAP's automatic placement.

The conditions came out one failure at a time.

Step	Result
Create with default parameters	Failed. `Volumes of this type must be at least 50GB`
Retry at 50 GiB	Failed. `Aggregates not matching FabricPool requirements: aggr1`
Retry naming the aggregate	Failed. `Minimum size is "400GB"`
400 GiB, aggregate named, `tiering.policy=none`, thin	Succeeded

Naming the aggregate works. But that aggregate name appears in neither the AWS console nor the Amazon FSx API. It has to be fetched with GET /storage/aggregates. Nobody hands the tenant an aggregate name, so that extra call is mandatory.

The 400 GB floor is configuration-dependent: the documented minimum is 100 GB per constituent, so the floor moves with the constituent count (the default is 8 per HA pair). And this extra step is only needed on the ONTAP REST API path — if you do not need to choose the aggregate yourself, creating the volume from the Amazon FSx API is shorter.

The same root cause was already known on the FlexCache side. There is a flag, use_tiered_aggregate, defaulting to false. It had not reached the volume-creation side. The same cause surfacing elsewhere with a different face is a familiar shape in a managed service that wraps upstream software.

The screen carries this note:

When creating a FlexGroup from this screen, the aggregate is named explicitly rather than placed automatically. FSx for ONTAP aggregates have tiering enabled and are excluded from automatic placement (measured: without it, creation fails with "Aggregates not matching FabricPool requirements"). With the default four constituents the minimum is 400 GB.

The capacity rebalance that never starts with the defaults

A FlexGroup places files across constituents by hash, so they drift apart over time. When one fills, the whole volume returns "no space" even though the others have room. A capacity rebalance is what corrects that.

Running it against the real cluster produced two constraints on max_runtime that are not in the API reference.

Constraint	Detail	Error code
Lower bound	Under 30 minutes is refused	`144182221`
Upper bound	Must be shorter than the time remaining until the next snapshot	`13107433`

Together, the window to start is:

30 min <= max_runtime < (time from start until the next snapshot)

ONTAP's default max_runtime is 6 hours. On a volume with the default snapshot policy (hourly at :05), 6 hours always violates the upper bound. With the value the portal offered as its default, it never started once. With the default policy the only window is the 30 minutes from :05 to :35 each hour. On a frequent schedule such as 5min, nothing starts until the policy is detached.

The boundary was pinned with a one-second A/B.

Time	`maxRuntime`	Until next snapshot	Result
16:06:12	`PT1H` (60 min)	58m47s	Refused (`13107433`)
16:06:13	`PT30M` (30 min)	58m46s	Succeeded

60 minutes > 58m47s refused; 30 minutes < 58m46s succeeded. Because the two calls are one second apart, the deciding factor is the max_runtime value and not elapsed time. ONTAP's own message names both remedies — "reduce the -max-runtime or disable the snapshot policy" — because of this structure.

The full observation log is in the FlexGroup capacity rebalance record. State values absent from the reference's volume-side list turned up there too: idle means running with nothing to move, scheduled means a reservation exists. The portal had been displaying "running" as "unknown".

A note on irreversibility

Starting a rebalance enables granular data on that volume. It is required for the rebalance, and it creates two multi-part inodes per moved file (ONTAP documentation). Stopping does not undo it. The vendor's documentation lists no way back other than deleting the volume or restoring from a snapshot taken before it was enabled.

Stopping is not a rollback either: files already moved stay where they are. Both were measured (granular_data remains true after a stop). The screen's confirmation says both.

The space that does not come back after a delete

Read the usage figure in a volume list naively and you will get it wrong, because space.used means a different quantity depending on the volume.

Snapshots that fit inside the snapshot reserve (5% by default) are not counted in space.used
Anything past the reserve consumes the same space as live data and is counted

The measurements. A 100 GiB volume reported used of 18.1 MiB while holding 77.3 MiB of snapshots. That sits inside the 5% reserve (5 GiB), so the usage bar does not move. On a different volume with a 0% reserve, used was 83,677 MiB: 81,934 MiB of live data and 1,743 MiB of snapshots. On 8 of 11 volumes, snapshots exceeded live data.

So the list was split from one usage figure into three: live data, snapshots, and reserve spill. Whether the space is not coming back because of snapshots or because live data fills it can now be read off the screen.

Sometimes the cause is on another volume and its own row does not show it. That is FlexClone. A clone is based on one of the parent's snapshots, and that snapshot stays locked while the clone exists. Deleting files on the parent frees nothing while the locked snapshot still references the blocks. Reading the parent's row alone cannot tell you that, so it is written on the FlexClone panel instead.

What to check when a FlexClone volume will not delete

After deleting a child volume created with FlexClone, deleting the parent volume is refused with has one or more clones. I reproduced that on real hardware. The awkward part is that the API cannot see that child volume. It is not in the volume listing, fetching it by name returns entry doesn't exist, and yet only the parent's delete refuses, saying a clone exists.

The cause is ONTAP's volume recovery queue, which retains deleted volumes for 12 hours by default. An A/B in the same environment, varying only whether the clone was split first:

Steps	Parent delete
Delete an unsplit clone, then delete the parent	Failed (still failing after 7 and 15 minutes)
Split the clone, delete it, then delete the parent	Succeeded within seconds

The exit was measured too. GET /api/private/cli/volume/recovery-queue reads the queue. POST .../purge goes through with fsxadmin privilege, the entry left the queue in about 20 seconds, and the parent delete succeeded immediately afterwards.

This is a place where I published something wrong and have corrected it. The earlier text said purge needs diag privilege, is unreachable as fsxadmin, and therefore the only option is to wait 12 hours. That was wrong. I blamed privilege without trying it.

The screen's error carries the causation and both exits (split first, or purge). Purge cannot be undone, so it also says to confirm the volume was yours before running it.

Four pitfalls in the S3-compatible API

Behaviours worth knowing before you call an FSx for ONTAP S3 access point from a browser or an SDK. Knowing them changes how long the investigation takes.

The 501 failure of a "write only if absent" request

PutObject with if-none-match: * returns 501 NotImplemented, with the body A header you provided implies functionality that is not implemented.

Request	Result
`PUT` + `if-none-match: *`	501 NotImplemented
`PUT` + `x-amz-checksum-crc32` (no `if-none-match`)	200 OK
`GET` / `ListObjectsV2`	200 (neither header is sent)

CRC32 flexible checksums pass. Only if-none-match fails. Reads are unaffected, so the symptom presents as "the listing works but every write fails."

This is not somebody else's problem. Amplify's Storage Browser translates its "do not overwrite" option into that header, so dropping it in unchanged breaks every upload and folder creation. If you need overwrite protection, look the key up before writing instead. That does not give you if-none-match's atomicity, and two concurrent writes to the same key can both decide it is absent.

GOVERNANCE and COMPLIANCE, indistinguishable from the error text

S3 Object Lock has two retention modes. GOVERNANCE lets a user holding a specific permission (s3:BypassGovernanceRetention) delete inside the retention period; COMPLIANCE lets nobody delete until the retention expires. Deleting an object written with one day of retention, compared across both modes:

Operation	GOVERNANCE	COMPLIANCE
Delete without bypass	`AccessDenied ... object protected by object lock`	Identical text
Delete with `BypassGovernanceRetention`	Succeeds	Refused with identical text

Because the text is identical, the response does not tell you why the bypass did not work. The first things to read are get-object-lock-configuration and head-object's ObjectLockMode rather than the IAM policy, because the same text covers both a missing permission and a mode that has no bypass.

The lock indicator that does not change under automatic locking

Assigning a policy whose schedules carry a retentionPeriod leaves the locking feature flag at false; only the assigned policy name changes. Reading the state alone suggests nothing will be locked, while every snapshot that policy takes is in fact locked. The judgement needs the assigned policy's retentionPeriod as well.

The usable endpoints, derived from the API rather than a config file

An alias copied into a config file looks equally correct for an access point that has been deleted and for one that is MISCONFIGURED. fsx describe-s3-access-point-attachments returns Lifecycle and the Internet or VPC origin, so derive the inventory from there. Always follow the pagination: read only the first page and an access point that exists goes missing in a way indistinguishable from one that does not.

Two places the documentation and the running system disagreed

Two things behaved differently from what I read before implementing them. Both changed the screen's wording.

Subject	Expected from the documentation	Measured (ONTAP 9.18.1P3D1)
Deleting a QoS policy while assigned	The CLI reference says it is refused without `-force`	REST accepts it, and the volume's assignment is silently removed (every ceiling becomes 0, meaning unlimited)
Enforcement after deleting a quota rule	The REST reference says it stays in effect until enforcement is cycled off and on	The deleted rule's ceiling disappeared from the usage report immediately

The QoS confirmation text was rewritten. The correct warning is not "it cannot be deleted" but "deleting it removes the ceilings from every volume using that policy". A note was added that releasing a single volume means assigning none, not deleting the policy.

The six defaults that can invalidate a measurement

Up to here the defaults decided whether an operation goes through. Measurement has defaults of the same shape. None of them returns an error: each returns a plausible figure while measuring something other than what you intended.

Where these figures come from: a different measurement from the environment above. 2026-09-01 to 02, ap-northeast-1, ONTAP 9.18.1P3D1, SINGLE_AZ_1, throughput tiers 128 MBps and 2048 MBps, client c5n.9xlarge. Repeatability across repeat runs of the same measurement was 0.02–0.14%.

The sixth is from a different measurement. 2026-09-06, ap-northeast-1, ONTAP 9.18.1P3D1,
SINGLE_AZ_2, throughput 6,144 MBps, an SMB SVM joined to AWS Managed AD, clients c5n.2xlarge and
c5n.9xlarge. A different generation and a different tier from the five above, so do not put the
figures side by side.

Default	What happens	How to notice	What to do
Linux NFS uses one TCP connection	Stops around 590 MB/s and does not respond to stream count	1 / 4 / 8 give the same figure	`nconnect=16`
`dd if=/dev/zero`	Zero blocks skip disk and return at 4x the published ceiling	Exceeds the published ceiling	Use incompressible data
Volume inline efficiency is on	Compressible or identical payloads collapse (4x on reads, under 5% on writes)	`space_savings.dedupe_percent` is high	Turn it off before writing
`DiskIopsConfiguration: AUTOMATIC`	3 IOPS/GiB, so IOPS becomes the limit	MB/s divided by IOPS is implausibly small	`USER_PROVISIONED`
A read only slightly over the cache size	The cache-served share dominates and the disk path is not measured	`DiskReadBytes` divided by `DataReadBytes` is a few per cent or less	Read at least twice the cache in one pass
SMB Multichannel ships disabled	One channel, and it holds while the session reports `dialect=3.1.1` (1 MiB sequential read 942 against 1,824 MB/s)	One established TCP connection to the SVM; `Get-SmbMultichannelConnection` returns nothing	`vserver cifs options modify -is-multichannel-enabled true`. It does not reach existing connections — restart the client's SMB service and remap

How the fifth one surfaced. I assumed 280 GiB would not fit in cache. Read through Amazon CloudWatch's AWS/FSx namespace, DiskReadBytes divided by DataReadBytes came to 1.4 / 0.9 / 1.5%. 98.5-99.9% of the bytes read never touched disk.

The 2048 MBps tier's in-memory cache is 256 GB, that is 238 GiB (FSx for ONTAP performance; ap-northeast-1 falls under the first-generation Single-AZ "all other Regions" table). 280 GiB exceeds it by 18%, not by a factor of two or three. "Past the cache" and "not served from the cache" are different things.

Inline efficiency can be restored, provided you wait.

# Rejected unless efficiency.op_state is idle
PATCH /api/storage/volumes/{uuid}  {"efficiency": {"compression": "inline"}}
PATCH /api/storage/volumes/{uuid}  {"efficiency": {"dedupe": "both"}}

But only turn it off on a volume you can discard. Immediately after writing, a background efficiency operation is running and the wait is not predictable.

Putting it in a runbook

#	What to write down
1	To create a FlexGroup through the ONTAP REST API, fetch the aggregate with `GET /storage/aggregates` and name it. If you do not need to choose the aggregate, create the volume from the Amazon FSx API instead
2	Set the rebalance `max_runtime` to at least 30 minutes and below the time to the next snapshot. Do not pass the default through
3	Do not treat starting a rebalance as an operation you can press at any time. `granular data` is enabled irreversibly
4	Do not show usage as a single number. Separate live data from snapshots, or nobody can reach the cause of "deleting files did not free space"
5	A teardown procedure for an environment with clones must include the split, or the recovery queue purge. "The clone was deleted, so the parent can go" does not hold
6	Errors arrive in ONTAP's vocabulary. Keep the AWS-to-ONTAP term mapping at hand before searching
7	A runbook for measuring performance states how to drop the defaults first. Leave any one of zero-filled data, inline efficiency, an unspecified `nconnect` or `AUTOMATIC` IOPS in place, and the figure produced is measuring something else

What I did not measure

"Unverified" is not "impossible". As last time, the boundary stated as a boundary:

Item	State
The actual effect of a rebalance (how far the skew is evened out)	Unmeasured. Only the conditions to start were measured
Performance impact after `granular data` is enabled	Unmeasured
A recovery queue with a changed retention	Unverified. Measured at the 12-hour default
Actual tiering into the capacity pool	Unverified. Needs a different aggregate configuration
Running SnapLock compliance	Deliberately not done. Irreversible, up to the file system
An alternative path for copies over 5 GiB (NFS or SMB)	Unmeasured. The portal stops at refusing

Part 6 covers the other side: what was left off the screen. How much of ONTAP stays usable alongside the S3 access point path, and the operational work that was handed to a schedule instead of a button.

Conclusion

Nothing in this part was visible while reading the documentation. Creating a FlexGroup through the ONTAP REST API needs the aggregate named, a capacity rebalance never starts with the defaults, space.used means a different quantity per volume, and a deleted FlexClone volume blocks the parent's delete while invisible through the API.

What they share is that the error text does not name the cause. That is why the causation and the exits live on the screen. As with the one-second A/B on the rebalance, another explanation looked just as correct until one control was added.

There is also one correction. I published "purge needs diag privilege, so wait 12 hours" and later found it goes through. I blamed privilege without trying it.

The figures come from one specific environment and configuration and will differ with workload and setup. The error codes were observed on ONTAP 9.18.1P3D1.

DEV Community: AWS Community Builders

Kiro Crew: qué es y cómo empezar a usarlo

Kiro Crew: qué es y cómo empezar a usarlo

1. ¿Qué es Kiro Crew?

2. Instalar Kiro Crew

3. Memory, Scheduled Tasks, Artifacts y Apps

Memory y Knowledge

Scheduled Tasks

Artifacts

Apps: mucho más que un chat

4. Seguridad, Channels y ejecución remota

Channels

Ejecución remota

5. ¿Qué probar primero?

Agentic AI Development with Kiro: The Hidden DevSecOps Layer — Closing the Loop

Open loop – Starter point a classical DevSecOps CICD system

The maturity model we use

Delivery

Security gates (the DevSecOps core)

Kiro adoption (the differentiator)

The Level 4 Vision: Agent-Integrated DevSecOps

The Level 4 loop:

What "closing the loop" requires

Principles that keep this safe

Why “company” is ready for Level 4

The flagship Level 4 initiative

Conclusion: Closing the Loop

FAIR Data for Agentic AI

The AI Transformation: From Predictions to Actions

Decoding AI Agents

The Building Blocks of an Agent

The Next Generation of Amazon SageMaker

A Unified Architecture

Transforming Raw Data into AI Wisdom

The Data Pipeline Steps

The Knowledge Pyramid

The Core: F.A.I.R. Data Principles

Key Takeaways

Conclusion

About the Author

References

Also Published On

From Data Lake to Lakehouse: Building Modern Analytics Platforms on AWS with S3 Tables & AWS Glue

The Headache of Traditional Data Architectures

Enter the Lakehouse and Apache Iceberg

The Maintenance Trap

S3 Tables to the Rescue

Structuring Data: The Medallion Architecture

Getting Data In: Choose Your Weapon

The "Wow" Moments: ACID, Time Travel, and Schema Evolution

Turning Data into Business Insights

The Future is Here: Claude + MCP Server

Key Takeaways

Conclusion

About the Author

References

Also Published On

I Tried Nx Plugin for AWS, Here's Why I'm Sold

A quick primer on Nx

What it actually gives you

Worthy callouts

The generators

The connection generator is the real workhorse

Prerequisites

Optional IDE extensions

Scaffolding a project

Build and deploy

Better together with AI assistants

Configure your AI assistant

Where to from here

Final thoughts

Links

EKS Auto Mode: correr Kubernetes sin andar peleando con los nodos

Mantener nodos es casi un segundo trabajo

Entonces, ¿Qué es EKS Auto Mode?

Lab: de cero a app corriendo, con nodos que aparecen solos

Lo que necesitas:

Paso 1: Crear el cluster con Auto Mode

Paso 2: Mirar el cluster recién creado

Paso 3: Desplegar una app