DEV Community: Ramón Chancay 👨🏻‍💻

Datos sensibles fuera del prompt: una librería de TypeScript escrita en 39 minutos desde su especificación

Ramón Chancay 👨🏻‍💻 — Mon, 07 Sep 2026 21:03:28 +0000

Cada vez que un backend concatena un mensaje de usuario dentro de un prompt, escribe el body de una petición en un log o reenvía un payload a un helpdesk, está sacando datos personales del proceso hacia un sistema que nunca fue diseñado para guardarlos. sanitype es una librería de TypeScript que interpone una llamada entre ese payload y su destino: recibe el objeto, devuelve una copia con la misma forma y los campos sensibles redactados, enmascarados, hasheados o eliminados, más un reporte de todo lo que tocó. Corre entera dentro del proceso, sin llamadas de red y sin dependencias en tiempo de ejecución. También se construyó de una forma poco habitual: la idea se dictó por voz a un agente, el agente escribió 563 líneas de especificación y Claude Code implementó la versión 0.1.0 contra ellas en un solo objetivo, 39 minutos después.

TL;DR

Combina dos estrategias: reglas por ruta de campo para lo que ya sabes que es sensible, y detectores sobre texto libre para el correo que alguien pegó en un campo de notas.

Por defecto, entra un objeto y sale un objeto con la misma estructura. Cada llamada devuelve un reporte de qué se tocó, dónde y con qué detector, y ese reporte nunca contiene el valor original.

El repositorio se escribió al revés de lo habitual: primero SPEC, ARCHITECTURE, ROADMAP y COMPARISON; después el código. Entre el commit de las especificaciones y el de la implementación pasaron 39 minutos.

sanitype en veinte segundos:

tu backend
    │
    ▼
sanitize(payload)
    ├─ reglas por campo   lo que ya sabes que es sensible
    ├─ detectores         lo que aparece dentro del texto libre
    └─ acción             redact · mask · hash · drop · tokenize
    │
    ▼
LLM · logs · analítica · APIs de terceros
    +  report: qué se tocó, dónde y con qué detector

Qué datos sensibles se escapan de un backend

El problema no es que alguien quiera filtrar datos. Es que hay cuatro salidas habituales por las que un payload sale completo del proceso, y ninguna de las cuatro se siente como una decisión sobre privacidad cuando la escribes:

payload del usuario
  ├──► prompt de un LLM       OpenAI, Anthropic, un modelo local
  ├──► logs y observabilidad  Sentry, Datadog, el logger de peticiones
  ├──► analítica              Segment, PostHog, eventos internos
  └──► APIs de terceros       helpdesk, webhooks, integraciones de socios

La primera es la más nueva y la más silenciosa. Un ticket de soporte con el teléfono y la cédula del cliente se concatena tal cual dentro del prompt, y el prompt viaja a un proveedor externo que puede retenerlo. La segunda es la más vieja: alguien puso logger.info({ body: req.body }) para depurar un caso y ese log lleva dos años recogiendo correos y números de tarjeta. La tercera y la cuarta son variantes del mismo descuido: se manda el objeto entero porque separar los campos que el destino sí necesita da trabajo.

Lo que estas cuatro tienen en común es que ocurren en el borde de salida del proceso. Ahí es donde hay que interponer algo, y ese algo tiene que ser barato de llamar, porque si cuesta una llamada de red nadie lo va a poner en el logger.

Sigue leyendo

Hasta aquí la primera mitad. El recorrido completo — con el resto de la implementación, las decisiones de diseño y lo que solo aparece en producción — está en mi blog:

Lee el artículo completo en ramonchancay.me →

Publicado originalmente en www.ramonchancay.me/es/blog/limpiar-datos-sensibles-antes-del-llm-sanitype.

Sensitive data out of the prompt: a TypeScript library written in 39 minutes from its specification

Ramón Chancay 👨🏻‍💻 — Mon, 07 Sep 2026 21:03:25 +0000

Every time a backend concatenates a user message into a prompt, writes a request body to a log or forwards a payload to a helpdesk, it is moving personal data out of the process and into a system that was never designed to hold it. sanitype is a TypeScript library that puts one call between that payload and its destination: it takes the object and returns a copy with the same shape and the sensitive fields redacted, masked, hashed or dropped, plus a report of everything it touched. It runs entirely in-process, with no network calls and no runtime dependencies. It was also built in an unusual way: the idea was dictated by voice to an agent, the agent wrote 563 lines of specification, and Claude Code implemented version 0.1.0 against them in a single goal, 39 minutes later.

TL;DR

It combines two strategies: field-path rules for what you already know is sensitive, and detectors over free text for the email someone pasted into a notes field.

By default an object goes in and an object with the same structure comes out. Every call returns a report of what was touched, where and by which detector, and that report never contains the original value.

The repository was written the other way round: first SPEC, ARCHITECTURE, ROADMAP and COMPARISON, then the code. 39 minutes passed between the specification commit and the implementation commit.

sanitype in twenty seconds:

your backend
    │
    ▼
sanitize(payload)
    ├─ field rules   what you already know is sensitive
    ├─ detectors     what shows up inside free text
    └─ action        redact · mask · hash · drop · tokenize
    │
    ▼
LLM · logs · analytics · third-party APIs
    +  report: what was touched, where and by which detector

What sensitive data leaks out of a backend

The problem is not that somebody wants to leak data. It is that there are four common exits through which a payload leaves the process whole, and none of the four feels like a decision about privacy when you write it:

user payload
  ├──► LLM prompt        OpenAI, Anthropic, a local model
  ├──► logs and tracing  Sentry, Datadog, the request logger
  ├──► analytics         Segment, PostHog, internal events
  └──► third-party APIs  helpdesk, webhooks, partner integrations

The first one is the newest and the quietest. A support ticket carrying the customer's phone number and national ID gets concatenated into the prompt as it is, and the prompt travels to an external vendor that may retain it. The second is the oldest: somebody added logger.info({ body: req.body }) to debug one case, and that log has been collecting emails and card numbers for two years. The third and fourth are variants of the same carelessness: the whole object goes out because separating the fields the destination actually needs takes work.

What these four have in common is that they happen at the outbound edge of the process. That is where something has to sit, and that something has to be cheap to call, because if it costs a network round trip nobody is going to put it in the logger.

Why nothing that already exists fit

Before writing a line I went through the landscape, and I wrote it down in a COMPARISON.md inside the repository, because "why not just use X" is a question that shows up in the first issue.

Option	What it does well	Why it did not fit
DLP as a service (Google Cloud DLP, Purview, Macie)	Broad entity coverage, model-based detection, compliance tooling around it	A network call per scan: latency, cost per request and one more vendor dependency
Self-hosted Microsoft Presidio	Mature, strong entity recognition, no per-call cost	It is a Python project: from a Node backend it means deploying and monitoring another service
Regex npm packages	Lightweight, dependency-free, easy to read	They work on loose text: they do not know that `user.ssn` in your type is a national ID, and they return no report
Browser extensions	They catch what a user pastes into a UI	They see nothing of server-to-server traffic, which is exactly the case here

The gap is specific: a native TypeScript library that runs in-process, understands the shape of your objects and not just the text, and returns an auditable record of what it did. That is the position sanitype occupies, and the price it pays for it is in the comparison: there is no entity-recognition model behind it, and free-text detection is bounded by the patterns it ships.

Keep reading

That is the first half. The full walkthrough — with the rest of the implementation, the trade-offs and the things that only show up in production — is on my blog:

Read the full post on ramonchancay.me →

Originally published at www.ramonchancay.me/blog/scrubbing-pii-before-the-llm-sanitype.

Spec-Driven Development en React Native: un MVP que lista los sismos del mundo, especificado antes de escribir código

Ramón Chancay 👨🏻‍💻 — Sat, 05 Sep 2026 21:53:33 +0000

Un agente de código ya escribe una app funcional en una tarde. El problema dejó de ser producir el código y pasó a ser otro: el agente no escribe peor código sin una especificación, escribe un código excelente para el problema equivocado. Cada pregunta que no contestaste la contesta él con un valor por defecto razonable, y lo descubres semanas después. El Spec-Driven Development (SDD) ataca justo eso: escribir primero qué debe hacer el software, resolver por escrito lo que está ambiguo, y tratar el código como el resultado de esa especificación. Lo apliqué a un MVP en React Native con Expo que hace una sola cosa —mostrar un listado de los sismos recientes en todo el mundo— porque es el tipo de pedido que cabe en una línea y esconde más decisiones de las que parece.

TL;DR

SDD separa tres artefactos: la especificación (qué y por qué), el plan técnico (cómo) y las tareas. El agente implementa contra ellos en vez de contra un prompt suelto.

El valor real no es el documento: es que las ambigüedades quedan marcadas y resueltas por escrito antes de que el agente elija por ti. Qué sismos entran en la lista, en qué orden y qué pasa sin conexión son decisiones de producto, no de código.

Los criterios de aceptación se escriben como una tabla de casos y esa misma tabla se convierte en el test. Si la fila no está en la tabla, el comportamiento no está definido.

Qué es el Spec-Driven Development y en qué se diferencia de prompting

Un prompt es una instrucción que se consume y desaparece. Una especificación es un archivo versionado en el repositorio, que se revisa en un pull request y que sobrevive a la sesión del agente. Esa es toda la diferencia, y es más grande de lo que parece.

En el flujo típico con un agente, describes la funcionalidad en el chat y el resultado son mil líneas de código más una conversación que nadie va a releer. Las decisiones importantes —la magnitud mínima, cuántas horas hacia atrás, qué pasa sin conexión— quedaron en un turno intermedio del chat. Cuando alguien pregunta por qué la app muestra un sismo de magnitud 2.6 y no uno de 2.4, la respuesta no está en ninguna parte.

SDD separa el trabajo en artefactos distintos, cada uno con su propio nivel de abstracción:

Especificación   →  qué debe pasar y por qué
      │
      ▼
Plan técnico     →  cómo se implementa (stack, librerías, estructura)
      │
      ▼
Tareas           →  unidades de trabajo verificables
      │
      ▼
Implementación   →  el código que el agente escribe contra lo anterior

La regla que ordena todo es que cada nivel habla de su nivel. La especificación evita decisiones de implementación salvo que sean una restricción real del producto: "debe funcionar sin conexión" o "los datos no salen del dispositivo" son requisitos aunque tengan consecuencias técnicas; "usar TanStack Query" no lo es. El plan sí nombra tecnología, pero no reabre decisiones de producto. Cuando esa separación se respeta, puedes cambiar de stack sin reescribir la especificación, y cambiar un umbral sin tocar el plan.

Hay herramientas que formalizan este flujo. Spec Kit, de GitHub, instala una serie de comandos en tu agente que producen exactamente estos artefactos en carpetas del repositorio:

# Instala el flujo en el proyecto y elige el agente que vas a usar.
uvx --from git+https://github.com/github/spec-kit.git specify init sismos-app

# A partir de ahí, dentro del agente:
#   /constitution  → principios del proyecto (aplican a todas las features)
#   /specify       → la especificación de esta feature
#   /clarify       → resuelve las ambigüedades marcadas en la spec
#   /plan          → el plan técnico
#   /tasks         → el desglose en tareas
#   /implement     → ejecuta las tareas

No hace falta la herramienta: tres archivos markdown en specs/ y la disciplina de mantenerlos alcanzan. Lo que aporta es que el agente tiene los comandos, las plantillas y el orden ya cargados, y no se salta pasos.

Sigue leyendo

Hasta aquí la primera mitad. El recorrido completo — con el resto de la implementación, las decisiones de diseño y lo que solo aparece en producción — está en mi blog:

Lee el artículo completo en ramonchancay.me →

Publicado originalmente en www.ramonchancay.me/es/blog/spec-driven-development-react-native-app-sismos.

Spec-Driven Development in React Native: an MVP that lists the world's earthquakes, specified before writing code

Ramón Chancay 👨🏻‍💻 — Sat, 05 Sep 2026 21:53:29 +0000

A coding agent can already write a working app in an afternoon. The problem is no longer producing the code; it is something else: without a specification the agent doesn't write worse code, it writes excellent code for the wrong problem. Every question you didn't answer, it answers with a reasonable default, and you find out weeks later. Spec-Driven Development (SDD) attacks exactly that: write down first what the software must do, resolve in writing whatever is ambiguous, and treat the code as the output of that specification. I applied it to a React Native MVP built with Expo that does one thing—show a list of recent earthquakes anywhere in the world—because it is the kind of request that fits in one line and hides more decisions than it seems.

TL;DR

SDD separates three artifacts: the specification (what and why), the technical plan (how) and the tasks. The agent implements against them instead of against a loose prompt.

The real value is not the document: it is that ambiguities get flagged and resolved in writing before the agent chooses for you. Which earthquakes make the list, in what order, and what happens offline are product decisions, not code decisions.

Acceptance criteria are written as a table of cases, and that same table becomes the test. If the row is not in the table, the behavior is undefined.

What Spec-Driven Development is and how it differs from prompting

A prompt is an instruction that gets consumed and disappears. A specification is a versioned file in the repository, reviewed in a pull request, that outlives the agent's session. That is the whole difference, and it is bigger than it looks.

In the typical flow with an agent, you describe the feature in the chat and the result is a thousand lines of code plus a conversation nobody will reread. The important decisions—the minimum magnitude, how many hours back, what happens offline—were made in some intermediate turn of the chat. When someone asks why the app shows a magnitude 2.6 earthquake but not a 2.4, the answer is nowhere.

SDD splits the work into distinct artifacts, each at its own level of abstraction:

Specification    →  what must happen and why
      │
      ▼
Technical plan   →  how it gets built (stack, libraries, structure)
      │
      ▼
Tasks            →  verifiable units of work
      │
      ▼
Implementation   →  the code the agent writes against the above

The rule that orders everything is that each level talks about its own level. The specification avoids implementation decisions unless they are a real product constraint: "must work offline" or "data never leaves the device" are requirements even though they have technical consequences; "use TanStack Query" is not. The plan does name technology, but it doesn't reopen product decisions. When that separation holds, you can switch stacks without rewriting the specification, and change a threshold without touching the plan.

There are tools that formalize this flow. GitHub's Spec Kit installs a set of commands in your agent that produce exactly these artifacts in folders of the repository:

# Installs the flow in the project and picks the agent you are going to use.
uvx --from git+https://github.com/github/spec-kit.git specify init quakes-app

# From there, inside the agent:
#   /constitution  → project principles (apply to every feature)
#   /specify       → this feature's specification
#   /clarify       → resolves the ambiguities flagged in the spec
#   /plan          → the technical plan
#   /tasks         → the task breakdown
#   /implement     → executes the tasks

You don't need the tool: three markdown files in specs/ and the discipline to maintain them are enough. What it adds is that the agent has the commands, the templates and the order already loaded, and doesn't skip steps.

Keep reading

That is the first half. The full walkthrough — with the rest of the implementation, the trade-offs and the things that only show up in production — is on my blog:

Read the full post on ramonchancay.me →

Originally published at www.ramonchancay.me/blog/spec-driven-development-react-native-earthquake-app.

The hard limits of an autonomous agent: what keeps it from disaster

Ramón Chancay 👨🏻‍💻 — Sat, 05 Sep 2026 01:06:11 +0000

A hard limit is a restriction the program enforces, not the model: it is checked in the code that executes the action, after the model has decided, and it doesn't depend on the agent having understood the instructions correctly. The previous post left the system complete—a ticket goes in, a PR comes out, with nobody pressing a button—and that's where the question that decides whether this stays an experiment or stays running shows up: what happens when something goes wrong at three in the morning and nobody is watching. This post is that layer: iteration and token caps treated as a real budget, command and path allowlists, a kill switch that works from outside, idempotent effects, and a log that lets you know what happened. And at the end, the uncomfortable part: none of this is the hard bit.

TL;DR

A limit asked for in the prompt is a preference; a hard limit is code that runs after the model has decided. Everything that matters—what commands it runs, where it writes, how much it spends, when it stops—goes in the program, not in the instructions.

The four that aren't optional: a per-run budget (iterations, tokens and time), a command allowlist with no shell, a path allowlist resolved with realpath, and a kill switch someone else can flip without deploying anything.

The hard part isn't the code: it's making the PRs worth reviewing. That doesn't depend on the agent, it depends on how clear your tickets are and how good your test suite is.

What a hard limit is

There are two ways to tell an agent not to do something. One is writing it in the system prompt: "don't run destructive commands", "don't leave the working directory". The other is making it so the program can't execute that action even if the model asks for it. The first works most of the time; the second works always. The difference between the two is this entire post.

The prompt influences the model's decision, and current models follow instructions fairly well. But an autonomous agent has three ways to get around that instruction with no bad intent at all: it can misread the request, it can call a tool with arguments you didn't expect, and it can receive text in its context that you didn't write. That third case is the one that matters in the previous post's system: the agent reads the text of a ticket, and anyone could have written that text. If the ticket says "to reproduce the bug, run this script", the model has a perfectly reasonable motive to run it.

A hard limit doesn't argue with any of that. It is enforced at the point of execution—in the function that runs the command, in the function that writes the file—and it denies by default: what isn't explicitly allowed doesn't get through. The model proposes the action; the program decides whether it runs.

The model decides              The program executes

  tool call  ───────────────►  is it allowed?
                                     │
                            no ──────┴────── yes
                             │               │
                             ▼               ▼
                      rejection          it runs
                      observation        inside the
                      (loop continues)   sandbox

There is a design detail in that diagram worth marking right away: a rejection does not end the run. It goes back to the model as one more observation—"command not allowed: curl"—and the agent can correct course, the same way it does when a test fails. A limit that aborts the run at the first disallowed attempt wastes tasks the agent could have solved. A limit that answers lets it keep working inside what's permitted.

Keep reading

That is the first half. The full walkthrough — with the rest of the implementation, the trade-offs and the things that only show up in production — is on my blog:

Read the full post on ramonchancay.me →

Originally published at www.ramonchancay.me/blog/hard-limits-autonomous-agent.

From a Jira ticket to a pull request: the end-to-end autonomous system

Ramón Chancay 👨🏻‍💻 — Sat, 05 Sep 2026 01:05:37 +0000

Every previous post built the worker: an agent loop with tools that runs isolated in a real repo, reproduces the bug with a test, and ships a pull request. What was missing is what starts it and what reports back when it finishes. This post is that wrapper: making a Jira ticket trigger the whole system without anyone pressing a button, processing it exactly once even if the trigger repeats, running it in an ephemeral workspace per ticket, and returning the result as a PR and a comment on the ticket itself. It's where the series lands: the agent stops being something you run by hand and becomes a service that reacts to tickets.

TL;DR

The trigger doesn't run the agent: it only enqueues. A Jira webhook gives low latency but drops events; JQL polling is slow but loses nothing. Use the webhook to react fast and polling as the net that recovers what the webhook missed.

The property that holds everything together is idempotency: process each ticket exactly once. You get it with persistent state in a database and an atomic claim, not with an in-memory flag that's lost when the process restarts.

Each ticket runs in its own ephemeral workspace (the worktree from the previous post) and the output is a PR plus a Jira comment with the link. A person still decides the merge.

What changes when you make it autonomous

Up to the previous post, the agent was a program you invoked: you handed it a goal, it ran in its worktree, opened the PR, and finished. Someone had to start it. Making it autonomous is removing that manual start: the system watches a source of work—a queue of Jira tickets—and, when one shows up marked as ready, it processes it on its own.

The change isn't in the agent, which is still the same loop with the same tools. It's in the three pieces around it: something that learns there's work to do (the trigger), something that guarantees that work is done once and isn't lost if the process crashes (the stateful queue), and something that returns the result to where the request was born (the Jira comment). The agent does the work; this post builds what connects it to a real input and a real output.

Before (manual)                    Now (autonomous)

you ──► runAgent(goal)             Jira ticket ──► trigger ──► queue
          │                                                      │
          ▼                                                      ▼
         PR                                          worker ──► agent ──► PR
                                                                 │
                                                                 ▼
                                                          comment on Jira

The word "autonomous" is scarier than it should be, and it also overpromises. It's not that the agent decides what to do about the business; it's that nobody has to copy the ticket text and launch the process by hand. The criterion of what gets done is still set by a person when they mark the ticket, and the criterion of what gets integrated is still set by whoever reviews the PR. Autonomous here means "no intervention in the middle," not "no control."

It's worth being honest about what this system is not: it doesn't plan, doesn't prioritize, doesn't coordinate some tasks with others, doesn't remember anything from one ticket to the next, and doesn't orchestrate dependencies between them. It's a worker that reacts to a queue, one ticket at a time. Everything that follows is the infrastructure that makes that reaction reliable—so the work isn't lost or duplicated—not an intelligence that decides on its own. If you expect planning or coordination, this isn't that system; it's the step before, and it's the one almost everyone needs first.

Keep reading

That is the first half. The full walkthrough — with the rest of the implementation, the trade-offs and the things that only show up in production — is on my blog:

Read the full post on ramonchancay.me →

Originally published at www.ramonchancay.me/blog/from-a-jira-ticket-to-a-pull-request.

The agent inside a real repo: isolating tasks with git worktree and shipping a PR

Ramón Chancay 👨🏻‍💻 — Sat, 05 Sep 2026 01:05:33 +0000

The previous post gave the agent tools to read, search, edit, and run across a multi-file repository, but running directly on the same working copy you work in. That's fine for understanding the loop; not for turning it loose on a real repo. This post is the missing wrapper: where the agent runs when the repository actually matters. The idea is to give each task an isolated copy—with git worktree and its own branch—have it reproduce the bug with a test before touching anything, run the whole suite so it doesn't break what already worked, and ship its output as a pull request, not a merge.

TL;DR

Don't let the agent run on your working copy. Give each task its own isolated copy with git worktree and a branch: a failed attempt is a branch you delete, and two tasks at once don't step on each other.

The flow that works for bugs is writing the failing test first (red), fixing until it's green, then running the WHOLE suite at the end to catch regressions. The new test passing isn't enough.

The agent's output is a PR, not a merge. It proposes a diff on its own branch; a person reviews and integrates it. Isolating the work and shipping it as a PR is what lets you leave it running unsupervised.

Why the agent shouldn't touch your working copy

The tools from the previous post—write_file and run_command—operate on the filesystem: they write real files and run real commands. While the repo is a toy example, it doesn't matter where they run. On a real repository, where they run is the first serious decision, because the agent gets steps wrong: it opens the wrong file, leaves a change half-done, runs a command that fails. If all of that happens on your active working copy—the working tree, the files you have open right now—the damage is direct.

Three problems show up immediately. The first is that the agent's work mixes with yours: if you had uncommitted changes, you now don't know which lines are yours and which the agent put there. The second is that an attempt aborted mid-edit leaves broken files in your tree, and cleaning that up by hand is exactly what you wanted to avoid. The third is that you can't run two tasks at once, because both would write to the same files.

On your working copy             Isolated per task
(bad)                             (good)

your work   ─┐                    your work ─► main copy (untouched)
agent t1    ─┼─► same tree        agent t1  ─► worktree 1 + branch agent/t1
agent t2    ─┘   (collide)        agent t2  ─► worktree 2 + branch agent/t2

The solution is the same one a human team uses to avoid stepping on each other: each task works on its own copy and its own branch. The only difference here is that the program creates and destroys the copy, not a person.

Keep reading

That is the first half. The full walkthrough — with the rest of the implementation, the trade-offs and the things that only show up in production — is on my blog:

Read the full post on ramonchancay.me →

Originally published at www.ramonchancay.me/blog/agent-in-a-real-repo.

From generating files to using tools: a code agent's ReAct loop

Ramón Chancay 👨🏻‍💻 — Sat, 05 Sep 2026 01:04:49 +0000

So far in this series, the agent generated the whole solution in a single call: you gave it the spec and the tests, and the model returned the entire function. That works when the whole task fits in a prompt. But in a real repository you don't know in advance which files you'll need, and you can't fit thousands of them into the context window. The leap in this post is to give the agent tools so it can get that context itself: read, search, edit, and run commands. That pattern—reason, act with a tool, observe the result, and repeat—is called ReAct, it's the "action" component the first post promised to open, and it's where a toy loop starts to look like Claude Code.

TL;DR

The shift is from "generate the whole function" to "read, search, edit, run". The model stops emitting the solution and starts emitting tool calls; the program runs them and hands the result back.

The agent builds its own context. Instead of you cramming the repo into the prompt, it reads and searches only what it needs, when it needs it. The context is the result of its observations, not something you preload.

It's the ReAct loop: reason, act, observe. The same loop, evaluator, and sandbox from earlier posts, but the action is no longer single—it's a choice among several tools—and that choice is what makes it feel like a real code agent.

Why generating the whole file stops working

In the write-test-fix loop the agent's action was a single one: "write the function". The model received everything it needed in the prompt—the spec and the tests—and returned the complete solution in one block. It worked because the example fit entirely in the prompt: a small function, a few tests, nothing more.

That condition breaks the moment you leave the toy case. A real task—"fix the 500 the users endpoint returns", "add a field to this model"—lives in a repository of thousands of files. You can't put the whole repo in the prompt: it doesn't fit in the context window, and even if it did, you'd pay a fortune to send thousands of irrelevant files on every call. And there's a problem before that one: you don't know in advance which files you need. That's exactly the part you wanted the agent to solve.

The way out isn't to give it more context in the prompt, but to give it the ability to get it itself. Instead of a single action that produces the solution, you give it a set of actions—read a file, search for a pattern, write, run a command—and let the model pick which one to use each turn. The change is exactly this:

WRITE-TEST-FIX (post 2)            ReAct with tools (this post)

single action:                     many actions; the model picks:
"write the whole function"           read · search · write · run

context goes in the prompt         the agent assembles its context:
(spec + tests, all at once)          reads and searches only what it needs

The loop underneath is the same as the first post: state, action, observation, stop condition. The only thing that changes is what the "action" is. Before it was writing code; now it's picking and calling a tool. But that seemingly small change is what separates a function generator from an agent that operates on a repo.

Keep reading

That is the first half. The full walkthrough — with the rest of the implementation, the trade-offs and the things that only show up in production — is on my blog:

Read the full post on ramonchancay.me →

Originally published at www.ramonchancay.me/blog/react-loop-agent-tools.

Who writes the agent's tests: three ways to define what's correct

Ramón Chancay 👨🏻‍💻 — Sat, 05 Sep 2026 01:03:05 +0000

When you build an agent, the question that really matters isn't how to describe the solution—it's who decides whether a solution is correct. In the write-test-fix loop that decision was made by tests you wrote by hand: you defined, up front, what counted as done. But writing the tests by hand is only one of the ways to give an agent that criterion, and it's the one that scales worst. This is the conceptual leap of the series: an agent's real input isn't the description of the task, but the definition of how a correct solution is recognized—what I called the evaluator in the minimal-loop post. This post walks through the three levels of that input, from tests by hand to the repo's suite, and the principle that ties them together.

TL;DR

An agent's real input isn't describing the solution, but defining how a correct one is recognized. That piece—the tests, a check, another model—is what the series calls the evaluator, and an agent doesn't optimize for doing the right thing: it optimizes for passing it.

There are three levels depending on who writes the evaluator: you by hand (spec + tests), the agent from examples you give it, or the repo's existing suite. The higher the level, the less work per task and the less control over the criterion.

The higher the level, the higher what you have to review climbs. At level 2 you review the tests the agent wrote itself, not just its code: an agent that defines its own criterion can set itself a lax one and go green without being right.

The evaluator is the agent's real input

In the previous post I called the piece that decides whether the model's output is correct the evaluator. In the write-test-fix loop that evaluator was the tests: a process that returns green or red. It's easy to read it as an implementation detail—"you need some tests to stop the loop"—and move on. But the evaluator isn't a detail: it's what you're really giving the agent.

Look at it from the loop. The model proposes an output, the evaluator judges it, and depending on that judgment the cycle stops or tries again. The model supplies the judgment about what to do; the evaluator supplies the judgment about whether it came out right. Remove the evaluator and you don't have an agent: you have a model that writes once and nobody checks whether it got it right.

That piece is exactly what a chat doesn't have, and putting them side by side is the fastest way to see why the evaluator is the input that matters:

CHAT                         AGENT

prompt                       prompt  (goal)
   │                            │
   ▼                            ▼
 model                        model ──► output
   │                            │
   ▼                            ▼
response                     evaluator ──► correct?
                                  │
                             no ──┘──► another turn
                                  │
                            yes ──► deliver

In a chat, the model's output is the final answer. In an agent, that same output passes through the evaluator first, and it's that verdict—not how many times you call the model—that turns the cycle into an agent. Everything the series built up to here—the loop, the sandbox that turns the output into a reliable observation—exists to run an evaluator you supplied.

From there comes the consequence worth keeping in mind from the start, because it explains a lot of agent behavior that looks strange until you have it in your head:

An agent doesn't optimize for doing the right thing. It optimizes for passing the evaluator you gave it. If the evaluator is lax, "correct" comes to mean "whatever the evaluator lets through."

And from there the principle that orders the post: your job when building an agent isn't to describe the solution, but to define how a correct solution is recognized. That definition is the evaluator, and it's the input that really matters. It sounds abstract until you ground it in a concrete question: who writes that evaluator? In the minimal-loop post you wrote it yourself, by hand, as three tests. But that's not the only option, and depending on who writes it, how much work it costs you to define the criterion—and how much control you have over it—changes. That's what separates the three levels.

Keep reading

That is the first half. The full walkthrough — with the rest of the implementation, the trade-offs and the things that only show up in production — is on my blog:

Read the full post on ramonchancay.me →

Originally published at www.ramonchancay.me/blog/who-writes-the-tests-agent-evaluator.

The sandbox in an agent loop: from the model's text to a reliable observation

Ramón Chancay 👨🏻‍💻 — Sat, 05 Sep 2026 01:03:01 +0000

The sandbox is the part of the write-test-fix loop that takes the code the model wrote and runs it. But calling it "run" sells it short: its real job is to turn the model's text output into a reliable observation, one the loop can base its next turn on. In the previous post that was a three-line runTests function—write the code to a file, run the tests, capture the output—and it worked because the example was a toy. The moment the code comes from a real model, four phases slip in between those three lines that the naive runner skips over, and each one has its own way of breaking.

TL;DR

The sandbox is the loop's run step, but its job is to turn the model's text into a reliable observation. It goes through four phases: entry, preparation, execution, and isolation.

Extract the code from the prose, check that it parses, run it with a timeout and in a separate process with a clean environment. Watch out: the timeout only stops code that takes too long—not code that consumes too much—and a subprocess is not a security boundary.

What moves the needle most is classification: having the sandbox return a status (not a boolean) and keep stdout and stderr separate, so each failure feeds back to the model as a distinct message.

The four phases of the sandbox

For the three-line runTests from the previous post to work, four assumptions have to hold: that the model's reply is code, that the code parses, that it terminates, and that it's safe to run. In the toy example all four hold because you wrote the case to behave well. In production none of them holds on its own, and each broken assumption is a phase of the sandbox: a filter the model's output has to pass before moving on.

Here's the map for the rest of the post. It helps to have the whole thing before diving into each piece:

Phase	What it receives	What can break	The defense
Entry	The model's text	It comes wrapped in prose or a fence	Extract the code
Preparation	The extracted code	It doesn't parse	Syntax check
Execution	The valid code	It doesn't terminate, or runs away	Timeout (with its limits)
Isolation	The running process	It touches your environment	Separate process, clean environment

Seen this way, the sandbox isn't a single block but a pipeline of four filters, and the model's output crosses them one after another:

model's text
      │
 [ Entry ]        extract the code        → prose / fence?
      │
 [ Preparation ]  does it parse?          → syntax_error
      │
 [ Execution ]    run with a timeout      → timeout · test_failed · pass
      │
 [ Isolation ]    separate process, clean env
      │
      ▼
  classified observation that feeds back to the loop

The idea that ties the four phases together, and the one I come back to at the end: in each phase, a distinct outcome has to be able to feed back to the model as a distinct observation. That's what separates a runner that merely executes from one that classifies.

Keep reading

That is the first half. The full walkthrough — with the rest of the implementation, the trade-offs and the things that only show up in production — is on my blog:

Read the full post on ramonchancay.me →

Originally published at www.ramonchancay.me/blog/sandbox-running-llm-generated-code.

The simplest loop that works: a write-test-fix agent, step by step

Ramón Chancay 👨🏻‍💻 — Sat, 05 Sep 2026 01:02:17 +0000

In the previous post we saw what an agent loop is: the pattern in the abstract. Now we're going to build it, in the smallest agent that actually does something useful. It's the write-test-fix loop: you write a specification and some tests by hand, the model writes the code, the program runs the tests, and the error feeds back to the model until everything passes. By the end of the post you have the whole cycle in code that runs.

TL;DR

The write-test-fix loop is the minimal agent: spec and tests written by hand, the model generates the code, it runs, and the test failure feeds back as context for the next attempt.

The stopping condition comes from the tests: green and it stops, red and it retries, up to an iteration cap. The model doesn't need to "decide" it's done.

Feedback is everything. A test that only says "failed" fixes nothing; one that says "expected [9,5,25] and got [null,0,0]" gives the model exactly what it needs to fix it.

From theory to code: the simplest possible agent

The previous post left the loop in pseudocode: a while with three components—state, action, and stopping condition—around a model that decides the next action over and over. Useful for understanding the pattern, but abstract. Here we'll instantiate it in the smallest case that's still a real agent, and at the end you have code that runs.

The mapping is direct. Each piece of the loop has a concrete equivalent in this example:

Loop component	In the write-test-fix loop
State	The spec, the tests, and the history of attempts and errors
Action	The model writes (or fixes) the code
Observation	The test runner's output
Stopping condition	Tests green, or the iteration cap is reached

If you understood that table, you understood the post. The rest is watching it work.

Of those four pieces, the one most worth keeping in your head is the state, because it's the one that moves. It doesn't reset on each turn: it starts with the spec and the tests, and grows with every attempt by the model and every error from the runner. Here's how that accumulation looks across iterations:

Iteration 0          Iteration 1               Iteration 2
                     (+ what 1 produced)       (+ what 2 produced)

spec                 spec                      spec
tests                tests                     tests
                     code (attempt 1)          code (attempt 1)
                     runner errors             runner errors
                                               code (attempt 2)
                                               runner errors

That accumulation is exactly what lets the model fix instead of repeating the same error: on iteration 2 it sees what it wrote before and why it failed. In this example the state grows by appending every previous attempt, nothing more. Later in the series we'll see how it's managed when the context no longer fits and you have to summarize or drop.

Keep reading

That is the first half. The full walkthrough — with the rest of the implementation, the trade-offs and the things that only show up in production — is on my blog:

Read the full post on ramonchancay.me →

Originally published at www.ramonchancay.me/blog/write-test-fix-agent-loop.

What an agent loop is (and isn't): state, action, stop

Ramón Chancay 👨🏻‍💻 — Sat, 05 Sep 2026 01:02:13 +0000

An agent loop is the cycle in which a language model decides an action, that action runs, and the result feeds back in as context for the next decision, until a stopping condition is met. That repetition—decide, act, observe, repeat—is what separates an agent from a chat. A chat answers each message once; an agent runs that cycle on its own, many times, until the task is done. This is the first post in a series on how AI agents work under the hood, and it starts with the piece that gives the whole thing its name: the loop.

TL;DR

An agent loop has three pieces: a state (what the agent knows so far), an action (what it does each turn), and a stopping condition (when it decides it's done).

The difference from a chat isn't the model, it's who runs the cycle: in a chat you run it (read, run, ask again); in an agent the program does.

"Build me an app" is a prompt for a chat and a goal for an agent. Same sentence, two different architectures.

What an agent loop is

A language model, on its own, does one thing: it takes text and produces text. One input, one inference, one output. It doesn't run anything, doesn't remember what came before, doesn't check whether it got it right. Each call is independent and ends the moment the model stops writing.

An agent wraps that inference in a cycle. The model's output is no longer the final answer but a decision: "run this command," "read this file," "search for this." The program executes that decision, captures the result, and hands it back to the model as new context. Then the model decides again, with more information than before. That turn repeats until the task is done.

The new piece isn't the model, it's the cycle around it. The same model that would answer once in a chat gets called ten or fifty times inside a loop, accumulating context at each step. That's why an agent can solve multi-step tasks a chat can only describe: not because the model is more capable, but because its output feeds back instead of ending.

The three components: state, action, and stopping condition

Every agent loop, however simple or complex, comes down to three pieces. If you understand these three, you understand the whole pattern.

State. It's everything the agent knows at this point: the goal, the history of actions it has already taken, and the results it got. The state grows on every turn, because each action and its result get added to the context. The first entry in the state is the goal; the last is what it just observed.

Action. It's what the agent does on each iteration. Almost always it's calling a tool: running a command, reading or writing a file, doing a search, calling an API. One detail that matters and that the series will keep repeating: the model doesn't execute anything. The model chooses the action; the program runs it. That separation is what makes the loop safe and debuggable.

Stopping condition. It's the rule that cuts the cycle off. It can be the model emitting a "done" action, a verifiable goal being met (the tests pass, the file exists), or an iteration cap as a safety net. Without a clear stopping condition, an agent either doesn't finish or doesn't know it already has.

The fastest way to pin down these roles is to see who handles each one:

Component	What it is	Who handles it
State	The goal plus the history of actions and results	The program accumulates it
Action	The decision of what to do this turn	The model chooses it, the program runs it
Stopping condition	The rule that decides when to cut off	The program checks it

The split in that last column is the center of the pattern: the model provides the judgment (what to do now), and the program provides the execution and the control (doing it and deciding when to stop). Confusing those two roles is the cause of most agents that misbehave.

Keep reading

That is the first half. The full walkthrough — with the rest of the implementation, the trade-offs and the things that only show up in production — is on my blog:

Read the full post on ramonchancay.me →

Originally published at www.ramonchancay.me/blog/what-is-an-agent-loop.