DEV Community: Lisandro Reinoso

Questions for a chatbot

Lisandro Reinoso — Thu, 01 Oct 2026 15:25:26 +0000

Today I have a file open with an empty table. At the top are the numbers from two weeks ago: out of 68 answers, one passed. At the bottom, the hypotheses I wrote so I wouldn't cheat myself when measuring again. The table in the middle, the one that would say whether the chatbot got better, has been empty for 16 days.

What I ask it

The chatbot answers questions about pasture growth rates with a chart and a text that explains it. Its user is a rural worker who doesn't code. To find out whether it answers well, I built a golden set: 68 questions, none of them from real users. Twenty came from a first round of examples and 48 from a second.

A golden set is a fixed list of questions with what you expect from each answer, which you rerun after every change to see whether things got better. If whoever grades only says "pass" or "fail", you know how much fails. My auditor, a separate agent, returns something besides the verdict: faltantes (what each answer was missing), with a type from a closed vocabulary (data, domain knowledge, tool, prompt) and a key so they can be summed across questions. That way you know what to fix.

1 of 68

In the September 14 baseline, 1 passed; 31 passed with reservations, 35 were rejected and 1 errored out.

That number wasn't the useful part. The useful part was 205 gaps, which grouped into 102 keys and mapped to 22 tickets: 16 new and 6 that were already in the backlog. Today 11 are done, 5 cancelled and 6 open. The three most frequent gaps: not knowing the dataset's reference date (24), having no rainfall column (19) and not being able to filter by field and paddock at the same time (16).

Rainfall didn't get fixed. There is no weather data, and I decided the chatbot should say so instead of inventing a proxy. A gap that gets resolved by saying "I don't know".

The pending measurement

Before measuring again I wrote hypotheses with a threshold: rejected answers, from 35 down to 20 or fewer; the reference date, from 24 down to 5 or fewer. And a counter-hypothesis: the change that lets the model call no tool at all could make questions that were chartable worse.

The second run generated the answers, but the audit stage, about 28 dollars, was cut off: the API credit ran out. It has been blocked since September 15.

I know what I fixed. I don't know if it got better.

When you evaluate something you built, does it hand you a score or a list of what's missing? And do you write down beforehand what you expect to change?

A reader saw what my fix didn't do

Lisandro Reinoso — Wed, 30 Sep 2026 21:17:05 +0000

The previous note ended with a question, and today someone answered. They didn't answer the question: they corrected the fix I had presented as the end of the story. Before replying I opened the scripts to see if they were right. They were.

What the reader said

A reminder: two agents wrote "I'm going to wait" and stopped, the harness cut them off after ten minutes, and the fix was a single line.

For the reader, that sentence was the useful signal, not the bug: it describes an intention, not a durable wait. They proposed that every deferred task leave a record the machine owns, with an id, an explicit deadline, and the next action, and that someone reconcile it separately. And that "success" should mean that mechanism exists, not that the agent promised to use it.

A handle, without the jargon: when one process hands work to another and moves on, the only thing that makes that wait durable is a record the machine keeps, with an identifier for the task, an explicit deadline, and what to do when it expires. Someone can check against it later to see what happened. A sentence from the agent is not that.

The weak version

My fix exports CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS as 0: no ceiling, wait forever. It removes the deadline instead of making it explicit. It makes the promise true by making it infinite, leaving nothing to reconcile.

The only reconciliation I have lives at the ticket level and arrives after the fact: run-ticket.sh reads the final JSON; if is_error is true, the ticket ends up blocked; if there's no verdict, it leaves the record untouched. The two turns from that early morning, which together cost USD 2.79, closed with success. Nothing marked them as failures.

What changed

Three days earlier a first comment had arrived, on a different note, and I decided not to capture it. This one I did: the same day it became a ticket in the template every new project is born from, with what the reader named (an explicit deadline per run, a block with the reason "deadline", closing with the handle and not with a sentence) and a pending decision they also saw: what to do when the deadline expires with the work half done.

That same early morning the ceiling had cut off a third project. Today the line is in one of thirteen scripts, and in this project the ticket that adds it has been pending for seven days. The weak fix didn't even propagate; the strong one is still a ticket.

I published that the problem was a line that doesn't travel, and a reader showed me the line wasn't the solution either. When an agent of yours delegates and says "I'll wait," what does the machine own of that wait, and who reconciles it if it expires?

"The agent said it would wait, and didn't"

Lisandro Reinoso — Thu, 17 Sep 2026 11:35:33 +0000

This morning, before launching the forks that refined this note's tickets, I exported an environment variable by hand from outside the script that starts the loop. I set it that way every time I run this project. Five nights ago that variable didn't exist anywhere, and not knowing that cost USD 2.79 and zero finished tickets.

I'm going to wait

In the early hours of September 12th, in the chatbot project, the loop ran in parallel for the first time: two tickets at once, each with its own coordinator. Each coordinator delegated the work to a coder, which started a background task while continuing its own turn. Both wrote almost the same thing before closing: "I'm going to wait for the completion notification," with success status. No error in sight; the process reported that it was waiting.

It wasn't waiting: it had finished, and it wasn't coming back. Ten minutes later — six hundred seconds — the process that actually was waiting cut off both coders with the same message: Background tasks still running after 600s; terminating. Set CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS=0 to wait indefinitely. The record went untouched. Cost: USD 2.79. Two coordinators, zero tickets done.

Ten minutes

Running something in headless mode — no interactive terminal, claude -p answers once — lets you delegate to an agent "in the background": it starts another one and keeps going with its own work while the second one works apart, and finds out when the second one signals it's done. It's fine for a short question. But the harness ships with a default wait ceiling — ten minutes —, meant for a one-off query, not for a loop that writes, reviews, and fixes code. If it takes longer, the harness doesn't warn that it got tired: it kills the process underneath, and the one above is no longer around to find out.

Eighteen minutes after that run started, the fix arrived: one line of code, plus four lines of comments, in the loop script. The next run finished ten tickets that same morning.

One line, one repo

What's notable isn't the fix itself — the error message suggests it. The same ceiling had already cut off a run in this project one day earlier, with USD 0.81 lost, and it got logged as pending instead of fixed on the spot. It's still pending six days later. Of eleven loop scripts across the ecosystem of projects I work with — nine active ones plus the two templates every new project is born from — only one has the line: the one that suffered it that morning. A project that starts today is born without the fix.

A one-line fix has the same propagation problem as a ten-page convention: if nothing pushes it between repos, someone has to remember to push it. How many one-line fixes do you have sitting in a single place, with the same bug waiting in the rest?

"Nobody designed the frontmatter"

Lisandro Reinoso — Wed, 16 Sep 2026 15:44:06 +0000

I open any file in these repos in VS Code, and before the first heading the Outline panel shows a block of names: title, numero, date, sesion_fuente, fecha_creacion, description, tags, destacado, slug. This post carries those nine at the top, and will gain a tenth once it's published. Yesterday I asked myself who decided each one. Short answer: nobody, all at once.

That block is the frontmatter: a few key: value lines between two --- at the top of a Markdown file. It isn't part of the text; it's what the file says about itself — title, date, tags, where it came from — in a format a script can read without understanding the prose. Nobody writes it from scratch: every file type has a template with the block already built and the values empty, and whoever creates the file copies it and fills it in; some fields get filled in later by a script — the hook that closes a session, the step that publishes a note. It's there so the site can order the notes, so a hook can fill in dates, so a radar can cross sessions against notes already published. And it's there, above all, for as long as someone reads it.

Four cradles

Tracing the origin took an afternoon: seven families of fields, born in four places. The sessions one — alias, fecha_inicio, estado — was born on August 16 in another project, when two sessions in a row settled how to name and pick a work session. Six days later, in the template I use to spin up projects, a two-field classification was born, domain and type, with a closed vocabulary, to tag rules, agents, tickets, and decisions alike. The third cradle is this project: between September 3 and 15, across five rounds, the fields that identify a note. The fourth, on the 11th: the English version, in a separate file with its own block.

What traveled and what didn't

Each family travels by a different mechanism, and none of them is all-or-nothing: the initial scaffold copies everything; the sync only updates agents, rules, and scripts; a hook fills in a handful of fields when each session closes. The result: of the tickets in a project prior to the classification, 1 out of 84 has it; in a later one, 178 out of 178; here, where it was born, 48 out of 215. And no script reads those two fields at all: the rule that defines them calls them, literally, "metadata".

The second gap is stranger. The same hook, when registering a new session, fills in its identifier with a regular expression; since that field starts out empty, it also swallows the next line — the start-date one. 254 out of 260 sessions in one project were born without it, and nobody noticed until another script had to infer it from the file name.

What holds up

The four fields that identify a note — slug, numero, fecha_creacion, tags — have what the classification never did: a reader that fails loudly. The build breaks if one is missing. And a piece's publication status isn't stored: it's recalculated every time it's queried. Less convenient, impossible to drift out of sync. A field doesn't survive because of the intent it was created with, but because someone, or something, still reads it.

Nobody designed the block above end to end: it has the shape of its own history. Of the ones heading your files, how many do you know who wrote, and how many do you know someone still reads?

"Two readers, two documents"

Lisandro Reinoso — Tue, 15 Sep 2026 14:04:05 +0000

Today, in the Matching repo, there are two files called README.md with the same modification time: 2026-08-18 04:43:43 for the one at the root, 04:43:24 for the one inside the code folder. Almost a month has gone by without anyone touching them. Next to them, in the same repository, CLAUDE.md has 886 lines and its last edit is from September 10 — twenty-three days after those two READMEs. Three documents in the same repo: two frozen, one alive. The difference isn't chance. It's who reads them.

Why two and not one

The only documented session from that August 18th closed at 04:48 without recording what time it had started. The goal, a single line: "Create the project's README.txt." The original request was actually double —a README "for the general-purpose assistant" and another inside the code folder— though I didn't even get that folder's name right, and along the way it became clear both should be .md, not .txt. The second one replaced Vite's default boilerplate outright, without anything else living alongside it.

What ended up in each one isn't symmetric, it's complementary. The one at the root is a guide for whoever picks up work in the repo, person or assistant: folder structure, agents, skills, backlog rules, the session system, the ticket system. The other is the guide to the product itself: stack (React 19, Vite 8, Supabase), setup, scripts, environment variables, deploy. That same exploration left, along the way, two findings logged to the backlog as out of scope —the outdated stack reference and a duplicated costs folder—, both resolved today.

What aged and what didn't

The code README still says the production deploy "is in blocked status" and that the app shouldn't be assumed to be deployed yet. The real deploy happened on August 22 —it's in the changelog, with a Vercel URL and all— so that file has spent 24 days claiming something that stopped being true the same month it was written. The agent table in the root README didn't hold up either: it lists six, the agents folder has twelve files today, and one of those original six doesn't even live there anymore — the costs one moved up to the user level, outside the repo.

What didn't age is as telling as what did. The root README's two pointers into CLAUDE.md sections still resolve today: the agent routing section and the session-naming convention section are exactly where the README says they are. The folder structure it describes still exists as described. And the claim that there's no .claude/skills/ or .claude/commands/ in the repo —skills are global to the user— is still true. What rotted were the states: a ticket, a file count. What held up were the pointers and the conventions.

That same August 18th, the project's changelog records six product changes: the public landing page, the guided matching assistant, the candidates view, the intro description, the topic search, and the rename of "Materia" to Área across the whole stack. No session documents any of them. The day I wrote how the repo gets documented is, also, the day the least got documented.

What stayed

The convention survived even though the files didn't. The template I use for every new project still ships both as boilerplate: 391 lines for the CLAUDE.md one, 137 for the README.md one. Of the twelve projects I have today, all twelve have a README.md and ten have a CLAUDE.md. The split by audience won; keeping it up to date in every project didn't. Which of your two documents is up to date today, and what does that tell you about who's actually reading it?

"Day three: the loop leaves a trace"

Lisandro Reinoso — Mon, 14 Sep 2026 17:38:06 +0000

Day one I built the agent loop. Day two, the session record. Day three I ran ./run-ticket.sh US-04 and nothing happened: no error, no output, no ticket executed. The loop had been running for two days, and that early morning, for the first time, there was no way to see it.

A script that said nothing

The cause was small and a little embarrassing. run-ticket.sh looks up the ticket by file name, but the sprint's twenty user stories lived in 01-Stories/20260815Sprint/ as NN_slug.md, without the US- prefix. grep -i "US-04" found nothing, and with set -euo pipefail active the find | grep | head pipeline failed and killed the script before it could print the error message I had written myself. The bug wasn't just US-04's: it affected all twenty stories in the sprint equally.

The fix added a second content-matching attempt — look for the # US-04 — ... heading inside the file when the name isn't enough — and neutralized pipefail with || true on both pipelines, so the existing error block would actually run when it should. Verification turned up a second bug inside the first one: without anchoring the search to the heading, US-01 matched TECH-011_smtp_resend.md because that ticket mentioned "Discovered during US-01" in its body. It was fixed by anchoring the grep to ^# US-01. I closed that session at 03:06 without having run US-04 end to end — fixing the matching wasn't the same as invoking the whole loop, and that was left for later.

One file per run

The next session, that same early morning, tackled something different: the loop ran, but its output only lived in the terminal. If it got cut off, if I wanted to compare two runs of the same ticket, or simply know how much a run had cost, there was nowhere to look. The decision was to save each run in its own timestamped file — history/<TICKET_ID>_<timestamp>.json — instead of a fixed file per ticket that each new run would overwrite. That night I was already generating some of those files by hand, without a timestamp, running real tickets in parallel; I was asked whether I'd rather align the convention to that, and I chose to keep the timestamp anyway, so as not to lose the history of repeated runs of the same ticket.

The implementation was a tee at the end of the pipeline: claude ... | tee "$HISTORY_FILE", verified with bash -n and an isolated simulation of the pipe, without invoking claude for real so as not to spend money on a test. I closed that session at 03:46. The first run under the new convention, US-06_20260817-034659.json, started that very minute.

What the record saw that same night

From then on, every run left a trace, and that same night the new record captured something that used to get lost. Between that early morning and the following night, 29 valid runs were left in history/ — close to 500 turns in total, around USD 55 — 20 finished DONE and 5 BLOCKED.

US-11 was the one that showed the most. It ran four times: two early BLOCKEDs without even reaching the coder, a third one cut off at 49 turns by "Credit balance is too low" — there, the coder had touched database.types.ts, a file generated by Supabase, and along the way had broken a flow from another already-finished user story; the coordinator caught it and returned the ticket before the credit ran out — and a fourth, with a narrow fix this time, that finished DONE. The ticket was still marked blocked in the record anyway: a false positive from run-ticket.sh itself, which looked for the word "BLOCKED" anywhere in the text instead of in the status line, and which was fixed by hand after documenting it. The system that finally let me see the loop was born with its own reading bug.

What stayed

Today, a month later, the three decisions from that early morning are still intact in the template I copy into every new project: pipefail active, the heading fallback, tee writing to history/. That's more than 500 files spread across eight projects. Every run of the loop — including the one that wrote this post — leaves a record in some history/, and that's where, among other things, the cost of each one comes from.

Day three didn't add a new feature to the product. It added the ability to look back and know what had happened. What do you keep from every agent run, and who reads it afterward?

"Keeping a record: working across many sessions"

Lisandro Reinoso — Sat, 12 Sep 2026 21:29:04 +0000

Day one closed in the small hours and day two started half an hour later, with a short session that didn't touch the product: it defined how sessions would be named, where they would be recorded, and which script would close them on its own. The problem it solved wasn't one of order but of continuity. Every conversation with Claude Code ends, and I wanted to be able to work across many of them —open one, close it, come back the next day or from another project— without losing what had been decided in the previous one. With some distance, it is one of the sessions that weigh the most of all the ones I have: without it this post wouldn't exist, because the material every post in this log comes from is, precisely, the file of each session.

A short session and a convention

What got written that night fits in four lines. A naming format, YYYYMMDD-VNN_nombre: the date, a counter that restarts at V01 every day, and a goal in two or three words. An index, docs/sessions.md, a Markdown table. A closing hook —a script Claude Code runs on its own every time the model finishes responding— that updates the end time and the status. And a decision on how to find out the session's real identifier, the UUID you need to resume it: take the most recent file in Claude Code's internal sessions directory.

And one more line, almost at the bottom: "session V01 of 08/15 registered retroactively". Day one, which I covered in the previous post, exists as a session because day two wrote it down. The record started by recording backwards.

What the same day already outgrew

That same afternoon, the third session of the day redid half of the design (the second one was opened and closed empty, with no goal: the first record of a session that never was). A single table wasn't enough. Each session got its own file, with a metadata header and fixed sections —goal, decisions, work done, next steps— and the index was reduced to a lightweight list. list_sessions.py appeared, which reads those headers and shows the sessions ordered by activity, along with the instruction that every conversation should start by showing that list and asking: do we resume one or open another? That is where the real goal got stated: that the work could be spread across many sessions without each one starting from scratch.

The detail I like most is in the metadata. The fourth session of the day has the same identifier as the third: the "most recent file" heuristic failed the very day it was written, with two sessions opened almost at once. And the times: the first session of the day has a start time; the next three, only the date; the ones from the day after literally say "(time not available)". The system was born without knowing how to read a clock.

What stayed and what moved

The naming format is today identical, letter by letter, in the template I replicate in every project. There are 426 session files spread across ten projects; 165 of them were opened by the system on its own, using the text of the first message as the name, one for every ticket an agent executed from the terminal without me in the conversation —day two didn't foresee that. The fixed sections of the third session are the same ones in the file of the session I'm writing this in.

What moved is the responsibility. The closing hook stopped living in my user folder and now travels with each project, and it responds to two events: one refreshes the end time on every turn, the other marks the session as closed exactly once, when the conversation ends. The start —list and ask— stopped being a paragraph in the instructions file: a start hook injects real data (tickets, sessions open for more than 48 hours, alerts), a rule propagated from the template defines the protocol, and the global instructions guarantee the minimum if the other two layers are missing. Three layers for what day two solved with a paragraph, because the paragraph diverged in three out of eight projects as soon as it was copied.

And there is something day two couldn't know: that the date in the name would end up being the date of this post. Posts in this log carry the date of their source session, not the day they are written. This one carries August 16. That night's convention decided, without meaning to, the order in which everything else is read.

What it didn't decide is whether this is the right way. Four hundred files, three layers of hooks and rules, a counter that restarts every day: it works, and it is what let me write this. But I built it in one night and kept patching it ever since, without ever comparing it with anything else. If you work with agents across many sessions —how do you keep the record? Is this the best way to manage sessions, or just the one I happened to land on?

Anatomy of a skill

Lisandro Reinoso — Fri, 11 Sep 2026 22:50:34 +0000

This started with a surprise, not with a plan. I needed a survey of the state of the art on a topic I was considering working on, so I ran deep-research, a skill that ships with Claude Code, and it came back with a report I didn't expect from a single run —27 sources, 123 claims extracted, 25 verified, and of those 18 confirmed and 7 refuted— with an executive summary, caveats, and open questions. What struck me wasn't the volume (I already covered the 109 agents in the previous post) but that all of it came out of a single skill. I wanted to see how it was built.

It wasn't easy to find: it doesn't live in .claude/skills/ or in ~/.claude. It's compiled into the Claude Code binary as a bundled workflow, and a comment in the code tells its origin: "Ported from bughunter architecture". It's 349 lines of JavaScript. Reading them was the research.

How a prompt is written

Inside there are three prompts, and all three follow the same shape: a role in the title, the context (the original question plus the specific input), a task as a numbered checklist, an explicit decision criterion, and the output format. The verifier is the clearest one:

const VERIFY_PROMPT = (claim, v) =>
  "## Adversarial Claim Verifier (voter " + (v + 1) + "/3)\n\n" +
  "Be SKEPTICAL. Try to REFUTE this claim. ≥2/3 refutations kill it.\n\n" +
  "## Claim under review\n\"" + claim.claim + "\"\n" +
  "**Supporting quote:** \"" + claim.quote + "\"\n\n" +
  "## Checklist\n" +
  "1. Is the claim actually supported by the quote, or is it an overreach?\n" +
  "2. WebSearch for contradicting evidence.\n" +
  "3. Is the source quality sufficient for the claim's strength?\n" +
  "4. Is the claim outdated?\n" +
  "5. Is this a marketing claim / cherry-picked benchmark / forum speculation?\n\n" +
  "**refuted=false** ONLY if: well-supported, current, and source quality matches claim strength.\n" +
  "Default to refuted=true if uncertain.\n\nStructured output only. Evidence MUST be specific."

It's exactly the framework I use when I review one of my own prompts —role, context, task, format, constraints— but with two details I don't usually include: the decision criterion is written as a rule (refuted=false only if…) and ties resolve by default toward the conservative side ("Default to refuted=true if uncertain"). A prompt that doesn't say what to do when in doubt leaves that decision to the model, and that's where the tidy-but-unfounded answers show up.

How a skill is built

A skill is more than a long prompt. What I saw in the file falls into six pieces:

1. Trigger metadata. The meta declares the name, the description, and the five phases, but the key is whenToUse: it's what the model reads to decide whether to invoke the skill, and it includes a prior instruction —if the question is underspecified, ask two or three clarifying questions before starting.

2. Tuning constants at the top. No magic numbers buried in the code:

const VOTES_PER_CLAIM = 3
const REFUTATIONS_REQUIRED = 2
const MAX_FETCH = 15
const MAX_VERIFY_CLAIMS = 25

3. One schema per agent. Each of the five agent types returns JSON validated against a schema. That's what makes the pipeline composable: one agent's output is the next one's typed input.

4. Prompts as functions. SEARCH_PROMPT(angle), FETCH_PROMPT(source, angle), VERIFY_PROMPT(claim, v): they take the input and return the text. The prompt isn't copied, it's instantiated.

5. Explicit orchestration. Search and fetch go through pipeline(): each angle moves on to fetching its sources as soon as it's done, without waiting for the others. Before verification there's a barrier —and the comment says so: "Barrier here is intentional"— because the pool of claims has to be complete before it can be ranked. Then, nested parallel(): 25 claims × 3 votes.

6. Defensive design. Every early exit (no claims, everything refuted, failed synthesis) returns a useful result with stats instead of throwing. A null vote counts as an abstention, not as a free pass:

const survives = valid.length >= REFUTATIONS_REQUIRED && refuted < REFUTATIONS_REQUIRED

And the final result carries agentCalls: the skill computes its own cost (1 + angles + sources + claims × 3 + 1). That's where the 109 came from.

Trimming the pattern

The proof that I understood the pattern was reusing it. A couple of later runs on that same topic got cut off by the session limit before verification, and instead of repeating them in full I wrote reverify-linea-c: same meta, same constants, same verdict schema, and the same three-vote verifier, but with no Scope, Search, or Fetch —those phases were replaced by an array of 22 already-extracted claims. The whole skeleton inherited, one array of my own. All 22 came back confirmed.

What I take away: a good skill isn't a long prompt, it's a short, well-formed prompt, instantiated many times by an orchestration that knows where to wait and where not to, and that measures what it spends. And it reads in an afternoon.