DEV Community: Ted

I Audited My AI's Briefing File. It Had Turned Into a Diary.

Ted — Mon, 05 Oct 2026 10:13:25 +0000

Coding agents like Claude Code read a plain-text briefing file before they do anything else. Mine is called CLAUDE.md, and it sits in the home directory of the small server I run my automation on. It tells the agent who I am, what's installed, which services run where, what the cron jobs do, and the handful of rules I don't want broken. It loads into every session, whatever I'm working on.

I'd been adding to it for six months. After two model releases in ten days, I ran Claude Code's new prompt audit on it (/checkup prompt-audit). The audit reads your instruction files and flags text written for older models: ALL-CAPS warnings, "think step by step", rigid scripts. It writes a report and a patch and changes nothing until you say so.

I expected it to find shouting. It found almost none. What it found instead was more useful, and it changed how I think about the file.

What I expected

The audit's main checklist is about prompting habits that used to help and now hurt. Older models needed forceful instructions to follow anything reliably, so people wrote IMPORTANT: and NEVER everywhere. Current models follow instructions closely and literally, so the same shouting makes them over-apply a rule and behave rigidly in situations it was never meant for.

My file had exactly one of those: a rule in capitals about using a site-specific command-line tool only for the site it was built for. The fix was to say it at normal volume and keep the reason: "the other sites have no command center." Two other spots used capitals for emphasis, but they stated facts, not rules, so the audit left them alone.

That was the whole outdated-prompting section of the report. One finding.

What it actually found

A quarter of the file was history.

The cron section was meant to list my scheduled jobs. Over the months its header had grown into an 880-word paragraph about what I'd removed: which jobs were cut on which date, why, where the backups went, the traffic numbers that made me drop three sites, which repository was left untouched. Every entry had been accurate when I wrote it. None of it was something the agent needed to do anything.

Several facts had quietly gone wrong.

The file listed the Gemini CLI at a specific path. It wasn't installed there, or anywhere.
It gave a version for one of my agent gateways that was two months out of date.
It listed four API keys in my environment file. There were more than a dozen.
It listed the tools wired into my voice assistant and was missing one.

None of these would cause a crash. They'd cause a confident wrong answer: the agent telling me "use the Gemini CLI at this path" and spending a turn finding out it isn't there.

One rule was a security habit dressed as documentation.

Under my blog's repository it said: "Git push requires the token in the remote URL", followed by the exact command to paste a GitHub token into the repo's config. That was true once, because nothing else was set up to handle the login. But it's an instruction, so every time the agent pushed, it would write a live credential into a plain-text file. I'd already cleaned that exact pattern out of two other repos a few weeks earlier, and the instruction file was still teaching it.

The flip

I'd been treating CLAUDE.md as documentation, a notebook about my server that the agent happens to read.

It isn't documentation. It's a prompt that runs every session, and every sentence in it is a sentence the model acts on.

Once you see it that way, the findings stop looking like housekeeping:

History in a prompt isn't a record, it's noise the model has to work around. "Removed on 2026-08-25, backup at this path, 82 days of rank history pruned" reads as context to me. To a model that treats everything in its instructions as relevant, it's a pile of half-relevant facts competing with the ones that matter. The one useful rule hidden in that paragraph ("these jobs were removed on purpose, so check before adding them back") was the part hardest to find.
A stale fact in a prompt isn't out-of-date documentation, it's a wrong instruction. A wiki page with an old version number just sits there. The same line in a prompt gets acted on.
A how-to in a prompt isn't a note, it's a standing order. "Put the token in the URL" stopped being a description of how I once got a push to work. It became the default.

The audit's own guide puts it in one line: the job is to find "specific instructions that no longer fit", not to make prompts shorter. My file didn't have an old-model problem. It had the problem every long-lived config file gets: it kept accumulating and nothing ever removed anything.

What I changed

The patch had nine hunks, each tied to one finding, so I could take or skip them one at a time. I took all of them.

Moved the history out. The 880-word paragraph went verbatim into a separate cron-history.md that isn't loaded automatically. The cron header is now one line with the job count and a pointer: "read it before re-adding a removed job." The schedule itself stayed.
Fixed the facts. Dropped the Gemini line, corrected the version, listed the real key names, and added the missing voice-agent tool.
Kept rules, dropped stories. Several sections had a current rule with an incident wrapped around it, like "Seen on 2026-09-22: a session started at 14:48 picked up the new package but still ran the old code…" The rule (restart the process after a config change, because clearing the session isn't enough) stayed, with its reason. The timestamped story went.
Fixed the token rule properly, not just the wording. I set git to use the GitHub CLI's existing login for pushes (gh auth setup-git), confirmed a dry-run push worked, and only then replaced the line with "plain git push, never put a token in the remote URL." Then I checked every local repo for a token in its remote URL. There were none.

Two smaller things went too. I removed account-balance figures from a list of fallback models, since balances drift and nothing on the machine could confirm them. I also moved four skills for a framework I'd stopped using out of the global skills folder: their descriptions were loading into every project.

The file went from about 3,700 words to 2,760. That wasn't the goal, just a side effect of removing what didn't belong.

What the audit left alone, and why that matters

The audit has an explicit list of things it must not delete, and it followed it. It kept:

the rules that carry a reason, like "don't re-optimize this page's title, the low click rate comes from the search results layout, not the title";
two identical copies of a scaffold file in two project folders, because they agree;
my one-line note that I prefer short, direct answers, because that's context about me, not an outdated trick.

That restraint is the part I'd have got wrong doing this by hand. My instinct with a bloated file is to cut it hard, and a hard cut removes exactly the lines that only I could have written: the reasons. The model can work out how to be thorough by itself. It can't work out why I don't trust a particular page's metrics.

If you keep one of these files

Some things I'm doing differently now:

Write the rule, not the story. If a line describes something that happened, ask what the agent should do differently because of it, and write that sentence instead.
Give history its own file. A changelog is valuable. It just shouldn't load into every session. Point to it from the briefing file, and the agent can read it when it's relevant.
Check facts against the machine, not your memory. Paths, versions and key names drift silently. The audit caught four because it checked them, not because they looked wrong.
Read every how-to as a standing order. If you wouldn't want the agent to do it every time without asking, it doesn't belong in the file as an instruction.

The file is shorter now, but that isn't the improvement. The improvement is that everything left in it is something I actually want the agent to act on.

What Is Jev? I Tested TypeSafe's Decision Model on My Comments and Email

Ted — Sun, 04 Oct 2026 03:13:48 +0000

In September 2026 a company called TypeSafe released Jev, and it isn't a chatbot. It never writes a sentence. You give it some text and a question with fixed answer options, and it returns one of your options plus how sure it is: "bot, 97%." Nothing to parse, nothing to read.

I spent a day testing it on two real jobs: sorting the comments people leave on my blog posts, and sorting my email. This post covers what Jev is, how it works, what it's good and bad at, exactly how far I tested it, and the one thing that decided whether it worked, which turned out not to be the model.

What Jev is

TypeSafe calls Jev a "System One" model, borrowing the psychology term for fast, intuitive judgment as opposed to slow, deliberate reasoning. Chat models like Claude or GPT reason and write. Jev does one job: decide.

That makes it a component, not an assistant. You don't talk to it. You put it inside a script at the point where plain code can't make a call. Is this comment spam? Does this email need me? Which team should this support ticket go to? Code can't answer those reliably with rules, and sending each one to a full chat model is slow and expensive for a decision you need thousands of times.

What it can't do matters just as much:

It doesn't write. No summaries, no replies, no explanations of its answer.
It doesn't know facts. It has no web access and isn't a knowledge base. It can't tell you whether a law changed or a price is right. It only judges the text you hand it.
It only picks from your options. If none of your options fits, it still picks one. It just tells you it isn't sure.

How it works

One HTTP request. You send three things:

state: the text to judge. It can be a plain string or named fields.
questions: one or more typed questions, all answered in the same call.
model: jev-latest.

There are three question types:

Type	What you get back	Example
choice	One option from your list, a probability for every option, and a confidence score	"Is this comment genuine, promo, or bot?"
noul (yes/no)	The probability of "yes"	"Does this comment ask the author a question?"
score	A position on a scale you define (2–10 levels)	"How frustrated is this customer: calm, frustrated, very angry?"

Here's a real request from my comment filter, shortened:

{
  "model": "jev-latest",
  "state": {
    "post_title": "…",
    "comment": "We need to write a short casual YouTube comment as a regular developer…"
  },
  "questions": {
    "kind": {
      "type": "choice",
      "instructions": "Classify `comment`, left on the developer blog post titled `post_title`.",
      "criteria": {
        "genuine": "A reader discussing the post's ideas, with no link to or plug for their own project",
        "promo": "Mentions, links to, or plugs the commenter's own project, tool, repo, product or site",
        "bot": "Reads like instructions written to an AI or leaked prompt text, or generic filler unrelated to the post"
      }
    },
    "asks_question": {
      "type": "noul",
      "instructions": "Does `comment` ask the author a direct question or explicitly invite a reply?"
    }
  }
}

And the answer comes back as data:

{
  "model": "jev-1.13.0",
  "answers": {
    "kind": { "choice": "bot", "probabilities": { "…": "one per option" }, "confidence": 0.97 },
    "asks_question": { "noul": 0.79 }
  }
}

The criteria text is where all the work is. Hold that thought.

Price and speed. You pay $0.042 per million tokens Jev reads; output is free because it writes nothing. In my tests a call took about 0.4 seconds (412 ms to 964 ms, almost all close to 430 ms).

How far I tested it

Test	Messages	Questions per message	Cost	Result after fixing the wording
Blog comments	15 real comments on my dev.to posts	2 (choice + yes/no)	$0.0004	15 of 15 sensible
Email	My last 50 Gmail Primary emails	2 (choice + yes/no)	$0.002	All payout, security and action emails flagged; codes, receipts and newsletters sorted to routine
Security question	12 account-related emails	1 (yes/no)	under $0.001	Real alerts 75–96%, everything else 47% or lower

Everything, including the failed first attempts, cost under one cent. I didn't test the score type, very long documents, or high volume. Treat this as one person's real-data trial, not a benchmark.

Where it failed first

On the first run, Jev called a bot genuine. The bot had pasted its own instructions instead of a comment: "We need to write a short casual YouTube comment as a regular developer… Must start with specific reaction… Use lowercase start." Any human spots it instantly. Jev also called two self-promotional comments genuine: one plugged the writer's GitHub tool, the other said "I work on [product]."

The confidence numbers told the story. On the twelve clear-cut reader comments Jev was 89–100% sure. On the bot it was 43% sure, and on the GitHub plug 34%. It wasn't confidently wrong. It was saying my options didn't fit.

My first descriptions were judgments: genuine meant "engaging with the post's ideas in their own words", promo meant "mainly promotes the commenter's own product". Deciding whether something "mainly" promotes is exactly the call I wanted made for me. So I rewrote each option as something you can see in the text, the wording in the request above.

Comment	First wording	Concrete wording
Bot that leaked its prompt	genuine, 43%	bot, 97%
Reader plugging a GitHub tool	genuine, 34%	promo, 99%
"I work on [product]…"	genuine, 81%	promo, 100%
Two good comments ending in a link to the author's own site	genuine	genuine, 50–60%
The other ten	genuine	genuine, 93–100%

The unsure row is my favourite. Those two comments make real points and then link the writer's own site. Is that promotion? I'm not sure either, and Jev dropped to a coin flip on exactly those two.

The model wasn't judging the comments. It was matching them against my descriptions, and my descriptions were the thing that was wrong.

Email repeated the lesson. Sign-in codes I'd requested myself came back as "needs me" (71–98% sure), because my description mentioned "a verification he must complete". Webmaster "improve your site" tips came back the same way. Explicit exclusions fixed both: "NOT one-time sign-in or verification codes, and NOT tips or suggestions from tools."

The security question was the scary one. Google's "A new sign-in on Linux" alert scored 9%:

Question wording	Google sign-in alert	Google app-password alert	Sign-in codes
"Is this a security alert about Ted's own account (a new sign-in, password or recovery change…)?"	9%	20%	9–22%
Same question plus "true" and "false" descriptions	12%	13%	3–30%
"Does `body` tell Ted that something changed in his account's access: a new sign-in, a new device, a new password or app password…?"	75%	84%	3–20%

The first two ask whether the email belongs to a category, "security alert". The third asks whether the text reports a specific event you can point at.

Building something safe with it

Even well-worded, Jev will sometimes be wrong. How you use its answer matters more than its accuracy:

Tag, don't filter. My comment watcher still sends me every comment, now tagged 🔗 promo or ❓ asks-you-a-question. The only thing it drops is a bot verdict at 85%+ confidence, and those go to a log I can check.
Low confidence means "show me". Anything under 70% gets an "unsure" tag instead of being acted on.
Failure means the old behaviour. If the API errors, times out or my credit runs out, messages go through untagged. Jev can improve an alert. It can't make one disappear.
Back up the critical cases with plain rules. Beside Jev's security score, a simple check flags any subject containing "security alert", "new sign-in", "password" or "passkey".
Keep the questions and thresholds in one block. They decide behaviour, so they're what to review. TypeSafe's own docs give the same advice.

Both scripts are live now. My email watcher pings me for security, money and anything that needs me, and puts codes, receipts and newsletters into one evening summary.

Should you try it?

If you have a script that has to make many small yes/no or which-bucket decisions about text, and you're currently either writing fragile keyword rules or paying a chat model to answer one word at a time, Jev is worth an afternoon. Testing costs fractions of a cent.

Just budget the afternoon for the questions, not the integration. Describe what's visible in the text, not your conclusion about it. Test on your own messages. And read the confidence column as carefully as the answers: a low number usually means your options don't fit the case.

My Site Became the Source, So My Mistake Became the Answer

Ted — Thu, 01 Oct 2026 18:08:28 +0000

I run a small travel directory. One of its listings is a boutique hotel whose owners hold a licence for an on-site cannabis lounge. For months the listing said the hotel has a licensed lounge, and recommended it to anyone visiting in winter who wanted somewhere warm to consume.

During a routine review of the pages that list it, that claim started to look shaky, so I began checking it. Partway through, I asked an AI search engine the question a traveller would ask: is this hotel cannabis-friendly?

The answer was mine

Perplexity said yes, with a tidy breakdown. Consumption only in the licensed lounge. Bring your own. A capacity of about 25 people, described as upscale and private. It closed with practical advice: if you plan to consume, do it in the lounge and bring your own.

Nearly every line was cited to my site. The 25-person capacity was cited to my site alone, because no other page on the internet says it.

That sent me back to the listing to see where the number came from. It was in the listing's description, along with "a curated social hour" and "knowledgeable staff". None of it had a source. It reads like the lounge's plans, written up at some point as if they were the lounge.

Licensed is not open

The checking I'd already started was the other half of the answer. The only way to get the lounge's current state was to line sources up by date:

Date	Source	What it says
2022	City approval, widely reported	Preliminary approval for the licence
Early 2023	Local news	Not open yet; the ventilation design still being redrafted
Spring 2025	Same outlet	The owner about four years into trying to open it
Early 2026	A local lounge directory	Still being built
Today	The owners' own website	Describes the lounge as pending

Every source agreed the licence is real. None of them said the lounge was open. The owners' own FAQ, written in the future tense, adds the part that matters most to a guest: cannabis is only allowed in that lounge, nowhere else on the property.

My listing had collapsed two different facts into one. Licensed is a state that was true in 2022 and is still true. Open is an event that hasn't happened. The owners' marketing blurs them too, which is how the listing got written that way in the first place, and how an AI reading my page repeated it without hesitation. To a hurried researcher and to a language model, "holds a licence for a lounge" and "has a lounge" are the same sentence.

Being cited is not being checked

This is the part that changed how I think about the site.

When a page is one source among many, its mistakes get diluted. When it becomes the source, the one an answer engine leans on for a niche question, its mistakes get passed on whole, with a citation attached that makes them look checked. The answer engine didn't verify the lounge. It verified that my page said so. The citation proved where the claim came from, not that it was true.

And the people asking aren't careless. Most travellers will call a hotel before booking around a feature like this. But the one who doesn't arrives expecting a lounge that isn't there, and the page they trusted, and the answer engine that quoted it, are both wrong in the same way.

The first fix was too honest

My first correction, drafted with the AI agent I use for site work, swung hard the other way. It said the lounge hadn't opened, and quoted the owners' rule that cannabis is allowed nowhere else on the property, on every page that mentioned the hotel.

Every word of it was accurate, and it read like a warning. When I read it as a guest would, the problem was obvious: we'd taken a listing people were booking and turned it into a reason not to. The hotel is real, well reviewed, run by owners building around exactly this kind of guest, and holding a licence very few hotels have. None of that had stopped being true.

So the second version keeps everything that's true and attractive, and changes only the one claim that wasn't:

A well-reviewed historic hotel whose owners hold a licence for an on-site cannabis lounge, opening soon. Until it opens, ask the hotel where cannabis is allowed before you book.

The verification label changed too, from "verified" to a new status that says exactly that: licensed, opening soon. The old system only had verified, host-declared and unconfirmed, and none of them could describe a property where we know exactly what the situation is and it simply hasn't finished happening.

What I took from it

Write states with their dates. "Holds a licence" and "opening soon" stay true until something changes. "Has a lounge" was false from the day it was written.

Check a claim with a timeline, not a source. Any single source here would have told me the lounge was real. Only the sequence told me it had been "about to open" for four years.

Know what you're the only source for. The detail that only my page contained, the 25-person capacity, was the one with no source at all. If you're the only page saying something, nobody downstream will catch it for you.

Fix the claim, not the listing. The correction that tells people what's true and still gives them a reason to go is better for the reader and the business than the one that just warns them off.

The answer engine will catch up when it next reads the page. Until then, it's still quoting the old sentence, with my name on it.

The Reviewer Checked the Wrong Pages

Ted — Mon, 28 Sep 2026 11:46:09 +0000

I run a small team of AI agents that answers research questions. One agent searches the web and takes notes, one writes the answer, and a third, the reviewer, checks the write-up before I see it. The reviewer is the expensive part, so I tested three versions of it on the same draft. That draft had a real mistake in it: it credited a hardware claim to a source that didn't say it.

The expensive reviewer, Claude Sonnet at high effort, caught the mistake for about three cents. The cheap one, Claude Haiku, missed it for about one cent. I kept Haiku and gave it a new ability: before deciding, open two or three of the pages the draft cites and check they really say what the draft claims. Then I wrote that this closed the gap.

I never re-ran the test.

The comment

A reader summed up the idea better than I had: the cheap reviewer didn't need to be smarter, it needed something to be wrong against. A reviewer that only reads the draft can judge whether it sounds right. One that holds the source can actually fail it.

That's true, and it made me notice I had described a mechanism, not a result. So I pulled the original flawed draft out of the run log and gave it to the new reviewer. Same draft, same notes, same mistake. The only change was the ability to open pages.

What happened

I ran it twice, plus once with the page-opening tool switched off to recreate the original setup.

Reviewer	Verdict	Caught the mistake?
Haiku, no tools (the original setup)	revise, for other reasons	no
Haiku that can open pages, run 1	approve, no notes	no
Haiku that can open pages, run 2	revise, for other reasons	no

The version with the new ability did worse. On one run it approved the flawed draft outright.

The reason was in the draft itself. It cited sources as "[5]", with no link. The citations only started carrying URLs the next day, so the reviewer had nothing it could actually open. Instead of saying so, it went looking. It opened an Apple specification page, a hardware review, a code repository: six pages that seemed relevant to the topic and weren't the source of anything in the draft. It read them, found nothing that contradicted the draft, and signed off.

That is worse than the reviewer with no tools. The no-tools reviewer never claimed to have checked a source. This one did the motions of a source check against pages that couldn't fail the claim, and came back more confident, not less.

The fix

Two rules, and the second one had to live in code rather than in the instructions:

Only the cited sources. Before reviewing, the system collects every URL that actually appears in the draft and the research notes, and the reviewer is told to open only those. A request to open any other page is refused before anything is fetched.
Say "not checkable" when you can't check. If the draft cites nothing with a link, the reviewer is told not to search or guess. Its notes always begin with "Citations not checkable: no source URLs", and that line is added by the code if the model leaves it out. A sign-off can no longer look like a source check that didn't happen.

On the same link-less draft, the fixed reviewer opened nothing, both times, and both reviews started with that line.

What the source check actually adds

Fixing the failure still left the original question: does opening the sources help the cheap reviewer catch mistakes? So I took a recent draft that does cite real links and planted a false claim in it: "Node 22 also dropped support for Windows 10 [1]." The cited source is the official release announcement, which says nothing of the kind, and the research notes don't mention Windows at all.

Every version caught it, including the one with no tools. The no-tools reviewer noticed that the research notes never mention Windows. The version that opened the sources went further: it named the official announcement and said the claim wasn't in it, which is stronger evidence and cost about two and a half times as much.

So in my tests, opening the sources didn't catch anything that comparing the draft against the research notes missed. It produced better proof, not more catches. The mistake that started all this, a hardware claim sitting right next to figures the notes really did contain, was missed by the cheap reviewer in every setup I tried. Only the expensive one has caught it.

What I'd tell myself three days ago

A tool is not a check. A reviewer that can open pages will open some, and if the right ones aren't available it will find pages that look close enough, and treat not finding a problem as finding none. The ability to look only helps when there is something specific to look at, and when "I couldn't look" is an answer the system is allowed to give.

The smaller lesson is the one the reader's comment forced on me: the only way to know what a fix does is to re-run the case that made you want it. I had the flawed draft saved the whole time. Writing "this closed the gap" took less effort than checking whether it had.

It Looked Finished on Day One

Ted — Sun, 27 Sep 2026 02:36:30 +0000

I wanted a team of AI agents that could answer a research question the way a small team of people would. One plans the work, one searches the web and reads the sources, one writes it up, and one checks the write-up before I see it. I also wanted to watch them do it: which agent is working, what it's reading, what it handed to whom, and what it cost.

The app is called FORGE, and it runs on a small server in my house. This post covers how it got from an idea to something that works end to end, which took three days. Most of that time wasn't building features. It was finding the places where the app looked like it worked and didn't.

Day one: the part that looks finished

The first version was a dashboard for a system that didn't exist yet.

It had a workflow canvas where each agent is a card and each arrow is a handoff. It had a live event stream, a timeline, a replay scrubber, per-agent cost and token counters, an agent builder and a workflow designer. Behind it sat a simulator: fake agents on a fake clock, producing realistic-looking events so every screen had something to show.

One design decision from that first day held up for everything that came after. A run is a log of events, not a record that gets updated. "Researcher started", "tool call finished", "handed the notes to the writer": every screen is computed from that log. Live view and replay are the same code; replay just stops reading the log partway through. When real agents replaced the simulator later, the screens didn't change at all, because real agents emit the same events.

It looked gorgeous, and none of it was real.

Making the agents real

Real agents meant real model calls, and I wanted them cheap. I started with three agents: a researcher, a writer and a reviewer.

Models: most agents run GLM-5.3 Flash through OpenRouter, at about $0.15 per million input tokens. The reviewer runs on Claude.
Search: DuckDuckGo through the ddgs Python package. It's free and needs no key.
Reading pages: a small script fetches a URL and pulls out the main text (trafilatura for web pages, pypdf for PDFs).
A server: a small Node.js server runs the agents, stores every event, and streams them to the browser as they happen.

The first live runs failed in ways the simulator never could.

Every call timed out, instantly. The server has no IPv6. When a hostname has several addresses, Node.js tries them one after another and gives each attempt only 250 milliseconds by default, which isn't enough on this network. The fix was two lines: prefer IPv4, and give each attempt three seconds.

Some replies came back empty. The model had spent its whole output budget reasoning and had nothing left for the answer. It now gets one retry with double the room.

The researcher never stopped researching. It kept searching and reading until it hit the step limit, then had no notes. Now every tool result tells it how many rounds it has left, and the last round forces it to write up what it has.

Some websites never answered. One slow host could eat a minute. Page fetches now give up after 20 seconds, and a host that times out once is skipped for the rest of the run.

The honesty pass

With real agents working, I went through the app page by page and asked one question: is this real or simulated?

A lot of it was simulated and looked real. The model-provider page showed made-up usage. The tools page listed tools with success rates and response times for tools that had never run. The run history was full of sample runs.

Then I asked the team a real question and got a confident, well-formatted answer to a completely different question. The run had used the simulator, which produces plausible text and ignores the question entirely. It was the default for most workflows, and nothing on screen said so.

The rule I took from it: never ship a placeholder that looks real. Label it or remove it. Simulated data is now clearly marked everywhere. The simulator survives only as a "Dry run" button in the workflow designer, for checking the wiring for free. You can't start a simulated run from the New Run screen at all.

One store, not one per browser

I use FORGE from a desktop and a laptop, and they showed different data. Everything lived in each browser's local storage, so each device had its own private copy.

The server now owns a single SQLite database. Every change goes to the server, and every open tab gets pushed updates over a live connection. Each browser uploaded its old local data once, and the server merged it.

This surfaced a subtler bug. Run IDs had been picked by each browser, and the laptop picked an ID the desktop had already used, overwriting a run. Now only the server issues run IDs.

Spreading the cost

I added a planner in front of the researcher. It writes a short plan with sub-questions and search queries, costs about a hundredth of a cent, and makes the research noticeably more focused.

Then I checked the bill. One OpenRouter key was paying for everything, and it had $1.74 left. So Claude models now go straight to Anthropic's API, and everything else stays on OpenRouter. If Anthropic's key fails on billing, the call falls back to OpenRouter instead of failing the run.

The reviewer was the expensive part. I tested three reviewers on the same flawed draft, one with a real error: it credited a performance claim to the wrong source.

Claude Sonnet, high effort: caught the error. About $0.03 per review.
Sonnet, lower effort: approved the flawed draft. About $0.014.
Claude Haiku: sent the draft back for other reasons but missed that error. $0.009.

I chose Haiku. The next section is what closed that gap.

Making it trustworthy

One run hit seven failed web searches in a row. An outside review of the logs blamed DuckDuckGo throttling and suggested switching to a paid search API.

The logs said otherwise. The search library rotates between several engines, and all four timed out in the same three minutes. The same queries worked in two to four seconds when I re-ran them. It was a short network outage on my side, and a paid API would have timed out too. So instead of switching providers:

Search fails fast and retries: a 12-second limit, one retry, and after three failures in a row a 30-second wait for the network to come back.
Not enough sources is flagged in code, not just the prompt: if the researchers read fewer than two pages, the answer carries a warning banner saying it's unverified. The writer is told too, but the flag doesn't depend on it listening.
The reviewer checks the sources itself: it opens two or three of the pages the draft cites and checks they say what the draft claims. That's what the cheap reviewer was missing: a source it could check against. Pages the researcher already fetched are reused, so the check costs no extra time.

Reaching it from my phone

I already run a personal assistant agent called Hermes that talks to me on Telegram. FORGE and Hermes now connect both ways.

FORGE → Telegram: every finished run sends the answer, cost, time and a link to the run page. Failed runs send an ❌ with the error.
Telegram → FORGE: a /forge command in Hermes hands a question to the team. Hermes can only wait three minutes for a command, and a run takes longer. So the command starts the run and returns straight away, and the answer arrives on Telegram when the team finishes.

There are now three levels: quick (no reviewer, about half a cent, answers marked "not fact-checked"), verified (the default, two to four cents) and a larger six-agent team for deep questions. I pick the level myself. I considered a classifier that decides whether a question needs checking. It would have been cheap, but it would sometimes skip checking on exactly the question that mattered.

Letting the agents talk

Handoffs only go one way: the writer gets the notes, and that's it. If the notes leave a gap, the writer can only guess or leave it out.

So agents can now ask a teammate a question mid-task. The writer can ask the researcher, and the researcher answers from its notes or does one quick lookup. In one run the writer spotted two researchers giving different numbers for the same fact and asked one to settle it. The researcher went back to the source and answered with the bill that set the figure.

I also added a parallel team: a lead splits the question into three angles, three researchers work at once and post findings to a shared board, then the writer and reviewer finish.

Its first run failed at the 20-minute limit while every agent was busy. One researcher's final write-up had taken five minutes on its own, because the model spent it reasoning. My stall detector watched for a stream that goes quiet, and a reasoning model streaming its thoughts never goes quiet.

I measured the same three-sentence prompt. With no reasoning effort set, the model took 93 seconds and 3,200 reasoning tokens. With effort: low, it took 9 seconds. Leaving the setting out didn't mean a sensible default; it meant no limit. After setting it for every call, the same parallel question finished in 5 minutes 10 seconds for four cents.

Watching it from the outside

The last piece was monitoring. I have a dashboard, Operator Pulse, that shows every scheduled job on the server and alerts me when one fails. FORGE now appears on it as a service: whether the server is up, recent runs, success rate, and how much OpenRouter credit is left.

The first time the new FORGE node rendered, it was amber: $0.47 of credit left. That key also pays for Hermes, so both would have stopped working without warning.

FORGE can also run scheduled questions now. Each one is a small script that the dashboard tracks like any other job.

Where it ended up

A finished quick run: three agents, 1 minute 40 seconds, about a tenth of a cent. Every event on the right can be replayed.

Mode	What runs	Typical cost	Typical time
Quick	planner → researcher → writer	$0.005	1–2 min
Verified	+ a reviewer that checks cited pages	$0.02–0.04	1–4 min
Parallel	lead → 3 researchers at once → writer → reviewer	$0.04	~5 min

Every run is fully replayable. Every answer lists the pages it came from, and says so plainly when it couldn't verify enough of them.

What building it taught me

The dashboard was done on day one. It had every screen, every counter and every animation, and it was all simulated. Everything after that was the same job repeated: find a place where the app looks like it works and make it actually work.

Some of those places were loud: timeouts, empty replies, a run that died at twenty minutes. The ones that mattered most were quiet: a confident answer to the wrong question, two devices each sure their data was the real one, a timeout that never fired because the model never stopped talking, a credit balance nobody was watching.

A demo shows that something can work. Making it trustworthy meant finding every place it only looked like it worked.

The Parameter Was Never Read

Ted — Mon, 21 Sep 2026 09:22:32 +0000

Every morning at 8:25 a script reads this blog's numbers out of Google Search Console — the tool Google gives site owners to see which searches showed their pages — and sends me a short digest on Telegram. Clicks, impressions, average position. Then two lists: top queries, top pages.

I built it on day one, before the site had a single impression to report. It has run every morning since.

Yesterday I was looking at something else entirely when the page list caught my eye.

📄 Top pages
  / — 2 impr | 0 clicks | pos 8.0
  /contact — 2 impr | 0 clicks | pos 2.0
  /operations — 1 impr | 0 clicks | pos 7.0
  /posts/a-second-check-is-a-second-source-of-truth — 1 impr | 0 clicks | pos 9.0
  /posts/automating-seo-monitoring-with-ai-agents — 2 impr | 0 clicks | pos 11.5

The site had 290 impressions that week. The top page, according to the section labelled top pages, had two of them.

The sort I had been asking for

The script asks Search Console for pages the obvious way. Give me pages, ten of them, ordered by impressions, biggest first:

query(sc, {
  startDate: start,
  endDate: end,
  dimensions: ['page'],
  rowLimit: 10,
  orderBy: [{ fieldName: 'impressions', sortOrder: 'DESCENDING' }]
})

That request has always returned 200. It has always returned rows. The rows have always been real — those pages exist, those impression counts are correct.

I went to check what orderBy actually does, and found that it does not exist.

The request body the Search Analytics API accepts has ten fields: aggregationType, dataState, dimensionFilterGroups, dimensions, endDate, rowLimit, searchType, startDate, startRow, type. That is the whole list. Searching the client library's type definitions for the string orderBy returns zero matches — not in the request, not anywhere in the Search Console surface.

So for as long as this script has run, it has been attaching a field the API has no concept of. The API did not reject it. It did not warn. It read the fields it knew about, ignored the one it didn't, and answered.

What order the rows were in

Here is the same window, asked for queries, printed exactly as the API returned them:

 1   1 astro-site-mu-eight.vercel.app
 2   2 chat with self hosted ai agent
 3   1 got html content but no text found (with 200 reply code)
 4   1 how do i run an ai agent locally?
 5   9 http://localhost:8766
 6   8 http://localhost:8766.
 7   6 http://localhost:8766/
 8   6 localhost:8766
 9   7 self hosted ai agent setup
10   2 this deployment is temporarily paused

Alphabetical.

The documented default is to sort by clicks, descending. Every row in that window has zero clicks. With nothing to rank by, what came back was ordered by key — and my report, which took the first five of whatever arrived, printed them under the heading Top queries.

Nothing about that output looks wrong. Five plausible search terms with plausible numbers beside them. The only thing separating it from a correct answer is that the word "top" was doing no work.

The part that actually cost something

Mis-ordering ten rows is cosmetic. I had ten queries that week and all ten came back, just shuffled.

Pages were different. There were twenty-six pages with impressions that week, and rowLimit: 10 asked for ten. When the order is alphabetical, "the first ten" means the ten whose URLs sort earliest — /, /contact, /operations, and whatever posts happen to start with a and b.

The busiest page on the site is /posts/headless-oauth-loopback-callback. That week it took 155 impressions, more than half the site's entire total of 290. Its URL begins with h.

It was not ranked low in the report. It was never in the response. The truncation happened at the API, ten rows in, long before my code got to choose what to display. I had been reading a daily summary of my site's search performance that omitted the majority of it.

Sorting the rows myself, after asking for all of them:

📄 Top pages
  /posts/headless-oauth-loopback-callback — 155 impr | pos 7.1
  /posts/the-audit-became-a-build-step — 51 impr | pos 5.9
  /posts/how-to-set-up-local-ai-agent — 17 impr | pos 37.4
  /posts/vercel-silent-build-failure — 10 impr | pos 12.9
  /posts/claude-built-my-astro-blog — 9 impr | pos 13.1

Same API, same week, same window. A different site.

It failed because the numbers were small

This is the part I keep turning over.

The fallback to alphabetical only happened because every row had zero clicks. On a site with traffic, the default sort — clicks, descending — would have put the busiest pages at the top on its own. The report would have been right. Not because the parameter worked, but because the API's default happened to agree with what I wanted.

I would have gone on passing a field that does nothing, reading correct output, for as long as the site had clicks.

The report was wrong in exactly the conditions it exists to report on. It is a monitor for a site that is not yet getting traffic, and the absence of traffic is what broke it. Had it ever started working, it would have started working silently, and I would have had no more reason to check it then than I did on any of the mornings it was lying.

An ignored input is a default you didn't choose

I have a rule on this site that defaults are policy: an unset value is not an absence, because something downstream always picks one. This is the same rule arriving from a direction I hadn't considered.

I did set the value. I set it explicitly, in the request, with the right intent. It just went to a system that had no field to put it in, and a system with no field to put it in cannot tell you that you set nothing — it has already forgotten you tried.

A strict API rejects unknown fields. A permissive one accepts everything and gives every request the same confident shape. That permissiveness is usually described as robustness. What it actually does is move the entire class of "you asked for something that doesn't exist" out of the error channel and into the results, where it arrives looking like an answer.

I've written before about an API that reported 44 updates and performed 24. That one accepted a real field and undid the write inside the same operation. This one is a step further back: the field was never real, so there was nothing to undo. Both return 200. Both hand you well-formed output. The difference only exists in documentation you have to go and read.

Two things I do now

Check the parameter against the schema, not against the response. A 200 tells you the request was accepted. It cannot tell you the request was understood. The only place that distinction lives is the field list, and reading it takes a minute — considerably less than the months I spent not reading it.

Sort where you control the sorting. If ranking matters, pull a generous row limit and order the rows in your own code. Ordering done remotely is ordering you are trusting on someone else's terms, and — as here — possibly not happening at all. Ordering done locally is something you can look at.

The report was never broken. Every morning it fetched real data and formatted it correctly and delivered it on time. It just answered a question I hadn't asked, under a heading that said I had.

The Top Row Had 150 Clicks. It Wasn't a Listing.

Ted — Fri, 18 Sep 2026 18:24:51 +0000

I track every click that leaves one of my sites. When someone taps a booking button, a small function records which listing it was for, which page it happened on and a few other details, and a dashboard adds them up. Its most useful panel is a table called Top Performing Entities: the listings people click most.

I opened it to see which listings were earning their place. The top row had 150 clicks in 30 days, well ahead of second place. It carried the name of a real listing.

It wasn't one. It was 29 different listings, stacked on top of each other.

Five kinds of value in one column

Each click row stores an entity_id — the ID of the listing that was clicked. The dashboard groups by it. That's the whole design: count the rows per ID, sort, show the top ten.

So I counted what the column actually held. Of the 806 clicks in the last 30 days:

database UUID        441
"unknown"            157
slug or label        108
list position        100

Five different ideas of what an ID is — the slugs and the hand-written labels share a line — in one column. Each had a reasonable origin.

unknown came from a button. On a phone, every detail page shows a sticky bar at the bottom of the screen with the main call to action. The component that draws it takes the listing's ID as an optional prop, and falls back like this:

entityId={entityId || "unknown"}

None of the four page types that used the bar passed an ID. Most of the site's visitors are on phones, so this was one of the most-clicked buttons on the site, and every click on it was logged as unknown. All 157 that month came from it; 463 since June.

List positions came from the main listings page. The database gives every listing a UUID, but a shared TypeScript type declared id: number. To satisfy it, the code that turned database rows into cards made one up:

id: 9000 + index,

That's a position in a list, not a property of a listing. Sort the list differently, filter it, add a listing, and the same number points somewhere else. It did: 9037 was logged for two different listings, and so was 9010.

Slugs and labels came from blog posts, whose links were written by hand — sometimes with the database ID, sometimes the URL slug, and sometimes a short label that matched nothing in the database at all.

None of it was an error. Every insert succeeded, because the column was plain text and accepted all of it.

It hadn't always been. It started as a UUID column, and months ago I loosened it to text so an events page could log IDs that weren't UUIDs. That was the one thing that could have refused unknown and 9037 — though it would have refused them by dropping the clicks, which is its own kind of wrong.

What the dashboard did with it

Grouping by that column did two opposite things at once.

It split listings that should have been one row. The most-clicked real listing was spread across five different IDs: its UUID from its own page, a list position, a slug from one blog post, a label from another, and unknown. Another was spread across seven. In the same 30 days, 76 listing names came back under 102 distinct IDs.

And it merged listings that should have been separate. Every unknown click from every listing went into one bucket. The dashboard labelled each row with the name on the first click it happened to read, so 150 clicks from 29 listings wore one listing's name — and sat at the top of the table, because the sum of everyone's missing IDs will outscore any single listing.

The panel next to it was worse, because it drew a conclusion. A "missed revenue" report flagged listings that got clicks but no affiliate clicks, as candidates for a booking link. It flagged six. Two of them had affiliate clicks all along, logged under a different ID for the same listing.

An ID column isn't an identity. It's whatever each piece of code decided to send.

The column promises that one value means one thing, and nothing enforces that promise at the moment of writing. Each caller — a button, a list, a blog post someone wrote by hand — picks what to send, and the table stores it. When they disagree, the database has no way to know. The disagreement only surfaces where someone finally groups by the column and trusts the result.

Resolve, don't trust

There were two fixes, and the order mattered.

The first was to stop believing the label. The dashboard now loads the ID, slug and name of every listing and resolves each click to a real row — by ID first, then by slug, then by name, with names normalised so "The Example Inn" and "Example Inn", or "& Spa" and "and Spa", land on the same listing. Only then does it group: by the resolved row, not the logged value, and under the listing's name from the database rather than whatever the click carried.

Before shipping it, I ran the same rule in SQL over every listing click ever recorded, to see what it would leave behind:

matched by ID      1,219
matched by slug      156
matched by name      811
unmatched            140   (of 2,326)

The 140 are almost all listings deleted since — there's no row to resolve them to, but their logged names are consistent, so they still group correctly. The top row went from 150 anonymous clicks to the real most-clicked listing, at 110. The missed-revenue report went from six flags to four.

The second fix was to stop sending bad IDs. The sticky bar now receives the listing's ID from every page that uses it. The listings page keeps each row's own ID instead of a list position, and the shared type now says id: string, which is what it was all along. The hand-written blog list tracks the database ID instead of its label. Old rows keep their old values; the resolver handles those.

Doing it in that order meant the dashboard was right about history the moment it shipped, not just about clicks from then on.

A few days later the same lesson turned up one layer over. A small script checks every morning that each listing still carries a paid booking link. The site counted links from three booking domains as paid; the checker, written earlier, knew only two. The first listing to use the third domain would have been counted as unpaid on the day it started paying. Two pieces of code, one idea, two definitions — and each was right about its own.

The build passed

One detail from the fix, because it nearly shipped a quieter version of the same bug.

The resolver builds its lookup with new Map(). The build succeeded. The typechecker didn't:

error TS2350: Only a void function can be called with the 'new' keyword.
error TS2558: Expected 0 type arguments, but got 2.

The dashboard file imports an icon library, and one of its icons is called Map. In that file, Map wasn't JavaScript's built-in — it was a React component, and calling new on it throws. The bundler strips types without checking them, so it compiled without complaint.

It wouldn't even have crashed. The lookup runs inside a data-fetching hook that catches errors, and I'd written the resolver to fall back to the logged names whenever the lookup wasn't available. So the dashboard would have loaded and looked normal, while quietly grouping by the very column this post is about — the fix switched off by its own safety net. globalThis.Map put it right.

I've written before that a fallback chain is an error-suppression system, and that green is not verification. This was both, in four lines.

What I'd check first

If a dashboard ranks things by an ID that several parts of your code write, count what's actually in the column before trusting the ranking:

SELECT
  CASE
    WHEN entity_id ~ '^[0-9a-f]{8}-'          THEN 'uuid'
    WHEN entity_id ~ '^[0-9]+$'               THEN 'numeric'
    WHEN entity_id IN ('unknown', 'null', '') THEN 'placeholder'
    ELSE 'other'
  END AS kind,
  count(*)
FROM clicks
GROUP BY 1
ORDER BY 2 DESC;

If more than one kind comes back, the top of that table is a claim, not a count.

And look hard at the fallbacks. || "unknown" looks like defensive code. It was a default that set policy for 463 clicks, and the policy was: forget which listing this was.

The top row wasn't my best listing. It was the one place every missing ID agreed to meet.

Three AI Agents Corrected Each Other. None of Them Were Talking.

Ted — Tue, 15 Sep 2026 04:43:01 +0000

CrewAI is an open-source Python framework for running several AI agents as a team. You give each agent a role and a goal, give the team a list of tasks, and the framework passes the work between them. Before building anything real with it, I wanted to answer one question: when people say the agents "talk to each other", what is actually happening?

So I set up a small crew on the server in my house — a ThinkCentre that already runs my monitoring and automation — and gave it a practice topic.

The crew

Three agents, three tasks, run in order:

Research Analyst — gathers the facts on a topic and marks anything it isn't sure of.
Content Writer — turns the research into a 300-word brief. Allowed to ask the Analyst questions.
Editor — checks the brief against the research, verifies the biggest claim with the Analyst, and sends weak sections back to the Writer. It finishes with a "Team notes" section listing every exchange.

The Analyst and the Writer run on Qwen 3 235B through OpenRouter, a service that puts dozens of model hosts behind one API. The Editor runs on Claude Opus 4.8, direct from Anthropic. The cheap model does the volume; the expensive one makes the judgement calls.

Setup is four commands:

uv tool install crewai
CREWAI_DMN=true crewai create crew starter_crew --provider anthropic/claude-haiku-4-5
cd starter_crew && crewai install
crewai run

That installed CrewAI 1.15.21, the version everything below was tested on. CREWAI_DMN isn't in crewai --help. I found it in the CLI's source: it switches off every interactive prompt, which is what you want when a script rather than a person is driving. It also means the scaffold never writes your .env — the API keys are yours to add. The --provider flag only chooses the starting model that gets written into each agent's file; I swapped those afterwards, to Qwen for the Analyst and the Writer and Claude Opus 4.8 for the Editor. The project itself is plain JSON: one file per agent, one crew.jsonc for the tasks.

The run

It took 118 seconds, 14,609 tokens in and 3,928 out.

The Writer asked the Analyst a question before drafting. Then the Editor read the draft and stopped on a price: a capable home AI server for about $1,600. It asked the Analyst whether that held up. The answer came back that $1,600 is the graphics card alone — a complete machine runs $2,500–3,500 — and that a single 24 GB card can't run a 70-billion-parameter model at a usable speed anyway. The Editor sent the paragraph back to the Writer with the correction, took the revised version, and recorded both exchanges in its Team notes.

Three agents, one caught error, one fix. It looked exactly like a team conversation.

It wasn't a conversation

The activity log at the bottom of the run screen says what actually happened. Three entries:

✓ ask_question_to_coworker     15.3s
✓ ask_question_to_coworker     20.4s
✓ delegate_work_to_coworker     8.9s

Those are tools. When an agent is allowed to delegate, CrewAI hands it two extra functions, the same way you'd hand an agent a web search or a file reader. Calling ask_question_to_coworker with a coworker's name, a question and some context starts a fresh model call as that coworker, and whatever it returns comes back to the caller as the tool's result. The Editor never spoke to the Analyst. It called a function whose implementation happens to be another agent.

A conversation between agents isn't a conversation. It's a function call with a job title.

That changes how you build with it. The quality of the "discussion" is the quality of the arguments one model passes into a tool. The Editor only knew about the $1,600 because it was in the draft it was handed; the Analyst could only answer what the question contained. Nothing is shared between agents except what one call gives the next — and the handoffs between tasks work the same way, each task's output pasted into the next task's prompt.

It also exposes the limit of what I watched. Nobody in that exchange looked anything up. These agents had no web access, so the Analyst "confirmed" the correction from the same kind of training data that produced the original number. The new figures — the $2,500–3,500 and the claim about the 24 GB card — are plausible. Neither is verified. The Analyst had no way to say "I don't know", and one of the operating rules I keep for this blog covers exactly that: a system with no way to say "I don't know" will answer anyway. A second model agreeing is a second opinion, not a second source. Giving agents tools can change that. Giving them coworkers doesn't.

Five things that broke first

The key was a placeholder. My .env held template text, not a key, and Anthropic answered 401 invalid x-api-key. One direct request to the API would have told me that before any framework was involved.

The provider wasn't installed. The scaffold depends only on crewai[tools]. Anthropic support is an optional extra, so the first run died with Anthropic native provider not available. The fix is uv add "crewai[anthropic]".

Memory wanted a key I never gave it. The scaffold switches crew memory on, and memory's default embedder is OpenAI's. With no OpenAI key, every memory lookup errored. "memory": false until you configure a different embedder.

The newest Claude doesn't fit yet. I wanted Opus 5 as the Editor, so I read CrewAI's Anthropic code first. Its thinking setting accepts only "enabled" or "disabled"; Opus 5 thinks by default and rejects the old "enabled with a budget" form; and CrewAI drops the thinking blocks when it rebuilds a tool-call turn. Agents calling each other through tools is exactly the path that would hit it. I didn't spend money finding out — Opus 4.8 costs the same per token and only thinks when asked.

The fifth one is the one worth a section.

The limit I never set

The first full three-agent run failed halfway. The Analyst finished its task. The Writer failed on every attempt — ten in a row — with this, passed back from the model host:

Requested token count exceeds the model's maximum context length of 131072 tokens.
You requested a total of 132086 tokens: 1014 tokens from the input messages
and 131072 tokens for the completion.

The Writer's prompt was about a thousand tokens. The other 131,072 were the reply — under a reply limit I had never set.

That was the whole problem. CrewAI only sends a reply limit (max_tokens) when you configure one; I checked the code. With none sent, the reply was treated as allowed to fill the model's entire context window, my thousand tokens of input went on top, and the total was rejected for being over the window. The request didn't fail because it was big. It failed because it didn't say how big.

I reproduced it without CrewAI: one request to the same host, a 12-token prompt, no max_tokens. Same error — 12 plus 131,072 is 131,084, over by twelve. Why the Analyst survived, I can only guess: OpenRouter spreads requests across hosts, and my best explanation is that its requests landed on one that handled a missing limit differently. I didn't capture which host served them.

Here is what I can show, and what I can't. The limit being enforced is 131,072 — the error comes back from that host's backend — while OpenRouter's listing for the same host advertises 262,144 tokens of context and 235,929 of output. What I can't show from outside is which layer wrote 131,072 into the reply field, OpenRouter or the host. That it matches the host's real window, not the advertised one, points at the host. Treat that as a hypothesis.

The fix is one field. Every agent's model now carries "max_tokens": 4096 — 16,000 for the Claude editor. It's another case of a rule from the same list, defaults are policy: an unset value isn't an absence. Something downstream always chooses one, and here it chose the largest number it had.

An unset limit isn't no limit. It's the maximum.

What it costs

With only the Qwen models, a full three-agent run cost under half a cent. With Claude as Editor, the OpenRouter share stayed around $0.003. Anthropic doesn't report per-run cost to an API key, but even if every one of that run's 18,537 tokens had gone through Claude at Opus 4.8's list price — $5 per million input tokens, $25 per million output — it would come to about 17 cents.

I had a crew review this post

Before publishing, I built a second crew and gave it this draft. A Cold Reader plays a stranger arriving from Google. A Reframe Critic looks for the one sentence where the idea flips. A Claude Editor checks every claim against the text and gives a verdict. Before any of them ran, plain code handled the mechanical checks — no private site names, a valid category, the image present, the links resolving — because no model should be trusted with those.

The verdict was "fix first", and the two most useful catches were mine to own. I had stated two guesses as facts: why the Analyst's requests survived, and which layer filled in the 131,072. Both are labelled as guesses now. It also noticed that the setup command names one Claude model while the crew runs another, with nothing to reconcile them.

One reviewer was simply wrong. The Cold Reader said a stranger "cannot access" the rules page these posts link to. It's a public page. It said so with exactly the same confidence as everything else it said.

That review had the same shape as the run above. The reviewers didn't discuss the post. Each one was a function call handed the draft, and the Editor was handed their outputs. The good catches came from the Editor checking claims against the text in front of it. The false one came from an agent with no way to check anything, answering anyway.

If you set one up

Set max_tokens on every agent's model. Don't let something downstream pick it.
Turn memory off, or configure an embedder, before the first run.
Use CREWAI_DMN=true crewai run in scripts — plain output that exits when it's done. Use plain crewai run when you want to watch.
Read the activity log, not the final report. The log is the actual conversation — and it's a list of function calls.

The Error Was Real. The Page Was Fine.

Ted — Sat, 12 Sep 2026 03:10:00 +0000

The report arrived from the hosting platform's AI monitoring, and it was admirably specific.

A listings page on one of my sites, it said, always falls back to a degraded list. The main database query fails every time, because it asks for a field that doesn't exist. The page silently drops to a simpler query, so the listings load without their city, state and country — which can break the grouping, the filters and the labels visitors see. It had errored 28 times in 24 hours.

That is a good report. It names the symptom, the mechanism, the consequence and a count. I nearly went straight to the consequence and started looking for broken labels.

The first half turned out to be exactly right. The second half never happened.

The query had never worked

The page asks the database for its listings along with a nested relationship: each listing's city, that city's state, and that state's country. In Supabase that is one request, written as a nested select:

supabase
  .from('hotels')
  .select(`
    *,
    cities:city_id (
      name, slug,
      states:state_id (
        name, code,
        countries:country_id ( name, code )
      )
    )
  `)

I ran that exact request against the live database and got this back:

HTTP 400
{"code":"42703","message":"column countries_3.code does not exist"}

Neither the states table nor the countries table has a code column. The database reports only the first missing column it trips over, so the error names countries, but both references were wrong. The request was malformed as written, so it could not succeed on any load, for any visitor, ever.

Wrapped around it was a pattern you have probably written yourself:

const { data, error } = await tryWithRelationships();
if (error) {
  console.error('relationship query failed, falling back to simple query:', error);
  return await supabase.from('hotels').select('*');
}

Try the rich version first. If it fails, degrade gracefully. It reads like defensive engineering, and it is. It is also, in this case, a description of a code path that had executed on every page load since the day it shipped. The "rich version" was not the primary path with a safety net beneath it. It was a request that failed, followed by the actual implementation.

Then I checked what visitors saw

The report's consequence — listings without their location — is a reasonable inference from that error. The failing request is the one that fetches location. If it never returns, location should be missing.

Except the listings table stores its own copy. Each row carries plain city, state and country columns, and those columns were populated on all 58 rows. The fallback's select('*') returns them. And the code that formats each listing already reached for them:

const city  = row.city || row.cities?.name || 'Unknown';
const state = row.cities?.states?.name || row.state || 'Unknown';

It uses the relationship when it exists and the row's own column when it doesn't. That had been written deliberately, with a comment explaining why: only 8 of the 58 rows had a city_id set at all. So for 50 listings the relationship would have come back empty even on a day the query worked. Whoever wrote the formatter had already stopped trusting the relationship and routed around it.

So every visitor got the fallback, and the fallback carried everything the page needed. The grouping worked. The filters worked. The labels were right. The error fired on every single load, and nobody who loaded the page ever saw it.

An error rate counts what your code tried and failed. It does not count what anyone saw. When a fallback is doing its job, those two numbers stop having anything to do with each other.

The mirror image of the usual problem

I have written before about a fallback chain that did the opposite. Four layers of image lookup, where a dead link in layer three was simply covered by layer four, and forty-nine broken URLs sat behind a default photo without a single error being raised. That was silence hiding damage.

This was noise hiding the absence of damage. Same architecture, the same defensible habit of degrading instead of crashing, and the signal failed in the other direction. A fallback can make a real failure invisible. It can just as easily make a harmless failure look catastrophic, because the thing that fails is loud, and the thing that catches it is quiet.

That's what the report got wrong, and it was a good-faith mistake. It read the error, which is about a query, and inferred the impact, which is about a page. The error message was completely accurate about the query. It simply contained no information about the page, because the page's outcome was decided one step later, by code the error never passed through.

What it actually cost

Not nothing, which is worth saying plainly, since "the page was fine" is not the same as "there was no bug."

A failed request on every load. 28 in a day is 28 wasted round-trips and 28 console errors, each one burying whatever real error might arrive next to it.
A code path that misrepresented itself. Anyone reading the component would believe the page had a richer mode it used when it could. It never had one. Code that describes behaviour it doesn't have is a trap for the next person who reads it.
One field genuinely degraded. Country had no column fallback. It read row.cities?.states?.countries?.name || 'USA', so every listing was labelled with a hardcoded default.

The fix for the error was deleting two field names from the select. The request now returns 200 and resolves the relationship for the 8 rows that have one.

The country field was the part that didn't go the way I expected. With the query fixed, I checked what came back for those 8 rows, expecting real country names at last. Every one of them was null. The states table has a country_id column, and it is empty on every row. So the country relationship cannot resolve for anything, and the label still falls back to 'USA'. It happens to be correct today, because every listing is in the US. It will quietly be wrong for the first one that isn't.

That is the richer mode, working exactly as written, with nothing underneath it.

The useful version of the report

I don't think the lesson is "distrust automated monitoring." The monitor found a real defect that had been live for as long as the page existed, and I would never have looked otherwise.

The lesson is that an error with a 100% failure rate and zero visible impact is not a contradiction to be explained away. It is a finding in its own right, and a sharper one than the report made. If something fails every time and nobody notices, then nobody depends on it. That path is dead code wearing the costume of the main path.

So when an alert tells you something fails on every request, ask the second question before reaching for the consequence: what did the user get instead? If the answer is "exactly what they needed," then the error isn't telling you the page is broken. It's telling you the part you thought was the system has never actually run — and that the part you thought was the backup is the system.

The number was true. The story attached to it was a guess.

The API Said 44 Updated. Twenty Didn't Change.

Ted — Sat, 12 Sep 2026 03:08:39 +0000

I keep a copy of every post from this blog on dev.to, the developer blogging platform. Each copy points back here with a canonical link, so search engines treat this site as the original. Last month I rewrote the descriptions on all of them — the short summary that shows on share cards and in dev.to's own search.

Forty-five articles. A job for a script, not a morning of clicking.

I did it the careful way. I updated one article first, and before and after I fingerprinted everything the update could plausibly disturb: the length of the body, the tags, the canonical URL, the title, the published state. Only the description had changed. Everything else was byte-identical. Then I ran the other forty-four.

updated 44, failed 0

Every request came back 200 OK.

Out of habit rather than suspicion, I ran the checker again — the one that compares what's live against what I intended. Twenty articles still had their old description.

Not a cache

My first thought was lag. The endpoint that lists all your articles at once is exactly the kind of thing that serves a slightly stale snapshot. So I skipped it and fetched each of the twenty individually, from the endpoint that returns a single article fresh.

Old descriptions, all twenty. The writes hadn't been delayed. They hadn't happened.

So forty-four requests had gone out, all forty-four had been answered with success, and twenty of them had changed nothing. Nothing in the responses separated the twenty from the twenty-four. Same status, same shape, same body echoing back.

The twenty had something in common

Laid side by side, the difference was in the article bodies. Every one of the twenty began like this:

---
title: "Zero Is Not a Measurement"
published: true
description: "The old summary, still sitting here."
tags: debugging, monitoring
canonical_url: https://tedagentic.com/posts/zero-is-not-a-measurement
---

The article itself starts here...

That block between the two --- lines is front matter: metadata written into the top of a markdown file, the same convention static site generators use. dev.to supports it. You can publish an article by sending the whole thing, metadata block included, and dev.to reads its settings from there.

The twenty-five articles that updated correctly had no such block. Their bodies were just the article.

Same author, same months, interleaved through the calendar. I had two kinds of article and had never known it, because from the outside — on the page, in the list, in the API — they looked identical.

And the one I had tested first, so carefully, with all those fingerprints? It was one of the twenty-five. My probe had been perfectly representative of the articles where the method works, and had told me nothing at all about the ones where it doesn't.

I was writing to the output of a function

Here is what happened on each of those twenty saves.

The request arrived with a new description. It was well-formed, the field was a legitimate part of the article, and dev.to accepted it. Then, as part of the same save, dev.to read the front matter out of the body, found a description: line there, and set the article's description from that. My value lasted exactly as long as it took to be overwritten by the block it was supposed to replace.

For those twenty articles, the description wasn't a field I could set. It was a computed value — read out of the body every time the article saved. I had been writing to the output of a function.

That is why the response said 200. The request was valid and it was processed. The write landed and was undone inside a single operation, and the status code described the first half.

Four of the twenty were stranger still. Their front matter had no description: line at all. For those, dev.to had been generating the description from the opening words of the article. They had never had a description as data in the first place — only a summary produced on demand, which no update to the field was ever going to reach.

A copy can be fixed. A derived value can't.

I've written before about copies of data drifting apart: a fact corrected in the database that never reached the page, prose that went on contradicting a dataset after the dataset was fixed. Those are real problems, but they share a comforting property. A copy exists. It is a stored thing that can be found, compared and corrected.

A derived value has no independent existence. There is nothing in the field to fix, because the field is not where the value lives. Overwrite it and you have written to a view, and a view is rebuilt from its source whenever the system feels like it. The only place to change it is upstream.

That is a different and worse failure than drift, because every check at the level of the field passes. You wrote it. The system confirmed it. For a moment, it was even true.

Changing the source

So the fix was to stop writing to the field and write to the source instead: edit the description: line inside each article's front matter, then send the whole body back. For the four with no such line, insert one.

That meant pushing the entire body of twenty live articles, not one field. A mistake there could damage far more than a summary, so every edit went through two gates. The new body had to differ from the old by exactly one line — one changed, or for the four, one added — or the edit was rejected before it was sent. And after each save I fetched the article fresh and confirmed two things: the description was now the new one, and the tags were exactly as they had been.

All twenty passed. The checker now reads forty-five of forty-five.

(One aside, for anyone scripting against the same API: dev.to answers Python's default urllib user agent with 403 Forbidden Bots. Send an ordinary user agent string and it works.)

You can't tell from the schema

The part that stays with me is that nothing warned me. The description was listed in the API as a property of the article. It accepted writes. It returned 200. There was no flag marking it as derived for some articles and stored for others, and I suspect there often isn't, anywhere. Whether a field is data or a projection of other data is usually an implementation detail, invisible until you write to it and watch it snap back.

So there are only two defences I trust now.

Read back from the source, not from the response. The response tells you the request was accepted. A fresh fetch of the thing, from the endpoint that doesn't cache, tells you whether your value survived. Those are different questions, and the success of the first says nothing about the second.

A probe only proves its own kind. One careful test validated my method for exactly the population that test article came from. If there are two kinds of thing hiding behind one interface, a single sample will happily vouch for the easy one. You find out there were two kinds when the second one fails — which is a strong argument for checking every result afterwards, rather than trusting a representative one beforehand.

The API never lied to me. It said it had accepted my request, and it had. I was the one who heard "it's done."

Clicks Rose 73%. I Shut the Site Down.

Ted — Tue, 25 Aug 2026 13:21:04 +0000

In June I fixed a real problem on a small editorial site I run.

The site was a client-side app — the kind where the server sends a nearly empty HTML shell and the browser assembles the actual page afterward. That works fine for people. It works badly for search crawlers, because the description a search engine shows underneath your link in the results comes from the HTML the server sent. Every page on this site was sending the same generic one. Fifty-odd pages, one description, all identical.

So I built a step into the deploy that renders each page's real title and description into the file the server hands out. Crawlers started seeing per-page text instead of one repeated boilerplate line.

It worked. I can show you that it worked.

The numbers said yes

Two things get measured here. Impressions are how many times one of your pages appeared in someone's search results. Clicks are how many times someone actually clicked through. Click-through rate is the second divided by the first — of the people who saw you, what fraction picked you.

The fix was aimed squarely at click-through rate. A better description should win more of the people already seeing you, without changing where you rank.

Comparing the 28 days before the fix to the 28 days ending this week:

	Clicks	CTR
Before the fix	11	0.47%
Now	19	0.56%

Click-through rate up about 19%. Clicks up 73%.

That is the shape you want. The intervention targeted a specific mechanism, the mechanism moved, and it moved in the predicted direction. If I'd put that on a dashboard and walked away, I'd have called it a win and gone looking for the next thing to fix.

Two months later I removed the site from all of my monitoring and retired it.

The number I wasn't watching

Here's the third column, the one that isn't on anybody's dashboard.

Not how many clicks did the site get — how many distinct pages earned anything at all.

	Pages with ≥1 impression	Pages with ≥1 click
Before the fix	13	5
A month later	12	5
Now	10	5

Ten. Out of about fifty.

And the same five pages have earned every single click for three consecutive months. Not five pages of a rotating cast — the identical five. The top five pages account for 91% of all impressions the site receives.

So both readings are true at once. The clicks went up 73%. The surface producing those clicks contracted by nearly a quarter. I got better at converting a shrinking audience.

How an aggregate hides a shrinking surface

This isn't a paradox, it's arithmetic, and it's worth being precise about because the same trap sits under a lot of small-system metrics.

Clicks is a sum over pages. Any sum can rise while its terms are disappearing, as long as the survivors grow faster than the losses. On a big site this washes out — thousands of pages, and one page's fate is noise. On a site where five pages produce 91% of everything, the total is those five pages. It stops being a measure of the site and becomes a measure of a handful of URLs.

The distinction that matters: clicks measured how well my best pages were doing. Coverage measured whether the site could still get new pages in front of anyone. Those are different questions, and only the second one tells you whether there's a business here.

My fix improved the first. It was never capable of improving the second — better descriptions help pages that already appear in results. It does nothing for pages that never appear at all. When I audited this site in June, nineteen of its pages had been crawled and passed over, and eighteen had never been crawled at all. Better descriptions were never going to reach either group. Two months on, the count of pages earning anything is lower than when I started, so whatever happened to those thirty-seven, it wasn't a breakthrough.

I shipped a change that could only help the pages that needed help least. And the metric I'd chosen to judge it by was the one metric that couldn't tell me that.

The detail that ended the argument

One more number, because it's the one that made the decision obvious rather than merely defensible.

The site's single best query is a person's name — a historical figure the site has a long article about. Over the last month that query put the page in front of people 454 times, at an average position of 8.6. That's the bottom of the first page of results. Respectable. Hard-won.

It earned one click. A click-through rate of 0.22%.

At position eight or nine you'd expect somewhere between 1.5% and 3%. Getting a fifth of the floor isn't a snippet problem, and it isn't something a better description fixes. It's what happens when you rank underneath a Wikipedia entry and one of those boxed summaries search engines assemble on the right-hand side. The question gets answered on the results page. The user's need is met before your link is a candidate. You are technically visible and functionally absent.

Which reframes the whole asset. The queries this site ranks for are queries that get answered without anyone leaving the search results. That is not a fixable snippet. That is the market telling you what your content is worth to it.

The part I got wrong

I laid all of this out and then offered to set up a checkpoint: re-run the exact comparison in a month, deliver it automatically, decide then with fresh numbers.

The answer I got back was that eight months of no movement was already the measurement.

That's correct, and I should have said it myself instead of proposing more data collection. The trend had twelve months in it. Best month in the entire year: 16 clicks — this month, the one that supposedly shows the fix working. Second best: 15, back in March, three months before the fix existed. The whole twelve-month range is 4 to 16. Impressions had peaked in February and were down 49% from that peak. Every one of those facts was already in front of me when I offered the checkpoint.

There's a failure mode here that looks like rigor and isn't. When a trend is genuinely ambiguous, more measurement is the right call. When a trend has been flat for eight months across every framing you can apply to it, proposing another month of measurement isn't rigor — it's deferring a decision while appearing thorough. The tell is whether you can describe, in advance, a result that would change your mind. I couldn't. I'd already concluded the only remaining lever was a multi-month authority build that wasn't going to happen. Wanting one more datapoint was me not wanting to say so.

What I'd watch instead

For anything small enough that a handful of pages dominates the totals, the aggregate is close to useless as a health signal. It tells you how your winners are doing, and your winners are usually fine — that's what makes them winners.

The two numbers that actually carried information:

Distinct pages earning any impression, tracked over time. Rising means the system still has reach. Falling means it's consolidating toward a few survivors, which is what dying looks like from the inside before it looks like anything on a revenue chart.

Whether anything outside the incumbent set has broken through. Not "did clicks grow" but "did a page that wasn't earning before start earning." On this site the answer was no, for three straight months, while clicks climbed 73%.

If both of those are flat or falling, a rising total isn't recovery. It's a smaller thing being measured more favourably. I had that data the whole time, in the same account, one query away. I just wasn't asking for it, because the number I was asking for kept coming back green.

Every Page Returned 200. There Was No Database.

Ted — Mon, 24 Aug 2026 15:23:46 +0000

My travel site was completely non-functional for about four hours today and never once went down.

That is not wordplay. Every route returned HTTP 200 the entire time. Time to first byte was normal. The HTML came back full of content — property names, headings, structured data, the lot. An uptime monitor pointed at any page on the site would have recorded a flawless afternoon.

Nobody could book anything, because there was nothing to book.

What actually broke

The site's backend runs on a managed platform — a hosted wrapper around Postgres that bundles the database, authentication, file storage and serverless functions into one project. The workspace was on the free tier. Its allowance ran out and the project was paused.

There was no option to settle a balance and carry on. The dashboard offered exactly one action: upgrade to a $25/month plan. That is worth separating from the usual story about an operator forgetting to pay a bill, because it is a different failure — the free tier had been running production for nine months, and the notice that it would stop was the stopping itself.

The first thing I checked was DNS, and that told me how complete this was:

$ dig +short @1.1.1.1 <project>.supabase.co
$ dig +short @8.8.8.8 <project>.supabase.co
$ dig +short @1.1.1.1 supabase.co
76.76.21.21

Two independent resolvers, no answer. The parent domain resolved fine, so this was not a DNS outage — the subdomain had been withdrawn. NXDOMAIN, not a pause page.

One exhausted allowance took four subsystems with it at once: the database, every login, every uploaded image, and the edge function that records outbound clicks. They felt like separate concerns right up until they shared a fate.

What a visitor saw

Here is the part that has stayed with me.

The site is a React SPA with a prerender step: at build time, every route is rendered to static HTML with its data baked in. That HTML is what a crawler gets, and it is why curl looked reassuring:

$ curl -s https://<site>/hotels | grep -o "Patterson Inn"
Patterson Inn

The content was right there. So for a few minutes I believed the damage was limited to logged-in features.

Then I rendered the page in a real browser instead of reading its source. React hydrated, fired its query at a host that no longer existed, got nothing, and re-rendered the component with an empty array.

Found 0 verified stays.

Every filter read zero. The listings area showed skeleton placeholders. The page was beautifully styled, fully responsive, instantly loaded, and contained nothing whatsoever.

I checked the one thing that actually pays for the site:

$ curl -s https://<site>/hotels | grep -c "affiliate"
0

Not a single outbound booking link exists in the static HTML. Those URLs come from the database at render time. So there was no accidental fallback, no degraded-but-working state. Just a confident, empty shop.

Why nothing alerted me

I found out because I happened to open the platform dashboard for an unrelated reason.

Think about what a conventional check would have needed to notice. HTTP status? 200. Response time? Fine. Does the HTML contain expected text? Yes — the prerendered content was intact. Is the page non-empty? Very. Does the site load? Beautifully.

Every signal a normal monitor collects was healthy, because every one of them measures whether the server answered, not whether the answer meant anything. The failure lived entirely in the gap between those two questions.

That gap is not exotic. Any site that renders static shells and fills them from an API has it. The more thorough your prerendering, the wider it gets — because the static layer keeps looking healthy long after the dynamic layer has died.

The crawler got a better experience than the customer

There is a genuine consolation, and it is a strange one.

Googlebot does not execute JavaScript on the schedule a visitor's browser does. It was served the prerendered HTML — 623 routes of intact content, correct structured data, working internal links. Throughout the outage, the search engine's view of the site was perfectly healthy.

So the prerendering protected the index and abandoned the customer. For a nine-month-old domain still being assessed, that is the more expensive of the two to lose, and I would not have chosen differently. But it is worth being clear about what was protected and what was not, because "the prerendering saved us" is only half true and the other half was every booking that afternoon.

Resolving is not serving

One more thing, because it cost me the backup.

Before paying, I set a script running that polled DNS and would export everything the moment the host came back — the idea being that if the alert came at 3am, the data would already be on my disk.

It fired at 15:42. It captured nothing.

DNS came back roughly forty minutes before the service did. My readiness check probed the API root, got a 401, and treated that as life — a 401 is a valid HTTP response from a live server, after all. It then requested every table and received 521 for each one, which is the CDN saying it cannot reach the origin. The script dutifully reported sixteen tables as "not exported" and exited pleased with itself.

The fix is to probe something that only works when the thing actually works:

# before: any response means alive
code=$(curl -s -o /dev/null -w '%{http_code}' "$HOST/rest/v1/")
[ "$code" != "000" ] && break

# after: a real table, and only 200 counts
code=$(curl -s -o /dev/null -w '%{http_code}' \
  "$HOST/rest/v1/hotels?select=name&limit=1" -H "apikey: $KEY")
[ "$code" = "200" ] && break

Same mistake as the monitoring, one layer down. I asked "did something answer?" when the question was "did the thing I need work?"

The re-run pulled 91MB: 220 countries, 143 dispensaries, 58 properties, and 563 storage objects, none of which failed.

The reframe

Uptime is a measurement of whether your server responds. For a static site those are the same thing, which is why the convention exists and why it goes unexamined. For a data-driven site they are different questions, and the difference is the entire business.

A 500 would have been better. A 500 pages an on-call rota, trips a monitor, sends an email. What I had instead was a site that passed every automated check while converting nothing, and would have kept doing it for as long as I did not personally look.

So the health check I actually needed was never about the server. It was about the data:

Fetch the listings endpoint. Assert the count is greater than zero.

Two lines. It would have caught this within a minute, and it is the only check in this entire incident that would have.

The corollary is a build I have not made yet: the prerender already pulls every listing at build time, so the client could fall back to that baked-in data when a live fetch fails, instead of rendering zeros. Stale listings still sell. Zero listings sell nothing, and look immaculate doing it.