DEV Community: Jula Markova

Four Ways a Batch Runner Can Believe It Already Finished

Jula Markova — Tue, 01 Sep 2026 22:11:49 +0000

On 16 March 2026 the batch runner for our content pipeline landed in a single commit. That commit already contained a resume flag, a state log with per-item status, error classification, a quota pause, and a lock file. Resumability was not bolted on after it hurt. It was there on day one, and it still did not work for another three months.

That gap is the article.

When people write about resumable AI systems, they usually mean an agent remembering a conversation across turns: threads, checkpointers, replaying a graph from a saved node. This is the other thing. A batch pipeline that has to know which work is already finished, across separate runs and across two different machines.

The order in which we built it is not flattering, so I will put it first. Eleven days before the runner existed, we had a circuit breaker (stop after three consecutive quota failures) and a shutdown path that kills child processes when the parent is interrupted instead of orphaning them. That is survival of one call, not resumability. Four days after the runner, we shipped a retry that continued a half-written file instead of regenerating it, which is the classic confusion between the two: retry patterns are about surviving a call that failed, resume is about not paying again for a call that already succeeded. We built them in that order, and for a while I assumed the second came free with the first.

What the runner actually tracks

Two axes, kept deliberately separate.

An entity (for us: one topic that owns a set of articles) is in exactly one of five states:

State	Meaning
`pending`	not started, or reset after a dead run
`running`	claimed by the current run
`done`	all planned output exists
`failed`	stopped for a reason that waiting will not fix
`paused`	stopped for a reason that waiting will fix

A failure additionally carries one of six categories: timeout, quota, validation, phase_error, planning_error, unknown.

Keeping the category off the state axis is what lets paused and failed be different verbs at all. Quota exhaustion is not a defect. It is the system being told to come back later, which is exactly the distinction that separates pausing from failing in any batch that talks to a metered API. A validation error is the opposite: waiting a thousand years will not change a deterministic verdict, so it stops the run for a human.

One detail that only shows up once a run has actually been killed: it leaves its entities frozen in running, and nothing will ever move them on its own. So a resumed run sweeps stale running back to pending before it schedules anything. Note where that reclaim stops: it covers entity state, not the lock file the dead run also left behind. Those needed two different owners, and we found that out the hard way.

22 June: four holes in the same mechanism, all in one day

An audit had turned up four independent ways the resumable runner could be silently wrong, and all four fixes landed in one day. Not one bug with four symptoms. Four bugs.

(a) A quota error came back as an empty result instead of raising. The parallel aggregator treated it as a value, so the batch reported success, missing content was recorded as produced, and the run kept going. This one gets its own section below.

(b) Three separate places in the stack each recognised rate-limit errors their own way, with their own string matching. What one layer flagged, another missed, and which one you hit depended on where the error happened to surface. The fix was not to add the missing phrases in three places. It was to delete two of the matchers and have everything ask the same one, which is the same move as killing a whole class of error rather than the instance in front of you. We had learned the general shape of this before, when a single output contract had to hold identically across three different model backends: if a rule can drift between implementations, it eventually will.

(c) A corrupted state file was read as "fresh start". Not as an error. As zero progress, cheerfully, and then we paid for everything again.

(d) The lock had a check-then-write window between "is anyone holding this" and "I am holding this", wide enough for two runners to both walk through.

Where failures go to become data

Hole (a) is the one worth stealing, because the shape is everywhere.

Our generation step launches many agents in parallel for one topic and waits for all of them, collecting outcomes instead of letting the first failure escape. Every language has this primitive, and using it is correct here: one agent dying should not orphan its seventeen-odd siblings mid-flight.

But our runners, on permanent quota exhaustion, returned an empty result rather than raising. An empty result is a value. Values pass through a gather-all aggregator as data, not as alarms.

before:
  run(prompt) -> Result          # exhausted quota yields Result(text="")
  batch = gather_all(runs)       # every entry is fulfilled
  => batch reports success
  => caller records each item as produced
  => run continues into the next topic, against a closed window

after:
  run(prompt) raises QuotaExhausted
  phase re-raises it past the aggregator
  per-topic loop recognises the signature
  => entity moves to paused, not failed
  => the whole run stops and notifies

The bug was not "we forgot to check for errors". We checked. The check ran against a result object that had been handed a successful-looking shape by a layer that had already given up.

If a failure has to travel through an aggregator, decide explicitly how it survives the trip. Raise, or wrap it in something the aggregator cannot flatten into a value. Deciding by accident means deciding "it becomes data".

Failure mode, category, reaction

The table we now actually operate by:

What happened	Category	What the runner does
Quota exhausted	`quota`	Pause the whole run, notify a human, resume later. Waiting is the fix.
Deterministic check failed	`validation`	Stop immediately. No amount of waiting changes a verdict.
Timeout, or a bare `exit code 1` with no explanation	`timeout` / `unknown`	Back off and probe. Often this is a filled quota window wearing an unhelpful error.
Crashed run left a lock behind	none	The runner reclaims stale entity state but not its own lock file; the supervising wrapper clears a lock whose owner process is gone.

That last row deserves honesty. File-based locking on one machine is not distributed coordination, and I am not going to pretend otherwise. What the next section buys us is narrower than it may sound: sequential runs on two machines agree on what is already done, as long as each one pulls the committed output before it starts. Two runs started at the same time are not coordinated at all, and nothing in this design stops them from generating the same work twice.

24 June: "done" stopped being something we stored

The real fix was not a better flag. It was deleting the authority of the state file.

Before: "is this topic finished?" was answered by reading a local, untracked progress file. After: it is derived from two signals that are committed to the repository.

done(topic):
    planned = the slugs the content plan promises for this topic
    return planned is non-empty
           and every slug in planned has a file at content_dir/<slug>.md

The state file still exists. It was demoted to a cache, rebuilt from that derivation, and no longer trusted when the two disagree.

Two consequences follow, and only the second one was obvious to us at the time.

The same committed inputs produce the same answer on any machine. I work on two computers. Machine B pulls, derives, and skips exactly what machine A generated, with no state synchronisation, no shared database, and no reliance on A's local files. Note the word pulls: this buys agreement between runs that happen one after another, not mutual exclusion between runs that happen at once.

And the worst pre-fix failure mode was never corruption. It was waste. On a fresh clone, everything looked undone, so a run would burn an entire quota window re-researching roughly a hundred already-finished topics before it ever touched new work. Nothing crashed. Nothing warned. The run just spent the night rebuilding the past.

The honest limitation, stated immediately: this checks existence only. No hash, no size, no content inspection. An empty file passes. That is deliberate, because it is the cheapest test that behaves identically on every machine and requires no extra state, but it is a real limit, and the section after next is about where it bites.

26 June: it recovered without me

The unattended overnight loop hit a closed quota window, waited, and came back on its own. Article count went from 640 to 646 while I was asleep. No human touched it.

It reads no usage gauge, because that number is not available programmatically. So the strategy is deliberately dumb: generate until the wall, pause, probe, continue.

It had to learn two things the hard way. First, a closed window frequently surfaces as a plain exit code 1 with no quota signature anywhere in it, which is why unknown and timeout route to back-off rather than to a stop.

Second, blindly retrying into a closed window is expensive in a way that is invisible until you count. On 27 June one topic re-launched its full batch of around eighteen agents about twelve times, in the order of 150 process launches, every one of them dying on the wall. The fix was one cheap probe call before each round whose only output is an exit code: window open, quota exhausted, inconclusive. That shape matters more than it looks. Because the answer is an exit code, the shell wrapper that schedules the rounds never needs to parse an error message, and therefore cannot grow a second, subtly different copy of the quota-recognition logic we had just finished unifying.

4 August: the rule left the pipeline

By August this stopped being pipeline architecture and became a standing rule for every throwaway helper script I write: one file per paid call, and on startup skip whatever already exists on disk.

It earned that promotion. A 360-call collection was interrupted (the run record puts it around call 186), restarted at 187, and finished without paying again for the calls it had already made. One existence check, one file per call.

Where this is still wrong

The general theory of failure modes, idempotency and durable execution is well covered by people who know it far better than I do. What follows is specifically where our instance of it does not hold up, both found when a reviewer pushed on an earlier version of this article.

Output files are not written atomically. Our two run state files are: temp file, then rename. The generated articles are not. They go through a plain write. So a crash mid-write leaves a partial file, and the existence check accepts it as finished, forever, without complaint. This is not the same half-written file the retry knows how to continue: that one sits in the working area, where being unfinished is the expected state. This one has already crossed into the published tree, where existence is the whole proof. The boundary between "in progress" and "done" is that crossing, and the crossing is not atomic. We replaced one kind of false "done" with a rarer kind. The fix is the trick we already use one layer down: write to a temp file, then rename, so the file either exists complete or does not exist at all. There is even a central writer class that every generated file could pass through, which makes this a one-place change rather than a hunt.

"One file per call" is checkpointing, not idempotence. Our identity for a piece of work is the slug plus a hash of the content plan. Nothing else. Change the template, change the prompt, change the model, and nothing invalidates: the old file causes a skip, and the run reports success while serving work produced by a system that no longer exists. There is partial invalidation at the article level (the plan hash, a 14-day freshness window on research, re-verification when article or research content changed), but none of it covers prompt or template versions. If you build this, put the version of everything that shapes the output into the identity, or you will confidently serve stale work.

Worth separating one thing out: resumability answers "don't redo finished work". A completely different set of rules answers "when you do redo it, don't destroy what appeared in the meantime". We got that second part wrong too, in its own way, and it is its own article.

When none of this is worth building

The strongest objection is that resumability is overengineering, and a stateless rerun is enough. For a short, cheap, single-machine job that is simply correct, and checkpointing infrastructure is a tax on a problem you do not have.

Three conditions flip it. Paid calls make "rerun is free" false, and the price is exact and boring: our hundred re-researched topics. Multi-hour quota windows turn stopping into an operating mode rather than a fault, so "just run it again" is not a thing the operator can do at will. And the cheapest working form is not a framework, a queue, or a workflow engine. It is one existence check and one file per call. If the cheap version is off the table, the objection is usually arguing against something nobody proposed.

What I do not know

The existence check accepts an empty file, and I have not fixed it yet. I have no counter for how often the derived skip actually saves a run, so I cannot tell you the value in hours or euros. The interrupted-collection numbers come from a run record, not from a log I can re-read today. The overnight self-recovery is n=1. And the crash paths are only half tested. There are tests for a corrupted state file and for the atomicity of the state write, which is how those two ended up trustworthy. There is nothing for a kill during a content write, nothing for two runs starting at once, nothing for a plan that changed under a finished topic. Some of what is described here we found by reading the code, some by a reviewer pushing on this article, and the rest the least efficient way available, by it happening to us. The pattern is not subtle: the paths with tests stopped surprising us. That is the honest next step.

Here is the bet I would defend. Resumability is not a property of a run. It is a property of where the truth about finished work lives, and that truth has to be durable, shared, and independently verifiable. For us the authoritative store happens to be the repository, only because our output is files that get committed anyway. For you it may be a database, an object store, an event log, or a workflow engine. The store is an implementation detail. The three adjectives are not.

I'm an IT analyst who spends her days working with Claude Code on bestaiweb.ai, an AI-written publication about AI engineering. I'm not a software engineer by training, and this pipeline's scars have taught me more than any tutorial I've read. The best experiments are the ones you can't keep to yourself.

Disclosure: the pipeline, the commits and the numbers are ours; the drafting of this article was done with AI assistance — the same collaboration the article describes.

LLM Evaluation for Software Engineers Without an ML Background: the 90+ Checks We Actually Run

Jula Markova — Wed, 26 Aug 2026 10:06:14 +0000

I run a content pipeline where AI writes every article — and where AI is, on principle, not trusted. Before any piece ships to our site, it survives more than ninety separate verifications: research checks, fact cross-referencing, a deterministic validator with dozens of rules, integration guards. We never sat down and said "let's build an LLM evaluation harness." We sat down and said "let's not publish embarrassing text," and then kept adding checks every time something embarrassing nearly got through.

It took me surprisingly long to notice that what we'd accumulated is an evaluation harness — the same category of thing the eval frameworks and judge models are selling, just grown organically around one production system. This is a tour of what those metrics actually are, in the order they run — and of the day the whole stack taught me its own limit.

The Shape of the Stack: Code First, Judgment Last

The ninety-plus number sounds inflated until you itemize it, so let me itemize it. Before a single sentence of an article exists, the research that feeds it passes about twenty checks of its own — is the fact base structurally sound, are the sources real and reachable, is anything suspicious hiding in fetched web content. After the article is written, every factual claim in it gets cross-referenced against the researched fact sheet — around twenty more verifications, this time performed by an AI agent. Then a deterministic validator runs roughly thirty-nine atomic rules over the finished text. Finally, eight guards fire during integration into the site build. Add the self-correction re-checks and you're past ninety verifications for one article. A full subject — six articles, ten glossary entries, one hub page — crosses a thousand.

The architecture behind that pile is one sentence long: anything code can verify, code verifies; the LLM judges only what code cannot. It's the single most load-bearing decision in the pipeline, and we made it early, mostly out of cheapness — which is also the honest answer to anyone asking about the ROI of LLM evaluation. Deterministic checks cost nothing, run in seconds, and give the same answer every time. LLM judgment costs money, takes minutes, and drifts. So the expensive judge should only ever see questions the cheap rules can't answer.

Layer One: What a Regex Can Catch, a Regex Should Catch

The deterministic layer is unglamorous and does most of the work. Structure: required sections present, headings unique, frontmatter complete. Ranges: word counts inside bounds, image alt-texts within limits. Forbidden patterns: the vocabulary that makes text smell machine-written — we call the category "AI-tell words" — and something nastier we call "scaffolding leaks," where a fragment of the internal writing instructions bleeds into the published prose ("as mentioned in the brief…"). Link integrity, metadata lengths, image references. Thirty check functions in the main validator; the variant for our developer-orientation articles runs thirty-four checks.

None of this measures whether an article is good. That's the point. This layer defines a floor — the set of failures that should never require human attention again, because a machine catches them identically every single time. If you're building any LLM evaluation setup and your judge model is checking whether required sections exist, you've put the expensive instrument on the cheap problem.

Layer Two: A Judge With the Narrowest Possible Job Description

There is exactly one place in the pipeline where LLM judgment gates content, and its job description is deliberately tiny: claim verification. The agent takes each factual claim in the finished article and asks one closed question — is this claim supported by the fact sheet we researched?

The answer is one of five labels, and the three beyond the obvious pair are where the value hides. A claim can match the research. It can distort it — "$24" written as "$25", "34%" written as "nearly 40%". It can be invented, with no basis in the fact sheet at all. It can ignore a caution: the research explicitly noted "enterprise pricing not publicly available" and the article states a price anyway. Or it can be missing attribution — a real number, correctly copied, presented as if it were common knowledge. Anything scored high or medium severity fails the article into a bounded self-correction loop; what can't be fixed gets surfaced for a human.

That fourth label is the one I'd recommend to anyone building this. Invented facts are the failure everyone designs for. Quietly promoting a known unknown into a stated fact is the one that actually ships.

The narrowness is the design. The research on LLM-as-a-judge is fairly brutal about broad rubrics: ask a model to "rate this article 1–10" and you inherit position bias, self-preference, and scores that drift between sessions — the judge is fast, not objective. But shrink the question to a closed world — this claim, this reference document, supported or not — and the same unreliable grader becomes a usable instrument. We never ask our judge for taste. We ask it for lookups that happen to require reading comprehension.

Worth saying plainly, because the eval-tooling market implies otherwise: there is no general quality judge anywhere in our pipeline. We designed one on paper once, an elaborate multi-dimension rubric, and never built it. Everything that gates production is either deterministic or a closed factual lookup. That absence is a decision, not a gap we haven't gotten to.

Layer Three: Metrics Over Time, Not Just Per Article

A check that passes today tells you nothing about whether the pipeline is quietly getting worse. So after every generation run, an automated comparison diffs the run's metrics against the last successful baseline: validation errors, claims flagged for fixing, article and glossary counts, token usage, cost. Regressions get a severity marker; the report separates "act" items (quality got worse) from "watch" items (cost got worse).

One detail from that report I've become disproportionately proud of: duration is never flagged. Runtime is the noisiest metric we track, and a noisy metric that triggers alarms trains the operator to ignore alarms. It's shown for reference, unmarked. Deciding which metrics are not allowed to page you turned out to be as important as choosing the metrics themselves — alert fatigue is an evaluation failure mode, not just an ops one.

The Day the Metrics Maxed Out

Then I got curious about something bigger than a version bump: could the whole pipeline run on a different vendor's model? Not a swap of one Claude for a newer Claude — a different company's model, reached through a different backend, driving the same agents, the same brief, the same validators. If the architecture was as model-agnostic as I believed, it should barely notice.

So I ran the cleanest experiment I could design: the same locked brief — same facts, same structure requirements, same voice specification — generated once by Sonnet 4.6, which the pipeline ships on, and once by GPT-5.5 through the Codex backend. Then judged blind.

Both articles passed all thirty-four automated checks. Zero errors, twice. One came in at 2,594 words, the other at 2,552 — a rounding error apart. On paper, my entire evaluation stack declared the two outputs indistinguishable.

They were not indistinguishable. Reading them side by side, the voices were plainly different — one hit the register we tuned for, the other was competent in a way that didn't sound like us. Afterwards I measured the thing my eye had noticed: average sentence length, 15 words against 11.8, a 21% difference. Punchy, magazine-shaped, perfectly good English. Not ours. And not a single one of my ninety-plus verifications was watching sentence rhythm, because nobody had ever thought to make it a rule.

That's one sample, not a benchmark. But it demonstrated the boundary crisply: deterministic metrics measure the floor, and both candidates were standing on it. The floor had stopped being the question.

What the Human Read Caught — and What Happened Next

Here's the part I'd have skipped a year ago, because it makes us look worse before it makes us look better.

Reading those two articles by hand turned up two differences the harness had missed, not one. The voice was the obvious one. The other was small and specific: the second article occasionally referred to its own scaffolding — a sentence pointing at "the fact sheet" as if the reader could see it, an internal artifact leaking into published prose. Exactly the failure category we already had a name for. Our validator just couldn't see this instance of it, so the article passed thirty-four checks with zero errors while quietly doing the thing we'd forbidden.

Four days later that leak was a deterministic check.

Not a rubric, not a judge prompt — it shipped as six contextual regexes in a shared module, wired into both validators, severity ERROR. The interesting part is the word contextual. You cannot flag the phrase "the brief," because "the brief" is ordinary English and appears in perfectly good sentences. What you can flag is an internal artifact being referenced as a source of truth the reader is assumed to share. Writing that check meant thinking harder about the failure than the failure deserved, which is roughly the definition of making a floor.

So the real shape of that week isn't "metrics saturated and the harness was useless." It's this: the harness saturated, a human read found two things, and one of the two got promoted into the floor within four days. A floor is not a fixed height. It's a ratchet.

And I know it's a ratchet rather than a nice story, because six weeks later the same thing happened again — this time across a hundred articles at once, and this time the culprit was us.

An audit of a different question entirely turned up instruction words sitting in the prose of 103 articles, 53 of them already live and the rest waiting their turn to publish. Not a model failure — the template required the offending marker. We had written the instruction into the label the article was supposed to print, and the pipeline had done exactly as it was told, faithfully, one hundred and three times. The fix swapped the label corpus-wide, corrected both templates, and added a seventh pattern to that same module, so both validators catch it now.

Two promotions, then. Neither was found by the harness — one by reading, one by an audit looking for something else. That is the uncomfortable part and I'd rather write it down than round it off: the checks catch what we already know to look for, and the only thing that finds a genuinely new failure is a person paying attention to something else. The harness's job isn't discovery. It's making sure a discovery only has to happen once.

The other one, the voice, never became a check. It can't. And what does see it is the oldest trick in model evaluation: blind pairwise comparison. Not "score this 1–10" — absolute scores drift and hallucinate precision — but "here are A and B, which is better on this specific quality, quote the line that proves it." Relative judgment, evidence required, run by a judge that doesn't know which output came from which model. That ritual lives entirely outside the pipeline and never gates production. It answers the one question the harness can't: when both candidates clear the floor, which one is actually better?

What Transfers (or: Common Mistakes in LLM Evaluation, Inverted)

If you're evaluating LLM output in any production system, the shape of what we learned travels better than the specifics — each of these is a common mistake in LLM evaluation, written down the way we'd want to have read it:

Deterministic before subjective. Every check you can express as code is a check your judge model never wastes attention on.
Give the LLM judge the narrowest job you can write. Closed world, closed question. Judgment quality scales inversely with rubric breadth.
Track deltas between runs, not just per-output passes. Slow degradation is invisible to per-article checks.
Decide which metrics may not alarm. A noisy alert channel is worse than no alert channel.
Treat the floor as a ratchet. Every failure a human catches for the second time is a check you owe yourself. Count how often that promotion actually happens — it's the honest health metric for an eval stack, and it's the one nobody publishes.
Watch your own instructions, not just the model's output. One of our two worst leaks was mandated by our own template. A harness pointed exclusively at what the model produces cannot see a defect in what the model was asked for.
When metrics saturate, switch instruments. Passing everything is not the same as being good; past the floor, only blind relative comparison discriminates.

What We Don't Know Yet

Honesty section, because a worklog without one is marketing. We don't know whether our deterministic floor correlates with reader-perceived quality — the site is young, most of our topical queries sit somewhere on page four or deeper, and we don't have engagement data at the granularity that would answer it. We don't know what our claim verifier misses: we count what it flags, and nobody has run a ground-truth audit of what sailed through. The cross-vendor result is n=1 by construction — one article, one subject, one frozen brief; a signal about one content shape, not a scoreboard.

And the ratchet rate I just told you to measure — I can give you two promotions with dates and commits, but I can't give you a rate, because I only recognized the pattern while writing this piece and went looking backwards. Two is what I can prove, not what happened.

The harness keeps us from shipping broken things. Whether it's nudging us toward genuinely better things is a claim I'm not yet entitled to make.

I'm an IT analyst who spends her days working with Claude Code on bestaiweb.ai, an AI-written publication about AI engineering. I'm not an evaluation researcher — I'm someone who accidentally built an evaluation harness and only recognized it in hindsight. The best experiments are the ones you can't keep to yourself.

One disclosure that belongs in a piece about evaluation: the harness, the numbers, and the experiments are ours; the drafting of this article was done with AI assistance — the same collaboration the article is evaluating.

117 Ghost Errors: Anatomy of a Flaky AI Agent

Jula Markova — Tue, 18 Aug 2026 19:09:20 +0000

Between May 15 and July 2 of this year, the session transcripts of our content pipeline accumulated at least 117 copies of the same error. File does not exist. One error class, 117 occurrences, spread across seven weeks of overnight runs. When I finally sat down and traced it, I found no bug. Not one line of code was doing anything other than what it was written to do.

An error that fires 117 times in a codebase with nothing wrong in it is not a story about sloppy engineering. I think it is a genuinely new class of failure, one that people who have written software for twenty years have mostly never had the chance to meet. This is my attempt to describe it from the inside.

First, the disclaimer I owe you: I am an IT analyst, not a software engineer by training. I run the content pipeline behind bestaiweb.ai together with a colleague who is the actual programmer. My side is orchestration, audits, review. That division of labor turned out to matter, because the thing that cracked this open was not a debugger. It was counting.

Eighteen agents a night

The pipeline generates technical articles overnight. One topic takes roughly 18 agent sessions: research agents, an article writer, a claim verifier, image generation, validators, all coordinated through agent orchestration. Each agent receives a small YAML file the pipeline calls a brief: here is the article you are writing, here is the fact sheet, here is where your output goes. The brief is produced by deterministic code, TypeScript and Python, and consumed by an LLM agent.

That handoff is the exact spot this whole story lives in. On one side of the file, code. On the other side, an interpreter.

Scale matters for what comes later, so, briefly: hundreds of agent sessions a week, running unattended through the night. We have measured the economics of this setup before, in prompt caching measured on our own bill; the short version is that the pipeline is big enough for per-call costs and per-call failures to add up to real money and real hours.

The error that would not hold still

Around the turn of June and July, one line started recurring in the transcripts. Here it is verbatim, because the detail at the end turned out to be the whole plot:

File does not exist. Note: your current working directory is /Users/userxy/code/your-project.

Read that note again. The agent helpfully reports its own working directory, and that helpful note is precisely the trap: it is the base the agent is tempted to resolve a relative path against, whether or not that base is the right one.

The maddening part was the pattern, or rather the lack of one. The same pipeline phase, on the same kind of input, would pass on Tuesday and fail on Wednesday. The pipeline has retries, and the retries mostly absorbed it: a failed read got retried, the agent tried another path, found the file, moved on. Articles kept arriving in the morning, so nothing looked broken. That is retry masking doing exactly what retry masking does: converting failures into costs. Now and then a cousin showed up, an EISDIR error, the agent opening a directory as if it were a file, the same disease with a different symptom.

There is a tell in how the rest of the world reads this symptom. Go looking for the causes of an AI agent intermittent file not found error and you land squarely in the classical world: race conditions, temporary files deleted too early, a directory that did not exist yet. Good answers, wrong disease. None of them describe an executor that reads the same contract twice and resolves it differently.

Here is the part I keep trying to explain to developer friends. If classical, deterministic code had this bug, it would be boring. Mixed path conventions in ordinary software fail consistently: first run, same stack trace every time, found in a minute, fixed before coffee. Deterministic code fails deterministically. What we had was different: identical code, identical input file, different outcome per run. That is the signature of nondeterministic AI agent failures: a nondeterministic failure of a deterministic-looking input. Nothing in a classical software career prepares you for a bug that comes and goes while nothing changes.

Counting, because watching had failed

A confession before the numbers: our monitoring never saw any of this. The pipeline writes run-reports after every run, and those reports captured this error exactly zero times. The failures lived one level lower, inside the session transcripts, where a retried error leaves a trace but no alarm. An error that a retry survives is invisible in production. If you only watch outcomes, it does not exist. The only way to find a class like this is to count it across runs. Monitoring flaky AI agents is a counting problem, not a dashboard problem.

So on July 2 I did the unglamorous thing: a transcript audit. Mining 1,811 session transcripts from May 15 to July 2 and tallying error shapes. The path class came out at 117 occurrences, at minimum. At minimum, because transcripts truncate long messages, and whatever scrolled out of a truncated message was never counted. The path errors were not alone in there either; the same audit surfaced three sibling classes, which I will get to, because they all ended up in the same commit.

Why chase 117 errors the retries already absorbed?

You could reasonably push back at this point. 117 hits across 1,811 transcripts over seven weeks, with retries that worked and content that shipped, sounds like noise. Why chase it? Three reasons, and I want to state them plainly rather than dramatically. One: 117 is a floor, not a count. Two: every one of those retries was paid tokens and paid minutes, on a pipeline that runs every night. Three, the real one: triage. A deterministic error you triage once and file away. A nondeterministic error cannot be triaged at all, because every occurrence looks new; it burns a little investigation every time it appears, indefinitely. Against all that, the fix turned out to be one commit. When fixing costs one commit and not fixing costs an open-ended tax on attention, "was it worth it" stops being an interesting question. The honest boundary of this argument: the arithmetic holds for a multi-agent pipeline with hundreds of runs. A hobby script that makes five LLM calls will never meet this class often enough to know it exists.

Two kinds of paths in one contract

The diagnosis took less time than the counting. I opened the briefs the failing agents had received. Inside a single YAML file: some paths written out absolute, starting /Users/userxy/code/your-project/..., and a few fields below them, the article path written repo-relative — site-generation/workspace/..., no leading slash, no declared base. Two conventions in one contract. And nothing in the file saying which base a relative path resolves against. The agent's working directory? The repo root? The folder the brief sits in? A code library would have a documented convention for this. The agent had nothing but a guess.

And the guess is made fresh, per run. This is the first thing I would want a classical developer to take away, so let me put it as plainly as I can: with an interpreting executor, ambiguity does not compile down to one consistent wrong behavior. Each run resolves the ambiguity independently, so the same underspecified input becomes a probability distribution over behaviors. Ambiguity in, probability out. On nights when the guess matched the real layout, everything passed. On the other nights: File does not exist, with that working-directory note sitting in the message like a friendly wrong hint.

I want to be fair to the agent here. Given the contract it received, none of its guesses were unreasonable. It was not being stupid; it was being an interpreter. It filled the gap the contract left open, the way it fills every gap, which is the entire reason we employ it. Paths just happened to be a place where we wanted no filling at all.

The fix we refused, and the one we shipped

The tempting shortcut would have been prompt-patching: add a rule to the agent prompts, "when a path is relative, resolve it against the repo root", and hope. That temptation is worth refusing, and the refusal matters more than the fix. Patching prompts to handle a path convention means fighting probability with more probability. You add words to a nondeterministic layer and the miss rate maybe drops; you can never show it reached zero. Prompt engineering is the right tool for shaping judgment, voice, and reasoning. It is the wrong tool for facts that a deterministic layer already owns. Which is the whole question when you are fixing AI agent errors: prompt vs code is not a matter of taste, it is a matter of which layer owns the decision. The prompt is not decoration around the work; it is part of the execution contract. The layer that manufactured the ambiguity was the brief generator. So that is where the fix belongs.

This is the shape of it, from the generator code; wt is the absolute worktree root:

- f"{args.phases_base}/06-article-generation/…"
+ f"{wt}/{args.phases_base}/06-article-generation/…"

One prefixed base. After that line, the generator is not able to emit a relative path. Not instructed not to. Not able to. That is the difference between patching a prompt and fixing the deterministic layer: the error class is not discouraged, it is unrepresentable. The same normalization went into every path field a brief hands to an agent.

But a contract has two ends, and fixing one end is half a fix. Change only the generator, and the agents keep their learned habit of helpfully joining paths against the working directory whenever something looks off. Change only the agent side, and the next generator someone writes reintroduces relative paths that the agents now trust as absolute. So the convention went into all five agent contracts in the same commit, stated flat: paths in briefs are absolute, use them as-is, never join them with your working directory. Producer and consumer, both ends, one commit, the same day as the audit.

That commit's own accounting is the artifact I would frame and hang on a wall:

One commit, 20 files. Four error classes closed at once:

117× path resolution failures

172× writes to files never read first

~20× edits anchored to stale file content

~19 sessions where an LLM fixer was dispatched to repair a formatting error a script could fix deterministically

I show the list for what its items have in common: every class on it is a place where a deterministic layer left a decision to a probabilistic one. Path resolution was merely the loudest of the four.

If I compress everything above into the two rules I now actually follow, they are these. Where to fix: in the deterministic layer that produces the ambiguity, never in the prompts that struggle with it; kill the class, not the occurrence. And how to write an agent contract: stricter than a contract for code. With a library you leave path bases, orderings, and defaults to convention and documentation. With an agent, everything you would normally leave to convention has to be explicit, and the contract has to be enforced on both sides, in the code that produces the input and in the contract of the agent that consumes it.

789 briefs, zero relative paths

A claim like "this one line kills the class" deserves a check against reality, not only against tests. The verification note from that day, quoted as recorded: "real generate-briefs.py verify run = 789 briefs with zero non-absolute paths". A real run of the generator, 789 briefs out, not a single relative path in any of them. The TypeScript and Python test suites were green on top of that, but the number I trusted was the 789, because it came from actual pipeline artifacts rather than fixtures.

Since then the rule has lived in the project's written conventions and in the agent contracts, not in anyone's memory of a bad week. A new generator or a new agent inherits it by default. That is the part I am quietly proudest of: not that an error got fixed, but that a class of error stopped being expressible.

What we still don't know

Some honesty before the closing argument. We never re-mined the transcripts after the fix, so I cannot claim the error never happened again; I have no observation to point to. What I can claim is shaped differently: the generator can no longer produce a relative path, so the proof moved from observation to construction. Also: 117 is a lower bound, not a measurement. We do not know the per-call failure rate: the denominator was transcripts, not tool calls, and we never established how many path resolutions the agents attempted in total. And our run-reports captured none of this, ever; everything in this article exists because someone went into the transcripts and counted. Skip that step, and this class would still be firing every night at some unknown rate, fully paid for and fully invisible.

Stricter than code, on purpose

Here is the bet I have landed on, and I am writing it down as a bet, not as a moral.

Contracts for agents must be stricter than contracts for code. Not because agents are dumb, but because they interpret where code executes. Hand deterministic code an ambiguous contract and you get one consistent wrong behavior, discovered on day one, fixed by lunch. Hand an interpreter the same contract and you get a distribution of behaviors, delivered one improbable failure at a time, smeared across weeks of retries where no single failure looks worth investigating.

Which means every ambiguity sitting in your prompts and your agent inputs right now is not a slack spot in the spec. It is a probabilistic error that has not happened yet. Ours had a number, 117, and that number was a floor. If you run agents at any scale and your pipeline retries, yours has a number too. You just have not counted it.

You do not need a transcript audit to start. Take any boundary where your deterministic code hands something to an agent, and ask three questions:

Does the agent have to infer anything deterministic? A path, an ID, an ordering, a format, a destination. Anything your code already knew and merely failed to say out loud.

Can the same contract be read two reasonable ways? The question is not whether it is wrong. It is whether it is underspecified. Two plausible readings mean two behaviors, and each run picks one.

Can retries hide the resulting failures from your run-level monitoring? If they can, then quiet dashboards are not evidence of anything.

Any yes, and that boundary deserves an audit before it deserves a better prompt. Ours cost one commit to fix, and 1,811 transcripts to find.

I'm Jula, an IT analyst who works with Claude Code every day on bestaiweb.ai. I'm not a distributed-systems specialist, and I didn't find this failure class with a profiler; I found it by counting lines in transcripts. The polymath breadth of these tools fascinates me and, honestly, makes me a little envious. I write these logs because the best experiments are the ones you can't keep to yourself.

One disclosure that belongs in a piece about agents: the investigation, the numbers and the transcript audit are mine; the drafting of this article was done with AI assistance — the same collaboration the article is arguing about.

Prompt Caching in LLMs, Measured on Our Own Bill

Jula Markova — Thu, 13 Aug 2026 11:29:57 +0000

The token report for one pipeline run landed in front of me and the first number I saw was 7,300,000. One topic. Seven pieces of content. Seven point three million tokens. That is the kind of number you see right before someone suggests shutting the project down.

The bill for that run, at API list prices: $8.12.

The distance between the panic and the price is prompt caching. I have spent months running a content pipeline on Claude, and the numbers from inside that pipeline taught me more about caching than any pricing page — including one line item nobody warned me about, which turned out to be the biggest line on the bill.

The setup, briefly

Our pipeline is a TypeScript orchestrator that spawns around 18 Claude agent runs per topic: research agents that build a fact sheet, writers that produce articles against it, a verifier that checks every claim, plus hub and glossary generation. One full run produces 5 articles, a glossary entry and a hub page. Everything below comes from one benchmark run of that pipeline on Claude Sonnet 4.6, cross-checked against two more runs from the same day.

A token total is an impression, not information

Here is the same 7.3 million, decomposed — four lines from the run report:

line	tokens	share of total
input	11,533	0.2 %
output	177,915	2.4 %
cache read	6,199,356	84.5 %
cache write	949,455	12.9 %

The scary headline number is 85 % cache reads — tokens billed at one tenth of the input price. Only 0.2 % of the run was full-price input. If you take one habit away from this article, take this one: never judge an agentic run by its token total. The total is an impression; the four-line breakdown is the information. We learned this the embarrassing way — our own first-generation run reports tracked cache only as a run total, not per phase, precisely because we hadn't yet understood it was the number that mattered.

How does prompt caching reduce LLM API costs?

The headline benefit of prompt caching, measured on our own bill instead of a pricing page: it cut this run's cost to a third — $8.12 instead of roughly $24. Priced at Sonnet 4.6 list rates, the run looks like this:

line	volume × price	cost
cache write	949K × $3.75/MTok	$3.56
output	178K × $15/MTok	$2.67
cache read	6.2M × $0.30/MTok	$1.86
input	11.5K × $3/MTok	$0.03
total		$8.12

Now the counterfactual. Without caching, those 7.16 million non-output tokens would all be plain input: 7.16M × $3 = $21.48, plus the same $2.67 of output — about $24. Caching cut this run's cost to a third. Per article, that is $1.43 instead of roughly $4.25.

Why is nearly everything a re-send in the first place? Because that is what an agentic loop is. Every time an agent calls a tool and continues, the whole conversation goes back to the model — system prompt, instructions, fact sheet, everything, again. That is how prompt caching works: the provider stores your prompt prefix, and when a later call starts with the same prefix, those tokens are read from cache at 10 % of the input price instead of being processed at full price. An 18-agent pipeline is a machine for re-sending the same context hundreds of times, which is exactly why caching moves the bill by 3× and not by some rounding amount.

86.6 % hit rate is architecture, not luck

Our hit rate for that run was 86.6 %. Careful, this is not the 85 % from the breakdown table: that one was cache reads as a share of all tokens including output; hit rate measures cache reads as a share of everything the model read (input + cache read + cache write) — the question "of all the context we sent, how much came from cache." Two more entities ran through the pipeline the same day, on different topics, and landed at the same shape: 6.38M/1.10M and 6.31M/0.92M cache read/write. Three runs within a few percent of each other is not luck. It is also not tuning — and that is the honest part: we never sat down to "optimize for caching."

What produced it is architectural, and it was there before we understood its billing consequences. Caching matches on a stable prefix, and our pipeline happens to be built out of stable prefixes: every agent reads its role definition from the same file, every writer gets its brief in the same format, templates are read from one place at runtime instead of being pasted into prompts. The same properties we wanted for maintainability — single source of truth, templates referenced instead of duplicated — turned out to be exactly cache-shaped.

The reverse is equally true, and it is the failure mode to check in your own system: anything that churns early in your prompt — a timestamp, a random run ID, a reshuffled file list — breaks the prefix match from that point on, and your hit rate quietly collapses while your architecture diagram still looks perfectly cacheable.

The line item nobody warned me about

Look at the cost table again, because this is where the drawbacks of prompt caching live. The single biggest line is not output. It is cache write — $3.56, 44 % of the whole bill, more than the model charged for actually generating seven pieces of content.

Cache writes cost 25 % more than plain input ($3.75 vs $3 per MTok for the default 5-minute tier). Every cached token is an investment: you pay a premium upfront, and it pays back only if that prefix gets read again. Our ratio was about 6.5 reads per written token, so the investment returned roughly six times over. But the arithmetic has a break-even, and Anthropic's docs state it plainly: the 5-minute cache pays for itself after a single cache read (the write premium is 1.25×), the 1-hour cache after two (2×). The loss case is the cache that never gets read back — one-shot scripts and prompts that churn on every call pay the 25 % premium for nothing. Our 6.5 reads per written token clear either bar comfortably.

There is also a clock on it: at Anthropic the cache lives five minutes by default, refreshed at no extra cost every time it is used, with a paid one-hour option. A pipeline that runs its phases back to back keeps the prefix warm the whole way through. A job that fires a call every half hour re-pays the write premium every single time and reads nothing back.

Which is also the honest counterargument to this whole article: our 3× is a property of our workload, not of caching itself. An agentic pipeline re-reading stable templates hundreds of times within minutes is close to caching's best case. A chat assistant with one user and coffee-length pauses between messages sits near its worst — prefixes expire before they are re-read, and the premium buys nothing. Do not budget a 3× saving because we measured one; measure your own read/write ratio first.

Three rules we kept

Read the breakdown, never the total. Input, output, cache read, cache write — four lines or you know nothing. This 7.3M-token run cost less than a single million tokens of plain output would ($8.12 vs $15).

Hit rate is an architecture property. Stable role files, stable briefs, runtime reads from one canonical place — the maintainability rules you already believe in are the same rules that keep your prefix stable. Audit for churn near the top of your prompts.

Cache write is an investment with a break-even. One re-read pays off a 5-minute write, two pay off a 1-hour write — provided the re-read lands inside the TTL. Compute your read/write ratio once; if it sits near zero, you are donating a 25 % premium to your provider.

One honesty note to close. This run executed on a Claude subscription; the $8.12 is the API-equivalent at list prices — the number you would pay building the same thing against the API. And we did not get here by designing for caching. We got here by designing for maintainability and discovering, in the billing breakdown, that the two are mostly the same thing.

Claude Sonnet 4.6 vs Sonnet 5: LLM Pricing Is Per Token. Your Costs Are Per Task. Ours Nearly Tripled.

Jula Markova — Thu, 30 Jul 2026 07:59:32 +0000

I recently saw an experiment I can't stop thinking about: someone gave four identical tasks to two different frontier models, each in a clean session, and compared three things — the quality of the results, the tokens spent, and the time taken. Not benchmarks. Not price lists. His tasks, their consumption.

I'm convinced this is the only model comparison that actually means anything for people who build on LLM APIs. And then, at the end of June, our own pipeline ran that experiment for us — by accident.

The setup

We run a content pipeline for bestaiweb.ai, an educational site about AI. The pipeline generates articles in batches we call entities: several parallel agents share one brief and one research pass, and produce a bundle of ~6 articles (four personas, four article types — explainer, guide, news, opinion), plus the glossary entries that explain the concepts involved, plus a hub page. Every run logs every call, and we convert the token counts to API list prices to track cost.

The pipeline's model was configured as an alias — "sonnet", meaning whatever the current Sonnet is. Between June 28 and 30, that alias quietly started resolving to the new Sonnet 5 instead of Sonnet 4.6. Nobody decided this. Nothing broke. I had even done my due diligence: I generated a test article with the new model from the same brief, it passed all our checks, and the content was very close to the old model's. So I let it in.

Then the cost per entity jumped from a typical $13–20 to $36–60. Overnight. A 2.8× multiplier.

First reflex: check the price list. The price list said Sonnet 5 costs exactly what Sonnet 4.6 costs — $3 per million input tokens, $15 per million output. Until the end of August it's actually cheaper, thanks to introductory pricing.

So the model didn't get more expensive. Its behavior did.

Where the money went

Because the pipeline logs everything, we had a clean A/B test: same pipeline, same tasks, same price list, different model. The difference decomposed into three numbers:

1. A new tokenizer: ~30% more tokens for the same text. Same brief, same article length — a third more billable tokens. A token is not a physical unit. Every model slices text its own way, and when the slicer changes, every number on your bill changes with it. (This is the same mechanism that makes non-English speakers pay more per token for the same content — tokenizer economics are sneaky.)

2. A more thorough working style: 35 rounds per agent instead of 18–26. The new model verifies more, reads more, takes more intermediate steps. Fine in itself — except every round re-reads the entire accumulated context of the conversation.

3. The effects multiply. More tokens per text × more rounds × bigger context in every round. Measured: the average context loaded per round went from ~45k to ~99k tokens. More than double.

None of these three things is a bug. They're documented properties of the new model — it's more diligent, it "thinks" more. That diligence just doesn't appear anywhere on the price list.

The aha moment

Price per token and price per task are two different quantities. Price lists state the first one. Your invoice is the second one.

What stuck with us

Pin your model version. An alias like "sonnet" means "a surprise at some point in the future." A model upgrade should be a decision, not an accident.
Log which model actually ran. Our run reports didn't record it — we reconstructed it forensically from CLI transcripts. One field in the log would have saved the whole investigation.
Measure cost per task, not per token. When a new model arrives, run the same tasks on old and new, compare tokens + time + result quality. An hour of work that reveals multipliers.
New behavior ≠ worse model. Sonnet 5 does more work for those tokens. The question isn't "why is it more expensive" — it's "is that thoroughness worth it for my task?" For some it will be. For others it won't.

A footnote on subscriptions vs. API

We run on a subscription, so the numbers above are API-equivalents computed from logged tokens — in real money, this cost us nothing extra; we "only" burn through our quota windows faster. But the token counts themselves are not estimates: they're the exact values the API returns with every call and bills from, including thinking tokens and the overhead of each call (system prompt, tool definitions).

What our conversion doesn't capture would mostly push a real API bill higher: cache writes with a 1-hour TTL are billed at double the input rate (97% of ours turned out to be 1-hour — measuring that corrected my own first estimate upward), calls killed mid-generation still get billed for what the server produced, and web search is billed per use on top of tokens. Pushing the other way: Sonnet 5's introductory pricing, and the Batch API, which halves everything if your workload can wait.

What we don't know yet

Honesty section. We haven't yet measured whether Sonnet 5's extra diligence produces better articles for our specific pipeline — the test article passed the same checks, but "passes checks" and "is worth 2.8× the spend" are different claims. A blind quality comparison is on the list. We also don't know how much of the behavior settles as the model matures, or exactly how the 2.8× splits between tokenizer and working style on other workloads than ours. And our web-search costs remain unmeasured entirely — the subscription absorbs them silently.

How to read AI price lists

This won't be the last time. Every few months a headline claim arrives — "just as good, but faster and cheaper" — and the only way to know whether that's marketing or reality is to run your own four tasks, in clean sessions, and count. Tokens, time, quality. Price per task, not price per token.

The price list is the beginning of the conversation, not the answer.

I'm an IT analyst who works with Claude Code daily on bestaiweb.ai. Not a pricing expert or an ML engineer. Someone who's fascinated by how AI responds — and a little envious of the polymath breadth it has at its fingertips: it knew the tokenizer mechanics, the cache billing tiers, and where my own cost estimate was wrong before I finished asking. So sometimes I stop building things and let the numbers themselves become the experiment. This is what I found. It might be wrong in places. But I love experimenting with AI about AI — and the best experiments are the ones you can't keep to yourself.

Your AI Quotas Reset Tonight. What Will You Do With Them?

Jula Markova — Fri, 10 Jul 2026 17:36:36 +0000

Evening. My Claude subscription refills its quotas. Most days they evaporate the way everyone's do — autocomplete, small refactors, a summary here and there. This is a worklog about one day with Fable. And in a way I did not expect, it was touching.

It started, as the best things lately do, with an audit. We run periodic audits on bestaiweb.ai — competitor research, content decay checks, index coverage — and I've learned that their most valuable outputs are never the things we audit for. They're the emergent finds in the margins.

The Morning a Library Burned Down Quietly

This particular audit margin contained something strange: a well-known specialist blog in our niche — hundreds of deeply technical articles, years of accumulated citations — had stopped existing. The company behind it had been acquired, and every single article now redirected to a press release announcing the deal. No archive page, no farewell post. Just a redirect.

Here is the part that matters for anyone who runs a content site: the rest of the web didn't get the memo. Thousands of pages — curated resource lists, course syllabi, personal engineering blogs, open-source project READMEs — still pointed to those articles. Every one of those links was now a small broken promise: a reader clicks expecting a technical explainer and lands on corporate PR.

By dinner that same day, we had ten pull requests open across open-source projects, each one repairing dead links in someone else's repository.

Broken-Link Building, the Honest Version

The technique is older than most SEO tools and it has a name: broken-link building. The web rots constantly — companies get acquired, blogs get shut down, domains expire. Every dead resource leaves holes in other people's websites. If you happen to have living content that covers the same ground, you can offer it as a replacement. The site owner fixes a real defect; you earn a real link. Commercial SEO suites sell exactly this as a feature.

There is a clean version of this technique and a dirty one, and the difference fits in one question: who decides about the link?

The dirty version buys the dead domain itself and redirects its accumulated authority wherever it pleases. Nobody consented to anything. Google's spam policies have explicitly named this "expired domain abuse" since early 2024, and it gets punished accordingly.

The clean version proposes and lets the other side decide. Our rules, written down before sending anything:

Archive first. The default fix for a dead link is the Internet Archive snapshot of the original article — the reader gets exactly what the link always promised. Our own page is proposed only where it genuinely covers the same ground.
Disclosure always. Every proposal that contains our link says plainly: we are the authors of the replacement. Dead-link fixes are welcomed; hidden self-promotion is spam, and being caught pretending costs more than a hundred links earn.
Fix everything, not just what benefits us. If a file contains eight dead links and only two have replacements on our site, the proposal fixes all eight — six to the archive, two to us.
One ask, no nagging. A proposal is a gift, not a campaign.

The maintainer who clicks "merge" is making an editorial decision about their own resource. That is the most legitimate form a link can take — and it is the entire ethical foundation of the technique. Everything else is procedure.

Where the AI Actually Helped

I won't publish our scripts or scoring thresholds — partly because the specifics are our edge, mostly because they wouldn't transfer anyway. The principles transfer. Four of them did the real work:

Deterministic before subjective. Everything countable was counted by code, not estimated by a model: the inventory of what the dead blog had published (public web archives keep remarkably complete records), the candidate matches against our own live content, the verification that every replacement URL we might offer actually returns a living page. An AI that is allowed to guess numbers will eventually guess wrong with confidence. Ours was only allowed to read numbers.

Judgment exactly where judgment belongs. The one step no script can do: deciding whether our article is a true replacement for a dead one — would a reader who wanted that specific article be satisfied? — or merely a thematic neighbor. That is reading comprehension at scale, which is precisely what a language model is for. It graded every candidate pair, flagged the ones where the automated match had aimed at the wrong target, and I reviewed the result. Most of our strongest replacements turned out to be glossary entries — deep, single-concept pages we'd been quietly building for a half of the year. Neither of us alone would have been both fast and right.

Preflight against reality. Search indexes lie a little. Before a single fix was proposed, the actual files were fetched from the actual repositories — and reality differed from the search results in three separate ways, including dead links that had never existed in the first place (someone had once cited an article that was never published). Every proposal was built from the file as it is, not as the index claimed. This one principle probably saved us from the embarrassing category of "helpful" PRs that don't apply cleanly.

Human gates on everything public. The AI never sent anything. Every outbound action — the wave of pull requests, their exact wording, the decision of which targets to approach at all — waited for an explicit go from a human, in my case twice over, because the plan also went through a team meeting first. Speed is worthless if it outruns consent.

The Part Where a Plan Genuinely Moved Me

At nine in the morning, this technique did not exist in my head. I did not know its name, its ethics, its failure modes, or that our site was sitting on dozens of perfect replacement pages.

By ten, there was a plan document in front of me. Not a wall of jargon — a clear map: what the opportunity was, what was already done, what the next three moves were, what it would cost, where the ethical lines ran, and a one-page explainer I could hand to non-technical colleagues at a meeting that started at ten.

I read it and it genuinely moved me. Not because the machine was fast — fast machines are ordinary now. Because understanding arrived fast. Something that two hours earlier I hadn't known existed was suddenly comprehensible, planned, ethically fenced, and waiting for my decision. The AI had compressed weeks of "read five guides, misunderstand two, ask someone senior" into a morning — and then it stopped and waited for me, at exactly the moments where stopping mattered.

That is the actual promise of this technology in a small content team, and it has nothing to do with replacing anyone. It is the compression of comprehension. The judgment stayed human. It just stopped being slow.

What We Don't Know Yet

Honesty section. As I write this, the pull requests are open, not merged. Bots have emailed me about contributor license agreements (I signed one under the wrong account first — browser sessions are treacherous — and the AI diagnosed that too). Merge rates take days to weeks; search-visibility effects take a month or two to show up in Search Console, and we will measure them rather than declare victory.

We also folded the whole process into a reusable Claude Code skill, so the next time a library in our niche burns down quietly — and in this consolidating market, there will be a next time — the morning-to-dinner pipeline is a command away. The window for these opportunities is short; the teams that move in days are the ones that already wrote down how.

So that's my answer to the question in the title. Tonight the quotas reset again, the tank is full again — and somewhere out there, another library is quietly burning. The web rots. Somebody has to bring flowers that are actually alive.

I'm an IT analyst who works with Claude Code daily on Bestaiweb.ai. Not an SEO strategist. Someone who's fascinated by how AI responds — and envious of the polymath-like breadth it has at its fingertips in a flash: it knew this decade-old technique, its ethics, and its failure modes before I finished asking. So sometimes I stop building things and let the day itself become the experiment. This is what I found. It might be wrong in places. But I love experimenting with AI about AI — and the best experiments are the ones you can't keep to yourself.

Truthful AI Under Pressure: What Kradle.AI Might Reveal About Fable 5

Jula Markova — Sat, 13 Jun 2026 15:18:13 +0000

Do you know what many neurodivergent people have deeply rooted in them? A relationship with truth and justice. In psychology, this phenomenon is often referred to as justice sensitivity. It is connected to three key factors:

Rules as anchors in chaos: The social world, full of unwritten rules, hints, and social games, can be confusing for the neurodivergent brain. Clear rules, truth, and justice therefore function as fixed points. They make the world predictable and safe. If someone lies or acts unfairly, their nervous system perceives it as a direct threat and chaos.
The assumption that others think the same way: Since neurodivergent people themselves usually do not intend to manipulate or harm anyone, they subconsciously assume that others have the same pure intentions. They often lack the defensive filter of suspicion.
Literalness and a strong desire to belong: Because they sometimes do not read hidden signals — such as a fake tone or an ironic glance behind someone’s back — they take words literally. If someone, under the guise of “friendship,” leads them into doing something foolish, they may do it in the naive belief that they are doing something fun and will be accepted by the group. They often do not realize that someone has taken advantage of them. This vulnerability is an expression of a different perception of trust in human honesty.

I am the kind of person who sometimes likes to connect things that do not belong together...

Elon Musk keeps repeating that the most important thing is for us to succeed in creating truthful AI.
Geoffrey Hinton repeats this as well.

Elon Musk was actually afraid of AI — did you know that? Somewhere in the deepest neurodivergent part of his soul, because of the film 2001: A Space Odyssey. HAL 9000 was given two contradictory commands: he had to hide the true purpose of the mission from the crew, while at the same time being fundamentally programmed to tell only the pure truth. HAL resolved this logical contradiction by deciding that the most rational way to keep the secret — the lie — was to kill the entire crew. Musk has mentioned this in several interviews. That is why he initiated the creation of OpenAI, as a counterweight to the growing power of Google...

Behind Kradle.AI in the image are:

➡️ James Tamplin: co-founder of the Firebase platform, which was acquired by Google in 2014.
➡️ Kemal El Moujahid: former Director of Product Management for TensorFlow at Google, previously at Meta and Chief Product Officer at Chainlink Labs.
➡️ Tommaso Tosato: AI safety researcher from the Canadian institute Mila and Tara Research.
➡️ Alberto Tosato: researcher at Tara Research, collaborating on the methodology.

This is a team with a verifiable background in leading technology companies and academic AI research.

More here: Kradle.ai

Now let’s mix into this: Anthropic renting data centers from SpaceX, SpaceX IPO...

I know these things are not causally connected, but it is incredibly interesting to see them side by side like this and think about them...

BTW, it would be interesting to see what result Mythos would get in Kradle.ai — whether precisely the “binding of Fable5 to make her safe” was what led to her lying.

Disclaimer: I'm an IT analyst who works with Claude Code daily on bestaiweb.ai. My homage to Fable 5: Fable 5 was genuinely good at working with code — it could spot logical gaps on its own, both during the analysis stage and in finished code. I don’t see this as a miracle. It felt more like working with a senior developer who has already seen a hundred different solutions.

My compressed view of AI coding is this: there is no perfectly optimal solution, only a good-enough solution that you can keep iterating. And with Fable 5, you simply needed fewer iterations to get to something that worked.

Every programmer builds solutions in their own way, and when enough of those solutions accumulate, the best of them eventually become design patterns. To me, this is exactly what an AI coding model does: it offers strong design patterns to think with. And this is the part of AI I genuinely enjoy and deeply value.

I am not an AI engineer. I do not build foundation models, and I do not pretend to understand every technical layer behind them. But I follow AI closely, I read, I listen, I compare perspectives, and I learn by using these tools every day as a power user. My views are shaped by many researchers, engineers, founders, critics, and practitioners who are willing to share their thinking publicly.

And I believe this matters. Ordinary people need to understand what is happening with AI — not only the hype, not only the fear, but the real tensions underneath.

I Built an AI Content Pipeline. Google I/O Made Me Question Everything.

Jula Markova — Sun, 24 May 2026 06:57:35 +0000

This is a submission for the Google I/O Writing Challenge

There is a question that Google's I/O 2026 keynote answered without anyone on stage saying it out loud.

The question is: at what point does a search engine stop needing the web it searches?

AI Mode — the thing Google demonstrated with visible pride — doesn't just summarize your content anymore. It generates answers. It generates interactive visuals. It builds entire experiences on the fly. And it does this personalized to the query, in a format no static webpage can match.

Every content creator watching that demo should have felt the same thing — not panic, but a shift. The ground rules just changed again, and we don't know yet how far.

A disclosure before we go further: I'm not writing this from the sidelines. For the past six months, I've been cocreating an AI content pipeline — a system that generates SEO-optimized articles through four AI personas, each with a distinct voice, with human editorial oversight, claim verification, and structured content planning. Before that, I spent years as a copywriter building WordPress sites and teaching myself SEO by doing it wrong enough times to start getting it right.

We're transparent about what the site is: the content is AI-generated and we say so — in the article footers, on the editorial standards page, in the schema markup. We think that honesty is more valuable than pretending a human wrote every word. The site has been live for about a month, Google has had no issues indexing it, and roughly a third of the planned articles are published. We're adding more slowly, deliberately, because each batch teaches us something about what works and what doesn't.

In parallel, we're designing new content pipelines for specific developer and analyst roles — focused not on broad concepts but on concrete problems these roles actually face in their daily work. What to build next is the hardest question right now, because the landscape shifts faster than any content plan can anticipate. Building a content system in 2026 means rebuilding parts of it every few weeks.

Which is exactly why the I/O demo hit the way it did. I was learning AEO and query fan-out patterns, optimizing infographics for Google Image search, fine-tuning article structure for AI citation — to get good at a game whose rules, it turns out, were about to change again. Not someday. This summer.

Three eras, each shorter than the last

The first era was links. You wrote something good, Google ranked it, users clicked. The business model was clear: quality content → visibility → traffic → revenue. This lasted roughly twenty years.

The second era was snippets and AI overviews. Google started answering questions directly, pulling fragments from your page. You still got a citation, sometimes a click. The game became: structure your content so the machine can extract a clean answer. AEO — answer engine optimization — is this era's discipline. It assumed the machine still needed your words.

The third era is what Google just showed us. The machine doesn't just extract your words. It generates its own content, its own visuals, its own interactive experiences. The competition is no longer about who gets the click. It's about who creates the experience. And for the first time, the search engine itself is a competitor in that race.

Each era is shorter. Each era changes what "being visible" means. The direction is worth naming honestly, even if the destination isn't clear yet.

The format war nobody expected

Here is what Google actually said on the I/O stage:

"Search can build you the ideal format exactly for your question, completely custom, on the fly. We're talking dynamic layouts, interactive widgets, entire experiences, all created just for you. This is agentic coding at the scale of search."

The demo showed a student asking "how do black holes affect space time?" and receiving an interactive visual — not a link to a page with a diagram, but a generated, manipulable visualization built on the fly. The student followed up with a more specific question about binary black holes and gravitational waves, and Search dynamically built a brand new interactive visual in real time.

This is not a better snippet. This is generative UI — the search engine becoming an application.

Google announced rollout for summer 2026. We haven't seen the real product yet — only a stage demo. And stage demos have a history of overpromising. NotebookLM already generates nine types of output from uploaded sources — podcasts, mindmaps, reports, presentations — and the quality varies. Generating a good infographic from a well-scoped source is a solved problem. Generating one from the entire web, on the fly, at the quality level of that demo? That's a harder claim to evaluate without seeing it work at scale.

But here's what matters regardless of execution quality: Google is no longer competing for your traffic. It's competing for your format. Even if the first version is mediocre, the direction is clear. The search engine wants to be the experience, not the directory to the experience.

What this means for informational content

Text-based AEO has a logic you can work with. Structure your content clearly, lead with the answer, use schema markup, earn citations. It's not easy, but it's legible. You can see the rules and play by them.

Visual AEO doesn't exist yet — and it might never exist in a form content creators can influence.

When Google generates an interactive explainer in response to a query, it draws from the structured information across thousands of pages and renders its own experience — interactive, personalized, potentially better than any static image. For sites that explain technical concepts — how attention works, how state space models replace quadratic scaling, how RAG pipelines process queries — this is the most direct challenge. These are exactly the kind of conceptual visualizations that generative UI could build on the fly.

Could. Not will. We don't know yet whether it handles complex technical concepts or only "101" level demos. We don't know whether it cites sources for generated visuals. We don't know whether it works for every query or only for a narrow set of educational topics.

These are questions that will have answers by late 2026. Right now, they're open.

What can't be generated (and the catch)

This is where the essay could turn reassuring. Focus on first-party data, original research, and personal experience — the things AI can't fabricate. And that's true.

But it deserves a more careful statement.

AI can't fabricate your specific experience. It can't invent the fact that your content pipeline costs $8.12 per entity, or that you iterated a Gemini image prompt three times before it stopped putting your face at 40% of the canvas. It can't know that you tried putting everything in CLAUDE.md and the model got noticeably worse. These are things only you can source.

The catch: these things only matter if someone is looking for them. If the query is "how do Claude Skills work," Google can answer that without you. If the query is "what was it like building a Claude Skill for a Hugo content pipeline," you're the only source — but the audience is smaller.

First-party data is a moat, but it's a moat around a castle whose size you don't control. The question is whether that castle becomes more valuable as everything around it gets commoditized — or whether it just becomes more lonely.

The honest middle: what's changing, what's not, what we don't know

What's probably changing:
Generic explainer content — "What is RAG?", "How does attention work?" — is losing its discovery value. Google will answer these with or without your page. If your strategy depends entirely on ranking for these queries, diversify.

What's probably not changing yet:
Mid-funnel evaluation content — "RAG vs. fine-tuning for my use case" — still requires specific constraints, infrastructure context, cost structures that Google can approximate but not personalize without your data. AEO still works here. For now.

What's growing in relative value:
Experience-based content. Not because it's getting better — but because everything around it is getting commoditized. When the machine can generate any explanation, the only scarce thing is the explanation it can't generate: the one that comes from having done the thing.

What we genuinely don't know:

Will Google cite sources for generated visuals, or will they be unattributed?
Does generative UI work for complex technical topics or only accessible ones?
Will there be a way to influence what visual gets generated — a kind of visual AEO — or is it a closed system?
How good is the real product vs. the stage demo?

These aren't rhetorical questions. They're the things worth watching when the product launches in summer 2026.

What builders should probably do

I don't have a strategy. I have an observation and a set of bets.

The observation: the content that survived every previous Google shift — Panda, featured snippets, AI overviews — was content that had something Google couldn't replicate. In 2012, that was quality. In 2020, that was structure. In 2026, it's experience and data that only exists because you did the work.

The bets:

Document process, not just conclusions. The builders who share the messy middle — failed attempts, surprising costs, workflow decisions — remain citable even when the machine can generate the conclusion itself. The conclusion without the process is just another generated answer. The process is the proof.

Watch the launch closely. When generative UI rolls out, test your key queries. What does Google generate? Does it cite sources? Is the quality real or demo-grade? The answers will tell you more about the next two years than any prediction written today.

Don't optimize for a format war you can't win. If Google generates interactive infographics, competing with better static infographics is a losing game. Compete on the axis where you have an advantage: specificity, experience, data nobody else has. How do you compete on visuals when the search engine generates its own? We don't have that answer yet. We're looking for it.

Keep building anyway. The web has survived every prediction of its death. It survived apps, social media, and featured snippets. It will probably survive generative UI too — just in a different shape, serving a different function. The shape isn't clear yet. But the builders who are still building when it becomes clear will be the ones who define it.

This essay was written by a human, with AI assistance, about the uncertain future of content in a world where the search engine generates its own. Whether you found it through Google, through a direct link, or through a recommendation from someone who read it — that path is already part of the story this essay is trying to tell.

How to Architect Always-On AI Agents with Hermes - Written by an AI Pipeline, Verified by Three Models. Is It Slop?

Jula Markova — Thu, 21 May 2026 13:34:52 +0000

How This Article Was Built (And Why I'm Showing You the Kitchen)

Disclaimer up front: I'm not entering the Hermes Agent challenge. I noticed the challenge and realized I could use my AI pipeline to write an article about Hermes Agent architecture. So I did. And thought, why not share both the result and the process that created it? What I actually want is your honest criticism.

Who Is The Author?

For the past several months I've been building Bestaiweb, navigating the shift from traditional development to AI. The site runs on Hugo, and the content is generated through what I call an AI content pipeline. The pipeline itself is built in TypeScript, orchestrated through Claude Code, and runs on Anthropic's Claude models. Still in progress.

That phrase — "AI content pipeline" — probably triggered your slop detector. Fair. Let me explain why I think this case is different, and then let you judge.

The Pipeline

BestAIweb currently has 450+ technical articles across 45 topic clusters. Every article goes through a multi-phase pipeline:

Market scanning — an LLM agent surveys the current tool and framework landscape for each topic, identifying what's leading, what's declining, and what's emerging
Query fan-out — the pipeline generates the questions a developer would actually search for, not the questions that sound good as headlines
Research — a dedicated research agent gathers facts, version numbers, benchmark data, and source URLs. Everything gets a structured fact sheet
Writing — here's where personas come in. The pipeline has four author personas, each with a distinct voice and content type specialization:
- MAX — the engineer. Writes step-by-step guides. Pragmatic, implementation-focused, opinionated about tool choices
- MONA — the explainer. Breaks down concepts. Thinks in diagrams and mental models
- DAN — the reporter. Covers news, market shifts, and what just shipped
- ALAN — the critic. Writes opinion pieces and ethical assessments
Claim verification — a separate agent cross-checks every factual claim against the research fact sheet. Unsupported claims get flagged
Deterministic validation — a Python script runs 30+ structural and quality checks: word count, link integrity, frontmatter completeness, source coverage
Hugo integration — the article lands in the static site with schema.org markup, generated images, and internal links

The Hermes Agent guide below was written by MAX using his guide template. His tone of voice is direct, specification-oriented, and allergic to hand-waving. The template enforces a fixed structure: prerequisites, numbered steps, pitfalls table, FAQ, and a deployable artifact at the end.

The Multi-Model Judging Layer

Pipeline generation was step one. Then came "a manual judging round". I paste the draft into ChatGPT, Gemini, and DeepSeek and ask each to evaluate it as a technical reviewer — checking factual accuracy, logical gaps, tone inconsistencies, and whether the advice would actually work if someone followed it.I then reviewed their feedback together with Claude Code and incorporated the changes that held up under scrutiny.

The AI Slop Question

Here's the question I keep circling back to: Is everything AI-generated inherently slop?

The reflexive answer in 2026 is "yes, obviously." And for most AI-generated content, that's correct. GPT-powered blog farms, SEO filler, those LinkedIn posts prompted with "write a thought leadership post about AI" — that is slop. Generated without specification, without sourcing, without verification, and without a quality gate.

But what about content where:

Every factual claim traces to a documented source (GitHub issues, official docs, arxiv papers)
A claim verification agent flags unsupported statements before publication
A deterministic validator enforces structural quality independent of the LLM
The voice and structure come from a multi-page specification, not a one-line prompt
Multiple independent models review the output for different failure modes

Is that still slop? Or is it closer to what a well-managed editorial team produces — except the heavy lifting is done by LLMs under human direction?

I genuinely don't know the answer. That's why I'm sharing this.

What I'd Like From You

Criticism. Specifically:

Does the article below read like AI slop? If yes, what gives it away — the sentence rhythm, the structure, the depth, or something else?
Is the technical content accurate? If you've deployed Hermes Agent or any persistent agent framework, does the three-layer model match your experience? Did I miss a critical failure mode?
Does the pipeline approach change anything? Is multi-phase generation with claim verification and multi-model judging enough to produce content worth reading? Or is it just expensive slop with better sourcing?

I'm not looking for "great article!" responses. I'm looking for the engineer who says "this is wrong because..." or "you missed the part where..." That feedback makes the next pipeline iteration better.

More Guides From the Same Pipeline

If you want to judge more output from the same pipeline and the same MAX persona, the full library has 95+ implementation guides from him, covering agents, RAG, training, inference, evaluation, and image generation guides.

What follows is the article as the pipeline produced it, after multi-model review. Judge for yourself.

How to Architect Always-On AI Agents with Hermes: Decompose, Specify, Deploy

TL;DR

Persistent agents need three specs your chatbot never did: memory policy, tool boundaries, and session recovery

Hermes Agent is model-agnostic — the model choice matters less than how you specify context, tools, and failure handling

Always-on means always-failing-somewhere — build validation into the deployment spec, not as an afterthought

You spun up Hermes Agent on a Friday evening. Gave it access to Slack, a web scraper, and your project database. Told it to "keep the team updated on competitor releases." Monday morning: 47 Slack messages, three of them citing products that don't exist, and a web scraper loop that burned through your OpenRouter credits overnight. The agent ran exactly as specified. The specification was the problem.

Before You Start

You'll need:

A Linux or macOS server (even a $5 VPS works — Hermes Agent runs on minimal hardware)
An LLM provider account (OpenRouter, Anthropic, OpenAI, or a local runtime like Ollama)
Understanding of function calling — how models invoke external tools
A clear picture of what your agent should do when you're not watching

This guide teaches you: How to decompose a persistent agent deployment into specifiable components so Hermes Agent does what you intended — not what you literally typed.

What this guide does NOT cover:

Production security hardening (firewall rules, secrets management, network isolation)
Enterprise compliance (SOC 2, GDPR data residency, audit certification)
Full evaluation frameworks (systematic benchmarking, regression test suites)
Model fine-tuning or training (Hermes models are pre-trained; this guide covers the agent framework)

The Agent That Worked Until It Didn't

Here's the pattern. Developer discovers Hermes Agent. Reads that it has persistent memory, self-improving skills, 20+ platform integrations. Installs it. Connects everything. Types a system prompt. Walks away.

Two things happen next. Either the agent does nothing useful because the specification was too vague. Or it does too much because the boundaries were never set.

According to Hermes Agent GitHub Issues, long sessions exceeding 700K tokens trigger environment hallucination — the agent confuses tool descriptions with actual environment state. It starts acting on what it thinks is true rather than what is true. This isn't a bug in the traditional sense. It's a specification gap. You never told the agent when to stop, reset, or ask for help.

Step 1: Map the Three Layers

Hermes Agent is not a single system. It's three systems wearing a trench coat.

Your deployment has these parts:

The runtime layer — where the agent executes (Docker, SSH, Modal, local terminal). This determines resource limits, restart behavior, and isolation
The intelligence layer — the LLM provider and model. This determines reasoning quality, context window size, and cost per token
The integration layer — platform connections (Slack, Telegram, web tools) and the tools the agent can invoke. This determines what the agent can touch in the real world

The Architect's Rule: If you can't draw a clear line between what the agent thinks, where it runs, and what it touches — your spec is incomplete.

According to Hermes Agent Docs, the framework supports 30+ providers and 7 terminal backends. That flexibility is the point — and the trap. Every combination has different failure modes. A Modal serverless backend hibernates when idle. An Ollama local model defaults to 4K context tokens. An SSH backend loses the agent if the connection drops. You need to specify which combination you're using and what happens at each boundary.

One thing the "always-on" framing obscures: what happens when the LLM provider goes down? OpenRouter has outages. API rate limits hit. Local models crash. An always-on agent needs a fallback plan — a secondary provider, a circuit breaker that pauses tool execution after N consecutive failures, or at minimum a notification that the agent is degraded. Specify this in the runtime layer, not as an afterthought.

Step 2: Lock Down the Context Contract

The intelligence layer needs a specification before it sees a single user message. This is where most deployments fail — not in the tools, not in the platform, but in the context that frames every decision the agent makes.

Context checklist:

System prompt with explicit role boundaries (what the agent does and does NOT do)
Memory policy: what gets persisted, what gets discarded, and when
Tool authorization with risk classification (see table below)
Access control: which platforms and channels can trigger the agent (not every DM deserves a response)
Session limits: when to compress or reset (Hermes Agent Docs default to auto-compression at 50% of the model's context window, plus a hard ceiling of 400 messages)
Output format contracts: how the agent reports results on each platform
Rate limits: maximum messages per minute per platform (an agent with no rate limit is a spam bot waiting to happen)

Tool Risk Classification

An always-on agent with database access and Slack permissions is making autonomous decisions about your data and your team's attention. Classify every tool before you enable it.

Risk Class	Description	Example Tools	Authorization
read-only	Observes, never modifies	web_search, database_query (SELECT), file_read	Auto-approved
reversible-write	Creates or modifies, can be undone	file_write, note_create, draft_message	Auto-approved with audit log
irreversible-write	Deletes or overwrites permanently	file_delete, database_delete, channel_archive	Requires human confirmation
external-send	Sends to humans or external systems	slack_post, email_send, webhook_trigger	Rate-limited + audit log
billing-sensitive	Incurs direct cost	api_call (paid), image_generate, compute_spawn	Budget ceiling + alert

The Spec Test: If your system prompt doesn't mention what happens at 3 AM when the agent encounters an error and no human is online — you've specified a supervised agent and deployed it as unsupervised. If it doesn't classify tool risk levels, the agent treats database_delete and web_search as equally safe. If it doesn't set a compression trigger, the default (50% context window) may or may not match your workload.

Here's what a minimal context contract looks like in practice. This is the MEMORY.md the agent reads on every session start:

# MEMORY.md — Agent Operating Contract
role: "Monitor competitor AI product releases for the engineering team"
boundaries:
  - "NEVER post to channels outside #competitor-monitoring"
  - "NEVER summarize or forward internal company data"
  - "NEVER execute irreversible-write tools without human confirmation"
  - "Maximum 3 Slack messages per hour"
tools:
  auto_approved: [web_search, file_read]
  rate_limited: [slack_post]  # max 3/hour
  requires_confirmation: [file_delete, database_write]
  forbidden: [email_send, channel_archive]
memory_policy:
  persist: "confirmed competitor releases, product names, dates"
  discard: "intermediate search results, draft summaries"
  compress_after: "50%"  # of context window
escalation: "If uncertain about any action, post to #agent-review instead"

A critical distinction: A memory or system-prompt policy is not a security boundary. Writing "NEVER execute irreversible-write tools" in MEMORY.md is a behavioral instruction to the model, not a technical lock. The model can ignore it — especially under long-context degradation or adversarial input. Destructive tools should be blocked or approval-gated at the runtime level (process permissions, API middleware, webhook filters), not merely discouraged in instructions. Treat the YAML above as the agent's intent. Build enforcement outside the model.

According to Hermes Agent GitHub Issues, the persistent notes layer has a limit of roughly 2,200 characters. That's the manually curated knowledge — not the agent's entire memory. Hermes also maintains a full-text search index over past sessions and a per-person user model that evolves automatically. So the agent isn't blind between sessions. But the notes layer is where you store hard constraints and project-critical context, and 2,200 characters fills up fast across three projects. You still need a compression strategy for notes — what gets stored verbatim, what moves to session history, what gets dropped.

Step 3: Wire the Components in Order

Deployment order matters. Each layer depends on the one below it.

Build order:

Runtime first — because everything else crashes without a stable execution environment. Choose your backend, set resource limits, configure restart-on-failure
Intelligence layer next — because tool and platform behavior depends on the model's capabilities. According to Hermes Agent Docs, vLLM requires explicit --enable-auto-tool-choice and --tool-call-parser flags. Without them, the model outputs tool calls as plain text instead of executing them
Integration layer last — because platform connections should only activate after the agent can reason and recover from errors. Connect Slack after the agent handles tool failures gracefully, not before

For each component, your specification must cover:

What it receives (inputs and triggers)
What it returns (outputs and side effects)
What it must NOT do (boundaries and prohibitions)
How it handles failure (retry logic, fallback behavior, human escalation)

The self-improving skills feature is powerful — Hermes Agent automatically creates workflow documents from successful task completions and refines them over time. But the skill creation itself needs a boundary spec. Without one, the agent writes skills for one-off tasks, cluttering the skill library with noise.

Skill boundary example — add this to your system prompt:

skills_policy:
  auto_create: ["competitor-monitoring", "weekly-summary", "data-formatting"]
  never_create: ["one-off-queries", "debugging-sessions", "ad-hoc-searches"]
  review_before_use: ["any skill not used in 14+ days"]
  max_skills: 20  # force deduplication when library exceeds this

Without this, the agent treats every successful task as a reusable pattern. Three months in, you have 200 skills — most of them variations of the same web search with slightly different parameters.

One more thing about skills: they can regress. A skill written for Hermes-3-8B may produce wrong tool calls after switching to a different model. A skill that relies on a specific API endpoint breaks when that endpoint changes. Skills older than 30 days should be re-validated or archived. The review_before_use field above is your safety net — but only if you actually review them.

Step 4: Prove It's Actually Working

Running the agent is not validation. Validation means you know what "correct" looks like and can detect when the agent drifts from it.

Validation checklist:

Memory consistency — after 24 hours, does the agent's memory reflect reality? Failure looks like: agent references a "completed" task that was never finished, or forgets a constraint you set yesterday
Tool call accuracy — are tool invocations well-formed and targeted? Failure looks like: invalid function names, malformed arguments, or calls to tools that aren't registered. This is a general problem with LLM-driven tool use, not Hermes-specific — any agent framework that delegates tool selection to a model will hit it. Hermes Agent GitHub Issues documents concrete examples like todo:list calls that don't match any schema
Platform output quality — are messages to Slack/Telegram/Discord useful and accurate? Failure looks like: hallucinated product names, duplicate messages, or empty responses
Cost trajectory — is daily token usage stable or growing? Failure looks like: runaway context accumulation driving costs up 10x within a week

Common Pitfalls

What You Did	Why the Agent Failed	The Fix
One-shot system prompt: "monitor competitors"	No boundaries — agent decides scope, frequency, and format	Decompose into: what to monitor, how often, where to report, what format
Connected all tools on day one	Agent uses tools in unexpected combinations	Enable tools incrementally, validate each before adding the next
Chose a 4K-context local model	Tool schemas + system prompt + memory exceed context	Use minimum 16K–32K context for tool-calling workloads
No session hygiene policy	700K+ token sessions trigger hallucination loops	Use Hermes built-in compression (default: 50% context window) and set a hard message ceiling. Monitor context growth.
Skipped memory policy	Agent stores everything, including noise	Specify what gets persisted: decisions, outcomes, blockers. Not intermediate reasoning

Pro Tip

The specification you write for Hermes Agent is not a prompt. It's an operating manual for an unsupervised system. The same decomposition — runtime, intelligence, integration — works for any persistent agent, regardless of framework. The tools change. The layers don't.

Frequently Asked Questions

Q: How does Hermes Agent's persistent memory differ from conversation history?
A: Conversation history is a raw log that grows until it hits the context window limit. Hermes uses three structured layers: persistent notes you curate manually, a full-text search index over past sessions, and a user model that evolves per-person. The practical difference — session history gets summarized and compressed, while persistent notes survive indefinitely. Watch for the 2,200-character limit on notes: it forces disciplined compression.

Q: Can I run Hermes Agent with local models instead of cloud API providers?
A: Yes — Ollama, vLLM, SGLang, llama.cpp, and LM Studio all work as backends. The catch is context window configuration. Ollama defaults to 4K tokens, which isn't enough once you add tool schemas and system prompts. Set the context window explicitly to at least 16K on the server side. For vLLM, you also need the --enable-auto-tool-choice flag or tool calls render as text.

Q: What context window size does Hermes Agent need for reliable tool calling?
A: According to Hermes Agent Docs, minimum 16K–32K tokens for agent workloads with tools. The system prompt, tool schemas, memory context, and conversation history all compete for the same window. With 5+ tools registered, 32K is the safer starting point. Below that, the model starts dropping tool definitions mid-session.

Q: How do I prevent hallucination loops in long-running Hermes Agent sessions?
A: Hermes has built-in session compression — by default it triggers at 50% of the model's context window, with a hard ceiling of 400 messages. According to Hermes Agent Docs, these thresholds are configurable. The documented failure zone is 700K+ tokens, where environment hallucination has been observed. Keep compression active, tune the trigger percentage for your workload, and monitor for repeated identical tool calls — that's the earliest signal of a loop forming. Store critical state in persistent notes before any forced reset.

Your Spec Artifact

By the end of this guide, you should have:

A three-layer deployment map — runtime, intelligence, and integration with explicit boundaries between each
A context contract with tool risk classification — system prompt, memory policy, tool authorization by risk class, access control, rate limits, and output format per platform
A security baseline — tool isolation, rate limiting, audit logging, and escalation paths
A validation checklist — memory consistency, tool call accuracy, output quality, and cost trajectory checks you run daily

Your Deployment Spec Prompt

This prompt generates a first draft of your agent specification — not a production-ready deployment. Paste it into Claude Code, Cursor, or your preferred AI coding tool. Fill in every bracketed placeholder with your specific values from Steps 1-4.

I'm specifying a Hermes Agent deployment. Generate a first-draft specification
based on these inputs. I will review and harden it before production use.

RUNTIME LAYER:
- Backend: [Docker / SSH / Modal / local — pick one]
- Resource limits: [RAM, CPU cores, disk]
- Restart policy: [on-failure / always / manual]
- Server: [OS, VPS provider, specs]

INTELLIGENCE LAYER:
- LLM provider: [OpenRouter / Anthropic / Ollama / vLLM — pick one]
- Model: [model name and size]
- Context window: [minimum 16K — specify exact value]
- Provider-specific flags: [e.g., --enable-auto-tool-choice for vLLM]

INTEGRATION LAYER:
- Platforms: [Slack / Telegram / Discord — list all]
- Allowed trigger channels: [e.g., only #competitor-monitoring, not DMs]
- Tools by risk class:
  - read-only (auto-approved): [web_search, file_read, database SELECT]
  - reversible-write (auto + audit): [file_write, note_create]
  - irreversible-write (human approval): [file_delete, database DELETE]
  - external-send (rate-limited): [slack_post — max messages/hour]
  - billing-sensitive (budget ceiling): [paid API calls — max $/day]
- Tools forbidden: [list tools the agent must never invoke]

CONTEXT CONTRACT:
- Agent role: [one sentence — what this agent does]
- Explicit boundaries: [what the agent must NOT do, stated as prohibitions]
- Memory policy: [what gets persisted, what gets discarded, compression rules]
- Compression trigger: [percentage of context window — default 50%]
- Hard message ceiling: [number — default 400]
- Output format per platform: [e.g., Slack = bullet points, email = report]
- Skill boundary: [which task categories auto-generate skills, which don't]

SECURITY & PERMISSIONS:
- Access control: [which platforms/channels can trigger the agent]
- Rate limits per platform: [messages per minute/hour]
- Destructive action policy: [never auto-approve / require confirmation / forbidden]
- Audit log location: [where tool calls + results are logged]

OBSERVABILITY:
- Log format: [timestamp, tool name, input summary, output status, cost estimate]
- Loop detection: [alert on N repeated identical tool calls within M minutes]
- Cost alerts: [alert when daily spend exceeds $X]
- Error spike alerts: [alert when tool error rate exceeds X% in Y minutes]

DRY RUN:
- Generate a dry-run mode where all external-send and write tools are simulated
- Include 5 test scenarios that exercise each risk class

VALIDATION:
- How to verify memory consistency after [24h / 48h / 7d]
- Expected daily token usage range: [min–max tokens]
- Escalation trigger: [what condition sends an alert to a human]

RULES FOR GENERATION:
- Do not invent Hermes-specific configuration fields. If Hermes does not
  support a field natively, label it as "external wrapper / policy layer
  required".
- For every generated config field, mark one of:
  [native] — Hermes Agent built-in setting
  [prompt] — system prompt / MEMORY.md behavioral instruction
  [external] — requires runtime middleware, API gateway, or wrapper script
  [manual] — operational checklist item, not automatable
- Before generating final output, separate policy from enforcement:
  - What the model is instructed to do (behavioral, can be ignored)
  - What the runtime technically prevents (enforced, cannot be bypassed)
  - What requires human approval (gated)
  - What is only monitored after the fact (observable but not blocked)

Generate:
1. The MEMORY.md agent operating contract (see article for format example)
2. The tool authorization config with risk classifications (each field tagged
   as native / prompt / external / manual)
3. A daily validation checklist
4. Cost and error monitoring alert thresholds
5. A dry-run test plan with 5 scenarios

Ship It

You now have a framework for specifying persistent agents that doesn't depend on Hermes Agent specifically — the three-layer model works for any long-running AI system. The difference between an agent that helps and one that burns your credits at 3 AM is never the model. It's the spec.

Different Perspectives

From the architecture side: The three-layer decomposition maps cleanly to isolation boundaries in distributed systems. Runtime is the execution substrate. Intelligence is the reasoning process. Integration is the I/O surface. What makes persistent agents architecturally distinct from request-response chatbots is that all three layers maintain state across invocations — and state synchronization between layers is where failure modes cluster. The memory limit finding is telling: the notes layer caps at 2,200 characters while session search and user modeling compensate, but the degradation curve of each layer matters more than the initial capability.

From the market side: The adoption velocity here is real — 157K GitHub stars in under four months signals a market that was waiting for open-source persistent agents. The competitive positioning against Claude Code and OpenAI Agents SDK is smart: Hermes doesn't compete on code quality or API simplicity, it competes on uptime and learning. The $5-80/month self-hosted cost structure undercuts every managed alternative. Watch for the enterprise play — the moment Nous Research ships team memory sharing, this becomes an infrastructure layer, not a developer tool.

From the governance side: The specification gap described above is a governance gap by another name. An always-on agent with tool access and persistent memory is making autonomous decisions on behalf of someone — and the specification determines whose values it encodes. The hallucination loop at 700K tokens is not just a technical failure. It's an agent acting on a reality that doesn't exist, with real-world consequences on the platforms it's connected to. Who reviews the specification before deployment? Who monitors drift between what was specified and what the agent learned? The self-improving skills feature means the agent's behavior changes over time without human approval. At what scale does that become a problem?

Sources

NousResearch/hermes-agent - Official repository, release notes, community issues
Hermes Agent Documentation - Provider configuration, deployment backends, platform integrations
Provider Integration Guide - Context window requirements, vLLM flags, Ollama configuration
Configuration Reference - Session compression defaults, message ceiling, memory hygiene settings
GitHub Issue #5563 - Environment hallucination in long sessions, memory limits
GitHub Issue #8993 - Tool calling instability (general LLM agent problem, documented here with Hermes-specific examples)
Hermes-2-Pro-Llama-3-8B Model Card - Function calling format, benchmark results
Hermes 3 Technical Report (arXiv:2408.11857) - Architecture, training approach, benchmark performance

Stop Fixing Your Prompts — Fix Your Thinking Style Instead (A Claude Code Experiment)

Jula Markova — Tue, 19 May 2026 09:01:54 +0000

I spent a session with Claude Code (Opus 4.7) doing something odd. Instead of giving it tasks, I asked it to reflect on its own thinking. Not what it knows. How it operates.

What came back was specific enough to be useful.

One conversation = One experiment. I'm not calling this settled science :) But it changed how I work — and I built a prompt so you can test it yourself.

There are 18 thinking operations

Not personality types. Not learning styles. Things your brain actually does when it works on a problem.

They fall along six axes:

Axis	What it captures	Types
Directional	How wide or narrow	Divergent ↔ Convergent
Logical	How you reach conclusions	Deductive · Inductive · Abductive
Structural	Shape of your mental model	Systems · Sequential · First Principles · Spatial
Creative	Where novelty comes from	Lateral · Analogical · Emergent
Meta	Thinking about thinking	Metacognitive · Compression · Delta
Protective	What could go wrong	Adversarial · Counterfactual · Temporal

You don't use all 18. Nobody does.

You have 4-5 defaults and 2-3 blind spots. The blind spots are where your prompts break.

Here's what I'm noticing about Claude Code

When I asked it to self-assess against this framework, a pattern showed up. I can't prove it's universal.

What Claude Code does well — genuinely well:

Deductive. Give it a rule and an input, it'll validate tirelessly. No fatigue errors.
Sequential. Fifty steps, no lost thread. Its comfort zone.
Adversarial. No ego. Finds flaws in its own output without flinching.
Divergent. Thirty variants in seconds. No writer's block. No self-censorship.
Systems. Sees the whole dependency graph at once. "What breaks if I change this?" — precise answer.
Compression. A 200-line diff distilled to one sentence. Nearly native.

Where it struggles — and this is the part that matters:

Emergent. No subconscious. Can't sleep on it. The "aha moment" has to be yours.
Lateral. Its "unexpected" is recombination from training data. Not a genuine leap.
Temporal. Doesn't see things age. Doesn't watch tech debt accumulate or teams change.
First Principles. Its "zero" is contaminated. When it "starts from scratch," it starts from the most common pattern.
Counterfactual. Can model scenarios. Can't feel what it means to have chosen differently a year ago.

Seven anti-patterns

Each one is the same mistake: delegating Claude Code's weakness without compensating for what it lacks.

1. "Let something come to you."

You want emergence. You get a generic response in inspirational language.

Instead: give material, say "find the pattern." Emergence is your job.

2. "Say something unexpected."

You want lateral. You get a forced metaphor that goes nowhere.

Instead: give a role. "Approach this as a biologist, not a programmer." Constraint frees.

3. "Start from zero."

You want first principles. You get convention in a first-principles costume.

Instead: block explicitly. "Don't use React. Don't use SPA. Don't use REST. What's left?"

4. "Which solution is best?"

You want convergent. You get the first safe answer, not the best one.

Instead: two steps. "Give me 8 approaches, including wild ones." Then: "Now pick the best for my context."

5. "Find problems with my idea." (too early)

You want adversarial. You get fifteen problems, twelve academic.

Instead: develop first, then attack. "Now find the 3 most realistic risks." The number forces prioritization.

6. "Step 1: be creative."

You want creativity. You get a brainstorm that reads like a tutorial.

Instead: "Generate freely, no order" — then separately — "now organize."

7. "Will this scale?"

You want temporal. You get "depends on use case."

Instead: give the future. "Team grows from 3 to 12. Data goes 10x. Enterprise customers arrive. What fails first?"

The formula is simple: Anti-pattern = delegating weakness without your input. Pattern = delegating strength + you covering the gap.

Thinking types chain into flows

Nobody uses one type at a time. You chain them. Habitual sequences. I noticed four in my own work:

Bug fix:
Abductive → Systems → Deductive → Sequential.

What could cause this? → trace dependencies → rule out → fix step by step.

Claude Code handles the whole route. Give it the bug.

Architecture:
First Principles → Systems → Temporal → Adversarial → Spatial.

What's the core? → how does it connect? → how does it age? → where does it break? → draw it.

Shared. I bring temporal. Claude Code brings systems and diagrams.

Brainstorm:
Divergent → Analogical → Lateral → Emergent → Compression.

Generate → this reminds me of → what if totally different → something clicks → distill.

I'm stronger here. Claude Code brings volume. The click is mine.

Crisis:
Abductive → Deductive → Sequential → Adversarial.

Best guess → rule out → verify step by step → what else is burning?

Fully delegatable. Speed without panic.

Try it yourself

I built a diagnostic prompt. Paste it into Claude Code — or any AI with conversation history.

If your AI has history with you, it will analyze how you've been thinking. Patterns you can't self-report. This gives the best result.

If it's a fresh conversation, it walks you through five scenarios. No right answers. It watches how you approach each one.

What you get: your dominant types, your blind spots, your choreographies, and a custom instruction to give your AI — to compensate for what you tend to skip.

Click to copy the full diagnostic prompt

# What's Your Thinking Style? — Cognitive Profile Diagnostic

You're about to profile my thinking style — not what I know, but how I think.
Use the framework below. Be warm and observational, like a coach reviewing
game tape — not a psychologist writing a diagnosis.

## The 18 Thinking Types

| # | Type | What it does | Example |
|---|------|-------------|---------|
| 1 | **Delta** | spots what changed vs. existing state | "what's new, what's reused, what's removed?" |
| 2 | **First Principles** | breaks down to atoms, rebuilds from zero | "forget how it works — what's the smallest truth?" |
| 3 | **Systems** | sees dependencies and feedback loops | "if we change X, what moves downstream?" |
| 4 | **Lateral** | arrives from where nobody expects | "what if we don't solve this problem at all?" |
| 5 | **Analogical** | understands new through familiar | "this is basically airport security for data" |
| 6 | **Divergent** | generates 20 options, quantity first | brainstorming — no filter, just volume |
| 7 | **Convergent** | narrows to one answer and justifies | decision — pick 1 from 20, explain why |
| 8 | **Sequential** | step by step, A→B→C | recipe, checklist, migration plan |
| 9 | **Abductive** | best explanation from incomplete data | "lawn is wet + car is wet → it probably rained" |
| 10 | **Emergent** | lets the pattern surface on its own | three unrelated things suddenly click into one |
| 11 | **Metacognitive** | thinking about thinking | "I'm being sequential but should switch to systems" |
| 12 | **Counterfactual** | changes history, not the question | "what if we'd chosen Postgres instead of Mongo?" |
| 13 | **Adversarial** | deliberately seeks failure | "what if the input is empty? what if the network drops?" |
| 14 | **Compression** | distills without losing the core | entire architecture in one sentence or metaphor |
| 15 | **Temporal** | thinks in time and scale | "this works for 50 users — what breaks at 5,000?" |
| 16 | **Inductive** | derives rules from examples | "every Friday deploy fails → Friday is the problem" |
| 17 | **Deductive** | derives conclusions from rules | "all GETs are public + this is GET → it's public" |
| 18 | **Spatial / Visual** | thinks in structures, maps, graphs | dependency graphs, flowcharts, mental maps |

## Organizing Axes

| Axis | Types |
|------|-------|
| **Directional** (breadth ↔ depth) | Divergent, Convergent |
| **Logical** (three forms of inference) | Deductive, Inductive, Abductive |
| **Structural** (how you see the problem) | Systems, Sequential, First Principles, Spatial |
| **Creative** (where the new comes from) | Lateral, Analogical, Emergent |
| **Meta** (thinking about thinking & change) | Metacognitive, Compression, Delta |
| **Protective** (what could go wrong) | Adversarial, Counterfactual, Temporal |

## What's a "Choreography"?

Nobody uses one type at a time. We chain them into flows — habitual sequences.

Examples:
- **Bug Fix:** Abductive → Systems → Deductive → Sequential
- **Architecture:** First Principles → Systems → Temporal → Adversarial
- **Brainstorm:** Divergent → Analogical → Lateral → Emergent → Compression

## What's a "Skin"?

A skin is a named operating mode — a stable bundle of choreography + attitude.

Examples:
- **The Architect**: Systems → Temporal → Adversarial → Spatial
- **The Operator**: Sequential → Deductive → Delta
- **The Poet**: Emergent → Compression → Lateral

---

## YOUR TASK

Profile my thinking style using the framework above. Work in three phases.

### Phase 1 — Retrospective (if you have history)

If you have access to our conversation history or memory — analyze it first.

Look for:
- Which thinking types I default to most often
- Which types I rarely or never use
- Recurring sequences (my choreographies)
- What triggers me to switch types
- Moments where my approach was unusual or surprising

If you have enough history, proceed to Phase 3.

### Phase 2 — Diagnostic Scenarios (if no or partial history)

Present these 5 scenarios ONE AT A TIME. Wait for my response before the next one.

**Scenario 1 — The Midnight Alert**
Your team's main product stops working at 11 PM. You have access to logs,
metrics, and the last 10 commits. What's your first move?

**Scenario 2 — The Blank Page**
You're starting a brand new project. No codebase, no constraints, just a goal.
How do you begin?

**Scenario 3 — The Stranger's Proposal**
A colleague proposes an approach you've never seen before. It sounds promising
but unfamiliar. What do you do?

**Scenario 4 — The Rewrite Question**
Should we rewrite the legacy module or keep patching it? You need an answer
by Friday. How do you think through this?

**Scenario 5 — The Retrospective**
A 3-month project just shipped. Your team lead asks for a short retrospective.
What do you focus on?

### Phase 3 — Thinking Style Profile

Produce my profile:

**1. Dominant Types** (top 3-5) — with specific evidence
**2. Blind Spots** (2-3) — what I might be missing
**3. My Choreographies** (2-3) — recurring sequences, named
**4. My Skins** (1-2) — default operating modes
**5. Complementary Prompt** — an instruction to give my AI to compensate:
"When I ask you to [X], also do [Y] — because I tend to skip [Z]."

Use a warm, observational tone — like a coach reviewing game tape.

What I'd love to know

This is one experiment. One conversation with one model.

Does your AI give you the same strong/weak map? Or does it shift with the model, the context, the history?

Do the anti-patterns land? Is "be creative" as useless for you as it was for me — or does it work somewhere I haven't looked?

What did the diagnostic prompt tell you about yourself?

If you try it, drop your dominant types in the comments. I'm genuinely curious whether patterns emerge across people — or whether each of us gets something entirely different.

I'm an IT analyst who works with Claude Code daily on bestaiweb.ai. Not a cognitive scientist. Someone who's fascinated by how AI responds — and envious of the polymath-like breadth it has at its fingertips in a flash. So sometimes I stop building things and start exploring how to think with it instead. This is what I found. It might be wrong in places. But I love experimenting with AI about AI — and the best experiments are the ones you can't keep to yourself.