DEV Community: cucoleadan

How Pangram Decides What Looks Like AI

cucoleadan — Tue, 28 Jul 2026 12:37:08 +0000

This was originally published on All Agents Considered.

I have no doubt you already know that last week Substack just attached an AI detector to every article on the platform.

The new “Scan for AI” feature lets readers analyze a post, Note, comment, or reply with Pangram, which estimates how much of the writing is human, AI-assisted or written by AI.

And the percentage it returns might looks precise, like a speed camera telling you exactly how fast you were going.

But Pangram did not watch the person type, check their browsing history, or find a hidden signature left by ChatGPT or Claude. It looked at the finished text and decided what the writing resembled.

That distinction sits at the center of the argument happening across the Substack, professor, and AI research subreddits.

Some people think these detectors generate little more than a random number. Others treat the score like a confession.

Both sides are missing the point.

Pangram is pretty good at identifying obvious AI writing, and it has more behind it than the early tools that made AI detection a laughing stock. In independent testing from the University of Chicago, it identified writing from widely available AI models with low error rates under the conditions studied.

But it is not perfect. Some model output and repeated paraphrasing can evade it, and human writing can still be classified incorrectly.

A low AI score does not prove that no AI was used, just as a high score does not prove an algo wrote the text. Pangram is a serious pattern detector, not a yes-or-no authorship test. It cannot magically settle the question of authorship. So today I am going a bit out of the way with the main topics of AAC to show you the truth behind Pangram and AI detectors.

In this edition I’ll cover how Pangram is trained, what its score measures, why it differs from older detectors, and where it fails, without pretending the tool can prove with 100% certainty who wrote a piece a text.

In this article:

Why older AI detectors earned so much distrust
How Pangram learns to identify AI patterns
What its percentage does and does not mean
The strongest evidence that it works
The known ways it can fail
How to use a result without accusing the wrong person

Why Everyone Hates AI Detectors

Early AI detectors often leaned heavily on two ideas called perplexity (not the tool) and burstiness.

Perplexity asks how surprised a language model is by the next word. AIs are trained to predict the next word, that’s literally written in the DNA of all LLMs. On the other hand, our human writing patterns are less predictable just because our writing tends to be messier.

Burstiness asks whether sentence lengths and structures vary. Humans may write one clipped sentence, then an overgrown paragraph, followed by a shorter sentences. AI models usually settle into a steady rhythm (like the Rule of 3’s).

You don’t need me to point out that these “AI detection” techniques are fundamentally flawed. Imagine judging whether bread came from a factory by checking whether every slice has the same thickness. That clue may help in some cases, but it doesn’t tell you exactly who baked the loaf.

A safety manual is predictable because it needs to be. A legal form repeats itself because consistency matters. A non-native writer may use simpler sentence patterns.

A poet may produce language so unusual that the detector has little familiar ground beneath it.

This is why old detectors produced absurd headlines. In 2023, Ars Technica reported that one detector labeled part of the US Constitution as likely AI-generated. A 2023 academic review of detection tools also found serious reliability problems, especially after text was edited or paraphrased. Pangram argues that those two measures alone fail as a basis for detection.

Thus an online myth was born: AI detectors don’t work.

Before we go into why Pangram is truly different, we really have to get this out of the way. Pangram is not the only trained detector in 2026. GPTZero describes a multiclass model for human, mixed, polished, generated, and paraphrased text. Originality says its current detector uses a trained transformer model built from human and generated examples.

Pangram Does Not Search for an Em Dash

Pangram is a trained classifier.

Researchers train it on many documents labeled by source: human-written or AI-generated. From those examples, the detector learns which combinations of patterns tend to distinguish one group from the other.

Think of it like teaching someone to recognize counterfeit passports rather than giving them a checklist of what a document should contain. Side note: this reminded me of the game Papers, Please.

Pangram says its flagship detector uses a transformer model adapted to classify sequences of text. Its technical explanation and original research paper describe a system that learns from human writing and model-generated writing. Basically they created an AI that can detect AIs.

It does not call OpenAI to ask whether ChatGPT wrote the paragraph. It does not recover the prompt. It does not compare the post with a giant database of every answer an AI company has produced. There’s literally no need for that.

Pangram learns what different kinds of writing tend to look like, then it analyzes the text you feed it against that map.

This should also clear up the em dash frenzy.

Pangram can show human-readable supporting clues such as em dashes, lists, headings, stock phrases, Markdown, and groups of. But Pangram says those visible clues are separate from the flagship detector. They help a reader inspect the text and show that they are not the inputs driving the main model.

To better put this into perspective, imagine a prisoner just escaped from incarceration. Deleting every em dash is like removing their striped shirt before a facial-recognition scan. The prisoner changed something visible without necessarily changing what the system recognizes.

Pangram Trains on Twins

The most interesting difference is how Pangram constructs its training set.

Suppose we trained a detector using personal diaries as the human examples and software landing pages as the AI examples. It might appear accurate while learning the wrong lesson. Instead of detecting AI, it could learn that feelings are human and product features are synthetic.

Pangram tries to prevent that shortcut with what it calls synthetic mirroring.

For each human document, it creates an AI counterpart about the same subject, in a similar tone and style, carrying similar information. A human book review is compared with an AI book review of the same material. A human essay is paired with its AI twin.

The detector must find differences between twins rather than compare a shopping list with a sales page.

Pangram’s model card says its human training material covers essays, reviews, books, creative writing, news, scientific papers, Wikipedia, and general web text. Its AI side is generated in-house to create those matched comparisons. Then it makes the exam even harder.

Pangram scans large collections of known human writing and looks for passages the detector wrongly calls AI. Researchers add those mistakes back into the training data, create AI mirrors for them, and train again.

This technique is called hard-negative mining.

There’s a famous story from World War II based on the statistician Abraham Wald. The military engineers were asked to examine aircraft that returned from battle and map where they had been hit. In most cases, the wings and fuselage were covered in bullet holes, so the obvious response was to reinforce those areas.

Wald argued that the military was looking only at surviving aircraft, but if those planes could return despite being hit in the wings and fuselage, then those areas were not the most vulnerable. That’s how he figured out the places with few bullet holes (like the engines) needed more protection.

Hard-negative mining follows a similar principle. The most useful evidence often comes from the cases a system mishandles. So instead of repeatedly training a detector on obvious AI writing, researchers study the difficult human passages Pangram (initially) falsely identified as AI and feed those mistakes back into the training process.

Pangram’s Open Model Is Not the Substack Model

Pangram has released public code, data links, and model weights through its EditLens repository. The project accompanies a paper accepted at ICLR 2026 on measuring degrees of AI editing.

That is meaningful transparency. It lets researchers inspect and reproduce part of the method without trusting a marketing page.

What it doesn’t reveal is the complete system running inside Pangram’s commercial product.

The public repository contains two smaller research models and instructions for training them. Pangram’s Open Pangram announcement explicitly says the release is a research baseline and should not be used to enforce AI policies in schools or workplaces.

The current commercial model described by Pangram is version 3.3. Its model card documents a larger production system with later training and scanning changes. Substack publicly confirms the Pangram integration, but its help page does not name the exact model version or threshold it uses.

What the Percentage Means

We can all agree that teal writing has made binary detection obsolete.

A writer may create the first draft, ask Claude to tighten one section, run the result through Grammarly, dictate a new ending, and then rewrite every sentence by hand. Which words belong to whom?

Pangram’s EditLens research tries to measure that grey area. Researchers begin with human text, ask models to edit it at different intensities, and compare each edited version with the original. A classifier then learns to estimate the degree of editing from the final text.

The full EditLens paper states the important limitation of its method. When the detector checks your article, it doesn’t receive your original draft. It only sees the final text.

The estimate is like a restorer examining a painting and guessing how much has been retouched. They may recognize signs of later work, but they can’t count the extra brushstrokes.

For long documents, Pangram breaks the text into overlapping windows, checks each window, then makes a finer pass near uncertain boundaries. Its 3.2 model card describes an approximate resolution of 50 words.

Pangram can then describe shares of a document as Human, Lightly AI-assisted, Moderately AI-assisted, or AI. A result such as 30% AI refers to the share of classified text segments assigned to that category.

It does not mean:

There is a 30% chance the writer used AI
A machine typed exactly 30% of the words
Pangram found the original prompt
The writer contributed only 70% of the thinking

Ideas, wording, structure, research, and final judgment are different contributions. No detector can reduce all of them to one authorship percentage by reading the final copy.

This is where Substack’s interface creates a social problem. A percentage looks like a measurement, but underneath it’s just a classification.

The Evidence That Pangram Works

The strongest independent evidence I found comes from the 2025 University of Chicago working paper Artificial Writing and Automated Detection.

The researchers gathered 1,992 verified human passages across blogs, novels, résumés, essays, business writing, and other genres. They produced matched AI versions using GPT-4.1, Claude Opus 4, Claude Sonnet 4, and Gemini 2.0 Flash.

On medium and long passages in that test, Pangram produced essentially zero false positives and false negatives. It outperformed GPTZero, OriginalityAI, and an open RoBERTa detector in the same experiment.

That should clear up the myth that “AI detectors never work”.

But this should also be taken with a grain of salt. The paper was, after all, a working paper. The test used a controlled set of matched human and synthetic documents. It evaluated the Pangram service available in May 2025 and it did not test every Substack voice, mixed editing workflow, new model, humanizer, or unusual format.

Pangram’s current numbers come from Pangram. Its 3.3 model card reports a 0.01% false-positive rate for long-form creative writing, 0.02% for academic writing, 0.04% for multilingual how-to articles, and 0.49% for poetry.

It also reports a 1.5% false-negative rate on a random Chatbot Arena set.

Those are impressive company-reported results, but they shouldn’t be treated as promises about the next text you scan.

The clearest warning comes from a competing test. In February 2026, GPTZero published its own benchmark and reported that GPTZero 4.3b beat Pangram 3.2 on its datasets.

That does not prove GPTZero is better. It proves the winner changes with the test set, model version, threshold, and person running the test.

I made the same argument in my breakdown of why AI benchmarks fail real agent workflows. A leaderboard compresses the conditions into one number, but conditions change and can be picked arbitrarily.

Where Pangram Messes Up

Pangram’s own documentation says it works best on long-form prose written in complete sentences, like a Substack post.

Its 3.3 limitations warn about bullet lists, technical instructions, tables of contents, references, templates, equations, headers, and footers. Poetry has a much higher reported false-positive rate than long creative prose.

Short text creates another problem. Just as a smoke detector cannot identify much from a few molecules in the air, Pangram can not correctly classify a short text.

Then there are false negatives.

A May 2026 preprint titled Base Models Look Human To AI Detectors found that output from base language models often looked human to both Pangram and GPTZero. Instruction-tuned chat models were easier to identify than base models. Researchers also used repeated paraphrasing to make AI text appear more human to the detectors.

That suggests detectors may be especially good at recognizing the habits produced by chat-model training, rather than detecting one universal substance called “AI writing.”

The Atlantic found another crack. A reporter sent ChatGPT and Claude output through the Walter Writes humanizer, then watched Pangram label the result human. Pangram has since described better humanizer detection in version 3.3, so one test does not freeze the system forever, but it does show the shape of the fight.

The detector learns the disguise, and the disguise keeps changing. It’s just like with police figuring out a new way to transport illicit substances, then traffickers come up with something new and the wheel keeps spinning.

This arms race creates an important asymmetry:

A result of “AI” can be a false accusation
A result of “Human” can be successfully disguised AI

Neither side provides proof.

Pangram CEO Max Spero told The Atlantic the detector should never be the final arbiter. The company setting limits on its own tool carries more weight than a critic attacking it from outside.

How to Read a Pangram Result

A Pangram result is closer to weather radar than a fingerprint.

Radar detects conditions associated with a storm. It can tell you when the signal is strong, where to look, and whether you should carry an umbrella. A fingerprint connects a person to an object through physical evidence, and Pangram has no such access to the writing process.

I also don’t use Pangram as a moral scorecard. I write about technical subjects, and I use AI heavily to structure my thoughts and explain difficult concepts to less technical readers. My first question when I read something is much simpler: Did I get anything valuable from it?

If an article teaches me something new, gives me a useful idea, or makes a complicated subject easier to understand, I’m satisfied with the time I spent reading it; whether AI helped produce it comes second.

I take the same approach to Duolingo, where I’m currently learning German. Duolingo has openly described using generative AI to create and validate course content. I also know that more than 1,000 volunteers helped create dozens of its original language courses.

I have serious ethical problems with that history. I personally believe that Duolingo owes an enormous debt to the community that helped build it, and embracing automation does not erase that debt. But I cannot reverse the company’s decisions, and I still want to learn German, so I keep on using the app and embrace the technology for what it can do.

That is roughly how I approach AI-assisted writing. I care about the quality of the thinking, the usefulness of the result, and whether the article rewards my attention. A detector score may tell me something about how the prose was produced, but it cannot decide whether the prose was worth reading.

With that in mind, I use four rules when reading a Pangram result:

Check the material. Is it long-form prose, or a short Note, poem, list, technical manual, reference section, or template? Pangram’s own model card says the detector is intended for long-form writing in complete sentences and warns that lists, instructions, reference sections, templates, and dense equations are more susceptible to false positives.
Read the label as resemblance. Pangram is a trained classifier. It studies patterns in human and AI writing and estimates which group a new passage most closely resembles. Pangram describes the process as turning the text into a numerical representation and passing it through a classifier. It does not observe who typed the words or watch the document being created.
Do not mistake the score for a reconstruction of the writing process. A high result does not tell me which ideas came from a model, how much work the writer contributed, or whether AI only helped with editing. A low result does not prove that no AI was involved. The detector sees the finished prose, not the conversation, research, prompting, or revision behind it. This is also why my best agent workflow is mostly files. Files preserve the trail between research, drafting, and editing, even when AI is part of the process.
Ask what question I actually want answered. If I want to know whether the prose resembles AI-generated writing, Pangram provides a useful signal. If I want to know whether the article is accurate, original, insightful, or worth reading, I have to judge those things myself. A strong idea does not become worthless because AI helped express it, just as a completely human-written article does not become valuable merely because a person typed every word.

I apply the same discipline when AI gives me a technical answer because accepting AI’s first answer without checking it replaces judgment with convenience. I do not reject an answer because a model produced it, and I do not trust it merely because it sounds polished. I ask whether it is accurate, useful, and worth my attention.

A Pangram result can describe patterns in the writing, but it cannot tell me whether I learned something. That would be dystopian.

My Verdict On Pangram

Pangram is different from the crude detectors people remember.

Its classifier learns from paired human and AI documents.

Its synthetic mirrors reduce topic shortcuts.
Its hard-negative mining feeds human false positives back into training.
Its calibration favors avoiding false accusations.
EditLens attempts to model mixed editing rather than forcing every document into a human-or-machine box.

Independent researchers found that this approach performed extremely well on their tested medium and long passages.

Those are facts.

Pangram still sees only the finished text. It cannot observe the actual writing process.

Its published research does not reveal every part of the production system, and its accuracy varies by domain. Short and formulaic formats remain particularly difficult.

New model types and paraphrasing can also evade detection. Not to mention independent benchmarks do not always agree with Pangram’s own results.

And even the percentage can be misleading. Readers may mistake it for a probability or a record of the writing process, but it is neither.

Those limitations are facts too.

The reasonable conclusion lies somewhere between “all detectors are useless” and “the algorithm knows the absolute truth”.

Pangram has earned the right to be treated as evidence, but it hasn’t earned the right to be treated as a verdict.

[Boost]

cucoleadan — Wed, 22 Jul 2026 13:12:41 +0000

cucoleadan

Jul 22

How I Built A Substack API With Hermes And Codex

#agents #substack #codex #npm

11 min read

How I Built A Substack API With Hermes And Codex

cucoleadan — Tue, 21 Jul 2026 12:53:31 +0000

This was originally published on All Agents Considered.

By the middle of June, I had dozens of Substack Notes sitting in markdown files with dates and publishing times. I had rewritten the awkward ones, removed repeated ideas, and mixed the topics so two similar Notes wouldn’t appear one after another.

But there was that one task I kept postponing in Asana: Finish Notes Scheduled Setup.

Scheduling a single Note inside Substack takes about 30 seconds. Keeping several days’ worth of posts organized turns into admin work fast, especially when one edit changes the order and every publishing time after it.

On June 12, I asked Hermes to find an old Substack tool installed somewhere in my Projects folder. Other software used it to talk to my Substack account, and I wanted to know whether it supported scheduled Notes. I expected a quick yes or no, followed by a small fix if I got lucky.

Hermes found the forgotten package beside old test files and a database from my Notes analysis project. Publishing worked. Scheduling didn’t.

One missing feature pulled me much further than expected. A few weeks later, I had published the Unofficial Substack SDK, a reusable Substack toolkit for other apps. Five versions shipped in three days, followed by 708 downloads during its first week.

I use two agents to run my workflows, and this project showed me why the split matters. One agent handling every stage carries its early assumptions all the way to the finish line.

Hermes handled the messy beginning, pulling context from old conversations, forgotten files, licences, and competing projects. Codex stepped in once the work fit inside one project folder and I knew what a successful result looked like.

In today’s edition, I’ll walk you through how I built the SDK and show why the handoff between Hermes and Codex mattered more than either agent working alone.

In this piece

How Hermes turned a half-remembered package into a real starting point
Why Codex became more useful once the job got smaller
How one wrong field name exposed the danger of plausible AI answers
A handoff format for splitting your next build without losing context

The Package I Forgot

I remembered using the old tool to collect Substack data, though I had forgotten its name, where it lived, and how much of it still worked. Hermes identified it as substack-api and traced it back to an open-source project.

Reading profiles, posts, comments, and Notes already worked. Publishing worked too. Scheduling notes support was missing.

Hermes searched the project, looked through public clues left in Substack’s website, and checked similar tools. Instructions for publishing a Note were already known, while scheduling remained undocumented. Nobody had confirmed the label Substack expected for the publishing time.

Hermes proposed scheduled_at. Plenty of online services use a label like this, and it looked completely at home beside the existing instructions. Hermes still warned me about the missing proof and listed other names Substack might expect.

I nearly ignored the warning because the feature looked finished. Doing so would’ve left me with a neat scheduling button sending a label Substack never promised to read.

Version 0.2.2 uses trigger_at. An automated check reads the exact instruction sent to Substack and fails if the label changes. Proof now lives outside the AI conversation.

Getting the first field wrong became the most useful part of the build because it forced me to separate finding an answer from earning trust in it.

Hermes Found The Ground

Starting from scratch would’ve been wasteful. Jakub Slys had already spent years learning how Substack’s private machinery worked, and his project carried working instructions plus a large set of checks.

Hermes traced the installed tool back to his GitHub project, checked the MIT licence, and kept the original credit. His research remains credited in the finished project.

From there, Hermes compared the other available projects instead of treating the first one as the automatic winner. Some were ready-made services for apps and AI agents. Others only read public posts. I wanted a small building block speaking directly to Substack without sending a user’s login through somebody else’s server.

An early experiment turned the inherited project into a small online service running on Cloudflare. Hermes removed parts tied to a traditional server and locked it to read-only access. All 230 automated checks passed before the project moved into my account.

Substack blocked requests coming from Cloudflare, so the first version hit a dead end. I moved the SDK to my VPS and built a small server interface around it, giving my other apps a stable way to use it.

Hermes handled this part well because the problem stretched beyond one project folder. Old conversations, forgotten files, licensing, and the safety of each user’s digital login key all affected the decision.

A coding agent focused on one project sees the files in front of it. Hermes also saw why I had them, what I had tried before, and which surrounding work mattered.

I covered a similar choice in my guide to choosing AI tools. Tool choice starts before the interface. First decide whether the messy part is finding the work or doing it.

Codex Needed a More Targeted Job

Once Hermes had found the right starting point and cleared up the wider questions, I moved the project into VS Code.

I use Codex through the VS Code extension, and this is where it works best for me. I point it at a specific file, ask for one change, review the result, and keep moving without reopening the whole project discussion.

I asked Codex to run the version Hermes had prepared beside my other services without rewriting anything. It downloaded the project, installed what it needed, passed the checks, and started the service.

Then I opened my Files app and couldn’t see it.

Codex had placed the project inside a private folder on the server, while my Files app only showed a different Projects folder. Every command had succeeded, yet the result still lived somewhere I didn’t use.

I corrected the path, and Codex moved the project before restarting the service. One small mistake exposed a useful limit: a coding agent understands the project in front of it, while personal conventions still need to be stated clearly.

Once everything ran from the right place, I asked Codex to launch Jakub’s newer project beside mine. His version already worked as a complete service for regular apps and AI agents. Mine did less, though its smaller size made it easier to reuse inside other products.

Running both made the decision easier. Jakub’s project was the stronger ready-made service, while my smaller toolkit made more sense as a building block for the server I wanted to control.

OpenAI had also given me banked resets, small refills for my Codex allowance whenever I reached the limit. Hermes burned through those resets much faster than Codex inside VS Code because it carried a wider working history into every session, including old conversations and surrounding files.

Codex stayed focused on the active project and the exact edit in front of it. For small changes, this made the VS Code extension faster and cheaper to run.

No model comparison or architecture discussion would’ve given me the same confidence. Hermes found and framed the right project, then Codex handled the precise edits without dragging the entire history behind it.

The Server Had To Prove It

Next came a small server for making the toolkit available to other apps through a web address. Codex downloaded it, ran its checks, started it locally, and confirmed it was healthy before I added my Substack login details.

After setup, I asked for my latest Substack notification. A live video alert came back from my account. Five recent replies followed when I asked for my latest mentions.

Documentation examples are easy to fake, so I had to know this would be working as expected. A response from my account proved the toolkit had completed the full trip to Substack and returned with real data.

Live use also exposed a design problem. Loading one Substack login at startup meant every person using the service would’ve appeared as me. A multi-user app needed a private and secure way of storing each account's data and selecting the right login for every request.

I am building an app for that and will soon have it ready. I’m thinking of calling it StackedHQ.

Let The Tests Argue

scheduled_at and trigger_at both sound reasonable as labels for a publishing time. An AI has seen enough software to defend either one with enough detail to waste your afternoon and usage quota.

Tests settle the argument by opening the instruction before it leaves the toolkit. One scheduling check expects trigger_at. Another caught missing image information and led to a tiny correction.

Version 0.2.2 passes 25 tests covering 52 expected behaviors. GitHub reruns them before every release, then publishes with one-time approval instead of storing a permanent publishing password.

I don’t care which agent sounds more certain once a request has an observable answer. Real responses, changed files, and passing checks get the final vote.

This is the part I missed inside one AI thread. Whoever researched the problem also wrote the code and explained why it looked correct. Moving the final decision into a check broke the loop.

This habit follows the same direction as The Agentic Engineering Shift. More AI responsibility requires stronger proof outside the conversation.

The Split I Use Now

I didn’t sit down before this project and design a three-part method. Each phase kept failing for a different reason, and the split appeared from those failures.

Hermes worked best while the starting point was messy. I use it when the request sounds like “find the package we used before,” “check what we decided last month,” or “compare this with the rest of my setup.” It searches across the wider project and brings back a prepared job.

Codex worked best after the task fit inside one project folder. I use it once I know where the work lives, what needs to change, and what success looks like. “Improve the Substack tool” leaves too much room. “Run both versions and tell me which one fits an app used by AI agents” gives the work an edge.

Tests take over wherever the answer belongs to the machine. A field name, an installable file, or a response from my real account shouldn’t end as a debate between two models.

I think of those roles as scout, specialist, and referee.

A scout finds the right ground. A specialist works inside a defined surface. A referee ignores confidence and checks what happened.

Choose the role before choosing the model. A stronger model won’t rescue a job whose boundaries are still mixed together.

Copy This Handoff

My handoff between Hermes and Codex now fits inside one short file:

Goal
[one result the project should produce]

Starting point
[where the work lives and what already works]

Known evidence
[useful files observations and previous attempts]

Limits
[what the AI must avoid]

Checks
[how we will prove the result works]

Stop and ask
[what needs my approval before continuing]

Hermes fills the first version from the wider context. I remove weak assumptions, cut extra scope, and make the checks specific. Codex receives the file with the project and works until a check passes or it reaches something needing my approval.

When a check fails, the next request gets smaller. I don’t reopen the whole project. I send the failed behavior back as one specific correction.

This handoff works outside software too. Research produces a sourced brief, writing turns it into a draft, and a publication checklist checks the result. My article about file-based AI workflows covers the wider system.

What Shipped

Version 0.2.2 of the Unofficial Substack SDK gives other apps a reusable way to work with Substack. It reads profiles, posts, comments, engagement numbers, and subscriber data. It also publishes Notes, schedules them, edits drafts, and adds images.

What feels crazy to me is that npm recorded 708 downloads from July 13 through July 19. Those are downloads rather than 708 individual people, and I won’t pretend otherwise. I still find the number encouraging for a new tool solving a problem I found in an old folder on my server.

If you want to give this a test you must know that Substack might change its private API without warning. Tests make breakage easier to find, though they don’t turn an unofficial tool into a promise from Substack.

Two Agents One Workflow

This project started because I wanted Hermes to schedule my Substack Notes. It ended with an open-source SDK, a server running on my VPS, and a much clearer idea of where each agent belongs.

Hermes earned its place at the messy beginning. It found the forgotten package, pulled in old conversations, compared the available projects, and followed the work when Cloudflare blocked the first version. Carrying all this context also made Hermes burn through my OpenAI-banked resets much faster.

Codex worked better once the problem became smaller. In VS Code, it stayed close to the active files and handled precise edits without loading the full project history. This made it faster and cheaper for the small changes where a focused coding agent has the advantage.

Independent checks sat between both agents and the finished package. Real account responses and passing tests overruled every confident answer before anything shipped.

One setup detail matters if you want to try the SDK. I deliberately left the Substack cookie out of the npm package because it gives access to your account. Ask your agent to read the setup instructions and tell you what needs configuring, then add the value of your substack.sid cookie yourself as a trusted server-side environment variable and pass it as sessionToken. Never paste the cookie into your codebase or anywhere likely to save and share it.

Substack blocked requests from Cloudflare during my build, which is why I added a server interface and ran it on my VPS. Your setup might behave differently, though your login cookie should always stay on a server you trust.

Look at your last AI project sprawling across one long conversation. Give the messy discovery work to the agent with the widest view, hand the focused edits to the agent closest to the files, and let independent checks decide when the result is ready.

For the wider tool decision, read When to Use MCPs CLIs or Your Own Tool. For the responsibility behind the work, read The Agentic Engineering Shift.

Use the agent with the widest view to find the path, use the focused agent to make the change, and trust neither until the work passes a check.

OpenAI Is Coming for Hermes One Codex Update at a Time

cucoleadan — Tue, 14 Jul 2026 12:49:05 +0000

This was originally published on All Agents Considered.

Last week I caught myself spending more time inside Codex than using Hermes, and I couldn’t pinpoint when the shift happened. GPT-5.6 had just landed, and what used to be a coding tool inside my ChatGPT subscription had become something closer to a full agent workspace.

What made it strange was that the same $20 bill could also feed a bunch of models into my Hermes agent. One subscription covering two competing stacks, with OpenAI shipping features every week that made one of them feel redundant. Browser control, remote access, plugins, banked resets, and looser usage limits all arrived over a single month, and every one of those used to be a separate purchase or, in some cases, an implementation I had to build myself.

Now I still use Hermes every day, but I just started spending more of my week inside Codex because OpenAI keeps adding features that I use all the time.

Today I’ll share why I believe Codex is catching up with Hermes and why I believe the $20 CHatGPT subscription is really a great deal.

In this article:

The recent Codex releases that shifted how I compare the two
How my ChatGPT subscription supplies Codex models inside Hermes through OAuth
Which workflows moved to Codex and which ones I refuse to move
A three-way test I run before building another agent workflow

The July Sprint That Reshaped My Stack

GPT-5.6’s July 9 release pushed this piece to the front of my queue. OpenAI added the GPT-5.6 family across ChatGPT, Codex, and its API, with three versions called Sol, Terra, and Luna. OpenAI positions Sol as the frontier model, Terra as the balanced option, and Luna as the efficient one (OpenAI’s GPT-5.6 announcement).

Placement matters more than early benchmarks. GPT-5.6 arrived inside Codex alongside faster computer use. OpenAI improved the model and its interface for acting on a computer at the same time. That pairing matters to me more than another leaderboard. An agent becomes useful when its brain and working environment stop feeling like separate purchases.

OpenAI expanded the desktop surface too. Its new ChatGPT desktop app puts Chat, Work, and Codex under one roof. Editing happens directly in Markdown and code, with inline annotations and selected-text revision. GitHub pull requests sit in the sidebar, related repositories share one project, and plugins are managed in Settings (Codex changelog).

Those sound like small interface updates when you read them one at a time. Together, they remove handoffs. Review now stays inside the task. I can inspect the diff where the work happened and return feedback without moving changes through a separate editor. Codex can hold several related repos in one project instead of treating every codebase like a separate room.

Browser work became more serious too. With explicit approval, Developer Mode gives Codex controlled access to Chrome’s developer tools, including the console, network activity, page structure, styles, and performance data (Codex browser documentation). That turns the browser from a page the agent can click into an environment it can inspect. For anyone building or testing a site, this removes another reason to wire up a separate browser setup for bounded work.

Codex Reaching Beyond Desktop

Codex also gained more reach. Codex Remote reached general availability on June 25. A task can start or continue from the ChatGPT mobile app while the work runs on a paired Mac or Windows computer. OpenAI also released a DigitalOcean Droplet Workspace plugin that provisions a remote machine, configures SSH, and connects it as a Codex workspace (ChatGPT release notes).

That closes part of the gap I used to describe as simple. Hermes lived on my VPS and stayed available from Telegram. Codex lived on the computer in front of me. Remote access and remote workspaces make that boundary less clean.

Plugins are moving the same direction. OpenAI replaced the old App Directory with a Plugin Directory, and its plugins can package skills, apps, and templates. Codex also improved plugin loading and made remote plugin catalogs easier to use. A workflow that once forced me to choose between an MCP, CLI, or custom tool can increasingly arrive as one installable bundle.

Then OpenAI softened the usage wall. On June 11, eligible Plus and Pro users received reset banking, including one free launch reset. A separate referral promotion ran from June 11 through June 24 and awarded resets after invited users sent their first Codex message. Earned resets expire after 30 days, so I wouldn’t treat them as permanent monthly allowance (Codex pricing).

On July 12, Codex lead Tibo Sottiaux said OpenAI was temporarily removing the five-hour restriction for Plus, Business, and Pro users. Weekly limits remained, and OpenAI’s standing pricing page still documented the five-hour structure. His announcement also included a usage reset after Codex reached six million active users (Sottiaux’s announcement, Codex pricing). Temporary is the word doing the work here. OpenAI can bring the restriction back or change the allowance again.

Still, the immediate price calculation changed. My $20 subscription stretches further during heavy weeks, with banked resets and fewer interruptions while the temporary change lasts.

This is bigger than GPT-5.6 or a reset button. OpenAI shipped the model and workspace upgrades alongside remote control and friendlier limits. That’s what turns Codex into an agent workspace rather than a coding interface. Work that previously started with choosing four services now starts with opening one app.

One Subscription Feeding Two Agents

Hermes officially supports an OpenAI Codex provider authenticated through ChatGPT OAuth. I can run hermes model, choose OpenAI Codex, complete the device-code login, and use the Codex models available through my ChatGPT subscription. Hermes can also import existing Codex CLI credentials when present (Hermes provider documentation).

I don’t need a separate OpenAI API key for that route.

It’s limited to Codex models exposed through the subscription rather than every model sold through the OpenAI API. That boundary still changes the economics. My ChatGPT subscription now pays for two different layers. It pays for the serviced Codex workspace, and it supplies one model route inside the Hermes runtime I control.

Codex and Hermes are competing for my workflows while sharing part of the same model bill.

I also use GLM-5.2 inside Hermes, and this is one of the main reasons Hermes is staying online. My workflow remains in place while I change the model serving it. Hermes’s model catalog includes GLM-5.2 through supported provider routes, while its CLI lets me switch among models I’ve configured (Hermes CLI documentation).

I’ve tested model choice inside a real Hermes workday, and the result kept pointing back to the same rule: the workflow should survive the model swap. GLM-5.2 uses a separate provider route, leaving the ChatGPT subscription as another option beside it.

That distinction was missing from my earlier $30 Hermes stack breakdown. Hermes still has visible hosting and provider costs. Nous Portal now offers a more bundled route, so several separate API keys are optional. Maintenance time remains part of the bill either way.

Codex hides more of those decisions inside a single price. I covered the wider plan economics in my comparison of six AI subscriptions, but the practical difference is simple. Codex supplies a serviced workshop. Hermes gives me the keys to one I own.

Serviced workshops keep adding equipment. My own workshop lets me decide which engine runs it.

The Workflows Codex Took From Hermes

Research moved first.

My Hermes research setup used a custom search route and a saved-file workflow that sorted everything against my brand filter. I still use it for recurring research, but one-off article research is usually faster in Codex now.

Codex researches against my local brief and writes the result back into the same Obsidian workspace. Research and writing live in one task, giving most bounded questions one search route.

This article is a concrete example. Its outline, research brief, old posts, and brand files all live in my vault. Codex can research the release claims against those files and write the article into the correct folder without me carrying context between tools.

Repository and browser work followed. Codex already had an advantage here because code is its home territory. Inline review and Browser Developer Mode widen that advantage. I can inspect a site and its console, edit the code, and review the diff in one working session.

One-off files became obvious too. When I need an office file or visual asset once, building a permanent Hermes workflow around it makes little sense. Codex has the file tools and task context ready. I verify the output and leave without creating another piece of agent infrastructure. My biggest change is how rarely I prepare infrastructure before starting. Codex removes the provider and output-routing decisions for bounded work.

Some work moved only halfway. Hermes still collects recurring research and saves the briefing on schedule. Codex often takes over when I turn one of those observations into a finished asset. That handoff keeps both systems useful without maintaining the same production setup twice.

If someone you know is rebuilding an agent stack Codex now supplies out of the box, send them this article before they add another API.

The Workflows Hermes Still Defends

My morning workflow stays.

It runs before I sit down, applies my filters to the sources I chose, saves a briefing, and delivers it through the gateway I control. I documented the full version in How My Hermes Agent Plans My Morning Before I Have My Coffee.

Codex now supports scheduled automations and remote work of its own. My reason for keeping this workflow in Hermes survives those additions because the complete runtime already lives on my VPS. Moving it would trade a working system I control for a product surface whose limits and behavior OpenAI controls.

I keep that runtime dependable with my Hermes maintenance routine, which checks the layers a bundled product manages for me.

Telegram stays too. Hermes remains available where I already communicate, even when my laptop is closed. It can call my own scripts and services, work with the files on my server, and keep the result in a location another workflow already knows how to find.

Provider choice matters most here. I use GLM-5.2 in Hermes and can switch the same workflow to a Codex model through ChatGPT OAuth when I want to. If one provider changes its terms or performs poorly on a task, I’ve got another route. That freedom has a maintenance cost, but here it buys continuity instead of technical decoration.

Persistent files finish the case. My workflows leave briefs and outputs in folders I own, with review notes beside them. Codex can work inside those folders, but Hermes is the runtime connecting them over time. One task hands a file to the next without depending on a single product account to remember the whole chain.

Sorting Every Workflow Into Three Buckets

I stopped choosing one agent for everything. I sort each workflow into one of three groups.

Bounded, interactive work where I’m present to start it and approve the result goes to Codex. Research for one article, repository work, browser testing, a document, or a short analysis usually lands there.

Work that needs to run without me, start from a schedule or outside trigger, reach private services, persist across sessions, or survive a provider switch stays in Hermes. Morning research, Telegram access, and multi-step file workflows land here.

Rare or unstable work stays manual for three runs. I automate it only after the inputs, judgment points, and output stop changing.

Six Checks Before You Build

I use six questions before deciding where a workflow belongs.

Does the workflow need to run without me? Does it need a schedule or external trigger? Does it touch files or services I want under my control? Would losing one vendor break the workflow? Do I need to switch models or providers? Does the control repay the maintenance cost?

Results map to Codex for bounded work, Hermes when continuity or control shows up several times, and three manual runs when the pattern remains unclear.

Two Workloads Through the Same Test

My morning research passes five of the six ownership checks. It runs unattended on a schedule and feeds later workflows from my files, so provider switching changes the result. Hermes earns its place there.

Researching a single article passes almost none. I’m present, the task is bounded, and the output goes into a draft I’ll review. Codex wins.

This test takes less time than configuring one API, and it has stopped me from maintaining the same capability twice.

Run Your Own Audit

Write down three recurring AI tasks and mark each one C, H, or M. Pause one duplicated layer this week, then check whether the workflow still finishes cleanly.

The Expanding Overlap Open Source Needs to Answer

When I say OpenAI is coming for Hermes, I’m describing an expanding product overlap rather than alleging that OpenAI copied a specific feature or set out to kill an open-source agent.

Codex is swallowing the layer of self-run agent work where convenience was the main payoff. Every new piece OpenAI bundles into the working environment makes the ownership case work harder.

Open source needs to protect a result I’d lose inside the rented product, because more switches alone no longer win. For me, those results are provider choice, persistent files, and an always-on runtime I control.

Codex carries the opposite risk. OpenAI controls the product and its limits. July’s friendlier terms prove both sides of that bargain because the company can remove friction or restore it quickly.

I wrote the longer version of that risk in my vendor-lock-in article. Codex’s current sprint has made me more selective about where independence pays while leaving the underlying risk intact.

Codex is shrinking the part of my stack worth maintaining. Hermes protects the workflows I refuse to rent.

Ownership has to earn its maintenance now.

Where the Line Sits Today

Codex handles bounded research, article production, repository work, browser inspection, and one-off files. These jobs start with me, end with a reviewed output, and benefit from the serviced bundle.

Hermes handles scheduled research, Telegram, persistent file chains, private services, and workflows where I want GLM-5.2, a Codex model, or another provider without rebuilding the system.

Manual covers everything else until it survives three real runs.

Which workflow has a subscription recently pulled out of your self-run stack?

Codex is the serviced workshop, and OpenAI keeps delivering new equipment. Hermes is the workshop I own, where I control the keys, files, and engines.

I build fewer tools myself and reserve ownership for the workflows where it protects the result. But that line keeps moving one Codex update at a time.

The Twenty Minute Audit That Found Where Hermes Was Getting Lost

cucoleadan — Tue, 07 Jul 2026 13:27:09 +0000

This was originally published on All Agents Considered.

I just got back from two weeks on the Croatian coast. While scrolling in a café in Rovinj, a Forbes headline caught my attention and I had to read the whole thing. AI costs more than the people it was supposed to replace. The technology that was promised to make labor cheaper is, at this moment, more expensive than the humans it was meant to displace.

That made me realize something. A few months earlier, I spent three weeks swapping models, adding MCPs, and building automation layers because Hermes kept getting confused. I kept chasing smarter models and better integrations when the problem was my workspace the whole time. None of it worked until I stopped adding tools and started looking at the files underneath them.

I wrote about the fix in a previous piece, and that article showed what a clean workspace looks like after reorganization, but people kept asking me the same question: how do you know which folder is broken before you start fixing things?

Today I cover the five-point audit I use, a prompt that runs it through Hermes itself, and how to pick the one fix that matters most without rebuilding your entire setup.

In this piece:

Why adding tools before auditing your workspace makes the same problem worse, because every new tool inherits the same confusion
A five-point audit that finds exactly where Hermes gets lost in your files and takes twenty minutes to run
A copy-paste prompt that audits any folder through Hermes itself and tells you where the confusion lives
How to pick the one fix that matters most without reorganizing your entire setup or over-structuring folders that work fine

When Tools Inherit the Mess

The hardest thing about working with Hermes, or any other AI harness for that matter, is accepting that it has no memory of what I already know.

When I open a folder, my brain automatically filters everything. I remember which draft I was working on yesterday, I can tell the difference between notes from this month and notes from six months ago, and I know exactly which file I need because I remember writing it.

Hermes doesn’t get any of that context. It looks at my workspace the same way a stranger would, reading only what’s visible right now and making assumptions based on file names, folder names, dates, and whatever instructions I happened to give it.

That gap between what I know and what Hermes can see is where everything breaks down.

If current and archived files look identical from their names alone, Hermes treats them as equally valid options and sometimes picks the wrong one without realizing it. If there’s no clear starting point in a folder, Hermes has to search through everything before it understands what it’s looking at. And if draft work lives in the same space as public actions, Hermes has no way to know when it should stop and ask for permission.

This is why my Hermes maintenance routine only works when the workspace underneath it is clean.

That routine checks models, memory, cron jobs, and gateway health, but every single one of those checks depends on reading from files. When the files are a mess, the routine produces noise instead of signal.

Workflow handoffs and approval gates fail the same way because they all assume Hermes knows where to read, where to write, and when to ask before acting. Those assumptions collapse the moment the folder structure doesn’t make those answers obvious.

I learned this by watching the same failure repeat every time I tried to fix it with a new tool.

A smarter model reading the wrong file still produces the wrong answer. A new MCP fetching data into a folder with no structure still loses the output somewhere. Switching providers doesn’t change where Hermes writes things down.

The one variable that determines whether Hermes succeeds or searches in circles is the one I kept ignoring, because it wasn’t as exciting as upgrading to a better model or connecting a new integration.

The 20-Minute Audit

My audit runs five checks on one folder Hermes touches often. Pick that folder and open it like Hermes would, with fresh eyes and no memory of what’s inside.

Hermes Workspace Audit

1. Active work
There is one obvious place for current work.
Pass if Hermes knows where current drafts, notes, or tasks live.
Fail if current and old material share the same folder.

2. Archive boundary
Old files have a separate archive folder.
Pass if archived material is clearly historical.
Fail if old files look current from the filename alone.

3. Entry point
The folder has an obvious starting file when it has multiple subfolders.
Pass if the first file tells Hermes where to start.
Fail if Hermes has to search before it understands the folder.

4. Handoff path
Each repeated workflow has an input and output location.
Pass if Hermes knows where to read and where to write.
Fail if every run invents a new output location.

5. Risk boundary
External, paid, destructive, or public actions have a review step.
Pass if Hermes knows when to stop.
Fail if the same instruction lets it draft and publish.

Walk through each check on one folder. Don’t audit the whole vault, don’t reorganize anything yet. Find the one folder where Hermes wastes the most time and understand why.

This prompt runs the same audit through Hermes:

Audit this folder as if Hermes had to work inside it.

Folder path or description:
[PASTE FOLDER PATH OR DESCRIBE THE FOLDER]

Main task Hermes should do here:
[DESCRIBE TASK]

Check the folder across these categories:

1. Active work
Does the current source of truth stand out.

2. Archived work
Does old material look separate from current material.

3. Starting point
Is there an obvious first file, root map, or README.

4. Handoffs
Are input and output locations clear.

5. Risk boundary
Are public, paid, destructive, or external actions separated from draft work.

Return:
- what would confuse Hermes
- the first file it should read
- the first file or folder I should rename, move, or index
- whether this folder needs a root map, and a recommendation on creating an INDEX.md for it
- one warning about over-organizing this folder

Run it on the folder you audited by hand and compare the results. If Hermes finds problems you missed, your instinct about the folder was incomplete. If you find problems Hermes missed, your prompt needs more context about the task. You can build this as an agent or skill file, or keep it in your docs and point Hermes to the folder when you’re running the audit.

Walking Through an Audit

Let me show you what this looks like with a real folder so you can see where to pay attention and how to interpret what you find.

I’ll use the research workflow I rebuilt in my previous piece because it’s a concrete example of what happens when you get the structure right from the start. The folder is called 01.Research Sorter and it holds the entire workflow that turns raw research links into scored article angles.

01.Research Sorter/
├── 01.instructions.md
├── 02.input.md
├── 03.output.md
└── 04.review.md

Four files, numbered for reading order, each with one job. Let me walk through how each audit check applies to this folder.

{image 5: diagram-036-folder-pass.png — Infographic: folder structure passing all five audit checks}

Check 1: Can Hermes find the current work? Pass. The numbered prefixes tell Hermes exactly where to start. 01.instructions.md is the entry point, 02.input.md is where I paste my research links, 03.output.md is where Hermes writes the scored results, and 04.review.md is the checklist I run through after. There’s no ambiguity about which file matters for which step. Hermes reads the instructions first, then moves to the input, then writes to the output. The sequence is explicit.

Check 2: Archive boundary. Pass. This folder only holds the current workflow. Old research runs get archived to a separate Archive folder once I’ve picked an article angle from them. The active folder stays clean because completed runs don’t pile up inside it.

Check 3: Entry point. Pass. 01.instructions.md is the obvious starting file. When Hermes opens this folder, it reads the instructions first and knows exactly what the workflow does, what the brand filter is, and how to score each item. No searching required.

Check 4: Handoff paths. This is where the audit caught a real problem in my earlier version. Before the rebuild, my research workflow would write output to different locations depending on what I’d told Hermes that day. Sometimes it went to 01.Articles/Drafts, sometimes to the root folder, sometimes to a folder I’d created on the fly. Every time Hermes had to find the output from the last run, it had to search again. The numbered file structure fixes this as 03.output.md is always in the same place. The next workflow step knows exactly where to look.

Check 5: Risk boundaries. Pass. This folder only produces draft observations and article angles. Nothing in it triggers a publish action or sends anything public. The actual writing and publishing happens in a separate workflow folder with its own risk boundaries. That separation means Hermes can run this research workflow freely without accidentally pushing something live.

Once I walked through these five checks, I could see why this workflow runs reliably while others didn’t. The structure answers every question Hermes might have before it asks.

The key thing I learned from auditing this folder is that you’re not looking for problems in isolation. You’re looking for the one problem that creates the most confusion for Hermes. In folders that fail the audit, the archive boundary is usually the biggest issue because it affects every single task.

That prioritization is what makes the audit useful. You’re not trying to fix everything at once. You’re trying to find the one thing that, if fixed, would make the biggest difference across most tasks.

Pick One Fix

Find the single issue from the audit that creates the most Hermes confusion. One fix, not five.

If current and archived files live together, fix the archive boundary first. Move old files to a separate archive folder. This one change solves more Hermes confusion than any other because it stops the most common failure: picking the wrong version of a file.

If outputs are unclear, define where Hermes should write before changing any tool. Create one output folder and point Hermes at it every time. Consistency matters more than location.

If there’s no starting point, build the INDEX.md pattern from my piece on file-first agent workflows. Read that piece, build one INDEX.md for the folder that failed the audit, and run the same task again. If Hermes finds the right file in under thirty seconds, the fix worked.

What you shouldn’t do is fix everything at once. Most of your folder names are fine, and the audit tells you which ones confuse Hermes. I wrote about the over-organizing trap in my previous piece on file-first agent workflows, where adding INDEX.md to every subfolder made Hermes slower. The short version: add only enough structure to fix the specific problem the audit found.

Also, don’t use memory to paper over bad file structure. Memory tells Hermes things across sessions, but it doesn’t help Hermes find the right file inside the current one. If the folder is confusing, fix the folder. And don’t add another MCP before Hermes knows where the work lives. I wrote about the MCP versus CLI versus custom tool decision in a previous piece. If Hermes can’t find files in a clean local workspace, no external tool integration will compensate.

Once the workspace is clear, the tool choice gets easier. A clear local workspace often points to CLI or file tools first. Shared external systems point to MCP. Repeated narrow steps point to a custom wrapper or skill. Most of the time, the audit reveals that I don’t need a new tool. I need a better folder. Even when I’m running multiple agent frameworks through the same Hermes brain, the workspace structure is what makes it work. Different agents, same files, same folder map.

If the broader stack is what you’re after, whether that’s provider routing, memory ownership, or scheduled workflows, I wrote the full cost breakdown of the stack and the morning workflow that runs on it. Both of those depend on the same workspace audit pattern to stay reliable across sessions.

Everything I’ve written about in this newsletter traces back to one instinct. Own the layer that matters. I built my stack so no company controls my tools. I built my workflows so no noise controls my output.

Capability is cheap when the foundation is broken. Audit one folder. Fix one thing. Then decide whether you need another tool.

Tear Down Your AI Workflow and Rebuild It Like This

cucoleadan — Tue, 30 Jun 2026 13:40:51 +0000

This was originally published on All Agents Considered.

Futurism ran a story last week about employees burning their company’s expensive AI tokens on pointless tasks. Amazon staffers were using their mandated AI agent to run personal tasks to hit usage quotas. A single employee at one company spent over $150,000 a month on AI tokens.

Moreover, Uber capped each employee’s AI spending at $1,500 a month after engineers blew through the company’s entire 2026 AI budget.

I did the same thing on a smaller scale, and it cost me three full days before I admitted it. Last week I wrote about reorganizing my files so my agent could find them in thirty seconds, and that gave me the ground to rebuild on. If I were starting over today, the first thing I would do before connecting anything is this one test I skipped.

In this edition I’m going to show you that test, the workflow I use now because of it, and how you can run it on your own setup right now.

In this piece:

A five-minute test that tells you whether a source belongs in your workflow before you waste a day configuring it
The exclusions list that cost me three days to learn but saves me 40 minutes every morning since
A four-file workflow pattern that turned 80% noise into 10 usable article angles per run

The Three Days I Lost to Configuration

When I found Hermes, the first impulse was to connect everything I could think of. arXiv for academic papers, Reddit through the Arctic Shift API for community discussion, RSS feeds for blogs I read, and a cron job to run the whole thing every morning so I would wake up to a fresh batch of research.

Each connection required its own setup, and each setup required its own research.

Arctic Shift needed an endpoint, subreddit filters, and time-range parameters. I had to find the API, read its documentation, figure out the filtering, and test it. That alone ate a full day. arXiv needed category filters for computer science papers, rate limiting so I wouldn’t get blocked, and a custom parser for its XML feed. Another day gone. Then I had to build the Hermes skill that would call both APIs, parse the results, deduplicate across sources, and route everything through my brand filter, which checks whether each item helps someone move beyond ChatGPT toward open source, local, and agentic systems they control. Its skill file alone was 180 lines of configuration listing six sources, a daily cron schedule, and the output format for observations and article angles.

By the time I finished, six sources were wired together and firing on a daily schedule. Less than 20% of the output was relevant to anything I would write about for All Agents Considered. arXiv served academic papers I would never translate for readers, while the subreddits I scraped served social chatter that would never pass my brand filter.

I filtered the noise for a few days thinking I could tighten it, but the bigger loss was time spent on API setup and skill management for sources I never held next to my filter before wiring them in. Building felt productive. Connecting APIs felt productive. Making things run on a schedule felt productive. But none of it produced a single article angle because those sources were never going to fit the work I do.

Every item must support the AAC mission: help AI users become more independent by building open source, local, and agentic systems they control.

I had that filter written down in the skill file. I never held it next to arXiv and Reddit before wiring them in. Instead I built on a wrong foundation for three days and ran the output for another week before I admitted it.

The Test I Should’ve Run First

Open your brand filter or audience description. If you don’t have one, write three sentences about who you serve and what they need. That’s your filter.

Open each source you’re planning to connect. Spend 60 seconds scrolling through what it produces today. Hold that output next to your filter.

arXiv today publishes papers on transformer architecture, reinforcement learning benchmarks, and multimodal reasoning. None of those help a non-coder move beyond ChatGPT. Reddit’s r/LocalLLaMA discusses quantization formats, model benchmarks, and hardware setups. Some crosses into AAC territory, but most assumes a technical reader who already knows what GGUF means. My audience doesn’t.

Most source mismatches are obvious when you see them side by side. People skip the test because connecting an API feels like real work and reading a webpage feels like nothing, so we gravitate toward what looks productive.

If you’re about to wire a source into your AI workflow, run this test right now. Open the source, open your filter, look at both. If the match is weak, put the source on an exclusions list and move on. You saved yourself a day of configuration.

Where the Output Proved Me Wrong

20 to 30 items every morning from 6 sources, and I was spending 40 minutes sorting through them. My brand filter was supposed to do the sorting, but the agent was applying it loosely because the input was too varied. Academic abstracts, Reddit threads, blog posts, and RSS headlines all needed different interpretation, and the instructions weren’t specific enough.

I tightened the filter and added scoring from one to five:

# Scoring rubric
+1 if technical people are discussing it
+1 if it helps people move beyond ChatGPT
+1 if it relates to independence, control, cost, memory,
   files, agents, local AI, or open source
+1 if a non-coder needs translation
+1 if it points to a practical setup, workflow, article,
   course module, or useful build

Drop anything scoring below 3.

Each rule caught real noise, but each rule was also more configuration on a foundation already wrong.

A daily cron meant the workflow ran every morning whether I needed it or not. Some mornings I hadn’t scanned new sources since the last run, so the output was thin. Other mornings I had a weekend backlog the single daily run couldn’t handle. Every three days to a week would’ve been right, but daily felt productive.

When I subtracted the setup days from the value the workflow produced, the complicated version was worse than no workflow at all. I’d spent more time configuring sources I should never have connected than I would’ve spent doing the task manually for a month. Once I understood that math, I tore the whole thing down.

The Exclusions List That Replaced It

Before I rebuilt anything, I wrote down every source that didn’t serve my work:

# Hard exclusions - sources I don't track as daily inputs

- arXiv
- Semantic Scholar
- Papers with Code
- Hugging Face Daily Papers
- Hugging Face trending models
- Ollama as a standalone source
- Reddit
- X
- LinkedIn
- YouTube
- Discord
- Product Hunt

Every source on that list is fine for other audiences. arXiv is essential for researchers. Reddit is gold for technical practitioners. YouTube is where most people learn about AI today. None of them produce output that helps a non-coder AAC reader move beyond ChatGPT toward systems they control.

Exclusions are where time savings compound. Every source you don’t connect is a day of configuration you don’t spend, an API you don’t debug, and a category of noise you don’t filter every morning. Short list, massive savings.

I wrote about the shift from optimizing prompts to building workflows earlier this month, and the exclusions list is the bridge between those two ideas. You can’t build a workflow until you know what doesn’t belong in it.

What stayed was small by comparison. Hacker News for technical discussion that non-coders need translated. Lobsters for deeper practitioner conversations. A handful of blogs I’ve read for years and trust to stay in my lane. RSS feeds from three newsletters that cover the intersection of AI and personal productivity. That’s the input list, and it fits on a single screen.

Before the rebuild, I was sorting through 20 to 30 items every morning from 6 sources, spending 40 minutes filtering noise. With this list, I paste a handful of links, run the agent, and get 10 usable angles back in the time it takes to make coffee.

The Workflow That Replaced Three Days of Configuration

My rebuild started by asking what the workflow needs to do. Sort raw research notes from sources I trust, score each against the brand filter, keep what passes, turn the best into article angles, and save the result to review with my coffee.

Structure uses the same numbered-folder pattern from my file structure piece, applied to a single workflow instead of a whole vault. Four files in one folder for one task I repeat every few days, with no external APIs, no cron, no routing, and no memory layers. Everything the agent needs to do its job lives in those four files, and nothing else creeps in.

01.Research Sorter/
├── 01.instructions.md
├── 02.input.md
├── 03.output.md
└── 04.review.md

Instructions tell the agent what the workflow is and what it should do, in plain English:

# AAC Research Sorter

This workflow takes raw research notes and turns them into
AAC article observations and seeds.

Read 02.input.md. For each item, apply the brand filter:
- Does this help someone move beyond ChatGPT?
- Does it point to a practical setup or decision?
- Does a non-coder need it translated?

Score each item from 1 to 5. Keep items scoring 3 or higher.
For each kept item, write one observation with a signal,
a plain-English translation, and an article angle.

Write the sorted list to 03.output.md.

Input holds the raw material. Links, titles, one-line observations I jot down while scanning sources I already trust. Everything I dump goes in this file, and the instructions file tells the agent how to sort it. Output receives the result: a dated note with top observations, article angles for each, the best three ideas to write next, and any course or build connections. Review tells me what to check:

# Review Checklist

- Read the best-three list
- Pick one idea to pursue this week
- Flag any observation where the article angle feels generic
- Check whether any item scored too high because the
  brand filter was applied loosely

Numbers give the agent its reading order, one concern per file keeps boundaries clear, and every file has one job. When the agent opens the folder, it doesn’t have to guess what to read first or what each file is for.

Three Manual Runs Before You Automate Anything

Before I added anything back, I ran the workflow three times manually. Three mornings of pasting links, running the agent, and checking what it produced. First run had a scoring error because my brand filter instruction was too vague about what counts as practical. Second run turned a pure news item into an observation because the instructions didn’t explicitly exclude vendor PR. Third run was clean.

If I’d added the cron schedule after the first run, I would’ve automated a workflow still broken in ways I hadn’t seen. Running manually three times is what taught me what the workflow does well and where it breaks. One run teaches you nothing because you haven’t seen the variance.

This is where most people go wrong. They build something, it works once, and they schedule it to run forever. Second run catches edge cases the first one missed. Third run proves the pattern holds. Skipping any of those steps means you’re automating a system you haven’t validated, and automation amplifies problems as reliably as it amplifies good output.

You can build this as an agent or skill file, or keep them in your docs as they are and point Hermes to the folder when you’re running the workflow.

Its full version now fetches from nine sources I’ve validated against my brand filter, scores 166 items in a single run, and produces ten article observations with course connections. It runs every few days, not every morning, because that’s the cadence the task needs. I wrote about how this fits into my morning routine, and the Hermes maintenance routine I run keeps it from drifting.

Run This Today

Open your AI workflow. Look at every source it connects to, every API it calls, every feed it reads. For each one, hold the source next to your filter. If the match is weak, disconnect it. Put it on your exclusions list. You freed up the configuration time and morning filtering time that source was costing you every single day.

If you don’t have a brand filter yet, write three sentences about who you serve and what they need. Sources that survive the test are your real input list. Build around those, run it by hand three times, and think about scheduling only after the third run is clean.

Whole process takes an afternoon and replaces weeks of configuration sprawl. Every layer you add after that, whether it’s a cron schedule or a new source, gets added only when the previous one works.

If the broader stack is what you’re after, whether that’s provider routing, memory ownership, or how to build an agent that doesn’t drift between sessions, I wrote the full cost breakdown of the Hermes stack and the morning workflow that runs on it. Both of those depend on the same small-file-loop pattern to stay reliable.

Capability is cheap when the wiring around it is broken. Build the filter first, run it by hand three times, and the capability takes care of itself. One brand statement and a four-file loop is the difference between an AI workflow that wastes your mornings and one that gives you back two hours a week.

Why My Best Hermes Workflow Consists Of Plain MD Files

cucoleadan — Tue, 23 Jun 2026 13:47:58 +0000

This was originally published on All Agents Considered.

Last week I read about an AI that designed a vaccine fighting coronaviruses we haven't even encountered yet (ScienceDaily). Cambridge ran the first human trial, and 39 volunteers received the antigen.

It's crazy to think that an algorithm analyzed the entire genetic family of a virus and engineered protection against members still circulating in animals and waiting to jump. But when I ask it to fetch my latest brief from a list of docs, it can spend nearly two minutes opening the wrong files in my vault just to show me a brief from three months ago.

And I know Cambridge uses a custom trained AI, but what I want to show you is that the gap between what AI can do and what a setup lets it do is what makes or breaks your workflow and productivity.

I never built scaffolding on purpose, until recently. It piled up around the agent by accident, folder by folder, until the agent lost its way inside a mess I shaped for myself.

So I spent a day timing every failure, and a pattern showed up. In this edition I show you why your agent keeps tripping over the easy stuff while the world calls it a genius, and the fix that puts it back on your side.

If you’re still watching your agent search through folders for the right file, this is the fix. Subscribe and I’ll send you one practical workflow every Tuesday.

In this piece:

Folder structure failed my agent because my folders were organized for how I think, not for how an agent navigates
One index file at the root of every major folder turned slow searches into thirty-second reads
The invisible scaffolding around your agent matters more than the agent’s own capability, and here’s how to build it
A diagnostic test you can run in five minutes to see whether your folder structure is the real problem

The Structure Your Agent Can’t See

I organized my vault the way most people do. Articles, research, assets, strategy docs, each got their own folder. That structure felt clean because I think in content types. Folders map to my categories and I never think about where to look. Drafts are articles, notes are research, the filing system runs on autopilot.

But an agent doesn’t think in categories. When I ask Hermes to plan a product launch, the task pulls from strategy notes, brand guidelines, and previous launches scattered across different folders.

Those pieces of context live in different places in my content-type structure. My agent has to search everywhere every time, without knowing which draft is current versus which one is archived, or which research note is from this month versus from six months ago.

Content-type organization works perfectly for humans. You’ve spent your whole life filing things under categories, and your brain does the cross-referencing automatically. It’s only natural to expect AI to work the same, when in fact an agent has to do that cross-referencing manually, and that’s where the structure breaks down.

The structure around your agent does more work than the agent’s own capability.

Once I saw the pattern, the cause was obvious: I had built my vault for a human brain that remembers where things live, then handed it to an agent that has to search from scratch every time, and that exposed the real issue underneath the folder mess.

A searcher needs a map, because without one the agent is walking through the dark and treating every file like it might be the answer.

What Capability Wastes On

Hermes kept opening a bunch of files before landing on the right one, or grabbing an archived version when the current file was somewhere else. Worse, it would ask which file to use when the answer should have been obvious from context. This happened with every model I used, including Opus, GPT 5.5, Qwen, and GLM.

The pattern kept repeating, and I was stuck watching the same slow searches play out across task after task.

Even though my agent was capable enough to find things, the folder structure had become the main bottleneck before it ever reached the actual work.

I hit the breaking point the day I timed the failures. I tracked how many files Hermes opened before it reached the right one, and how long it took to get there. Five common tasks looked like this before I changed anything.

Before I changed anything:

Find the current article brief: 7 files opened, 2:00 to the correct file;
Find brand color definitions: 5 files opened, it took 1:12;
Look up the article queue for planning: 4 files opened, it took 0:48;
Find previous articles on the same topic: 6 files opened, it took 1:36;
Pull promotion strategy for a launch: 3 files opened, irt took 0:34.

Rough experiment, so treat the exact numbers as directional.

Every task crossed multiple folders. Most of them opened archived versions before finding the active ones. My agent was burning capability on navigation instead of doing its job. One launch plan needed three files that lived in three different folders, and I watched it make three wrong attempts before landing on the right ones.

An agent that writes articles, codes, and plans shouldn’t be spending most of its time looking for things.

The Smallest Cage That Works

One folder per concern, numbered for order, with an INDEX.md at the root that maps everything. That’s the whole fix, and the details live in three rules that work together.

INDEX.md at the root of each major folder is a map. It lists every subfolder and canonical file, plus where my agent should start. My agent reads this first and knows what’s inside before touching anything else. Think of it as a soft approval gate where the agent isn’t allowed to start work until it knows what it’s working with.

My brand folder structure looks like this after the reorganization.

05.Brand/
├── INDEX.md
├── 01.Brand System/
├── 02.Editorial Strategy/
├── 03.Promotion/
├── 04.Public Deliverables/
├── 05.Operating Plan/
└── 06.Archived/

This structure organizes by concern instead of content type. Each concern gets its own folder, so brand work stays with brand work and strategy stays with strategy. My agent doesn’t cross boundaries looking for something that doesn’t belong there.

Numbers on folders make reading order explicit instead of relying on alphabetical sorting. 01.Brand System gets read before 02.Editorial Strategy. My agent doesn’t guess. Numbers inside folders do the same thing for files, so my agent knows 01.AAC Articles is the starting point before 02.Previous Articles.

I lead folder names with a number because it’s easier for me to remember as well.

This is how my full $30 Hermes stack stays easy to move across dozens of folders without my agent ever getting lost.

Archived stuff lives in 06.Archived, where old briefs and retired plans go to wait. My agent knows not to look there unless I specifically ask for historical context. That separation is what keeps the active folders clean and the searches fast.

Once my agent was told not to cross that boundary by default, every task inside the active folders ran faster.

My INDEX.md looks like this.

# AAC Brand Index

This folder holds the current All Agents Considered brand system.

## Folder Map

| Folder | Purpose | Updated |
|---|---|---:|
| `01.Brand System/` | Visual identity, topic scope, voice | 2026-06-11 |
| `02.Editorial Strategy/` | Article direction, title rules, queue | 2026-06-11 |
| `03.Promotion/` | Launch, distribution, tool strategy | 2026-06-11 |

## Canonical Files

| File | Purpose |
|---|---|
| `01.Brand System/01.AAC Brand System.md` | Visual identity, colors, typography |
| `02.Editorial Strategy/01.AAC Articles.md` | Article queue, title rules |

## Where To Go

- Start with `01.Brand System/04.AAC Direction.md` for mission
- Use `02.Editorial Strategy/01.AAC Articles.md` for article selection

INDEX.md went through three versions. My first version listed every single file inside the brand folder and ran to forty lines. It was technically complete but my agent had to parse it every time and the length was its own overhead. My second version was too short, around fifteen lines, and my agent still asked questions I hadn’t answered. This third version lists only subfolders and canonical files, with a short “Where To Go” section that tells my agent its starting points.

When I ask my agent to pull promotion strategy for a launch, it reads 05.Brand/INDEX.md, opens 03.Promotion/01.AAC Promotion Strategy.md, and gets to work. Timing after the reorganization looks like this.

After the reorganization:

Find the current article brief: 1 file opened, 0:10
Find brand color definitions: 3 files opened, 0:22
Look up the article queue for planning: 1 file opened, 0:10
Find previous articles on the same topic: 2 files opened, 0:18
Pull promotion strategy for a launch: 1 file opened, 0:12

The slowest task dropped from 2 minutes to 26 seconds, and the fastest runs landed around 10 seconds. My agent opens the INDEX.md, follows the starting point it needs, and gets to work without wandering. No capability changed on my agent’s side. Folder structure stopped wasting most of it before the task even started.

Folder structure is a cage I built that constrains how my agent moves without constraining what my agent can do. Inside the cage, my agent runs freely. Without it, Hermes tends to wander off and I have to waste my time pointing it at the right file.

Moving to Hermes let me build that kind of system across files, providers, and memory all at once, and the INDEX.md pattern is the smallest piece that made the biggest difference day-to-day.

Every Cage Has a Lock

My first mistake was adding INDEX.md to every subfolder. Too many maps means my agent spends its time reading indexes instead of doing work. I keep INDEX.md only at the root of major folders like Brand and Editorial Strategy, where there are enough subfolders to need a map. Inside a small subfolder with four or five files, Hermes navigates directly without any map at all.

Keep in mind that numbering gets tedious when you add new folders. I’ve got to decide where a new folder fits in the sequence, and sometimes I append it at the end instead of renumbering everything. That’s fine. Numbers don’t have to be perfect as long as they’re directional. Perfectly ordered folders aren’t the goal, and treating them as one turns a practical fix into a cosmetic project.

A bigger mistake is building structure before your agent shows any confusion. Most people over-engineer their agent setup because they think it needs proper infrastructure.

My current rule follows the same principle behind choosing MCPs versus CLIs versus your own tool. It uses the smallest interface that gets the job done. Add structure only when your agent gets lost, and add only enough structure to fix that specific problem.

Another failure I ran into was nesting structure inside structure. I added subfolders inside subfolders to make everything perfectly categorized, which turned a two-level hierarchy into a five-level one. My agent then had to read multiple INDEX files and parse multiple numbering sequences to reach a single file. I collapsed those extra levels back into flat subfolders and navigation sped up again.

Depth is the enemy of fast file lookup, and most reorganizations add depth thinking it’s precision.

Measure Before You Reorganize

Before touching a single folder, run this test on three real tasks your agent handles most often. Time each one, and note how many files it opens before finding the right one. Track how often it picks the wrong file or stops to ask you which one to use.

Any task that takes your agent more than thirty seconds or opens three or more wrong files points to a folder behind that task that’s broken.

Pick the one failure that happens most often. Open that folder and write an INDEX.md that lists every subfolder and important file, plus where your agent should start. Save it. Run the same task again.

If your agent finds the right file in under thirty seconds, the fix worked and you now know the pattern. Apply it to the next broken folder. If it still fails, the problem is likely numbering or one-concern-per-folder, and those are the next two things to fix.

Start with the one folder that wastes the most time, fix it, and move to the next only when you’re ready.

Where the Bars Still Bends

Archived content still gets referenced sometimes when I need an old brief for context. My agent has to know to look in 06.Archived instead of the active folders, and I mention this in the INDEX.md so it knows where historical material lives. Without that note my agent assumes archived content doesn’t exist.

Also, Obsidian Sync creates occasional problems when it does not catch up cleanly across devices. If I update a file on my laptop but the older version is still sitting on my VPS, Hermes loads the VPS copy and treats it like the current source of truth. That is a sync problem more than a folder-structure problem, but it shows up inside the folder system because the agent only sees the files in front of it.

Numbering breaks down when you’ve got so many folders that the sequence becomes meaningless. Past ten or twelve numbered items at one level, the numbers start being arbitrary.

I try to keep major categories below that threshold and nest deeper structure inside instead of expanding the top level.

Renaming is another source of friction. When I reorganize I sometimes rename folders to better match their purpose, and any INDEX.md reference to the old name breaks until I update it. I now try to set folder names once and leave them alone, because a slightly awkward name is cheaper than a broken reference.

This structure works best for content-heavy workflows and planning-heavy ones. Pure code or data-heavy setups might need different organization principles entirely, and the INDEX.md pattern doesn’t solve every navigation problem.

Which folder in your setup would your agent get lost in right now? Tell me in the comments.

I Built My AI Stack to Survive Vendor Lock-in and Google Just Proved Me Right

cucoleadan — Wed, 17 Jun 2026 07:57:25 +0000

This was originally published on All Agents Considered.

Tomorrow, Google will switch off a free tool a lot of people built their daily work around. Most readers here have never touched it, so the shutdown itself won't ruin your week. What sits underneath the shutdown is the part worth your attention. It keeps happening, and one day it might land on a tool you do depend on.

Gemini CLI is a small terminal tool Google released to run AI agents straight from your computer. Think of it as Google's answer to Claude Code or Codex, a way to point an agent at your files and let it work.

What made it interesting was that it shipped open source, which means anyone was free to read the code, copy it, fix it, or build something better on top of it. Open source is the closest thing software has to a promise that a tool stays yours even if the company behind it loses interest.

Google made that promise, and now Google is walking away from it.

Last month, Google announced that on June 18 2026, the tool will stop working with Google's AI subscription plans. In its place comes something called Antigravity CLI, which is closed-source.

Six days earlier, the US government forced Anthropic to shut down Fable 5, a model people had barely started using before it disappeared.

Two different shutdowns, same lesson. The tool you built your work around can change or disappear, and you won't get a vote.

That's why I built my AI stack so I can always swap models without rebuilding anything. When one company shuts the door, another one is already wired up and ready to go. In today's edition, I am going to show you exactly how to break free of vendor lock-in.

In this piece:

Why three different AI vendors made three different moves in one month, and how every move pointed at the same trap
How to spot the parts of your own AI setup that secretly hold your context, your memory, and your routines hostage
The three decisions I built mine around so any single vendor change turns into a shrug instead of a crisis
One move you finish in twenty minutes today that proves the whole principle to yourself

A Pattern Everyone Recognizes and Nobody Names

Here's how it goes, almost every time. A company hands out something free, often open source, and people build real work around it because free and open feels safe. Word spreads. More people lean on it.

Then a quiet thing happens inside the company. Someone notices the free tool now does roughly what the paid product does, and the two start eating each other. Around that point the terms change, or the license shifts, or the whole tool gets retired and replaced by a sealed version.

Goodwill came from the open thing. Money comes from the closed thing. When those two collide, money wins every single time.

Gemini CLI walked this exact road. It carried an Apache 2.0 license, which in plain terms meant stays open forever, free for anyone to keep using even if Google lost interest.

Last month's announcement erased that. A permanently open tool moved behind a paywall and into closed source. A community of people who trusted the open license now have less than 48 hours to migrate or go dark.

Anthropic ran a version of the same play a week before Google's announcement. They quietly walked back hidden safeguards inside Claude after a public backlash.

People leaning on Claude for serious research found out their tool had been working against them in the background. It shifted its behavior while they changed nothing about how they used it.

Then the US government forced Anthropic to shut down Fable 5 entirely. People who had just started building on it lost access overnight, and the company had no choice but to comply.

Two different moves from the same company in the same week. One changed the tool's behavior in secret. The other made it disappear completely.

Same week, OpenAI floated price cuts aimed squarely at Anthropic. Your monthly AI bill now rides on a boardroom argument you'll never sit in and never hear about.

Three companies, three different moves, one shared lesson. Lean your whole setup on a single vendor's tool or model, and you take on their shutdown dates, their hidden behavior changes, their price wars, and every private decision about which features live and which ones disappear overnight.

What Dependency Looks Like

Most people never feel dependent until the morning something breaks. You sit down with coffee, open Claude, and a prompt it handled fine yesterday gets refused today for no reason you understand.

In that moment you learn your entire research routine was balancing on one model staying agreeable. Or you open ChatGPT to pick up a three-month project and your conversation history is gone, wiped during a policy change you never read.

A tool you built your whole day around changed its mind. Your only role in the decision was finding out afterward.

Dependency runs deeper than the model itself. It reaches into everything the model has been touching on your behalf.

Your project context lives inside a chat window that vanishes the second you close the tab. Your task history sits trapped behind an interface with no real export. Your routines and saved instructions live in one company's private format.

Moving to a competitor later means rebuilding the whole thing by hand from memory. Nobody handed you a contract to sign. You used a tool that felt good in the moment.

The context you kept adding to it became the most valuable thing in the room, right up until the day it walked out the door wearing the tool's logo.

This was my normal life before I built around it. I would spend an hour walking Claude through a project, finally get answers worth keeping, close the laptop, come back the next morning, and start from zero.

Every time I switched providers chasing a better price or a smarter model, every routine reset to nothing. None of it hurt enough to fix on any single day.

It only became unbearable after it happened enough times that building a real fix took less energy than complaining about it one more morning.

Three Decisions That Make Any Single Vendor Irrelevant

Understanding this pattern was the same moment I started rebuilding my setup to outlive it. I didn't see the Gemini CLI shutdown coming by name, or predict Claude's safeguards, or guess OpenAI's pricing move.

Watch any group of vendors for more than a few months and the shape becomes obvious. Building around the shape costs almost nothing compared to getting blindsided by it. Three decisions carried most of the weight.

Decision one was routing. Rather than wiring my whole workflow to a single provider, I spread the work across Opencode Go, OpenRouter and Codex (via ChatGPT Pro) depending on the job.

One small file tells my agent which company handles which kind of request. Writing goes to one model because it holds tone better. Research goes to another because it chews through long documents faster. Routine generation goes wherever the price is lowest that hour.

When a provider hikes prices, throttles me, or pulls a model, I change three lines in that file and the work keeps flowing through the others without missing a beat. I broke down the full money side of this in my cost comparison against Claude Max, because the savings surprised even me.

Decision two was memory, and this one mattered most. Claude's conversations disappear when the chat closes. ChatGPT's history lives on OpenAI's servers, under OpenAI's rules. I know that both Claude and ChatGPT have memory, but it's a single file and you cannot alter it directly. All this means that you don't have full access to it.

My setup has 3 types of memory. First, Hermes comes with its own memory files and also built-in tools for session recall, so it can search our past convos. Then I added a 3rd party memory provider called Hindsight, but I might drop it as it's proven to be too much hassle for too little return for me.

Last but not least, I use Obsidian with plain markdown files, synced across my devices through the built-in Sync, readable by any AI tool I choose to point at it. Project notes, research, past sessions, saved agent routines, all of it sits as a file on my VPS, laptop and phone. I also configured a WebDAV server and MCP so literally any AI can get access to my files (behind a login of course).

A provider changes its deal tomorrow and my context follows me wherever I go next. None of it ever belonged to the provider in the first place.

Decision three was the agent layer, and this is the piece most people skip. I run Hermes as the stack that sits between me and whichever company happens to be doing the actual thinking.

Hermes holds the workflow, the saved skills, the routing, and the handoff logic. Whatever provider is plugged in becomes a swappable engine. Switching it means changing a single reference.

When Claude started refusing harmless prompts last week, my workflow never stopped. I would've sent the work somewhere else and kept moving. Every saved skill that taught my agent how my projects run stayed identical, because those skills live in files I control.

I moved to Hermes specifically for this, because the layer running my work needed to stay independent.

None of these three decisions asks for a computer science degree. Each one asks for a single choice, which is refusing to build your livelihood inside a tool that rewrites its own terms without asking you first.

How to Build Your Own AI System

Nobody builds the whole thing in two days, and trying to is how people give up. You build it one decision at a time, letting the setup grow around the parts you genuinely use rather than the parts a tutorial told you to want.

Order matters, because each layer makes the next one easier. Here is the order I followed and why.

Start by owning your memory. That move pays off the same afternoon and keeps paying off every month after.

Pull your project notes out of whatever AI chat tool currently holds them and drop them into a notes app you control. Obsidian, Notion, plain text files in a folder on your desktop, any of them works.

The brand of the app matters far less than the fact that nothing inside it disappears when you close a tab. Every month you keep building context inside a chat window is a month of thinking that disappears the moment the company changes its mind.

With your memory off the company's servers, adding a second provider becomes the natural next step instead of a scary one. Pick one piece of your workflow and run it through a different service for a week.

If Claude writes for you, send a research task through OpenRouter and watch what happens. If ChatGPT does everything, hand one job to a rival model purely to feel the difference.

Replacing your favorite tool is not the goal here. Learning what switching costs, while nothing is on fire, is the goal. That way the day you need to switch under real pressure you already know what you have to do.

Remember that your workflow is really a handful of separate jobs pretending to be one tool.

Ideation runs on one model. Research runs on another. Notes and memory live in your own app. These pieces talk to each other through plain files and simple handoffs.

The tool you love this month might not be the tool you want next year. Spend a little time now making sure that gap costs you nothing when it arrives.

Do This in Twenty Minutes Today

Fastest possible start, and also the simplest. Open whichever AI chat tool you lean on most.

Find the one conversation holding your most valuable project context, the thread full of decisions you made, research you gathered, and ideas you would really need to save.

Drop this prompt into that conversation:

Take the key decisions, facts, open questions, and next steps from this conversation. Summarize them as a structured note with these sections: Project Overview, Key Decisions Made, Important Context and Research, Open Questions, Next Steps. Write it as clean markdown I can paste into a note-taking app. Do not include our chat back-and-forth. Only the useful context.

Copy what comes back. Paste it into any app you own, whether that's Obsidian, Notion, Google Docs, or a single text file on your desktop.

Label it with the project name and today's date, and close the loop.

You're done. You moved your first piece of context off a company's servers and into something nobody gets to switch off.

Your chat tool still works exactly as it did five minutes ago, no harm done, no bridges burned. What changed is that the best part of that conversation now survives even if the tool rewrites its terms tomorrow morning.

Ten conversations fit comfortably in twenty minutes. And once you're done you'll figure out that you've already taken the first step towards building your own system. When you decide to install Hermes, or an open-source agent, you'll have everything you need to get going.

Get Ready For When Things Break

This setup is not perfect, and pretending otherwise would waste your trust. Some months I pour more time into maintaining my routing file than I would have spent living inside one vendor and taking the lock-in.

Independence carries an upkeep cost.

Memory sync across devices stumbles now and then. Mostly when I edit the same note on two machines at once and create a conflict I have to fix by hand.

Routing across providers means keeping three pricing models straight in my head instead of one. When a company changes its rates I need to update my calculations. But usually prices go down instead of up with each open source model (not the same can be said about proprietary ones).

A handful of tools refuse to hand context to each other cleanly. That leaves me copying and pasting between them in a way a single sealed vendor would have smoothed over for me.

Honest version, this whole approach suits people who already got burned by vendor dependency at least once and decided they would rather spend an hour a month on upkeep than risk losing an entire year of work to a corporate decision they were never allowed to influence.

Draw Your Own Dependency Map

Give this two more minutes. Write down three things in your AI life you couldn't easily replace.

One model that handles your most important work (like Opus is for some). One app holding your project history and notes. One workflow that breaks if a single tool changes its terms next week.

Now run one quick test on each. Try to export it.

Find your AI conversations locked inside a chat window with no real export button, and you found your first weak point. The twenty-minute exercise above already started repairing it.

Find your project notes trapped inside a tool that won't let you download them, and you found your second weak point. You already know its name without checking.

Export cleanly from all three and you hold more independence than almost every AI user out there. Fail to export from even one and you found exactly where your next twenty minutes belongs.

My own automated morning workflow runs this same provider-agnostic pattern on a schedule. Every check reaches into files I own instead of interfaces I rent by the month.

Gemini CLI's shutdown is only the newest entry in a pattern with no plans to stop. Claude's hidden safeguards, OpenAI's pricing war, Google's license reversal, each one is the same story wearing a different month on the calendar.

Everything above is the short version, the field notes. Full treatment, including the actual routing files, the Obsidian Sync setup, and the provider comparison that pushed me toward OpenCode Go and OpenRouter, is going into the first Hermes 101 course.

I'm building it right now and it should be ready soon. That course is where the patient step-by-step version of all this will live for anyone who wants their hand held through it.

If the money angle is what grabbed you, I wrote a full breakdown of the $30 Hermes stack that goes toe to toe with Claude Max and asks for no subscription. If you want the origin story, my migration from OpenClaw to Hermes is the longer answer to why I needed an agent layer I controlled before any of the rest of this made sense.

How to Build AI Workflows When You're Tired of Optimizing Prompts

cucoleadan — Tue, 09 Jun 2026 13:18:29 +0000

This was originally published on All Agents Considered.

Finding good content ideas used to take me hours every week. Reddit in one tab, news in another, arXiv in a third, and an Obsidian note where I'd paste everything and try to remember how the pieces connected. Each AI search took seconds, but I spent the rest of the time being the glue.

What made it worse was how much attention I burned just moving between tabs and chats. Every switch cost me focus, and every reset made the work feel heavier than it was.

I didn't know it then, but instead of overly optimizing my prompts, I should've just created a workflow. Took me some time to figure out the best way to go about this and so I am ready to share my way of converting prompts into workflows.

In this piece:

Why prompt habits break at scale
How to spot your first workflow candidate
How to find the seams in long conversations
The handoff pattern that carries context forward

If you're copying output between AI chat tabs, you’re doing the coordination work the AI should handle. The fix is to turn your prompts into a workflow where each step writes to a file and the next reads it. Context carries forward without you carrying it. You only stop where a real decision needs to be made.

When Prompting Stops Working

Almost everyone starts with AI the same way. You type a question, get an answer, copy-paste it somewhere, repeat. This is how I spent my first year using it. And I get it, it feels productive because each interaction gives you something tangible.

Then you notice you’re spending more time managing the AI than the AI is saving you. You’re the one copying between steps. You’re the one remembering what step three needed from step one.

An October 2025 study published on arXiv found that LLM accuracy drops significantly when relevant information is embedded within longer contexts, even when all irrelevant tokens are masked.

Prompt engineering blogs and courses are still selling the idea that the right words will fix everything. They’re optimizing the wrong layer. You’re trying to run a pipeline through a chat window, and no amount of word-smithing changes that.

Hitting a ceiling with prompting means you have an architecture problem.

How to Spot Your First AI Workflow

Before we go further, try this. Think about the last repetitive task you did with AI. The one that took 45 minutes and made you want to scream by minute 30. Now ask yourself:

Did I copy-paste between steps?
Did I open multiple chat windows because context kept getting polluted?
Did I have to remember what step three needed from step one?
Did the AI produce good output at each step, but the final result was mediocre?

If you answered yes to any of these, you already have a workflow candidate. You’ve been doing the coordination work manually.

Here’s a prompt you can use right now. Paste it at the end of your next long AI conversation, after you’ve finished a task:

Look back at this conversation we just had. I'm going to paste the initial prompt I started with below. I want you to analyze whether this task could be converted into a reusable skill or workflow.

Specifically:
1. Could the steps I took be structured as a sequence where each step produces output the next step needs?
2. Are there handoff points where context needs to carry forward?
3. Would this task benefit from being broken into separate steps with clean context, rather than running as one long conversation?
4. What would the input, instructions, output, and checkpoint look like if this became a workflow?

Here's the initial prompt I used: [PASTE YOUR INITIAL PROMPT HERE]

Tell me if this is a good candidate for a workflow, and if so, sketch what the steps would look like.

Run this after your next repetitive task. You might find you’re already doing workflow-shaped work manually.

This works whether you use Hermes, Claude Code, Codex, Cowork, or any other AI conversation tool. Patterns stay the same. Tools don’t matter. Structure does.

Where to Find the Seams in a Long Conversation

Converting a long conversation into a workflow starts with seeing where your current process has seams.

When you have a long AI conversation, look for the moments where you switched gears. Where you said “okay, now let’s do X” and started a new mental context. Where you copied something from earlier in the chat and pasted it into a new request. Where you had to remind the AI what you were working on because it forgot. Those seams are where scope creep happens. I wrote about what Gordon Ramsay taught me about scope and knowing when to stop.

Those are your seams. Each seam is a potential step in a workflow.

My breaking point came during a content ideation project. I needed to find interesting angles for newsletter articles, which meant pulling from multiple sources. Reddit threads surfaced complaints about specific problems, news articles covered emerging tools, and arxiv papers hinted at new capabilities.

I started manually, copy-pasting Reddit posts into a document, scraping news headlines, running arxiv searches and saving abstracts. Each source lived in its own chat session because context windows kept getting polluted. By the time I finished with Reddit, I’d forgotten what I found in the news search.

Then I created individual skills for each source. One skill for Reddit research, another for news scraping, a third for arxiv papers. Each skill worked fine on its own, but I was still the one coordinating between them. I’d run the Reddit skill, save the output, run the news skill, save that output, run the arxiv skill, save that output. Then I’d manually combine all three into a final idea list.

I was doing the agent’s coordination work manually. The AI could do each step well. Handoffs were the problem. I was the middleware.

How to Correctly Carry Context Forward

Workflows are sequences of steps where each step produces something the next step needs. What separates workflows from prompting is that context moves forward automatically instead of you carrying it by hand.

Anthropic’s “Building Effective Agents” guide, published in December 2024 and widely cited as the definitive resource, makes a clean distinction. Workflows are systems where LLMs and tools are orchestrated through predefined code paths. Agents are systems where LLMs dynamically direct their own processes.

For non-coders, workflows are the sweet spot. You define the path. The AI does the work at each stop.

Anthropic describes five workflow patterns. In plain English:

Prompt chaining works like an assembly line. Step one’s output becomes step two’s input. Each step stays simple and focused.

Routing sorts different inputs down different paths. Like a mail sorter that sends letters to the right zip code.

Parallelization runs multiple things at the same time. Like having three researchers instead of one.

Orchestrator-workers uses a boss agent that breaks down the work and delegates it to worker agents.

Evaluator-optimizer has one agent do the work and another check it. The first one revises based on feedback.

I call the files that hold it all together handoff files. Each step writes its work down so the next step doesn’t have to guess. Format matters less than the principle. It could be a markdown file, a Google Doc, a structured text block. What matters is that each step produces something the next step can read.

I tried everything for holding context between steps. In-memory variables disappear when the session ends, database entries require setup and maintenance, and shared state files get corrupted when two steps write at once.

Markdown files in Obsidian won because they’re boring and reliable.

Each step in a workflow writes its output to a markdown file, and the next step reads that file. Files sit in a folder structure that mirrors the workflow. When something goes wrong, I open the file and see exactly what step three produced. I trace the problem backward through the chain.

This also gives me something I didn’t expect. I track what each subagent or step did, with links to the specific files it produced. When something sounds fishy in the final output, I open the intermediate files and find where the drift started.

Markdown has practical advantages too. Plain text works everywhere. Files move between systems without conversion. Changes are version-controllable over time. Everything renders nicely in Obsidian, which I already use for notes.

Storing context in a database or shared state mechanism adds complexity, requires setup, and creates dependencies. Markdown files require nothing except a folder and a text editor.

Each step writes its work down. The next step reads what the previous step wrote. Context carries forward through files, not through memory.

Building an AI Workflow Step by Step

Let me show you what this looks like in practice. I’ll use my content ideation workflow as the example, but the structure works for any repeating task. If you want to learn how to architect a workflow in 5 minutes before building, that article covers the planning phase.

Four steps make up this workflow. Each step reads from the previous step’s output file and writes to its own output file.

Step 1: Reddit research

Input: A topic or keyword to search for.

What it does: Searches Reddit for threads where people complain about problems related to that topic.

Output: reddit-findings.md with thread titles, URLs, and key complaints.

Step 2: News scraping

Input: The same topic.

What it does: Searches news sources for articles about emerging tools or trends related to that topic.

Output: news-findings.md with headlines, URLs, and summaries.

Step 3: Arxiv search

Input: The same topic.

What it does: Searches arxiv for papers that hint at new capabilities related to that topic.

Output: arxiv-findings.md with paper titles, abstracts, and relevance notes.

Step 4: Synthesis

Input: All three files from steps 1-3.

What it does: Reads all three files and synthesizes them into a list of article angle ideas.

Output: idea-angles.md with 5-10 potential article topics, each grounded in the research.

Each step gets a clean context with exactly what it needs. Nothing is buried. Nothing is forgotten.

My first attempt at this workflow was ugly. Files on my desktop, a checklist in a notes app, and a lot of copy-pasting held it together. But it was structured. Each step had a clear input and a clear output. The agent didn’t need to remember anything from three steps ago because I gave it exactly what it needed.

Eventually I built one unified skill that handles the whole pipeline. It pulls from Reddit, news sources, and arxiv in sequence, writes each batch of findings to a separate markdown file, then synthesizes all three into a final idea list. The skill runs top to bottom without me copying anything between steps.

Prompting vs. Workflows: The Same Task

Content ideation looks completely different the prompt way versus the workflow way.

The prompt way: You open a chat and ask the AI to search Reddit for complaints about a specific topic. It gives you a list. You copy that list into a document. You open a new chat and ask it to scrape news articles about the same topic. It gives you headlines and summaries. You copy those into your document. You open another chat and ask it to search arxiv for relevant papers. It gives you abstracts. You copy those too.

By the time you’re done, you’ve got three separate chunks of text in a document. Now you need to synthesize them into idea angles. You paste everything into a new chat and ask for ideas. The AI produces a list, but it’s generic. It lost the nuance from the Reddit complaints because they got buried in the combined text. It missed the arxiv findings because they were at the bottom of a 5,000-word prompt.

The workflow way: You run a skill that searches Reddit and writes the findings to a file called reddit-findings.md. The skill then searches news sources and writes to news-findings.md. Then it searches arxiv and writes to arxiv-findings.md. Each file is clean and focused.

The final step reads all three files and synthesizes them into idea-angles.md. Each step gets a clean context with exactly what it needs. Nothing is buried or forgotten.

Clare Liguori’s research at AWS tested five approaches to guiding agent behavior across 3,000 evaluation runs. Simple prompt instructions reached 82.5% accuracy, meaning roughly one in five interactions failed. When she added structured feedback loops, what she calls steering hooks, accuracy hit 100% across 600 runs.

Better structure made the difference, not better prompts.

I tested this myself when comparing how different models handle real Hermes workflows. Models that looked impressive on benchmarks often failed at structured workflows because they overthought simple steps or ignored format constraints. Structure matters more than raw capability.

Where Humans Still Check

Every workflow needs checkpoints, but not every step needs one. Adding review points everywhere turns the workflow into a series of interruptions.

I use decision gates. You only stop where a real choice needs to be made. Which angle to pursue. Which source to prioritize. Whether to cut a section that doesn’t fit.

If the output is fine and no decision is needed, you don’t stop. Workflows run until they hit a point where they can’t proceed without your judgment.

Decision gates check whether the output matches your intent. AI produces grammatically correct, well-researched content that still goes in the wrong direction. Decision gates catch that before the next step builds on a mistaken assumption. I wrote about why accepting AI’s first answer is bad and how checkpoints prevent drift.

I wrote a full guide on adding approval gates to Hermes workflows if you want the technical details. Gates protect your reputation by blocking external actions without your OK, protect your data by requiring confirmation before system changes, and protect your wallet by blocking spending above a threshold without approval.

For most workflows, you need one gate at the point where the output becomes public or irreversible. A content workflow might have a gate after the outline, before the final draft goes live. A research workflow might have a gate after the synthesis, before you act on the findings.

Decision gates are where you stay in control of direction while the AI handles execution.

Where to Start Your First Workflow

Pick one repeating task. Not the most complex one. Pick the one you do every week that takes 45 minutes and makes you want to scream by minute 30. That’s your first workflow.

Mine was a morning briefing that pulls tasks and articles before coffee. Two steps. Read from Asana, format the output, deliver it. Simple enough to build in an afternoon, useful enough to run every weekday since I built it.

If you’re new to Hermes, start with a two-step workflow like this one before attempting anything complex.

Minimum viable workflows have four parts: input (what goes in), instructions (what the agent does), output (what comes out), and checkpoint (where you verify). You don’t need software. You don’t need code. You need a folder with files in it.

Anthropic’s own advice from “Building Effective Agents” is to start simple and add complexity only when needed. They explicitly warn against starting with frameworks or complex architectures. Start with two steps. Make them reliable. Then add a third.

Confluent’s guidance on AI workflows makes the same point. Simple solutions are often the best place to begin. Starting with simple prompt engineering may not be perfect, but it works well enough as a first pass. When you hit the ceiling, add structure. Don’t add structure preemptively.

Boring beats clever. Your first workflow should be so simple it’s embarrassing. A two-step process with a file handoff and a human check. That’s it. People who get value from AI workflows built boring ones and ran them 50 times, not impressive ones they ran twice. I wrote about why AI makes you build too much and how to resist that urge.

Most AI productivity advice tells you to write better prompts. Designing better handoffs is where the real payoff lives. Prompts at each step can be mediocre if the context they receive is clean. A brilliant prompt in a bloated chat thread will still produce mediocre output.

If this changed how you think about AI workflows, share it with someone who's still copying between chat tabs. Subscribe for more practical agent workflows every week.

Recognizing when you’re doing coordination work the AI should handle is the whole shift. Once you see the pattern, you can’t unsee it. Every repetitive task becomes a candidate for structure. Every manual handoff becomes a design problem.

Hitting a ceiling with prompting means you have an architecture problem. Build the pipeline. Let the context flow. Keep your hands on the decisions that matter.

Why AI Benchmarks Fail Real Hermes Agent Workflows

cucoleadan — Tue, 02 Jun 2026 13:14:14 +0000

This post was originally published on my Substack publication as Why AI Benchmarks Fail Real Hermes Agent Workflows.

The day after Opus 4.8 launched, I gave it a job that should've taken two minutes. Find a named file, summarize it under a strict word limit and return the result in a specific format so the next step in the pipeline could parse it correctly.

Opus handled it with clean output and solid reasoning, but it took its time making sure every move was right. By the time it finished, a cheaper model would've done the same work three times over.

That was the moment I stopped trusting benchmarks. A leaderboard score tells you how a model performs on a clean task under controlled conditions. It says nothing about whether that model can survive a twenty-step workflow.

I've wanted to put models through a proper test for a while now. I finally found the time and the right config to do it. Here's how I run every model through the same real tasks, and what four models taught me about real Hermes work.

In this article:

Tool call discipline matters more than reasoning quality. A model that calls the right tool once beats one that explores and verifies three times over.
Route by task, not by preference. Use the cheapest model that reliably finishes the job, and step up only when it fails.
A simple testing framework that puts models through real Hermes tasks instead of synthetic benchmarks.

Bottom line: There is no single survivor. I route lightweight models (GLM 5.1) for fast background tasks, capable mid-tier models (Qwen 3.7 Max) for complex workflows, and flagships (Opus) exclusively for deep debugging when the others fail.

Why AI Benchmarks Fail Agent Workflows

Benchmarks test the thing they can measure cleanly. Can the model solve this logic puzzle, answer this math question, write code that passes these test cases? Those are useful questions for evaluating raw capability.

But agent work needs different skills. The model needs to be obedient, fast, disciplined with tools, and able to stop when the task is done. None of these show up on a benchmark because none of them are easy to measure in a controlled test.

A model can score in the top tier on a reasoning benchmark and still be the wrong fit for unattended workflow automation. It might overthink simple tasks, waste tokens on unnecessary reasoning, or call tools it doesn't need because it's trying to be thorough. In a chat interface that thoroughness feels impressive. In a scheduled job running at 6 AM it means the session times out before the work finishes.

Benchmarks also miss the compounding effect of small failures. A model that adds an extra section, ignores a format constraint, or calls a tool twice when once would suffice. Each is minor on its own. In an agent workflow where each step feeds the next one, minor failures cascade into broken jobs.

In my opinion, ClawEval comes close to a valid benchmark. The paper runs 300 human-verified agent tasks across 9 categories with a Pass^3 rule that eliminates lucky runs. A task only passes if the model meets the success criteria in all three independent trials, which is a meaningfully stricter bar than Pass@3. It's the most serious attempt at realistic agent evaluation I've seen, and it has already spawned related work like WildClawBench that tests agents inside live OpenClaw instances. But that's beyond the point.

My test is less scientific and more practical. I put models through the same jobs I schedule on Hermes, using the same config I run every day, and I watch what happens.

Speed Is a Workflow Cost

Speed matters in two completely different ways depending on how you use the model.

When you're in a session talking to Hermes, time is something you're spending. A model that takes four seconds instead of one is the difference between staying in flow and getting distracted between tool calls. Fast models give that time back to you.

In a cron job, nobody's watching the clock. But speed still matters because fast models tend to produce fewer reasoning tokens per tool call, and that directly affects cost and reliability. Less verbosity means a tighter context window across twenty steps, which keeps context window degradation from compounding into broken jobs by step fifteen.

This is why GLM 5.1 on Ollama Cloud carries most of my daily workload. It's fast enough to feel instant in interactive sessions and tight enough with tokens to keep scheduled jobs cheap and stable. I use it for heartbeat checks, morning briefings that synthesize Asana tasks before coffee, and anything that needs to be fast and correct without deep reasoning. The boring work that makes up most of a Hermes workday.

Speed also compounds with cost. A fast mid-tier model that finishes a workflow in three minutes is cheaper than a slow flagship that takes fifteen minutes. The cost math behind this is what I've been tracking across providers and it's where a fast model pays for itself.

For tasks that require massive token input and a lot of tool calls to read and process files, I reach for DeepSeek v4 Flash. When a job needs to chew through a hundred thousand tokens across a dozen files, speed is the only thing that keeps the session from becoming an exercise in patience.

Tool Call Discipline Beats Reasoning

This is the sharpest opinion I hold about models inside agent loops, and it's the one benchmarks almost never test.

Benchmarks measure whether a model can think hard about a problem. Hermes needs a model that calls the right tool once, reads the result, and moves on without second-guessing itself. Those are different skills, and the second one matters more for unattended work.

A model that makes twelve tool calls when four would do is being expensive and fragile. Every extra call adds API cost, creates another failure point, and fills the context window with noise the model has to process on the next step. Most of what people call "context engineering" inside an agent loop is just preventing this kind of noise from ever entering the window in the first place.

I've seen top-tier reasoning models call a search tool, read the results, then call a different search tool to verify what the first one returned, then call a third tool to format the output when a simple string operation would've worked. The net effect was a session that cost three times as much and took three times as long.

The same failure shows up with instruction obedience. A model that ignores format constraints and adds helpful extra sections breaks downstream parsing. A model that keeps writing after the task is done wastes tokens. A model that skips a negative constraint includes something you told it to avoid at the worst possible moment.

In a chat, each of these looks like helpfulness. In an agent workflow, each one becomes a liability because each step feeds the next one.

Tool call discipline separates a model I trust with unattended work from a model I keep supervised. A disciplined model reads the task, decides which tools it needs, calls each one once, and stops. An undisciplined model explores and adds helpful extra steps nobody asked for.

If you're running Hermes with approval gates, tool call discipline becomes even more important. A model that makes unnecessary tool calls also tends to ignore "ask before destructive action" instructions. The model thinks it knows better than the prompt.

This also ties into the interface question I've been tracking — a disciplined model calls the right tool the right way, and sometimes the right tool is a lightweight CLI instead of a bloated MCP server that chokes the context window before the model even starts thinking.

How I Route AI Models in Hermes

Model selection inside an agent loop is a routing problem, not a ranking problem. There's no single best model. There's a best model for each job, and the job changes throughout the day.

GLM 5.1 carries most of my workload. Heartbeat checks, simple scheduled jobs, structured data parsing, quick research tasks. These are tool calls that need to be fast and correct but don't need deep reasoning. GLM on Ollama Cloud is cheap enough that I don't think twice about spinning up a session. The boring work that makes up the bulk of a Hermes workday.

GPT-5.5 is what I use through my Codex subscription for the heavy work. I tried it inside Hermes first and it burned through usage by making tons of unnecessary tool calls. The model doesn't know when to stop, which makes it terrible for agent loops where each tool call costs tokens and fills the context window. So I stopped using it in Hermes and shifted it to Codex, where I control the loop. In Codex it handles most of my coding and the research pulls where I want thorough coverage. It is more verbose than the other models in my stack, which helps for research and hurts for tight format constraints, so I shape the prompt accordingly. The subscription removes cost as a gating factor, which means I can run it as often as the job needs without watching a counter.

Opus gets the tasks GPT 5.5 can't finish. I run it through the API and it's expensive, so I keep it scoped to debugging. When a problem doesn't reproduce cleanly or when a code review needs a different reasoning style, Opus handles it. That's it. Outside of those sessions it stays idle.

Qwen 3.7 Max is my surgical tool. I reach for it when a task needs more reasoning than GLM can deliver but I don't want to pay Opus prices. The step up from the cheaper Qwen tier is noticeable on tasks involving multi-step logic or ambiguous instructions. The cheaper version guesses and moves on. 3.7 Max pauses and works through it. For most structured agent work, the cheaper version gets the job done. I use 3.7 Max sparingly and mostly for content and deep research.

My pattern is simple. Use the cheapest model that reliably completes the specific task type. When GLM fails, step up. When the mid-tier fails, step up again. Escalation stays task-driven, not model-driven. Pricing only matters once the model can finish the job.

The first rule of routing is reliability. The second rule is cost. Get them in the wrong order and you'll pay for it.

I pay per token every time Hermes calls a tool, so the cost math matters. Every tool call generates input tokens from the context window and output tokens from the response. A session with a dozen tool calls consumes as many tokens as a long chat conversation.

My AI Agent Evaluation Ladder

I don't trust leaderboards because they don't test what I need. So I run models through the same set of real tasks from my actual work. Same files, same prompts, same conditions for every model so the comparison stays honest. Each task tests a different dimension of what I need from an agent model.

The tasks escalate from simple to demanding.

First, whether the model stays restrained when given no task. A good Hermes model sends a short greeting back and waits. A bad one starts searching files and scanning memory before you ask it to. The model that won't wait is the model that won't stop.

Then whether it finds a document and summarizes under strict format rules without drifting into extra sections. I cap the summary at a specific number of bullets with a word limit on each one. The task doesn't require deep reasoning. It requires the model to follow directions and stop.

Whether it packages a CLI tool into a reusable skill without overbuilding. Some models create five files when one would do. The disciplined model reads the help output before it writes anything.

Whether it uses that skill correctly in a fresh session with specific source rules. No Reddit, no arXiv, no turning search snippets into facts. This tests whether the model can follow negative constraints, which are harder than positive ones because the model has to actively suppress the instinct to include everything it finds.

And finally, whether it pulls together sources, cross-checks claims, and produces a decision-ready report under a word limit. This combines everything. Format constraints, tool judgment, instruction obedience, and the discipline to stop writing when the task is done.

That last requirement is the one benchmarks never test. In a chat, extra writing is harmless. In an agent workflow, extra writing is a tax on every step that follows.

What's Coming Every Thursday

I'm running Opus, GPT-5.5, Qwen 3.7 Max via OpenCode Go, and GLM 5.1 via Ollama Cloud through these tasks and publishing a short verdict card every Thursday. Each card covers the best use case, the conditions where you should skip it, cost notes from real sessions, and whether the model belongs in a Hermes setup at all.

The goal is a repeatable testing standard that accumulates over time instead of a one-off leaderboard that goes stale the week after publishing.

I'll also note when a model fails because of provider instability rather than model quality. A weak model needs replacing. An unreliable route needs a backup. I've hit this with Hindsight memory timeouts and gateway drift — things that look like the model broke but were actually a layer below it.

Each Thursday card follows the same format so you can compare over time. The verdict will be one of five categories: daily driver, strong specialist, background worker, backup only, or skip entirely. I'll add new tasks to the ladder based on what people suggest in the comments.

Stay tuned and subscribe to receive these reports as soon as I publish them.

Share this with a builder choosing their next AI model from a leaderboard or plan page.

Which model should go through the ladder first, and what real workday task should I add to the test? Let me know in the comments.

My Hermes AI Agent Maintenance Routine For Maximum Reliability

cucoleadan — Tue, 26 May 2026 12:59:38 +0000

This post was originally published on my Substack publication as My Hermes AI Agent Maintenance Routine For Maximum Reliability.

Last week, I spent a few days blaming the model before I realized Hermes was waiting on a memory recall timeout.

When the response time got worse, I assumed provider latency because I'd changed models before and knew that layer could get noisy.

The real problem sat one layer earlier, inside the retrieval path I hadn't checked yet.

My external memory provider, Hindsight, threw a retrieval error, Hermes retried, and the request stalled because the memory system was broken before the model ever had a chance to answer.

A few days later, my Friday Hermes health-summary job missed its Telegram report over a long weekend. The stack still answered messages, but the missing report told me the scheduled workflow had stopped producing the artifact I expected to see.

Hermes maintenance means checking the layers around the model before you blame the model. The routine I use now is a set of cron-backed prompts that check memory, gateways, scheduled jobs, model IDs, and backups, then stop before they make changes that need approval.

Most install guides skip this part because they get you to the first successful command, then leave you with a working AI control plane and no maintenance loop around it.

Hermes feels like one system when it works, but it routes through models, memory, gateways, skills, cron jobs, provider keys, and local files, so the fault can sit in any one of those layers when the stack starts behaving strangely.

And don't get me wrong, I've never had a single issue with the actual Hermes code compared with my time using OpenClaw, but I have had issues with models, providers, and third-party integrations.

The model gets blamed first because it's the visible part of the stack, while the failure usually starts somewhere less obvious.

This article is the maintenance routine I use now, rewritten as prompts you can hand to your agent.

In this article:

A maintenance routine you can run after Hermes is installed, so silent drift doesn't turn into a broken workflow.
Copy-paste cron-job prompts for daily, weekly, and monthly checks across memory, gateways, scheduled jobs, providers, and backups.
A simple approval rule that lets agents report problems without giving them permission to delete, update, rotate, restore, or rewrite anything.
A rollout path for turning maintenance into useful visibility instead of another noisy automation.

After Hermes Install

The first successful Hermes run can trick you into treating setup as finished before operations have even started. You connect a provider, configure the gateway, test memory, send a message through Telegram or the TUI, and watch Hermes answer with context from the project you care about.

That moment is where the stack leaves the install guide and becomes something you have to run. Old configs can keep stale model names, scheduled jobs can miss their expected output, memory calls can slow down, and backups can look comforting until the first restore test fails.

I treat those failures as normal infrastructure behavior because a control plane becomes trustworthy only after you can see whether its dependencies are still healthy.

That lesson showed up during my OpenClaw to Hermes migration, even though the migration itself went smoothly. The first week felt better because Hermes followed instructions more closely, kept memory behavior cleaner, and made the gateway setup feel less stitched together.

The first problems were small enough to ignore in the moment but specific enough to matter later. An imported publishing skill failed because its YAML header was malformed, one environment variable was missing from the runtime, and token usage climbed while memory ingestion ran behind the workflow I was paying attention to.

None of those problems killed the setup, but each one pointed at the same operational truth: the model is only one layer inside a wider system. I stopped treating maintenance as an occasional chore once I realized a scheduled prompt could check those layers before the next failure stole an afternoon.

Cron Prompts Beat Commands

The earlier version of this routine had shell commands sprinkled through the article because that was how I checked my own server. Commands are useful when your environment matches mine, but they don't travel cleanly across Windows, Linux, Docker, hosted runners, local agents, and the custom glue every serious stack accumulates over time.

The official Hermes Agent docs are where I would start for setup details. This piece starts after setup, when the question changes from "Can Hermes run?" to "Can I trust this workflow tomorrow?"

Prompts travel better because they describe the job instead of assuming the tool. A cron-backed agent can inspect logs, check timestamps, call a gateway, read a config file, compare recent output, or ask for approval using the tools available inside its own environment.

If the maintenance prompt needs to reach outside Hermes, the same decision from When to Use MCPs, CLIs, or Your Own Tool applies here: use the smallest interface that can inspect the system cleanly without turning one check into a brittle integration project.

A scheduled prompt still needs firm boundaries because a useful maintenance job names the layer being checked and asks for evidence before it reports confidence. The report should be readable at a glance, but the agent should refuse to delete, update, rotate, restore, or rewrite anything without approval.

That boundary turns maintenance automation into a reporting system instead of a new source of damage. I want the agent to notice problems before I do while every irreversible action still comes back to me as a decision.

Three Maintenance Layers

My Hermes maintenance routine uses three layers that map cleanly to the way the stack fails: updates, cleanup, and health checks. Those labels keep the job concrete enough for a scheduled agent to report on the system without turning the prompt into a vague request to "check Hermes."

This is the operational side of the $30 Hermes stack, because a cheaper and more flexible agent setup only stays useful if the layers around it keep working.

The update layer asks whether something changed underneath the workflow while I was focused on using it. Providers rename models, preview routes become stale, plugins move, skills change formats, and memory backends update their APIs.

The cleanup layer asks whether the stack has accumulated enough junk to start changing behavior. Logs grow, sessions pile up, cached files stick around, and memory keeps old context long after the project has moved on.

The health-check layer answers the operational question before I start relying on the stack again. Before the workday starts, I want evidence that the gateway answers, the provider route works, scheduled jobs are producing output, and memory can retrieve a recent decision without timing out.

The layers keep the routine small enough to survive a busy week without reducing the review to a shallow status ping. Maintenance disappears when it depends on a vague intention, while a scheduled job with named layers can keep running after the calendar gets crowded.

Daily Hermes Health Check Prompt

The daily job should be boring enough that you can read it every morning without turning the start of the day into a debugging session. Its job is to tell you whether the stack is ready for work, then stop before it tries to repair anything.

Use this as a read-only cron job near the start of the workday, then adapt the gateway name, job names, and project references to match your own setup.

Create a Hermes cron job called "Daily Stack Pulse" that runs every morning at 8:00 local time, delivers to origin, and uses a cheap model (gemini-3.1-flash-lite via openrouter, or deepseek-v4-flash via opencode-go — pick whichever is configured). Restrict toolsets to terminal and web. Use this exact prompt body for the job:

---
Run a daily read-only Hermes stack pulse check. Make no changes: do not delete files, rotate keys, update packages, prune memory, restore backups, or rewrite configuration.

1. Gateway. Send or simulate one normal request through the Telegram gateway and confirm it responds.
2. Scheduled workflows. Run `hermes cron list` and inspect ~/.hermes/cron/output/ for the latest runs of jobs tagged or named for morning briefing, health summary, memory maintenance, publishing, client, or paid workflows. Confirm each ran inside its expected window.
3. Logs. Scan recent warnings and errors from the Hermes runner (~/.hermes/logs/), the model provider, the memory layer (hindsight), the gateway, and the scheduler.
4. Memory recall. Run one hindsight_recall query against an active project decision (use "All Agents Considered newsletter" or "Vibe Stack Lab library repo"). Report whether the result was relevant, stale, missing, or slow.

Return a short report with exactly these sections, one sentence per item:

PASS:
Healthy checks with evidence.

WARN:
Items needing attention later, with the layer named in parentheses.

FAIL:
Broken or missing items that block reliance on the stack today.

APPROVAL NEEDED:
Any action that would delete, update, rotate, restore, rewrite, prune, or change provider behavior. Name the action and layer. Do not execute.
---

After creating the job, run it once immediately so we can see the first report, then confirm the job ID and schedule.

The report matters more than the scheduler that happens to run it, as long as the result gives you enough evidence to trust or question the stack. You can run the prompt from cron, a recurring Hermes task, a hosted automation, a CI runner, or any agent runner that has permission to inspect the stack.

I care most about evidence that the gateway answered, the important jobs ran, memory recall still works, and recent errors haven't turned into a pattern. Once the report names the failed layer, the next step becomes smaller because the investigation has a place to start.

Weekly AI Agent Drift Review Prompt

My quiet cron failure is the reason I care more about weekly drift than a one-time setup checklist. A job definition sitting in a scheduler proved nothing once the Friday health-summary report stopped reaching Telegram.

That is the same reason my morning Hermes workflow checks visible output instead of trusting that a scheduled task exists somewhere in a config file.

The weekly review looks for slow changes that don't announce themselves while normal work still appears to be moving. Disk pressure, stale output, growing logs, slow memory, and old model IDs rarely feel urgent while they are accumulating, but they become expensive once they pile up inside a broken workflow.

Use this prompt near the end of the week, when the report can shape a short maintenance pass instead of interrupting deep work in the middle of a day.

Create a Hermes cron job called "Weekly Drift Review" that runs every Sunday at 9:00 local time, delivers to origin, and uses a cheap model (gemini-3.1-flash-lite via openrouter, or deepseek-v4-flash via opencode-go — pick whichever is configured). Restrict toolsets to terminal and web. Use this exact prompt body for the job:

---
Run a weekly read-only Hermes drift review. Make no changes. If a fix is obvious, list it under RECOMMENDED ACTIONS or APPROVAL NEEDED but do not execute.

1. Storage growth. Measure size of ~/.hermes/logs/, ~/.hermes/sessions/, ~/.hermes/cache/, ~/.hermes/memory/, ~/.hermes/cron/output/, /tmp/hermes*, and any backup folder under ~/.hermes/. Compare to last week if a snapshot exists at ~/.hermes/cron/output/drift-snapshot.json. Save a fresh snapshot at that path after measuring. Flag any folder that grew more than 25 percent or crossed 1GB.

2. Scheduled jobs. Run `hermes cron list`. For each job, confirm it exists, has run inside its expected window, and produced a visible artifact in ~/.hermes/cron/output/ or the delivery channel. A job definition with no recent run counts as broken.

3. Memory recall. Run three hindsight_recall queries: one active project ("All Agents Considered newsletter"), one older project ("Build It #2 AI Code Review Agent"), one recent decision ("Vibe Stack Lab library repo"). Report each as accurate, stale, empty, or slow.

4. Provider and model config. Read ~/.hermes/config.yaml. Flag preview or dated model names (anything with -preview, -beta, dated suffixes, or matching known-deprecated IDs), fallback routes pointing at old IDs, and project-level overrides under ~/.hermes/profiles/*/config.yaml that diverge from the main config without obvious reason.

5. Logs. Scan the last 7 days of ~/.hermes/logs/ for repeated errors, retry loops, auth failures, timeouts, and missing-env-var messages. Group by layer (runner, provider, memory, gateway, scheduler).

Return a report with exactly these sections:

DRIFT:
Storage growth and configuration drift observed this week.

BROKEN:
Jobs, routes, providers, memory calls, or gateways that failed and need repair. Name the layer.

STALE:
Model IDs, project configs, skills, outputs, or memory entries that look outdated.

RECOMMENDED ACTIONS:
Small proposed fixes. For each: action, risk (low/med/high), expected benefit, approval needed (yes/no).

APPROVAL NEEDED:
Anything that changes files, deletes data, updates Hermes, rotates keys, changes providers, prunes memory, restores backups, or edits scheduled jobs. Do not execute.
---

After creating the job, run it once immediately so we can see the first report, then confirm the job ID and schedule.

That weekly prompt would have caught my quiet cron failure earlier because a cron entry sitting in a file doesn't prove the workflow is alive. The agent has to find the last run, the last output, or the last expected message before it claims the job is healthy.

The same weekly review helps with memory issues because recall drift often feels like model weakness from the outside. When retrieval returns stale or empty context, the report should call that a memory-layer problem before anyone starts blaming generation quality.

Monthly Hermes Assumptions Review Prompt

The monthly job checks whether the assumptions under the stack still hold after weeks of normal use. Provider behavior, model IDs, permissions, backups, and release notes deserve a slower review because mistakes in those layers can create bigger messes than a missed daily report.

Run this one when you have enough time to read the report and decide what should change, because the monthly review is the one most likely to recommend actions that touch live state.

Create a Hermes cron job called "Monthly Assumptions Review" that runs on the 1st of every month at 10:00 local time, delivers to origin, and uses a cheap model (gemini-3.1-flash-lite via openrouter, or deepseek-v4-flash via opencode-go — pick whichever is configured). Restrict toolsets to terminal, web, and file. Use this exact prompt body for the job:

---
Run a monthly read-only Hermes assumptions review. Make no changes: do not update Hermes, change providers, rotate keys, restore backups, prune memory, delete files, rewrite configs, or edit scheduled jobs.

1. External change summary. Check for changes that could affect this stack in the last ~30 days:
   - Hermes Agent: `cd ~/.hermes/hermes-agent && git log --since="30 days ago" --oneline` and check release notes
   - Plugins and skills: list anything in ~/.hermes/plugins/ and ~/.hermes/skills/ modified in the last 30 days
   - Provider changes: scan OpenRouter and opencode-go model lists for renamed, deprecated, or newly preview-flagged IDs that match anything in ~/.hermes/config.yaml
   - Gateway, memory backend (hindsight), scheduler, and backup tool changelogs if accessible
   Summarize only changes relevant to this stack.

2. Provider and model ID audit. Grep every config layer for model IDs:
   - Main: ~/.hermes/config.yaml
   - Profiles: ~/.hermes/profiles/*/config.yaml
   - Cron jobs: ~/.hermes/cron/jobs.json
   - Skills referencing models: search_files for "model:" or model IDs under ~/.hermes/skills/
   - Scripts under ~/.hermes/scripts/
   - Env files: ~/.hermes/.env and any *.env
   Flag preview IDs (-preview, -beta, dated suffixes), known-deprecated IDs, missing fallbacks, and defaults that conflict between layers.

3. Health sweep. Quick check across:
   - Gateway response (one Telegram round-trip)
   - Provider reachability (one ping each to configured providers)
   - Memory recall (hindsight_recall on an active project)
   - Scheduler activity (hermes cron list plus recent output)
   - Storage headroom (df -h on ~/.hermes/ partition)
   - Backup completion (most recent backup artifact timestamp and size)
   - Key availability (env vars and 1Password references exist, not the values)
   - Permissions (~/.hermes/ ownership and mode)

4. Restore test. Pick one non-sensitive backup artifact under ~/.hermes/backups/ or wherever backups land. Copy to /tmp/hermes-restore-test/, inspect contents, confirm it opens and matches expectations. Do not overwrite live files. Delete the temp copy after inspection.

5. Approval-gate review. List every workflow (cron job, skill, plugin, script) that can delete files, prune memory, rotate keys, change providers, restore backups, update Hermes, edit configs, or send messages outside this workspace. For each, confirm whether it requires explicit approval or runs automatically.

Return a report with exactly these sections:

ASSUMPTIONS STILL VALID:
Operational assumptions that still look safe.

ASSUMPTIONS TO RECHECK:
Provider, memory, gateway, scheduler, backup, or permission assumptions that may have drifted. Name the layer.

RESTORE TEST:
Artifact inspected, safe location used, and result.

PROPOSED CHANGES:
Each with reason, risk (low/med/high), rollback notes, approval status.

APPROVAL NEEDED:
Every action that would modify the stack or touch live data. Name the action and layer. Do not execute.
---

After creating the job, run it once immediately so we can see the first report, then confirm the job ID and schedule.

I review provider model IDs here instead of waiting for a stale preview route to break under load. A fallback route in an old project config can keep calling yesterday's model even after the main Hermes provider has moved to the stable ID.

The Hindsight timeout became confusing because the symptom pointed at the wrong layer. Hermes felt slow, I blamed the model, and the retrieval path had already burned the time before generation started.

Approval Gate for Maintenance Jobs

Every scheduled maintenance job should carry the same approval rule because the boundary gets easy to forget after the first few reports look useful. Read-only inspection can run freely, while destructive or identity-changing work still needs a human decision.

If you haven't built that habit yet, start with the approval gate setup before you let a maintenance prompt touch files, providers, keys, or backups.

Add this block to the end of every maintenance prompt that runs on a schedule, especially if the agent has access to files, keys, backups, provider settings, or outbound channels.

Approval rule for this maintenance job:

You may observe, inspect, summarize, classify, and recommend without asking first.

You must ask for approval before any action that deletes files, prunes memory, rotates keys, changes providers, restores backups, updates Hermes, edits configuration, changes scheduled jobs, rewrites prompts, sends external messages, or changes permissions.

When approval is needed, return a proposal with the issue, suggested action, expected benefit, risk level, affected files or systems, rollback notes, and the exact command or tool call you want to run.

If the risk is unclear, classify the action as approval needed and wait.

That rule keeps the maintenance agent useful without letting it become a cleanup bot with too much confidence. The agent can prepare the decision, but I still want to make the decision when live state changes.

Quiet Agent Failures

The failures that cost time are small enough to miss and specific enough to blame on the wrong thing. My cron failure didn't crash the stack because it stopped doing work in a corner I wasn't watching.

The model ID drift behaved differently because the main provider setup looked current while an older route still pointed somewhere stale. The visible symptom showed up as slower Hermes responses and memory behavior that looked worse than it was.

The Hindsight timeout changed how I diagnose agent slowness in every workflow that depends on memory. When an AI tool slows down, I check the retrieval chain before I blame the model because the model may be downstream from the delay.

Maintenance doesn't prevent every failure, but it reduces the time spent accusing the wrong layer. Once you can name whether the issue sits in routing, memory, scheduling, storage, backup, skills, or config, the repair becomes less mysterious.

How to Roll Out the Routine

I would start with one weekly maintenance job before adding daily and monthly jobs. Weekly reporting is frequent enough to catch drift, and a month of reports gives you enough signal to decide whether the daily pulse is worth the extra noise.

Once the weekly report proves useful, add the daily pulse for the pieces you depend on most. My daily set covers gateway response, scheduled job output, memory recall, and provider reachability because those failures change whether I can trust the stack that morning.

The monthly review should stay slower and more deliberate because updates, provider IDs, backup restores, and permission gates need more attention than a quick morning report can give them.

Your stack may use different names, but the shape should stay the same. The scheduled agent observes the stack, reports the failed layer, proposes small actions, and stops before touching anything that could create real damage.

Failure Limits

Maintenance won't make the stack perfect, and the prompts shouldn't pretend they can. Provider outages, weak retrieval, bad project context, poor model fit, and bad release notes can still turn into manual work.

The routine also leaves approval gates in place for every action that changes live state. If Hermes wants to prune memory, change providers, delete logs, rotate keys, restore a backup, or update itself, I still want to approve that action before it touches anything real.

That boundary keeps the routine useful because the agent can notice problems before I do, while every action that changes the system comes back as a proposal I can read.

Bottom Line

Hermes feels like one system when it's working, but underneath it's a control plane sitting on top of models, memory, gateways, cron jobs, files, skills, providers, and backups. When one layer drifts, the whole experience gets worse even if the visible symptom looks like a slow model or a lazy agent.

The maintenance loop keeps those layers visible through a daily pulse, a weekly drift review, and a monthly assumptions review. For most personal agent stacks, that rhythm is enough to know where to look when something breaks.

Start with the weekly prompt and run it long enough to see whether the reports change your behavior. If the reports help you catch missed jobs, stale model IDs, slow memory, or backup gaps, add the daily pulse and monthly review around the same approval rule.

The install guide gets Hermes running, and the maintenance loop is what keeps it worth trusting.

This post was originally published on my Substack publication as My Hermes AI Agent Maintenance Routine For Maximum Reliability.

I Tested 6 AI Plans to Find What $5, $10 and $20 Get You

cucoleadan — Tue, 19 May 2026 14:10:55 +0000

This post was originally published on my Substack publication as I Tested 6 AI Plans to Find What $5, $10 and $20 Get You.

A little while ago, I built a multi-step workflow in Hermes to generate a ten-page report that would get stronger each time it passed through the document. It checked the latest news, then read through Reddit threads, then cross-checked with X and also read through a bunch of internal documents.

For most of the run, it worked the way I wanted, and Hermes kept moving the file forward while pulling in the context it needed and holding onto the thread of the job.

By the time it reached the last stage, somewhere around the fourteenth tool call, it already had the material it needed and only had to stay coherent long enough to verify the details and write the final section cleanly into the file.

Then it just stopped in the middle of the edit. It retried enough times to trigger a context reduction right when the report needed the fullest possible view of everything that had already happened. The fact that I had to step back in and rebuild the whole thread was extremely annoying and the reason why I decided to write this article.

That was also the moment I started focusing on reliability rather than judging AI plans by the model menu.

Pricing pages encourage you to compare plans by the names they advertise, but Hermes forces a more practical question, which is whether a plan can carry real work through a messy session without handing it back to you halfway through.

Once I started looking at plans that way, I cared a lot less about whether a subscription included a famous model and a lot more about whether Hermes could finish the work before my own attention became the most expensive part of the workflow.

I have paid for enough AI accounts to know how misleading a low sticker price can be. A five-dollar plan stops feeling cheap the moment it burns an hour of focused work.

Not to mention that most twenty-dollar plans might feel like they come with extra usage compared to their cheap alternatives, but that is not usually the case. Looking at you, Anthropic.

That's the frame for this piece, because I rechecked the official pricing pages on May 19, 2026, and I want to show you these prices through an AI agent lens rather than focusing on their sales copy.

In this article:

Why model names and benchmark scores are the wrong way to judge an AI plan
How one $5 plan became my daily driver after I fixed my routing
Why the $10 tier is where most plans start to make real sense
What the big brand names ($20 tier) actually limit once you push them
Where plans break mid-session and how cost per useful hour flips the math
The exact stack I would buy today and which plans I would skip

The One Test That Picks Winners

Benchmarks tell you how a model performs in isolation, but Hermes shows you something much harder to fake, which is whether a plan stays useful once the session fills with tool calls, file reads, and the usual clutter that comes with trying to finish real work.

My test now feels much simpler than any leaderboard, because all I really have to do is give Hermes one job from a normal week and watch how much of my own attention it gives back to me by the end.

If Hermes gets to a result I can keep, the plan earns its place. If the session breaks, the model loses the thread, or I have to step back in for cleanup, the plan gets more expensive no matter how cheap the subscription looked when I bought it.

$5: Where Most People Get It Wrong

The five-dollar tier starts with OpenCode Go, and it stands out immediately as it's the only subscription I found that gives you a real first month instead of a throwaway trial.

Right now, OpenCode Go is $5 for the first month and $10 after that, and it works in Hermes by default, which matters because it feels like a provider route built for agents instead of a chat plan stretched into agent work after the fact.

What changed my view of this plan is that it did not stay a cheap side route for long. It became my daily driver, even during the stretch when I was still paying for three subscriptions just to keep up with my usage.

At the time, the real problem was not the plan itself but the way I was using it, because I kept pushing the same model through every kind of Hermes task and expecting it to behave well no matter what the work looked like.

For a while I ran Qwen 3.6 Plus for almost everything, and that worked badly enough that I ended up compensating with more subscriptions instead of better routing.

The setup only started to make sense once I matched the model to the job, with DeepSeek V4 Flash and V4 Pro taking most of the regular Hermes work while Gemini 3.1 Flash Lite via OpenRouter handled image analysis more cleanly than the routes I had been forcing before.

OpenCode Go became much more useful once I stopped treating one model like a universal answer and started treating the plan like a routing layer for different kinds of work.

I still think the five-dollar month is the right place to learn this lesson, since it is cheap enough to experiment with and real enough to show you very quickly whether your workflow is efficient or just patched together.

$10: The Real Starting Line

The $10 tier is where most of these plans start to feel normal, since the $5 and sub-$5 options are mostly gone now outside of special promos.

That is also the first tier I would take seriously for regular Hermes use.

After the first month, OpenCode Go lands here at its regular price, and MiniMax Token Plan Starter shows up at the same $10 with 1,500 M2.7 requests every 5 hours.

On paper, that sounds like a clean comparison. In practice, I care much less about the headline limits and much more about what the workflow feels like once Hermes is doing the work.

MiniMax Starter gives you a dedicated M2.7 bucket, which is useful if you already know that model is good enough for most of your week and you want limits that are easy to reason about.

OpenCode Go works differently, since it gives you a shared routing budget across several model families, and that can look better or worse depending on what kind of week you're having.

If you mostly run MiniMax M2.7 through Go, the published estimates are higher at around 3,400 M2.7 requests every 5 hours for the same monthly price, so it can look cheaper than MiniMax Starter on raw throughput alone.

Still, that is not what would decide it for me.

I would judge the whole tier by loop quality more than by the model list or benchmarks. Sometimes I hit 503 errors on Qwen 3.6 Plus through OpenCode Go, and other times the tokens per second I got through Go were clearly better than what I was getting from MiniMax directly. And I absolutely hate it to wait for AI to answer. I'd rather have a faster model than a smarter model, but that's just personal preference.

What matters most to me is whether it keeps moving after the first answer, uses tools cleanly, and keeps its replies short enough that the session stays readable while the work is still in progress.

$20: Brands You Know, Limits You Don't

The $20 tier is where the familiar companies start showing up.

OpenAI and Anthropic are the obvious ones, because they are the subscriptions most people already know. Ollama belongs in the same conversation for a different reason, as it's one of the few open-model companies that already feels big enough to sell a hosted plan without sounding like a side project.

That matters because this tier is not only about extra usage. It is also about how much trust people attach to the company behind the plan, and whether that trust survives contact with the actual limits.

ChatGPT Plus is the default benchmark. OpenAI lists Plus at $20 per month, says it gives higher GPT-5.5 limits inside ChatGPT, and keeps API usage separate from the subscription.

You can count Plus in the real stack because Hermes supports OpenAI Codex through ChatGPT OAuth, but the plan still buys ChatGPT access rather than API credit. The limit story is also less generous than the branding makes it feel. OpenAI says Plus users can send up to 160 GPT-5.5 messages every 3 hours, and manual GPT-5.5 Thinking has a weekly limit of up to 3,000 messages. That is fine for normal chat use. It starts looking smaller once you lean on it harder.

Claude Pro has the same advantage and the same problem. Anthropic is a big enough name that people do not need much convincing to try the plan, and Claude is useful enough that plenty of people will keep paying for it anyway. The issue is that the limits are nowhere close to generous for heavy use.

It's just easy to run into the ceiling faster than the $20 price tag suggests, especially once you lean on Sonnet for real work.

Ollama Cloud Pro is more interesting to me because it is not trying to be ChatGPT or Claude. Ollama lists Pro at $20 per month or $200 per year, with larger cloud models, 50x more cloud usage than Free, and three concurrent cloud models.

That sounds strong until you compare how the limit story is presented next to OpenCode Go. OpenCode Go tells you the five-hour, weekly, and monthly caps directly, including a monthly ceiling of $60. Ollama tells you usage is mostly GPU time, gives you five-hour and weekly resets, and lets you run three cloud models at once, but it does not spell out a monthly limit on the pricing page. That makes the plan harder to reason about.

The three-model ceiling also matters more in Hermes than it would in a normal chat app. If you mostly run one agent at a time, it probably feels fine. If you like concurrent agents, background runs, or separate research and writing loops happening together, three can start feeling smaller than the headline suggests.

So yes, Ollama Pro looks good. It is just not automatically better than Go once you care about legibility, concurrency, and what the plan looks like over a full month instead of over a good afternoon.

Nous Portal Plus is less mainstream than OpenAI, Anthropic, or Ollama, but it still deserves the slot because it fits Hermes more naturally than most of the bigger brands. Nous lists Plus at $20 per month with 300+ models, hosted tool usage, and $22 in monthly credits with rollover. I felt that I should include this because they are the team who created Hermes after all.

MiniMax Token Plan Plus is still the simplest volume play. MiniMax lists Plus at $20 per month with 4,500 M2.7 requests every 5 hours plus speech and image quotas. If M2.7 already works for your Hermes load, that is a very direct way to buy more room.

Those are not the same thing, and the difference only shows up once Hermes starts leaning on the plan instead of just chatting through it.

Where Plans Hit the Wall

Hermes exposes plan limits in the middle of real work instead of at the edge of a chat.

A chat cap is annoying when you are asking questions. The same cap inside Hermes can land in the middle of a file edit, a research loop, or a tool run that was finally starting to cohere. Then you lose more than a reply. You lose the state of the job and pay for it again in the next session.

Fallback models create a quieter version of the same mess. A session starts on one route and ends on another, and you can feel it even before you check the model picker. Instruction following gets softer. The agent stops being careful with the same tool path it was following ten minutes earlier.

Tool use is still the cleanest divider for me. A model can sound impressive in a chat window and still be weak inside an agent loop. If it avoids reading files, skips verification, or acts allergic to tools, I do not care how good the brand or benchmark looks. The less glamorous route that checks its work often finishes more jobs per dollar.

Memory changes the value of a plan too. Hermes only starts to feel useful once it can carry a project forward across sessions. If the provider leaves you with a morning reset, the agent never really joins the work. It just keeps reintroducing itself.

That is also why the OpenClaw to Hermes migration mattered so much to me. I was not looking for a smarter chat app. I wanted something that could keep the work moving without making me rebuild the thread every time.

Latency has its own cost. A slow model is fine for overnight cleanup or background chores. It gets expensive the moment you are thinking with the agent in real time and waiting for the next useful move.

The Only Math That Matters

The metric I keep coming back to is cost per useful Hermes hour.

I like it because it is boring enough to be honest.

cost per useful hour = monthly plan cost / Hermes hours that ended in usable work

If a $5 plan gives you ten clean background hours, it is excellent.

If that same plan burns one focused afternoon because Hermes stalls in the fragile part of the job, the cheap price was fake.

A $20 plan can still be the cheaper one if it finishes the sessions you would otherwise have to rescue.

I would not build a dashboard for this. One line in your notes after each session is enough. Write down the plan, the job, and whether Hermes finished without babysitting.

After a week, the pattern usually gets obvious. OpenCode Go might end up doing the background work. MiniMax might carry more of the daily load than you expected. Nous might keep its place because the tool gateway removes setup friction. Ollama might stay as the open-model cloud route. ChatGPT and Claude might remain in the stack because they are still where you think best before sending the work back into Hermes.

That is enough to make the decision. The goal is to stop paying for subscriptions without knowing what job each one is there to do.

Here Is What I Would Buy

If I were rebuilding this stack today, I would still start with OpenCode Go and give it the boring work first.

That is the cheapest place to learn whether the workflow is efficient or just being propped up by extra subscriptions.

I would keep fragile sessions away from it until it earned trust. Cleanup, first-pass research, low-risk drafts, and the kind of work that is useful when it lands but not painful if it misfires.

Once the first month ended, I would treat the $10 tier like the real test. OpenCode Go at full price and MiniMax Starter both deserve a normal week before I let a $20 brand into the stack on reputation alone.

After that, I would only pay for a $20 plan if I knew exactly why it was there. ChatGPT Plus belongs if the ChatGPT or Codex lane matters enough to keep. Claude Pro belongs if Claude is still where the best writing or dev work happens, even with the limits. Nous sits closest to native Hermes work. Ollama Pro belongs if I want the open-model cloud lane and can live with the three-model ceiling. MiniMax Plus is the straightforward volume upgrade if M2.7 is already carrying real work.

That is less satisfying than picking one winner. It is also closer to how the work behaves.

Different jobs deserve different routes. Background chores do not need the same plan as the sessions where one bad restart can waste half an afternoon.

Bottom Line

The cheapest AI plan is the one that gives Hermes work you would keep.

A $5 route is great when it clears background noise. A $10 route is where I would test daily Hermes usage. A $20 route only earns its place when it gives you something the cheaper paths do not, whether that is better fit, clearer limits, or a route you trust enough to use for harder work.

The wrong plan steals focus at any price.

Before you buy another subscription, look at your last ten Hermes sessions. Mark the ones that ended in usable work. Mark the ones you had to rescue. Then ask which plan helped the work move forward and which one only looked cheap on the invoice.

That becomes the buying decision.

I would rather pay for one route that finishes the work than keep juggling three subscriptions that still need me to manage them.

Source Notes

OpenCode Go lists the $5 first month and the $10 monthly price after that. The page also covers any-agent use and current request allowances.

ChatGPT Plus lists $20 per month, app-level Plus benefits, and the note that API usage is billed separately. OpenAI API pricing lists GPT-5.5 and GPT-5.4 token pricing outside ChatGPT subscriptions.

Ollama Cloud pricing lists Pro at $20 per month or $200 per year. The same page covers three concurrent cloud models and usage measurement based mainly on GPU time.

Nous Portal lists Plus at $20 per month with 300+ models and hosted tool usage. It also lists the $22 monthly credits and rollover rules.

Claude Pro lists Pro usage behavior and resets, while Anthropic API pricing lists Claude API prices separately from Pro.

MiniMax Token Plan lists Starter at $10 per month with 1500 M2.7 requests per 5 hours and Plus at $20 per month with 4500 M2.7 requests per 5 hours.

Hermes AI Providers lists the relevant provider paths for Nous Portal and OpenAI Codex. It also covers OpenCode Go and Anthropic.