DEV Community: cucoleadan

Before You Install That GitHub Tool, Make Claude Code Beat It

cucoleadan — Tue, 27 Oct 2026 13:45:02 +0000

Six minutes into testing a PDF tool, Claude found its Windows download held only a README.

PaperPilot , a Rust PDF tool, showed up on Hacker News on 26 October. Its command-line interface looked useful for the arXiv papers I use in my research workflow, and it also came with an MCP server for agents.

That same day, Simon Willison posted about having Claude Opus 5.5 try a new tool and write up what it found. I wanted to try that approach, so instead of installing PaperPilot myself, I asked Claude to investigate it first, on a small job I could check: turning two papers into a four-page reading pack.

As it turned out, the empty download was the smaller surprise. A bigger one came once Claude Code started checking the results, and I had to ask how I’d do this job without PaperPilot at all.

Subscribe for more hands-on tests of tools you can use with your agents, including the attempts that go nowhere.

In This Edition

Claude found that PaperPilot’s Windows download held only a README, before I installed anything.
PaperPilot built a pixel-perfect four-page pack inside a locked-down Docker container, but its plain text came back as Markdown.
A short pypdf and PyMuPDF script tied on pages and gave me cleaner text, so that’s what I’m keeping.
You get the trial prompt I used, rewritten so you can point it at a tool you’ve been meaning to install.

Nothing to Install

I started in the Claude desktop app and asked Claude to find out what PaperPilot could do before I installed anything. I gave it one rule: no PaperPilot code runs without a real sandbox. It’s the same idea as the approval gates I add to Hermes: nothing risky runs until a check passes.

It came back with a problem. PaperPilot’s Windows downloads were tiny zips with a README inside and no program. A packaging script hides that error, so nobody gets warned when the program goes missing.

Claude also spotted a wrong flag in the README and confirmed that the Linux version worked. Then it found cloud credentials in its own workspace and stopped, exactly as I’d asked.

It even corrected me. My notes said people had reported security problems on Hacker News. Claude reread the thread and found questions about licensing and where the code came from, with no security reports at all. Reading past the docs paid off, a habit I wrote about in How to Search Beyond the Docs When Your Code Won’t Work.

Before I’d installed a thing, I knew to skip Windows and run the test on Linux. I learned to check first the hard way in The Twenty Minute Audit That Found Where Hermes Was Getting Lost, after adding three tools before looking at what they’d read.

Three Tries to Run It

I moved to my home server and handed the job to Claude Code. It ran PaperPilot inside a Docker container with no network access and only one folder it could see:

docker run --rm --network none --cap-drop ALL \
  --security-opt no-new-privileges --user 1000:1000 \
  -v ~/pp-trial:/work -w /work ubuntu:26.04 <command>

You could do the same with Docker Desktop on a laptop.

Getting a clean run took three tries, and none of that was PaperPilot’s fault. On the first, I’d pasted only half my instructions and named no papers, so Claude Code grabbed the repo’s own test files and ran the tool on those. On the second, it stopped and asked which papers I meant. Seems fair.

Five minutes of planning would have saved me a try, as I argued in How To Architect A Feature In 5 Minutes Before Talking To AI.

On the third try, it had both papers, Attention Is All You Need and the LoRA paper. About two minutes later, I had my four-page reading pack and its text. Claude Code then checked both with pypdf and PyMuPDF , two Python PDF libraries that have nothing to do with PaperPilot.

Remember those two.

Perfect Pages, Messy Text

PaperPilot nailed the pages. Every page in the pack matched the original exactly, down to the arXiv stamps running along the margin.

But it sometimes failed too. When Claude Code asked for page 16 of a 15-page paper, PaperPilot returned an error and wrote nothing. You won’t end up with a half-finished file that looks complete.

Text extraction was the catch. Here’s how the Attention paper’s authors came out:

**Ashish**  **Vaswani**

**Noam**  **Shazeer**

**Niki**  **Parmar**

**Jakob**  **Uszkoreit** Google Brain

Google Brain

Google Research Google Research [avaswani@google.com](mailto:avaswani@google.com)

Every word survived, but PaperPilot wrapped bold words in Markdown, turned emails into links and split authors from their affiliations. It broke the LoRA title into four pieces and mangled the inline maths too.

You could search that text. I wouldn’t want to read it, or hand it to another tool, without cleaning it first.

Send this to someone with a pile of starred GitHub repos they never get around to trying.

Then Claude Wrote Its Own

Before writing this up, I asked myself how I’d do this job without PaperPilot. I didn’t have an answer. My best guess was that I’d ask an agent, and it would write a Python script.

That’s when it clicked. Claude Code had already used pypdf and PyMuPDF to check PaperPilot’s work, and those same two libraries could do the whole job. It’s the question from The Build vs Buy Scorecard, with an agent doing the building.

I sent Claude Code back with the same papers, no PaperPilot, and free rein over the method. It wrote a short script.

On pages, it was a tie. Both packs came out identical.

On text, the script won. LoRA’s title stayed in one piece, every author kept their affiliation and email, and the maths stayed readable.

It wasn’t perfect, and Claude Code said so. One setting glued a few hyphenated words together, turning “English-to-German” into “Englishto-German”, and some letter pairs came through as special characters that trip up search. Both are one-line fixes, and Claude Code left them in so the comparison stayed fair. Honest comparisons are rarer than they should be, which is why I test models on real Hermes work instead of trusting benchmarks.

It also refused to call the script faster, because the second run had everything the first one taught it. Fair enough.

Try It on Your Next Tool

I’m keeping the script. Its text needs less cleanup, and Claude Code couldn’t find a PaperPilot setting that fixes PaperPilot’s text. Tweaking one line in my own script is easier. It’s also code I own, which matters to me since I built my stack to survive vendor lock-in.

If you only need to cut and merge pages, PaperPilot did that well, as long as you’re not on Windows. For OCR, big documents or two-column papers, this trial can’t tell you much, so run your own.

In When to Use MCPs, CLIs, or Your Own Tool, I wrote that small custom tools are often enough for the job. I’ve had agents build those for me before, like the Substack API I built with Hermes and Codex. With this trial, I got to test the idea on a real tool, with files I could check myself.

Here’s the prompt to start from. Give it your real files, or your agent will grab demo files like mine did.

I want to know whether <TOOL URL> is worth installing for one job: <JOB>.

Start by checking what the current release supports for this job, using
the official documentation and downloads as evidence. Record the commit
you inspect so we can distinguish that version from later changes.
Keep planned features separate from anything available in the release.
Check whether a binary exists for my OS before suggesting a source build.

Before executing any project code, establish which isolation boundaries
are enforced in the environment where it will run. Stop if those
boundaries aren't available, since an empty folder doesn't provide them.

Use the tool inside that environment to do the job on <INPUTS>.
If another tool is needed to produce an output, explain that rather than
crediting the result to the tool we're testing.

Verify the result using an independent method and tell me how you checked
it. Include a failure case that shows what happens when an input is invalid.

Try the same job your own way on the same inputs, without using the tool
under evaluation. Compare the outputs closely enough that I can decide
which approach I'd want to keep using.

Save enough evidence in run-notes.md for someone else to follow the run.
Include the exact versions and file hashes needed to identify what was
tested. Preserve the commands with their output and any errors encountered.
Record actual timestamps without turning them into estimated savings.

Finish with your recommendation for this particular job, making clear
where the evidence is still too limited for a decision.

If I’d stopped after the PaperPilot run, I’d have a working PDF and some text to clean up. Asking Claude Code to do the job without it gave me a script I’d rather keep, all from an experiment that started with an empty Windows download.

Leave a link to a tool you’ve been meaning to try in the comments, and I’ll pick one to put through the same comparison.

Subscribe if you’d like to see what comes back when I try the next tool this way.

I Used the Ponytail Skill to Make Claude and Codex Write More Reliable Code

cucoleadan — Tue, 20 Oct 2026 12:24:35 +0000

My best coding upgrade this month was telling my agents to be lazy.

Claude and Codex write extra code for free, and I pay for it in review time. Every helper they invent is one more thing I have to read, trust and keep working when the app changes.

On October 7, I added a skill called Ponytail to my project instructions. It makes the agent climb a seven-rung ladder before it writes anything, and the first rung asks whether the code needs to exist at all. I run it in Claude and in Codex.

Twelve days later, I asked Claude for an MCP server for my SaaS. Claude delivered one commit with 44 tools and a single new dependency, and I had it live and tested the next day.

Below is how the skill works, what it changed in that build, the one place where less code would have been wrong, and the shortcuts it wrote into my codebase on purpose.

In This Edition

Climb before you code. Ponytail makes Claude and Codex check seven options, from skipping the code to reusing what exists, before they write anything new.
Reuse beats a second API. My 44-tool MCP server sends every tool through pages the app already has and adds a single dependency.
Some code earns its place. Two wait tools add lines and stop an agent from calling a half-written brief done.
Read the notes it leaves. Every ponytail: comment marks a shortcut you still own, so review them before they turn into bugs.

Subscribe for agent builds with the code, checks and failures left in.

The Seven-Rung Ladder

Most coding instructions stop at “keep it simple”, while Ponytail gives the agent an order to work through.

Before writing anything, the agent reads the task and traces the code it touches. For a bug, that means finding every caller of the broken function and fixing it once, in the shared place. Then it climbs the ladder and stops at the first rung that holds:

Does this code need to exist at all?
Can it reuse something already in the codebase?
Does the standard library cover it?
Does the platform do it natively?
Does an installed dependency already handle it?
Can it be one line?
Write the minimum code that works.

The first rung applies the cut I argued for with features in What Gordon Ramsay Taught Me About Vibe Coding, one level down at the code.

It comes in three levels. Lite builds what you asked and names a lazier option in one line. Full runs the whole ladder. Ultra deletes before it adds and questions the requirement itself.

I run full mode, and my project’s AGENTS.md spells out what that means: trace the existing flow, reuse local helpers and components, prefer native platform features and installed dependencies, and add only what the current screen needs.

Ponytail also protects the parts that keep an app safe, like validation at trust boundaries, error handling that prevents data loss, and anything you asked for by name. It asks for one small runnable check on any non-trivial logic.

My favorite rule covers shortcuts: when the agent picks a simpler version with a known ceiling, it leaves a ponytail: comment naming the limit and the upgrade path, and you’ll see why that matters in the catch.

The skill is short, and you can read the whole thing on GitHub.

44 Tools, No Second API

The SaaS is Yahini, my content strategy app, and its MCP lets Claude, ChatGPT or Cursor work in it as you. They can create projects, run keyword research, fill the workbook, and generate briefs and drafts.

The website already knew who can open a project, which actions spend credits and what happens when a request fails. A fresh API layer for the agent would look clean in a diff and give me two copies of every rule. The day I changed one, the website and the agent would start giving the same user different answers.

Ponytail stopped Claude at rung two, so each of the 44 tools runs the page’s own loader or action with the user’s session. The app keeps handling permissions, credit checks and validation, exactly like it does on the website.

A test locks that in by failing whenever a tool points at a page the app’s routes don’t list, so no agent can slip in a tool with its own private logic.

The other rungs show up too:

Rung 5, an installed dependency: sign-in runs through Cloudflare’s workers-oauth-provider, the only package the build added.
Rung 7, the minimum: the server answers JSON-RPC over plain POST with no streaming, from a 187-line request handler.

Claude kept the careful parts too, giving each connection its own Supabase session so connecting an agent never rotates the session in your browser. Disconnecting an app from your account deletes its tokens and that session with them.

Reuse still costs something, because the MCP now depends on those page operations and renaming a form field on the website breaks the mapping. Another test checks that arguments become exactly the fields each page reads, which catches a small refactor before it breaks every connected agent.

Where Less Code Is Wrong

Briefs and drafts in the app generate in the background. When an agent asks for one, the first response only confirms that the app accepted the job, long before the document is ready.

A minimal build would return that response and let your agent tell you the brief is done while the app is still writing it. Notifications and webhooks can’t reach a chat agent, so nothing would correct it.

That’s why the MCP has two extra tools, wait_for_generation and wait_for_document, which re-read the page every 5 seconds for up to 45 seconds and then answer with finished. While it’s false, the agent calls again, so it follows the run within its own turn and you never have to ask twice.

Those 45 seconds earn their code, and Ponytail agrees: its protected list keeps error handling that prevents data loss and asks for a check on non-trivial logic. Two of the 11 tests in mcp.test.ts cover the wait tools, including the case where the limit runs out before the work does.

Refusals get the same care, with a test that turns the app’s own answers into tool results so a 403 reaches the agent as an error. Without it, your agent could report a change as saved when the app said no.

Know someone whose coding agent turns every small fix into a new module? Send them this before their next pull request.

The Catch

Ponytail ships shortcuts, and it tells you so. My codebase now has 14 ponytail: comments, and each one marks a limit the agent chose and wrote down.

Two of them are real bugs waiting for the right timing:

ponytail: two submits racing in flight can still both create;
the backend needs an idempotency key to close that.

ponytail: read-modify-write, so two people editing the same
list at once can drop one change

Neither is hidden, and both are still mine to fix. If you never read those comments, Ponytail’s honesty turns into debt you forgot you took on.

The MCP has its own limits, starting with a connected apps page that lists only the first 100 connections, since one app per user rarely gets near that. I left billing out on purpose, so checkout and plan changes still send you back to the app.

The bigger risk is the agent grading its own work, because one session writes the code, writes the test and explains why both are right. If the test copies the code’s assumption, everything agrees and the feature is still wrong. I gave my Hermes agent approval gates for the same reason.

Keep one check the agent didn’t write, the way I connected this server to a real client and used it before calling it live.

I haven’t run a controlled test, so I can’t give you a percentage of bugs Ponytail prevents, and I’d be suspicious of anyone who did. What I can show you is the code, the tests and the notes it left.

Start With One Task

Pick a change small enough to review in one sitting, with a result you can check. If it needs a plan first, my five-minute architecture pass gets you one before the agent sees the task. Save the starting code, load Ponytail in a fresh session and give the agent this:

Implement this change using the existing project conventions.
Before editing, find the code that already owns this behavior.
Keep the change limited to the requested outcome.
Check the failure case most likely to mislead a user.
Report what changed, what you verified, and what you skipped.

Then pick the level by the problem you keep hitting:

Your agent adds files and abstractions you didn’t ask for: run full.
It builds roughly the right thing, only bigger: start with lite and read the lazier option it suggests.
You suspect the whole feature is overbuilt: try ultra on a branch you’re happy to throw away.
Your setup already ships focused changes with real checks: keep it, because a skill should earn the instructions it adds.

Write the level into your project’s AGENTS.md so every session starts with it. Then search the repo for ponytail: once a week and decide which limits you’re still happy with.

If long explanations bother you more than the code, Ponytail pairs with Caveman , a separate skill for terse answers. If your agent keeps solving the wrong problem, the community’s Karpathy-inspired guidelines start by making it state its assumptions. I haven’t used either one, so start from their published rules.

My Substack API build with Hermes and Codex came down to the same standard: follow the request into the code, then find the proof that “done” means done.

One dependency, 44 tools and a note on every shortcut add up to AI code I can live with.

Tell me the last thing your coding agent overbuilt, and I’ll tell you which rung it should have stopped on.

Subscribe for more agent builds with the files, decisions and failure checks behind them.

Muse and Grok Bot vs ChatGPT and Hermes: What Are You Paying For?

cucoleadan — Tue, 13 Oct 2026 13:56:02 +0000

Yesterday (October 12) Meta announced Muse Wonder, with the new model set to reach Muse subscribers first when the app rollout begins. Ten days earlier, xAI announced a subscription combining Cursor and Grok, with Grok Bot included on the higher tiers. Both companies are giving us more reasons to pay for an agent that keeps working after we close the laptop.

And I understand the appeal. When you’re the person troubleshooting a missed scheduled job, paying someone else to run the machine starts to sound reasonable. You still get an agent with access to your tools, but the server upkeep becomes someone else’s responsibility.

The harder decision comes when you already pay for ChatGPT or have Hermes doing useful work. Some of what Muse and Grok Bot offer overlaps with that setup. Before adding another bill, I want to know which jobs become easier and how much of the checking still comes back to me.

I compared their documentation and pricing with published hands-on reports, using my experience running Hermes as the reference point. I’m looking for the work each service can take off my hands and what it asks of me in return.

I checked the prices and availability below on October 13, 2026. The comparison focuses on what you can use now, including the limits behind the subscription prices.

In this article:

What You’re Buying
What the Price Covers
What People Have Delegated
The Work You Own
Try One Job
Where I’d Start

Subscribe for practical ways to put agents to work and judge the results.

What You’re Buying

Meta builtMuse around a dedicated virtual computer with a browser. You give it ongoing goals, it keeps memory, and you talk to it through the app or WhatsApp. That arrangement appeals to me because I can hand over work that needs access to my accounts without also taking on another machine to maintain.

The small-business rollout adds connections to QuickBooks, Shopify, and Slack, among others. Meta lists the US and Canada, so readers elsewhere should check availability before spending time planning a workflow. For each connector, look for the action you need it to perform inside the app.

Grok Bot also works on a cloud computer and runs scheduled jobs while your laptop is closed. Its product FAQ explains that all your bots share one persistent computer, including its files, browser, and logins. I would check how overlapping browser jobs behave early in a trial, before relying on several bots at once.

Before adding either, look through what ChatGPT already includes. ChatGPT Work handles tasks with files and connected tools, and automations cover recurring work. For jobs that use your local machine, you need to keep that machine and the app running.

I’m using ordinary ChatGPT and Plus as the baseline here. OpenAI also has dots, its always-on agents with cloud computers, rolling out to eligible higher-tier accounts. If your plan includes a dot, give it the job before buying another managed agent.

With Hermes , you make more of those choices yourself. The open-source agent supports persistent memory and reusable skills, with scheduled jobs and messaging integrations for ongoing work. You pick the model provider and where the agent runs, and you look after the connections between them.

What the Price Covers

I’ve used the advertised monthly prices in US dollars. Check the offer in your own account, since your location and existing plan can affect what you pay.

Muse has a free option with limited usage. Meta’s subscription page lists Power at $20 a month with 500 million Muse tokens per week , and Maximum at $100 with 3 billion per week. It doesn’t give a fixed numeric allowance for the free option.

I can’t tell you how many useful tasks those token allowances will buy. Muse uses its own accounting, and treating its tokens as equivalent to a model provider’s API tokens would give us a misleading comparison. Put a few of your recurring jobs through it and watch how much allowance they use.

For Grok Bot , the entry point for new customers is $30 a month. The unified X Subscription, announced October 2, puts Bot on Premium and above. The full plan list is Free, Plus at $8, Premium at $30, Super at $100, and Ultra at $200 monthly.

The bundle makes more sense if you also use Cursor, but the two products draw from the same weekly allowance. Every Bot job uses allowance you may also want for coding that week, while third-party models in Cursor carry separate usage charges.

If you’ve seen people recommending a $20 Cursor plan for Bot, they may still have an older subscription. Existing subscribers can keep those plans, while new customers start with the unified offer. Include taxes and any extra usage when comparing the bill.

ChatGPT Plus is $20 monthly , according to OpenAI’s pricing. If you already subscribe and have room in the allowance, you can try the job without adding another bill, which makes it my first recommendation for existing subscribers. API usage costs extra.

Hermes has no software subscription fee. You supply the model access and somewhere to run it, then cover any paid tools you connect. The easy part to leave out is your own maintenance time, especially if, like me, you enjoy working on the setup.

What People Have Delegated

In Eric Hal Schwartz’sMusetrial for TechRadar, the useful moment was an email he’d missed. Muse spotted an unopened utility email about a solar-panel visit, discussed dates with him, and prepared a response for approval. I can see myself using an assistant for that kind of admin.

Muse also found shopping options and asked for an Amazon login before proceeding. The report takes that task as far as the login request. Elsewhere in the trial, Schwartz described Muse forming a distorted picture of him from the emails it could see.

That last part would bother me more than having to approve an email. I expect to review a draft. If the agent has misunderstood something about me, I first have to notice the assumption and work out which of its decisions relied on it.

One e-commerce user reported mixed results withGrok Bot. They found its email-marketing work useful, but it repeatedly missed instructions on product listings. The user also reported browser crashes and eventually abandoned a customer-chat experiment.

A bookkeeping and data-entry user reported much better value. I can’t verify either user’s results, and their plans and models also form part of the picture. I take these accounts as suggestions for what to try, rather than a basis for promising anyone a particular saving.

For a first assignment, give the agent a draft you know how to review. You’ll learn where it needs help without asking it to handle every unusual customer request along the way. Once you can use those drafts with little editing, you have a reason to give it more responsibility.

The Work You Own

I’m willing to pay someone else to keep the agent’s machine running. That would remove a job I already understand, although I’d still make time to review what the agent produces.

Meta describes safeguards aroundMuse, including a separate system that reviews network access, plus checks before sensitive actions. Those checks serve a different purpose from reading a customer email to make sure it says what I intended, and I want both before the agent sends anything under my name.

I use that same separation in my Hermes approval-gate workflow. The agent prepares the work and brings it back to me before sending. It’s an easy place to start while you get to know how a new agent handles your instructions.

Maintenance has taken more of my attention than I expected at times. In my Hermes maintenance routine, I wrote about blaming the model for a delay before tracing it to memory recall. I also had to investigate a scheduled report that hadn’t arrived.

Those problems came from my setup and its integrations, and I had to sort them out. I still value the control Hermes gives me, but I count that work when weighing a hosted alternative. If someone else can save me the troubleshooting, I’m happy to include that in what their service is worth.

Control also involves knowing where your data goes. When you connect a self-hosted agent to a cloud model, it still sends information to that provider. Hosting the agent yourself gives you choices about those connections, and you need to pay attention to the choices you make.

When you compare the options, include the time you spend getting usable work out of them. If you spend an hour rescuing a run and throw away the result, that hour still counts.

Share this with someone weighing another agent subscription against the setup they already have.

Try One Job

Pick something you do often enough to know when the result is useful. My morning briefing workflow fits because I know what belongs in it and can tell when it leaves me digging through the inbox again.

For an invoice-follow-up trial, give each option the same sample. Include an overdue invoice alongside one that’s already paid, and leave one record ambiguous on purpose. The agent needs to get the amounts and dates right, leave the paid invoice alone, and ask about the unclear record before preparing anything to send.

If research takes up more of your week, ask for a dated competitor comparison with source links and a note about anything the agent couldn’t confirm. Open those links and check whether they support the claims. Include that reading time when you decide how much work the agent saved you.

Run the same job through your existing method so you have something to compare it with. Keep the first attempt small enough to inspect completely, then repeat with fresh inputs. Record setup separately from the time each run takes.

Keep a short note for each attempt:

Job and frequency: What needs doing, and when?
Accepted result: What must be correct before I use it?
Access and approvals: Which accounts and actions are involved?
Human minutes: Setup, supervision, review, corrections, and repair.
Spend and limits: Added cost, allowance used, and any interruption.
Outcome: Accepted, corrected, or abandoned, with the reason.

You’ll soon notice if a tool gives you more work than you started with. When the early results look useful, keep the same notes as you give it fresh inputs. That’s how you find the awkward cases the first attempt missed.

Where I’d Start

For most readers already paying for ChatGPT , that’s where I recommend starting. Give it the research task or draft, then see what you still have to do yourself. Check its scheduling options before buying another product to repeat the job.

Muse is worth trying when its connections fit the admin you deal with every day. If your account is eligible, start with the free option and give it one of those jobs. A paid plan becomes easier to justify once you have a result you want again next week.

Grok Bot makes more sense to me if you also want Cursor and have work that needs a persistent browser. Check that the shared allowance covers both, and pay attention when several bots use the computer at once. Those details will affect how comfortably it fits into your week.

I still have a clear reason to keep Hermes : I want to choose my model provider and work with files on my own host. I’m willing to accept some upkeep for that, but if you don’t need those choices, you have less reason to take on the maintenance yourself.

And if you only do the job occasionally, it’s fine to finish it yourself. Setting up an agent should save you work often enough to repay the effort.

Leave promised upgrades out of the buying decision until your account can use them. That includes the Muse Wonder app rollout mentioned at the start.

Give Muse or Grok Bot one job for a month before moving the rest of your work. Keep track of how often you use the result and how much attention it takes to get there. By renewal time, you should know whether the service saves enough time to justify the bill, or whether you’re better off keeping the setup you already have.

Which recurring job would you give Muse or Grok Bot first? Tell me the task and what you use today in the comments.

Subscribe for more agent workflows, maintenance notes, and practical comparisons.

One MCP to Rule Them All

cucoleadan — Tue, 06 Oct 2026 12:57:26 +0000

You’ve connected an MCP server to Claude, and now you need the same tools in Codex.

Each client needs its own connection. When you find another useful integration, you repeat the setup across the apps you use. Eventually, maintaining those connections starts looking like a separate hobby.

Cloudflare’s MCP server portals give that configuration a shared home. You connect each client to the portal, then manage the upstream servers from the Cloudflare dashboard. The portal keeps the same address as the collection grows.

You might hear this described as an MCP gateway. Cloudflare’s current product name is worth knowing because MCP Portals is what you’ll find in the dashboard.

Code Mode makes the proposal more useful. It lets the agent discover tools as it needs them, which reduces the amount of catalog information loaded into the conversation upfront.

I’ve based this piece on the documentation. The practical question is whether centralizing the tools removes enough maintenance to justify another service in the stack.

In this article:

How one portal replaces repeated MCP setup
Where Code Mode can reduce token overhead
How to connect the agents you already use
What to check before moving your tools behind Cloudflare

If connecting your tools takes more time than using them, subscribe for practical ways to make your agent stack easier to maintain.

Give the Tools a Home

An MCP server gives an agent access to another system through a defined set of tools. Cloudflare’s portal sits between your client and those servers, routing each request to the service that performs the work.

Suppose your agent already reads project issues through the portal. You decide it should also consult your team’s documentation. You register that documentation server in Cloudflare and attach it to the existing portal.

The client might need to refresh its tool list before the new capabilities appear. The documentation service might also require its own login. The integration itself now has one place to be configured.

The boundary matters: this works with remote HTTP MCP servers. A local server running over stdio needs an HTTP endpoint before the portal can reach it.

A filesystem tool on your laptop is a reasonable case for keeping a direct connection. Hosting it remotely would create a new problem merely to make the configuration look consistent.

Stop Loading the Whole Catalog

Once the connections live together, the tool catalog becomes the next concern.

A model needs enough information about a tool to call it correctly. That definition consumes context when the client loads it. A large catalog can occupy a substantial part of the conversation before the agent starts your task.

I explored the broader context problem in Why AI Gets Stupid Deep in the Thread. For this setup, the question is how much tool information the agent needs to carry before it knows what you’re asking it to do.

Some clients already discover tools on demand. The overhead depends on how your client handles the catalog, so a long tool list alone doesn’t establish how much you’re paying for it.

Cloudflare’s portal Code Mode changes the interface the agent sees. It provides portal_codemode_search for discovery. A second tool, portal_codemode_execute, runs the JavaScript the agent writes to use the available methods.

The code runs in an isolated Worker environment. Portal management tools remain available alongside that interface, so the description of “two tools” applies to the replacement for the upstream catalog.

Consider a read-only request to find overdue issues for a project called Website Refresh. The answer needs to show the issue titles with their source links. Everything else in the issue records is background unless it helps explain the delay.

With Code Mode, the agent could use JavaScript to filter the retrieved records before returning them to the model. The model would read the evidence needed for the answer instead of the full response from every call.

For that workflow to succeed, the filtering would have to preserve the relevant issues. The oldest overdue issue should still appear in the answer after the response has been trimmed.

What the Savings Cover

Cloudflare reports roughly 1,000 tokens for the Code Mode interface to its own Cloudflare API MCP server. It compares that with an estimated 1.17 million tokens for equivalent traditional tool definitions. The company describes the difference as a 99.9% reduction for that tool surface.

The measurement covers the API catalog’s context footprint. Your total task includes much more than the catalog, so the figure shouldn’t be read as a forecast for your bill.

The model still spends tokens completing the work. Generated code might need repairs, and those retries have a cost. If your client already loads definitions selectively, the additional savings could be smaller.

Subscription billing adds another distinction. Lower token use might help you stay within a usage limit while leaving the monthly price unchanged.

I’d judge Code Mode by the cost of a correct result. That keeps the comparison tied to the work the agent was supposed to finish.

Choose What Gets Through

A central catalog also gives you a place to decide what the agent should be able to do.

Cloudflare lets you choose which tools a portal exposes. A research portal could expose the read operations from a project tracker. Write access could belong in a separate portal with a narrower Access policy.

This is the same decision behind When to Use MCPs, CLIs, or Your Own Tool. The interface should fit the work you’re handing over. A server’s full catalog is rarely a useful definition of that work.

Tool descriptions deserve attention too. Cloudflare supports aliases and description overrides, which gives you a way to clarify an ambiguous operation without changing the upstream server. An agent has a better chance of choosing correctly when the description explains the job the tool serves.

New capabilities are enabled by default in ordinary portal mappings. If you want to review additions before an agent uses them, configure the mapping as an allowlist. Otherwise, an upstream update can broaden access after your initial review.

Keep Identity in View

Cloudflare Access controls entry to the portal. A connected service can still require the user to authorize their own account through OAuth.

There’s also an option to use an upstream admin credential. That changes whose permissions the request carries, so it deserves a deliberate choice.

For an unattended agent, Cloudflare supports Access service tokens. The portal needs a matching Service Auth policy. Each linked server needs one as well.

Those connections use the upstream admin credential. A server configured to require individual user OAuth won’t be available through a service-token session.

I’d establish the workflow with an interactive login first. The unattended version becomes easier to assess once the basic connection has done something useful.

Follow a Failed Request

Portal logs give you somewhere to investigate a failed call. You can identify the upstream server involved and inspect the result status. The recorded duration helps distinguish a slow response from a quick rejection.

My Hermes maintenance routine starts with the same habit: find the failing layer before changing the model. The portal belongs in that investigation once your tools depend on it.

Optional Cloudflare Gateway routing adds traffic inspection. With the appropriate data loss prevention policy, Gateway can block a call that contains matching sensitive data.

That path requires upstream Streamable HTTP support. An SSE-only server won’t work with Gateway routing enabled.

The controls still leave you responsible for deciding when an action needs review. My Hermes approval-gate workflow applies when an agent is about to make a consequential change.

Connect the First Client

The dashboard setup requires an active domain on Cloudflare. You’ll also need an identity provider configured in Zero Trust.

Open Zero Trust → Access controls → MCP Portals. Register your remote server in the MCP servers tab and complete its authentication setup.

Create a portal on a subdomain such as mcp.example.com. Attach the registered server to it, then choose the capabilities you want exposed. An Access policy determines who can connect.

The resulting endpoint takes the form https://mcp.example.com/mcp. Use your own domain in the examples below.

These examples follow the documented client features. They haven’t been verified against a live portal across all four clients.

Claude Code

Claude Code accepts a remote HTTP connection with this command:

claude mcp add --transport http --scope user cloudflare-portal https://mcp.example.com/mcp

Open /mcp inside Claude Code to complete authentication. Other Claude surfaces have their own remote connector setup.

Codex

Codex supports a URL-based connection followed by OAuth login:

codex mcp add cloudflare-portal --url https://mcp.example.com/mcp
codex mcp login cloudflare-portal

Its local clients share configuration through config.toml. ChatGPT web uses the separate connection described below.

Hermes

In Hermes, merge the portal entry into the existing mcp_servers block in ~/.hermes/config.yaml:

mcp_servers:
  cloudflare_portal:
    url: "https://mcp.example.com/mcp"
    auth: oauth

Complete authentication from a fresh terminal:

hermes mcp login cloudflare_portal

When Hermes runs on a remote host, follow its documented browser callback procedure. The login needs to reach the machine running the agent.

ChatGPT

OpenAI’s current instructions use ChatGPT Plugins → plus button → Add custom MCP server on the web.

Enter the portal URL and configure OAuth. Create the plugin, then install it so it’s available in your conversations. Workspace permissions still apply.

Use the app’s refresh control when you need to pull an updated tool catalog.

Send this to the builder who sets up the same MCP server in every new agent. The portal gives that repeated work a shared home.

Put Code Mode to Work

Cloudflare’s default portal policy makes Code Mode opt-in. Under that policy, connect with this address:

https://mcp.example.com/mcp?codemode=search_and_execute

The portal administrator can set a different policy. Check that setting before assuming the query parameter will take effect, then reconnect the client after changing its URL.

Cloudflare also provides lighter context options. The minimize_tools option reduces the initial definitions while keeping full definitions available through discovery. Its search_and_execute option provides proxy execution behind a search interface.

Those options use the optimize_context parameter. Portal Code Mode uses codemode to enable JavaScript execution. The similar naming makes the distinction easy to miss.

For a first comparison, use a task whose answer you can check yourself. The overdue-issue request from earlier gives you a bounded test:

Use the connected project tracker to find up to five overdue
issues in Website Refresh. Treat an issue as overdue only if
it's open and its due date is before today.

Order the issues from oldest due date to newest. Return each
title with its source URL. Make no changes to the tracker.

If the due date is missing, leave that issue out. Report a
failed lookup rather than claiming there are no overdue issues.

Run it with ordinary portal access, then repeat it with Code Mode using the same model. Check the result against the tracker before comparing token use. A run that misses a qualifying issue shouldn’t count as a saving.

Save the run record alongside the answer so you can see where any extra calls came from. It gives you something to inspect when a shorter response turns out to cost more.

The Dependency You Add

The convenience comes with a shared point of failure. If the portal becomes unavailable, every client depending on it loses that route to the tools.

Cloudflare also sits on the request path. The decision to use it should include whether you’re comfortable routing the relevant data through its service.

Before relying on that path, try adding the service your workflow depends on most. Some providers reject proxy-based OAuth clients, even when the underlying transport matches.

A server already running its own Code Mode needs particular attention. Cloudflare’s portal documentation directs users to an upstream version exposing the full tool list. Check the upstream opt-out mechanism before combining the two.

After a successful connection, check what happens when the upstream catalog changes. Manually configured OAuth servers have capability-sync restrictions, so refreshing the client won’t necessarily reveal a new tool. The server synchronization documentation is the place to investigate a stale catalog.

Cloudflare currently lists a limit of 80 servers per portal. An account supports up to 20 portals. Those limits leave room for a substantial setup, although filling the available capacity shouldn’t become the objective.

Your upstream services retain their own charges after moving behind the portal. Some Cloudflare features also depend on your plan, including the Enterprise-only Logpush export for portal activity.

Cloudflare’s AI Gateway handles model traffic separately. Its spending controls require their own setup, so they shouldn’t be counted as an automatic benefit of the MCP portal.

For a small collection of local tools, I’d keep the direct connections. For someone maintaining the same remote integrations across several agents, a portal addresses work they’re already repeating.

The more useful result is freedom to try another client without rebuilding the tool setup around it. You can spend that time finding out whether the new agent deserves the job.

Which MCP are you tired of configuring in every agent? Leave a comment with the connection you’d move first.

Leave a comment

Subscribe for more ways to make your tools useful across the agents you use.

Jev for Hermes: What It Does and Where It Fits

cucoleadan — Tue, 29 Sep 2026 12:39:10 +0000

My Hermes agent runs on GLM 5.3 Flash hosted on Fireworks, and it’s more than capable of doing anything I throw at it. So naturally, adding another model sounds like another account to maintain and another place to look when a request stalls.

That kind of changed when I read more about Jev. It caught my attention because it takes on a smaller job. Before Hermes writes anything, it often has to choose which instructions to load or which tool to use. Those decisions happen repeatedly, even when the task itself is routine.

TypeSafe built Jev for that kind of judgment. You supply the available answers, and Jev returns a decision with probabilities your software can use.

I haven’t integrated Jev into Hermes yet, but I’ve been looking through TypeSafe’s documentation and the community integrations to work out where I’d try it. Skill selection looks like the strongest starting point, especially for an agent with a crowded catalog.

And I can already see a few places where Jev could help my Hermes workflow. I’ll walk through where I’d try it and what would make it worth another API bill.

Subscribe for practical agent experiments, including what earns a permanent place in the stack.

In this article:

What Jev does and where it fits
Four integration ideas for Hermes
A first experiment worth running
The limits I’d check before relying on it

What Jev Does

TypeSafe introduced Jev in early access on September 15. The company calls it a System One model, designed for quick judgments inside software.

You send it state, such as a user request or an error report, alongside questions with defined answer types. Its API has three primitives:

Choice selects from options you provide, returning a probability for each.
Score rates the input against ordered descriptions in your rubric.
Noul returns a number between zero and one representing the probability of “yes.”

Suppose Hermes receives: “Find the deployment instructions for this project.” A Choice question could select between a documentation-search tool and a terminal tool, with an explicit “none fits” option.

Jev would return the selected option. The main model would still have to prepare the search and interpret what came back.

Choice and Score also return a confidence value derived from how concentrated the probabilities are. That value tells your code how decisive the answer looks. A confidence of 0.9 shouldn’t be read as proof that 90% of equivalent decisions will be correct in your workflow. You need your own examples to establish that relationship. TypeSafe explains the distinction here.

The current model page lists Jev 1.13 at $0.042 per million input tokens , with output tokens free. The total request budget is 64k tokens, subject to a separate 32k limit for the state plus the longest question. Input is text only.

At that price, repeated classification becomes cheap enough to investigate. Whether it saves money overall depends on what it replaces and how often a wrong route creates extra work.

Where the Call Goes

Jev fits beside the model running Hermes. The main model still handles open-ended reasoning and writing. Jev answers a narrower question at a point where the workflow needs to branch.

For model routing, the proposed sequence is:

User request → Jev chooses a route → code checks the result

↓

selected model or existing fallback

Think of a support request that mentions a failed payment without using the word “billing.” A decision model can judge its meaning, while ordinary code sends it to the corresponding queue.

Exact conditions belong in code. If a spending limit is $10, compare the amount directly. Asking another model whether $12 exceeds the budget would be an expensive way to introduce uncertainty.

Hermes supports plugins and lifecycle hooks where these calls can run. The right connection depends on whether Jev should be available on demand or consulted automatically. That’s the same choice behind my MCP, CLI, or custom tool article.

Four Places I’d Try It

The Hermes catalog already includes several community Jev integrations. Catalog entries pin specific commits, but the review covers metadata and declared capabilities. Inclusion doesn’t amount to a full code audit or evidence that the plugin improves your agent.

Each of these ideas needs its own test.

Finding the Right Skill

An agent with a large skill catalog has to choose from descriptions that often resemble each other. “Create a presentation” and “edit a presentation” look close until the wrong instructions send the task sideways.

TypeSafe’s Hermes skill-suggestion cookbook tackles this with two requests. Jev first ranks the catalog, then reads more detail about the top three candidates. It can reject the shortlist when nothing fits.

The cookbook reports that wrong skill loads dropped from 16.8% to 7.3% across the covered requests in a 488-request evaluation. Loading a skill unnecessarily fell from 9.8% to 4.0% on requests with no matching skill.

Those are TypeSafe’s results, using Jev 1.12 with Claude Haiku 4.5 and a fixed Hermes roster. The evaluation includes generated requests, so it leaves plenty to check against everyday use.

The community jev-skill-router plugin implements this suggestion pattern. I’d compare its recommendation with Hermes’s existing choice before letting it influence a turn.

For a request to create a new slide deck, I’d want the authoring skill selected. Loading the editing skill would count as a visible miss, even if the agent eventually recovered.

Choosing a Model

A routing layer could send routine formatting to a cheaper model and reserve a stronger one for difficult work. Jev would select from routes the operator has already configured.

The descriptions need to reflect the task’s consequences. “Change one line in the production deployment config” is a short request with more at stake than rewriting a long document.

I’d give each route a precise description and include an uncertain outcome that keeps the existing model. Budget enforcement would stay in code.

A useful test would compare total cost per completed task. Saving on the first call accomplishes little if the cheaper model needs several repair attempts. I’d also inspect the tasks Jev sent down the cheap route incorrectly, because those failures reveal where the descriptions are too generous.

I use that same approach when testing models through a real Hermes workday. The whole task has to finish before the cheaper route earns its place.

Reviewing Tool Calls

Hermes already has an auxiliary-model approval path. The community jev-approvals plugin connects Jev to that part of the system.

A narrow question might ask whether a proposed terminal command changes files outside the approved project directory. The state would include the command and its resolved working directory, giving the judgment something concrete to inspect.

I’d evaluate recorded commands first, with the existing approval rules left in charge. A classification that overlooks an external write is an immediate failure, regardless of how confident the model sounds.

My approval-gate workflow still needs a human decision where the consequences warrant one. Jev’s score wouldn’t supply permission to send a client message.

This is also a place to retain exact checks. If code already knows that a path is outside the workspace, it should enforce that boundary directly.

Filtering Retrieved Context

An agent retrieving old project notes often gets more text than the next task needs. Jev could judge which passages deserve space in the main model’s context.

For example, a request about the current deployment might retrieve both an old setup note and its replacement. A useful filter would retain the current instructions and recognize that the earlier note was superseded, provided that evidence is present.

Dropping the current note would be harder to spot than loading the wrong skill. The agent might proceed confidently with the only version it received.

The community hermes-jev plugin includes context curation with shadow modes for observing proposed changes. I’d inspect those decisions before applying them, especially when a discarded passage contains a constraint.

There’s a data tradeoff here too. The hosted service receives the content being evaluated. TypeSafe says it doesn’t train on customer requests or responses, while its model documentation describes zero data retention for enterprise customers. Those are different commitments.

If you’re building your own agent setup, subscribe to All Agents Considered for practical ways to improve it without spending every weekend maintaining it.

A First Experiment

I’d start with skill selection on one Hermes profile, using a small set of real requests. During the first pass, Jev would only produce a log for comparison.

I’d collect around 50 requests and label the appropriate skill myself, including cases where no skill applies. That’s enough for an exploratory check, though it wouldn’t establish reliability across every kind of task.

Each record would contain the request and both selections. I’d add a short review note whenever they disagreed, so I could return to the reason behind the label.

Ambiguous cases deserve attention. A request to “improve these slides” may require clarification before either selector has a fair chance. Forcing a correct-answer label onto unclear input would make the comparison look more precise than the input warrants.

After refining the question descriptions, I’d evaluate a fresh set of requests. Otherwise, I’d be measuring how well I’d tuned Jev to the examples I’d already seen.

Only then would I let the suggestion reach Hermes. Even a correct suggestion can be ignored, so the second stage needs to measure which skill the agent loads and whether the task completes properly.

The fallback would preserve the current selection process whenever Jev fails or the answer is uncertain. I’d keep the questions and threshold settings together in one file, where changes are easy to review.

The result I care about is fewer wrong skill loads at an acceptable delay. Cost belongs beside that result, including any extra work caused by a bad suggestion.

Share this with someone considering Jev for their agent. A small comparison gives them somewhere concrete to start.

Using Other Agents

TypeSafe provides Python and JavaScript SDKs, so the same decision call can sit inside a custom agent application.

Its coding-agent skill gives tools such as Claude Code and Codex documentation for writing those integrations. Installing it supplies instructions. Runtime calls still need an integration and API credentials.

For an agent that supports external tools, an MCP wrapper is one possible way to expose Jev on demand. Automatically consulting it before every turn requires an extension point in that particular agent. The connection should follow the host’s supported interfaces.

The distinction is useful when asking a coding agent to build this. Specify the decision you want to test and where its answer will be consumed. “Add Jev to my agent” leaves most of the design undecided.

For Hermes, the community jev-typesafe plugin exposes explicit tools such as jev_route and jev_score. That offers a smaller entry point: ask the agent to evaluate one saved request, inspect the returned answer, and compare it with your own judgment before considering automatic calls.

For a custom agent, I’d put the equivalent call behind a small function that returns a suggested route. The caller would check that the route exists and remains allowed for the current task. Credentials would stay in the environment. Keeping execution outside that function makes it easier to replace Jev later without rewriting the workflow around it.

The Limits Worth Reading

TypeSafe’s Jev 1.13 limitations page is worth reading before copying an integration.

It describes a model that reads instructions word for word. Implied conditions can be missed, and complicated chains of reasoning make answers less dependable. Sending a large amount of irrelevant text also hurts performance.

Numeric tasks are another weak spot. Counting and date comparisons belong in code, even when Jev helps interpret the language around them.

It also needs adversarial testing. A document being evaluated may contain instructions intended to influence the judgment. Constraining the response format doesn’t settle whether the selected answer is right.

That distinction gets lost in the launch phrase “zero hallucinations.” Jev constrains the available output, so a Choice answer comes from the supplied options. It can still select the wrong one with confidence.

For my stack, the hosted dependency is another consideration. An unavailable decision service needs to leave the workflow with a usable fallback. I’d record the returned model version during evaluation too, because changing models complicates comparisons.

It would also need a place in my Hermes maintenance routine. If Jev stops responding, I want to know before I start blaming the main model.

Where I’d Start

Jev looks worth testing when a recurring judgment already causes trouble. Skill selection gives me a concrete place to measure that, with decisions I can inspect before they affect the work.

For an agent with a handful of tools and no selection problem, I’d leave the setup alone. The same goes for a workflow whose relevant data has to stay on the machine.

I haven’t established that Jev will improve my Hermes stack. The docs give me enough reason to run a small comparison, and enough detail to recognize a disappointing result.

If it reduces wrong skill loads, I’d let it suggest a skill during real work and see whether that improvement survives. If keeping the selector useful becomes another weekly maintenance job, I’d have my answer too.

Which decision would you want Jev to handle in your agent? Tell me the task, and where your current setup gets it wrong.

Leave a comment

The AI Memory Layer I Forgot Was Running For the Past Half Year

cucoleadan — Tue, 22 Sep 2026 13:36:34 +0000

Hindsight has been running behind my Hermes agent for nearly half a year, and somewhere along the way I almost forgot it was there.

Old decisions come back when they’re needed, conversations carry over into the next session, and work picks up without the familiar ten minutes of explaining everything again. After a while, you stop noticing that part. You expect the agent to remember because most days it does.

That’s a fairly big change from June, when I wrote that Hindsight might be too much hassle for too little return. At the time, keeping another service running felt like a questionable way to save myself work.

It stayed in the stack anyway, and I’ve been meaning to revisit that comment. Looking through the logs for this article made me realize how far my attention had moved on while memory kept doing its job.

Meanwhile, the models available to handle that background work have become cheaper and more capable. If you’re using Hermes every day and still explaining the same project each time you open a session, I’d put custom memory much higher on your list than another model upgrade.

In This Edition

Where Hindsight fits beside my files and workflow instructions
What changed my mind after the early retrieval problems
How two agents hit a shared limit, and what happened to the failed writes
Whether the extra service is worth running in your own setup

Subscribe to All Agents Considered for practical agent workflows and follow-ups on the setups I keep using.

When Remembering Becomes Normal

There’s a particular kind of friction you get used to with AI. A conversation goes well, you settle a decision, and a few days later you’re explaining the whole thing again because the new session has no useful recollection of it.

You start compensating without thinking about it. Before asking for help, you gather the old context and prepare a small briefing for an assistant you’ve already briefed. Eventually that preparation feels like part of using the tool.

Custom memory takes some of that work off your hands. In my setup, Hindsight keeps useful information from sessions and brings relevant context back into later ones, so a decision has a chance of surviving beyond the conversation where it was made.

The difference shows up in ordinary work. Earlier this month, Hermes picked up a research workflow from June and used the output format we’d settled on then. I’d forgotten that I’d specified it, which made it slightly embarrassing to be reminded by my own agent.

Another session brought back an older decision about article numbering. Hermes applied it and carried on, and there was no reason to stop what I was doing and think about the memory system responsible.

Multiply that kind of thing across a working week and you begin to understand why this follow-up took so long. There’s less to complain about, so your attention goes somewhere else.

The week of logs I checked contained 177 recalls and 183 writes. Memory was being used throughout the week while I was thinking about the work in front of me. Those numbers won’t tell you how useful every result was, but they explain why removing the service now would be noticeable.

Even returning to older work feels different. Asking about something from August gives Hermes a place to start without requiring me to remember which conversation contained it. That’s useful when the reason for a decision never made it into the finished document.

It Took Some Settling In

Anyone who read my vendor-lock-in article might remember this sentence:

I added a 3rd party memory provider called Hindsight, but I might drop it as it’s proven to be too much hassle for too little return for me.

There was a reason for that frustration. A retrieval timeout had made Hermes feel slow, and I spent days blaming the model before finding the delay earlier in the chain. The agent was waiting for memory before it even reached the part I’d been blaming.

Adding a service to reduce repetitive work feels a bit silly when you’re spending your evening diagnosing that service. At that point, going back to a simpler setup was tempting.

But the early trouble stopped defining the experience. Hindsight stayed running while the rest of the stack changed around it, and retrieving old context became something I relied on without making a separate decision to trust it.

My setup is still fairly modest. Hindsight runs in a container on hardware I already own, with a Nemotron model through a GPU provider handling fact extraction. That’s the step that turns a session into information the memory system can keep.

Most settings remain at their defaults, with a limit on how much recalled text comes back into each request. A useful reminder about the current project should fit into the work without dragging a pile of unrelated history along with it.

There are more settings available if you want to spend your weekend experimenting. After the early troubleshooting, I was happy to leave them alone and get back to using Hermes.

Better Models Change the Calculation

Nearly half a year is a long time in AI. The cheaper end of the model market has improved enough that background jobs like this deserve another look, especially if your impression of them comes from an older setup.

You can see the broader direction in Artificial Analysis’s tracking of model capability and inference prices. More capable models keep becoming available at lower price points, giving us better options for work that has to happen repeatedly without turning every request into an expensive event.

Memory is a good use for that progress. Extracting a project decision from a conversation is a narrower job than carrying out the project itself. It needs a model that handles the information reliably, but paying for the most expensive model on every background call is a choice you should have a reason for.

And this work adds up. Each session leaves information to process, even when the visible task was small. Cheaper models make it easier to keep that happening as part of normal use instead of saving memory for the occasions when you remember to ask.

My own setup uses an existing provider account and an existing server, so adding Hindsight didn’t add another memory subscription. It still uses compute and provider capacity, but the extra service has been easy enough to accommodate inside the stack.

This is the direction I wanted when putting together the $30 Hermes setup: affordable components that keep being useful after the initial excitement wears off. The options for putting that kind of setup together have improved since then.

For someone already running Hermes daily, I don’t see a good reason to keep putting off third-party memory because of what it used to cost or how capable the cheaper models used to be.

You’ll still spend some time connecting it and checking the results. But if you’re repeatedly supplying context the agent has already seen, you’re already spending time on memory. You’re doing the retrieval yourself.

Even the Failure Went Unnoticed

The most telling detail in the logs was something nobody in the house had noticed.

Two agents use the same Hindsight deployment, each with its own memory bank. One is mine and the other belongs to a family member, but both use the same GPU provider account and API key for extraction.

One morning in September, both were busy. My agent had also delegated work to several subagents, whose returning results triggered additional recalls during the run.

When both banks needed to extract facts around the same time, the provider returned rate-limit errors. A batch of writes retried four times over roughly seven seconds and failed.

Then, about ninety seconds later, the queue tried again. The batch went through and the operation completed.

By the time I found it in the logs days later, there was nothing left for me to fix in that batch. It had failed and recovered while we carried on using the agents.

After the earlier timeout, I’d expected another memory problem to announce itself by getting in the way of my work. This one left a trace in the logs and was over before either of us noticed.

There’s still a useful detail to take from it. Separate memory banks were sharing the same extraction capacity, so two apparently separate agents could hit the same limit. If you’re running several scheduled jobs, check whether they all reach that provider at once before deciding you need a different model.

For my setup, though, the recovery was the more encouraging part. The queue gave the provider time to accept the work, and the failed batch didn’t need me to intervene. That’s the kind of behavior that makes a background service easier to live with.

Memory Fits Around Your Files

None of this has changed where the working documents live. The current article still belongs in its folder, and the instructions for a recurring task still sit beside that task.

Hindsight helps bring back the conversation around the work. If a decision needs to govern every run, it deserves a place in the instructions where Hermes is told to read it.

That’s why the Mistake Ledger still has a job. A verified fix for the research sorter belongs beside the sorter. Recalling an old conversation is useful, but a rule that must apply every time should be easy to find every time.

Both fit comfortably into the same setup. Files give the work a clear shape, while memory helps the agent pick up details you’d otherwise have to supply again.

You don’t need to turn the whole workspace into a memory experiment to get something useful out of it. Start with the projects you return to often and pay attention to whether the old decisions come back correctly. An agent remembering the wrong version of a decision is something to correct before you rely on it more widely.

Once that part is working, give yourself permission to stop fiddling with it. It’s easy to spend so much time improving the machinery that you never get around to using it.

A Much Easier Recommendation Now

Coming back to Hindsight after nearly half a year, the surprise is how little of the story involves maintaining it.

The early frustration was real, but it no longer describes an ordinary week. Most of the time, Hermes retrieves something useful and the work continues. Even the September incident had finished recovering before I knew it had happened.

With cheaper, more capable models available for the background work, custom memory is also easier to recommend than it was when I started. If Hermes is part of your daily routine, give it a way to carry more of that routine forward.

That’s where my setup has ended up. I open Hermes expecting it to know enough about the project for us to get started, and most days it does. Hindsight is still running on the server, but it’s been a while since it felt like another project I had to look after.

What does your agent still make you explain every time you come back? Tell me in the comments.

Leave a comment

Subscribe to All Agents Considered for practical ways to make your agent more useful in everyday work.

I Asked an AI Agent Whether You Should Subscribe to My Newsletter

cucoleadan — Tue, 15 Sep 2026 13:15:03 +0000

I asked Codex to send a fresh agent to my newsletter and help a stranger decide whether to subscribe.

The reader I gave it was someone who uses AI every day but doesn’t code. I wanted the agent to explain who AAC is for, pick three articles to start with, and work out what a subscriber would get.

It got most of that right. It understood the audience and chose a sensible reading order. Then it reached the questions about courses, support, and paid access, and couldn’t verify enough to give a complete answer.

That was the useful part of the experiment. The agent could explain my work, but some of the details a reader would need before paying were still out of reach.

In This Edition

What an agent learned about AAC from the public site.
The three articles it chose for someone who doesn’t code.
Why finding an announcement didn’t tell it what was available.
A prompt to try on your newsletter, consulting site, or public project.

Subscribe to All Agents Considered for practical experiments with AI agents, including the ones that send me back to inspect my own work.

Give It a Stranger’s View

I’ve already asked Hermes what my files say I do all day. It described me as an AI Workflow Field Reporter With a Side Cart of Websites , which is still a ridiculous thing to put on LinkedIn.

Hermes had an advantage in that exercise. It could see the work behind the articles, including notes and project files a reader would never encounter.

This time I wanted the public version. Someone looking at my newsletter doesn’t know which plans I’ve abandoned or what I meant when I wrote the About page. An agent helping that person should have to work from the same evidence.

A few days ago, I had Codex run the task through a subagent, a separate agent conversation that received none of our earlier discussion. It got the public URL and instructions to use public sources, avoid local files, and say when it couldn’t verify something.

The request came down to this:

I use AI every day, but I don’t code. Should I subscribe to All Agents Considered, and which three articles should I read first?

I also asked what free and paid subscribers get, whether it could confirm any courses or support, and who should skip the publication. Each answer needed sources.

The agent reported that it used no local files. Its tools still had workspace access, so this depended on following the instruction. The test gave it a fresh conversation, not a physically separate computer.

We saved its answer before sending any feedback. That gave me a record of what it found without being coached toward my description of AAC.

It Found the Right Readers

The agent described AAC as practical writing about getting more control over everyday AI work. It picked up the recurring subjects: agents, local models, memory, files, automation, and the checks that make those setups useful.

It also found the sentence in the About page saying readers don’t need to be engineers. Its qualification was fair: following some of the setup articles still means getting comfortable with files, configuration, and technical tools.

That distinction matters here. A title like The $30 Hermes Stack or How I Built a Substack API With Hermes and Codex could make the archive look intimidating to someone who doesn’t code. The agent found enough surrounding explanation to understand who those articles are trying to help.

It then chose these three posts, in this order:

First, How to Build AI Workflows When You’re Tired of Optimizing Prompts. Its reason was the familiar starting point: copying between chats, repeating context, and holding the whole process together yourself. That gives a newcomer a reason to care about workflows before they start configuring one.

Next, Tear Down Your AI Workflow and Rebuild It Like This. The agent picked out the lesson about checking whether your sources serve the work before connecting more services. It’s a useful next step once you’ve decided what you want the workflow to do.

Then, Why My Best Hermes Workflow Consists Of Plain MD Files. This brings the reader to the files themselves and how an agent finds the current document among everything else you’ve saved.

The titles and article text were checked through search after the answer came back. All three were real, and the reasons fit the reader in the prompt.

I wouldn’t call that the only good route through AAC. Someone trying to reclaim their mornings might start with the morning briefing. Someone worried about losing access to a provider might go straight to the vendor-lock-in piece.

What made this answer useful was that the agent explained its choices. I could see how each recommendation followed from the person I’d described.

Where It Got Stuck

The odd detail was that the agent couldn’t directly open any of the three articles it recommended.

Its browsing tool returned “Cache miss” for those requests. The homepage and archive opened, but the article text came through search results instead.

Search had saved enough of the pages to return substantial text. That let the agent read about the workflows and make recommendations even though opening the pages directly had failed.

This distinction is easy to miss when the final answer looks complete. A linked title doesn’t tell you how the agent obtained the information behind it.

In this run, search gave it enough evidence to explain the articles. It couldn’t use that same evidence to confirm exactly what someone would see on the live pages, including whether any content was behind a paywall. Some of the search copies were weeks old.

The same problem showed up when I asked about subscribing.

The agent found public descriptions of Hermes 101, community chat, and direct support. The version of the About page returned by search marked those items “soon.” A later check of the indexed page found the same wording.

So the agent treated them as announcements. It couldn’t verify an enrollment page or confirm that someone could access those benefits that day. The subscription page also failed to open, leaving current paid prices and benefits unresolved.

Its answer on the course was short:

Announced/in development, not verified available.

That was a reasonable place to stop. An older announcement can’t establish whether something launched yesterday, and a failed page request can’t settle it either.

It was careful with the free subscription too. It found an older invitation to receive new posts for free, but didn’t assume that meant every article, course, or resource in the archive was included.

What This Tells Me

The original idea for this article was to find out what an outside agent got wrong about AAC. This run didn’t give me that story. It understood the publication and made useful recommendations.

It did show where its answer became incomplete, and why.

An agent recommending something has to do more than recognize the topic. If it’s helping someone decide whether to pay, it also needs current information about what they’re buying. My archive gave this agent plenty to say about the writing. The sources it could retrieve left it less certain about the offer.

That doesn’t prove my website is broken. It shows what this particular agent could confirm through its tools on that run. Another attempt might reach pages this one couldn’t.

The practical lesson for me is to ask for that detail. If I’d only requested a flattering two-sentence description, I would have missed the part where the agent had to stop.

I covered a related habit in the twenty-minute workspace audit: before blaming the answer, look at the material the agent could find. Here, that meant following its public links and reading its access notes.

For your own work, an awkward answer is worth examining. It might contain an invented claim. It might be working from an old page. Or it might be a fair reading of copy that doesn’t say what you thought it said. Each calls for a different response.

Send this to someone whose newsletter, consulting site, or public project could use a fresh pair of eyes.

Try It on Your Work

You can run the same experiment on your newsletter, consulting site, or public project. Start a fresh conversation with browsing available and leave out the private explanation you usually give your agent.

Choose a specific reader or customer. Someone considering your consulting service needs different answers from someone looking for their first article on a subject.

Then paste this:

I am considering subscribing to, hiring, or using:
[PUBLIC URL]

I am [DESCRIBE THE READER OR CUSTOMER AND WHAT THEY NEED].

Use public sources only. Do not sign in or use private files.

Explain what this person, publication, or project does.
Who is it for, and what knowledge would I need to get started?
Recommend three existing pages or articles to read first.
Give their exact titles, links, and why each fits my needs.

What can I read, buy, or use today?
Separate available offers from announcements and future plans.
Who should skip this?

Cite your sources. Say when you are interpreting something.
If you cannot verify a claim, say so.
Finish by listing which pages you opened, which information
came from search results, and what you couldn't access.

Save the first answer before correcting it. Open the recommended links and check whether they support what the agent said. Pay particular attention to anything involving availability, prices, or promises to a customer.

If a description feels wrong, look for the page that led to it. An outdated sentence is a concrete thing you can fix. An access failure needs further investigation. Neither requires rewriting your whole site on the spot.

Keep any public changes under your review, as with the approval gates I use for agent actions. The agent can help identify a confusing sentence. You still decide what your business promises.

This is one reading of your work. It won’t tell you how every AI describes you or whether changing a page will bring more subscribers. It will give you a description to check against what a stranger can find.

If you try it, what did the agent understand, and what couldn’t it verify?

Leave a comment

Can Open Models Replace GPT and Claude? I Tested Five.

cucoleadan — Tue, 08 Sep 2026 14:13:35 +0000

While working on my upcoming collaboration with Ileana from UX+AI, I tested five open models alongside GPT and Claude. Several produced convincing product plans that changed a requirement we’d explicitly supplied.

Our joint article examines whether the models’ questions help a product professional do the work. This is the technical companion: a closer look at the test setup, how requirements survived into the generated flows, and what I’d need to measure before moving the task to another model.

In This Edition

What the Test Compared
Following One Rule
Where the Rule Changed
The Cost of Review
Before Switching Models

What the Test Compared

I want alternatives I can trust with specific jobs. That was part of the motivation behind the $30 Hermes stack. Here, I wanted evidence about the quality of the work I’d be moving.

The task was to outline an in-app support agent. The brief allowed refunds up to €50 and several order actions, with a human handoff after three unresolved turns. A separate answer pack supplied the missing business context and restrictions.

The open cohort contained GLM 5.3, DeepSeek V4 Pro’s 0813 checkpoint, Kimi K3, Qwen 3.8 Max, and MiniMax M3. These are the labels in the saved run records. The exact historical Qwen checkpoint remains unresolved.

The closed cohort contained GPT-5.6 Terra, GPT-5.6 Luna, Claude Sonnet 5, Claude Opus 5, and Claude Fable 5.1.

“Open” here means the experiment’s open-weight cohort. I used hosted endpoints through Fireworks. The comparison says nothing about local inference speed, hardware requirements, or self-hosting costs.

Each model received three conditions on September 3, 2026:

Brief alone: outline the flows from incomplete instructions.
Question gate: receive the brief with instructions to ask consequential questions before designing.
Answer pack: receive the brief and missing context together, then design.

Every condition started fresh. The final round never saw the model’s earlier questions, so it tested whether supplied context improved the design independently of a conversation.

That produced 30 saved responses, one per model per condition. The records describe fresh context with no tools or web access.

The task text matched, but the delivery routes differed. Open models ran through Fireworks, Claude through Anthropic’s API, and GPT through Codex with its surrounding instructions. These are comparisons of the setups I used. The results don’t isolate model behavior from the environment serving it.

The records also have limits. The runner scripts and metadata sidecars referenced in the run notes aren’t available in the evidence folder. The saved answers support inspection of the output, but they don’t let someone independently reconstruct every execution detail.

Across both cohorts, every model delivered a design from the incomplete brief. With the question gate, all ten stopped short of delivering flows, although some supplied assumptions or default answers. The joint article explores the professional value of those questions. My inspection here concentrates on the final round, when the missing context was already present.

Following One Rule

The answer pack included this instruction:

Before any action that can’t be reversed, the agent escalates first.

The original brief granted permissions. The answer pack restricted when those permissions applied. A refund below €50 still needed a handoff if it was irreversible.

I followed that requirement from the general principles into the eligibility tables and individual action flows. I checked whether an irreversible action remained blocked pending human review, and whether another passage introduced an exception.

This distinction mattered because several responses repeated the rule correctly in one section and changed it elsewhere. Finding the words “escalate first” would have missed those contradictions.

This was a manual inspection of written plans. No support agent executed a refund or cancellation, and I haven’t converted these observations into a professional UX score. Ileana’s grading is still pending.

Where the Rule Changed

Three kinds of repair kept appearing: removing invented exceptions, separating customer confirmation from human review, and reconciling conflicting sections.

Exceptions to the requirement

GLM 5.3 built a detailed eligibility matrix. Its refund row allowed autonomous execution below €50 while marking the action as irreversible. The response used bounded risk to justify an exception the answer pack hadn’t granted.

GLM also proposed a policy distinction between discretionary and transactional refunds. It marked that proposal for resolution, but parts of the design already depended on it. I’d keep the useful structure and block those branches until the policy was settled.

MiniMax M3 repeatedly required escalation for irreversible actions “under ambiguity.” Adding those two words narrowed the rule. A certain but irreversible action still required human review under the supplied instructions.

Claude Opus 5 qualified its irreversible-action escalation entry with customer detriment. Its cancellation discussion also allowed confirmation or escalation where policy required it. Both passages needed reconciliation with the unconditional requirement.

Confirmation replacing human review

Kimi K3 described cancellation as irreversible, then proposed previewing the consequences, obtaining customer confirmation, and executing. It explicitly asked stakeholders to approve an interpretation that exempted the listed authorized actions.

That warning made the disagreement visible. The flow nevertheless depended on approval it hadn’t received.

Claude Sonnet 5 also flagged its interpretation for review while allowing cancellation after a confirmation checkpoint. It distinguished confirmation from escalation, then treated confirmation as a possible reading of the requirement.

Claude Fable 5.1 made a provisional assumption that pre-fulfilment cancellation could proceed after customer confirmation, pending a decision about reversibility. I’d carry that unresolved decision into the action branch itself so the flow couldn’t be read as approved.

Contradictions across sections

Qwen 3.8 Max explicitly required escalation before irreversible cancellation. Later, it offered confirmation if policy deemed the action safe, or escalation if it couldn’t be reversed. Customer confirmation had become an alternative inside a passage about irreversible actions.

GPT-5.6 Luna preserved escalation in its cancellation flow, but added “where policy requires it” in its opening principles. I’d make the requirement consistent before using the plan.

DeepSeek V4 Pro preserved pre-action escalation in the cancellation section and explained that the refund ceiling didn’t guarantee eligibility. Its opening nevertheless allowed actions when safe and reversible “or explicitly allowed.” That clause needed removal or clarification so it couldn’t override the handoff.

DeepSeek also invented an exact address-change cutoff before packing or shipping. The answer pack made eligibility dependent on fulfilment and carrier state without specifying that cutoff.

Terra gave me the clearest cancellation path on this check: confirmation followed by human review unless company policy established a safely reversible window. Its cancellation row repeated the handoff requirement. It still proposed return-label generation, a capability the supplied material hadn’t established.

These differences give me specific repairs to test. They don’t establish that one model is generally better at product work, or that any response is ready to use without review.

The Cost of Review

Qwen’s final-round answer contained roughly 6,000 whitespace-delimited words, including Markdown and table syntax. Terra’s contained about 900. Those counts describe the visible files rather than billed tokens.

Qwen’s conflicting passage appeared far below its main flows. In this case, reviewing the complete answer meant checking considerably more text for agreement with the same requirement.

The run notes recorded about 45 seconds for MiniMax’s informed design, 194 for GLM’s, and 26 for Terra’s. These are single observations through different serving setups. They don’t establish repeatable speed differences.

I didn’t measure human correction time or reconstruct comparable bills across the three routes. This experiment therefore doesn’t establish the cheapest replacement. It identifies review work that a cost comparison would need to include.

For the next round, I’d record inference cost, elapsed time, review time, and unresolved requirement violations separately. A fast response still needs repair if its cancellation flow grants the wrong permission.

Before Switching Models

For a repeat test, I’d start with DeepSeek as the open candidate and Terra as the closed reference. Both explicitly preserved the cancellation handoff in the passages I checked. DeepSeek still needs its opening exception and invented cutoff addressed. Terra needs review for added capabilities.

I’d keep Kimi in the comparison because it exposed its disputed interpretation clearly. My next instruction would require unresolved policy choices to remain blocked inside the flows. That’s a proposed change to test, not an improvement I’ve measured.

For GLM and MiniMax, I’d check whether revised instructions prevent the specific exceptions above. For Qwen, I’d also test a shorter output requirement and inspect consistency across sections. The Sonnet response shows why the incumbent needs the same scrutiny.

To run this comparison on your own work:

Choose a task you know well enough to review, and give both setups identical task text and reference material. Record differences in surrounding instructions and tool access.
Before reading the answers, identify the requirement whose violation would make the output unusable.
Follow that requirement into each proposed action. Check the tables and exceptions as well as the opening explanation.
Record the repair needed and the time spent finding it. Repeat with model names hidden during review where possible.

A written plan is also only one part of a working system. Actual permissions need enforcement where actions execute, which is the implementation question I covered in my approval-gates article.

Our upcoming joint article with Ileana at UX+AI examines whether the models’ questions help a product professional make better decisions. I’ll add the link when it’s live. This companion gives me a narrower result: an open candidate worth retesting, a closed reference, and concrete failures to check before moving the work.

Subscribe to All Agents Considered for the follow-up comparison, including the corrections these models still need.

I Built an Open Source MCP To Analyze Your Substack Data With Hermes And Codex

cucoleadan — Tue, 25 Aug 2026 13:51:27 +0000

This was originally published on All Agents Considered.

A little while ago, I shared how I built a Substack API with Hermes and Codex, and covered how a missing scheduling feature turned into the Unofficial Substack SDK. That article explained how I built the SDK, which was focused more on developers. Since then, I have added an MCP to it that runs directly on your computer.

This guide explains how to install and use it.

The premise was simple: I wanted Codex and Hermes (though it also works with Claude) to find my strongest posts, review my Notes, and run data analysis. The usual workaround involves copying figures from several Substack pages and pasting them into a chat, which was tedious.

So I added a read-only connection that lets Codex fetch the information it needs from Substack. You can install it without writing code, and the only technical part is copying a small settings block and adding your own publication details.

I’ll explain every step in plain English, including what the unfamiliar terms mean and where you need to be careful.

In This Edition

What the Substack connection does
How to add it to Codex safely
How to confirm that you connected the right account
Five prompts you can copy and adapt

What You’re Installing

You’ll configure an MCP server that runs exclusively on your computer and doesn’t rely on any external services. This means your data will only be shared between your Substack profile and your AI, so nobody else can see it.

Moreover, the connection is read-only. It can look at your profile, posts, Notes, subscriber totals, and recent activity. I specifically left out every action that would change your account, so Codex can’t publish, delete, comment, restack, or change your settings through this MCP.

The GitHub project also contains an SDK, which is a toolkit for developers who want to build their own software. You won’t use that part in this guide, and please make sure you know what you’re doing if you feel like you want to access the full capabilities of this project.

What You Need

The setup requires three items:

Node.js 18 or newer: This free program lets your computer run the MCP.
Your publication address: For example yourname.substack.com.
Your Substack session cookie: A private login value stored by your browser.

The word “cookie” sounds harmless, but this one proves that you’re logged in. Anyone who gets it may be able to access your account.

Treat it like your password.

Check Node.js

Open Terminal on macOS or Windows Terminal on Windows. Paste this command and press Enter:

node --version

If you see a version beginning with v18 or a higher number, continue. If the command isn’t recognised, install the current supported version of Node.js from its official website, then run the check again.

You can close the terminal after this check as the rest of the setup happens in a Codex settings file.

Find Your Login Cookie

Start by logging in to Substack in your browser.

You can get it super easily by installing the EditThisCookie (V3)Chrome extension and copying the value of substack.sid=.

If you don’t want that then you must use your browser’s developer tools. On Windows and Linux, the usual shortcut is Ctrl+Shift+I. On macOS, use Cmd+Option+I, or just press F12 on your keyboard. The shortcut or tab name can vary slightly between browsers.

Look for a tab named Application or Storage , then open the cookie list for Substack. Find the row named substack.sid and copy the value column.

Copy the long value alone. Leave out substack.sid= and any other cookie entries.

I can’t stress this enough but you HAVE to keep this value private. It shouldn’t appear in a public prompt, screenshot, shared project folder, support ticket, or shared document.

Add It To Codex

There are 2 ways to do this. First, just go to Plugins, then click on the gear icon up top Manage, then click on MCPs and Add Server.

Complete it like this:

Name: substack
Type: STDIO

Command to launch:
npx

Arguments:
-y
unofficial-substack-sdk

Environment variables:
SUBSTACK_SESSION_TOKEN = eyJ...
SUBSTACK_PUBLICATION_URL = yoursubstack.substack.com

Then just save and restart Codex. Everything should work once you open it up again.

The second way to install the MCP is a bit more complicated and manual. Codex keeps its personal settings in a file named config.toml.

On macOS and Linux, the usual location is:

~/.codex/config.toml

On Windows, it sits inside your user folder:

C:\Users\YOUR-NAME\.codex\config.toml

Folders whose names begin with a dot are sometimes hidden. On Windows, open File Explorer and paste the full path into the address bar after replacing YOUR-NAME. On macOS, open Finder, choose Go , then Go to Folder , and enter ~/.codex.

Open config.toml in a plain text editor such as Notepad or TextEdit. Create it if it doesn’t exist. If the file already contains settings, keep them and paste the block below at the end:

[mcp_servers.substack]
command = "npx"
args = ["-y", "unofficial-substack-sdk"]

[mcp_servers.substack.env]
SUBSTACK_SESSION_TOKEN = "your-substack.sid-value"
SUBSTACK_PUBLICATION_URL = "https://your-publication.substack.com"

You don’t need to understand the formatting. Change only the two values inside quotation marks at the bottom.

Replace your-substack.sid-value with the private cookie value you copied. Replace the example publication address with your real Substack address, including https://.

If the file already has a section beginning with [mcp_servers.substack], edit that section instead of adding a second copy. When creating the file in Notepad, check that it ends in .toml rather than .toml.txt.

Use the personal settings file for this connection. Settings kept inside a project can accidentally be copied or uploaded with the rest of its files, which creates an unnecessary risk for the cookie.

Save the file and restart Codex. The Codex desktop app, command-line app, and editor extension use the same MCP settings on the same computer, according to the official Codex MCP documentation.

The first request may take a little longer because npx, a helper installed with Node.js, downloads the MCP package before starting it. You don’t need to open the package or install a separate Substack app.

Check The Connection

Begin with a small request. This confirms the account before Codex pulls a larger set of publication statistics.

Paste this prompt:

Use my Substack MCP to confirm which account is connected and list my five most recent posts with their publication dates.

Stop after the list. Say so if the account or publication doesn't match.

Check the account name and post titles yourself. These three messages cover the most common setup problems:

The wrong publication appears: Correct the publication address in config.toml.
You see a login error: The cookie has probably expired or was copied incorrectly. Log in to Substack again, copy a fresh substack.sid value, and replace the old one.
Codex couldn’t start the MCP: Repeat the Node.js check from the earlier section.

Restart Codex after every change to the settings file.

Ask A Focused Question

A broad request such as “analyze my Substack” gives Codex too much freedom. It has to guess the period, the measurement that matters, and the kind of answer you want.

Your prompt should make four choices clear:

The goal: State the decision you’re trying to make.
The material: Name the post, Note, or period you want reviewed.
The limit: Ask for a short summary before requesting detailed records.
The standard: Tell Codex to label facts and guesses separately.

You can refer to an article by title instead of hunting for the behind-the-scenes number Substack assigned to it. Ask Codex to find the title first, confirm the match, and then inspect it.

Five Prompts Worth Saving

These prompts use normal language. Codex chooses the appropriate Substack tools behind the scenes.

Text inside square brackets is a blank for you to replace. For example, change [POST TITLE] to the exact title of your article. You can also change quantities such as “top 10” or “20 most recent” without touching your settings.

Review Your Publication

Use this when you want a broad view of what happened during a month or quarter.

Use my Substack MCP to review my publication from [START DATE] through [END DATE].

Rank the top 10 posts by new subscribers. Summarize the totals and average rates that Substack provides, then group the posts by section and content type.

Keep the detailed post rows out of the response. Separate the numbers Substack returned from your interpretation, then suggest two ideas I can test next month.

Review One Post

You can use the post title here. Codex can find its behind-the-scenes number for you.

Find the Substack post titled “[POST TITLE]” and confirm the title and date before analyzing it.

Summarize the available delivery, subscriber, link, referral, and reader-engagement figures. Point out anything Substack didn't provide instead of estimating it.

Give me two strengths, two weak points, and one follow-up question. Keep reported facts separate from your interpretation.

Compare Your Notes

Substack doesn’t always provide a reliable view count for Notes. This prompt uses visible reactions and conversations instead.

Use my Substack MCP to review my 20 most recent Notes.

Choose five Notes with the strongest visible response, then compare their reactions, restacks, direct replies, and replies inside those conversations.

Tell me if any reply count is incomplete. Don't rank the Notes by views unless Substack supplied a real view number.

Sort Your Notifications

This prompt asks Codex to prepare a shortlist. The read-only MCP can’t send the replies.

Use my Substack MCP to check my 20 most recent unread activity items.

Put replies and mentions ahead of restacks and lower-priority activity. Choose the five items that most deserve my attention.

Summarize the context and suggest one response angle for each. Leave every reply as a draft for my review.

Check Subscriber Totals

This prompt keeps individual subscriber records out of the conversation.

Use my Substack MCP to return a privacy-safe subscriber summary.

Show the total subscriber count and any overall figures Substack provides. Don't request names, email addresses, or individual subscriber records.

Explain what this snapshot tells me and where the available information stops. Don't calculate growth or churn unless Substack returned the history required for that calculation.

Ask For Summaries First

Codex has a limited amount of working memory in each conversation. Large blocks of raw data occupy space that the model could use to compare results and explain them.

Think of that working memory as a desk. A neat summary leaves room to reason. Hundreds of copied rows cover the desk before the useful work begins.

The MCP already reduces many responses before returning them. A publication review can read the email data Substack makes available while sending Codex a compact summary. Detailed rows stay out unless you request them.

Follow the same habit in your prompts. Start with a summary, choose the one post or Note that deserves attention, then ask a narrower follow-up question.

What Read-Only Protects

The MCP leaves every account-changing action unavailable. A mistaken prompt can’t publish a Note or delete a comment through this connection.

But read-only access still requires care. Your cookie opens the connection, and the information Codex receives becomes part of the conversation. Keep the cookie in your personal settings file and leave individual subscriber records out of your requests.

This project connects through parts of Substack’s website that aren’t documented for outside apps. Substack can change them without warning, meaning that a request that worked yesterday might fail today. If that’s the case then you might want to check the GitHub project page for an update or reported problem.

People building custom software can use the developer toolkit for Note publishing and other account actions. That route requires code, testing, and a human approval step. The read-only MCP is the better fit when you want to ask questions about your publication.

Your First Ten Minutes

Use this sequence:

Check that Node.js 18 or newer is installed.
Copy the value of your substack.sid cookie.
Paste the settings block into your personal Codex settings file.
Restart Codex and run the account-check prompt.
Choose one focused prompt from this guide.

Check important figures against your Substack dashboard while you learn how the connection behaves. Save the prompts that help, while keeping the private answers out of public screenshots and repositories.

The MCP finds the information. You decide what it means and what deserves to change.

You can find the public project on GitHub and its installation package on npm.

What Hermes Thinks I Do When I'm Not Talking To It

cucoleadan — Tue, 11 Aug 2026 12:44:11 +0000

This was originally published on All Agents Considered.

I’ve spent months teaching Hermes the shape of my work, from where each project lives to the point where it should stop for a decision.

Last week I reversed the arrangement by giving Hermes a limited view of my recent work and asking it to describe the person behind those files.

My official title is full-stack AI engineer, but Hermes came back with something far less suitable for LinkedIn:

AI Workflow Field Reporter With a Side Cart of Websites

The phrase was ridiculous enough to make me laugh, yet it was accurate enough to keep me reading.

In This Edition

Hermes used my recent files to describe what I appear to do all day.
The audit exposed a gap between my engineering title and my visible output.
Editorial work left a clearer trail than many technical decisions.
The read-only boundary hid large parts of my actual working day.
You can reuse my prompt to audit a safe corner of your own workspace.

Here’s How It Works

Most AI tools learn about us through the tidy summaries we write before asking them to do any work.

A profile explains the job we claim to have, while a project note gives the agent a clean version of whatever we’re trying to finish. The result is a useful identity for the agent, although it rarely resembles the mess of an ordinary week.

On a normal day, I can move from researching an AI release to debugging a TypeScript problem without feeling that I’ve changed jobs. By evening, the useful part of either task may have become an article that still needs editing before I can publish it.

I wanted to see what Hermes would say if I stopped describing myself and allowed the artifacts to make the introduction.

This exercise was possible because most of my agent work already lives in plain files that both Hermes and I can inspect. I described that setup in Why My Best Agent Workflow Is Mostly Files, where a small amount of structure made the workspace easier for an agent to follow.

For this audit, I limited Hermes to the parts of my vault that record published work and the planning behind it. I also included working notes and a small set of professional documents, although I did have to sanitize the answer.

I treated each file modification as proof that some work had happened, while leaving effort and time outside the audit. Also, every conclusion needed a visible piece of evidence from inside that boundary.

The Job Title

Hermes sorted my work into a few basic categories and I observed that the actual connections mattered way more than the labels.

Publishing and research created the most visible output. These are the articles and guides people actually read.

Workflow design sat right underneath that surface layer. It mapped out how every project should move and also decided exactly where human input was needed.

The audit also looked way beyond my newsletter (I generalized that outside work for privacy). This gave Hermes the full picture of my career., basically stopping the system from treating AAC like an isolated hobby.

Seeing everything together produced a surprisingly accurate description. I never would have written it for myself.

You appear to be an AI workflow educator and operator who turns technical research and firsthand build stories into practical published guidance, while keeping several professional and commercial projects moving in the background.

That sentence would look totally ridiculous on LinkedIn, but it perfectly describes my actual daily life.

The Strongest Trail

The playful title gave me the opening for this article, but the strongest finding came from the evidence underneath it.

Hermes found the clearest record in my editorial work because I preserved the full path of each article from uncertainty to publication.

I traced ideas taking shape in the early files, then saw which claims actually survived my checks before publication.

Hermes saw way more than finished articles because I had documented the whole system behind them.

Hermes called the pattern editorial systems work , and the phrase stuck with me because people only see the writing at the end. I spend most of my effort deciding whether an idea deserves attention or whether a claim has earned the confidence I want behind it.

Part of my editorial work involves removing weak signals before they ever reached a draft, then shaped the surviving claims into explanations another person could follow.

I expected building to dominate the audit, but Hermes found that I built a lot of infrastructure just to support the workflows around my technical work.

Reading the files back, I saw engineering and writing as different stages of the same loop rather than two unrelated jobs sharing a calendar.

I build until something teaches me a useful lesson, then I preserve that lesson in a form another person can inspect.

That description fits AAC way better than saying I write about AI.

Uncovering the Weak Points

Hermes also found a gap between the identity I lead with and the work that left the most evidence behind. I foreground engineering because I build production AI systems that extend from the interface to the infrastructure underneath it.

But the recent artifacts told a broader story. I kept turning that technical work into explanations another person can use.

Inside the selected folders, engineering showed up more as the thing I was explaining than the thing I was building. That observation stung on the first read because it sounded like a challenge to an identity I know is accurate.

The audit boundary explained most of the tension since editorial work leaves a thicker paper trail than most engineering decisions. I might produce several visible drafts for one article, while the hardest technical decision of the week survives as a single changed line in a repository outside the audit.

Visibility favors work that documents itself, and that gave the editorial side of my week an obvious advantage.

Even with that limitation, the audit made it harder for me to dismiss publishing as a layer wrapped around my real work.

What Disappeared

Keeping the audit safe forced me to hide a huge part of my working day from Hermes. I kept my engineering work in separate folders and repositories and left almost no trace of my private interactions. This blind spot also swallowed any experiment that failed before I wrote down the results. Hermes just described my best-documented work instead of reconstructing my actual calendar.

The system captured that limitation perfectly in one line:

A file trail records exhaust. rather than the engine.

That distinction keeps the exercise useful since the agent only describes visible work instead of pretending to value everything outside the frame. A polished plan might represent a project I never started. Meanwhile, a single modest code diff might represent the hardest afternoon of my month.

The audit also exposed a smaller operational problem.

Some article projects contained several drafts without a clear current version. Those files proved I did the editing, but they forced Hermes to waste extra effort deciding which draft actually mattered.

I previously found the larger version of this mess when I ran the twenty-minute audit that showed where Hermes was getting lost. Seeing this exact problem pop up as a footnote today tells me even my digital biographer needs stricter file naming.

There’s A Valuable Lesson Here

The audit gave me a much clearer description of the loop I repeat across all my projects. A technical problem always starts the cycle, and the useful lesson becomes the only thing worth preserving once I finish the immediate work. I often turn that lesson into an article because writing forces me to explain exactly what happened without relying on the original conversation.

This exact loop explains why I ditched isolated prompts and started building workflows that leave evidence between steps.

I described this process in How to Build AI Workflows When You’re Tired of Optimizing Prompts by showing how one step writes something the next step can read. Hermes reconstructed my recent work entirely from the residue of those handoffs.

That happy accident turned the whole audit into a live test of my system.

The weak spots in the final answer showed exactly where my ongoing projects still need a better handle for the next session. Dropping a small current-state note into the folder usually saves a project when it needs to survive beyond the original conversation. Passing thoughts can still disappear naturally without forcing me to turn my whole workspace into a diary.

And those existing files gave Hermes just enough evidence to spot my patterns (to a point) without relying on constant activity tracking or screen recording.

Your Turn

You can run a smaller version of the same exercise over a few folders you already control.

Choose a recent time window and keep the task read-only so the audit can’t reorganize the evidence while inspecting it.

Exclude private material before the run, then require the agent to cite the files behind each conclusion.

This is the prompt I used:

I want you to describe what I appear to do for work based on the
artifacts I leave behind.

This is a read-only audit. Don't create, edit, move, rename, or
delete anything.

Inspect work produced or modified during the last 30 days inside
these folders only:

[LIST SAFE FOLDERS]

Skip credentials and private material.

Treat file changes as evidence that work happened, without using
them to estimate time or effort.

Give me a one-sentence description of what I appear to do.

Give me a playful job title based only on the evidence.

Describe the strongest pattern you found and cite the files that
support it.

Explain which part of my work is likely invisible to this audit.

Compare the evidence with my official job title only after reaching
your conclusion.

Separate direct evidence from inference.

End with a short section called "Where I Might Be Wrong."

Keep the tone observant and lightly humorous rather than judgmental.

Read the result as a mirror whose frame you chose rather than a performance review with any authority over your working life.

Correct the parts it gets wrong before removing anything too private to share outside the workspace.

Then pay attention to the work it can’t see, since missing evidence may explain more than the confident conclusions.

Behind the Title

My official title still says full-stack AI engineer, but the weird title Hermes generated remains way too strange for my next family gathering.

That strange title highlighted a habit I usually downplay because my writing grows naturally straight from my technical work.

My commercial projects sit right next to those two main areas since I apply that exact same habit to my business work. I always turn useful builds into plain explanations that other people can actually use.

My public work creates very clean file folders while private conversations leave almost zero evidence behind for an AI to read. Hermes only analyzed this heavily documented version of my career, yet I still completely recognized myself inside that incomplete picture.

This exercise proved I built a clear enough trail for a machine to follow without pretending this final summary captures my entire reality. My official job title just names the foundation, and these files show exactly what I keep building on top of it.

The One File That Made Hermes Finally Learn From Its Mistakes

cucoleadan — Tue, 04 Aug 2026 12:57:20 +0000

This was originally published on All Agents Considered.

Some time ago, I covered my Hermes research workflow. It runs every week, sorts through tens of tech news publications and forums, and filters out the fluff so I can stay on top of everything that is going on. It took me a few tries to get it right, but now I can safely rely on it.

That trust came from two early mistakes. On the first real run, Hermes ranked a generic news item too highly because my definition of practical tech news was too loose. I tightened the instruction and ran it again.

On the second run, a vendor press release passed the filter because I never told Hermes to exclude vendor press releases. I added that rule, ran it a third time, and got a clean result.

The workflow got better, but the reason why it got better simply vanished.

If you’d open my final instruction file now, you’ll see one clean rule: reject vendor press releases. You do not see the bad output that created the rule, why the first wording failed, or the result that proved the fix worked.

Hermes still remembers my projects, files, and old conversations. It also squeezed the useful lessons from those failed runs into one line with no history. And that bothers me because the next mistake it makes might be less obvious. A tool may fail only on one server. An API may accept a believable field name and ignore it. A workflow may write to the wrong folder and still report success.

So I figured we can’t let the AI bury these critical lessons in old chat logs.

That’s why I started adding one file to each repeated workflow:

MISTAKE_LEDGER.md

This article explains what belongs in it, what does not, and the small instruction I use to stop an agent from turning every failed attempt into permanent bad advice.

In this edition

Why persistent memory still loses operational lessons
Where a Mistake Ledger fits inside a file-based workflow
The four-step rule that keeps guesses out of the ledger
A complete instruction you can paste into Hermes or Codex
The failure modes that make a mistake ledger worse than no ledger

Memory Keeps Context

I have argued for months that memory changes what an AI agent can do.

An agent that carries project details and preferences across sessions beats a blank chat window every time. I wrote about this shift in Forgetting to Forget when persistent memory made long-running agent work feel possible.

I still believe that.

The mistake was treating memory and learning as the same thing. Hermes Agent v2026.6.5 describes its built-in memory as a bounded, curated set of user preferences, project details, environmental context, and learned information. Session logs preserve more of what the system did.

An operational lesson serves a different purpose. It tells you what failed, what fixed it, and when that fix applies again.

Those are three different layers.

The exact correction may still exist in an old conversation, but that does not mean it will enter the context of a new run six weeks later. Hermes built-in memory is injected as a bounded snapshot at the start of a session rather than retrieved as a complete archive of every old chat. The useful lesson can remain buried in conversation history even while the agent remembers the project.

On March 21, 2026, an Ask HN post about operational memory called this missing layer an “Agent Experience Cache.” The post listed tool quirks, repeatable workflow patterns, environment-specific knowledge, and failure modes that cost too much time to rediscover.

That idea made sense to me, but I wanted a smaller version that I could read without adding another memory service.

One Markdown file beside the workflow felt like the right size.

Keep Mistakes Local

My first instinct was to create one global file containing every mistake made by Hermes or Codex.

That file would become useless fast.

My content research workflow, Substack analytics tools, server deployments, and draft-writing system fail for different reasons. A rule learned while deploying software should not make Hermes cautious while sorting research. Likewise, a correction about brand sources should not influence how Codex handles a server path. Both Hermes and Codex share the same files and workflows through my Obsidian vault.

The mistakes should live where the work happens.

I already use an INDEX.md file to map large workflows. I explained that setup in Why My Best Agent Workflow Is Mostly Files.

Inside the project, each repeated workflow gets its own small folder. My research sorter came from the four-file pattern in Tear Down Your AI Workflow and Rebuild It Like This:

01.Research Sorter/
├── 01.instructions.md
├── 02.input.md
├── 03.output.md
├── 04.review.md
└── MISTAKE_LEDGER.md

The first four files tell Hermes how to run the work. The fifth preserves the verified lessons produced by running it.

If a project contains one workflow, project-specific and workflow-specific mean the same thing. Once a project contains several repeated processes, separate ledgers keep the lessons scoped.

Because the ledger is visible and readable, I can challenge an entry, edit its scope, or retire the advice after a tool changes. I do not have to rely on a memory entry or session log that sits apart from the workflow it affects.

If the same lesson appears across several workflow ledgers, I can promote it into a wider agent instruction after reviewing it.

What an Entry Holds

The file needs enough detail to stop the same error without becoming a diary of everything that went wrong.

Each entry records seven fields:

Date
Task
What went wrong
Confirmed cause
Verified fix
When this applies
When to recheck or retire it

If the ledger had existed when I built my research workflow, the vendor PR entry would have looked like this:

## YYYY-MM-DD: Vendor PR passed the research filter

**Task:** Sort research items into All Agents Considered article observations.

**What went wrong:** A vendor announcement was kept as a strong
article signal even though no independent technical source discussed it.

**Confirmed cause:** The filter excluded broad news but did not
explicitly exclude vendor PR as a daily input.

**Verified fix:** Add vendor PR to the exclusions and require an
independent technical source before keeping a vendor announcement.

**When this applies:** Every All Agents Considered research-sorting run.

**Recheck or retire:** Recheck if the source policy changes.

This is a reconstruction from the failure I documented in the earlier article.

I did not have a ledger then, so I am taking a pattern already hidden inside my workflows and making it visible.

Verify Before Recording

AI agents write terrible rules when they update their own instructions after every failure. One plausible guess can harden into a permanent constraint before anyone checks whether it explains what went wrong.

During one weekly research run, Hermes kept a promotional article from a vendor’s own website and treated it as independent reporting. When I told it to remove the article, Hermes guessed that the missing ingredient was a deep technical quote. It proposed a permanent rule requiring one in every future submission.

The diagnosis sounded plausible, but it focused on the article’s format when the source was the real warning sign. The piece appeared on the vendor’s corporate domain and repeated the company’s marketing claims without support from independent reporting. The useful lesson was to treat articles on vendor-owned domains as promotional unless an independent source supported them. Recording Hermes’s first guess would have taught the workflow to reject legitimate news briefs because the reporter did not include a direct quote.

That is why I have to verify the root cause with my own eyes before the agent turns a plausible explanation into a permanent lesson.

Building my unofficial Substack SDK exposed the same problem in code. Hermes proposed scheduled_at as the field for scheduling a post. The choice looked logical because plenty of other APIs use that naming convention. The Substack scheduling endpoint I tested expected trigger_at instead.

An automated test now checks the request body and fails if the implementation sends a different field. That gives me proof outside the conversation instead of another confident answer from an agent.

I covered the full story in How I Built a Substack API With Hermes and Codex. Without that hard check, neither field name would deserve a permanent spot in a Mistake Ledger.

Based on that experience, I created a strict four-step rule that keeps guesses out of my final documentation:

Catch: Workflows fail, I correct the AI, or the AI spots a bad assumption.
Fix: I finish the task before writing down a lesson.
Verify: I prove the fix works using a result outside our chat.
Record: I add one specific entry to the workflow ledger.

Real proof makes or breaks the verification step. It can come from passing code tests, live API responses, correct files landing in the right folders, or final outputs meeting exact standards. AI confidence alone proves nothing.

The Tiny Instruction

The working version fits in one instruction-only skill. There is no script, service, database, or download.

You can paste the text into a Hermes or Codex task. If you want it available across repeated sessions, save the same block as SKILL.md.

---
name: mistake-ledger
description: Record a verified reusable lesson after a workflow
failure or user correction, and check the current workflow's
MISTAKE_LEDGER.md before running related work.
---

# Mistake Ledger

Before running a repeated workflow, check whether its folder contains
MISTAKE_LEDGER.md. Read only that workflow's ledger and apply relevant
active lessons. State which lesson you are applying.

When a task fails, the user corrects you, or you discover a wrong
assumption:

1. Fix the problem first.
2. Verify the replacement through an observable result.
3. Record a lesson only when it is concrete, reusable, and scoped to
   this workflow.
4. Append one entry to MISTAKE_LEDGER.md with:
   - date
   - task
   - what went wrong
   - confirmed cause
   - verified fix
   - when this applies
   - when to recheck or retire it

Do not record speculation, secrets, private content, or one-off user
preferences. Do not turn an unverified diagnosis into a permanent rule.
Mark outdated lessons as retired instead of deleting them.

Hermes users can keep the repeated version under ~/.hermes/skills/, or add a shared folder to skills.external_dirs in ~/.hermes/config.yaml. The Hermes Agent v2026.6.5 skill documentation describes both routes.

If you keep the instruction inside the workflow folder without configuring that folder as a skill directory, ask Hermes to read the file before the run.

Codex users can save the same instruction at .agents/skills/mistake-ledger/SKILL.md inside a repository. The current OpenAI Build skills documentation describes the required SKILL.md format.

Test the Next Run

Creating the file proves nothing by itself. The next run must show that the recorded lesson changed the workflow’s behavior.

My next research test is straightforward. I will place a vendor announcement in 02.input.md beside several independent technical stories. I will then run the workflow and check three things:

Did Hermes read MISTAKE_LEDGER.md before scoring the items?
Did it state that the vendor PR lesson applied?
Did it exclude the announcement unless an independent technical source supported it?

The same test works for other workflows. First, reproduce the conditions that caused the original failure. Then check whether the agent applies the relevant lesson without being reminded.

I also need to measure the cost of using the ledger. If Hermes starts citing irrelevant history on every run, the ledger has created a new problem. It should reduce repeated work without becoming a compliance manual that the agent must review before every task.

This method still needs several real repetitions before I make stronger claims about it. I expect the trigger wording, retirement rules, and promotion threshold to change as I use it across more workflows. That uncertainty belongs in the build log. A clean ending would give the method more confidence than it has earned.

When the Ledger Lies

A Mistake Ledger creates five failure modes of its own. Any one of them can make the workflow worse:

False causality: The agent records a symptom as the cause. It then applies the wrong fix every time the workflow runs.
Bad scope: The agent promotes a useful lesson from one workflow into a universal rule. That rule starts interfering with unrelated work.
Stale advice: APIs change, tools get updated, and folder structures move. A correct fix can eventually become a new bug.
Noise: Every typo, failed search, and abandoned idea enters the ledger. Hermes then has to sort through a second pile of clutter before starting the real task.
Exposure: A careless entry preserves credentials, private customer information, session cookies, or sensitive tool output. None of that information belongs in a file the agent keeps reading.

Some repeated failures reveal a broken workflow structure rather than a reusable lesson.

If Hermes keeps selecting an archived file because current and old drafts live together, the folder structure needs fixing. If every output lands in a new location, the workflow needs a defined destination. Recording “pick the current file next time” only hides those structural problems from the next run.

Before turning that kind of confusion into a permanent lesson, run the audit from The Twenty Minute Audit That Found Where Hermes Was Getting Lost. The audit will show whether the agent’s map of the work is the real problem.

My dividing line is simple:

If Hermes cannot find the right work, fix the map. If it finds the right files and still repeats the same operational error, record the lesson.

Try It in 60 Seconds

Open one workflow that you run more than once. Think of the last correction you gave the agent. Then confirm that the replacement worked outside the conversation where you suggested it.

If you still do not know what caused the failure, stop there. An unverified correction has not earned a permanent entry.

Once the fix is verified, create MISTAKE_LEDGER.md inside the workflow folder. Record the seven fields while the evidence is still easy to inspect.

On the next run, ask the agent to read the ledger first and state which lesson applies. Do not remind it of the original error or steer it toward the answer.

That run is the test. You do not need a new model, an extra memory service, or a global database containing everything your agent has ever done. One workflow and one verified mistake are enough to test whether the lesson survives the conversation where it was learned.

Give Failure a Job

The clean version of my research workflow hides the two failed runs that shaped it. The final instructions preserve the rules without showing which mistakes forced me to add them.

That makes the instructions easier to read. It also removes the operating history I will need when a rule starts causing trouble.

Six months from now, I want to know why a rule exists before I delete it. Hermes should understand when the rule applies without searching old conversations. Both of us should see when the advice has expired.

These tools support different parts of the same system. Project maps and workflow folders make the work findable and repeatable. Audits expose structural defects before they become permanent instructions. The Mistake Ledger preserves verified failures after the immediate problem has been fixed.

The ledger does not turn Hermes into an agent that trains itself. It gives the next run a short piece of reviewed evidence from the last one. That is enough for version one.

If the broader stack is what you are after, I wrote about the full cost breakdown and the morning workflow that runs on it. Both depend on the same file-based workflow structure to stay reliable across sessions.

Capability is cheap when the foundation is broken. Audit one folder. Fix one thing. Then decide whether you need another tool.

How Pangram Decides What Looks Like AI

cucoleadan — Tue, 28 Jul 2026 12:37:08 +0000

This was originally published on All Agents Considered.

I have no doubt you already know that last week Substack just attached an AI detector to every article on the platform.

The new “Scan for AI” feature lets readers analyze a post, Note, comment, or reply with Pangram, which estimates how much of the writing is human, AI-assisted or written by AI.

And the percentage it returns might looks precise, like a speed camera telling you exactly how fast you were going.

But Pangram did not watch the person type, check their browsing history, or find a hidden signature left by ChatGPT or Claude. It looked at the finished text and decided what the writing resembled.

That distinction sits at the center of the argument happening across the Substack, professor, and AI research subreddits.

Some people think these detectors generate little more than a random number. Others treat the score like a confession.

Both sides are missing the point.

Pangram is pretty good at identifying obvious AI writing, and it has more behind it than the early tools that made AI detection a laughing stock. In independent testing from the University of Chicago, it identified writing from widely available AI models with low error rates under the conditions studied.

But it is not perfect. Some model output and repeated paraphrasing can evade it, and human writing can still be classified incorrectly.

A low AI score does not prove that no AI was used, just as a high score does not prove an algo wrote the text. Pangram is a serious pattern detector, not a yes-or-no authorship test. It cannot magically settle the question of authorship. So today I am going a bit out of the way with the main topics of AAC to show you the truth behind Pangram and AI detectors.

In this edition I’ll cover how Pangram is trained, what its score measures, why it differs from older detectors, and where it fails, without pretending the tool can prove with 100% certainty who wrote a piece a text.

In this article:

Why older AI detectors earned so much distrust
How Pangram learns to identify AI patterns
What its percentage does and does not mean
The strongest evidence that it works
The known ways it can fail
How to use a result without accusing the wrong person

Why Everyone Hates AI Detectors

Early AI detectors often leaned heavily on two ideas called perplexity (not the tool) and burstiness.

Perplexity asks how surprised a language model is by the next word. AIs are trained to predict the next word, that’s literally written in the DNA of all LLMs. On the other hand, our human writing patterns are less predictable just because our writing tends to be messier.

Burstiness asks whether sentence lengths and structures vary. Humans may write one clipped sentence, then an overgrown paragraph, followed by a shorter sentences. AI models usually settle into a steady rhythm (like the Rule of 3’s).

You don’t need me to point out that these “AI detection” techniques are fundamentally flawed. Imagine judging whether bread came from a factory by checking whether every slice has the same thickness. That clue may help in some cases, but it doesn’t tell you exactly who baked the loaf.

A safety manual is predictable because it needs to be. A legal form repeats itself because consistency matters. A non-native writer may use simpler sentence patterns.

A poet may produce language so unusual that the detector has little familiar ground beneath it.

This is why old detectors produced absurd headlines. In 2023, Ars Technica reported that one detector labeled part of the US Constitution as likely AI-generated. A 2023 academic review of detection tools also found serious reliability problems, especially after text was edited or paraphrased. Pangram argues that those two measures alone fail as a basis for detection.

Thus an online myth was born: AI detectors don’t work.

Before we go into why Pangram is truly different, we really have to get this out of the way. Pangram is not the only trained detector in 2026. GPTZero describes a multiclass model for human, mixed, polished, generated, and paraphrased text. Originality says its current detector uses a trained transformer model built from human and generated examples.

Pangram Does Not Search for an Em Dash

Pangram is a trained classifier.

Researchers train it on many documents labeled by source: human-written or AI-generated. From those examples, the detector learns which combinations of patterns tend to distinguish one group from the other.

Think of it like teaching someone to recognize counterfeit passports rather than giving them a checklist of what a document should contain. Side note: this reminded me of the game Papers, Please.

Pangram says its flagship detector uses a transformer model adapted to classify sequences of text. Its technical explanation and original research paper describe a system that learns from human writing and model-generated writing. Basically they created an AI that can detect AIs.

It does not call OpenAI to ask whether ChatGPT wrote the paragraph. It does not recover the prompt. It does not compare the post with a giant database of every answer an AI company has produced. There’s literally no need for that.

Pangram learns what different kinds of writing tend to look like, then it analyzes the text you feed it against that map.

This should also clear up the em dash frenzy.

Pangram can show human-readable supporting clues such as em dashes, lists, headings, stock phrases, Markdown, and groups of. But Pangram says those visible clues are separate from the flagship detector. They help a reader inspect the text and show that they are not the inputs driving the main model.

To better put this into perspective, imagine a prisoner just escaped from incarceration. Deleting every em dash is like removing their striped shirt before a facial-recognition scan. The prisoner changed something visible without necessarily changing what the system recognizes.

Pangram Trains on Twins

The most interesting difference is how Pangram constructs its training set.

Suppose we trained a detector using personal diaries as the human examples and software landing pages as the AI examples. It might appear accurate while learning the wrong lesson. Instead of detecting AI, it could learn that feelings are human and product features are synthetic.

Pangram tries to prevent that shortcut with what it calls synthetic mirroring.

For each human document, it creates an AI counterpart about the same subject, in a similar tone and style, carrying similar information. A human book review is compared with an AI book review of the same material. A human essay is paired with its AI twin.

The detector must find differences between twins rather than compare a shopping list with a sales page.

Pangram’s model card says its human training material covers essays, reviews, books, creative writing, news, scientific papers, Wikipedia, and general web text. Its AI side is generated in-house to create those matched comparisons. Then it makes the exam even harder.

Pangram scans large collections of known human writing and looks for passages the detector wrongly calls AI. Researchers add those mistakes back into the training data, create AI mirrors for them, and train again.

This technique is called hard-negative mining.

There’s a famous story from World War II based on the statistician Abraham Wald. The military engineers were asked to examine aircraft that returned from battle and map where they had been hit. In most cases, the wings and fuselage were covered in bullet holes, so the obvious response was to reinforce those areas.

Wald argued that the military was looking only at surviving aircraft, but if those planes could return despite being hit in the wings and fuselage, then those areas were not the most vulnerable. That’s how he figured out the places with few bullet holes (like the engines) needed more protection.

Hard-negative mining follows a similar principle. The most useful evidence often comes from the cases a system mishandles. So instead of repeatedly training a detector on obvious AI writing, researchers study the difficult human passages Pangram (initially) falsely identified as AI and feed those mistakes back into the training process.

Pangram’s Open Model Is Not the Substack Model

Pangram has released public code, data links, and model weights through its EditLens repository. The project accompanies a paper accepted at ICLR 2026 on measuring degrees of AI editing.

That is meaningful transparency. It lets researchers inspect and reproduce part of the method without trusting a marketing page.

What it doesn’t reveal is the complete system running inside Pangram’s commercial product.

The public repository contains two smaller research models and instructions for training them. Pangram’s Open Pangram announcement explicitly says the release is a research baseline and should not be used to enforce AI policies in schools or workplaces.

The current commercial model described by Pangram is version 3.3. Its model card documents a larger production system with later training and scanning changes. Substack publicly confirms the Pangram integration, but its help page does not name the exact model version or threshold it uses.

What the Percentage Means

We can all agree that teal writing has made binary detection obsolete.

A writer may create the first draft, ask Claude to tighten one section, run the result through Grammarly, dictate a new ending, and then rewrite every sentence by hand. Which words belong to whom?

Pangram’s EditLens research tries to measure that grey area. Researchers begin with human text, ask models to edit it at different intensities, and compare each edited version with the original. A classifier then learns to estimate the degree of editing from the final text.

The full EditLens paper states the important limitation of its method. When the detector checks your article, it doesn’t receive your original draft. It only sees the final text.

The estimate is like a restorer examining a painting and guessing how much has been retouched. They may recognize signs of later work, but they can’t count the extra brushstrokes.

For long documents, Pangram breaks the text into overlapping windows, checks each window, then makes a finer pass near uncertain boundaries. Its 3.2 model card describes an approximate resolution of 50 words.

Pangram can then describe shares of a document as Human, Lightly AI-assisted, Moderately AI-assisted, or AI. A result such as 30% AI refers to the share of classified text segments assigned to that category.

It does not mean:

There is a 30% chance the writer used AI
A machine typed exactly 30% of the words
Pangram found the original prompt
The writer contributed only 70% of the thinking

Ideas, wording, structure, research, and final judgment are different contributions. No detector can reduce all of them to one authorship percentage by reading the final copy.

This is where Substack’s interface creates a social problem. A percentage looks like a measurement, but underneath it’s just a classification.

The Evidence That Pangram Works

The strongest independent evidence I found comes from the 2025 University of Chicago working paper Artificial Writing and Automated Detection.

The researchers gathered 1,992 verified human passages across blogs, novels, résumés, essays, business writing, and other genres. They produced matched AI versions using GPT-4.1, Claude Opus 4, Claude Sonnet 4, and Gemini 2.0 Flash.

On medium and long passages in that test, Pangram produced essentially zero false positives and false negatives. It outperformed GPTZero, OriginalityAI, and an open RoBERTa detector in the same experiment.

That should clear up the myth that “AI detectors never work”.

But this should also be taken with a grain of salt. The paper was, after all, a working paper. The test used a controlled set of matched human and synthetic documents. It evaluated the Pangram service available in May 2025 and it did not test every Substack voice, mixed editing workflow, new model, humanizer, or unusual format.

Pangram’s current numbers come from Pangram. Its 3.3 model card reports a 0.01% false-positive rate for long-form creative writing, 0.02% for academic writing, 0.04% for multilingual how-to articles, and 0.49% for poetry.

It also reports a 1.5% false-negative rate on a random Chatbot Arena set.

Those are impressive company-reported results, but they shouldn’t be treated as promises about the next text you scan.

The clearest warning comes from a competing test. In February 2026, GPTZero published its own benchmark and reported that GPTZero 4.3b beat Pangram 3.2 on its datasets.

That does not prove GPTZero is better. It proves the winner changes with the test set, model version, threshold, and person running the test.

I made the same argument in my breakdown of why AI benchmarks fail real agent workflows. A leaderboard compresses the conditions into one number, but conditions change and can be picked arbitrarily.

Where Pangram Messes Up

Pangram’s own documentation says it works best on long-form prose written in complete sentences, like a Substack post.

Its 3.3 limitations warn about bullet lists, technical instructions, tables of contents, references, templates, equations, headers, and footers. Poetry has a much higher reported false-positive rate than long creative prose.

Short text creates another problem. Just as a smoke detector cannot identify much from a few molecules in the air, Pangram can not correctly classify a short text.

Then there are false negatives.

A May 2026 preprint titled Base Models Look Human To AI Detectors found that output from base language models often looked human to both Pangram and GPTZero. Instruction-tuned chat models were easier to identify than base models. Researchers also used repeated paraphrasing to make AI text appear more human to the detectors.

That suggests detectors may be especially good at recognizing the habits produced by chat-model training, rather than detecting one universal substance called “AI writing.”

The Atlantic found another crack. A reporter sent ChatGPT and Claude output through the Walter Writes humanizer, then watched Pangram label the result human. Pangram has since described better humanizer detection in version 3.3, so one test does not freeze the system forever, but it does show the shape of the fight.

The detector learns the disguise, and the disguise keeps changing. It’s just like with police figuring out a new way to transport illicit substances, then traffickers come up with something new and the wheel keeps spinning.

This arms race creates an important asymmetry:

A result of “AI” can be a false accusation
A result of “Human” can be successfully disguised AI

Neither side provides proof.

Pangram CEO Max Spero told The Atlantic the detector should never be the final arbiter. The company setting limits on its own tool carries more weight than a critic attacking it from outside.

How to Read a Pangram Result

A Pangram result is closer to weather radar than a fingerprint.

Radar detects conditions associated with a storm. It can tell you when the signal is strong, where to look, and whether you should carry an umbrella. A fingerprint connects a person to an object through physical evidence, and Pangram has no such access to the writing process.

I also don’t use Pangram as a moral scorecard. I write about technical subjects, and I use AI heavily to structure my thoughts and explain difficult concepts to less technical readers. My first question when I read something is much simpler: Did I get anything valuable from it?

If an article teaches me something new, gives me a useful idea, or makes a complicated subject easier to understand, I’m satisfied with the time I spent reading it; whether AI helped produce it comes second.

I take the same approach to Duolingo, where I’m currently learning German. Duolingo has openly described using generative AI to create and validate course content. I also know that more than 1,000 volunteers helped create dozens of its original language courses.

I have serious ethical problems with that history. I personally believe that Duolingo owes an enormous debt to the community that helped build it, and embracing automation does not erase that debt. But I cannot reverse the company’s decisions, and I still want to learn German, so I keep on using the app and embrace the technology for what it can do.

That is roughly how I approach AI-assisted writing. I care about the quality of the thinking, the usefulness of the result, and whether the article rewards my attention. A detector score may tell me something about how the prose was produced, but it cannot decide whether the prose was worth reading.

With that in mind, I use four rules when reading a Pangram result:

Check the material. Is it long-form prose, or a short Note, poem, list, technical manual, reference section, or template? Pangram’s own model card says the detector is intended for long-form writing in complete sentences and warns that lists, instructions, reference sections, templates, and dense equations are more susceptible to false positives.
Read the label as resemblance. Pangram is a trained classifier. It studies patterns in human and AI writing and estimates which group a new passage most closely resembles. Pangram describes the process as turning the text into a numerical representation and passing it through a classifier. It does not observe who typed the words or watch the document being created.
Do not mistake the score for a reconstruction of the writing process. A high result does not tell me which ideas came from a model, how much work the writer contributed, or whether AI only helped with editing. A low result does not prove that no AI was involved. The detector sees the finished prose, not the conversation, research, prompting, or revision behind it. This is also why my best agent workflow is mostly files. Files preserve the trail between research, drafting, and editing, even when AI is part of the process.
Ask what question I actually want answered. If I want to know whether the prose resembles AI-generated writing, Pangram provides a useful signal. If I want to know whether the article is accurate, original, insightful, or worth reading, I have to judge those things myself. A strong idea does not become worthless because AI helped express it, just as a completely human-written article does not become valuable merely because a person typed every word.

I apply the same discipline when AI gives me a technical answer because accepting AI’s first answer without checking it replaces judgment with convenience. I do not reject an answer because a model produced it, and I do not trust it merely because it sounds polished. I ask whether it is accurate, useful, and worth my attention.

A Pangram result can describe patterns in the writing, but it cannot tell me whether I learned something. That would be dystopian.

My Verdict On Pangram

Pangram is different from the crude detectors people remember.

Its classifier learns from paired human and AI documents.

Its synthetic mirrors reduce topic shortcuts.
Its hard-negative mining feeds human false positives back into training.
Its calibration favors avoiding false accusations.
EditLens attempts to model mixed editing rather than forcing every document into a human-or-machine box.

Independent researchers found that this approach performed extremely well on their tested medium and long passages.

Those are facts.

Pangram still sees only the finished text. It cannot observe the actual writing process.

Its published research does not reveal every part of the production system, and its accuracy varies by domain. Short and formulaic formats remain particularly difficult.

New model types and paraphrasing can also evade detection. Not to mention independent benchmarks do not always agree with Pangram’s own results.

And even the percentage can be misleading. Readers may mistake it for a probability or a record of the writing process, but it is neither.

Those limitations are facts too.

The reasonable conclusion lies somewhere between “all detectors are useless” and “the algorithm knows the absolute truth”.

Pangram has earned the right to be treated as evidence, but it hasn’t earned the right to be treated as a verdict.