DEV Community: 鄭宏宇

I measured whether my coding agent follows its rules

鄭宏宇 — Thu, 17 Sep 2026 01:53:23 +0000

My agent made the same mistake four times in one conversation.

It kept writing files through a shell heredoc, and the heredoc kept eating a backslash, so the file looked right and ran wrong. Three notes in its memory store described exactly this mistake, and all three had been retrieved and read. I went back through the transcript to make sure. Then it did it again.

Work in this space gets filed under "agent memory", and most of the tooling is about retrieval: getting the right text in front of the model. That part mostly works. What I've spent the last few weeks measuring is whether the text changes what the model does, and so far the answer is much less than I expected. The fixes I tried also failed in ways that were easy to miss.

Delivering the rule didn't make it stick

My first theory was timing. Rules load at session start and decisions happen an hour later, so the fix should be to put the rule in front of the model at the moment it matters.

I had already built that. A small core stayed loaded, and the detail lived in files behind pointers: "before writing code, read this file." Then I measured it. Across 89 sessions, 81 reached a point where a pointer told the model to go read the file. It opened the file in 4 of them, or 4.9%.

So I moved everything back to always-loaded, with every rule in context all the time. That same day, with all 59 of them sitting in front of the model, it broke them six times.

Loading rules up front and fetching them on demand turned out to be two versions of the same thing. Neither one could stop an action once the model ignored the rule. Stopping it takes hooks that the agent host runs at two points, right before a tool call executes and when the model finishes a turn, where either one can refuse and make the model try again.

The same pointer file behaved very differently in another agent host: it was opened in 109 of 144 sessions, 96 of them within the first three actions. That host reads the whole rules file at the start of every session anyway, so it never has to follow a trigger. How well a rule lands depends partly on host habits you don't control.

Turning rules you can't check into rules you can

A refusal needs something to match on. "Don't write files through a heredoc" has one, because the command contains << and a backslash. Plenty of rules have nothing like that. "Verify before you say it's done" happens inside the model, where nothing outside can see it.

What worked was changing what the rule asks for. Instead of "verify first", the rule became: a message that claims something is done has to say what was checked, such as a test count, a command, or a file that was read. That part is visible, so a check at the end of the turn can refuse the message.

This also changes what skipping costs. Skipping verification used to leave no trace. Now the only way past the check without doing the work is to write down evidence that doesn't exist, which is a much harder thing to do by accident.

Checks fail silently, and in specific ways

I added checks expecting that to be the end of it. Instead, every one of them turned out to have some way of not applying, without saying so, while the setup still looked armed. Five examples follow.

The check only looked at the end of the turn

A turn is usually: say something, call a tool, say something else. The person watching sees all of it, but the check read only the final message, so a false claim made before a tool call was out of reach of every rule. I found this when the agent told me something had never been tested. It had been tested the day before; that work had been summarised out of its context, and the agent took "I can't see it" to mean "it didn't happen." A check written for exactly that sentence still couldn't have reached it.

The same tool has a different name in each host

One rule guarded the Bash tool. Another host calls its shell tool Shell. Names were compared exactly, so the rule never fired there, and nothing reported it.

A cap meant for search also capped enforcement

To keep things fast, only a limited number of rule files were read per turn. The limit was supposed to bound discovery, and it bounded checking as well. In one large test store, 500 armed rules came out as 12 live per turn.

Saving a note changed where its fields lived

When a note was saved, the memory store reformatted its metadata and nested the fields one level down. The check read top-level fields only. Those rules looked armed and passed lint, and they blocked nothing.

Automatic tuning switched off a good rule

I had logic that relaxed rules which fired too often. One rule fired 6 times and blocked all 6, and was switched off for two weeks, because a hit rate can't separate "too broad" from "the agent keeps doing this." A rule in the second situation is doing its job. I removed the tuning.

None of these raised an error, so from the outside each of them looked like a quiet day.

Replaying history against the logs

What catches this kind of failure is replaying history through the same matching code and comparing the result with what the checks logged at the time. Every night, replay the day's transcripts and reconcile. A hit the replay finds that the log doesn't have is a miss. The replay has to reuse the live check's code, since a replay with its own interpretation can disagree for reasons that tell you nothing.

Some cases can't be told apart this way. A short quoted sentence that breaks a rule looks the same as a legitimate quotation, and I let quotations through, so for that case I count how often it happens instead of trying to decide.

My tests passed anyway

While I was building this, my suite passed 54 of 54, and four independent reviews each said not to ship. One of the things they reproduced was the documented install command silently skipping a whole step and reporting success anyway.

Later, an acceptance script I'd written to confirm that "every leftover file is named" turned out to be skipping every file, because the temporary directory happened to sit under a path the filter excluded. It exited 0 having confirmed nothing, and a reviewer caught it, not me. A test suite written by the same person who wrote the code mostly checks what that person already expected.

Memory costs something

Retrieved context stays in the conversation and gets re-read on every later call until the conversation is compacted. When I measured a week of real sessions, retrieval was by far the largest token cost in the setup, and the checks were cheap by comparison. Two findings:

A deduplication key included a per-prompt label, so a note already delivered got sent again whenever that label changed: 1,404 identical lines re-sent within a single stretch of conversation, a median of three prompts apart.
Using a loose proxy (was the note's name mentioned later?), about 4% of delivered project notes and 7% of delivered reference notes got used.

If you only count what gets delivered, adding more memory will always look free.

What I'd tell someone building this

When an agent breaks a rule, ask what should have stopped the action and whether it ran, before asking whether the rule was in context. Where you can't observe the act a rule cares about, require the sentence that reports it. Test every check end to end in every host you support, using an input it should refuse; until you've watched it fire, assume it's off. Replay history against your logs, because silent misses only show up in that comparison. Anything that gets summarised can lose facts about work already done, and the agent won't know. Measure what memory costs, and drop what gets delivered and never used.

Disclosure: I build Epitype, an open-source memory governance layer for Claude Code and Codex, and every number above comes from building it. I've kept it out of the body on purpose, because these failure modes don't depend on which tool you use.

Your coding agent remembered the rule, and broke it anyway

鄭宏宇 — Wed, 02 Sep 2026 11:14:09 +0000

This week, two different coding agents did three things on my machine. One asked whether it could operate the browser, although I had granted that permission days earlier. One answered a question about my desktop cleanup with a status line I wrote on July 22. It had been out of date for six weeks. One ran a shell heredoc with double backslashes three times in one day while the rule against it sat in the instruction file.

If you use Claude Code or Codex daily, some of this will look familiar. People usually diagnose the problem as forgetting, then add more memory: extract more, embed it, retrieve the top matches. Sometimes that is the right fix. It would not have fixed these three. The first needed my "yes" captured from my own words. The second retrieved the memory and still got the answer wrong. The third had the rule in writing and ignored it. Remembering the right fact is not the same as acting on it. Epitype is a governance layer for that gap, built for Claude Code and Codex. It also clears a pile of small annoyances you would otherwise wire yourself: the right context injected at session start, relevant cards recalled on every prompt, a recovery map written before context compaction, the gate before each tool call, host files backed up first. One install command, and none of those needs separate attention.

It does not build a second warehouse

Epitype leaves your host's native memory directory as the storage authority. It does not copy cards into its own database, sends nothing to a hosted service, and depends on nothing outside the Python standard library. During installation it backs up any host file it touches and merges only entries it marks as its own. Uninstall removes what it owns and leaves every card where it was. I start here because it is the first thing I would ask about a memory plugin: will it eat the memory I already have?

Three jobs need separate rules

Agent memory has at least three different governance jobs. A single warehouse treats them all the same.

Habits are rules the agent should follow every time without stopping to decide. Their governance problem is size. Epitype keeps them in primary memory, and the session-start index that carries them is bounded, so they stay visible without turning into a wall of text.

Scars are lessons from incidents. When your hand moves toward a hot stove, the flinch comes before the decision. A scar card in Epitype can carry a trigger made from a tool pattern plus an input pattern. When a tool call matches, the hook blocks it before it runs, returns a safer route, and writes an audit row. The matcher examines the executable and the unquoted argument positions. It follows shell and interpreter wrappers while ignoring quoted strings, comments, and heredoc bodies, so a comment containing a forbidden word does not block a legitimate command. After I turned the heredoc rule into a scar with a trigger, the gate blocked my next attempt. A scar retires when the hazard gets a mechanical guard somewhere better than prose.

Pending work has another lifecycle. It needs an owner and a next check, and it should disappear when done. The July 22 line that misled my agent was a pending item with no exit. Left in place, it quietly hardened into "the current state". Epitype keeps pending work in its own block, and a lint reports the entries that have gone stale.

Two rules apply across all three. Every decision has exactly one current version. Its decision card carries a stable key, a status of active or superseded, an effective time, and who decided. By default, every read path excludes superseded cards while retaining them for provenance, so last week's decision cannot return as today's. And a permission you grant is stored in your words and tied to you, instead of depending on whether the agent remembered to take a note.

Where it runs

Epitype uses four host events. At session start, it provides a bounded index of the vault. Each prompt gets a few relevant cards. Before each tool call, the gate runs. Before context compaction, it writes a recovery map so the next context knows where things stand. Every injection is capped at 10 KiB, fails open within three seconds, and is labelled as advisory data that cannot override instructions or grant tool authority. That label matters because injected memory is itself a prompt injection surface.

In practice, a run looks like this:

old decision   superseded  -> excluded from recall
new decision   active      -> surfaces on a matching prompt
tool call      scar match  -> denied, safer route returned, audit row written

What I measured, and what it does not prove

The repository ships 17 component selftests covering the tools, both host adapters, the installer, the exam engine, and the privacy gate. Before tagging 1.0.0, the release gate also passed a strict 300-case behavior corpus and a 15-seed review written from the incidents above. Those two are not in the repository, and all of this evidence is synthetic. It proves that the mechanisms behave as specified in a small vault. It does not prove that they improved anyone's agents over months.

One number came from my own machine, produced by a tool you can run yourself. The pending lint found 54 stale pending lines across 32 cards, the oldest 80 days old. That is what a memory looks like when nothing governs exits.

Hooks can only govern the events and tools the host exposes. The budgets mean the agent never sees the whole vault, so selection can be wrong. Gate precision depends on the triggers people write, and a malformed card fails open instead of taking control. Two hosts are tested. The bundled tests are synthetic.

Look before it changes anything

Read the repository, then run:

pip install epitype
epitype-graft install --dry-run

The dry run prints what would be written, which hosts it found, and where the vaults are. If that matches what you expect, the README walks through install, doctor, and the Codex trust check. If it does not, nothing has changed.

MIT. https://github.com/leavemagic-cyber/epitype

My review loop ran 21 rounds. The last five found nothing.

鄭宏宇 — Wed, 19 Aug 2026 09:47:07 +0000

I manually checked 100+ startup launch directories — here are 16 free ones to start with

鄭宏宇 — Sat, 20 Jun 2026 22:56:47 +0000

If you've ever tried to "submit your startup everywhere," you know the pain: you Google "startup directories", open a listicle from 2021, and half the links are dead, paywalled, or just gone. You waste an afternoon and still don't know which ones are actually worth it.

So I did the boring part: I went through hundreds of startup / SaaS / AI launch directories, opened each one, threw out the dead links, and noted whether submission is free or paid and what's required.

Here are 16 verified ones to get you started — all checked in June 2026.

16 verified launch directories (free to start)

Directory	Submit URL	Free / Paid	Best for
Product Hunt	https://www.producthunt.com/launch	Free (optional paid promo)	General / SaaS / AI
BetaList	https://betalist.com/submit	Free (or paid skip-queue)	Startup / Pre-launch
Indie Hackers	https://www.indiehackers.com/products	Free	Indie / Startup
SaaSHub	https://www.saashub.com/submit	Free	SaaS / General
Uneed	https://uneed.best/submit-a-tool	Free (paid options)	Indie / SaaS / AI
Peerlist Launchpad	https://peerlist.io/launchpad	Free	Dev / Indie / SaaS
Dev Hunt	https://devhunt.org/	Free	Dev tools
MicroLaunch	https://microlaunch.net/	Free (paid Pro)	Micro-SaaS / Indie
Fazier	https://fazier.com/submit	Free (optional promo)	Indie / SaaS
PeerPush	https://peerpush.com/	Free (paid fast-track)	SaaS / AI
Launching Next	https://www.launchingnext.com/submit/	Free	Startup / General
OpenAlternative	https://openalternative.co/submit	Free	Open-source
SideProjectors	https://www.sideprojectors.com/	Free (paid premium)	Side projects
Tiny Startups	https://tinystartups.com/	Free (paid options)	Indie / Startup
Futurepedia	https://www.futurepedia.io/	Free (paid fast-track)	AI tools
Startup Stash	https://startupstash.com/add-listing/	Free	SaaS

How I check each one (so you don't get burned)

A directory list is only useful if it's true today. For every entry I keep:

A live-link check — if it 404s or the submit page is gone, it's out. No graveyard links.
A last-checked date — so you know how fresh it is, instead of trusting a 3-year-old blog post.
Free / Paid + what's required — login, approval, or payment — so you don't waste time or money.
A niche tag — AI tool vs SaaS vs indie product, because not every directory fits every launch.

That's it. No "domain rating" guesses, no "do-follow backlink" promises, no ranking guarantees — just a clean, current list you can actually work through in an afternoon.

Want the full list?

The 16 above are a free starting point. I keep a maintained spreadsheet of 100+ verified, dated directories (CSV + Excel) with the same columns — free/paid, submission requirement, niche, last-checked date — so you can filter and plan your whole submission run at once.

If that saves you a weekend of Googling, it's here: https://launchlistlab.gumroad.com/l/zblii

Either way — go ship. 🚀

Disclaimer: this is a curated data file from publicly available sources, for informational use. It doesn't guarantee rankings, traffic, approvals, or links — those depend on each platform and your product. Directory info can change after the last-checked date.