I mined 1,398 corrections out of my coding agent logs and turned them into a rules file

My coding agent asks me the same things constantly. New file or add to the existing one. Do you want a test for this. Should I clean up the function next to the one I am touching.

I have answered all of these. Dozens of times. The answers were sitting in my chat history and nobody, me included, was reading them back.

So I wrote a tool that does.

What it actually is

jot is an extractor plus a knowledge base. The extractor finds the session stores your coding agents leave on disk, reads them, and pulls out the moments where you stopped the agent and said no, do it this way instead. Those get clustered into rules, each with a reason and a stated exception, and written to markdown.

You load that markdown as an agent skill. Your agent reads it before it starts working.

npx skills add HarjjotSinghh/jot -g

Then in any agent that supports Agent Skills:

/jot should this state be global or local?
/jot review this PR
/jot which model should these subagents run on?

That command installs mine, which is really a worked example. The part worth your time is pointing the extractor at your own logs.

No training. No GPU. No fine-tune. Text files.

The version I sat on for a year

For about a year I wanted to fine-tune a small model, 8B to 20B, on every file I own. Text, audio, PDFs, spreadsheets, decks. A model of me, trained on me.

I never wrote a line of it. Not once did I open a terminal.

Then I read Kun Chen's post about distilling himself into a skill, and the thing that clicked was that the hard part of "clone yourself" was never the weights. It was the preferences. And preferences are small enough to just write down.

By preferences I mean the boring calls with no correct answer, only my answer. Refactor now or ship and come back. Test first or backfill before merge. Ask before touching anything with real users on it, or just go. A generic model has a sensible default for every one of these. Several of mine are not the default, and that gap is the entire reason the agent keeps interrupting me.

Do not interview yourself

The obvious way to build this is to sit with a model and answer questions about how you work. It produces beautiful, useless output, because everyone is aspirational about themselves.

Ask me if I write tests first and I will say yes with a straight face. Look at what I actually shipped and you find spikes with no tests and coverage backfilled right before merge. The second version is the true one.

So I skipped the interview and went to the logs.

The highest-value thing on my disk is not my code. It is every time I typed something at an agent that was one keystroke away from doing the wrong thing. Every agent saves those. Nobody reads them back.

The extraction

Six adapters: Claude Code JSONL, Codex rollouts, Grok history, Gemini CLI logs, OpenCode, and Cursor, which stores every message as a row in a 1.5 GB SQLite file.

python scripts/extract_corpus.py
python scripts/extract_corpus.py --agents claude codex cursor --min-score 5

The pipeline:

1. Normalize everything to turns. Six agents, six schemas, one shape.
2. Pair each thing I said with the agent turn it was reacting to. "No, not like that" is meaningless on its own. It only carries information next to the thing it rejected.
3. Score against a lexicon weighted toward corrections, rejections, and stated preferences.
4. Dedupe hard, because I complain about the same three things constantly and 40 instances of one gripe is one rule, not 40. First pass: 1,398 distinct judgment events across 1,290 conversations. 505 corrections. 380 stated preferences.

Every rule in the knowledge files carries an evidence tag pointing back at the source:

• [observed Nx] extracted from N distinct real sessions, dated
• [stated] I wrote it down as a rule
• [inferred] derived from adjacent behaviour, lower confidence, flagged at use time A rule with no evidence is a vibe. If it cannot be traced, it does not go in.

Layout

A thin skill stub, and a knowledge base that lives in the repo.

skills/jot/SKILL.md   loader; local checkout first, raw repo URLs second
ENTRY.md              routing table plus the decision procedure
PRINCIPLES.md         durable rules that survive a framework change
ENGINEERING.md        backend, debugging, git, review
FRONTEND.md           visual taste, design parity, slop detectors
AGENTIC.md            model routing, autonomy grants, parallelism
WORKFLOWS.md          named sequences: feature, bug, refactor, review, EOD
TOOLS.md              what I reach for, and the friction I hit
BOUNDARIES.md         hard stops, read before anything outward-facing
VOICE.md              how I write; loaded only for text going out under my name
OPERATING.md          attention, cadence, escalation
CONTEXT.md            who I am, what is active
state/                mined evidence, changelog, pending promotions
bench/                the benchmark that says whether any of this works

The stub never changes. All churn happens in the knowledge files, so every agent pointed at the repo stays in sync without a separate update step.

private/ and state/evidence/ are gitignored and never fetched over the network. The public layer holds preferences. The private layer holds the specifics that make them actionable: named colleagues, clients, money, positioning. The skill only loads those from a local checkout.

Does it work

A rules file that sounds like you but decides differently is worse than no file at all. It makes confident wrong calls in your name and you are not there to catch them. So bench/ answers the question properly.

Three arms. A control model with no skill. The same model with /jot loaded. And me, answering blind.

python scripts/bench_run.py --set v1
python scripts/bench_score.py --run <timestamp>

Ten questions, each one a decision where a competent generic model has a plausible default that is not what I actually do. Both model arms run in clean-room sessions that never saw the conversation where the rules were written, because a session that helped write the answers already knows the answers. Their responses go into sealed files I do not open until I have committed to mine.

Scored 0 to 2 against my answer. Minus one for fabricating a preference I do not hold.

First run: skill 13/20, control 7/20.

The number that matters is the delta between the two model arms. A skill that scores well only because any competent model would have scored well is recording things that did not need recording.

Then I sorted the results by where each rule came from

I had written two kinds of rules without really noticing. Ones mined out of real transcripts, and ones I had simply believed about myself and typed in.

mined from transcripts:  2, 2, 2, 2
believed about myself:   0, 1, 0, 0

No exceptions in either direction. The mined rules carried the entire gain. Everything I had assumed about myself was dead weight or actively wrong, and it was wrong in exactly the same confident tone as the stuff that was right.

The worst one: my file said that once I give a blanket approval, the agent should stop asking. I had generalized that from exactly one line I typed mid-task, once. The benchmark asked what to do at a sub-decision that approval never covered. The skill told the agent not to ask. My real answer was "I would ask first."

One occurrence is an anecdote. My own precedence rules say three occurrences make a rule. I broke my own precedence rules inside the file that contains them.

That is how this goes wrong. Not by being wrong about you. By being right about you in one situation and then applying it everywhere.

Patched four rules, re-ran, 18/20

And one question that had scored 2 dropped to 0.

The rule I added to fix "just take the two-hour refactor now" got applied to a two-minute fix on a live customer list. Which is the opposite of what I do, because that is not my code and those are somebody's real customers.

A rule that fixes one case and breaks another is written too broadly. Misses now become golden cases in bench/golden/, and each one turns into a rule with an explicit Doesn't apply when line. Boundaries outrank every principle in the repo instead of politely tie-breaking with them.

What it is like to use

Uncanny. Not "wow, impressive model" uncanny. Specifically: you ask it something you have never thought about before, it answers, and you get a small jolt of yes, that is exactly what I would have done, and that is exactly why.

It is talking to yourself, except this version of you has read every correction you have ever issued and never gets tired.

It disagrees sometimes, and the disagreements turned out to be more useful than the agreements. It was never being stupid. It was faithfully applying a rule I had written badly. The gap between the two model arms measures my preferences. The gap between the skill and me measures my writing. Two different problems, fixable separately.

Worth doing even if nobody else installs it. Writing a rule with a reason, a stated exception and a piece of evidence forces you to find out whether you actually hold it. A few of mine did not survive the format.

Not bad for an idea I sat on for a year because I thought it needed a GPU.

Try it

npx skills add HarjjotSinghh/jot -g

Repo: https://github.com/HarjjotSinghh/jot

Prior art: https://github.com/kunchenguid/kun, which made the case that the repeatable part of your judgment is worth externalising, and that it was never the part that was your moat. Read Kun's first.

If you run the extractor on your own logs, I want to know two things. How many events it found, and how many of the rules you wrote by hand survived contact with a benchmark. My guess is fewer than you expect.