DEV Community: Harjot Rana

I mined 1,398 corrections out of my coding agent logs and turned them into a rules file

Harjot Rana — Thu, 10 Sep 2026 13:08:17 +0000

My coding agent asks me the same things constantly. New file or add to the existing one. Do you want a test for this. Should I clean up the function next to the one I am touching.

I have answered all of these. Dozens of times. The answers were sitting in my chat history and nobody, me included, was reading them back.

So I wrote a tool that does.

What it actually is

jot is an extractor plus a knowledge base. The extractor finds the session stores your coding agents leave on disk, reads them, and pulls out the moments where you stopped the agent and said no, do it this way instead. Those get clustered into rules, each with a reason and a stated exception, and written to markdown.

You load that markdown as an agent skill. Your agent reads it before it starts working.

npx skills add HarjjotSinghh/jot -g

Then in any agent that supports Agent Skills:

/jot should this state be global or local?
/jot review this PR
/jot which model should these subagents run on?

That command installs mine, which is really a worked example. The part worth your time is pointing the extractor at your own logs.

No training. No GPU. No fine-tune. Text files.

The version I sat on for a year

For about a year I wanted to fine-tune a small model, 8B to 20B, on every file I own. Text, audio, PDFs, spreadsheets, decks. A model of me, trained on me.

I never wrote a line of it. Not once did I open a terminal.

Then I read Kun Chen's post about distilling himself into a skill, and the thing that clicked was that the hard part of "clone yourself" was never the weights. It was the preferences. And preferences are small enough to just write down.

By preferences I mean the boring calls with no correct answer, only my answer. Refactor now or ship and come back. Test first or backfill before merge. Ask before touching anything with real users on it, or just go. A generic model has a sensible default for every one of these. Several of mine are not the default, and that gap is the entire reason the agent keeps interrupting me.

Do not interview yourself

The obvious way to build this is to sit with a model and answer questions about how you work. It produces beautiful, useless output, because everyone is aspirational about themselves.

Ask me if I write tests first and I will say yes with a straight face. Look at what I actually shipped and you find spikes with no tests and coverage backfilled right before merge. The second version is the true one.

So I skipped the interview and went to the logs.

The highest-value thing on my disk is not my code. It is every time I typed something at an agent that was one keystroke away from doing the wrong thing. Every agent saves those. Nobody reads them back.

The extraction

Six adapters: Claude Code JSONL, Codex rollouts, Grok history, Gemini CLI logs, OpenCode, and Cursor, which stores every message as a row in a 1.5 GB SQLite file.

python scripts/extract_corpus.py
python scripts/extract_corpus.py --agents claude codex cursor --min-score 5

The pipeline:

Normalize everything to turns. Six agents, six schemas, one shape.
Pair each thing I said with the agent turn it was reacting to. "No, not like that" is meaningless on its own. It only carries information next to the thing it rejected.
Score against a lexicon weighted toward corrections, rejections, and stated preferences.
Dedupe hard, because I complain about the same three things constantly and 40 instances of one gripe is one rule, not 40. First pass: 1,398 distinct judgment events across 1,290 conversations. 505 corrections. 380 stated preferences.

Every rule in the knowledge files carries an evidence tag pointing back at the source:

[observed Nx] extracted from N distinct real sessions, dated
[stated] I wrote it down as a rule
[inferred] derived from adjacent behaviour, lower confidence, flagged at use time A rule with no evidence is a vibe. If it cannot be traced, it does not go in.

Layout

A thin skill stub, and a knowledge base that lives in the repo.

skills/jot/SKILL.md   loader; local checkout first, raw repo URLs second
ENTRY.md              routing table plus the decision procedure
PRINCIPLES.md         durable rules that survive a framework change
ENGINEERING.md        backend, debugging, git, review
FRONTEND.md           visual taste, design parity, slop detectors
AGENTIC.md            model routing, autonomy grants, parallelism
WORKFLOWS.md          named sequences: feature, bug, refactor, review, EOD
TOOLS.md              what I reach for, and the friction I hit
BOUNDARIES.md         hard stops, read before anything outward-facing
VOICE.md              how I write; loaded only for text going out under my name
OPERATING.md          attention, cadence, escalation
CONTEXT.md            who I am, what is active
state/                mined evidence, changelog, pending promotions
bench/                the benchmark that says whether any of this works

The stub never changes. All churn happens in the knowledge files, so every agent pointed at the repo stays in sync without a separate update step.

private/ and state/evidence/ are gitignored and never fetched over the network. The public layer holds preferences. The private layer holds the specifics that make them actionable: named colleagues, clients, money, positioning. The skill only loads those from a local checkout.

Does it work

A rules file that sounds like you but decides differently is worse than no file at all. It makes confident wrong calls in your name and you are not there to catch them. So bench/ answers the question properly.

Three arms. A control model with no skill. The same model with /jot loaded. And me, answering blind.

python scripts/bench_run.py --set v1
python scripts/bench_score.py --run <timestamp>

Ten questions, each one a decision where a competent generic model has a plausible default that is not what I actually do. Both model arms run in clean-room sessions that never saw the conversation where the rules were written, because a session that helped write the answers already knows the answers. Their responses go into sealed files I do not open until I have committed to mine.

Scored 0 to 2 against my answer. Minus one for fabricating a preference I do not hold.

First run: skill 13/20, control 7/20.

The number that matters is the delta between the two model arms. A skill that scores well only because any competent model would have scored well is recording things that did not need recording.

Then I sorted the results by where each rule came from

I had written two kinds of rules without really noticing. Ones mined out of real transcripts, and ones I had simply believed about myself and typed in.

mined from transcripts:  2, 2, 2, 2
believed about myself:   0, 1, 0, 0

No exceptions in either direction. The mined rules carried the entire gain. Everything I had assumed about myself was dead weight or actively wrong, and it was wrong in exactly the same confident tone as the stuff that was right.

The worst one: my file said that once I give a blanket approval, the agent should stop asking. I had generalized that from exactly one line I typed mid-task, once. The benchmark asked what to do at a sub-decision that approval never covered. The skill told the agent not to ask. My real answer was "I would ask first."

One occurrence is an anecdote. My own precedence rules say three occurrences make a rule. I broke my own precedence rules inside the file that contains them.

That is how this goes wrong. Not by being wrong about you. By being right about you in one situation and then applying it everywhere.

Patched four rules, re-ran, 18/20

And one question that had scored 2 dropped to 0.

The rule I added to fix "just take the two-hour refactor now" got applied to a two-minute fix on a live customer list. Which is the opposite of what I do, because that is not my code and those are somebody's real customers.

A rule that fixes one case and breaks another is written too broadly. Misses now become golden cases in bench/golden/, and each one turns into a rule with an explicit Doesn't apply when line. Boundaries outrank every principle in the repo instead of politely tie-breaking with them.

What it is like to use

Uncanny. Not "wow, impressive model" uncanny. Specifically: you ask it something you have never thought about before, it answers, and you get a small jolt of yes, that is exactly what I would have done, and that is exactly why.

It is talking to yourself, except this version of you has read every correction you have ever issued and never gets tired.

It disagrees sometimes, and the disagreements turned out to be more useful than the agreements. It was never being stupid. It was faithfully applying a rule I had written badly. The gap between the two model arms measures my preferences. The gap between the skill and me measures my writing. Two different problems, fixable separately.

Worth doing even if nobody else installs it. Writing a rule with a reason, a stated exception and a piece of evidence forces you to find out whether you actually hold it. A few of mine did not survive the format.

Not bad for an idea I sat on for a year because I thought it needed a GPU.

Try it

npx skills add HarjjotSinghh/jot -g

Repo: https://github.com/HarjjotSinghh/jot

Prior art: https://github.com/kunchenguid/kun, which made the case that the repeatable part of your judgment is worth externalising, and that it was never the part that was your moat. Read Kun's first.

If you run the extractor on your own logs, I want to know two things. How many events it found, and how many of the rules you wrote by hand survived contact with a benchmark. My guess is fewer than you expect.

How we built a 14-agent pipeline that ships a deployed app + launch assets in ~7 minutes

Harjot Rana — Mon, 01 Jun 2026 01:54:28 +0000

Most AI app builders stop at "deployed." You prompt, you get a repo, maybe a preview URL, and then the actual work starts: wiring a domain, writing the landing copy, cutting screenshots, drafting the launch thread. We wanted the pipeline to stop at "launched" instead, so we built one. This is how it works under the hood, including the parts that broke.

The product is Moonshift. One prompt triggers 14 specialized agents across 10 phases. Average run is ~7 minutes and ~$3 in API spend, with a hard $5 ceiling that aborts the run. Everything ships to your Vercel, your GitHub, your database. This post is the engineering, not the pitch.

The core problem: parallel agents drift

The naive version of "many agents build an app" falls apart fast. If a backend agent and a frontend agent both work from a vague English spec, they invent incompatible contracts. The backend returns { user_id }, the frontend reads userId, and you find out at runtime in production.

Our fix is a planner that emits a JSON contract first. Before any code is written, one agent produces a typed contract: routes, request/response shapes, table schemas, env vars, page list. That contract is the single source of truth. Backend, frontend, database, and test agents all build against it in parallel instead of against prose.

Then a contract-validator agent runs after the parallel build and diffs the actual code against the contract. When the frontend's fetch shape doesn't match the backend's handler, the validator doesn't just flag it. It patches the mismatch. This one agent removed the largest single class of "looks done, 500s on click" failures we had.

The 10 phases

Plan - generate the JSON contract.
Scaffold - lay down the framework skeleton (Next.js, config, deny-globs that protect files agents shouldn't touch).
Backend - API routes and server logic against the contract.
Frontend - pages and components against the same contract.
Database - schema + migrations (Drizzle + Turso).
Validate - contract-validator reconciles 3-5 in parallel, auto-fixes drift.
Test + fix - generated tests run; a fixer loop addresses failures.
Deploy - ships to your Vercel via your token.
Audit - security and a11y passes on the live deployment.
Market + publish - a marketer agent drafts X and LinkedIn launch posts in your voice, image-gen produces hero images, and a publisher gates everything behind your one-tap approval.

The interesting phases are 6, 7, and 10. Everyone has a code-gen step. Almost nobody has a reconcile step, a real fixer loop, or a phase whose only job is launch assets.

Reliability: not every failure is equal

Long multi-agent runs fail in boring ways: a rate limit, a flaky deploy, a model that returns prose where you asked for JSON. If you retry all of them the same way, you either give up too early on transient errors or burn money death-looping on deterministic ones.

We run a failure classifier that buckets every failure into transient, deterministic, or permanent:

transient (429s, network blips, stream idle) - retry with backoff.
deterministic (a test that fails the same way every time) - hand to a fixer agent, don't blindly retry the same call.
permanent (bad auth, missing token) - stop and surface it. No point spending more.

Retries are capped on three axes at once: per-phase, per-agent, and a global per-run ceiling. The global cap is what keeps a single bad run from quietly turning into a $40 bill. Combined with the hard $5 abort, the worst case is bounded and visible instead of a surprise invoice.

A subtle one we hit: a long LLM stream can go idle without erroring (the upstream connection gets severed but the socket never closes). A naive loop waits forever. We added an idle watchdog in the agent loop so a silent stall is treated as a transient failure and retried, instead of hanging the whole run.

Design constraints that shaped everything

Your infra, zero lock-in. Code lands in your GitHub, the app deploys to your Vercel, the database is yours. Cancel the subscription and you keep a working product. This forced the deployer to operate purely through user-supplied tokens, which is more work than deploying to our own infra but is the entire point.
The publisher physically cannot post without you. Social publishing is gated per post, per platform, behind an explicit human tap. Autonomy ends at the point where it would speak as you in public. Generation is automatic; publishing is not.
Hard cost ceiling. $5/run, enforced mid-run, not reconciled after. Agents check remaining budget before expensive calls.

Stack

Next.js for web, Drizzle + Turso (libSQL) for data, Playwright for browser automation in the marketing/audit phases. The orchestrator is a separate runtime from the web app, spawned per run from source so a fix ships without a full rebuild.

Honest lessons

A typed contract beats a smarter prompt. We spent weeks trying to make agents "just agree." Making them agree on a machine-checkable artifact was the actual fix.
A reconcile phase is worth more than a better code-gen model. Catching drift after the fact, cheaply, beat every attempt to prevent it perfectly up front.
Classify failures before you retry them. Uniform retry is how multi-agent systems burn money and still fail.
Bound the blast radius in money, not just time. A per-run dollar cap is the single most important guardrail in an autonomous pipeline that calls paid APIs in a loop.

If you want to see the output, the first run is free with your own API key at moonshift.io. Happy to answer architecture questions in the comments.