research-ledger: a skill that helps one person and several AI models not fool themselves

#ai #research #llm #opensource

I am not a professional researcher. I work in IT support, and in the evenings I explore ideas with several AI models: Claude, GPT, Gemini, Grok. Over a few months, that work produced a lot of ideas and very few results. It also showed me a pattern: the main risk was never a lack of ideas, it was fooling myself.

Out of that experience came research-ledger, an MIT-licensed skill (a SKILL.md folder) that turns any assistant that reads skills into a disciplined research partner. This post explains what it does, where it came from, and what it has caught so far.

The problem: four ways to fool yourself with AI

Models agree with each other. Three models saying "yes, this is a real connection" feels like evidence. It isn't. They share much of their training data, so agreement is a reason to check the algebra, not a result.
Metaphors dress up as mechanisms. "X in physics is like Y in learning" sounds deep. Usually nobody can write the step that actually transfers.
Thresholds move after the result. A 0.62 becomes "promising" because you wanted it to be.
"I didn't find it in the literature" becomes "it's new." That sentence is a statement about your search, not about the world.

What the skill does

idea ──► sieve (5 min) ──► translation ──► prior art ──► preregistration
                                             │
                     LANDED ◄────────────────┤
                                             ▼
                       pilot ──► confirmatory test ──► blind round (milestones)
                                             │
                                             ▼
            DEAD · LANDED · BRIDGE CANDIDATE · BRIDGE · EXPLANATION ──► ledger

Sieve first. Most ideas die in five minutes against a short checklist. They get no paperwork.
Translation. Every metaphor must become a measurable test or a precise statement with a proof goal. For each analogy, the skill asks one question: mechanism or theme? If you can't write the equation or step that transfers, it's a theme. It goes into the ledger, not into an experiment.
Prior art with scope. "Not found" is recorded together with the date, the search terms and the nearest source, never as novelty.
Preregistration. Predictions, thresholds, controls and statistical power are frozen before the run. In a Git ledger, a separate PREREG commit pushed before the run is the proof.
Pilot ≠ confirmatory test. A pilot may tune settings, but it never gives a PASS.
Blind rounds. At milestones, the same seed goes to two or three models from different labs, in clean conversations, with no project context.
Five verdicts.
- DEAD: wrong, a tautology, or killed by a control.
- LANDED: you rediscovered a known bridge. That's calibration, not defeat.
- BRIDGE CANDIDATE: known pieces, connection not found, and a new prediction.
- BRIDGE: only after a confirmatory test, a blind round and an independent check.
- EXPLANATION: a known mechanism explains the pattern.
Append-only ledger. Every "no" is kept, together with what it shows and the regime where its lesson holds.
Glossary. Every new term gets its standard name and a reference, or an explicit "no standard term known". A new name is not a new idea.

On first use the skill asks six to eight setup questions and writes a small config: your field, where the ledger lives, which models you have for blind rounds, and how strict to be. It also asks which of your own favourite hypotheses must never be used as premises.

Where the idea came from

It grew out of my own log of tests, roughly forty numbered experiments so far, in toy quantum models, combinatorics, small neural networks and numerology of physical constants. Every rule in the skill is a scar from one of them. A few examples:

"Zero error" can mean nothing. One model reproduced a set of measured ratios with error 0.0000. Then we ran the forgery control: random "fake" target ratios. About 3,700 formula "keys" fit the real numbers, and just as many fit the fakes. The fit was empty. That became the lock-leakage check: count how many keys fit the real lock and how many fit forged ones, before celebrating.
The translator inserts what it already knows. I fed fringe and amateur texts through models to look for ideas the mainstream had missed. Most "insights" turned out to be the model translating the text into a known concept. Of the leftovers that survived translation, 6 of 8 were known science that the amateur had rediscovered. Hence the rule: check every leftover against the literature before combining it with anything.
A control that can't differ from the treatment proves nothing. In a continual-learning test, the "noise control" turned out to be algebraically identical to the treatment, so the test had no power at all. The rule now: write down, before freezing, why control and treatment cannot coincide by construction.
Equal strengths hide everything. Several effects worked only when all components had the same weight and vanished with heavy-tailed weights. The rule now: always retest with heavy tails.

The seed style for blind rounds was added this month, after I studied the published problem prompts in openai/math. Their prompts say prove **or* refute* (never "show that"). They fix the conventions explicitly, list what is not in the setting, give no method hints and use no analogies. I adopted all of that.

What it has caught (honest results)

Most ideas die, and that is the method working. The ledger is mostly DEAD and LANDED entries. A pile of LANDED verdicts is useful in its own way: it measures what the translator already knows, not progress toward something new.

An example of a clean kill. One idea combined local learning rules (no backpropagation) with "selectivity": only units that match the input get updated. We preregistered kill criteria and a strong control: a fly-olfaction-style model with fixed random features. The result:

the selective rule made units go silent and starve, so it learned worse than updating random units;
learned features beat the fixed random ones by a lot, and even a 16× wider random model stayed behind;
plain feedback alignment without any gating did just as well as the gated version.

So the idea is not refuted in general, but it brings nothing in that setting. Written down, with the regime, so nobody tests it again.

Cross-lab review catches different things, and sometimes catches the reviewer. This week I used the method on a side project: a machine-readable atlas of proof techniques extracted from the 719 manuscripts in openai/math. Forty extracted "signatures" were reviewed blind by Claude (Anthropic) and Grok (xAI), with a frozen rubric.

Grok's first attempt: 40 out of 40 "OK", every justification field empty, and in its own words "spot checks". It missed an objective error, a multiplicity of 0 where the source says 1. We marked that review invalid, not as "disagreement".
Grok's second attempt, with a required quote from the source for every "OK", in four batches of ten: a real review.
Agreement: the two reviewers agreed on 35 of 40 items. Each caught something the other missed: Claude was stricter on missing preconditions, Grok caught a claim taken from lines outside the cited range.
Result: both reviewers passed 32 of 40 signatures (80%, 95% interval 65–90%). An earlier 6 of 10 had an interval of 31–83%, which told us almost nothing.

The lesson went straight into the method: a review needs proof that the reviewer actually read the source, and planted faulty items that it must catch.

A test registered before it can be contaminated. For the atlas we froze two "needs" from an open learning-theory question before anyone searched the atlas with them in mind. The test pits the atlas against plain grep on the same corpus, with a blind evaluator. My own bet, written in the preregistration: most likely undetermined. We'll see.

Honest limits

The method reduces self-deception. It does not produce discoveries.
Blind rounds remove my context, not the models' shared training data.
The checks come from a specific history. Each one is labelled with its regime, and the skill asks you to rate it useful or not for your problem instead of applying it blindly.

Try it

git clone https://github.com/Freeky7819/research-ledger
cp -r research-ledger/research-ledger ~/.claude/skills/

It also works in Claude.ai (upload the zipped folder) and in Codex (the folder includes agents/openai.yaml). Then just talk to it: "here's an idea…", "evaluate this paper", "this test gave 0.62, what now?", "write a blind-round seed for this".

If a rule fails you, open an issue with the case. That is exactly the kind of "no" the ledger is for.

Disclosure: the skill, the experiments and this post were developed together with AI models (mainly Claude, also GPT, Gemini and Grok). The questions and ideas are mine; most of the code and much of the wording were written with them.

DEV Community