DEV Community

Cover image for Your LLM Wiki Remembers Facts. Teach It How You Work, Too.
Vincent for AFFiNE

Posted on

Your LLM Wiki Remembers Facts. Teach It How You Work, Too.

Most LLM wiki setups I've seen are very good at one kind of memory and have almost none of the other.

Karpathy's LLM Wiki pattern is the one everyone copies now: raw sources go in, the model writes linked Markdown pages, and answers come back with sources. That's great for facts. What the release process looks like, which service owns sync, what we decided about the queue.

What it doesn't store is how you work. The fifth time my agent wrote release notes, it did the same five things it did the first four times: read the git log, searched the wiki for our changelog style, read that page, drafted, proposed the page for review. It rediscovered that path from scratch every time, burning context to do it.

That's procedural memory, and it was all over my feed this week:

  • Perplexity's Computer agent was reported this week to have a "Brain" that creates and updates personalized skills from past sessions and tool calls, with a separate improvement agent that only proposes changes when the evidence is strong.
  • A Google Cloud AI / DeepMind / Penn State paper on Self-Improvement Memory (SIM) was reported this week. It logs the reasoning behind each harness edit so the optimizer doesn't repeat the same mistakes.
  • On X, a few people described having a Hermes agent turn its own LLM-wiki usage into a repeatable skill.

Same idea three times: watch what the agent actually does, find what repeats, and write it down as a skill. The part I care about is the second half of the Perplexity description, though. Proposes, with strong evidence. An agent that rewrites its own playbook every night is how you end up with a wiki full of confident, slightly wrong procedures.

So this post builds the boring local version: a ~200-line, stdlib-only Python script that reads agent session logs, finds recurring tool-call sequences, and drafts skills/*.md pages into a proposals/ folder only when the evidence clears a bar. A human approves. It never touches an approved skill, and it tells you when an approved skill has stopped matching reality.

Pipeline: session logs go into skill_miner.py, which writes proposals; a human reviews them and approved pages move into skills/, which the agent reads next session

Facts vs. procedures

Here's the split I use:

Facts (what the wiki already has) Procedures (what it's missing)
Example "Changelog style lives in docs/style/changelog.md" "Release notes = git log → style search → read → draft → propose"
Source documents, meetings, decisions the agent's own tool calls
Goes stale when the world changes the way you work changes
Wrong one costs a bad answer a bad run, repeated every time

The last row is why I don't let the agent write skills directly: a wrong procedure fails the same way on every run.

The input: session logs

Most harnesses can already dump a transcript. I normalized mine to one JSONL file per session: a start event with a task label, one tool event per call, and an end event with whether the result was accepted.

{"event": "start", "session": "s03", "date": "2026-10-27", "task": "release-notes", "prompt": "Draft release notes for 0.26.1"}
{"event": "tool", "name": "git.log", "args": {"range": "v0.26.0..HEAD"}}
{"event": "tool", "name": "wiki.search", "args": {"q": "changelog style"}}
...
{"event": "end", "success": true}
Enter fullscreen mode Exit fullscreen mode

"Success" here just means a human accepted the output. It isn't a model grading itself, and it's the most valuable field in the log.

The fixture in this post is a sample I made up: 19 fictional sessions from a fictional team between Sep 24 and Nov 19, 2026. There are five recurring tasks plus two approved skills that already exist. I seeded it with the messy stuff real logs have: one run that retried wiki.search twice, one failed release, an FAQ workflow that gets rejected a third of the time, and a weekly-metrics skill the team quietly stopped following.

The miner

Three ideas, each a few lines.

1. Count sequences per session, not per occurrence. Every contiguous run of 4 to 8 tool calls is an n-gram. If a session loops the same four calls three times, that's still one session of evidence. Repetition inside a single run is usually a retry, not a habit.

2. One decision per task, longest passing sequence wins. For each task label I keep the longest sequence that clears all thresholds: at least 3 sessions, at least 75% accepted, last seen within 30 days. If nothing clears them, the script prints the best candidate and the exact reason it was rejected. That turns out to be as useful as the proposals.

3. Confidence is a Wilson lower bound, not a success rate. 3 out of 3 is "100%" but it's thin evidence. The Wilson lower bound for 3/3 is 0.44 and for 4/4 it's 0.51, which matches how much I actually trust those numbers.

Core of skill_miner.py (candidates, pick, stale/contradicted check)
def pick(cands, today, min_sessions, min_success, max_age):
    """One decision per task: the longest sequence that clears every threshold,
    or, if none does, the best-supported one and the reason it failed."""
    groups = defaultdict(list)
    for c in cands:
        groups[c["name"]].append(c)
    out = []
    for name, cs in groups.items():
        ok = [c for c in cs if not judge(c, today, min_sessions, min_success, max_age)]
        if ok:
            out.append((max(ok, key=lambda c: (len(c["steps"]), c["support"])), None))
        else:
            c = max(cs, key=lambda c: (c["support"], c["rate"], len(c["steps"])))
            out.append((c, judge(c, today, min_sessions, min_success, max_age)))
    return sorted(out, key=lambda x: (x[1] is not None, -x[0]["support"], x[0]["name"]))

def check_skills(skills, sessions, today, window):
    """Flag approved skills that recent sessions no longer follow."""
    recent = [s for s in sessions if (today - s["date"]).days <= window]
    flags = []
    for name, sk in skills.items():
        if sk["meta"].get("status") != "approved":
            continue
        steps, task = sk["steps"], sk["meta"].get("task", name)
        matched = [s for s in sessions if contains(s["steps"], steps)]
        last = max((s["date"] for s in matched), default=None)
        same_task = [s for s in recent if s["task"] == task]
        hits = [s for s in same_task if contains(s["steps"], steps)]
        if same_task and len(hits) * 2 < len(same_task):
            variant, n = Counter(tuple(s["steps"]) for s in same_task
                                 if not contains(s["steps"], steps)).most_common(1)[0]
            flags.append(("CONTRADICTED", sk["path"],
                          f"{len(hits)}/{len(same_task)} recent '{task}' runs match; "
                          f"{n} use: {' > '.join(variant)}"))
        elif not same_task and not any(contains(s["steps"], steps) for s in recent):
            when = f"{last} ({(today - last).days} days ago)" if last else "never"
            flags.append(("STALE", sk["path"], f"last matched {when}, 0 runs in last {window} days"))
    return flags
Enter fullscreen mode Exit fullscreen mode

The write path has three guards, and each one has a unit test:

  • If skills/<name>.md exists and is approved, the proposal goes to proposals/<name>.update.md with supersedes: in the frontmatter. The approved file is never opened for writing. The test compares its bytes before and after.
  • If a proposal is already waiting for review, it's kept (KEEP), so a human's half-finished edits never get clobbered by the next nightly run.
  • --dry-run prints the decisions and writes nothing.
$ python3 -m unittest test_skill_miner
...........
----------------------------------------------------------------------
Ran 11 tests in 0.012s

OK
Enter fullscreen mode Exit fullscreen mode

Running it on the sample fixture

$ python3 skill_miner.py fixture --today 2026-11-20
skill_miner: 19 sessions (2026-09-24..2026-11-19), 2 approved skills
PROPOSE  incident-postmortem  6 steps, 4 sessions, 100% ok, conf 0.51 -> proposals/incident-postmortem.md
PROPOSE  release-notes        5 steps, 3 sessions, 100% ok, conf 0.44 -> proposals/release-notes.md
PROPOSE  weekly-metrics       4 steps, 3 sessions, 100% ok, conf 0.44 -> proposals/weekly-metrics.update.md (approved weekly-metrics.md exists; not touched)
SKIP     customer-faq         success 67% < 75% (2/3 sessions)
SKIP     dependency-bump      best sequence seen in 2 sessions < 3
SKIP     onboarding-checklist best sequence seen in 1 session < 3
STALE        skills/onboarding-checklist.md: last matched 2026-09-24 (57 days ago), 0 runs in last 21 days
CONTRADICTED skills/weekly-metrics.md: 0/3 recent 'weekly-metrics' runs match; 3 use: metrics.query > dashboard.snapshot > draft.write > wiki.propose
Enter fullscreen mode Exit fullscreen mode

Terminal output of skill_miner.py on the 19-session sample fixture: three proposals, three skips, one stale skill and one contradicted skill

What a reviewer should take from that:

  • release-notes has five sessions in the logs but only 3 count as evidence for the full five-step path. One run failed before proposing, and one retried wiki.search twice, which breaks the contiguous match. The miner is strict about contiguity. I'm fine with that. A skill should describe the clean path, not the one with retries.
  • customer-faq happens just as often but gets rejected a third of the time. Turning that into a skill would just teach the agent to fail faster. The rejection reason tells you the real fix is upstream (the ticket clustering step, probably), not in the playbook.
  • weekly-metrics is the interesting one. The approved skill says "export to a spreadsheet, update the wiki page in place." For the last three weeks every run has done something different: snapshot the dashboard, draft, and propose the page for review. Nobody updated the skill. The miner proposes the new path as an .update.md and flags the old one as CONTRADICTED in the same run, so the reviewer sees both sides together.
  • onboarding-checklist hasn't been used in 57 days. Onboarding is seasonal, so that doesn't make it wrong, but it deserves a glance before the agent trusts it again.

A second run changes nothing: all three proposals come back as KEEP.

What a proposed skill page looks like

Rendered proposals/release-notes.md: frontmatter with status proposed, owner human, evidence s03 s16 s19, confidence 0.44, then steps, a why section and a review checklist

Two choices here are deliberate. owner: human means the agent is never the owner of its own procedures. And the "why" section cites session ids, which gives you the SIM idea in miniature: when someone asks in February why the agent always reads the changelog page first, the answer is three linked sessions, not a vibe.

The review checklist at the bottom of every page ("nothing in the args is secret or one-off") matters more than it looks. Session args are where tokens, customer names and one-time paths leak into a doc that every future session will read.

Where skills live in a team knowledge base

A skills/ folder in a repo works for one developer. For a team, I'd put skill pages in the same place as the docs and decisions they depend on. A skill that says "read the changelog style page" should link to that page, so a reviewer sees the procedure and its inputs side by side.

In AFFiNE, this is how I would set it up. Skills become ordinary pages with properties (status, owner, evidence, confidence), in the same workspace as the docs and the whiteboard where the team sketches the process. The miner runs against a Markdown export and its proposals land in a review collection. Approving one would mean a person flips status to approved. If you're choosing knowledge base software for this, look for queryable page properties and backlinks, so "which approved skills cite this doc?" is one query when the doc changes.

The review flow is the same one teams already use for SOPs, just with better evidence attached. If you've written process documentation before, the shape will feel familiar: trigger, steps, owner, review date. The difference is that the first draft comes from what actually happened, not from what someone remembers happening. And if you're comparing tools for the human side of that loop, this rundown of SOP management software covers the approval and versioning features that matter.

I'd also keep this layer somewhere you control. Procedural memory says more about your team than the facts do: which systems you touch, in what order, and where you cut corners. That's an argument for an open source Notion alternative you can self-host, rather than one more SaaS that learns your workflows. It's also why I think the "agents write their own skills" trend belongs inside your AI knowledge base software, not in a side database owned by whichever agent vendor you use this quarter.

What it does badly (on purpose)

It needs a task label. Grouping by task makes proposals readable and keeps release notes from merging with postmortems. If your harness doesn't log intent, you'll need to cluster prompts first, and that's a harder problem.

Contiguity is strict. One retry and the run doesn't count. You could allow gaps (subsequence matching), but then the miner starts "finding" skills that are really just "every session reads a wiki page." I'd rather have strict and slow.

It only sees tool names. It can't tell that wiki.search for "changelog style" and wiki.search for "who owns sync" are different steps. The args shown on the proposal page are hints for the reviewer, not part of the match. A next version could bucket args. I'd still keep the human gate.

If you want the bigger picture of how this fits with the fact side, here's a good primer on what an LLM wiki is and how teams structure one.


Try it: export a week of your own agent transcripts to JSONL, run the miner with --dry-run, and read the SKIP lines before the PROPOSE lines. In my sample fixture, the skills it refused to write were just as informative as the ones it wrote.

Top comments (1)

Collapse
 
aistackguide profile image
AI Stack Guide •

The facts/procedures split is the useful framing here, and the "a wrong procedure fails the same way on every run" line is why I think your review gate is the whole design rather than a safety feature.

Two places I would push the same idea further, from running a notes-and-transcripts pipeline over the last few months:

  1. Meeting transcripts are where procedural evidence actually lives, and they are the worst possible input for this. They have no task label, the tool calls are implied ("I'll check the tracker") rather than logged, and one hour of speech produces a dozen near-duplicates of the same decision. If you ever mine them, deriving the task label from the calendar event title is more reliable than clustering prompts afterwards — my attempt at clustering produced topics, not intents.

  2. The contiguity rule is right, and I would keep it even for transcripts. What I would add is speaker attribution on the steps: "who asked for the approval" is often the part of the procedure that matters, and it is also the part that leaks personal data. Your "nothing in the args is secret or one-off" checklist should run over names and pricing, not just tokens, the moment the evidence source is a customer call.

The Wilson bound on 3/3 being 0.44 is the kind of number that changes a reviewer's mood. Printing the rejection reason next to the proposal is the part I would steal.