DEV Community: Fancy39

I built a coding agent skill that writes you a course and quizzes you on it

Fancy39 — Tue, 18 Aug 2026 07:28:11 +0000

I use Claude Code every day. It writes my tests, fixes my refactors, and explains things I was too tired to read. But when I wanted to actually learn something new, the mechanics of async/await, how flexbox really works, the math behind a normal distribution, I kept leaving the agent and going to YouTube, a blog post, a textbook I'd give up on by page four.

The agent was right there. It could search the web and explain things back to me. Why was I still going elsewhere to learn?

The honest answer is that "explain X to me" is a bad prompt. It gives you a blob. You read it, feel like you understood, and a week later you can't recall any of it. That's not a model problem. It's a structure problem. Learning needs prerequisites, pacing, worked examples, and some way to check whether the knowledge actually went in. A single chat reply gives you none of that.

So I built a skill that does give you that. This post is about what I built, what the skill format lets you do that a web app can't, and where I went wrong.

What the skill actually does
You tell your agent "I want to learn X." The skill takes over. It researches sources with the agent's own search tools, cited not invented, cuts an outline with prerequisites mapped as a directed graph, caps each lesson at seven new concepts, and writes the lessons through Gagné's nine events (hook, explain, worked example, faded practice, independent practice, recap). The output is a folder of Markdown files on your disk.

The skill's entry point is a SKILL.md that routes intent to one of five sub-skills:

agent-mentor/
├── SKILL.md # intent router
└── skills/
├── generate-course-from-topic/ # "I want to learn X"
├── review-course/ # "quiz me"
├── maintain-course/ # "update my course"
├── export-course/ # "export to PDF / Anki"
└── publish-course/ # "put it online"
The frontmatter is the part the agent reads first:

name: agent-mentor-skill
description: "\"Use for EVERY learning request, even unnamed —"
\"I want to learn X\", \"teach me X\", learn X.
Never answer inline; build a structured self-paced course.

Also: review/quiz, maintain, export, publish online."

That description is the routing key. When the user says "I want to learn regex," the agent pattern-matches the description and loads the full workflow from the matching sub-skill. No other skill loads. That's progressive disclosure: 100 tokens to decide, the full instructions only when it's relevant.

The part I actually cared about: review
Writing a course is a one-time effort. Making sure someone learned from it is the harder problem, and it's where most of the engineering time went.

Most review tools show you the same card until you memorize it. That tests recall, not understanding. The review skill generates fresh questions each session from the course material, grades your answers locally, and reschedules based on how you did. The question generation runs through a hosted model, but the grading logic is auditable and the model only sees the question, not your entire course.

The review queue is a JSON file on disk, not a database. Run npm run review:queue and you get today's batch:

{
"items": [
{
"course": "css-flexbox-zh",
"kind": "term",
"id": "display: flex"
},
{
"course": "css-flexbox-zh",
"kind": "term",
"id": "justify-content"
},
{
"course": "css-flexbox-zh",
"kind": "term",
"id": "align-items"
},
{
"course": "css-flexbox-zh",
"kind": "term",
"id": "flex item"
}
]
}
That's what the review skill reads. No server, no account. You can version-control your learning state alongside your code.

The scheduling uses FSRS (the algorithm behind the srs-benchmark project). Each item has a stability score that grows with correct reviews. Wrong answers drop stability and bring the item back sooner.

Interactive blocks, not screenshots
Lessons ship with interactive blocks that render in the reading site. Here's a real one from the git basics course:

{
  "id": "learn-git-basics-02-staging-check",
  "label": "认出暂存区的职责",
  "prompt": "哪条命令把文件从工作目录移进暂存区，让它可以被提交？",
  "answer": "git add",
  "accept": ["git add .", "git add --all", "git add -A"]
}

That block renders as a real input in the reading site. You type an answer, it judges it locally, and the result feeds back into the review queue. There are five block types: agentmentor-check (free text judgment), agentmentor-order (reorder steps), agentmentor-predict (predict output), agentmentor-trace (trace variable state), and agentmentor-code (a pseudo-IDE drill). They're all Markdown, so they version-control cleanly and work offline.

Why a skill, not a web app
I went back and forth on this. A web app is easier to distribute. People can try it without installing anything. SEO works. You can put a free trial in front of it.

But a web app means your courses live on someone else's server. It means the learning happens in a browser tab that competes with forty other tabs. It means the agent, the thing you already use to write code, is not the thing doing the teaching.

The skill runs inside the agent you already have. Your Claude Code or Codex or opencode reads the SKILL.md, follows the workflow, and produces the course files right where you are. When you finish a lesson and want to practice, you stay in the same agent. When you want to publish a course so other people can read it without installing anything, that's a separate command that publishes to a hosted URL. The local learning and the public sharing are two different things on purpose.

The whole thing lives on your disk. No account, no cloud, no one reading your learning history. If you bring your own API key for the hosted parts (in-place answers, web research during a lesson, course podcasts), those calls go straight to your provider. You pay them directly.

Where I went wrong
I spent too long trying to make course generation work with every agent at once. The skill format is an open standard now (Claude Code, Codex, Gemini CLI, Cursor, and others all read the same SKILL.md structure), but the host agent's search and browse capabilities vary a lot. Codex has solid built-in web retrieval. Claude Code works fine if you pair it with a search skill. Some setups need you to paste source links manually. I should have picked one agent, made it perfect there, and expanded.

I also underestimated how much of the work would be in the review loop, not the course generation. The course generation is the part people see first, so I spent most of my early time on it. But a course you read once and forget is not a course. The review loop, the question generation, the scheduling, the local grading, that's where the actual retention happens, and it took roughly three times as long as the generation pipeline.

The last mistake was pricing communication. I priced it at $29.90 one-time because I didn't want to charge a subscription for something that runs on your own machine. But I spent weeks explaining to people that "one-time" means one-time, not "one-time plus credits." The hosted services (cloud publishing, the AI features if you don't bring your own key) run on a credit system, and the credits don't expire. That distinction took longer to communicate clearly than it took to build.

Where it is now
Nine sample courses are readable right now at agentmentor.dev without buying anything: git basics, CSS flexbox, HTTP fundamentals, JavaScript async/await, regex, music theory, personal finance, terminal basics, and a meta course on putting an agent to work. The English and Chinese versions are mirrors of the same course, not separate courses.

If you want to see what a generated course looks like before installing anything, read the flexbox one: agentmentor.dev/css-flexbox. The install takes about two minutes if you decide you want the rest.

I gave an LLM agent write access to my cloud drive. Three bugs taught me how to constrain it.

Fancy39 — Tue, 07 Jul 2026 09:44:44 +0000

I wanted a media library that knew the difference between what should exist and what does. Most automation I tried picked one side. Some tools search well and never track what you already have. Others move files and assume the move worked. I wanted the gap between those two things to be the thing the software acted on.

So I built Mediary Scout. You name a movie or a show. An LLM agent searches your indexers, transfers the best match into your own cloud drive, then reads the drive back to confirm what landed and what is still missing. It runs self-hosted. You bring your own drive, your own model, your own metadata key. There are desktop builds for Mac and Windows if you just want to double-click and run it, and a read-only demo if you want to watch one acquisition play out first.

The drives it speaks today happen to be Chinese cloud storage (115, Quark, GuangYaPan). That detail does not matter for the rest of this post. The part that took real work was different: handing an LLM tools that move and delete files, and stopping it from doing something dumb with them. Three bugs taught me most of what I now believe about that.

The shape of the thing

The web app does almost nothing interesting. It writes a row to a Postgres queue and returns. A long-running worker picks up the row and starts a sandboxed agent.

The agent gets a small set of tools: search resources, transfer a candidate, list a directory, move files into a season folder, mark episodes as obtained. Every tool runs through a deterministic workflow that owns the actual side effect. The agent proposes. The workflow decides whether the proposal is allowed, performs it, and reads the world back.

That split is the whole design. The model is the part I cannot fully predict, so it gets the smallest possible blast radius. The deterministic code around it holds every irreversible action and every check. When I violated that split, things broke. They broke in the order below.

Bug 1: the agent searched sixteen times and picked eleven copies of one season

A twelve-episode season needs, at most, one good complete pack. My agent decided to cover it with eleven overlapping season packs, after running sixteen separate searches to find them. One planning pass spent 6.8 minutes inside the model loop.

The transfer loop then tried to receive all eleven packs before deduplication ran. 115 has a per-operation API budget that exists to stop exactly this kind of hammering. The run tripped it (PAN115_RATE_LIMIT) and died. The whole acquisition failed because the agent was being thorough.

My prompt had a line that said overlap was safe because a later step would dedup. That line was a lie. Dedup ran after the transfers that spent the budget. The cleanup I promised the model never got a chance to happen.

I spent an afternoon rewording the prompt to ask for restraint. The model ignored all of it. Asking a model to be economical works about as well as asking water to be less wet.

The fix lived in deterministic code, not in English. A greedy set-cover function (trimToMinimalCoveringCandidates) computes the fewest packs that cover every wanted episode and drops the rest before any transfer happens. A second gate caps distinct searches at eight and dedups identical queries at the tool boundary, so the same keyword can only hit the provider once. The model can ask for sixteen searches. It gets eight, and the redundant ones come back as cached snapshots.

The lesson generalizes past my project. If your agent produces a number (how many times to search, how many things to select, how many calls to make), put a deterministic ceiling on that number. Do not trust the model to limit itself, and do not trust a future step to clean up the mess. The gate has to sit between the model and the irreversible action.

Bug 2: the coverage check I kept trying to automate

Coverage is the question of which episodes I have. The honest answer requires looking at the real files in the drive. I kept trying to compute it mechanically instead, and my own design notes kept telling me to stop.

I wrote a re-read that hit the drive after a mark to "verify" the file was present. I wrote a filename parser that guessed episode numbers from titles. I wrote a check that called a movie acquired if the directory had any video file. Each one felt reasonable. Each one put a mechanical guess in the seat where the agent's judgment belonged.

The discipline I landed on draws a hard line. The agent decides coverage by inspecting real files, after it has moved and flattened them into place. It then declares which episodes it got. That declaration is a plain statement with no file IDs and no hidden re-read. The system records it and moves on. The system never counts files on its own to second-guess the agent, and it never parses a filename to decide truth.

The bookkeeping side is just as strict. I have whatever the agent marked. I should have whatever the metadata says aired. The missing set is the subtraction between those two. A scheduled sweep only wakes the agent for shows the subtraction says are incomplete. Thousands of finished shows never get scanned, because there is nothing to scan for.

The transferable point: let the model judge the messy, real-world question, and make it judge against the actual world rather than its own narration. Keep your deterministic bookkeeping separate and dumb. Trouble starts the moment those two responsibilities bleed into each other.

Bug 3: a progress bar that rendered nothing across three pull requests

This one cost me the most pride.

Users behind a slow model reported that the inline progress bar looked empty during an acquisition. I opened a PR. The bar's phase mapping had a step in the wrong band, so I fixed the band math and shipped it.

Still empty. Second PR. The progress events fire only when a tool call happens, and one search was taking ninety-four seconds of a three-minute run, so the bar froze between events. I added a client-side trickle that animates between server updates. Shipped it.

Still empty. The author of the bug report kept insisting there was no green at all, that anything above ten percent would be visible by eye. I had "verified" my two fixes by reading style.width off the element and seeing a sensible percentage. The number was correct every time.

The third PR found it. The fill was a <span>. A span defaults to display: inline, and width has no effect on an inline box, so the fill collapsed to zero by zero pixels and painted nothing. My demo component used a <div> and worked fine. I copied its logic into the production badge and swapped the element type without thinking about it.

style.width reported five percent, then twenty-one percent, then thirty-four. getBoundingClientRect() reported a width of zero the entire time. I had been reading the value I set, not the pixels the browser drew. The two had nothing to do with each other for a collapsed inline element.

The fix was one CSS line. The lesson was larger than the fix. Test the artifact the user sees, not the proxy that is convenient to read. A green bar is green pixels on a screen. It is not a string in a style attribute. I trusted the proxy for three rounds and let a confident-sounding number lie to me.

What held up

The pattern under all three fixes is the same. A probabilistic core, wrapped in deterministic plumbing that owns every irreversible action, gates every number the model emits, and verifies against the real world instead of the model's account of it.

The agent in Mediary Scout is small on purpose. It reads search results and picks. It cannot decide that a run is done, cannot count files to fake coverage, cannot blow past a search budget, cannot touch a directory outside its sandbox. Those constraints live in code the model never sees and cannot argue with. Everything I trust about the system, I trust because a deterministic function checked it, not because the model said so.

If you are building agents that do real, irreversible things, that is the part worth most of your attention. The model will surprise you. Your job is to make the surprises cheap.

The project is open source and self-hosted at github.com/fancydirty/mediary-scout, with a live demo at demo.mediaryscout.app and desktop builds at mediaryscout.app. The drive backends are domain detail. The plumbing is the part I would reuse anywhere.

One practical note, since the drives it speaks today are Chinese and most readers won't use them. If this pattern is useful to you and you'd want your own drive supported (Google Drive, Dropbox, a regional one), the drive layer is a self-contained plugin behind a brand registry: a client plus a transfer executor. I'd welcome a PR and I'm happy to help scope one. That's the payoff of keeping the model's blast radius small. The boundaries end up clean enough that adding a drive is bounded work, not surgery.