You can't quiz someone into writing tests

Every course I've taken has a lesson about testing. It explains what a unit test is, has you write one into a box, marks it green, and moves on. I passed several of those lessons. I did not start writing tests.

The same is true of git. There's a lesson on commit messages, usually with a multiple-choice question about the imperative mood. Nobody's commit history changes because of it.

That's not the courses being lazy. It's that these aren't facts, they're habits, and a habit is visible in a year of work rather than in one answer. You can quiz somebody on what a conventional commit is. You can't quiz them into writing one on a Tuesday when they're tired.

So when I built my own Python course, I left that lesson out and wrote something else instead: the course reads your repositories and grades what's already there.

What it actually does
There are seven probes and eleven awards between them. Each probe opens a repository you point it at, runs the tools that repository already uses, and reads the output. Awards pay experience, which is what the XP column below is.

Probe What it looks for XP
git commit subjects that follow a convention 40
git branches named for their work 25
lint a tree your linter has nothing to say about 40
lint formatting nobody has to argue about 25
tests a test suite that exists at all 50
tests a suite that is actually green 75
types signatures that say what they mean 40
docs a README someone else could walk through 25
docs decisions written down where you can ask about them later 40
ci something other than you checking the work 30
ci a pipeline that stays green 45
Totals put you on a ladder: Starting out, then Finding the habits at 100, Habits that hold at 250, Dependable at 500, Seasoned at 1000.

Every award carries evidence — the actual number it saw — because an award you can't check is a sticker.

The three problems that were actually interesting
Writing the probes was the easy half. The hard half was deciding what a probe's answer means.

"I found nothing" and "I couldn't look" are different answers
The CI probe wants to know whether your last few pipeline runs were green. That needs the GitHub CLI installed, signed in, and a network.

If gh isn't there, the probe could reasonably say "no green pipeline". That's a disaster: the first time you scan on a laptop without gh, the course quietly takes away an award you earned, for a pipeline that is still green.

So a probe returns one of two things, and the difference is load-bearing:

def evaluate(self, repo: Path) -> list[Award] | None:
"""What this probe finds now, or None if it could not look.

The difference matters. An empty list means the repository
does not earn this, and the scan takes the award away. None
means the probe could not run at all, and the scan leaves
what was already held alone - losing an award because ruff
was not installed today would be a lie about the work.
"""

A list — possibly empty — means I looked, and this is what's there. None means I couldn't answer, and the scan leaves whatever was already held alone.

It ended up as two probes rather than one, because "does a workflow file exist" can always be answered and "were the last runs green" often can't:

Two probes rather than one, because they fail differently. [...] One probe for both would revoke a green pipeline every time gh was missing, which is the lie the None return exists to prevent.

Awards have to be revocable, and that's uncomfortable
The obvious design is that experience only goes up. It feels better, it's what games do, and it avoids an awkward conversation.

It's also a lie with a delayed fuse. If you earn "a suite that is actually green" and then break your tests and leave them broken, a number that still says otherwise is wrong — and you're the one person guaranteed to find out.

So a scan can take an award back:

An award that survives the work that earned it being undone is a lie the learner will eventually notice.

A probe can't do this, because a probe only sees now. Only the scan knows what was held before, so only the scan can remove it.

Re-earning is not re-paying
If running a scan twice paid twice, the meter would measure how often you run scans.

So an award's id identifies the achievement, not the occasion. Scan again and the evidence is refreshed, so it describes your repository as it stands today — but the date you first earned it is left alone, and nothing is paid twice.

The threshold is 80%, on purpose
The git probe looks at your last 100 commit subjects and wants 80% of them to follow the convention. Not all of them.

I went back and forth on this and settled it with a comment that is still in the file:

An occasional "wip" is a human being, not a failure of craft.

A tool that demands perfection to acknowledge a habit teaches you to perform for the tool. The same reasoning sets the floor at ten commits: below that there's no habit to judge, just a handful of commits.

Why it runs on your machine and not on a server
Here's the part I'd have got wrong if I'd built it the obvious way.

Running these probes means running your test suite. On a hosted service, that is executing a stranger's code on my server on their say-so — the exact thing the project's sandboxing decisions exist to prevent. So the hosted version refuses:

403 — The craft meter runs real test suites, so it is for a local install
So it runs where your code already is: you run the application on your own machine and point it at your own repositories. Your code is never uploaded, because there is nowhere to upload it to. No token, no OAuth scope, no third party with read access to your private repos.

One honest caveat, since this is the part people ask about: the local install isn't something you can go and get yet. The hosted course is live, but the craft meter is the half that needs the application on your own machine, and packaging that up is still ahead of me. If this is the bit you want, say so and I'll know it's worth doing sooner.

I'd like to claim that as a privacy feature. Honestly, it's a consequence — I couldn't safely build it the other way, and the constraint produced the better design. That happens more often than I expect it to.

What it doesn't do
It doesn't read your code and tell you it's good. It reads what your own tools say about it. If your linter is configured to permit everything, a clean tree means very little, and the probe can't tell.

It won't make you employable on its own. It measures habits an interviewer can also see in thirty seconds of scrolling your GitHub — which is rather the point. Those are the habits worth having, so they are the ones worth counting.

And it is genuinely slow. It runs pytest over every repository you give it. That's not a bug to optimise away; it's what makes the number mean something.

This is part of LexipawsPY, a free Python course I'm building. Every lesson is free and stays free — the whole course, the drills, the hints and the solutions. It's at py.lexipaws.eu if you want a look, though fair warning: the beginner path is still being written. It runs to sixty-eight lessons today, through loops, lists and tuples, and stops just before dictionaries.

If you've built something similar, or you think grading habits is a terrible idea, I'd like to hear it.