Session log in, reusable skill out. Every tool with that shape has the same four lines, and two of them are guesses:
candidates = mine(session, min_support, max_span) # two thresholds, four error curves
groups = merge(candidates, merger) # where it goes wrong quietly
skills = verify(groups) # necessary, and not sufficient
shortlist = rank(skills, by=description_length)
Repo: https://github.com/dev48v/auto-skill-builder - PUBLIC, MIT, standard library only, 41 tests, no API key, no network. Running in your browser: https://dev48.infy.uk/agentlab/vol2-03-auto-skill-builder.html
Counting is not mining: the six most frequent verbs, in frequency order, occur as a sequence once in 53 actions. What the job needs is gapped subsequence support with a window, because the same workflow turns up with different arguments and twice with a whole second procedure interleaved step for step.
No cell of the grid reaches zero
Thirty-two cells of support x span, both mistakes counted separately - a skill emitted that nobody performed, and a real procedure missed.
| min_support | max_span | emitted | found | spurious | missed |
|---|---|---|---|---|---|
| 2 | 6 | 59 | 3 of 4 | 56 | 1 |
| 3 | 9 | 38 | 1 | 37 | 3 |
| 4 | 9 (shipped) | 12 | 1 | 11 | 3 |
| 5 | 5 | 3 | 0 | 3 | 4 |
The shipped default emits exactly one skill and misses three of the four procedures. The most any cell finds is three, at the price of 56 nobody performed. That is a property of the input: the two-step fragment test.run -> git.commit sits inside two procedures at support 7, above every real procedure at best 4, and closed-pattern filtering cannot remove it because nothing longer ties its support.
Generalising the arguments, both directions, never added together
Thirteen labelled run-pairs. Seven are one skill with parameters; six are two different skills. An over-merge ships a skill that does the wrong thing; a refusal ships two near-identical skills.
| merger | correct | over-merged | refused |
|---|---|---|---|
| strict (identical runs only) | 6/13 | 0 | 7 |
| variance (if it varies, parameterise it) | 7/13 | 6 | 0 |
| effect-aligned (substitution rewrites the effects) | 9/13 | 2 | 2 |
| oracle, handed the answers | 13/13 | 0 | 0 |
The rule almost everyone implements gets 7, and all six of its mistakes are in the dangerous direction.
Both of my assumptions about which part was hard
I expected mining to be the working core, and it is the part that fails. I expected replay to be the filter. At min_support 3, max_span 10 the unfiltered candidate list is 36 and the verified list is 36 - every one binds back to the runs it claims and reproduces their effects, because its steps really happened and really had those effects. Replay catches a skill that is wrong. It cannot catch a skill that is not a skill, and the top-ranked candidate is three procedures spliced at the seams, replaying 3/3.
The only signal that separated a procedure from a chimera was not in the plan and is not statistical: a real procedure hands a value between its own steps - the branch you check out is the branch you push. Filtering on that cuts 36 candidates to 4 and puts a real procedure first. The cost is stated rather than hidden: it discards every real procedure whose steps share no argument, and one of the four is exactly that.
The shipped default therefore emits one skill, replays 3/3, saves 16 tokens, and parameterises --no-verify - the flag that decides whether your pre-commit hooks run. It is wrong, every test passes, and no downstream check can see it, because skipped hooks leave no trace in the log at all.
Agent Lab Vol 2, Project 03: https://dev48.infy.uk/agentlab.php
Top comments (0)