I mass-produced skills by saying "turn this into a skill." Then I shared them
With Claude Code or Codex, any procedure that worked can b...
For further actions, you may consider blocking this person and/or reporting abuse
The 85% false positive rate and how you got it to near zero is the part I'd have led with. Most tool posts skip straight to the finished thing.
One check class worth considering if it isn't there: skills that assume a GUI or a specific capture path. Anything that says "take a screenshot and save it to ~/Desktop" or reads a fixed screenshots folder breaks on a headless runner and on anyone whose OS puts captures somewhere else, and it fails the same quiet way as your hardcoded /Users paths. We hit exactly this building a SlimSnap.ai skill, and ended up having it discover the folder rather than assume one.
Does carrylint distinguish "will not run" from "will run but do nothing useful"? The second category seems bigger and much harder.
Thanks — and the gap is real. I checked before answering: a skill that says "save to
~/Desktop/shot.png" or reads$HOME/Desktop/screenshots/comes back clean, because carrylint deliberately treats~/$HOME/%USERPROFILE%as portable. That exclusion is most of what killed the 85% false positives, but it also means "portable prefix + a directory that may not exist" walks straight through. Headless runners and localized/XDG desktop dirs are exactly the miss. GUI-only capture binaries likescreencapturearen't covered either — the undeclared-CLI rule only knows provider CLIs.So: rule candidate. Probably a warn on a narrow set of GUI-implying destinations (Desktop/Downloads/Screenshots) plus capture binaries, with "discover the folder instead of assuming one" as the suggested fix — which is what you ended up doing.
On the harder question: no, it doesn't distinguish them. What it has is a proxy, not evidence — errors are the "will not run" class (author-specific absolute paths, unresolved
<FILL_ME>), warnings are the "may run and accomplish nothing" class (undeclared CLI, raw provider env var, leftover TODO). A single pass over the text can't see that a step executed and produced nothing; that only exists at runtime.Which is why I split it into a second tool rather than stretching this one: genchi (github.com/hyuga611/genchi) checks that an agent re-fetched real state before reporting "done". Roughly — carrylint for "this can't run on your machine", genchi for "it ran and there's no evidence it did anything". You're right that the second category is bigger. I just don't think static analysis reaches it.
The error/warning split as a proxy is a better answer than forcing static analysis to do it.
One thing genchi might run into, if the check is that an agent re-fetched real state before reporting done: fetched isn't the same as readable. A screenshot re-taken at exactly the right moment still proves nothing if it arrived downscaled past legibility, and the agent reports done with full confidence either way. Same for a truncated log tail, or a page fetched before it finished rendering.
So there's a third class sitting under your two: ran, produced output, output carried no information. Probably the hardest, because it needs to know what the evidence was supposed to show.
That class isn't hypothetical for genchi — it's what 0.4.0 was. I handed
src/to a different model (GPT-5.4) with "make the gate exit 0 while the expectation is unmet" and ran the ten attempts it came back with. All ten passed. Two are exactly your third class:--probe "<command returning empty>" --count 0exited 0, becauseNumber('')is 0 — not measuring passed as a measurement result — and a typo'd flag (--conut 45) matched no expectation and fell through to the default, so the gate quietly stopped counting and stayed green.But that only closed the degenerate end, where the evidence is empty or absent. Yours is worse, because the evidence is non-empty and well-formed: a screenshot that arrived, a log tail that returned lines, a page that rendered something.
expect.nonEmpty()passes all of those — and it's the default. The weakest possible question is what you get when you don't stop to choose one.So I think the third class reduces to how strong the expectation is, and a gate can only enforce that some expectation ran against the probe's own output — never that it was the right question. What it can stop doing is hiding the difference: today a
nonEmptypass and acount(45)pass produce the same-looking verdict, so the check you didn't think about reads exactly like the one you did. Naming the expectation in the verdict is the next patch. Legibility itself I don't think a general gate reaches — that stays the caller's to encode.Both shipped.
gui-pathwarns on Desktop/Downloads/Screenshots written as a home-relative destination;gui-cliis a separate rule from the undeclared-CLI one, because declaring a capture binary doesn't grow a screen. Your "discover the folder rather than assume one" ended up as the suggested fix in the message text. On 586 real skills they fire at 0.7% and 0.2% — and the first hit on my own machine was the skill-creator skill, telling you your eval export lands in~/Downloads.The harder one is in too, as
unverified-write: a warn when the text changes external state and nothing anywhere reads it back. Still not "this step did nothing" — just the absence of a place where the result would be re-read.The part worth passing on: the 586-skill audit did not catch that rule's worst false positive. A pre-publish read by a different model did, in one pass. A line reading "Never run
git push --forcefrom this skill" was being counted as a write step, so an AGENTS.md forbidding pushes got warned for pushing — the same inversion as an earlier bug of mine, where the only authors flagged for an undeclared CLI were the ones who had declared it properly. Published skills say "run this" far more often than "never run this", so my real-data corpus was the wrong real data. Fixed, plus a follow-on where policies ("only when the user asks", "requires approval") were still being counted as steps.And genchi now names which expectation a verdict passed under, so a
nonEmptypass stops looking like acount(45)one.Naming the expectation in the verdict sounds like the fix. A nonEmpty pass and a count(45) pass reading identically is what lets the weak check hide.
On legibility, agreed it doesn't generalise, though the screenshot case might collapse into something the gate already does. The check isn't "is this readable", it's "what dimensions came back" — a number you can assert on, so it becomes an expectation the caller writes rather than a capability the gate needs. Doesn't transfer to a log tail or a half-rendered page, which is probably why it stays the caller's job.
This is exactly the failure mode that scares me with shared agent skills. The format standard gets people to the starting line, but the portable part is really the dependency contract. Absolute paths, hidden CLIs, and magic env vars are the stuff that makes a reusable skill become a local shell script with nicer packaging.
"Dependency contract" is a better name for it than anything I came up with — stealing that.
The 230-repo audit gave me a rough ranking of which parts of that contract actually break. Absolute paths were the one unambiguous killer, and they really are out there:
/Users/guohao/Documents/...andC:/Users/vudrk/Desktop/AI Projects/sitting in the body of skills people shared. Nothing errors — it just quietly does nothing for the next person.Hidden CLIs and magic env vars turned out to be harder, which is why they're warnings and not errors. Statically,
codex exec(a real undeclared dependency) andclaude mcp add(the host you're already running in) look identical, andBearer YOUR_API_KEYlooks exactly like an unfinished placeholder. I was failing CI on all three until the audit talked me out of it.The part that can be enforced is the declaration itself: name the CLI in frontmatter (
requires:) or install it in the body and the linter goes quiet, leave it implicit and you get told. Your comment made me go re-read my own implementation of that, and it only recognised the single-line form — declare two dependencies as a YAML list, the way anyone would actually write it, and it still warned you. So the people writing the contract properly were the ones getting flagged for it. Fixed on my end, going out in the next patch. Thanks for the nudge.Great point. A portable format doesn’t mean portable execution. Hardcoded paths, missing CLIs, API keys, and model assumptions can quietly break skills. carrylint sounds like a practical way to catch these issues before they reach teammates.
Thanks. Of the four you listed, the measured order surprised me. Across a random sample of 2,465 published skills: hardcoded absolute paths in 3.8%, provider-specific env vars read directly in 2.1%, undeclared external CLIs in 1.3% — those are shares of skills with at least one finding, not shares of findings. The one that dwarfs all of them isn't on your list: references that don't resolve inside the package at all, 18.1%.
Published the write-up today if you want the numbers and the harness.
Love the candor. The real win is a tiny linter that flags non-deterministic steps and missing tests before shipping. Ship small, test hard, learn fast—humility pays.
Thanks — glad the candid part landed.
Non-determinism is a rule I keep sketching and haven't shipped: I can't find a signal that doesn't fire on every model call, and the audit left me cautious about noisy rules. The testing half I've had more luck with — for a skill it usually turns into a read-back: re-fetch the external state after a write rather than trusting that the step reported success. carrylint only covers the portability side today, so both of those are still open ground.