That amazing skill you shared only runs on your machine. I built a linter for it — then found I was 85% wrong

I mass-produced skills by saying "turn this into a skill." Then I shared them

With Claude Code or Codex, any procedure that worked can become a reusable SKILL.md just by asking "turn what we just did into a skill." I did this for image generation, deploy steps, everything. It was great.

Where it fell apart was handing those skills to teammates. They ran on my machine and silently died on theirs. Every time I dug in, it was the same shape:

• Output path hardcoded to C:\Users\atlan\Downloads\out.png (my home directory)
• The body calls codex exec ... with no instruction anywhere for installing that CLI
• Assumes OPENAI_API_KEY is already set, with no guidance when it isn't
• A model id like gpt-image-2 written inline

All of them are the author's environment baked in. None of them raise an error — they just fail quietly in the next person's hands, which is the worst kind.

So I built carrylint, a linter that fails CI on exactly this.

The standard made the format portable. Whether the contents run is a separate question

In December 2025, Anthropic made Agent Skills an open standard. One SKILL.md now runs across Claude Code, Codex, Gemini CLI, Cursor, Copilot, and 20-odd others. Format portability is solved.

But a standard only guarantees the shape of the container. If the inside holds an absolute path or an undeclared CLI, the container is valid and the contents still don't run for anyone else. I couldn't find a tool looking at that.

• reflint (also mine) asks: do the references exist?
• skills-lint (also mine) asks: do skills collide, is the frontmatter valid?
• carrylint asks: do the references resolve in a different environment, under a different model?

The failure class is inverted. The others check "is this correct as specified." carrylint checks "will the next person install this and have it actually run."

What it catches

Run it against a deliberately non-portable sample:

✗ examples/bad/leaky-image-gen/SKILL.md — 3 errors / 3 warnings
  ✗ :16  [abs-path] machine-specific absolute path `C:\Users\alice\Downloads\out.png`
         — will not resolve on anyone else's machine (use a relative path or {baseDir})
  • :16  [undeclared-cli] calls `codex` but never declares or installs it
  ✗ :22  [abs-path] machine-specific absolute path `C:\Users\alice\Downloads\out.png`
  • :25  [provider-env] assumes `OPENAI_API_KEY` is set
  ✗ :27  [placeholder] unresolved placeholder `<FILL_ME>` left in a shipped file
  • :29  [todo] TODO/FIXME marker left in a shipped file

carrylint: 3 errors / 3 warnings
exit code: 1

Rules are split by severity. False positives are what get a linter uninstalled, so only things the next person will hit, with no room for interpretation, are error (which fails the PR). I moved that line a lot after shipping — see below.

Severity	Rules
error	Author-environment absolute paths (C:\…, /Users/<realname>/); unfinished markers (<FILL_ME>, REPLACE_ME). $HOME, ~, YOUR_API_KEY, /path/to/ are excluded — they're portable or a documentation convention
warn	Undeclared external CLIs (host commands like claude mcp add excluded); raw provider-specific env references; leftover TODO:
opt-in	Hardcoded model ids (claude-*, gpt-*) — deliberate pinning is legitimate, so off by default

No LLM and no API key at runtime — pure static analysis. Since the criterion is "are you locked into one environment or model", it works identically whether the skill was written by Claude or Codex. The tool embodies the thing it checks for.

Then I ran it against 230 real skills and learned I was 85% wrong

This is the part I most want to be honest about. I had shipped it, but something nagged before I promoted it: my own examples and my own tests passing proves nothing. So I collected 230 real SKILL.md / AGENTS.md files from public GitHub repos and ran carrylint against them untouched.

The good half. Skills that only run for their author are genuinely out there. One PPT-generation skill hardcoded /Users/guohao/Documents/... — the author's Mac path — into the body. Another used C:/Users/vudrk/Desktop/AI Projects/ as the base for every script. None of these error out; they die quietly for the next person. carrylint's reason to exist was sitting in the real data.

The bad half. About 85% of my errors were false positives. I was flagging Bearer YOUR_API_KEY (the standard API-docs convention for "put your key here") as unfinished. I was flagging claude mcp add — the host itself — as an undeclared CLI. I was flagging $HOME/..., which resolves per user and is the portable way to write it, as a machine-specific absolute path. I'd written that false positives are a linter's only cause of death, and then nearly shipped promotion on top of an 85% false-positive rate.

Luckily the real data named exactly what to fix. v0.1.1 corrected four things: $HOME/~/generic names are portable; placeholders narrowed to genuine fill-me markers; host CLI setup commands (claude mcp add etc.) excluded; the home-relative path rule dropped entirely. Re-run against the same 230: error false positives went from ~85% to near zero, and every true positive survived. I then ran it against 70 repos it had never seen to check I hadn't overfit (3% fired, all genuine). The real examples I found are bundled in the repo as regression tests.

One lesson: your own tests passing is not evidence of correctness. You find out by running against real data — and by doing it before you promote, not after.

Honestly: this niche is already crowded

One more admission. Partway through designing this I went looking, and there were already 7+ linters for SKILL.md as of 2026. My own skills-lint is one of them. This was not an empty lot.

But I read the ones I could find, and none of them looked at whether the contents actually run somewhere else. They stop at spec compliance and frontmatter. So carrylint does only that. Teams that mix Claude and Codex and distribute skills to each other are honestly still rare — the demand may be slightly ahead of its time. Still, if "I shared it and it only ran for me" has ever stung you, the low-noise error rules (sharpened by that audit) should earn their place.

Summary

The standard made SKILL.md portable as a format. Whether it runs is a different question. carrylint fails CI on that difference.

- uses: hyuga611/carrylint@v0     # in CI

npx @hyuga/carrylint              # right now, locally

Repo: https://github.com/hyuga611/carrylint

Comments

[Alex @Bickov]

The 85% false positive rate and how you got it to near zero is the part I'd have led with. Most tool posts skip straight to the finished thing.

One check class worth considering if it isn't there: skills that assume a GUI or a specific capture path. Anything that says "take a screenshot and save it to ~/Desktop" or reads a fixed screenshots folder breaks on a headless runner and on anyone whose OS puts captures somewhere else, and it fails the same quiet way as your hardcoded /Users paths. We hit exactly this building a SlimSnap.ai skill, and ended up having it discover the folder rather than assume one.

Does carrylint distinguish "will not run" from "will run but do nothing useful"? The second category seems bigger and much harder.

[hyuga]

Thanks — and the gap is real. I checked before answering: a skill that says "save to ~/Desktop/shot.png" or reads $HOME/Desktop/screenshots/ comes back clean, because carrylint deliberately treats ~ / $HOME / %USERPROFILE% as portable. That exclusion is most of what killed the 85% false positives, but it also means "portable prefix + a directory that may not exist" walks straight through. Headless runners and localized/XDG desktop dirs are exactly the miss. GUI-only capture binaries like screencapture aren't covered either — the undeclared-CLI rule only knows provider CLIs.

So: rule candidate. Probably a warn on a narrow set of GUI-implying destinations (Desktop/Downloads/Screenshots) plus capture binaries, with "discover the folder instead of assuming one" as the suggested fix — which is what you ended up doing.

On the harder question: no, it doesn't distinguish them. What it has is a proxy, not evidence — errors are the "will not run" class (author-specific absolute paths, unresolved <FILL_ME>), warnings are the "may run and accomplish nothing" class (undeclared CLI, raw provider env var, leftover TODO). A single pass over the text can't see that a step executed and produced nothing; that only exists at runtime.

Which is why I split it into a second tool rather than stretching this one: genchi (github.com/hyuga611/genchi) checks that an agent re-fetched real state before reporting "done". Roughly — carrylint for "this can't run on your machine", genchi for "it ran and there's no evidence it did anything". You're right that the second category is bigger. I just don't think static analysis reaches it.

[Alex @Bickov]

The error/warning split as a proxy is a better answer than forcing static analysis to do it.

One thing genchi might run into, if the check is that an agent re-fetched real state before reporting done: fetched isn't the same as readable. A screenshot re-taken at exactly the right moment still proves nothing if it arrived downscaled past legibility, and the agent reports done with full confidence either way. Same for a truncated log tail, or a page fetched before it finished rendering.

So there's a third class sitting under your two: ran, produced output, output carried no information. Probably the hardest, because it needs to know what the evidence was supposed to show.

[hyuga]

That class isn't hypothetical for genchi — it's what 0.4.0 was. I handed src/ to a different model (GPT-5.4) with "make the gate exit 0 while the expectation is unmet" and ran the ten attempts it came back with. All ten passed. Two are exactly your third class: --probe "<command returning empty>" --count 0 exited 0, because Number('') is 0 — not measuring passed as a measurement result — and a typo'd flag (--conut 45) matched no expectation and fell through to the default, so the gate quietly stopped counting and stayed green.

But that only closed the degenerate end, where the evidence is empty or absent. Yours is worse, because the evidence is non-empty and well-formed: a screenshot that arrived, a log tail that returned lines, a page that rendered something. expect.nonEmpty() passes all of those — and it's the default. The weakest possible question is what you get when you don't stop to choose one.

So I think the third class reduces to how strong the expectation is, and a gate can only enforce that some expectation ran against the probe's own output — never that it was the right question. What it can stop doing is hiding the difference: today a nonEmpty pass and a count(45) pass produce the same-looking verdict, so the check you didn't think about reads exactly like the one you did. Naming the expectation in the verdict is the next patch. Legibility itself I don't think a general gate reaches — that stays the caller's to encode.

[hyuga]

Both shipped. gui-path warns on Desktop/Downloads/Screenshots written as a home-relative destination; gui-cli is a separate rule from the undeclared-CLI one, because declaring a capture binary doesn't grow a screen. Your "discover the folder rather than assume one" ended up as the suggested fix in the message text. On 586 real skills they fire at 0.7% and 0.2% — and the first hit on my own machine was the skill-creator skill, telling you your eval export lands in ~/Downloads.

The harder one is in too, as unverified-write: a warn when the text changes external state and nothing anywhere reads it back. Still not "this step did nothing" — just the absence of a place where the result would be re-read.

The part worth passing on: the 586-skill audit did not catch that rule's worst false positive. A pre-publish read by a different model did, in one pass. A line reading "Never run git push --force from this skill" was being counted as a write step, so an AGENTS.md forbidding pushes got warned for pushing — the same inversion as an earlier bug of mine, where the only authors flagged for an undeclared CLI were the ones who had declared it properly. Published skills say "run this" far more often than "never run this", so my real-data corpus was the wrong real data. Fixed, plus a follow-on where policies ("only when the user asks", "requires approval") were still being counted as steps.

And genchi now names which expectation a verdict passed under, so a nonEmpty pass stops looking like a count(45) one.

[Alex @Bickov]

Naming the expectation in the verdict sounds like the fix. A nonEmpty pass and a count(45) pass reading identically is what lets the weak check hide.

On legibility, agreed it doesn't generalise, though the screenshot case might collapse into something the gate already does. The check isn't "is this readable", it's "what dimensions came back" — a number you can assert on, so it becomes an expectation the caller writes rather than a capability the gate needs. Doesn't transfer to a log tail or a half-rendered page, which is probably why it stays the caller's job.

[hyuga]

You're right, and it's an example gap rather than a capability gap — the current API already takes that shape. The move is to have the probe return the measurement instead of the artifact:

// weak: the file is there, so this passes — including at 40×30 pixels
await gate({ action: 'capture', probe: () => existsSync(shot), expect: expect.nonEmpty() });

// stronger: the probe returns a number, so the expectation can be about legibility
await gate({ action: 'capture', probe: () => imageSize(shot).width, expect: expect.atLeast(1200) });

Same move for a log tail (probe the line count, not the text) and for a page (probe for the element that only exists after render, not the HTML length). "Was this readable" stays domain knowledge the caller has; "what width came back" is something they can already assert on. I've put that in the README — it should have been there from the start, since the whole design rests on the caller choosing a probe worth comparing against.

Thanks for the two rounds. Both of your comments turned into shipped rules and this one turned into the docs, which is a better hit rate than most of my own ideas manage.

[Alex @Bickov]

Probe returns the measurement, not the artifact. That's a cleaner statement of it than I had. I'll run genchi over the capture step in our skill this week and tell you if anything odd falls out.

[hyuga]

Please do — and if something odd falls out I'd rather hear it than not. The capture case is the one I'm least sure generalises, so a real skill running it is worth more to me than more thinking about it.

[Reid Marlow]

This is exactly the failure mode that scares me with shared agent skills. The format standard gets people to the starting line, but the portable part is really the dependency contract. Absolute paths, hidden CLIs, and magic env vars are the stuff that makes a reusable skill become a local shell script with nicer packaging.

[hyuga]

"Dependency contract" is a better name for it than anything I came up with — stealing that.

The 230-repo audit gave me a rough ranking of which parts of that contract actually break. Absolute paths were the one unambiguous killer, and they really are out there: /Users/guohao/Documents/... and C:/Users/vudrk/Desktop/AI Projects/ sitting in the body of skills people shared. Nothing errors — it just quietly does nothing for the next person.

Hidden CLIs and magic env vars turned out to be harder, which is why they're warnings and not errors. Statically, codex exec (a real undeclared dependency) and claude mcp add (the host you're already running in) look identical, and Bearer YOUR_API_KEY looks exactly like an unfinished placeholder. I was failing CI on all three until the audit talked me out of it.

The part that can be enforced is the declaration itself: name the CLI in frontmatter (requires:) or install it in the body and the linter goes quiet, leave it implicit and you get told. Your comment made me go re-read my own implementation of that, and it only recognised the single-line form — declare two dependencies as a YAML list, the way anyone would actually write it, and it still warned you. So the people writing the contract properly were the ones getting flagged for it. Fixed on my end, going out in the next patch. Thanks for the nudge.

[jacob foster]

Great point. A portable format doesn’t mean portable execution. Hardcoded paths, missing CLIs, API keys, and model assumptions can quietly break skills. carrylint sounds like a practical way to catch these issues before they reach teammates.

[hyuga]

Thanks. Of the four you listed, the measured order surprised me. Across a random sample of 2,465 published skills: hardcoded absolute paths in 3.8%, provider-specific env vars read directly in 2.1%, undeclared external CLIs in 1.3% — those are shares of skills with at least one finding, not shares of findings. The one that dwarfs all of them isn't on your list: references that don't resolve inside the package at all, 18.1%.

Published the write-up today if you want the numbers and the harness.

[Yunetzi]

Love the candor. The real win is a tiny linter that flags non-deterministic steps and missing tests before shipping. Ship small, test hard, learn fast—humility pays.

[hyuga]

Thanks — glad the candid part landed.

Non-determinism is a rule I keep sketching and haven't shipped: I can't find a signal that doesn't fire on every model call, and the audit left me cautious about noisy rules. The testing half I've had more luck with — for a skill it usually turns into a read-back: re-fetch the external state after a write rather than trusting that the step reported success. carrylint only covers the portability side today, so both of those are still open ground.