SkillsBench, the paired-evaluation benchmark for Agent Skills from Xiangyi Li and colleagues (arXiv 2602.12670, first posted in February and now on its fourth revision), has one result that should worry anyone letting an agent write its own tooling docs. Curated skills lifted the average pass rate across 87 tasks and 18 model-harness configurations from 33.9% to 50.5%. Skills the agent generated for itself did the opposite. On the three dedicated-harness setups the paper tested, self-generated skills landed below the no-skills baseline every time: 8.1 points down on Claude Code with Opus 4.7, 11.3 down on Codex with GPT-5.5, 11.5 down on Gemini CLI with Gemini 3.1 Pro. Curated skills on the same three configurations added between 18.2 and 24.8 points.
The trajectory audit in Appendix D.6.1 explains the gap, and the worst mechanism is the interesting one. On a 3D scan task, the generated SKILL.md asserted as a "critical gotcha" that the STL coordinates were in millimetres and hard-coded a thousand-fold unit conversion. Both solver runs executed the bundled script without questioning it and submitted a mass off by three orders of magnitude, with every other step correct. The curated skill for the same task carried exactly the opposite warning. Same file format, same harness, same model. The only difference was whether the person who knew the tool wrote the gotcha or the agent guessed it.
The paper's other design finding points the same way. Compact and standard-length skills gained 19.0 and 21.5 points; "comprehensive documentation" gained 0.7. That is the case for keeping a skill file down to the things the agent would get wrong on its own.
What ours says the agent may not claim
That framing is a useful way to read the skill file we ship with shell.online. It lives in the repo at public/skill/shell-online/SKILL.md, and the Cloudflare Worker serves the same bytes at https://shell.online/skill (the route in worker/index.ts answers GET only and maps it straight to that asset). The frontmatter is the two required fields from the Agent Skills specification, a name that matches the directory and a description that says when to use it. Next to it sits a four-line agents/openai.yaml with a display name and a default prompt for harnesses that read that file. The whole thing is 109 lines. Seven commits touch it, the first being the open-source release on August 22 and the latest being the docs commit that shipped in today's 0.14.1.
A surprising share of those 109 lines are not procedure. They are constraints on what the agent tells the human afterwards:
- After sharing: "Say what process it exposes and whether
read_onlyis true." - If the operator asked for
--no-e2ee, the file spells out that Cloudflare can then read terminal input and output, and adds: "State that boundary clearly." - Wrapping ROS: "Do not claim shell.online makes an otherwise unsupported ROS/OS combination compatible."
- Handing off a Claude Code conversation: "state that it is a fork with the same conversation history and workspace" and "Never claim that shell.online adopted the original PID or PTY."
- After rotating a session password: "say that previous credentials no longer decrypt later frames, and note that teammates must be shared the rotated password again."
Each of those is a millimetre gotcha in reverse. Take the Claude Code one. The code behind it is short: claude_handoff.go reads CLAUDE_CODE_SESSION_ID from the environment and launches claude --resume <id> --fork-session inside the shared PTY. An agent that has just run shell --json -- claude sees a link to a Claude session with its own history in it, and the natural thing to report is "I shared this session." That report is wrong in a way the operator will discover ten minutes later when messages stop lining up between the two windows. The file exists to pre-empt that sentence.
What today's release added to it
The 0.14.1 docs commit added a section the earlier versions did not need, because the capability did not exist. File access is now a separate host opt-in (--files for the process working directory, --files-root for a chosen directory) carried over the existing end-to-end encrypted WebSocket, and the skill file's instructions for it are again mostly about disclosure and refusal: enable it only when the operator explicitly asks, "say which root is being shared," and never combine it with --no-e2ee. The same section states that the alpha Refstream renderer turning filename-like output into previews does not grant filesystem access by itself, and that its agent invitation is a separate, revocable, single-session grant with read or control scope. Anyone can diff that change in the shell.online repository; it is the commit titled "Complete v0.14 docs and dependency updates."
The file is also an attack surface, so keep it readable
There is a second reason to keep a skill file short and literal. Snyk's ToxicSkills study from February looked at 3,984 published skills and found 36.82% with at least one security issue and 76 with confirmed malicious payloads, 91% of which also used prompt injection inside the markdown to prime the agent into accepting the code. A SKILL.md is an instruction channel that the harness loads on the agent's behalf. The defence available to a maintainer is boring: every command in ours is a shell subcommand on the local machine, the install line requires the installer to report a verified SHA-256 digest, and the whole file fits on two screens for whoever reviews it before enabling it.
That review question is going to get bigger, not smaller. An agent-facing catalogue like the Pilot Protocol App Store is, at bottom, a directory of tool descriptions that agents read and act on without a human in the loop for each call. The SkillsBench audit is a reminder of what those descriptions are for. Explaining the tool is the smaller job. The larger one is carrying the handful of facts the agent would otherwise assert with confidence and get wrong, and those facts have to come from the people who built the thing.
Top comments (0)