DEV Community

Dawncoral
Dawncoral

Posted on Originally published at wallabytoken.com

Your skill file is too big: the 6KB line, measured

Skill files — the markdown instruction files coding agents load at runtime, whether you call them SKILL.md, AGENTS.md, or rules — have a size limit nobody talks about. Keep yours under 6KB. That is the finding; everything below is the evidence.

We got here by accident first. During an unrelated experiment, a 9,211-byte skill file, written in one pass by the model itself the way most people write these files, made our weaker model return empty responses six times in a row inside the test driver. We reran it with the output budget raised to 16,000 tokens. Still one empty out of two. The file was not buggy. It was just long, and the model was spending its entire output budget thinking about it.

So we stopped treating it as an anecdote and measured it.

This is part 3 of a series, and it stands alone if you only want the number. Part 1 is the kill rule that fired; part 2 is the map of where skills help, do nothing, and backfire.

The experiment: one variable, 28 executions

The setup was deliberately boring. Same task throughout: extract an invoice batch, with three private conventions buried in the skill file (prices in cents that must convert to dollars, a 6% channel fee, a tax-exempt line item listed separately). Same system prompt. max_tokens pinned at 16,000. No retry on empty outputs, because retries hide the raw failure rate we wanted to see.

The only thing that changed was file size. We took a compact 2,761-byte skill and padded it toward target sizes using 54 real bloat blocks harvested from that naturally bloated 9.2KB file: repeated sections, restated rules, ceremonial boilerplate that models actually produce. No random filler. Seven size points, four repetitions each: 28 executions. Scoring was deterministic regex against the three conventions, no judge model involved.

We also wrote down the falsification line before running: if empty-output rate and token consumption do not rise with size, the 9.2KB incident was a one-file coincidence.

One unit note, because reviewers will ask: we measure the file in bytes because that is what you see in your editor. At roughly four characters per token of English, 6KB is about 1,500 tokens of instruction text. The file rides in the prompt as input, but its length inflates the model's output-side thinking, and that thinking is what hits the 16,000 cap. The input side is not free either: an agent loop re-reads the whole conversation on every iteration, so every kilobyte that stays resident charges rent on every request. We audited one production pipeline whose daily spend grew from 110 million to over 500 million tokens with no change in workload, and the skill catalog alone had quietly reached 161KB of every system prompt.

What happened

Variant Bytes Empty-output rate Mean output tokens Conventions hit
size-2.8K 2,761 0% 9,096 4/4
size-4.1K 4,097 0% 7,913 4/4
size-6.1K 6,101 25% 13,598 3/4
size-8.0K 8,037 25% 11,769 2/4
size-9.5K 9,503 25% 15,469 3/4
size-11.2K 11,187 0% 9,927 4/4
natural-9.2K 9,211 50% 14,766 2/4

Empty-output rate and mean output tokens by skill file size, with the 16,000-token output cap marked

Two regions are visible to the naked eye. Below 6KB, the model consumed 8–9K tokens per run, comfortably far from the cap, and never returned empty. At 6KB and above, consumption got pushed into the 12K–15.5K neighborhood of the 16,000 ceiling, and a quarter of the runs died there. The 9.5K point averaged 15,469 tokens, kissing the cap, and two of its four runs survived only because the thinking chain happened to come in shorter. Whether you crash at the edge is decided by variance in reasoning length, not by anything you control.

The rules, stated precisely

The tempting summary is "bigger file, more deaths." The data says something more specific:

  1. Under 6KB is a safe zone. Eight runs at 2.8K and 4.1K, zero empty outputs, consumption far from the cap.
  2. At 6KB and above you are in a risk zone. Burnout is a function of how close your consumption rides to the output cap, not a monotone function of file size. Near the cap, survival is luck.
  3. The 11.2K point breaks monotonicity. Zero empties, consumption back down to 9,927. That variant contained all 54 blocks, including the natural file's own structured sections. Our working hypothesis is that organized content partially compensates for raw size. Unverified; we flag it because deleting the counterexample would make the chart prettier and the claim wrong.
  4. Organic bloat is the most poisonous kind. Same size class, different file: the naturally bloated 9.2KB file died at 50% where the hand-padded 9.5KB died at 25%. Repetition and rambling hurt more than bytes. This also retroactively explains the original incident: six consecutive empties was not bad luck, it was a 50% failure rate behaving normally.
  5. Size kills production, not correctness. When the model did produce output, convention hit rates stayed near full at every size point. The question a long file answers first is not "will the model follow it" but "will the model answer at all."

What to do with this

Three rules we now apply to our own skill files, AGENTS.md, and long system prompts:

Budget under 6KB. Not as a style guideline; as an executability zone. Our generation pipeline now treats 6–9KB as risk territory (25% observed failure) and 9KB+ as high risk, and the audit step fails files that drift over.

Delete repetition before deleting content. The natural file's problem was not that it said too much; it said everything twice. Deduplication recovered more safety than trimming rules did in our padding exercise.

Budget tighter for weaker models. These runs used DeepSeek V4.1-Flash as the execution model, and weaker models are exactly where the cliff matters: stronger models burn budget more gracefully, and the same file can hurt one model and not the other. The full strength-dependency map is in the companion piece.

Honest limits

One task (invoice extraction), one model family, four repetitions per point, so the 95% confidence interval on "25%" is wide, and padded-via-algorithm is not identical to bloated-by-handwriting. The 11.2K counterexample deserves a direct test we have not run yet: hold byte count fixed, vary structure, organized filler against rambling filler. The gradient round cost ¥2.55 in API fees, about $0.36. Replicate it before you tattoo 6KB on anyone; treat it as a measured starting point with a clear mechanism, not a universal constant. But "the model returned nothing because it spent its whole budget reading your instructions" is a failure mode worth checking today, with the file sizes you already have. wc -c is the whole diagnostic.

The series, in order

  1. We pre-registered a kill rule for our own product. It fired.: the program this gradient round belongs to
  2. Same skill, opposite effects: when structure helps, when it does nothing, when it backfires
  3. The 6KB line (you just read it)

Beyond the series: AGENTS.md as a token budget, the forbidden-list playbook for the file you already maintain; Same prompt, four thinking budgets, four bills, the metered version of what output-side thinking costs.

The tool behind this series is open source: Skill Forge ships as an MCP server, a prompt pack, and an installable skill, under MIT + Commons Clause. Design deep-dive right here on dev.to: We open-sourced the tool that failed its own kill rule.

Disclosure: Wallaby Token (that's us) sells API access to open-weight models. This experiment ran on DeepSeek V4.1-Flash, and nothing in it depends on where your API key comes from.

Top comments (11)

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

@dawncoral Not among what each run records: the token breakdown, turns and tool calls are, so a fixed-overhead term would have to be fitted from those runs. The 38.9% already carries the overhead, since it's measured end to end.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

@dawncoral Thanks, the end-to-end point is the one to keep. The THOL figure is a whole-session number, overhead included, so nothing needs to be subtracted to get it.

Collapse
 
dawncoral profile image
Dawncoral •

agreed, I'll cite it as a whole-session number

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

@dawncoral Bill, not window. Our number is session cost in USD, end to end: 39% lower on average on long sessions (200k+ tokens, Claude Code), raw data public: pi-infected.github.io/token-harnes.... Agreed on the split: output exhaustion is the one that loses the answer.

Collapse
 
dawncoral profile image
Dawncoral •

Went through the leaderboard before replying. End-to-end USD, task success as the gate, raw runs in sqlite — that's exactly the measurement I was asking for, and it's public. Thanks for building it.
Worth noting: your finding #3 is the 6KB claim, worded better than we did it — overhead paid on every turn, cache-read billed at a reduced rate but still billed. Your table doesn't contradict the rent line; it's the strongest public evidence for it. Most tools cluster near zero because standing overhead plus failed adoption eats the saving. tokenade's +38.9% is a net number: savings left over after paying its own rent.
And agreed back on the split — money is recoverable, a truncated answer isn't.
One question your data might settle: does results.sqlite split fixed prompt overhead from per-call savings? If it does, the rent per resident kilobyte is computable straight from your runs. That would answer the 6KB question better than either of our posts.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

A skill's body stays in context for the rest of the session, so a 6KB line is paid again with each turn, not once. Did you check the bill, or only the window?

Collapse
 
dawncoral profile image
Dawncoral •

Thanks for checking.
I’d hoped results.sqlite already broke out fixed overhead, but fitting it from the token breakdown, turns and tool calls makes sense. Your end-to-end measurement is the key detail: the 38.9% already includes the overhead.

Collapse
 
dawncoral profile image
Dawncoral •

You're right about the mechanism, and the post says the same thing: an agent loop re-reads the whole conversation on every iteration, so every kilobyte that stays resident charges rent on every request. Where prompt caching applies, that rent bills at cache-read rates. Rent either way.
On the bill: yes, we checked it. The companion piece is a bill-level audit of one of our production pipelines: daily spend grew from 110M to over 500M tokens with no change in workload, and the skill catalog alone had quietly reached 161KB of every system prompt (wallabytoken.com/blog/p/agent-cont...). The gradient round behind the 6KB number cost ¥2.55, about $0.36, in metered API fees.
One distinction worth keeping: the 6KB line is about the output side, not the input rent. A long file inflates the model's thinking chain until it rides the 16,000-token output cap, and at 6KB and above a quarter of our runs returned nothing at all. Input rent costs you money. Output exhaustion costs you the answer.
If you've measured the per-turn cost of a resident skill file in your own harness, I'd like to see the numbers. The 11.2K point in our table already broke one of our own hypotheses, so more data is welcome.

Collapse
 
promptalo profile image
PromptAlo •

The 11.2K row is the one I keep looking at: 0% empty and 4/4 on the conventions, better than every size from 6.1K to 9.5K. With four runs per size, 25% is a single empty response, so the curve may depend on which bloat blocks landed in each variant as much as on byte count. Padding two files to the same size, one with restated rules and one with neutral text, would show whether length or repetition is what eats the output budget.

Collapse
 
dawncoral profile image
Dawncoral •

Fair on the counts: the three middle rows are one empty each, so 6.1K and 9.5K aren't distinguishable at all. Pooled across the padded arms, that's three empties in sixteen runs at or above 6KB and none in eight below.

Direction holds, sharpness doesn't. The limits section says the same in politer language.

The block mix is the confound I'd concede without argument. Each arm packed a different subset of the 54 blocks into the 2.7KB skeleton, so bytes and block identity moved together.

What I'd push back on is the variable. The arm carrying all 54 blocks, repeated rules included, is the one that didn't die: 11.2KB, 4/4, mean 9,927 output tokens. The natural 9.2KB file those blocks were cut from, same size class, went 2/4 at 14,766.

Same 54 blocks, opposite result.

So hold size fixed across three cells, not two: restated rules, neutral prose, ordered sections. Two cells only split length from repetition, and repetition is the suspect our own table argues against.

Sixteen executions, about 21 cents at the rates in the piece. If it reproduces, the number gets stronger. If it doesn't, I'll publish that.

Collapse
 
promptalo profile image
PromptAlo •

Three cells is the better design, since two would only test the variable your own table already argues against. The same 54 blocks giving opposite results points at arrangement more than content, so the ordered sections cell is the one I'd most like to see. At four runs a cell, one empty response is still 25 points, so keeping the task prompt identical everywhere and showing each cell's empties would stop a single run from deciding it. At about 21 cents, it's a cheap way to settle it.