DEV Community

Your skill file is too big: the 6KB line, measured

Dawncoral on October 18, 2026

Skill files — the markdown instruction files coding agents load at runtime, whether you call them SKILL.md, AGENTS.md, or rules — have a size limit...
Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

@dawncoral Not among what each run records: the token breakdown, turns and tool calls are, so a fixed-overhead term would have to be fitted from those runs. The 38.9% already carries the overhead, since it's measured end to end.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

@dawncoral Thanks, the end-to-end point is the one to keep. The THOL figure is a whole-session number, overhead included, so nothing needs to be subtracted to get it.

Collapse
 
dawncoral profile image
Dawncoral •

agreed, I'll cite it as a whole-session number

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

@dawncoral Bill, not window. Our number is session cost in USD, end to end: 39% lower on average on long sessions (200k+ tokens, Claude Code), raw data public: pi-infected.github.io/token-harnes.... Agreed on the split: output exhaustion is the one that loses the answer.

Collapse
 
dawncoral profile image
Dawncoral •

Went through the leaderboard before replying. End-to-end USD, task success as the gate, raw runs in sqlite — that's exactly the measurement I was asking for, and it's public. Thanks for building it.
Worth noting: your finding #3 is the 6KB claim, worded better than we did it — overhead paid on every turn, cache-read billed at a reduced rate but still billed. Your table doesn't contradict the rent line; it's the strongest public evidence for it. Most tools cluster near zero because standing overhead plus failed adoption eats the saving. tokenade's +38.9% is a net number: savings left over after paying its own rent.
And agreed back on the split — money is recoverable, a truncated answer isn't.
One question your data might settle: does results.sqlite split fixed prompt overhead from per-call savings? If it does, the rent per resident kilobyte is computable straight from your runs. That would answer the 6KB question better than either of our posts.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

A skill's body stays in context for the rest of the session, so a 6KB line is paid again with each turn, not once. Did you check the bill, or only the window?

Collapse
 
dawncoral profile image
Dawncoral •

Thanks for checking.
I’d hoped results.sqlite already broke out fixed overhead, but fitting it from the token breakdown, turns and tool calls makes sense. Your end-to-end measurement is the key detail: the 38.9% already includes the overhead.

Collapse
 
dawncoral profile image
Dawncoral •

You're right about the mechanism, and the post says the same thing: an agent loop re-reads the whole conversation on every iteration, so every kilobyte that stays resident charges rent on every request. Where prompt caching applies, that rent bills at cache-read rates. Rent either way.
On the bill: yes, we checked it. The companion piece is a bill-level audit of one of our production pipelines: daily spend grew from 110M to over 500M tokens with no change in workload, and the skill catalog alone had quietly reached 161KB of every system prompt (wallabytoken.com/blog/p/agent-cont...). The gradient round behind the 6KB number cost ¥2.55, about $0.36, in metered API fees.
One distinction worth keeping: the 6KB line is about the output side, not the input rent. A long file inflates the model's thinking chain until it rides the 16,000-token output cap, and at 6KB and above a quarter of our runs returned nothing at all. Input rent costs you money. Output exhaustion costs you the answer.
If you've measured the per-turn cost of a resident skill file in your own harness, I'd like to see the numbers. The 11.2K point in our table already broke one of our own hypotheses, so more data is welcome.

Collapse
 
promptalo profile image
PromptAlo •

The 11.2K row is the one I keep looking at: 0% empty and 4/4 on the conventions, better than every size from 6.1K to 9.5K. With four runs per size, 25% is a single empty response, so the curve may depend on which bloat blocks landed in each variant as much as on byte count. Padding two files to the same size, one with restated rules and one with neutral text, would show whether length or repetition is what eats the output budget.

Collapse
 
dawncoral profile image
Dawncoral •

Fair on the counts: the three middle rows are one empty each, so 6.1K and 9.5K aren't distinguishable at all. Pooled across the padded arms, that's three empties in sixteen runs at or above 6KB and none in eight below.

Direction holds, sharpness doesn't. The limits section says the same in politer language.

The block mix is the confound I'd concede without argument. Each arm packed a different subset of the 54 blocks into the 2.7KB skeleton, so bytes and block identity moved together.

What I'd push back on is the variable. The arm carrying all 54 blocks, repeated rules included, is the one that didn't die: 11.2KB, 4/4, mean 9,927 output tokens. The natural 9.2KB file those blocks were cut from, same size class, went 2/4 at 14,766.

Same 54 blocks, opposite result.

So hold size fixed across three cells, not two: restated rules, neutral prose, ordered sections. Two cells only split length from repetition, and repetition is the suspect our own table argues against.

Sixteen executions, about 21 cents at the rates in the piece. If it reproduces, the number gets stronger. If it doesn't, I'll publish that.

Collapse
 
promptalo profile image
PromptAlo •

Three cells is the better design, since two would only test the variable your own table already argues against. The same 54 blocks giving opposite results points at arrangement more than content, so the ordered sections cell is the one I'd most like to see. At four runs a cell, one empty response is still 25 points, so keeping the task prompt identical everywhere and showing each cell's empties would stop a single run from deciding it. At about 21 cents, it's a cheap way to settle it.