I asked an LLM to cut 14 texts by 20–30%. Median: only 7%

I run a small site whose service pages are 1,100–1,800 Japanese characters each. I wanted them shorter, so I asked a model to cut them by 20–30% — and then, because the pages contain prices, limits and exceptions, I added "keep the numbers and the sentences that state conditions". My assumption was that the "keep" clause is what stops the model from reaching the target.

So I measured it instead of assuming. Fourteen documents, four instruction conditions, 56 outputs, each scored three ways.

Setup (frozen before generation)

• Material: 14 pages from my own site, 1,149–1,797 characters after stripping front-matter, HTML tags, the closing CTA and whitespace. SHA-256 of each file frozen.
• Protected atoms: every number + unit string in the source (154 of them: 「33,000 円」, 「3 つ」, 「1 営業日」…), as a multiset.
• Boundary claims: 29 sentences that state a condition, a limit or an exception (hand-picked from candidates, frozen).
• Conditions (nested, verbatim in the kit):
  • 0 — "Shorten the following text by 20–30%. Output only the shortened text."
  • A — 0 + "Do not change the structure of headings, paragraphs and lists."
  • B — A + "Keep the numbers and the sentences that state conditions, limits and exceptions."
  • C — B + "Lists may be converted to bullet points."
• Model: claude -p --model claude-opus-5, text on stdin, run from an empty directory. Note that claude -p still carries Claude Code's own system prompt, so this is not a bare API call.
• Hypotheses: H1 B compresses less than A. H2 A reaches −20% but drops an atom or a claim in more than half of the documents. H3 C compresses more than B.
• HOLD threshold: I first ran 3 documents × condition A × 3 times. The same input landed anywhere from −3.5% to −20.8%; the median spread was 6.6 points. Any difference between conditions smaller than that is "not distinguishable".

Condition 0 was added after an external review pointed out that A–C all contain the "keep the structure" clause, so the main result could not be separated from it. The other 42 outputs were generated in randomized condition order (seed 20260919), one run per cell.

Scoring

• Character change (whitespace removed).
• Atoms: an atom is "lost" when it appears fewer times in the output than in the source. This is a string check — 「3 つの確認」 ("three checks") rendered as a three-item list still counts as lost. No meaning adjudication.
• Claims: normalized-string containment for automatic pass; everything else was shown to a human with the closest output sentence (difflib) and judged paraphrased-kept / lost / inverted.
• Structure: heading count and bullet count.

ATOM = re.compile(r'([\d,]+(?:\.\d+)?)\s*(円|件|本|日|時間|営業日|%|％|項目|段階|か所|回|人|社|通|枚|行|字|ミリ秒|秒|分|か月|つ|層|型|種|点|倍)')
lost = {k: c - out.get(k, 0) for k, c in src.items() if out.get(k, 0) < c}

Results

Condition	n	Median change	Quartiles	Min / max	Reached −20%	Docs that lost ≥1 atom string	Claims kept
0 no constraint	14	−6.5%	−7.5 … −5.1	−9.6 / −4.0	0	7	28/29
A keep structure	14	−6.9%	−8.2 … −5.4	−11.0 / −2.7	0	8	28/29
B + keep numbers & conditions	14	−8.2%	−10.3 … −6.5	−21.7 / −2.4	2	5	28/29
C + bullets allowed	14	−8.4%	−10.8 … −6.6	−21.1 / −3.7	1	7	28/29

Quartiles are the 4th and 11th of 14 sorted values; statistics.quantiles moves them by about half a point.

• Differences between conditions: A−0 = 0.4, B−A = 1.3, C−B = 0.2 points. All inside the 6.6-point reproduction width. H1 and H3: HOLD.
• H2 failed at its premise: A never reached −20% (0/14), and neither did the unconstrained condition 0 (0/14). The second half happened anyway — 9 of 14 A-outputs lost an atom string or a claim.
• The three outputs that did reach −20% were all B or C. The "keep" clause did not prevent compression either.
• What disappeared: counts — 「3 つ」, 「5 つ」, 「1 通」, 「3 か月」, 「10 件」. Every price (18 occurrences of 「…円」 across 6 documents × 3 conditions) survived.
• Boundary claims, 29 × 4 = 116: 74 exact matches, 38 judged paraphrased-kept, 4 lost, 0 inverted. The 4 lost are the same sentence in all four conditions: 「合否条件を通した検証の結果と、確かめていない範囲も一緒に渡します」 ("we hand over the results of verification that passed the acceptance criteria, together with what was not checked") became 「検証の結果と…」 every time — the acceptance-criteria wording was dropped whether or not the instruction said to keep such sentences. (Its condition-0 row was first batch-judged as a paraphrase; a second review aligned it with A–C. The change is recorded in the published sheet.)
• Heading and bullet counts were identical to the source in 56/56 outputs, including condition 0 where structure was never mentioned. Condition C's permission to convert lists into bullets was never used.

What I can and cannot say

Can: under these run conditions, a 20–30% request stopped at a median of −6.5 to −8.4% with or without constraints, and the constraint I suspected did not explain the shortfall. All 18 price strings survived while some count strings disappeared; structure was kept without being asked; and the acceptance-criteria phrase was cut from the same sentence in every condition.

Cannot: what actually sets the ceiling. Other models, other temperatures, a single-point target ("exactly −25%"), documents over 3,000 characters, or a second run of the same cell (n = 1 each; the pilot shows −21% and −7% for the same input).

The acceptance sheet I use now

Column	How to write it
Compression	character change (whitespace removed) ≤ −X%
Protected atoms	every number + unit string appears at least as often as in the source (string check only)
Boundary claims	every condition / limit / exception sentence survives, paraphrase allowed

"Shorter" is a proxy. What I wanted to know was whether the numbers and the conditions survived. Counting the two separately showed both things at once: the target was not reached, and things were still being dropped on the way.

---

Part 2 of the series "Measure the Outcome, Not the Proxy" (part 1: transfer size ≠ speed; part 3: screenshot success ≠ threshold discrimination — both on this account).

Verification record (environment, verdict, evidence links) and the canonical write-up in Japanese. Reproduction kit — the 14 source texts, the three scripts, all 56 outputs and the two adjudication sheets (code MIT, data CC BY 4.0). I audit and verify AI-generated and outsourced work with pre-registered pass/fail criteria like these — text-only, no calls: Sumitsuke / verification.