Removing two edge-case examples from a system prompt cost me more than the token savings were worth — here's the exact breakdown.
I run an ad analytics SaaS on Cloudflare Workers. One skill had a 4,200-token system prompt: persona, output format, three example pairs, edge-case guidance. At $3/M input tokens with ~800 calls/day, that's $10/day just in system prompt tokens. I assumed prompt caching would soften the blow. Then I checked the logs:
[2025-06-08 03:14:22] INFO cache_creation_input_tokens: 4218
[2025-06-08 03:14:22] INFO cache_read_input_tokens: 0
[2025-06-08 03:14:23] INFO cache_creation_input_tokens: 4218
[2025-06-08 03:14:23] INFO cache_read_input_tokens: 0
Cache hit rate: zero. Workers spins up a fresh instance per request, so the TTL window almost never catches a repeat. The "caching makes long prompts free" assumption doesn't hold in this infrastructure.
So I cut aggressively — down to 820 tokens. Dropped all three example pairs, both edge-case blocks, compressed the persona to two sentences. Cost dropped from $10/day to under $2. Looked like a win for 72 hours.
Then the report quality collapsed. Numeric formatting became inconsistent across outputs. Worse: on inputs that previously returned "analysis unavailable," the model started confidently producing wrong conclusions. One bad insight nearly shipped to a client report.
I ran a recovery experiment — re-adding each removed section one at a time and grading outputs. The results were pretty clear-cut:
| Removed section | Tokens saved | Quality impact |
|---|---|---|
| Persona compression | ~400 | Negligible |
| Format example: 1 pair → 0 | ~600 | High — format consistency broke |
| Edge-case guidance (2 cases) | ~500 | Very high — wrong reasoning increased |
| Prohibited behaviors list | ~300 | Low |
Persona and scope descriptions are trimmable. Format examples and edge-case handlers are not — remove them and the model fills the gap with its own judgment, which isn't always the judgment you want.
I now run at ~1,800 tokens. That's not a universal number; a simple classification task might be fine at 800, and a multi-step reasoning pipeline might justify 2,500. The only real calibration method is cutting, then watching actual outputs for 3–5 days before declaring victory.
I wrote up the full breakdown — including the exact 5-section prompt structure I settled on, and the conditions where caching does make long prompts defensible — over on riversealab.com.
Top comments (0)