DEV Community: Jay Pokale

Caveman vs Ponytail vs Chisle: I benchmarked the Claude Code token-saving plugins on 20 tasks

Jay Pokale — Tue, 06 Oct 2026 20:14:43 +0000

If you use Claude Code, you've probably seen the two popular plugins that promise to cut your token bill: caveman, which makes Claude talk like a caveman, and ponytail, which pushes it to write less code. I built a third one, Chisle, and benchmarked all three against Claude with no plugin at all: 20 live tasks, 59+ model runs on Haiku and Sonnet, with the arms differing only in the injected ruleset.

TL;DR: caveman cut the bill to 80%, ponytail to 68%, and Chisle to 52%, nearly half. Chisle also had the smallest worst day (173% vs 424% and 227%) and backfired once in 20 tasks, where caveman backfired 6 times and ponytail 8.

20 live tasks, billed output vs no plugin	total bill	average task	worst case	backfires
no plugin (baseline)	100%	100%
caveman	80%	98%	424%	6 / 20
ponytail	68%	91%	227%	8 / 20
Chisle	52%	69%	173%	1 / 20

In the July re-verification run, every answer from every arm was graded correct. None of these tools buys its savings with wrong answers. Every number comes from committed raw transcripts in the Chisle repo.

Where the savings come from: coding vs explanation

	tasks	caveman	ponytail	Chisle
coding (wants working code)	12	74%	59%	44%
explanation (wants prose)	8	103%	104%	87%

On coding, Chisle bills 44% of a bare model, a third less than ponytail, the closest thing to a dedicated "lazy code" tool. On explanation prompts both specialists go above 100%: tools built to write less made Claude write more than using nothing. Chisle is the only one that stays under.

Split by answer length, the gap widens: on long answers Chisle bills 45%, caveman 79%, ponytail 59%.

Caveman alternative: great prose compressor, no engineering judgment

caveman is genuinely good at compressing prose, and on some short prose prompts it's a hair leaner than Chisle. But it has no judgment about what to build. Asked to "add caching", it produced three implementations (330 tokens). Chisle gave one @cache decorator and a one-line upgrade path (151 tokens). Its worst day cost 4.2× a bare model.

Ponytail alternative: right instinct on code, padded prose

ponytail has the right instinct: smallest thing that works. But it pads prose so much that it backfires: on a "retry logic" prompt it ran 227% of baseline. A tool whose whole job is writing less wrote more than twice as much.

Installing both to cover both axes doesn't fix it either: they fight over prose style and double per-session overhead. On one task the pair did worse (605 tokens) than Chisle alone (595).

Chisle compresses three things; caveman and ponytail compress one each

	prose	code judgment	input / context	publishes failures
caveman	✅	❌	❌	❌
ponytail	❌	✅	❌	❌
Chisle	✅	✅	✅	✅

Output prose: no filler, no hedging, no manufactured structure.
Output code: a YAGNI "efficiency ladder" (use what exists, ask instead of guessing, skip speculative abstractions). Safety, error handling, validation and accessibility are never cut.
Input context, the axis neither rival touches. Across 171 real Claude Code sessions, tool output was 67.5% of the context window, re-billed on every later request. Chisle's PostToolUse hook trims oversized tool output by ~46% before it re-enters context (Read/Edit/Write are never touched).

Same prompt, same model

"Add debounce to a search input that currently fires an API call on every keystroke." Verbatim committed output:

No plugin: 142 lines, 1,506 tokens. A generic useDebounce<T> hook in its own file, then Option 2, Option 3, a comparison table and caveats.
Chisle: 35 lines, 602 tokens. setTimeout in the effect you already have, two lines on why, and "use lodash.debounce if it's already installed."

useEffect(() => {
  const timer = setTimeout(async () => {
    if (query.trim()) { /* fetch */ }
  }, 300);
  return () => clearTimeout(timer);
}, [query]);

Not golfed, just boring: one less file, one less abstraction, same behaviour.

The fine print (because the repo publishes it)

The 20-task total includes one cache task where the bare model wrote an unusually long answer (4,910 tokens, against 375–813 in later runs), and a few cells that an older ruleset example may have primed. Without those cells, Chisle's 20-task total is 70%.
A later, smaller rerun (13 prompts × 2 seeds on the newest plugin versions) again put Chisle lowest: 83%, vs caveman 102% and ponytail 105%. It was the only one of the three below a bare model.
On short answers, every tool breaks even or worse; there's little to cut in a three-line reply.

Chisle is the only tool in this class that publishes the runs where it lost: benchmarks/results.

Install Chisle in Claude Code

npx chisle             # installs for Claude Code and any other agents it finds
npx chisle --dry-run   # preview first

Zero dependencies, MIT licensed. The same ruleset ships to Cursor, Codex, Gemini CLI, GitHub Copilot, Windsurf, Cline, OpenCode, Kiro, Antigravity, Hermes and Pi. Claude Code and Pi also get the input-side compressor and live modes (lite, full, ultra). /chisle-audit flags over-engineered code and bloated prose/docs in one ranked report; ponytail's audit is code-only and caveman has none.

FAQ

Is Chisle better than caveman?
Across 20 tasks: 52% vs 80% of the bare-model bill, worst case 173% vs 424%, 1 vs 6 backfires. On coding prompts 44% vs 74%. caveman is a little leaner on some short prose prompts.

Is Chisle better than ponytail?
Across 20 tasks: 52% vs 68%, worst case 173% vs 227%, 1 vs 8 backfires. On explanation prompts ponytail goes above 100% (104%); Chisle stays at 87%.

Can I install caveman and ponytail together instead?
You can, but they fight over prose style and double the overhead. On one task the pair did worse (605 tokens) than Chisle alone (595).

Does it make answers wrong?
In the July re-verification every answer from every arm graded correct. The rule is necessary, not fewest characters, and safety is never cut.

Where's the raw data?
All transcripts are committed: benchmarks/results. Full tables: docs/benchmarks.md.

JayPokale / Chisle

Cut your AI coding agent's token bill on three axes: terse prose, YAGNI-first code, and tool-output compression. Claude Code, Pi, Cursor, Codex, Gemini + 4 more. Zero deps, published benchmarks including the runs it loses.

Chisle

Your AI talks less, builds less, reads less, and says more. Like a senior dev who bills by the syllable.

The only tool in this class that publishes the runs where it lost. Here's why.

Built for Claude Code: coding answers come back 33% shorter and 24% cheaper, while caveman and ponytail make them longer · 12 agents · zero dependencies · one command

Chisle is a Claude Code plugin that makes Claude cheaper to run without making it dumber. It cuts what Claude writes: no filler, no hedging, no speculative abstractions, just the smallest code that works. It also cuts what Claude reads: a PostToolUse hook trims oversized tool output by ~46% before it re-enters the context window, where it would be re-billed on every later request. In agent-loop tests Claude with Chisle passes the same tasks as Claude without it. One npx chisle installs it, and the…

View on GitHub

Site: chisle.jaypokale.me. If you run your own comparison against caveman or ponytail, I'd like to see it, especially where Chisle loses.

My speech-flaw detector flagged 41 false alarms a minute. One line of math fixed it.

Jay Pokale — Tue, 06 Oct 2026 20:04:50 +0000

I built Podium, a tool that compares your reading of a speech with a great delivery of the same text (JFK, Reagan) and tells you exactly where you drift and why: "14.3 syllables/s here vs 7.7 in the reference (+85%)", pinned to a time range.

The first version worked beautifully on my test set. Then I gave it a different speaker, and it flagged 41 "flaws" per minute on a perfectly clean recording.

This post covers why that happened, the one-line fix, and the evaluation set-up that caught it.

Step 1: put both recordings on the same word grid

Comparing two deliveries frame by frame is hopeless: they never line up. Comparing them word by word is easy, as long as both are force-aligned to the same transcript. I used torchaudio's MMS_FA (a wav2vec2 CTC aligner):

bundle = torchaudio.pipelines.MMS_FA
model, tokenizer, aligner = bundle.get_model(with_star=False), bundle.get_tokenizer(), bundle.get_aligner()
emission, _ = model(wav)
spans = aligner(emission[0], tokenizer(words))   # one span list per word

Now word 17 of your reading is word 17 of JFK's. For every word I compute speaker-normalised features with Praat (via Parselmouth) and librosa:

pitch in semitones relative to your own median (a deep and a high voice with the same intonation give the same curve);
loudness in dB relative to your own level;
articulation rate, syllables per second of actual speech, pauses excluded, over a 5-word window;
the pause before the word, and voiced sound inside that pause (more on that below);
high-frequency energy above 2.5 kHz, which drops when consonants get swallowed.

Step 2: build a dataset where the answers are exact

There's no public dataset that pairs a good delivery with bad deliveries of the same words. So I made one: 207 recordings, starting from three public-domain speeches.

The trick is that I inject the flaws myself, so every label is sample-accurate:

Flaw	How it's injected	Severity 1 / 2 / 3
rushed / dragged	phase-vocoder time-scale of 4–9 words	×1.3/1.6/2.0 · ×0.8/0.65/0.5
monotone	WORLD vocoder, pitch contour squashed toward its mean	50/75/95% removed
mumbled	gain down + low-pass	−6 dB @ 3 kHz … −16 dB @ 1.1 kHz
awkward pause	room-tone silence inserted mid-phrase	0.7 / 1.3 / 2.2 s
missing breath	a natural pause squeezed out	50 / 20 / 0% kept
stutter	word onset repeated	1 / 2 / 3 repeats

Edits are joined with 5 ms fades that preserve length, so the label times never drift. My first version used 15 ms crossfades, which quietly shifted every later label by 15 ms per edit.

I also had the same texts read by open Piper TTS voices, with flaws injected into those too. That's the "different speaker" test.

And I split it honestly: thresholds are tuned only on JFK, then frozen and tested on Reagan plus an unseen voice.

Step 3: the bug that wasn't a bug

Version 1 scored each word by how far it departed from the reference, as a robust z-score (median/MAD) after removing your overall offset. On Reagan's own audio with injected flaws it got F1 0.64 with zero false alarms.

On a TTS voice reading the same text:

	F1	False alarms on clean audio
Same speaker	0.64	0 / min
Different speaker	0.03	41 / min

A synthetic voice doesn't breathe where Reagan breathes or lift the words he lifts. Measured against Reagan, everything it does is a deviation. Technically correct, but useless as feedback: nobody wants to hear that every sentence is wrong because they aren't Reagan.

The fix was to separate style from flaws. A flaw is something that's off compared with the reference and off compared with your own delivery. So every word gets two scores:

z_ref  = robust_z(participant - reference)   # departs from the reference, beyond your overall style
z_self = robust_z(participant_feature)       # stands out within your own recording
flaw   = np.fmin(z_ref, z_self)              # a soft AND

A consistently different voice moves z_ref everywhere but z_self almost nowhere, so it's reported once as style ("33% faster and flatter than the reference") instead of 40 times as flaws.

False alarms on clean different-speaker recordings went from 41/min to 3/min.

Step 4: a stutter that looked like a pause

Injected stutters kept being reported as "awkward pause". The cause: the aligner usually places the word at its last restart, so "w- w- we" becomes a long gap followed by a normal "we".

The fix was one feature: seconds of voiced audio inside the gap. A real pause is silent; a gap full of voiced sound is a restart.

gap_voiced = np.sum(~np.isnan(f0[prev_end:word_start])) * HOP

Long pauses now require gap_voiced < 0.15 s, and stutters can trigger on gap_voiced > 0.12 s. Stutter recall went from 0.11 to 0.44 on the held-out set.

Results on the held-out set (Reagan + an unseen voice)

detected regions overlap the true flaw with mean IoU 0.81; overall F1 0.51;
awkward pauses, missing breaths, mumbling and stutters are located within 0.00–0.06 s;
0 false alarms per minute on untouched audio of the reference speaker;
the rubric score falls monotonically from L0 to L4 on every excerpt (Spearman ρ = −1.00).

What doesn't work yet:

Pacing is weakest (F1 0.29–0.35). Slowing a phrase also stretches its pauses, so "dragged" is often reported as "awkward pause"; the confusion matrix shows exactly that.
Different speakers are still hard (F1 0.21). Fixed noise floors are the next thing to replace with learned ones.

Three things I'd tell anyone building an evaluator

Make your labels exact before tuning anything. Injecting the flaws yourself turns an argument about subjective judgement into a number.
Hold out a speaker, not just files. My calibration F1 (0.62) and held-out F1 (0.51) are close because the split was honest. Same-speaker numbers alone would have hidden the 41-per-minute problem.
Decide what shouldn't count before deciding what should. The hardest part wasn't detecting deviations, it was deciding which deviations matter.

Code, dataset (207 labelled recordings) and the full evaluation are open source:

JayPokale / podium

Contrastive speech delivery analytics: find exactly where a delivery drifts from a great speech, and why. Multimodal AI Hackathon 2026, Track C.

🎙️ Podium: contrastive speech delivery analytics with temporal flaw grounding

Read a great speech. Podium shows exactly where your delivery drifts from it, to the tenth of a second and explains each flaw with the numbers behind it.

Built for the Multimodal AI Hackathon 2026, Track C: Contrastive Speech Analytics & Temporal Flaw Grounding.

What it does

Pick a reference: an excerpt of a great public-domain speech (JFK, Reagan).
Give your delivery: record yourself reading the same text in the browser, upload a file, or try one of the 207 dataset samples.
Get grounded feedback
- a deterministic 0-10 rubric for pacing, pauses, pitch, volume and fluency;
- flaw regions with exact start/end times, shaded on time-series overlays of your pitch, loudness and pace against the reference (time-warped onto your timeline word by word);
- a causal explanation for every region, built from the measured delta ("14.3 syllables/s here…

View on GitHub

Built for the Multimodal AI Hackathon 2026 (Track C), with heavy help from Claude Code, an AI coding agent, for implementation and evaluation runs.

Outside Quest: I asked a local Gemma to name birds. It couldn't, so it sends you on a scavenger hunt instead

Jay Pokale — Tue, 06 Oct 2026 06:05:08 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Outside Quest is a photo scavenger hunt that runs on Google's open-weight Gemma 4, entirely on a laptop with Ollama. It's designed so the screen is the shortest part of the experience:

Before the walk (about 2 minutes): type where you're going. Gemma writes a six-item quest card that fits the real place and season. Print it, or glance at it once. Then a full-screen "Phone away 🌳" card takes over.
On the walk (an hour): the phone stays in your pocket. You only take it out to photograph a find.
Back home (a few minutes): drop in your photos. Gemma checks which quests each photo completes and says what it saw. You get points, badges and a walk journal: a timeline from the photos' timestamps and a small route map drawn from their GPS tags, with no map tiles and no internet.

It's for anyone who says "I should go outside more" and then doesn't: families with kids, friends who need an excuse for a walk, or me after a day of staring at code.

The quest card adapts to where you are. For Ambazari lake garden, Nagpur, India in October it asked for "a seed pod or fruit on the ground" and "moss or damp earth" (post-monsoon tropical). For Central Park, New York it asked for "crimson leaf under a sturdy oak" and "a squirrel busy gathering nuts". That only happened after I told the prompt to think about the real local climate; the first version cheerfully asked Nagpur for "autumn colour change".

Demo

The results above use six Wikimedia Commons photos (credited in the repo) with made-up timestamps and GPS points, so I could test the journal and map.

Code

JayPokale / outside-quest

Two minutes on a screen, an hour outside: an offline photo scavenger hunt powered by Gemma 4 + Ollama. Hacktoberfest 2026 Touch Grass.

Outside Quest 🌿

Two minutes on a screen, an hour outside. An offline photo scavenger hunt powered by Google's open-weight Gemma 4, running on your own computer with Ollama.

Before you go: tell it where you're walking. Gemma writes a 6-item quest card that fits the real local season ("a seed pod on the ground" in tropical Nagpur in October, "crimson leaves under an oak" in New York) Print it or glance at it once.
On the walk: phone in your pocket. You only take it out to snap a photo of each find.
Back home: drop in your photos. Gemma checks which quests each photo completes, with the visible evidence and builds a walk journal: a timeline from the photos' timestamps and a little route map from their GPS tags.

No account, no upload, no internet needed. Your photos (and their location data) never leave the computer.

…

View on GitHub

Try it

ollama pull gemma4:e2b
git clone https://github.com/JayPokale/outside-quest && cd outside-quest
pip install -r requirements.txt && python3 server.py   # http://127.0.0.1:8777
# optional, for durable checking:
temporal server start-dev --db-filename ~/.outside-quest/temporal.db

No account, no API key. There's also a Quick card button that works with no model at all.

How I Built It

Stack: gemma4:e2b through Ollama on a 6 GB GTX 1660 Ti laptop GPU · a Python standard-library server plus Pillow (for EXIF) · an optional local Temporal server for durable checking · one HTML file, no framework. A quest card takes ~30 seconds, checking a photo ~30 seconds.

Plan A failed, and that's the most useful thing I learned

My first idea was the obvious one: a bird and plant identifier that works on the trail with no signal. Before building the UI, I tested the model on six photos: a Common Myna, a neem tree, a Monarch butterfly, a fly agaric mushroom, a peacock and a storm cloud.

Photo	gemma4:e2b said	gemma4:e4b said
Common Myna	"Jackdaw", confidence high	"Weaver Bird", confidence high
Monarch butterfly	"butterfly, undetermined"	"Heliconius"
Neem leaves	"deciduous trees, undetermined"	"trees / forest canopy"
Peacock	Peacock ✓	Peacock ✓

Asking for a top-3 list didn't rescue it: the Myna never appeared. Gemma 3 4B did worse and broke its JSON. A tool that names the wrong bird with high confidence is worse than no tool, especially outdoors, where people make decisions about mushrooms and berries.

But in every run, the coarse answer was right: there's a bird, a butterfly on a flower, red-capped mushrooms, a storm cloud, a canopy of leaves. So I flipped the design: the model only answers the question small open models are good at, "does this photo show a bird?", and the naming is left to you and a field guide. A scavenger hunt turned out to be a better way to get people outside anyway.

Making the judge strict

The first judge was too generous: it gave "a leaf" points for the butterfly photo because there were leaves in the background. Word-matching heuristics made it worse (it then rejected the peacock for "a bird"). What worked was asking Gemma, through a JSON schema, two explicit yes/no questions for every quest on every photo:

{ "id": "q2", "evidence": "A butterfly (an insect) is clearly resting on the flower.",
  "is_main_subject": true, "completed": true }

A quest only counts if both are true. On a test card (a bird, an insect on a flower, a mushroom, a stormy cloud, leaves up close, something red) and the six test photos, that gave 5 correct matches and no false ones. The miss was the storm cloud, which it read as a seascape. It isn't perfect: in the run shown above it gave "a leaf bigger than your hand" to a canopy photo, where nobody can judge leaf size. But it errs strict far more often than generous, and for a game that's the right way round.

Safety lives in code, not in the prompt

Quests the model writes pass through plain-code filters before anyone sees them. Anything about climbing, entering water, touching, picking, eating, feeding or chasing animals, roads, railway tracks, private land or night walks is dropped and replaced from a safe built-in pool. Mushrooms, berries, nests and eggs always get "(photo only, don't touch)". If the model is down or returns junk, the card fills from the same pool, and there's an instant "Quick card" button that skips the model entirely.

Making the photo check durable with Temporal

Checking is the slow part: about 30 seconds per photo, so a 40-photo walk is 20 minutes of a laptop GPU working. The first version looped over photos in the browser. Close the tab, let the laptop sleep, or have Ollama run out of memory on photo 23, and the rest of the batch was simply gone.

So the Gemma judge now runs inside a Temporal workflow, on the open-source Temporal dev server on the same laptop. Each walk is a CheckWalk workflow and each photo is a check_one activity:

Photos are saved to disk first, so you can close the tab as soon as the upload finishes. Reopen the page later and it picks the walk back up.
If Ollama is down, out of memory, or returns broken JSON, the activity is retried with backoff (5 seconds up to 2 minutes, 30 attempts) instead of being skipped.
If the server dies, the workflow resumes from the next unchecked photo. Each answer is cached next to its photo, so nothing is checked twice.

To test it, I stopped Ollama in the middle of a six-photo walk and then killed the app server too. The red bar is the photo that kept failing while Ollama was down. Attempt 4 succeeded once it came back, and the walk finished with all six photos checked:

If Temporal isn't running, the app works exactly as before. Durable mode is an extra, not a requirement.

What a Reddit reader caught

After I shared Outside Quest on Reddit, a reader pointed out something I hadn't tested: WhatsApp strips EXIF data, and iPhone sharing can leave location out. So for many people the route map and timeline would just be empty, with no error. The journal now says so ("no location in 2 of 5 photos") and suggests copying the originals over USB or AirDrop. Fixing that also turned up a bug: the walk length mixed camera times with file dates, which made one test walk "3175 minutes" long. It now uses camera times only.

They also asked whether the card changes with the season. It does. Same place, Central Park, three months: October asked for "a fallen maple leaf", January for "ice crystals on a puddle", and April for "a daffodil or early tulip bloom".

Why Does Open Innovation Matter?

Your photos are a map of where you were. Walk photos carry GPS coordinates and timestamps. Sending them to a cloud API would hand someone a log of when and where you walk. Here the model and the photos stay on your laptop.
It works where the walk is. No API key, no signal needed, no per-photo cost, so a family can check a hundred photos after a holiday without a bill.
I could test it honestly and change course. Because the model runs locally and is free to call, I could benchmark e2b against e4b and Gemma 3 on the same photos in an evening, see exactly where it fails, and design around that instead of trusting a demo.
Where closed would win: a frontier cloud model would almost certainly name the Myna. For this app that's the wrong trade: I'd rather have a private, free, offline game that's honest about what it knows.

My Agent Session

Built with heavy help from Claude Code (an AI coding agent): it ran the model benchmarks above, wrote most of the code, and drove the UI in a browser to test it. The design decisions (dropping species ID, strict judging, code-level safety filters) came out of those test runs.

Prize Categories

Best Use of Gemma
Best Use of Temporal