Coding agents waste tokens in obvious ways. They explain code that already explains itself. They create helpers for one call site. They add a dependency before checking the standard library.
The first response to that waste was simple: tell the agent to be shorter.
Caveman applies that idea to prose. Ponytail applies it to code. Both identified a real frustration for developers, and both became popular because the problem is easy to recognize.
But shorter output is not one problem.
A utility function, a landing page, and a message from one agent to another do not fail in the same way. A rule that works for one can damage another.
That is why GreenPT built Honey, an open-source product, and why we decided to benchmark it instead of relying on a few appealing before-and-after examples.
Three different kinds of waste
Honey uses three separate levers.
1. Less code
Before adding code, the agent checks whether the need exists, whether the codebase already solves it, whether the standard library or platform covers it, and whether an installed dependency can do the job.
This is ordinary engineering discipline. The important part is the stopping rule. Once a simple solution satisfies the task, the agent stops.
That does not mean removing input validation, error handling, authentication, accessibility, or anything the user requested. A shorter implementation that silently drops a requirement is not efficient. It is unfinished.
2. Less prose
Many agent replies contain a wind-up, a summary of the prompt, a narration of readable code, and a second summary at the end.
Honey removes that material when the user wants an implementation. It keeps explanations when the explanation is the deliverable, such as a design decision, trade-off, or learning question.
3. Denser agent handoffs
Agent-to-agent communication has different needs from human communication.
A human wants readable prose. Another agent often needs structured records. Pretty-printed JSON repeats the same keys for every row, which costs tokens without adding information.
Honey uses compact JSON by default and can use ESON for repeated record arrays. Keys are declared once, records remain addressable by stable identifiers, and row counts provide a simple truncation check.
The goal is not compression at any cost. The next agent must still be able to recover every fact it needs.
How we measured it
The benchmark contains 23 tasks in three groups:
- 14 coding tasks with executable checks
- 7 user-facing tasks with structural and accessibility checks
- 2 agent-handoff tasks with adversarial recovery questions
Each task runs against four variants:
- Plain baseline
- Caveman
- Ponytail
- Honey
The prompt and model stay the same. Only the skill changes.
Runs collapse by median, then each task is paired with its own baseline. We report the median of those paired changes. This matters because a ratio of total tokens can be dominated by one unusually long task.
Correctness uses objective tests where possible. A cross-model judge panel scores qualities that are harder to reduce to a test. The rubric does not reward short answers.
The method, task definitions, raw results, and reporting code are all in the repository.
The current result
On Claude Opus 4.8, Honey produced 29% less median output across the whole suite, with p=0.020.
It passed 100% of the objective tests.
Overall judged quality showed no measurable difference from the baseline, with p=0.648.
The output reduction also reproduced on GPT-5.5, where Honey produced 20% less output with p=0.004.
A separate live benchmark of focused Honey skills found larger reductions on narrow workflows. The Honey review skill used 70% less output at passing correctness. This is why we present the result as 29% across mixed tasks and up to 70% in focused review workflows, instead of pretending one number fits every use case.
The result varies by task type, which is the main point.
Code
Honey reduced output most strongly on code tasks while retaining a 100% unit-test pass rate.
This is where a stop-at-the-first-working-rung rule pays off. The agent avoids speculative helpers, options, and abstractions, but keeps the behavior the tests require.
User-facing work
On UI tasks, Honey did not produce a statistically meaningful output reduction. That is a useful result.
Visual polish, responsive behavior, and accessibility require code. A skill should not remove them just to improve a token chart. Honey's carve-out kept the only 100% accessibility pass rate in this tier.
Agent handoffs
Honey's strongest result appeared in structured handoffs. It retained 100% recovery while using about half the output.
Caveman and Ponytail produced shorter handoffs, but lost information under adversarial questions. A compact format only helps when the receiver can reconstruct the data.
This tier contains two tasks, so we do not claim a p-value. It is promising and needs a larger task set.
What we cannot claim
We cannot claim that Honey reduces hidden reasoning tokens.
We cannot yet claim proven dollar savings. Model pricing, prompt caching, harness behavior, and task mix all affect the bill. On this sample, the cost result was not statistically significant.
We also cannot claim that 23 tasks represent every coding workflow. The benchmark is evidence, not a universal law.
Finally, judge panels have biases. Objective tests are stronger, which is why the benchmark keeps correctness checks separate from style judgments.
We corrected our own numbers
Earlier versions of the project reported blended figures based on ratios of arm totals. That method allowed long tasks to dominate the result.
We replaced it with paired per-task medians and published the correction. The headline output reduction moved to 29%. Some competitor results also changed, and two earlier outlier claims were retired.
This is not an embarrassing footnote. It is what a benchmark should make possible. If the method improves, the claim should change.
Try it or challenge it
Honey is an open-source GreenPT product with an MIT license. It works with Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenClaw, Kiro, and other agents that support skills or project instructions.
The repository includes installation paths, the benchmark method, raw results, and commands needed to reproduce the report:
https://github.com/Green-PT/honey-for-devs
The most useful contribution is not a star. Run the benchmark on another model or harness. Add a task where brevity is dangerous. Find a requirement Honey removes. Then open an issue with the evidence.
Shorter is useful only when the important parts survive.``
Top comments (0)