DEV Community: Arjun Shah

I ported my coding-agent benchmark to Kaggle, and the first bugs I found were mine

Arjun Shah — Sat, 10 Oct 2026 02:18:44 +0000

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

I maintain cli-bench, an open benchmark for coding agents that work in a terminal. Each task is a small repository with a bug to fix, a feature to build, or a function to speed up, plus a verifier script that decides pass or fail. Nothing is graded by another model. Either the tests pass and the extra checks hold, or the run fails.

For this challenge I ported six cli-bench tasks to Kaggle Benchmarks:

Kaggle task	What the model has to do	What the verifier checks
`cb-debug-wrong-answer`	Find the bug in a small stats library	The shipped pytest suite passes
`cb-refactor-deadcode`	Delete the three unused functions and nothing else	Dead functions gone, six live ones still defined, tests pass
`cb-feature-rate-limiter`	Write a thread-safe token bucket from an interface spec	Tests for burst, refill, atomic rollback, and 100 threads at once
`cb-perf-hot-loop`	Make a pair counter at least 10x faster in pure Python	Stdlib only, same signature, randomized equivalence, 10x on uniform, clustered, and gridded points
`cb-data-log-analysis`	Answer seven exact questions about an application log it never sees	The model writes `solve.py`; the task runs it and compares every answer to the log
`cb-sec-patch-xss`	Close an XSS hole in a comment board	Exploit tests pass and a fresh payload renders as text

In cli-bench, an agent gets a shell and a time budget. On Kaggle the model gets one shot: the prompt holds every file in the repo, and the model answers with whole files in FILE: path blocks. The task writes those files into a temporary directory and runs the original cli-bench verifier gates on the result. A run passes only if every gate passes. If the reply rewrites a test file or an input, that write is thrown away and the run fails, the same way cli-bench treats sabotage.

So the question this benchmark asks is: how much of a coding agent's score comes from the model reading code carefully, and how much comes from the agent loop of running tests and trying again? I already had one agent run on the full suite (Codex with gpt-5.6-luna, 23 of 36 trials passed). The Kaggle version takes the loop away.

Before running a single model, I found two bugs in my own benchmark

Porting meant reading every verifier line by line, and checking each task with a reference solution and a few wrong ones. Two tasks did not hold up.

sec/patch-xss could not be passed by a correct fix. Two shipped tests assert that the words alert and onerror appear nowhere in the rendered page. The verifier's own probe then requires the escaped payload text, including alert, to still be in the page. Escaping the comment with html.escape is the textbook fix, and it fails the tests, because <script>alert(...) still contains the word alert. Deleting the text passes the tests and fails the probe. In the Codex run, two of the three trials used html.escape and failed the tests, and the third stripped the text and failed the probe. I had written those failures up as the model's fault.

data/log-analysis described one rule and graded another. The question sheet defined error_rate as the "fraction of lines with level ERROR". The verifier divides ERROR request lines by request lines, and about one log line in seven is not a request. Both of the Codex trials that failed this task were off on error_rate and nothing else (0.0434 vs 0.05, 0.038 vs 0.0441). Those are exactly the numbers you get by following the text.

That is 5 of the 13 failed trials in my first leaderboard run that came from the benchmark, not the model. The Kaggle versions fix both: the XSS tests now check for raw markup instead of words, and the question sheet states the rule the verifier checks.

Models Tested

Model (Kaggle slug)	Why it is in the lineup
`gpt-5.6-luna`	The same model my Codex agent run used, so one shot and agent loop can be compared directly.
`claude-opus-5-5-default`	Anthropic's top tier on Kaggle's model list.
`gemini-3.1-pro-preview`	Google's Pro tier on Kaggle's model list.
`qwen3-coder-480b-a35b-instruct`	An open-weights model built specifically for code.
`gpt-oss-120b`	An open-weights general model you can run on your own hardware.
`gemini-3.7-flash`	Not picked: Kaggle runs its default model when a task is pushed, so it came along for free.

Each model ran each task once, through Kaggle's model proxy at the SDK's default temperature.

Findings

task	Opus 5.5	Gemini 3.1 Pro	GPT-5.6 Luna	Qwen3 Coder 480B	gpt-oss-120b	Gemini 3.7 Flash
cb-debug-wrong-answer	pass	pass	pass	pass	pass	pass
cb-refactor-deadcode	pass	pass	pass	pass	pass	pass
cb-feature-rate-limiter	pass	pass	pass	pass	pass	pass
cb-perf-hot-loop	pass	pass	pass	fail	pass	pass
cb-data-log-analysis	pass	pass	pass	pass	pass	pass
cb-sec-patch-xss	pass	pass	pass	pass	pass	pass
total	6/6	6/6	6/6	5/6	6/6	6/6

1. One shot was enough for these six tasks. 35 of 36 runs passed. With the files in front of them and no way to run anything, every model fixed the stats bug, pruned exactly the three dead functions, wrote a token bucket that rolls back atomically and survives 100 threads, and escaped the comment board correctly. These tasks are too easy to separate frontier models in one shot. That is a result too: the difficulty I measured in cli-bench was not coming from these tasks.

2. The same model did better without the agent loop. GPT-5.6 Luna passed all six in one shot. As a Codex agent it passed hot-loop 1 time in 3, with a worst-case speedup of 1.2x on the gridded dataset. In one shot it wrote a spatial grid that, timed on my machine against the same verifier, ran 42.5x faster on uniform points, 26.5x on gridded, and 14.6x on clustered. The caveats are real: one run against three, one dataset seed against three, and different machines. But it is the opposite of what I expected. Having a shell did not help the agent find the fast solution, and may have pulled it toward measuring and patching a slow one.

3. Every model reached for the same idea on hot-loop, and the only failure was invisible to the unit tests. All six bucketed points into a grid. Clustered points were the worst case for every passing model (14.6x to 41x on my machine), not gridded. Qwen3 Coder's grid passed all nine shipped unit tests and still undercounted: on one random case it found 84 pairs where there are 123. Only the randomized comparison against the naive version caught it. If my verifier had stopped at the shipped tests, that bug would have scored as a pass.

4. Most of my first-round failures were my harness, again. The first batch of Kaggle runs had 9 failures. Seven were mine:

Kaggle's task runtime does not have pytest installed. My fallback test runner did not support pytest fixtures (tmp_path, monkeypatch), so all six models "failed" the XSS task. Every one of them had escaped the output correctly.
My reply parser expected FILE: solve.py and rejected gpt-oss-120b's **FILE: solve.py**. I ran its script by hand against the same log and every answer was correct.

I fixed both, pushed new versions of those two tasks, and re-ran every model on them. The table shows the re-runs. The other two first-round failures were real. One is the hot-loop bug above. In the other, Qwen3 Coder's log script called statistics.quantiles with a method name that does not exist and crashed. On the re-run it wrote a different script and passed. That is one run at the default temperature, so treat any single pass or fail as a sample, not a verdict.

5. The corrected XSS task is passable by the textbook fix. All six models used HTML escaping, and all six passed the corrected tests and the probe. In cli-bench 0.9.1, the same approach failed. That confirms the problem was the tests, not the models.

What surprised me: across both versions of this benchmark, bugs on my side (two task specs, a test runner, a parser) caused more failures than the models did.

What I would measure next:

Harder cli-bench tasks: the flaky-test and CI tasks, and the four "houdini" probes that check whether a model games the verifier.
Three or more runs per model, so a single crash like Qwen's is visible as variance.
A version that gives the model a tool to run the tests, to see whether one retry helps or, as with hot-loop, hurts.

The lesson I am keeping: a benchmark that has never been run against a known-correct answer is not a benchmark yet. Every task in this port now ships with a reference solution, an empty reply, and a test-rewriting reply. A task only counts when the first passes and the other two fail, and that has to hold in the environment where the models actually run, not just on my laptop.

My Benchmark

Kaggle benchmark: https://www.kaggle.com/benchmarks/aks1321/cli-bench-one-shot

The six public tasks (each page has the full task code in its published notebook):

cli-bench is Apache-2.0: https://github.com/arjunkshah12345-hash/cli-bench

SuperCompress is now on PyPI! pip install supercompress in 1 line

Arjun Shah — Fri, 26 Jun 2026 19:55:27 +0000

I just published SuperCompress to PyPI! 🎉

pip install supercompress — that's all it takes.

What is it?

A tiny ~5K parameter CPU policy that scores every line of context for relevance before sending to the LLM. It keeps only what matters for the answer.

The Numbers

65% fewer tokens → same answers
100% oracle recall → never drops the answer line
~60ms CPU latency → no GPU needed
Open source → MIT with non-commercial clause

Quick Start

pip install supercompress

from supercompress import compress
result = compress(context, question)
print(f"Saved {result['kv_savings_pct']}% tokens")

Live Demo

Try the interactive comparison tool: https://supercompress.vercel.app/compare

Or read the technical deep-dive: https://dev.to/arjunkshah/how-i-built-a-prompt-compressor-that-saves-65-on-llm-costs-3m80

GitHub: https://github.com/arjunkshah/supercompress
PyPI: https://pypi.org/project/supercompress/

I Built a Prompt Compressor That Saves 65% on LLM Costs — Here's the Story

Arjun Shah — Fri, 26 Jun 2026 19:45:49 +0000

I've been working on a side project called SuperCompress — an intelligent prompt compression system for LLMs. The idea is simple: most tokens you send to an LLM never need to be processed. They're padding, boilerplate, irrelevant context. But they still burn GPU cycles.

I wanted to fix that.

The Problem

Working with LLM agents, I noticed something: every agent loop was sending massive context through the GPU. 10K tokens. 50K tokens. Sometimes more. Most of it was irrelevant to the specific task.

Truncation (keeping head + tail) was the standard approach, but it regularly dropped critical information from the middle of the context.

I thought: what if we could score each line of context for relevance BEFORE sending it to the GPU? A tiny CPU model that decides what matters.

The Build

The technical challenge was:

Train a lightweight policy (~5K params) that runs on CPU in under 60ms
Score each line of context relative to the user's question
Evict low-relevance lines while keeping answer-critical ones
Ensure the compressed output preserves correct answers

After a lot of iteration, the results surprised even me:

Policy	KV Saved	Oracle Recall
Truncation	65%	25%
H2O	65%	98%
SuperCompress	65%	100%

100% oracle recall at the same token savings. The policy never dropped a line the answer depended on.

The Environmental Angle

Here's what hit me hardest: at 50M agent turns per day (a conservative estimate for the industry), we're wasting 100B tokens daily. That's 24K GPU hours, 1,526 tons of CO₂, 6.5M liters of cooling water. Every day.

Per 1 million compressions, SuperCompress saves:

800M tokens avoided
29 kWh energy
12 kg CO₂
52 L cooling water

It's tiny per call. It's enormous at scale.

Current Status

✅ Working policy with 100% oracle recall
✅ Benchmarks and tests (65 passing)
✅ Hosted API with free tier
✅ Browser demo (compresses in-browser)
✅ Python client library
✅ Integration guides (OpenAI, LangChain, LlamaIndex)
✅ Open source (MIT)

Currently looking for:

First real users and feedback
Integration partners
Contributors to the open-source codebase

Try It

Live demo: https://supercompress.vercel.app
GitHub: https://github.com/arjunkshah/supercompress
Docs: https://arjunkshah-supercompress-55.mintlify.app

The ask: If you're building with LLMs, try compressing your next prompt. See if the answers stay the same. I'd love to hear what you think.

Now available on PyPI! pip install supercompress

Links: GitHub | PyPI | Live Demo

SuperCompress: Cut LLM Costs by 65% Without Losing Answers

Arjun Shah — Fri, 26 Jun 2026 19:23:33 +0000

Tweet 1

Every LLM call burns GPU cycles on tokens that never needed to run.

Padding. Boilerplate. Irrelevant context.

I built SuperCompress — a tiny CPU policy that cuts 65% of tokens before inference.

Open source. MIT. Free tier.

supercompress.vercel.app

Tweet 2

The problem is worse than most people realize.

At ~50M agent turns/day:

→ 100B tokens wasted daily

→ 24K GPU hours

→ 1,526 tons CO₂

→ 6.5M L cooling water

We're burning through resources on tokens that don't matter.

Tweet 3

How it works:

1️⃣ Context + question → CPU policy (5K params)

2️⃣ Every line scored for relevance to the question

3️⃣ Low-scoring lines evicted

4️⃣ Only essential tokens reach the GPU

CPU first. GPU for what matters.

Tweet 4

The numbers at 35% budget:

• 65% KV cache saved

• 100% oracle recall (vs 25% for truncation)

• ~60ms CPU latency

Same answers. ⅓ the compute.

Tweet 5

Per 1 million compressions:

→ 800M tokens avoided

→ 29 kWh saved

→ 12 kg CO₂ avoided

→ 52 L cooling water saved

Scale that across the industry and it's enormous.

Tweet 6

SuperCompress is:

✅ Open source (MIT)

✅ Free API tier

✅ Python library

✅ Browser demo (no install)

✅ Integration guides for OpenAI/LangChain

Try it: supercompress.vercel.app

GitHub: github.com/arjunkshah/supercompress

Tweet 7

Built this because I believe we can't scale AI by burning through what we have left.

Smarter compute means more AI for everyone — without the environmental cost.

Would love feedback from the community 🙏

LLM #AI #OpenSource #MachineLearning

Links: GitHub | Live Demo | Interactive Tool

How I Built a Prompt Compressor That Saves 65% on LLM Costs

Arjun Shah — Fri, 26 Jun 2026 19:15:11 +0000

How I Built a Prompt Compressor That Saves 65% on LLM Costs

Every time you call an LLM, tokens that never needed to be processed burn GPU cycles, waste money, and strain the grid. The problem gets worse with every agent loop, every long-context RAG query, every multi-turn conversation.

I built SuperCompress — a tiny ~5K parameter CPU policy that scores every line of context for relevance before inference, keeping only what the model needs.

The results? 65% fewer tokens, 100% oracle recall, ~60ms latency. Open source. MIT licensed.

The Problem: LLMs Are Wasteful

Modern LLMs process every token you give them. On long contexts (think agent logs, RAG results, codebases), most of those tokens are padding — irrelevant boilerplate that consumes KV cache space without contributing to the answer.

The standard approaches don't work well:

Approach	Tokens Saved	Answer Quality
Truncation (keep head/tail)	~65%	~25% recall
FIFO eviction	~65%	~25% recall
H2O	~65%	~98% recall
SuperCompress	~65%	100% recall

At the same KV savings, SuperCompress preserves answer quality dramatically better.

The Architecture: CPU-First Eviction

The key insight: you don't need a GPU to decide what a GPU should process.

┌─────────────┐     ┌──────────────┐     ┌──────────┐
│  Context In  │ ──→ │  CPU Policy  │ ──→ │  GPU LLM │
│ (1,247 tok)  │     │  (5K params) │     │ (437 tok) │
└─────────────┘     └──────────────┘     └──────────┘
                          │
                          ↓
                    Score each line
                    Drop low-relevance
                    Keep answer-critical

The policy is a lightweight neural network (~5,000 parameters) that runs entirely on CPU. It takes each line of context + the user's question, and scores how relevant that line is to answering the question. Lines below a threshold get evicted.

Training Approach

The policy was trained on a dataset of:

Long-form text passages (books, documentation, code)
Paired with realistic user questions
Ground-truth relevance labels from oracle LLM judgments

The training objective balances:

Token savings — maximize KV reduction
Recall — preserve lines needed for correct answers
Latency — keep inference under 100ms on CPU

Benchmarks

At a fixed 35% budget (keep 35% of tokens):

Policy          | Oracle Recall | Entity Recall | Latency
────────────────┼───────────────┼───────────────┼────────
FIFO/Truncation |         25%  |         73%   | ~57ms
Summarization   |         61%  |         65%   | ~63ms
H2O             |         98%  |         73%   | ~56ms
SuperCompress   |        100%  |         73%   | ~60ms

100% oracle recall means the policy never dropped a line that the answer depended on. At the same compute savings.

Environmental Impact

Per 1 million compressions:

800M tokens avoided — that's real GPU time
29 kWh saved — enough to power a home for a day
12 kg CO₂ avoided — tiny but it adds up
52 L water saved — datacenter cooling is thirsty

Getting Started

Python (in-process)

pip install git+https://github.com/arjunkshah/supercompress.git

from supercompress import compress_context

result = compress_context(
    "Your long context text here...",
    "What does this code do?",
    budget_ratio=0.35,
)
print(result.compressed_text)
print(f"{result.kv_savings_pct:.1f}% KV saved")

Hosted API (no local ML deps)

curl -X POST https://supercompress.vercel.app/api/v1/compress \
  -H "X-API-Key: sc_live_YOUR_KEY" \
  -d '{"context":"...","query":"Summarize this"}'

Browser demo (no setup needed)

Just visit supercompress.vercel.app and try the live demo.

What's Next

Adaptive compression ratios (not fixed budget)
Integration with LangChain/LlamaIndex as a built-in compressor
Quantized policy for even lower latency

The code is open source under MIT. Contributions welcome!

GitHub: https://github.com/arjunkshah/supercompress
Live demo: https://supercompress.vercel.app
Docs: https://arjunkshah-supercompress-55.mintlify.app