DEV Community: Olivia Mckelvey

How three OSLabs engineers built a CLI to catch you overpaying Claude

Olivia Mckelvey — Thu, 24 Sep 2026 02:45:22 +0000

I’m one of three developers behind PennyWyze, an open-source CLI that audits which Claude tier (Opus, Sonnet, or Haiku) is the cheapest one that still passes your quality bar. You point it at your prompt and a handful of real examples you already know the right answer to, and it tells you, in real dollars, whether you’re overpaying.

How PennyWyze works

You give it two things:

Your prompt — the exact instructions you already send to Claude in production.
A golden dataset — real inputs paired with the answer you already know is correct, one JSON object per line:

{"input": "I was charged twice this month", "expected": "billing"}
{"input": "The app crashes on upload", "expected": "technical"}

Then you run it:

pennywyze audit --prompt prompt.md --dataset dataset.jsonl --pass-rate 90

PennyWyze sends every example in your dataset to Opus, then Sonnet, then Haiku, using your real prompt over the real Anthropic API, and grades each answer against the one you said was correct. It prices every call from the token counts the API reports back, never an estimate. It prints a report showing each tier’s accuracy and projected monthly cost at your real volume, and names the cheapest tier that still cleared your bar. If a tier mathematically can’t recover, say it’s already missed more than your --pass-rate allows with half the dataset still to go, PennyWyze stops calling that model immediately instead of burning your API budget to prove a foregone conclusion.

Here’s a real run, exactly as it printed:

✓ opus audited — 50 questions
✓ sonnet audited — 50 questions
✓ haiku audited — 50 questions
PENNYWYZE AUDIT REPORT
┌───────────────────────────┬────────────┬────────────────┐
│ MODEL                     │  ACCURACY  │ EST. COST / MO │
├───────────────────────────┼────────────┼────────────────┤
│ claude-opus-5             │ 49/50 PASS │   $205.94 / mo │
├───────────────────────────┼────────────┼────────────────┤
│ claude-sonnet-5           │ 48/50 PASS │    $77.30 / mo │
├───────────────────────────┼────────────┼────────────────┤
│ claude-haiku-4-5-20251001 │ 49/50 PASS │    $26.26 / mo │
└───────────────────────────┴────────────┴────────────────┘
VERDICT  Switch to claude-haiku-4-5-20251001 — save ~$179.68/mo.
ℹ Audit cost: $0.15

Opus and Haiku tied at 49/50. Sonnet didn’t just cost more than Haiku, it scored worse too: 48/50 for nearly three times the price. On this task, there’s no accuracy gradient to pay for at any tier above Haiku, and there was no way to know that without running the numbers. Fifteen cents to find out. This isn’t an “8x savings for a little accuracy” story. We got nothing at all for the extra $180 a month.

What makes it different

Plenty of tools already rank which model is “best” in some general sense. PennyWyze tells you which model is cheapest for your prompt, graded against your own definition of correct. The verdict is only as generic as your dataset, and your dataset is built from your own real inputs.

One real scope limit, worth stating here rather than leaving for the roadmap section: today’s grading is exact-match, which means PennyWyze works for tasks with one correct answer — classification, extraction, routing — not open-ended generation like drafting a reply or summarizing a document. Broader grading (an LLM-as-judge mode) is on the roadmap, not built yet.

Why the grading is stricter than you’d expect

Both sides of a comparison get cleaned up before grading: surrounding quotes, code fences, capitalization, and trailing punctuation are stripped from both the model’s answer and your expected answer, then compared for exact equality, not “contains.” A model that answers "Billing" when you expected billing still passes; decoration doesn't count against it. But a model that answers "I think the answer is billing" fails, on purpose. If your production system needs a bare category name back and the model buries it in a sentence instead, that's a real compliance failure, not a grading technicality. PennyWyze is built to catch that gap between knowing the answer and following the instruction.

Setup

npm install -g pennywyze

Add your own Anthropic API key in a .env file (get one at the Anthropic Console), write a prompt and a golden dataset, and you're auditing against the real models in minutes.

New to how Claude’s pricing tiers work?

Anthropic, like every major LLM provider, sells the same basic capability at multiple price points, based roughly on how much reasoning power sits behind the model. Opus is the most capable and the most expensive. Haiku is the fastest and cheapest. Sonnet sits in the middle. The prices aren’t small differences: multiples, not percentages, per token processed.

Almost every team building on top of these APIs does the same thing: default every feature to the smartest tier, “just to be safe,” and never revisit it. That’s a reasonable instinct when you’re launching something for the first time. But most of what AI features do in production is narrower than “understand anything a human could say.” Sort a support ticket into a category. Extract a field from a form. Classify a message’s intent. Tasks like that rarely need the most expensive model in the lineup, just a model that’s reliably right, and the cheapest one that clears that bar is worth real money every month, at scale.

Teams already believe this in theory. Checking it requires building a whole evaluation harness: a dataset, a grading method, a way to run the same task across multiple models and compare. Nobody wants to build that just to answer one cost question, so they don’t check. They just keep paying for Opus. PennyWyze is that evaluation harness, already built.

How this actually came together

The short version of how we got here: the “we spotted a gap in the market and built a solution” story isn’t quite true. Every OSLabs team pitches eight or more product ideas before settling on three, and ours started as a conversation memory layer, my teammate Edward’s idea, to stop AI chat apps from re-sending their entire history on every message. We cared more about the theme underneath it, token efficiency, than that specific idea, which was lucky, because Mem0, Zep, and Letta (formerly MemGPT) were already solving conversation memory, and Anthropic had started shipping automatic context compaction natively. Every twist we tried hit the same wall: already built. We left one evening without an answer; Olivia came back the next day with a different one, auditing which model tier you actually need instead of defaulting to the priciest. By the time we wrote our final pitches, PennyWyze was one of them.
I didn’t fully get the problem myself until I’d spent real time learning how AI is priced in production, not just how to use it, a few weeks of catching up that ended the first time we watched a real audit print the report above.

Want to go deeper? (architecture, for the curious/technical)

Everything above is what you need to use PennyWyze. Everything below is for anyone who wants to see under the hood, or is thinking about contributing.

The whole audit loop is built around two small, swappable contracts instead of one monolithic pipeline:

ModelProvider — anything that can take a prompt and a question and return an answer plus the tokens it cost. The real Anthropic provider implements this. Adding a new model provider — say, for a future non-Anthropic tier — means writing one new file that implements this interface, not touching the loop itself.
Scorer — anything that can grade a model's answer against the expected one and return true or false. Our exact-match scorer is the only one that ships today, but the loop doesn't know or care how a Scorer makes its decision.

One more thing that’s useful if you’re contributing rather than just using PennyWyze: there’s a free, built-in fake provider behind a --fake flag, which runs the identical pipeline against canned responses instead of the real API. It doesn't tell you anything about your own prompt's real cost or accuracy, so it's not a way to "try PennyWyze out." It's how we build and test the tool itself day to day without spending real API money on every change.

The early-stopping math is one line: allowedFailures = Math.floor(datasetLength * (1 - passBar)), computed once per audit and checked after every question. Cross that number and the run for that tier ends immediately, no more API spend on a tier that's already lost. Every cost figure comes from one formula, applied to real token counts from the API response: (inputTokens × inputPrice + outputTokens × outputPrice) / 1,000,000. Anthropic quotes pricing per million tokens, and it's the one line in the whole codebase where forgetting that division would make every dollar figure wrong by six orders of magnitude, and nobody would notice until the invoice arrived.

Some things that actually tripped us up

Every project write-up says “we learned a lot,” and it’s usually too vague to mean anything. These four are pulled straight from our own commit history, not memory.

Opus doesn’t always put the answer first

When we first wired up the real Anthropic provider, we assumed the model’s text answer would be the first item in the API response’s content array, a reasonable assumption since for most models it is. Opus’s adaptive thinking broke it. The response can carry a thinking block before the text block, which meant we were occasionally grading an internal reasoning fragment as if it were the model's real answer. The fix was one line: search the response for whichever block has type === "text" instead of assuming position zero. Getting there meant noticing our grading results looked subtly off specifically on Opus, and tracing that back to a response shape we hadn't accounted for.

A quoted answer with a period after it failed grading

Our grading function strips code fences, then lowercases and strips trailing punctuation, then strips surrounding quotes, before comparing two strings for equality. Early on, that last step ran before the punctuation strip instead of after. A model answering "billing"., quoted, with a period after the closing quote, would only get its leading quote stripped, since the string's actual last character was a period, not a quote. That left a dangling quote mark stuck to the end of the string once punctuation was stripped, so it never matched the expected billing. A perfectly correct, slightly decorated answer failed for a reason that had nothing to do with correctness. The fix swapped the order: strip trailing punctuation first, so quote-stripping runs last against the string's real boundary. We added a regression test for that exact input so it can't quietly break again.

Early stopping almost never stopped

The whole point of early stopping is to stop paying for a model that’s already mathematically lost. The threshold is datasetLength × (1 − passBar), and it depends entirely on passBar being a fraction between 0 and 1. For one stretch of commits, it wasn't. cli.ts was already converting your --pass-rate flag from a percentage into a fraction before handing it off, and the audit loop divided it by 100 again on top of that. A 90% pass bar meant to allow a handful of misses out of fifty questions instead worked out to a pass bar so close to zero that a model would have to fail almost every question before the run would ever cut short. The feature was there, it just quietly couldn't trigger. It took a few commits across the same evening to notice the math didn't add up and settle on doing that one conversion exactly once.

Would the same audit give us the same answer twice?

This one worried us the most, because if the answer was no, the whole tool would be useless. Adaptive thinking means the same prompt can produce a different number of output tokens on different runs, even against the same model, which meant our cost projections could wobble between two audits of the exact same prompt and dataset. We didn’t want to ship something whose core promise, “here’s your cheapest passing tier,” might change if you ran it twice. So we ran the same real audit, against the real API, five consecutive times and compared every result. The dollar figures moved by a few percent run to run, exactly as expected from non-deterministic token counts. But the accuracy scores and the verdict itself, which tier to switch to, held identical across all five runs. That’s why we’re comfortable telling you PennyWyze’s verdict is something you can trust, not just something we hoped was true.

Where we’re taking this next

PennyWyze does one thing well today, and we’d rather ship that one thing solidly than half-ship five more. But there’s a real list of what’s next:

Cross-provider audits — not built yet, but the plan is to run the same dataset against OpenAI, Google, and Grok models alongside Claude, so the verdict isn’t just “cheapest Claude tier” but “cheapest model, period.” Today, PennyWyze audits Claude only.
LLM-as-judge grading — exact match runs first as it does today, but a judge model would review everything exact match doesn’t pass and report how often it agrees with your own grading, for tasks where “correct” isn’t a single fixed string.
pennywyze init — a guided builder for your first golden dataset, for teams who know they want to audit something but don't have 20 labeled examples sitting around yet.
A GitHub Action — re-run your audit in CI against a committed baseline, so a prompt change that quietly breaks your cheap tier gets caught before it ships.

Plus: structured JSON output and grading, a shareable HTML report, flexible dataset formats beyond JSONL, and prompt trimming — because the cheapest model is only half the savings; a leaner prompt is the other half.

The full, current list lives in the repo’s Roadmap, and every item on it is genuinely open for someone else to pick up.

If you want to help build it

PennyWyze is open source, and the two-contract design above is exactly why: most new features are one new file implementing ModelProvider or Scorer, not a change to the core loop. Fork the repo, branch, and open a PR against main — the CONTRIBUTING guide in the repo walks through building from source and running the test suite.

Meet the team

Olivia McKelvey
Edward Zgonc
Maia Bard

Don’t pay for wasted intelligence. Audit it.

If any of this sounds like a problem you have too: npm install -g pennywyze, point it at your own prompt and a handful of real examples, and see what it tells you.

If you run it, I’d like to know: which prompt did you point it at, and what tier did it say you actually needed?

Explore PennyWyze on GitHub

PennyWyze — built at OSLabs

We Built a CLI to Find Out If You’re Overpaying for Claude

Olivia Mckelvey — Thu, 24 Sep 2026 02:35:56 +0000

How three OSLabs engineers built PennyWyze—and almost built something completely different.

Hi, I’m Maia. I’m one of three developers behind PennyWyze, an open-source CLI that audits which Claude tier—Opus, Sonnet, or Haiku—is the cheapest one that still passes your quality bar.

You point it at your production prompt and a handful of real examples where you already know the correct answer.

PennyWyze runs those examples against each Claude tier, measures accuracy, calculates the actual API cost, and tells you which tier clears your bar for the lowest cost.

In other words: instead of assuming you need the most expensive model, you can test it.

How PennyWyze Works

You give it two things:

Your prompt — the exact instructions you already send to Claude in production.

A golden dataset — real inputs paired with the answer you already know is correct.

For example:

{"input": "I was charged twice this month", "expected": "billing"}
{"input": "The app crashes on upload", "expected": "technical"}

Then run:

pennywyze audit --prompt prompt.md --dataset dataset.jsonl --pass-rate 90

PennyWyze sends every example to Opus, Sonnet, and Haiku using your real prompt and the real Anthropic API.

It grades each response against your expected answer and calculates cost using the token counts returned by the API—not an estimate.

It then produces a report showing each model's accuracy and projected monthly cost at your actual volume.

A real audit

Here's what one run looked like:

✓ opus audited — 50 questions
✓ sonnet audited — 50 questions
✓ haiku audited — 50 questions

PENNYWYZE AUDIT REPORT

┌───────────────────────────┬────────────┬────────────────┐
│ MODEL │ ACCURACY │ EST. COST / MO │
├───────────────────────────┼────────────┼────────────────┤
│ claude-opus-5 │ 49/50 PASS │ $205.94 / mo │
│ claude-sonnet-5 │ 48/50 PASS │ $77.30 / mo │
│ claude-haiku-4-5-20251001 │ 49/50 PASS │ $26.26 / mo │
└───────────────────────────┴────────────┴────────────────┘

VERDICT Switch to claude-haiku-4-5-20251001 — save ~$179.68/mo.
ℹ Audit cost: $0.15

Opus and Haiku tied at 49/50.

Sonnet actually scored lower than both while costing nearly three times as much as Haiku.

On this particular task, there wasn't an accuracy difference to justify the additional cost.

And finding that out cost $0.15.

Why We Built It

There are already plenty of tools that try to answer which AI model is "best."

That's not the question we wanted to answer.

We wanted to answer:

Which model is cheapest for my prompt while still being good enough for my application?

Those are different questions.

A model that performs well on a general benchmark isn't necessarily the cheapest model that will reliably handle your specific production task.

PennyWyze uses your own examples to define what "correct" means.

That makes the verdict specific to your application instead of based on a generic benchmark.

The Problem With Always Using the Biggest Model

When you're building an AI feature for the first time, defaulting to the most capable model makes sense.

You want to know that your feature works.

The problem is that it's easy to never revisit that decision.

A lot of production AI work isn't open-ended reasoning. It's things like:

Classifying support tickets

Extracting fields

Routing messages

Detecting intent

Categorizing requests

For those tasks, you may not need the most expensive model.

But checking requires building an evaluation harness: a dataset, a grading system, and a way to run the same task against multiple models.

Most teams don't want to build all of that just to answer one cost question.

So they don't check.

PennyWyze is that evaluation harness.

One Important Limitation

PennyWyze currently uses exact-match grading.

That means it works best for tasks where there is one correct answer:

Classification

Extraction

Routing

It isn't designed yet for open-ended generation such as drafting an email or summarizing a document.

LLM-as-a-judge grading is on the roadmap.

Why Our Grading Is Strict

We wanted PennyWyze to catch a specific production problem: a model can technically know the right answer while still failing to follow the output format your application requires.

Before grading, both the model response and expected answer are normalized.

We remove things like:

Surrounding quotes

Code fences

Capitalization differences

Trailing punctuation

Then we compare the results for exact equality.

So:

Billing

passes when the expected answer is:

billing

But:

I think the answer is billing

doesn't.

That's intentional.

If your application expects a bare category name and the model wraps it in a sentence, that's a real production failure—not something we want to hide with a fuzzy grading system.

Getting Started

Install PennyWyze globally:

npm install -g pennywyze

Then add your Anthropic API key to a .env file, create a prompt and golden dataset, and run your first audit.

You can start testing against real models in minutes.

The Part We Didn't Expect

The interesting thing about PennyWyze is that it wasn't our original idea.

Every OSLabs team pitches multiple product ideas before settling on a few.

We initially started with a completely different problem: conversation memory.

The idea was to stop AI chat applications from repeatedly sending their entire conversation history with every message.

What we actually cared about was the broader problem underneath it:

token efficiency.

But as we explored the idea, we kept running into projects that were already solving the problems we were trying to solve. Mem0, Zep, and Letta were already working on conversation memory, and Anthropic had started shipping automatic context compaction.

We eventually left one evening without an answer.

The next day, Olivia came back with a different idea:

Instead of trying to reduce tokens by changing how conversations were stored, what if we looked at which model you were using in the first place?

That became PennyWyze.

What We Actually Learned Building It

The most useful lessons weren't the ones we expected to learn.

Opus doesn't always put the answer first

When we first connected the real Anthropic provider, we assumed the model's text response would be the first item in the API response's content array.

That assumption worked for most models.

Then Opus's adaptive thinking broke it.

The response can contain a thinking block before the text block, which meant we were sometimes grading an internal reasoning fragment instead of the actual answer.

The fix was simple:

Instead of assuming position zero, we search for the response block where:

type === "text"

The difficult part wasn't the fix.

It was noticing that our results were subtly wrong specifically for Opus and tracing the problem back to the response structure.

A quoted answer with a period failed grading

Our normalization logic removes code fences, lowercases the response, removes punctuation, and strips surrounding quotes.

At one point, those operations happened in the wrong order.

A response like:

"billing".

could leave a dangling quote after punctuation was removed.

The answer was semantically correct, but our grader marked it wrong.

We changed the order of operations and added a regression test for that exact case.

Early stopping almost never stopped

PennyWyze can stop auditing a model once it mathematically cannot reach the requested pass rate.

The basic calculation is:

allowedFailures = floor(datasetLength × (1 - passBar))

But at one point, we were converting the pass rate from a percentage to a fraction twice.

A 90% pass rate was effectively being divided by 100 again.

The feature existed.

The feature just almost never triggered.

It took several commits to notice that the math didn't add up and make sure the conversion happened exactly once.

Would the same audit give us the same answer twice?

This was probably the question that worried us most.

Adaptive thinking means the same prompt can produce different output token counts between runs.

That means the cost projection can move.

We ran the same real audit against the real API five consecutive times.

The dollar figures moved by a few percent, as expected from the variation in token counts.

But the accuracy scores and the resulting model choice stayed the same across all five runs.

That gave us confidence that PennyWyze's core result wasn't simply an artifact of one API run.

Under the Hood

For anyone interested in the architecture, the audit loop is built around two small, swappable contracts:

ModelProvider

Anything that can take a prompt and question and return an answer plus the tokens it cost.

The real Anthropic provider implements this interface.

Adding another provider means implementing the contract rather than rewriting the audit loop.

Scorer

Anything that can grade a model's answer against the expected answer and return true or false.

Exact-match scoring is the only scorer currently included.

But the audit loop doesn't need to know how the scorer makes its decision.

That means adding a future LLM-as-a-judge scorer can happen without rewriting the core audit pipeline.

We also built a fake provider behind a --fake flag.

It runs the same pipeline using canned responses instead of the real API, which lets us develop and test without spending API money on every change.

What's Next?

PennyWyze currently audits Claude models.

Some things we're exploring next:

Cross-provider audits across OpenAI, Google, and Grok

LLM-as-a-judge grading for open-ended tasks

pennywyze init to help users build their first golden dataset

A GitHub Action for running audits in CI

Structured JSON output

Shareable HTML reports

More flexible dataset formats

Prompt trimming

The goal is to eventually look beyond model selection.

The cheapest model is only half of the cost question.

The other half is how much you ask it to process.

If You Want to Try It

PennyWyze is open source.

npm install -g pennywyze

Then point it at your own production prompt and a handful of examples where you know the correct answer.

See what it tells you.

GitHub: https://github.com/oslabs-beta/PennyWyze

If you try it, I'd genuinely like to know:

What prompt did you audit, and what did PennyWyze tell you?

PennyWyze was built at OSLabs.

codesmith-builds #opensource #ai #javascript #devtools #llm