Vikramaditya Khupse

Posted on Aug 27 Edited on Sep 1

I Got Tired of Guessing How Much My LLM Workflows Would Cost

#ai #privacy #opensource #productivity

NoRefund is an open-source desktop tool that locally calculates tokens, context fit, LLM costs, and GPU VRAM requirements.

I had a 526-page PDF, a stack of personal course notes, that I wanted to feed into an agentic workflow I was building. As an AI engineer, I already know every model has a different context window and a different price per token, that part isn't new.

Before I sent anything anywhere, I wanted three answers: How many tokens is this? Will it actually fit? And how much will it cost me?

Finding all three turned out to be surprisingly annoying.

The problem

I opened one of the usual online token counters. Most of them are built for pasting a paragraph, not processing hundreds of pages, the page froze, truncated my text, or just gave up.

The ones that did handle larger input were optimized for quick text checks rather than large-document and multi-model workflows: often only a couple of models, a rough word-to-token approximation instead of the real tokenizer, and no awareness of pricing tiers, so a document crossing a long-context threshold got priced as if it hadn't. None of them touched memory either, if I wanted to self-host a model, nothing told me whether the weights and KV cache would actually fit my GPU.

And the document itself was real content I didn't want sitting on a server I don't control.

So I built NoRefund.

(Following is the actual result export of my 526 page PDF document)

Model	Provider	Tokens	Context %	Fits	Input $	Output $	Total $
GPT-5.6 Sol	OpenAI	60,771	5.8%	✓	0.2431	0.0205	0.2636
Claude Sonnet 5	Anthropic	~61,322	6.1%	✓	0.1226	0.0102	0.1329
Gemini 3.5 Flash	Google	~61,322	5.8%	✓	0.0920	0.0092	0.1012
DeepSeek V4 Flash	DeepSeek	62,455	6.0%	✓	0.0275	0.0014	0.0288

That comparison, run once, told me more than any single "token count" ever did: which models could even take the file, and what the cheapest option that fit actually was.

So I built NoRefund

NoRefund is a local-first LLM workload analyzer: tokens, context limits, cost, and self-host memory requirements, all computed on your machine.

Document analysis — process files or folders locally and get real token counts.
Context checks — see whether a workload fits a model's context window before you send it.
Cost comparison — the same document, priced across every supported model.
Local inference fit — estimate whether a model's weights and KV cache fit your GPU's VRAM.

Everything runs locally. Your documents are never uploaded to a server, network access is only used when you explicitly download a tokenizer or refresh currency rates.

On tokenizer accuracy, I want to be precise rather than just say "real tokenizers": OpenAI, DeepSeek, Llama, Qwen, and Mistral models run their actual published tokenizer. Anthropic and Google don't publish a downloadable tokenizer for Claude or Gemini, so those fall back to a close approximation, and the app marks every approximate count as such rather than presenting it as exact.

How it works

NoRefund parses the file locally (PDF, DOCX, PPTX, TXT, or Markdown), runs it through the real tokenizer for whichever model you pick, then applies that model's context limit and pricing rules, including tiered long-context rates where a provider has them. You get a token count, a fit verdict, and a cost, without a single network call.

The feature I didn't expect to care about

"Token counter" is easy to picture. "Will this model actually fit on my 24 GB GPU?" is a more interesting engineering problem, and it turned out to be the feature I use most.

Fit Check estimates a model's weight memory, KV cache, and activation overhead against a GPU, Apple Silicon chip, or cloud instance's usable VRAM, before you rent hardware or wire a model into an agent that assumes it'll fit.

Beyond a single document

The same problem shows up worse inside an agentic workflow. Retrieved documents, tool outputs, conversation history, and prior model responses all consume context, a workload that looks cheap at step one can get expensive by step five.

Document:        120k tokens
Tool output:       20k
Conversation:       40k
                  ─────
Total:            180k tokens

That's the kind of number NoRefund is built to surface before the request goes out, not after the bill arrives.

What I learned

The interesting part wasn't that token counting is hard. It's that a raw token count, on its own, doesn't actually answer the question I cared about.

The useful question is: will this workload fit, what will it cost, and can I run it on the hardware I already have? A number without those three answers just gets me back to guessing, only with more confidence than I should have.

DEV Community