How I Cut LLM API Costs by 60% With 2 Lines of Code

Our OpenAI bill tripled in 60 days.

User growth was up 40%. Revenue was up. But the API bill was growing 3x faster than everything else.

I spent a week digging into why. What I found was embarrassing and completely fixable.

What the data showed

After analyzing thousands of real API calls, the same four patterns kept showing up:

1. Bloated system prompts

Most system prompts accumulate over time. Teams add instructions for edge cases, add clarifications, add reminders and never remove anything. The result: system prompts that say the same thing four different ways.

Here's a real example before and after:

# Before - 89 tokens
You are a helpful customer support assistant. Please make sure to always
be polite and professional in your responses. It is very important that
you respond to customer questions in a helpful manner. Make sure to note
that you should always try to resolve the customer's issue. Please be
concise but also make sure to be thorough. Always maintain a professional
tone and make sure to be empathetic to the customer's situation.

# After — 18 tokens
You are a polite, professional customer support assistant.
Resolve issues concisely and empathetically.

Same model behavior. 80% fewer tokens on every single call.

2. No semantic caching

Exact match caching catches identical prompts. But users don't ask the same question the same way twice.

"How do I reset my password?" and "I forgot my password, what do I do?" should return the same cached response. Without semantic caching, both hit the API and cost tokens.

We were running a customer support bot with hundreds of near-duplicate requests every day. Every one was hitting the API.

3. Stuffed context windows

We defaulted to sending full conversation history on every call. Turns out only the last 3-4 turns actually influenced the output. We were paying for 20 turns of context the model was mostly ignoring.

4. Zero visibility

We had no idea which feature was burning the most tokens. No breakdown by endpoint. No cost per feature. Just a monthly invoice.

Turns out our onboarding flow was costing 10x more per user than our core product and we'd never thought to optimize it.

The fix: two lines of code

I built Tokoscope to solve this. It wraps your existing LLM client and handles everything automatically:

JavaScript

import OpenAI from 'openai'
import { wrap } from 'tokoscope'

// Before
const client = new OpenAI()

// After — that's it
const client = wrap(new OpenAI(), {
  apiKey: 'ts_live_...' // from app.tokoscope.com/settings
})

// All your existing calls work unchanged
const response = await client.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'Hello' }]
})

Python

from openai import OpenAI
from tokoscope import wrap

# Before
client = OpenAI()

# After
client = wrap(OpenAI(), api_key='ts_live_...')

# All your existing calls work unchanged
response = client.chat.completions.create(
    model='gpt-4o',
    messages=[{'role': 'user', 'content': 'Hello'}]
)

Works with Anthropic and Gemini too, same pattern.

What happens automatically

Prompt compression

Every prompt gets scored for waste. High-waste prompts are automatically rewritten to their minimum effective form using Claude Haiku before being tracked.

Real result from our own testing:

# Original — 113 tokens
Please note that it is very important that you make sure to respond to
my question. As an AI, I want you to please make sure that you
understand that I need you to help me. Make sure to note that what I am
asking you is the following question which is important:
What is the capital of France? Please make sure to answer clearly.

# Compressed — 8 tokens
What is the capital of France? Answer concisely.

90% token reduction. Same answer from the model.

Semantic caching

Two-layer caching system:

Layer 1 — Exact match: Hash the prompt. Identical requests return cached responses instantly.

Layer 2 — Semantic match: Generate an embedding for the prompt. Compare cosine similarity against cached embeddings. At 85%+ similarity, return the cached response.

Console output when it fires:

⚡ Tokoscope cache hit [semantic (89.3% match)] — saved 93 tokens ($0.000049)

"What is the population of Japan?" and "How many people live in Japan?" scored 89.3% similarity and correctly served the cached response. Saved a full API call.

Cost attribution

Every call is logged with token counts, cost, waste score, endpoint, and user ID. The dashboard breaks it down so you can see exactly which feature costs the most.

The numbers

Three techniques, compounding:

Technique	What it catches	Typical saving
Prompt compression	All calls	~30%
Exact match caching	Identical prompts	~10–15%
Semantic caching	Similar prompts	~20–25%
Total		60–70%

These compound because they operate at different layers. Compression reduces the size of every call. Caching eliminates calls entirely. Attribution tells you where to focus first.

Per-user tracking

If you're building a multi-tenant app, you can pass a user ID to see token usage per end user:

// JavaScript
const client = wrap(new OpenAI(), {
  apiKey: 'ts_live_...',
  userId: currentUser.id
})

# Python
client = wrap(OpenAI(), api_key='ts_live_...', user_id=current_user.id)

Users show up in the dashboard with individual token usage, cost, and waste scores.

Getting started

npm install tokoscope
# or
pip install tokoscope

Free tier monitors up to 500K tokens per month. No credit card required.

The dashboard is at app.tokoscope.com and you can have it running in under 5 minutes.

What I learned

The biggest surprise wasn't how much waste there was, it was how invisible it was. Without token-level visibility, you're flying blind. You can't optimize what you can't measure.

The good news: once you can see it, the fixes are usually simple. Compress the system prompt. Add semantic caching. Trim the context window. Three changes, compounding savings.

The bad news: most teams find out about token waste the same way we did — when the bill arrives.

---

If you're building with LLMs and your API costs keep growing — try Tokoscope. It's free to start and takes 2 minutes to integrate.

Feedback welcome in the comments — especially from anyone who's tried different semantic similarity thresholds or embedding models for caching.