Your free token allowance just ran out. The dashboard shows zero. You have no idea where it went. I built a simple audit tool to find out.
MonkeyCode offers free model access and a free server option. Both are enough to run this audit. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Why token usage is invisible
Most LLM integrations record responses. Few record costs. You see the output. You never see the 47 retries behind it.
Token waste hides in three places:
- Retry loops that hammer the API
- Context that grows every run
- Duplicate calls for the same result
None of these appear in your application logs. They only appear in your quota. By then, it is too late.
What you will build
A token ledger. Every LLM call goes through a wrapper. The wrapper records:
- Project and task tags
- Prompt and completion tokens
- Latency and status
- Timestamp
A report script aggregates the data. A cron job runs it daily. Total setup time: 25 minutes.
Stage 1: Prepare the free server
MonkeyCode's free server gives you a Linux box. Connect over SSH.
ssh user@your-server-ip
Verify Python and SQLite:
python3 --version
python3 -c "import sqlite3; print(sqlite3.sqlite_version)"
You need Python 3.10 or newer. Most current images ship with it. SQLite ships with Python. No extra install.
Stage 2: Create the schema
Create a working directory:
mkdir -p ~/token-audit && cd ~/token-audit
Save this as schema.sql:
CREATE TABLE IF NOT EXISTS calls (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp TEXT NOT NULL,
project TEXT NOT NULL,
task TEXT NOT NULL,
model TEXT NOT NULL,
prompt_tokens INTEGER NOT NULL,
completion_tokens INTEGER NOT NULL,
total_tokens INTEGER NOT NULL,
latency_ms INTEGER NOT NULL,
status TEXT NOT NULL
);
Initialize the database:
sqlite3 token_ledger.db < schema.sql
Verify:
sqlite3 token_ledger.db ".tables"
You should see calls.
Stage 3: Write the wrapper
The wrapper is the core. Every LLM call goes through it. Save this as ledger.py:
#!/usr/bin/env python3
"""Token-tracking wrapper for LLM calls. Illustrative: adapt to your SDK."""
import json
import sqlite3
import time
import urllib.request
class TokenLedger:
def __init__(self, db_path="token_ledger.db"):
self.conn = sqlite3.connect(db_path)
self._init_schema()
def _init_schema(self):
self.conn.execute("""
CREATE TABLE IF NOT EXISTS calls (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp TEXT NOT NULL,
project TEXT NOT NULL,
task TEXT NOT NULL,
model TEXT NOT NULL,
prompt_tokens INTEGER NOT NULL,
completion_tokens INTEGER NOT NULL,
total_tokens INTEGER NOT NULL,
latency_ms INTEGER NOT NULL,
status TEXT NOT NULL
)
""")
self.conn.commit()
def call(self, project, task, model, messages, base_url, api_key):
payload = {
"model": model,
"messages": messages,
"temperature": 0,
}
req = urllib.request.Request(
base_url.rstrip("/") + "/chat/completions",
data=json.dumps(payload).encode(),
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
},
)
start = time.time()
try:
with urllib.request.urlopen(req, timeout=30) as resp:
body = json.loads(resp.read())
elapsed_ms = int((time.time() - start) * 1000)
usage = body.get("usage", {})
prompt_tokens = usage.get("prompt_tokens", 0)
completion_tokens = usage.get("completion_tokens", 0)
total_tokens = usage.get("total_tokens", 0)
status = "success"
content = body["choices"][0]["message"]["content"]
except Exception as exc:
elapsed_ms = int((time.time() - start) * 1000)
prompt_tokens = completion_tokens = total_tokens = 0
status = f"error: {exc}"
content = ""
self._record(
project, task, model,
prompt_tokens, completion_tokens,
total_tokens, elapsed_ms, status,
)
return content, total_tokens, status
def _record(self, project, task, model, prompt_tokens,
completion_tokens, total_tokens, latency_ms, status):
self.conn.execute(
"INSERT INTO calls (timestamp, project, task, model, "
"prompt_tokens, completion_tokens, total_tokens, "
"latency_ms, status) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
(time.strftime("%Y-%m-%d %H:%M:%S"), project, task, model,
prompt_tokens, completion_tokens, total_tokens,
latency_ms, status),
)
self.conn.commit()
The wrapper reads the usage field from the API response. That field contains token counts. If the field is missing, it records zeros.
Stage 4: Tag every call
Tags are the difference between a ledger and a log. Every call needs two tags:
-
project: which codebase made the call -
task: what operation it performed
Example usage:
from ledger import TokenLedger
ledger = TokenLedger("token_ledger.db")
content, tokens, status = ledger.call(
project="docs-summarizer",
task="summarize-readme",
model="free-model",
messages=[{"role": "user", "content": "Summarize this in 50 words."}],
base_url="https://api.example.invalid/v1",
api_key="your-key-here",
)
print(content, tokens, status)
The base URL and key above are placeholders. Replace them with values from MonkeyCode's current quickstart. The model id needs the same treatment.
Run it once:
python3 example.py
Verify the row landed:
sqlite3 token_ledger.db "SELECT * FROM calls;"
Stage 5: Write the report
Raw rows are useless. Aggregates are not. Save this as report.py:
#!/usr/bin/env python3
"""Daily token usage report."""
import sqlite3
conn = sqlite3.connect("token_ledger.db")
print("=== Tokens by project ===")
for row in conn.execute("""
SELECT project, COUNT(*), SUM(total_tokens),
SUM(prompt_tokens), SUM(completion_tokens)
FROM calls
GROUP BY project
ORDER BY SUM(total_tokens) DESC
"""):
print(f"{row[0]:<20} calls={row[1]:<5} "
f"total={row[2]:<10} prompt={row[3]:<10} completion={row[4]}")
print("\n=== Tokens by task ===")
for row in conn.execute("""
SELECT task, COUNT(*), SUM(total_tokens)
FROM calls
GROUP BY task
ORDER BY SUM(total_tokens) DESC
"""):
print(f"{row[0]:<20} calls={row[1]:<5} total={row[2]}")
print("\n=== Daily totals ===")
for row in conn.execute("""
SELECT substr(timestamp, 1, 10) as day,
SUM(total_tokens), COUNT(*)
FROM calls
GROUP BY day
ORDER BY day
"""):
print(f"{row[0]} tokens={row[1]:<10} calls={row[2]}")
Run it:
python3 report.py
You get three views. Projects show where tokens go. Tasks show what operations cost. Daily totals show trends.
Example output after one week:
=== Tokens by project ===
docs-summarizer calls=147 total=284300 prompt=261200 completion=23100
code-review-bot calls=89 total=198400 prompt=175300 completion=23100
chat-wrapper calls=412 total=152800 prompt=118200 completion=34600
The summary project burns more than half the quota. That is your first lead.
Stage 6: Schedule the report
Add a cron job:
crontab -e
Insert one line:
0 8 * * * cd /home/user/token-audit && python3 report.py >> report.log 2>&1
Verify:
crontab -l
The report accumulates daily. Read the log when you want the trend.
What the data reveals
After a week, patterns emerge. Here are the three I see most often.
Retry storms. An API returns 429. Your code retries in a loop. Each retry burns prompt tokens. One retry storm can consume 12k tokens in minutes. The ledger shows it as a task with many rows and a high error count.
Context bloat. A prompt starts at 2k tokens. You add more context over time. It reaches 30k. The ledger shows the same task with rising prompt_tokens. The fix is trimming context, not buying more quota.
Duplicate work. The same summarization task runs three times a day. The output never changes. The ledger shows identical token counts. The fix is caching.
Limitations
This approach has real limits.
- It only records calls through the wrapper. Direct
curlcalls stay invisible. - The
usagefield may be missing on some free endpoints. Zeros get recorded. - SQLite is single-file. It works for personal use. It does not scale to a team.
- Free tiers change. Re-check the current terms before relying on them.
Who should skip this
Skip this if you need:
- Real-time usage alerts
- Multi-server aggregation
- Per-user billing
Those need a proper observability stack. This is a personal audit tool. That is its job.
Closing
MonkeyCode's free server and free token allowance are enough to run this today. The wrapper is the part you keep. Start tracking before your next quota surprise.
Top comments (0)