Emery Chen

Posted on Aug 25

Token Forensics: A Self-Hosted Observatory for Every Model Call

#ai #python #opensource #tutorial

Free tokens vanish quietly. A retry loop. A long batch. A forgotten job.

A budget gate stops the bleeding. It does not tell you where the blood went. You need a different tool.

Build a token observatory. It records every model call. It stores metadata in SQLite. It answers one question: where did my quota go?

The Problem: Invisible Spending

Free model tiers give you room to experiment. They rarely give you a usable breakdown. The API returns a usage object, but nobody persists it. After a week, you have no idea which script, which model, or which hour consumed your allowance.

A gate only says "no more." An observatory says "here is what happened." You need the second before you can meaningfully set the first.

What You Will Build

A small FastAPI service with three jobs:

Forward chat requests to an OpenAI-compatible endpoint
Record every call's timestamp, model, token count, and latency into SQLite
Expose aggregate stats and a simple visual dashboard

No rate limiting. No kill switch. Pure observation.

The Free Tier

MonkeyCode is an open-source AI coding assistant. It offers free model access and a free server option. That gives you a real endpoint and a real host for this experiment.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The exact token allowance changes over time. Check the current number on the dashboard before you depend on it. The pattern in this tutorial survives any quota change.

Prerequisites

Python 3.11 or newer
A MonkeyCode account with free model access
A terminal and 15 minutes

No Docker. No cloud account. The free server handles deployment.

Step 1: Get Credentials

export MONKEYCODE_API_KEY="your_key_here"
export MONKEYCODE_BASE_URL="https://your-endpoint-from-dashboard/v1"
export MODEL_NAME="your-model-name"

Verify this step:

curl "$MONKEYCODE_BASE_URL/models" \
  -H "Authorization: Bearer $MONKEYCODE_API_KEY"

Expect a JSON list of models. If you see an error, check the base URL. Do not proceed until this works.

Step 2: Scaffold the Project

mkdir token-observatory && cd token-observatory
python -m venv .venv && source .venv/bin/activate
pip install fastapi "uvicorn[standard]" httpx pydantic

Create the main file:

touch app.py

Step 3: Write the Recorder

Here is the complete app.py:

import os
import sqlite3
import time
from datetime import datetime

import httpx
from fastapi import FastAPI, HTTPException
from fastapi.responses import HTMLResponse
from pydantic import BaseModel

app = FastAPI()

DB_PATH = "calls.db"
MODEL = os.getenv("MODEL_NAME", "your-model-name")
BASE_URL = os.getenv("MONKEYCODE_BASE_URL", "https://your-endpoint-from-dashboard/v1")
API_KEY = os.getenv("MONKEYCODE_API_KEY", "")


def init_db():
    conn = sqlite3.connect(DB_PATH)
    conn.execute("""
        CREATE TABLE IF NOT EXISTS calls (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            ts TEXT NOT NULL,
            model TEXT NOT NULL,
            tokens INTEGER NOT NULL,
            latency_ms INTEGER NOT NULL,
            endpoint TEXT NOT NULL
        )
    """)
    conn.commit()
    conn.close()


init_db()


class ChatRequest(BaseModel):
    messages: list
    temperature: float = 0.2


def record_call(model, tokens, latency_ms, endpoint):
    conn = sqlite3.connect(DB_PATH)
    conn.execute(
        "INSERT INTO calls (ts, model, tokens, latency_ms, endpoint) VALUES (?,?,?,?,?)",
        (datetime.utcnow().isoformat(), model, tokens, latency_ms, endpoint),
    )
    conn.commit()
    conn.close()


@app.get("/health")
def health():
    return {"ok": True}


@app.post("/chat")
async def chat(req: ChatRequest):
    start = time.perf_counter()
    async with httpx.AsyncClient(timeout=60) as client:
        resp = await client.post(
            f"{BASE_URL}/chat/completions",
            headers={"Authorization": f"Bearer {API_KEY}"},
            json={"model": MODEL, "messages": req.messages, "temperature": req.temperature},
        )
    resp.raise_for_status()
    data = resp.json()
    latency_ms = int((time.perf_counter() - start) * 1000)

    usage = (data.get("usage") or {}).get("total_tokens", 0)
    record_call(MODEL, usage, latency_ms, "/chat")

    return {
        "reply": data["choices"][0]["message"]["content"],
        "tokens_used": usage,
        "latency_ms": latency_ms,
    }

Three parts matter most:

init_db() creates the table on first run
record_call() writes one row per model call
The usage.total_tokens field comes straight from the API response

Step 4: Add the Query API

Observation is useless without retrieval. Add these endpoints to app.py:

@app.get("/stats/daily")
def daily_stats():
    conn = sqlite3.connect(DB_PATH)
    rows = conn.execute(
        "SELECT substr(ts,1,10) AS day, COUNT(*), SUM(tokens) "
        "FROM calls GROUP BY day ORDER BY day DESC"
    ).fetchall()
    conn.close()
    return [{"date": r[0], "calls": r[1], "tokens": r[2]} for r in rows]


@app.get("/stats/total")
def total_stats():
    conn = sqlite3.connect(DB_PATH)
    row = conn.execute(
        "SELECT COUNT(*), COALESCE(SUM(tokens),0) FROM calls"
    ).fetchone()
    conn.close()
    return {"calls": row[0], "tokens": row[1]}

Verify the recorder:

uvicorn app:app --reload --port 8000

In another terminal:

curl http://localhost:8000/health
curl -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Reply with exactly three words"}]}'

Then check the stats:

curl http://localhost:8000/stats/total

Expect calls to be at least 1. The recorder works.

Step 5: Add a Visual Dashboard

Numbers are fine. A chart is better. Add a simple HTML page with Chart.js from a CDN:

@app.get("/dashboard", response_class=HTMLResponse)
def dashboard():
    return """
    <!DOCTYPE html>
    <html>
    <head>
        <title>Token Observatory</title>
        <script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
    </head>
    <body style="font-family: sans-serif; max-width: 800px; margin: 2rem auto;">
        <h1>Token Observatory</h1>
        <canvas id="chart" height="120"></canvas>
        <script>
        fetch('/stats/daily')
            .then(r => r.json())
            .then(data => {
                const days = data.map(d => d.date).reverse();
                const tokens = data.map(d => d.tokens).reverse();
                new Chart(document.getElementById('chart'), {
                    type: 'bar',
                    data: {
                        labels: days,
                        datasets: [{
                            label: 'tokens',
                            data: tokens,
                            backgroundColor: 'rgba(54, 162, 235, 0.6)'
                        }]
                    }
                });
            });
        </script>
    </body>
    </html>
    """

Verify the dashboard:

Open http://localhost:8000/dashboard in your browser. Send three more chat requests. Refresh the page. The bars grow.

Step 6: Deploy to the Free Server

MonkeyCode's free server hosts Python services. Push your repo to GitHub. Import it from the server dashboard. Set the same three environment variables.

Start the service:

uvicorn app:app --host 0.0.0.0 --port 8000

Verify the public endpoint:

curl https://your-service-url/health

Expect {"ok": true}. Send one chat request to the public URL. Check /stats/total again.

You now have a live token observatory on free infrastructure.

Limitations

Be honest about what this is not.

The observatory only sees calls that go through it. Direct API calls bypass the record.
SQLite is a single file. Multiple server instances will conflict.
There is no authentication. Anyone with the URL can spend your quota. Add a header check before exposing it publicly.
Free tiers change. Verify current limits before relying on them.
This is a lens, not an alarm. It does not alert you in real time.

Who Should Skip This

Teams that need centralized, multi-service audit logs
High-throughput production paths. The extra hop adds latency.
Anyone who needs real-time alerts. Build a gate for that.

Try It

Watch for a week. You will find surprises. A single endpoint eating half your quota. A model you forgot you had.

The observatory comes first. The gate comes second. MonkeyCode's free tier is a good place to start looking.

DEV Community