The Prompt Was the Bottleneck, Not the Model

Your latency graph is not your real problem. Your prompt size is. I stopped staring at p99 and started weighing requests instead.

Latency is noisy on shared endpoints. Tokens are countable, cheap, and honest. So I built a ledger before I touched a single tuning knob.

Start with a ledger, not a flame graph

The idea is boring. One JSON line per request. Each line splits the prompt into parts I control.

Here is the weighing step. It is a sketch, not a benchmark. Run it against any OpenAI-compatible client.

# budget.py - token accounting per prompt component
import os, json, time
from collections import defaultdict

LEDGER = os.environ.get("LEDGER", "ledger.jsonl")

def weigh(text):
    # Rough proxy: ~4 chars per token. Replace with a real counter.
    return max(1, len(text) // 4)

def parts(messages, system, chunks):
    return {
        "system": system,
        "history": "".join(m["content"] for m in messages[:-1]),
        "retrieved": "".join(c["text"] for c in chunks),
        "question": messages[-1]["content"],
    }

Now the call wrapper. It records both my estimate and the provider's reported usage. Never trust one without the other.

# agent.py (sketch) - one ledger line per call
from openai import OpenAI

client = OpenAI(base_url=os.environ["BASE_URL"], api_key=os.environ["API_KEY"])

def ask(messages, system, chunks, model):
    payload = [{"role": "system", "content": system}, *messages]
    resp = client.chat.completions.create(model=model, messages=payload)
    w = {k: weigh(v) for k, v in parts(messages, system, chunks).items()}
    with open(LEDGER, "a") as f:
        f.write(json.dumps({
            "ts": time.time(),
            "model": model,
            "weights": w,
            "reported_prompt_tokens": resp.usage.prompt_tokens,
            "reported_completion_tokens": resp.usage.completion_tokens,
        }) + "\n")
    return resp

Weigh the parts, not the total

A total tells you nothing actionable. A share tells you which line to delete. So the report prints the share of prompt tokens per component.

And it prints percentiles. Averages hide the fat tail of long conversations.

# report.py - share of prompt tokens per component
import sys, json
from collections import defaultdict

def pct(xs, p):
    xs = sorted(xs)
    i = min(len(xs) - 1, max(0, round((p / 100) * (len(xs) - 1))))
    return xs[i]

def bar(share, width=28):
    n = int(round(share * width))
    return "#" * n + "." * (width - n)

def load(path):
    with open(path) as f:
        return [json.loads(line) for line in f if line.strip()]

for path in sys.argv[1:]:
    rows = load(path)
    shares = defaultdict(list)
    for r in rows:
        total = sum(r["weights"].values()) or 1
        for part, w in r["weights"].items():
            shares[part].append(w / total)
    print(f"\n{path}  n={len(rows)}")
    for part in sorted(shares):
        p50 = pct(shares[part], 50)
        p95 = pct(shares[part], 95)
        print(f"  {part:<10} p50={p50:.0%} p95={p95:.0%} {bar(p50)}")

The reconciliation check matters more than the chart. Compare your estimate sum to reported_prompt_tokens. If the ratio drifts past roughly 15%, your proxy is wrong. Recalibrate before you trust any share.

The experiment: flip exactly one knob

Freeze the prompt. Freeze the question set. Freeze concurrency at one. Then change one variable.

Run A replays twenty history turns and eight retrieved chunks. Run B replays four turns and three chunks. Same forty questions, same model, same day.

LEDGER=run-a.jsonl MODEL=$MODEL python agent.py --history 20 --chunks 8
LEDGER=run-b.jsonl MODEL=$MODEL python agent.py --history 4  --chunks 3
python report.py run-a.jsonl run-b.jsonl

Compare the share column, not the raw totals. Share survives model changes and prompt edits. Raw totals do not.

And keep the completion tokens in view. A shorter prompt that triggers longer answers is a lateral move, not a win.

The graph I kept

I kept the share bars. They sit next to the latency dashboard, and they explain it.

When p99 moves, the bars usually moved first. When the bars are flat, latency risk lives somewhere else, often in your network path.

You cannot ship a bar chart to your users. But you can ship the decision it implies.

p50 share of prompt	First move
history > 40%	cap turns, summarize older ones
retrieved > 40%	dedupe chunks, cut top-k
system > 25%	shorten rules, move constants into code
question < 5%	prompt is fine, look elsewhere

Those thresholds are heuristics, not laws. Treat them as a starting order, not a verdict.

Where free access changes the loop

Audits are repetitive. You run the same forty questions many times, and each sweep costs real budget.

Free model access removes that friction. I use it for the boring sweeps the method needs, so a knob flip does not need a purchase order. That is why this article exists at all.

The free server option solves the second problem. The ledger collector is a cron job plus a tiny endpoint. It does not deserve a paid instance.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator states a free token allowance and a free server option. Check the current terms yourself before you build on them. Availability and limits can change, and I will not pretend otherwise.

One habit keeps this honest. Run the same sweep on a second model before you conclude anything. A share pattern that survives a model swap is a prompt fact. One that vanishes was model behavior.

Limitations worth stating out loud

Character-based token estimates drift across languages and code blocks. Recalibrate per corpus, not per project.

Prompt caching also distorts the picture. A cached prefix is cheaper than its token count suggests, so weigh cost and weight separately.

And log hygiene matters. Ledgers contain user text. Hash or redact before you ship them anywhere, free server included.

Who should skip this

If your monthly spend is one coffee, the ledger is overhead. Skip it and go fix something user-visible.

Skip it too if you are under a fixed-price contract and cannot change the payload. The measurement will only make you sad.

And if compliance forbids third-party hosting for your prompts, do not move the collector off your own machines for convenience. Run it locally and accept the manual sweep.

Close the loop, then close the laptop

The model gets blamed for a bill that the prompt wrote. Weigh the parts once and you will see it.

So what is your history share right now? If you cannot answer, you have your next hour planned.