I Stabilized the Prefix. The Cache Came Back.

My agent loop got slower with every turn. The model was innocent, and so was the network.

The culprit was a timestamp I put in my own prompt. One field, changing every turn, was quietly forcing a full prefill on every request.

Sound familiar? Everyone argues about which model is fastest this week. Almost nobody asks why their own prompt keeps moving.

The symptom nobody graphs

Turn one feels fine. Turn twenty feels wrong, and you start pricing faster inference.

Then you open a fresh session and it snaps back. That gap is your first clue, and it points at your context, not the GPU.

Keep the graph anyway. A flat model-time line next to a rising wall-clock line is the signature you want.

Log the field that decides

Most providers expose some form of cached-token count in the response usage block. Field names differ, so adapt this to yours.

If that number holds steady, your prefix is being reused. If it collapses to zero turn over turn, your prefix moved.

# log one line per turn; usage field names vary by provider
for turn, resp in enumerate(stream, start=1):
    u = resp.usage
    print(turn, u.input_tokens, u.cached_tokens, hash_prefix(messages))

That fourth column is the trick. I compute the fingerprint myself instead of trusting the provider's number.

Fingerprint the prefix

The cache key is basically your serialized prompt bytes. So hash them, and strip fields that change for no good reason.

import hashlib, json

VOLATILE = {"ts", "timestamp", "run_id", "trace_id", "started_at"}

def strip_volatile(obj):
    if isinstance(obj, dict):
        return {k: strip_volatile(v) for k, v in sorted(obj.items())
                if k not in VOLATILE}
    if isinstance(obj, list):
        return [strip_volatile(v) for v in obj]
    return obj

def hash_prefix(messages):
    blob = json.dumps(strip_volatile(messages), sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(blob.encode()).hexdigest()[:12]

Twelve hex characters are plenty. You are not securing anything here, you are detecting drift.

The diff that ends the argument

Capture two consecutive request bodies, then diff them. It takes a minute and it settles the debate.

diff <(jq -S . turn1.json) <(jq -S . turn2.json) | head -40

In my case the diff was small and ugly. A started_at, an absolute working directory, and a tool list whose order followed dictionary iteration.

None of that was information the model needed. All of it was enough to invalidate the prefix.

A test that fails when you regress

Freeze the behaviour in a test. Two builds with identical logical state must hash the same, even when wall-clock time differs.

def test_prefix_is_stable():
    a = hash_prefix(build_messages(state, now="2026-01-01T00:00:00Z"))
    b = hash_prefix(build_messages(state, now="2026-01-02T00:00:00Z"))
    assert a == b, "prefix moved; expect a full prefill on every turn"

Run it in CI. When it fails, you find out before the invoice does.

The fix itself is boring. Treat the prefix as append-only, and push anything volatile into a trailing block. Canonicalise tool ordering with an explicit sort key.

Where MonkeyCode fits

I used MonkeyCode for two reasons here. Its free model access meant I could run the two-payload A/B without spending a paid key on a failed experiment. The free server option let the harness sit close to the endpoint, so network jitter stopped polluting the cache comparison.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source coding agent project, and the free tier is the part that matters for this workflow. I am told the free allowance is 10M tokens as of this writing, alongside a free server option. Verify that on the project page before you plan around it, because quotas move.

The experiment is simple. Run your harness once with volatile fields left in the prompt. Run it again with the normalizer switched on. Same code path, one flag.

One run should show cached tokens holding steady. The other should show them dropping to zero, with latency following behind. You do not need my numbers. You need yours.

Who should skip this

Skip it if your prompts are short. Prefix caching usually has a minimum length, and below that you are optimising nothing at all.

Skip it if you genuinely need fresh retrieval every turn. Restructure the prompt instead of deleting information the model depends on.

Watch the other direction too. A warm cache can hide a stale context bug, because the model keeps reusing something you forgot to update.

And if your provider does not do prefix caching, this whole article is a nice theory. Read the docs first.

So what is actually slow in your loop? Model time, network time, or your own prefix drifting under your feet?

If you want to run the two-payload test before buying keys, the MonkeyCode free tier is one honest way in. Go try it, then graph your own prefix hash.