You Keep Paying for the Same Prompt

You do not mainly pay for the next tool call. You pay to recite the conversation that justified it. That is the part most free-lane plans miss.

A chat loop looks like one job in your head. The API sees a growing document. Turn one is a short ask. Turn four is the ask, plus three tool results, plus your earlier reasoning, shipped again as input. The new thought is a rounding error. The echo is the bill.

Call it replay tax. You already spent tokens to produce a tool result. Then you spend again to remind the model that the result exists. If the model cannot see prior turns unless you resend them, every extra tool hop is also an extra copy of history. Free capacity does not cancel that copy. It only decides who waits while the copy is processed.

This is not an intelligence debate. It is operations. Public talk about agents keeps circling whether the loop is “real.” Cost ops should ask a colder question: what fraction of each request is new work, and what fraction is a photocopy of work you already did? Loop engineering fails in public when the loop never stops. It fails in private when each iteration inflates the prompt until your cheap lane is busy copying text you already paid to generate.

Picture a warehouse that restickers every crate before the forklift moves. The move looks cheap. The resticker line is not. Your context window is that line. The model is the forklift. Hiring another forklift does not shrink the labels. It just makes a longer line of machines waiting to read the same stickers.

So you split the meter. Stop logging “tokens per request” as one heroic number. Log two. Replay tokens are the prompt prefix that already existed before this turn’s new user or tool text. Fresh tokens are only the delta. If replay over fresh stays high, you are not thinking more. You are photocopying.

Here is a harness you can run locally. It does not call a vendor and it does not invent an invoice. It estimates tokens with a blunt chars/4 rule so you can see the shape of the bill before you spend real quota. Treat the counts as a ranking tool. Label them estimates until a real tokenizer sits in the same function.

"""Replay-tax harness. Estimates, does not bill."""
from dataclasses import dataclass, field
from typing import Literal

Kind = Literal["system", "user", "assistant", "tool"]

def est_tokens(text: str) -> int:
    # Rough, portable, wrong in the tails. Good enough to catch echo.
    return max(1, len(text) // 4)

@dataclass
class Turn:
    kind: Kind
    text: str

@dataclass
class LoopBill:
    turns: list[Turn] = field(default_factory=list)
    rows: list[dict] = field(default_factory=list)

    def add(self, kind: Kind, text: str) -> dict:
        fresh = est_tokens(text)
        replay = sum(est_tokens(t.text) for t in self.turns)
        total_in = replay + fresh
        ratio = replay / total_in if total_in else 0.0
        row = {
            "n": len(self.turns) + 1,
            "kind": kind,
            "fresh": fresh,
            "replay": replay,
            "input": total_in,
            "replay_ratio": round(ratio, 3),
        }
        self.turns.append(Turn(kind, text))
        self.rows.append(row)
        return row

    def should_stop(self, max_turns: int, max_input: int, max_ratio: float):
        if not self.rows:
            return None
        last = self.rows[-1]
        if last["n"] > max_turns:
            return "turn_cap"
        if last["input"] > max_input:
            return "input_cap"
        if last["n"] >= 3 and last["replay_ratio"] > max_ratio:
            return "echo_cap"
        return None


def demo_tool_loop() -> None:
    bill = LoopBill()
    bill.add("system", "You are a repo assistant. Prefer tools over guesses.")
    bill.add("user", "Why does checkout fail on order 1842?")
    tool_blob = "stack=trace " + ("frame " * 80)
    for i in range(1, 6):
        bill.add("assistant", f"Calling get_logs id={i}")
        bill.add("tool", tool_blob + f" id={i}")
        reason = bill.should_stop(max_turns=8, max_input=4000, max_ratio=0.85)
        print(bill.rows[-1], "stop=", reason)
        if reason:
            break

if __name__ == "__main__":
    demo_tool_loop()

Run it with python replay_tax.py. Watch fresh stay almost flat while replay stairs up. That stair is the argument against treating the loop as one cheap completion. You can swap est_tokens for a real tokenizer later. Do not wait for the perfect tokenizer to put a cap on the loop.

Caps belong in the same function that enqueues the next model call. A turn cap stops “just one more tool.” An input cap stops a log dump from eating the window. An echo cap is the one people skip. It fires when the conversation is mostly a recording of itself. Tune the 0.85 threshold if you must. Do not skip the measurement.

A tiny self-check keeps the rule honest. It is not a benchmark. It only proves the stair is visible.

def test_replay_outgrows_fresh():
    bill = LoopBill()
    bill.add("user", "short question")
    blob = "x" * 800
    bill.add("tool", blob)
    bill.add("tool", blob)
    last = bill.rows[-1]
    assert last["replay"] > last["fresh"]
    assert last["replay_ratio"] > 0.5
    assert bill.should_stop(8, 4000, 0.5) in {"echo_cap", "input_cap", None}

Wire the same counters around a real client the same way. Before you send, snapshot prompt size. After the tool returns, snapshot again. Log fresh, replay, wait_ms, and lane. The lane field is the cost-ops part. Free model access is a lane with a queue. A free server is a lane with noisy neighbors. Neither lane deletes replay tax. They only change how long you stand there while history is tokenized again.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option. That pairing is useful here as a lab, not as a production contract. Point the harness at a spare model, keep the loop on a free server, and learn whether your agent is thinking or reciting. Use the run to decide whether the next environment should even be free. If replay ratio climbs above your echo cap on turn three, a faster queue will not save you. You need a smaller transcript.

When is free capacity the wrong bet for this failure mode? When the job is a loop whose prompt is dominated by prior turns, and a human is waiting on the answer. You are not buying a better thought. You are buying another copy operation behind other copy operations. The wait is real. The tokens are mostly echo. Putting that work on a free lane trains you to ignore both. Deadline work has the same smell, but the mechanism is different: here the clock dies because the prompt is a novel you insist on rereading aloud.

Compress before you resend. Tool output is the usual villain. A fat stack trace you include “for context” becomes a fatter prompt two turns later. Summarize the tool result into the fact you actually need. Keep the raw dump in your own store. The model does not need the dump twice. It needs the conclusion and a pointer.

Drop a clip in front of bill.add("tool", ...) so the warehouse stops accepting unlabeled crates.

def clip_tool(raw: str, keep: int = 600) -> str:
    raw = raw.strip()
    if len(raw) <= keep:
        return raw
    head, tail = raw[: keep // 2], raw[-keep // 2 :]
    return f"{head}\n...\n{tail}\n[clipped {len(raw) - keep} chars]"

Clipping is not insight. It is inventory control. You are deciding what re-enters the prompt. If you cannot state the fact in two sentences, you are not ready to pay replay tax on the full blob. The same rule applies to your own assistant messages. Long chain-of-thought pasted back in as “memory” is just echo with better manners.

A second control is checkpointing the goal, not the chat. After two tool turns, write a standing brief: question, facts so far, next action, stop condition. Start the next model call from that brief plus the new tool result. You throw away the theater. You keep the state. This is the opposite of “let the agent see everything.” Seeing everything is how echo becomes the product.

BRIEF = """Question: {q}
Facts: {facts}
Next: {nxt}
Stop if: {stop}
"""

def next_prompt(q, facts, nxt, stop, latest_tool: str) -> str:
    body = BRIEF.format(
        q=q,
        facts="; ".join(facts[-6:]),
        nxt=nxt,
        stop=stop,
    )
    return body + "\nLatest tool:\n" + clip_tool(latest_tool)

That function is the cost control. The model still gets a loop. You no longer resend the novel. If a later turn needs a raw log, fetch it from your store as a new tool call, clip it, and forget it again. Do not keep a souvenir.

Debug it like an incident, not like a prompt-craft session. First print the last row of LoopBill on every turn. If replay_ratio crosses your cap, dump the turn kinds, not the full text. You want to see tool, tool, tool, not another stack trace in your pager. Second, time the send. If wait_ms grows while fresh does not, you are queued on echo. Third, only then change lanes. Moving an obese transcript onto a “better” model is how teams spend money to copy faster.

Limitations are sharp. chars/4 will mis-rank CJK text, minified code, and tokenizer-specific specials. Echo ratio is a heuristic; a retrieval turn can look fresh while still being junk. Caps can cut off a legitimate long investigation. Tokenizer-free estimates will not match a vendor invoice. If you ship customer-facing agents with legal or safety constraints, do not use a free lane as the only backstop. Put a hard server-side turn limit in front of any model, including a paid one.

Who should not use this approach: anyone who needs a hard latency SLO on the next token, anyone whose tool output is the artifact itself (you cannot clip a contract and call it done), and anyone who has not yet proven the loop terminates. If the product is “keep calling tools until the user is happy,” you do not have a budget. You have a vibe. Free model access will hide the vibe until the queue does not.

Do not confuse this note with a claim about model quality. A stronger model can still drown in its own minutes. A weaker model on a short brief can finish. Your job is to stop paying for the minutes. Measure replay. Cap the echo. Keep free capacity for the experiments that still fit in a small prompt. When the transcript is the product, pick a lane that prices waiting honestly, or stop looping.

If you need a quiet bench to watch the ratio move, MonkeyCode’s free model access and free server option are enough to run the harness against a real loop. Treat the session as measurement, then take the echo cap with you even if you never use that lane again.