My AI Bot Choked on a 9,000-Word README. I Built a Token Gate.

A 9,000-word README broke my AI bot in under a second. The output was confident, detailed, and completely wrong. The root cause was boring: I never measured the input before sending it. The README blew past the model's context window, and the model quietly invented the rest. The fix was a small token gate that checks every request before it reaches the API.

I was building a docs summarizer for a side project. It worked beautifully on the two sample files I tested. Then I pointed it at a real repository. The first call returned a summary that mixed real features with ones I never wrote. A second call returned a different hallucination. The variance was the clue: the model was not reading the whole document.

I built the bot on MonkeyCode's free models, and I exposed it through a public endpoint hosted on MonkeyCode's free server. Both fit the budget of a $0 side project. But free access to a model does not remove the model's context limit. The model reads tokens, not prices. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The lesson: context is the real budget. Quotas reset monthly. The context window resets on every request, and most developers never look at it until the bot starts lying.

Count tokens before you send

The simplest fix is measuring the request on the client side. Token counting is a solved problem. The tiktoken library handles most common encodings, and the call is short:

import tiktoken

def count_tokens(text: str, encoding: str = "cl100k_base") -> int:
    return len(tiktoken.get_encoding(encoding).encode(text))

A full count costs CPU time on every request. For a gate, a rough estimate is usually enough. Prose sits near four characters per token. Code sits closer to two. I used the cheap estimate for the decision and the exact count for logging.

def quick_count(text: str) -> int:
    return max(1, len(text) // 4)

The gate in front of the model

I wrapped the counting logic in a small class. It takes the parts of a request, a priority order, and a hard limit. It returns what fits and what did not.

class TokenGate:
    def __init__(self, limit: int, reserve: int = 500):
        self.limit = limit
        self.reserve = reserve

    def budget(self) -> int:
        return self.limit - self.reserve

    def decide(self, parts: dict, order: list[str]) -> tuple[int, dict]:
        used = 0
        kept = {}
        for name in order:
            tokens = count_tokens(parts[name])
            if used + tokens > self.budget():
                return used, kept
            kept[name] = parts[name]
            used += tokens
        return used, kept

The reserve is the safety margin. Model outputs cost tokens too. A response can easily eat five hundred of them. If the response overshoots, the whole call fails. Keeping part of the window unused is not waste; it is the only plan that survives real usage.

Priority order matters. My requests had three parts: a system prompt, a question, and a document. The system prompt is core behavior, so it stays. The question is short, so it stays. The document is the first thing to cut. My order reflected that: ["system", "question", "doc"].

When the document does not fit, cutting the middle rarely hurts as much as people expect. The first part of a README explains what the project is. The last part usually covers configuration or license. The middle holds history and examples. A trim that keeps both ends preserves most of the signal:

def trim_to_budget(text: str, room: int) -> str:
    head = text[: room * 4]
    tail = text[-200:]
    return f"{head}\n...[trimmed in the middle]...\n{tail}"

This is a heuristic, not a guarantee. A README with a critical install command in the middle will lose it. When that matters, truncation is the wrong tool.

When trimming is not enough

Some documents are structured, and both ends are not enough. The correct fallback is chunking: split the document into pieces, summarize each piece, then answer from the summaries. I kept this as a pseudocode spec for the cases the gate could not handle alone:

for chunk in split(document, chunk_size=1500, overlap=200):
    summary = summarize(chunk)
bucket = join(summaries)
answer = generate(system_prompt + bucket + question)

Each chunk fits comfortably inside the budget. The cost is latency: one call per chunk plus one final call. The benefit is a full read of a long document. Cap the chunk count. Without a cap, a pathological document turns into a hundred calls and a very patient user.

Hosting the gate on the free server

A gate is only useful if every request goes through it. I put it in front of the same public endpoint that was already running on MonkeyCode's free server. The wrapper is small:

from flask import Flask, request, jsonify

app = Flask(__name__)

@app.post("/ask")
def ask():
    payload = request.get_json(silent=True) or {}
    question = str(payload.get("question", ""))[:400]
    doc = str(payload.get("doc", ""))
    used, kept = gate.decide(
        {"system": SYSTEM, "question": question, "doc": doc},
        ["system", "question", "doc"],
    )
    if "doc" not in kept:
        return jsonify({"warning": "document trimmed", "answer": answer(question, SYSTEM)}), 200
    return jsonify({"answer": answer(question, SYSTEM + kept["doc"])}), 200

app.run(port=8080)

The question length cap is not politeness. It is protection. A four-thousand-character question would eat the whole budget and leave nothing for the answer. The free server handled this workload because the requests are small and stateless. A free server is best-effort. I kept a local fallback and treated the endpoint as an experiment, not an SLA.

The full strategy fits in one table:

Input size	Strategy
fits in the budget	send as-is
document too long, edges carry the signal	trim the middle
structure matters across the whole doc	chunk and summarize
still too big after both	reject with a clear error

Reject early. An empty response is the worst failure mode.

Who should not adopt this

Small projects with tiny inputs do not need a gate. If your documents are five paragraphs, the gate is overhead with no payoff. Teams with a real uptime requirement should not build their critical path on a free server, no matter how convenient. Lossless pipelines should never truncate; the chunking path is the only honest option.

Free models changed the economics of side projects. Free servers removed the last excuse to keep tools local. But nobody removed the context window. I stopped trusting my bot the day it failed silently. I started trusting it again after I added a counter that refuses to guess.

Before you ship your next AI helper, feed it one real input by hand. If it lies, measure what you sent. The meter is ten lines of code, and it will save you a much stranger debugging session.