The Free Model Read Only the First 2,000 Tokens and Still Sounded Certain

A legal-tech prototype started misclassifying long lease agreements, and I initially blamed the free model for weak reasoning. The same model handled short support tickets without any problem, so the failing inputs seemed to expose some deeper limitation in the model itself. I tuned prompts, adjusted roles, and even tried a different temperature before I checked the least glamorous number in the response: how many tokens the provider actually counted for my prompt. That number was less than one eighth of what I had sent.

The symptom depended on input length

Most documents under 2,000 tokens produced perfect classifications and extracted the correct clauses. Once the text grew past a certain size, the model would still answer confidently, but it would miss conditions buried in the later sections. It was not random noise; the errors always involved information from the second half of the document. That pattern should have pointed to truncation immediately, but the responses were so fluent that I kept assuming the model had read everything and simply failed to weigh the right part.

Think about that for a second: would you rather get an error when the input is too long, or a smooth answer that silently ignores half the facts? The free endpoint I used chose the smooth answer, and it made the failure much harder to notice.

The first clue was a suspicious usage field

I added one temporary log line to print the entire response envelope, not just the generated text. That line exposed a mismatch between what I sent and what the model reported seeing.

import sys

# After a successful POST to the completion endpoint:
print('status=%s raw=%s' % (status, raw_response.text), file=sys.stderr)

The body contained a usage object that looked like this:

"usage": {
  "prompt_tokens": 2016,
  "completion_tokens": 164,
  "total_tokens": 2180
}

I had sent a contract that my local tokenizer measured at about 18,400 tokens. The provider reported only 2,016 prompt tokens. That meant the model never saw the rest of the document. It was answering a legal question based on the first few clauses and inventing plausible completions for the rest, which is exactly why the output sounded confident but missed key obligations.

Building a minimal reproduction with the free model

To confirm this was not a one-off or a bug in my logging, I sent a series of explicitly sized inputs to the same free model from a free server worker. I used MonkeyCode's free model access and a small free server slot for this test because it let me run dozens of requests without worrying about spend. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The reproduction script is simple: submit inputs of increasing token length and record the provider's reported prompt token count.

import requests

ENDPOINT = "https://api.example.com/v1/chat/completions"  # replace with the free endpoint

def call_model(text):
    response = requests.post(
        ENDPOINT,
        json={
            "model": "free-model",
            "messages": [{"role": "user", "content": text}],
            "temperature": 0.0
        },
        timeout=30
    )
    response.raise_for_status()
    return response.json()

for length in [1000, 3000, 6000, 12000, 24000]:
    # Generate a deterministic filler text with a unique marker at the end
    filler = "word " * length
    text = filler + " The secret clause is: pay the tenant in full. "

    result = call_model(text)
    usage = result.get("usage", {})
    print(f"sent~{length} tokens, provider_prompt_tokens={usage.get('prompt_tokens')}, "
          f"model_mentioned_secret={('pay the tenant' in result['choices'][0]['message']['content'].lower())}")

Running this produced a clear cliff. For shorter inputs, provider_prompt_tokens tracked my rough estimate and the model repeated the secret clause. Once my input exceeded roughly 2,000 tokens, the reported prompt count stopped growing and the model never mentioned the clause, even though it still returned a well-formed answer. The silent cutoff was real and reproducible.

Where the truncation happened

Three possible layers could have dropped the extra tokens: my HTTP client, the gateway in front of the free model, or the model itself. I eliminated the client first by checking the raw request body bytes before they were sent. The gateway was the likely culprit because free endpoints often enforce a shorter context window than the underlying model advertises. The server accepted my request, returned HTTP 200, and quietly sliced the prompt to fit its limit. There was no warning, no finish_reason=length, and no error field. The only evidence was the difference between my local token count and the provider's usage.prompt_tokens.

The fix: measure before you send

The first rule is to know the token count before the request leaves your code. I added a local tokenizer that approximates the provider's token count. It will not be exact for every model, but it is close enough to detect when the input approaches the limit.

import tiktoken

def estimate_tokens(text, model="cl100k_base"):
    encoding = tiktoken.get_encoding(model)
    return len(encoding.encode(text))

MAX_INPUT_TOKENS = 1800  # keep some headroom under the provider's reported limit

def prepare_task(doc):
    count = estimate_tokens(doc)
    if count > MAX_INPUT_TOKENS:
        raise ValueError(f"document too long: {count} tokens > {MAX_INPUT_TOKENS}")
    return doc

If you must process longer documents, split them into overlapping chunks or use a retrieval step to select only the relevant sections. Do not rely on the free endpoint to handle long text gracefully, because it may not tell you when it cuts the input.

After the request returns, I verify that the provider actually received the same number of prompt tokens the client intended to send:

def validate_usage(sent_tokens, usage):
    received = usage.get("prompt_tokens")
    if received is None:
        return
    if received < sent_tokens * 0.8:
        raise RuntimeError(
            f"prompt likely truncated: sent ~{sent_tokens}, provider counted {received}"
        )

That mismatch check would have caught the bug in a single request instead of after a full day of prompt engineering.

What I changed in the worker

1. Added estimate_tokens before every model call and rejected inputs above a safe limit.
2. Logged usage.prompt_tokens alongside every response for later discrepancy checks.
3. Added a post-call validation that raises a loud error when the provider's prompt token count is far below my estimate.
4. Stored the original input length and the reported token count in the request metadata so future failures can be correlated.

These changes are small, but they convert a silent data-loss problem into a visible mismatch. The moment the provider starts truncating again, the worker stops and tells me instead of returning a confident-sounding answer built on half the facts.

Who should not use this approach

If you already use a paid model with a guaranteed context window and a documented finish_reason=length response, you can skip some of this ceremony. But even paid providers can truncate through middleware or upload limits, so checking usage.prompt_tokens is still good hygiene. If your input is always short, do not add a tokenizer just because this article suggested it; measure the real distribution first and apply the check only where long documents are possible.

The free server option is fine for this kind of reproduction and for low-traffic prototypes. It is not a replacement for capacity planning, an SLA, or a production-grade document processing pipeline. Free model limits can change without notice, and the tokenizer you choose may drift from the provider's actual counting. Keep the headroom conservative.

Would you have caught this?

A long document can silently become a short document, and the model will still answer as if it read the whole thing. That is a compiler-level silent failure: the output type-checks, but the semantics are wrong because the input was truncated behind your back. Do you log the provider's usage.prompt_tokens on every model call? If not, the next long report you process might earn a confident and completely wrong conclusion.