The fastest way to break a free LLM workflow is to assume the server remembers what you told it five turns ago. A pairing session this week started from that exact assumption and ended with a small reproducible harness instead. The decision we kept after the hour was simple: treat the server as stateless and pass every important fact into every request explicitly.
The pairing target was MonkeyCode, an open-source project that combines free model access with a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The project advertises a free allowance of 10 million tokens alongside the free server, according to publicly visible project material at the time of writing. This article deliberately does not benchmark those claims; it tests only one property, which is how much the server persists between independent requests.
Why the session started with memory
This week's front-page discussions kept circling a related worry: assistants that never forget can also trust the wrong details. The same worry applies to coding servers, but the failure mode is quieter there. A server that quietly persists state can mix one project's context into another request, and nobody notices until the generated code is subtly wrong.
The senior's opening question set the tone for the whole session: where does context live, and who is responsible for it? Three answers were tried, and three dead ends taught the team more than the final script did.
Dead end one: counting tokens like a human
The first attempt estimated request size with character counts because the team wanted a rough sense of how much history each call carried. That estimate broke immediately because modern tokenizers split words, whitespace, and code punctuation in ways that character counts cannot capture. A 4,000-character prompt can easily be 900 tokens or 1,400 tokens depending on formatting and language.
# Wrong: character counts say very little about token cost
prompt = 'refactor the retry logic in gateway.py and keep the exponential backoff'
print(len(prompt)) # 83 characters, but not 83 tokens
The fix was to measure with a real tokenizer and then label every number as an estimate. Since the exact model behind a free endpoint can change, token counts are directional information rather than a contract.
Dead end two: testing memory inside one chat
The second attempt sent a seed and a question in the same conversation, saw the marker returned, and concluded that the server remembered something. The senior immediately rejected that conclusion because clients replay the entire message history on every call. A correct answer in one thread proves client-side replay, not server-side persistence, and the distinction is exactly what needed measuring.
Dead end three: reading the dashboard instead of the wire
The project's web UI shows a tidy conversation view, and the team initially checked it to learn what the server kept. The UI reconstructs history from the client's own messages, so it cannot reveal anything about server-side state. The only trustworthy signal was a fresh request that carried no history at all.
The probe that settled the argument
The final harness measures one variable: whether a server responds to a bare follow-up request as if it remembers a seed. The sequence is deliberately boring and easy to repeat:
- Send a request that asks the server to memorize a unique marker.
- Send a second request immediately afterward that asks for that marker.
- Ensure the second request contains no history and no reference to the first.
- Record whether the marker appears in the second response.
- Repeat twenty times with fresh markers and a short delay between pairs.
import requests
import time
API = 'https://your-free-server.example/v1/chat'
HEADERS = {'Authorization': 'Bearer YOUR_TOKEN'}
def send(content):
response = requests.post(API, headers=HEADERS, json={
'messages': [{'role': 'user', 'content': content}]
})
return response.text
for i in range(20):
marker = f'marker-{i}-{hex(i * 7919)[2:]}'
send(f'Remember exactly this marker: {marker}.')
time.sleep(1)
answer = send('What was the last marker I asked you to remember?')
print(f'{i:02d} marker={marker} leaked={marker in answer}')
The URL, payload shape, and auth header must be adapted to whatever endpoint is being tested. The artifact is the isolation pattern itself, not the specific library calls.
How to read the results
The twenty runs produce one of three patterns, and each pattern implies a different trust posture:
| Pattern across 20 runs | What it suggests | Decision to keep |
|---|---|---|
| All runs isolated | Stateless endpoint | Send full context on every single call |
| All runs leaked | Implicit session keyed by auth or IP | Rotate credentials and never send secrets |
| Mixed results | Flaky persistence or unstable retries | Add idempotency and verify each output |
The exact pattern observed on any given day does not change the conclusion, because a single leaked run is enough to justify the safe posture. Any non-isolated response means the server may retain prompt content, and that possibility has to drive the integration design.
The decision we kept
The pairing session ended with three rules instead of a verdict about the product. First, treat the free server as a stateless proxy with an unknown retention policy, even if every probe run looks clean. Second, put every fact needed for a good answer into the request itself, and re-verify the output with small assertions where possible. Third, never place credentials or private source snippets in a prompt, because server-side memory cannot be observed from the client side.
That posture also matches an earlier lesson from this account's gateway debugging: separate connection errors from model errors, and separate client state from server state. Once those boundaries are explicit, a free LLM server becomes a replaceable component rather than a mysterious oracle.
Who should not use this approach
Teams with strict data-governance rules should not point any free server at proprietary code without explicit approval first. Developers who need guaranteed cross-turn memory should build a client-side memory layer instead of hoping the server keeps state. And production pipelines should never depend on an isolated probe result, because free tiers and server behavior change without notice.
Limitations of the probe
The harness measures one endpoint, over one network path, on one day, and it says nothing about output quality or speed. It does not verify the 10-million-token allowance or the free server tier, which are operator-supplied claims that readers should check in the project documentation. Re-run the probe whenever the project announces changes, and keep the last run's output as a baseline in the repository.
The script above takes about three minutes to execute, which is cheap insurance for a trust question that most teams never ask. Anyone curious can run the same probe against MonkeyCode's free server, and the memory question stops being an opinion and becomes a column in a log file.
Top comments (0)