I asked a chatbot my name. It told me. Ten seconds later it had no idea who I was.
Nothing about the model changed between those two questions. I deleted a list on my own laptop, and the memory went with it, because that list was the memory.
This post is about what I found building Prompt Lab, a CLI chat tool for Google's Gemini models that prints the tokens and cost of every reply. The code is on GitHub. I'm using the google-genai SDK and gemini-3.5-flash-lite.
Statelessness is the whole design constraint
Language models retain nothing between requests. No session, no history on the server. So "memory" cannot live in the model, and it doesn't. It lives in a list in your process, and it works by re-sending the entire conversation every time.
Which means memory and cost are the same thing. Every turn pays again for every turn before it.
Commands like /reset and /stats are intercepted before the network call, so they cost nothing. Everything else appends to history, then the system instruction plus the entire history goes to the API.
Building the request
The history is a list of types.Content, each carrying a role:
from google.genai import types
history: list[types.Content] = []
def add_turn(role: str, text: str) -> None:
"""role is "user" for my messages, "model" for the bot's replies."""
history.append(
types.Content(role=role, parts=[types.Part.from_text(text=text)])
)
The first gotcha is here. Gemini has no role="system". OpenAI passes the system prompt as a message inside the list; Gemini passes it as configuration alongside the list:
def config() -> types.GenerateContentConfig:
return types.GenerateContentConfig(system_instruction=system_prompt)
def send_once() -> tuple[str, types.GenerateContentResponseUsageMetadata]:
response = client.models.generate_content(
model=MODEL,
contents=history, # the whole conversation, every single time
config=config(),
)
return (response.text or "").strip(), response.usage_metadata
That contents=history line is the entire memory mechanism. One turn looks like:
add_turn("user", user_input) # 1. append what I said
reply, usage = send_once() # 2. send the WHOLE history
add_turn("model", reply) # 3. append what it said
Step 3 is the one that's easy to miss. Skip it and the model never sees its own previous answers, so it starts contradicting itself.
Reading the bill
Usage rides along on the response object:
usage = response.usage_metadata
usage.prompt_token_count # input: my messages + system prompt + history
usage.candidates_token_count # output: what the model generated
usage.total_token_count # what you are actually billed on
Input and output are priced separately, so the cost function bills them at different rates. Google publishes per million tokens:
PRICING_PER_1M = {
"gemini-3.5-flash-lite": (0.30, 2.50), # (input, output) USD per 1M tokens
}
def cost_of(input_tokens: int, output_tokens: int) -> float:
return (
input_tokens / 1_000_000 * INPUT_PER_1M
+ output_tokens / 1_000_000 * OUTPUT_PER_1M
)
Note the ratio: 2.50 / 0.30 means output costs roughly 8x more per token than input. The /stats command prints the running bill split by direction:
In that session, 74% of the tokens were input, which is history being re-sent. But look at the cost column: 156 output tokens cost $0.000390 while 451 input tokens cost only $0.000135. A quarter of the tokens, three quarters of the price. That gives you two independent levers, not one: trim history to cut input, and ask for brevity to cut output.
The measurement
The same four words, asked seconds apart, either side of a /reset:
You: what is my name
Bot: Your name is Salima!
[turn 22] history: 44 msgs | tokens in: 822 out: 6
You: /reset
History cleared. Dropped 44 messages.
You: what is my name
Bot: I don't have access to your personal information, so I don't know your name
yet! What should I call you?
[turn 23] history: 2 msgs | tokens in: 29 out: 27
822 input tokens down to 29. Twenty-eight times cheaper for an identical question. The 822 wasn't my question. It was 44 messages of history being re-sent.
Across another session input went from 41 tokens on turn 1 to 654 by turn 18, while my typed messages stayed the same length and the replies got shorter. Growth is closer to quadratic than linear: turn 10 pays for turns 1 through 9 again.
Two more gotchas
total_token_count is not always input + output
I assumed it was addition. Testing gemini-3.6-flash, one request reported 4 input tokens, 2 output tokens, and a total of 62. The missing 56 were internal reasoning tokens, billed but never shown in the reply. Trust total_token_count, not your own arithmetic.
With streaming, usage arrives on the final chunk
Switching to generate_content_stream() doesn't break cost tracking, but the usage data isn't there from the start. Keep the last one you're handed:
for chunk in client.models.generate_content_stream(
model=MODEL, contents=history, config=config()
):
if chunk.text:
print(chunk.text, end="", flush=True)
if chunk.usage_metadata:
usage = chunk.usage_metadata # only the final chunk carries it
Streaming changes perceived speed, not price.
Failing without corrupting state
Two things mattered here. First, status codes need different handling, because retrying a rejected key is pointless while retrying a rate limit is sensible:
if code in (400, 401, 403):
return "Your API key was rejected. Check GEMINI_API_KEY for typos."
if code == 404:
return (
f"The model '{MODEL}' was not found. Your key is fine - a 404 means "
"auth succeeded and only the model name is wrong."
)
if code == 429:
return "Rate limit or quota exceeded. Wait a minute and try again."
That 404 branch exists because I wasted half an hour assuming my key was broken. A 404 means authentication succeeded.
I triggered each of these deliberately rather than coding for them and hoping. Here's an invalid key, reported and survived:
Two details in that output matter. The tool keeps running instead of exiting. And the session totals report zero turns and zero messages in history, which proves the rollback below actually fired.
Second, a failed request must not leave a half-finished conversation:
try:
reply, usage = send_once()
except errors.APIError as err:
history.pop() # roll back the user turn we just appended
print(explain_api_error(err))
continue
Without that pop(), history ends on an unanswered user turn, which breaks the shape of the next request.
What I'd fix next
cost_of() takes input and output, which means it misses thinking tokens entirely. On a model that uses them, I'm under-reporting by roughly 10x. My code contradicts my own write-up on that point.
History is also never trimmed. It only grows, so a long enough conversation will stop fitting in the context window. The fix is dropping oldest turns in pairs to keep roles alternating, or summarising the old prefix into one synthetic turn.
Takeaway
Making a stateless model appear to remember means re-sending everything, and that is exactly what you pay for.
If you're building on an LLM, print your token counts once and watch them climb. It changes how you think about conversation length, system prompt size, and when to throw history away.
Code, architecture notes and a day-by-day build log: github.com/Iddrisusalima/prompt-lab
Top comments (3)
"Memory and cost are the same thing" is the cleanest statement of this I have seen, and building a tool that prints tokens per reply is the right way to internalise it. Once you can watch the number climb, the quadratic shape of a long conversation stops being abstract.
Two things that follow from the same constraint. Prompt caching changes the economics without changing the design, because the re-sent prefix gets billed at a lower rate as long as it stays byte-identical, which quietly makes prefix stability a design goal rather than an accident.
And the compaction decision is a retrieval problem in disguise: when you summarise old turns you are choosing what evidence survives, and a summary that drops the one constraint the user set in turn 3 fails in exactly the way a bad retriever does.
The 822 → 29 token drop after /reset on an identical question is the most concrete illustration I've seen of how quadratic context cost actually feels in practice. Twenty-eight times cheaper, same question, same model — purely because 44 turns of history isn't being re-sent. The two independent levers framing (trim history for input, ask for brevity for output) is useful because most people treat "LLM cost" as one dial when it's actually two with very different ratios — your 8x output/input price difference makes that concrete. The rollback on failed requests (popping the user turn before re-raising) is the kind of error handling that most tutorials skip but that matters a lot in production: a half-finished conversation corrupts every subsequent request shape.
Wow, very impressive project. This blog captures everything one needs to know about LLM and memory management, also how system prompts contribute to the overall cost of input tokens.