Summarize Locally, Send Less: A Mobile LLM Pattern for Free-Tier APIs

Your mobile AI assistant has been chatting for ten minutes. Every turn, the client re-sends the entire conversation history to the server, and you have not noticed because the UI stays smooth. Then the error appears: quota exhausted. You check the logs and find that 80% of your token spend was repetition, not new information. This is the hidden cost of naive context management, and it becomes visible the moment you run against a free-tier backend.

Here is my position: a mobile LLM client should summarize its own history before sending anything to the server. On-device summarization cuts token waste, reduces latency, and keeps sensitive details off a shared backend. The free server and token allowance from MonkeyCode are a useful place to test this pattern, because they force you to care about every token. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why Your Client Repeats Itself

Most chat clients treat the conversation as an immutable array and send the whole thing on every request. That works for a few turns, but the context window grows linearly while the useful information grows sublinearly. A 20-message conversation about debugging a network issue might contain three key facts: the device model, the error code, and the fix that worked. The other seventeen messages are filler that still costs tokens.

Server-side solutions like session memory or prompt caching exist, but they are not always available on free tiers, and they do not help when the server restarts or the quota resets. The client is the only layer you fully control, so that is where the compression should happen. The goal is not to lose context; it is to reduce the redundant representation of that context.

The On-Device Summarization Pattern

The pattern has three steps: split the history into an old segment and a recent segment, summarize the old segment locally, and send the summary together with the recent messages. The summary can be as simple as an extractive heuristic or as sophisticated as a local transformer model. The key is that the summarization runs on the phone, not in the prompt.

Here is a minimal TypeScript implementation that uses extractive summarization for the old part and keeps the last four messages intact:

// summarize.ts
interface Message {
  role: 'user' | 'assistant';
  content: string;
}

function summarize(messages: Message[], maxLength: number): string {
  // Extractive summary: take the first sentence of each message.
  // For production, swap in a local ML model (e.g., TFLite) for better quality.
  return messages
    .map(m => m.content.split('.')[0] + '.')
    .join(' ')
    .slice(0, maxLength);
}

export function buildPayload(
  history: Message[],
  summaryLength: number,
  recentCount: number
) {
  const recent = history.slice(-recentCount);
  const old = history.slice(0, -recentCount);
  const summary = old.length > 0 ? summarize(old, summaryLength) : '';
  return { summary, recent };
}

Then you construct the prompt from that payload instead of the raw history:

const { summary, recent } = buildPayload(history, 200, 4);
const prompt = `Previous conversation summary: ${summary}\n\nRecent messages:\n${recent
  .map(m => `${m.role}: ${m.content}`)
  .join('\n')}\n\nUser: ${newMessage}`;

The old messages are reduced to a single paragraph, and the model still has enough context to answer coherently. If the summary is too aggressive, the recent messages provide the immediate thread; if the recent messages are too short, the summary fills the gap.

Measuring the Savings

Token savings are easy to measure if your API returns a usage object. Compare the prompt_tokens from a naive request against a summarized request with the same conversation. For a 20-message history, the summarized version often uses 50–70% fewer prompt tokens, depending on how verbose the conversation is. The exact number depends on your summarization length and recent count, so instrument both paths in your client.

You can also measure the byte size of the outgoing request as a rough proxy. A smaller request means less upload time on cellular networks and less battery drain from the radio. That matters on mobile even when the token quota is not the bottleneck.

A Test Plan with a Free Server

MonkeyCode's free server is a reasonable target for validating this pattern, because the token allowance is real and the server may be slower than a paid endpoint. Run this five-step test on a physical device and record the OS and framework versions:

1. Build a debug build that logs prompt_tokens for every request.
2. Start a 20-message conversation with a scripted user and assistant exchange.
3. Send one message using the naive full-history payload and record the token count.
4. Send the same message using the summarized payload and record the token count.
5. Verify the assistant's answer quality is still acceptable for your use case.

Expected observation: the summarized payload uses significantly fewer tokens, and the answer remains useful. If the answer degrades, increase summaryLength or recentCount until the quality returns. This is a calibration exercise, not a one-time decision.

Limitations and Tradeoffs

Summarization is lossy. If a user mentions a critical detail early in the conversation and the extractive heuristic drops it, the model will not know about it. A local ML summarizer reduces that risk but adds model size, memory pressure, and battery consumption. You need to measure whether the token savings justify the on-device cost.

Another limitation is that the summary is static once created. If the user asks about something from the old segment, the summary may not contain the exact detail they need. One mitigation is to keep a rolling summary that updates after every turn, but that adds complexity and more local compute.

Finally, the free tier's advertised 10-million-token allowance is a moving target. Read the project README before relying on that number in your design. The pattern works regardless of the exact quota, but the urgency of implementing it depends on how tight the budget is.

The Opinion, Restated

A mobile LLM client that sends its full transcript to every request is wasting the user's data, battery, and patience. On-device summarization is not a hack; it is the correct architecture for constrained environments. The free server from MonkeyCode is a good place to feel that constraint early, because it turns an abstract quota into a concrete measurement. Build the summarizer, measure the savings, and let the free tier teach you how to design for limits. The pattern will serve you long after you move to a paid backend.

MonkeyCode provides free models that can run this workflow.