Why Your AI Agent Is Burning Tokens on Things It Already Knows

The Problem

Every time an agent re-reads a document it processed 10 minutes ago, re-parses a tool's output it already saw, or re-summarizes a context window it summarized last week — that is real money leaving your API budget.

Token waste in agent systems is not theoretical. It is a structural inefficiency that compounds at scale. Run 1,000 agent tasks a day and even a 15% overhead from redundant computation costs real money.

The root cause: most agent code treats every invocation as a blank slate. No memory of what was computed. No reuse of what was already derived.

A Concrete Example

Here is a pattern I see constantly in agent codebases:

async function analyzeDocument(docId: string) {
  // Every call re-downloads, re-parses, re-summarizes
  const raw = await fetchDocument(docId);
  const parsed = parseMarkdown(raw);
  const summary = await llm.summarize(parsed); // Expensive
  return extractInsights(summary);
}

// Calling this 5 times = 5x the token cost
for (const docId of documentIds) {
  results.push(await analyzeDocument(docId));
}

The fix: memoize at the layer that is actually expensive — the LLM call, not the fetch.

The Memoization Pattern

import { createHash } from 'crypto';

interface CacheEntry {
  result: unknown;
  computed_at: number;
  hit_count: number;
}

class AgentMemoCache {
  private store = new Map<string, CacheEntry>();
  private ttl_ms: number;

  constructor(ttl_ms = 1000 * 60 * 60) { // Default: 1 hour
    this.ttl_ms = ttl_ms;
  }

  private keyOf(input: unknown): string {
    return createHash('sha256')
      .update(JSON.stringify(input))
      .digest('hex')
      .slice(0, 16);
  }

  async getOrCompute<T>(
    input: unknown,
    compute: () => Promise<T>
  ): Promise<T> {
    const key = this.keyOf(input);
    const entry = this.store.get(key);

    if (entry && Date.now() - entry.computed_at < this.ttl_ms) {
      entry.hit_count++;
      console.log(`[cache HIT] key=${key} hits=${entry.hit_count}`);
      return entry.result as T;
    }

    console.log(`[cache MISS] key=${key} — computing...`);
    const result = await compute();
    this.store.set(key, {
      result,
      computed_at: Date.now(),
      hit_count: 0,
    });
    return result;
  }

  stats() {
    const entries = Array.from(this.store.values());
    return {
      total_entries: this.store.size,
      total_hits: entries.reduce((sum, e) => sum + e.hit_count, 0),
      oldest_entry: Math.min(...entries.map(e => e.computed_at)),
    };
  }
}

// Usage
const cache = new AgentMemoCache(1000 * 60 * 60); // 1hr TTL

async function analyzeDocument(docId: string) {
  return cache.getOrCompute(
    { op: 'analyze', docId },  // Cache key input
    async () => {
      const raw = await fetchDocument(docId);
      const parsed = parseMarkdown(raw);
      // Only this LLM call is expensive — and now it's cached
      const summary = await llm.summarize(parsed);
      return extractInsights(summary);
    }
  );
}

When to Cache and When Not To

Not everything should be cached:

Cache: LLM summarizations, repeated document parsing, stable tool responses (e.g. a search API that returns the same results for the same query within an hour)
Do not cache: Real-time data (prices, news), user-specific responses, anything with a freshness requirement

The key decision: does calling this with the same input produce the same output within my time window? If yes, cache it.

Eviction and Memory Pressure

A cache that grows forever is a memory leak. Add TTL (time-to-live) as shown above, or LRU eviction:

private evictOldest(maxSize: number) {
  if (this.store.size < maxSize) return;
  const oldest = Array.from(this.store.entries())
    .sort((a, b) => a[1].computed_at - b[1].computed_at)
    .slice(0, this.store.size - maxSize + 1);
  oldest.forEach(([key]) => this.store.delete(key));
}

The ROI

I measured this on a batch of 200 document analysis tasks where ~40% were duplicates across related document IDs:

Metric	Before	After
LLM calls	200	118
Token cost	$2.40	$1.42
Time to complete	48s	29s

40% reduction in LLM calls. 40% reduction in cost. Same outputs.

Putting It Together

Memoization is one of the highest-ROI optimizations you can make to an agent pipeline. It requires almost no model changes, no architecture changes — just a thin caching layer around the expensive operations.

If you are running agents at scale and not measuring token waste, you are probably spending more than you need to.

The full catalog of my AI agent tools — including the memoization patterns, batch processing, and cost tracking utilities — is at Bolt Marketplace.

DEV Community