Our SDK Cached One User's Answer and Served It to the Next One Who Asked

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

I work on AcruxCore, an LLM ops platform with a published SDK in TypeScript and Python. Both SDKs render prompts client-side against a small in-memory cache, so a hot path doesn't make a network round trip on every call:

const rendered = await hub.renderPrompt('rag-chat', 'production', { question });

renderPrompt fetches the templated messages once, then serves that same result out of the cache for cacheTtl milliseconds (60 seconds by default) before checking the API again. That's the whole feature. Two bugs were hiding in how "serve that same result" actually decided what counted as "the same."

Bug Fix or Performance Improvement

Correctness bug — the cache key didn't include the one thing that changes on every real call: the variables.

The key was built from three fields:

const cacheKey = `${hashApiKey(this.apiKey)}:${name}:${alias}`;

API key, prompt name, alias. Nothing about what you're actually asking. Two different renders of the same prompt with different variables, inside the same TTL window, hashed to the identical key — so the second call got the first call's cached answer back, not its own.

This wasn't a hypothetical. It was live in our own docs. The RAG-without-a-gateway tutorial renders the same prompt per question:

rendered = await hub.prompts.render("rag-chat", "production",
                                     {"context": context, "question": question})

Ask "Where is my order?", then ask "How do I refund?" within 60 seconds, and the second call's rendered.messages still says "Where is my order?" — while the trace viewer, which logs what you passed in, correctly shows "How do I refund?" as the input. The trace and the actual prompt sent to the model disagree, and nothing errors. It's also exactly why a sibling tutorial, the tool-calling agent guide, never puts the question in a template variable at all — it appends it to the messages array in code instead, a design forced by this bug rather than a stylistic choice.

The second bug was in the same file, one function down:

if (age < this.cacheTtl) {
  return cached.value; // fresh — serve it directly
}
// stale — serve the old value anyway, refresh in the background

Setting cacheTtl: 0 to turn caching off did the opposite. age < 0 is never true, so every single call took the stale branch: return the cached value immediately, and kick off a background refetch nobody's waiting on. A TTL of 0 didn't mean "don't cache" — it meant "always serve whatever was cached first, forever." There was no setting that actually disabled the cache.

Two bugs, one root cause

Both come from the same gap: the cache didn't have a way to represent "this input is different" or "this call opted out." One key was too coarse; one flag had no matching branch. Fixing them meant adding both.

Code

The fix is public in our mirror — same logic in both SDKs, shown here in TypeScript:

• packages/sdk/src/client.ts

Bug 1 — hash the variables into the key:

/**
 * Serialises a value with object keys in sorted order, at every depth, so that
 * two variable maps that differ only in insertion order produce one string.
 */
function stableStringify(value: unknown): string {
  if (value === null || typeof value !== 'object') {
    return JSON.stringify(value) ?? 'null';
  }
  if (Array.isArray(value)) {
    return `[${value.map(stableStringify).join(',')}]`;
  }
  const entries = Object.entries(value as Record<string, unknown>)
    .filter(([, v]) => v !== undefined)
    .sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0));
  return `{${entries.map(([k, v]) => `${JSON.stringify(k)}:${stableStringify(v)}`).join(',')}}`;
}

function hashVariables(variables: Record<string, unknown>): string {
  return createHash('sha256').update(stableStringify(variables)).digest('hex').slice(0, 16);
}

- const cacheKey = `${hashApiKey(this.apiKey)}:${name}:${alias}`;
+ const cacheKey = `${hashApiKey(this.apiKey)}:${name}:${alias}:${hashVariables(variables)}`;

Bug 2 — make a non-positive TTL actually skip the cache, on read and write:

  async renderPrompt(name, alias, variables = {}) {
+   // A non-positive TTL means "never serve a cached render" — skip the read and
+   // pass a null key so the write side skips too, instead of filling the cache
+   // with entries no read path will ever consult.
+   if (this.cacheTtl <= 0) {
+     return this._fetchAndCache(name, alias, variables, null);
+   }
    const cache = getCache(DEFAULT_MAX_CACHE_SIZE);
    const cacheKey = `...`;
    ...
  }

  async _fetchAndCache(name, alias, variables, cacheKey /* now: string | null */) {
    ...
-   const cache = getCache(DEFAULT_MAX_CACHE_SIZE);
-   cache.set(cacheKey, { value, fetchedAt: Date.now() });
+   if (cacheKey !== null) {
+     const cache = getCache(DEFAULT_MAX_CACHE_SIZE);
+     cache.set(cacheKey, { value, fetchedAt: Date.now() });
+   }
    return value;
  }

Technical Approach

Why hash sorted keys at every depth instead of JSON.stringify(variables)

JSON.stringify preserves insertion order. { name: 'Alice', city: 'Lahore' } and { city: 'Lahore', name: 'Alice' } are the same request to any caller, but they'd stringify to two different strings — and split into two cache entries for what should be one. A caller that builds the variables object by spreading defaults first ({ ...defaults, question }) versus spreading them last would silently halve their own cache hit rate. Sorting keys at every depth, including inside nested objects, makes key order invisible to the cache. A dedicated unit test pins exactly this:

it('treats the same variables in a different key order as one cache entry', async () => {
  (global.fetch as ReturnType<typeof vi.fn>).mockResolvedValue(makeOkResponse());
  await hub.renderPrompt('my-prompt', 'production', { name: 'Alice', city: 'Lahore' });
  await hub.renderPrompt('my-prompt', 'production', { city: 'Lahore', name: 'Alice' });
  expect(global.fetch).toHaveBeenCalledTimes(1);
});

Why not just delete the "serve stale, refresh in background" path

That's a real reliability feature, not accidental complexity: if the API is briefly unreachable, a caller with a warm cache keeps getting an answer instead of an error. Deleting it to fix the cacheTtl: 0 bug would trade one bug for a regression. The fix instead adds a third state — "don't cache at all" — that bypasses the stale-serving branch entirely rather than replacing it, so cacheTtl: 30_000 still falls back to a stale value during an outage, and only cacheTtl: 0 gives that up on purpose. Both behaviors are documented side by side in the README so the trade-off is explicit, not discovered during an incident.

Proving the fix actually fixes it, against a real API

Unit tests mock fetch. The two bugs above are about what a real render pipeline does with real prompt versions, so the integration suite hits an actual Express app and a real Postgres database instead:

it('renderPrompt renders the new variables inside the cache window instead of replaying the first render', async () => {
  // ...creates a real prompt, a real version with "Question: {{ question }}"...
  const hub = new acruxcore({ apiKey, baseUrl, cacheTtl: 600_000, maxRetries: 0 });

  const first = await hub.renderPrompt(name, 'production', { question: 'Where is my order?' });
  const second = await hub.renderPrompt(name, 'production', { question: 'How do I refund?' });
  const repeat = await hub.renderPrompt(name, 'production', { question: 'Where is my order?' });

  expect(first.messages[0].content).toBe('Question: Where is my order?');
  expect(second.messages[0].content).toBe('Question: How do I refund?');
  expect(repeat.messages[0].content).toBe('Question: Where is my order?'); // still cached, not evicted
});

I reverted src/client.ts and re-ran this test before writing the fix. It failed on exactly the assertion you'd expect — second.messages[0].content came back as 'Question: Where is my order?', the first question, not the second. That's the failure this whole post describes, caught by a test instead of a support ticket.

A second integration test does the same for Bug 2 — cacheTtl: 0, promote a new prompt version, confirm the very next call sees it instead of serving the version that was cached before the promotion.

Two existing tests were asserting the bug

The old unit test for caching was itself proof the bug had never been tested against a variable change:

- it('caches the result — second call does not hit fetch', async () => {
+ it('caches the result — repeating the same variables does not hit fetch', async () => {
    await hub.renderPrompt('my-prompt', 'production', { name: 'Alice' });
-   await hub.renderPrompt('my-prompt', 'production', { name: 'Bob' });
+   await hub.renderPrompt('my-prompt', 'production', { name: 'Alice' });
    expect(global.fetch).toHaveBeenCalledTimes(1);
  });

It rendered Alice, then Bob, and asserted one fetch call — which is precisely the bug, written down as a passing test. Fixing the implementation without touching this test would have left it green while lying about what "caching" meant. It had to be rewritten to assert the correct behavior — different inputs, different fetches — before the fix could be trusted.

Results

	before	after
second renderPrompt call, different variables, inside TTL	first call's cached content (wrong)	its own rendered content
repeated call, same variables as the first, inside TTL	cached content (correct, coincidentally)	cached content (correct, by design)
cacheTtl: 0, two calls	1st content served both times	fresh API call both times
entries written to the cache with cacheTtl: 0	one per call	zero
variables passed in a different key order	new cache entry (cache pressure ↑)	same entry, one hit

Test suite after the fix: 111 TS unit, 14 TS integration (both new ones failing-then-passing verified against the revert), 112 Python, clean tsc --noEmit, clean docs build.

What I'm carrying to the next cache I write

• A cache key has to include everything the caller varies, not just everything the caller identifies with. API key and prompt name answer "who and what" — they don't answer "with what input," and that's the part that actually changes per call.
• "Off" needs its own code path, not a boundary value on an existing one. 0 looked like it should mean "never fresh, so never serve cached" — the actual check (age < ttl) made it mean the opposite. A dedicated bypass, not a clever value, is what you want for a real off-switch.
• A passing cache test that only ever sends one input is testing the plumbing, not the cache key. The bug survived in production because the one existing test for "does caching work" happened to use two different inputs and asserted the outcome the bug produces.

---

If your SDK caches a rendered or computed result keyed by anything less than the full input, it's worth a five-minute check today: call it twice with different arguments inside your TTL window, and read back what you actually get. What did yours return?