DEV Community: Babak Abbaschian

Prompt cache, finally typed: shipping llm-ports 0.1.0-alpha.19

Babak Abbaschian — Fri, 12 Jun 2026 20:10:03 +0000

TL;DR. @llm-ports 0.1.0-alpha.19 ships a typed, provider-neutral surface for prompt caching across Anthropic, OpenAI, and Gemini. One optional cacheControl field on every request, four modes. Plus a BREAKING field rename: cost.cacheDiscountUSD → cost.cacheSavingsUSD. This is the third of four shape-locks before beta.0 on 2026-06-30.

Install: npm install @llm-ports/core@alpha resolves to it now.

Why prompt cache needed a port surface in the first place

Three providers, three completely different mechanisms.

Anthropic does explicit cache_control: { type: "ephemeral" } markers placed on message-content blocks. You decide which blocks to cache. The provider's cache lookup is keyed on the prefix of cached blocks. You pay a write rate on the first call and a read rate on subsequent calls.

OpenAI does implicit always-on caching. There is no API to opt in or out. The system caches what it caches, you pay the discount rate when it hits. No control surface, no API contract.

Gemini does createCachedContent returning a handle. You call the cache-creation API once with the content you want cached, get back an opaque handle, then pass that handle on subsequent requests instead of the original content.

If you write an application against all three, you write three completely different caching code paths. If you want to switch providers, you rewrite that code path. If you want to add a fourth provider, you add a fourth path.

This is exactly the situation ports-and-adapters exists to fix. We just hadn't fixed it yet.

The locked shape

import type { CacheControl } from "@llm-ports/core";

interface CacheControl {
  mode: "auto" | "manual" | "preCreated" | "off";
  ttlSeconds?: number;
  breakpoints?: Array<{ at: "tools" | "system" | "message-index"; index?: number }>;
  cachedContentHandle?: string;
  namespace?: string;
}

cacheControl? is now optional on GenerateTextOptions, GenerateStructuredOptions, StreamTextOptions, StreamStructuredOptions, RunAgentOptions. Omitting it is equivalent to { mode: "auto" }: every adapter does whatever its provider does by default. Code that worked under alpha.18 without touching cache control works identically under alpha.19.

The four modes map onto the three provider patterns:

Mode	Anthropic	OpenAI	Gemini
`auto`	place marker at last static block	no-op (implicit cache always on)	no-op
`manual`	place markers at supplied breakpoints	no-op	no-op
`preCreated`	no-op	no-op	uses `cachedContentHandle`
`off`	strip `cache_control` from blocks	no-op (no API to disable)	no-op (no API to disable)
`ttlSeconds`	300 or 3600	ignored	passed through

namespace partitions cache lookups by tenant when you front the provider with Helicone or a similar caching proxy. Setting namespace: "tenant:acme" keeps tenant A's cache from spilling into tenant B's lookups, which is the kind of thing that doesn't show up in a benchmark but does show up in your cross-tenant data-leak postmortem.

The breaking change

cost.cacheDiscountUSD is renamed to cost.cacheSavingsUSD on every result object.

The semantics are unchanged: USD the caller did not pay because they hit prompt cache. The field is still optional and only populated when the provider returned cache telemetry. What changed is the name, and the name was wrong.

"Discount" implied the provider was generously giving you a price reduction. They aren't. It's your money you didn't spend because you re-sent the same context. OpenInference's llm.cost.cache_savings, Helicone's dashboards, Langfuse's cost attribution — every observability vendor in the field already uses "savings". We were the odd one out.

- if (result.cost.cacheDiscountUSD !== undefined) {
-   metrics.cacheSavings.record(result.cost.cacheDiscountUSD);
- }
+ if (result.cost.cacheSavingsUSD !== undefined) {
+   metrics.cacheSavings.record(result.cost.cacheSavingsUSD);
+ }

TypeScript catches every read site. Runtime code that hand-rolled the old field name silently resolves to undefined, so check your dashboard queries too. Migration guide with the full diff and a couple of gotchas: docs/migration/alpha-18-to-alpha-19.md.

Worked example

A multi-turn Anthropic conversation with a long stable system prompt and a short turn-by-turn user message:

import { createAnthropicAdapter } from "@llm-ports/adapter-anthropic";

const adapter = createAnthropicAdapter({ apiKey: process.env.ANTHROPIC_API_KEY! });
const port = adapter.createLLMPort("claude-opus-4-7", "claude-opus");

const result = await port.generateText({
  taskType: "longform-summary",
  instructions: theBookEqualsLongSystemPrompt,
  prompt: thisTurnsShortUserQuestion,
  cacheControl: {
    mode: "manual",
    breakpoints: [{ at: "system" }],
    ttlSeconds: 3600,
  },
});

console.log(result.usage.cacheReadTokens);   // tokens served from cache
console.log(result.usage.cacheWriteTokens);  // tokens committed to cache
console.log(result.cost.cacheSavingsUSD);    // dollars you didn't pay

The same call sites work on OpenAI and Gemini. The cacheControl field is accepted everywhere; it's a no-op where the provider has no equivalent. Switching providers does not require rewriting the cache path.

Shape is locked. Behaviors aren't.

This is the part of the contract worth understanding.

The shape of CacheControl is locked at alpha.19. Every field, every literal, every union member is committed. beta.0 ships it unchanged.

Per-mode adapter behaviors mature across beta minors without breaking the shape:

Better static-block detection for Anthropic's mode: "auto".
Helicone proxy header forwarding for namespace.
Gemini createCachedContent handle lifecycle helpers (creation, refresh, expiration) under @llm-ports/capabilities.

If you write call sites against this shape today, the breakpoint placement Anthropic does in beta.1 is "more correct" placement of the same markers your code already specified. Your call sites don't change. The shape is the contract; the implementation matures behind it.

What's coming

The next 18 days finish the shape-lock sequence:

alpha.20 (Tue 2026-06-17). BudgetScope. Five tiers from tenant → customer → user → agent → session. Minute-grain windows because some providers (Cerebras's 30 RPM, Groq's 10 RPM on certain models) have rate limits that can't be expressed in cost:N/day no matter how creatively you squint at the math.
alpha.21 (Fri 2026-06-20). Observability hook signatures aligned with OpenTelemetry's gen_ai.* semantic conventions. onCost, onTokenUsage, onFallback, onValidationRetry, onCacheHit. Drop-in Langfuse / Phoenix / OpenLLMetry / Datadog wire-up. Zero adapter code on your side.
beta.0 (Tue 2026-06-30). Scope-closed. Contract goes load-bearing. Surface stops moving.

Then beta minors stop locking and start delivering things that should have existed already:

beta.1 (Tue 2026-07-28). Resilient fallback preset (no more rolling your own circuit breaker around runtimeFallback.shouldFallback). Adapter-anthropic stops wasting your money on extended-thinking models that spend their entire output budget on hidden reasoning. Auto multi-turn cache_control placement for stable system prompts. Four capability factories you've been writing yourself: createDetector, createTagger, createAnswerer, createResponder.
beta.2 (Tue 2026-08-11). Persistent budget backend. Your cost gates survive process restart. Token-denominated mode for "$X per 1M tokens per tenant per day". Pluggable CacheBackend so your exact-prompt response cache lives where you want it (Redis, file, your own KV). Three more factories: createRewriter, createDecider, createExpander.
beta.3 (Tue 2026-08-25). RerankPort gets its first adapter. Three harder factories: createRedactor for PII-safe prompts, createAgent ergonomics so multi-turn tool-use stops feeling like assembling IKEA without instructions, capability-wrapper around RerankPort.
1.0.0 (Mon 2026-09-08). The contract goes load-bearing.

Test stats

626 passing across 7 packages, up from ~615 in alpha.18. 11 new tests in packages/core/tests/cache-control.test.ts cover the shape lock and the rename. Two existing cost tests updated. Zero behavioral regressions.

Stop scattering LLM SDK/API calls across your codebase. Here is the 2-file rule that fixed mine

Babak Abbaschian — Sat, 23 May 2026 05:48:40 +0000

I upgraded an LLM SDK and expected a routine version bump.

Instead I had to touch 15+ files, fix breaking changes across four providers, and spend the rest of the day hoping I had not missed one. That was the second time it happened. I knew there would be a third.

If you have ever shipped a production LLM system, you probably recognize the smell:

An SDK minor version renames maxTokens to maxOutputTokens and now 15 files break at runtime, not compile time.
Switching one classification task from Claude to a cheaper model means editing import paths and type signatures in business logic.
You have written classifyEmail, scoreLead, triageTicket, and categorizeRequest, and they are all the same function with a different prompt string.

This is not an SDK problem. It is an architecture problem. Here is how I fixed it, and the open-source library that came out of it.

The 2-file rule

I made one rule: only two files in the entire codebase are allowed to import the LLM SDK. One adapter that translates my interface into SDK calls, and one provider registry that creates clients from config. Everything else talks to a typed interface and has no idea which provider, model, or SDK is in play.

This is just hexagonal architecture (ports and adapters, per Alistair Cockburn) applied to LLMs. You already do this for databases and message queues. Nobody scatters raw SQL across business logic. LLM providers belong in the same category. They are infrastructure, not application logic.

The dependency flow goes from this:

Application code
  ├─ direct SDK call
  ├─ direct SDK call
  └─ model router leaking SDK types

To this:

Application code
  ↓  llmClassify(), llmDraft(), llmScore() ...
Capabilities
  ↓
LLM Port  (TypeScript interface, zero SDK imports)
  ↓
Adapters + Provider Registry  (the only 2 files that touch the SDK)
  ↓
OpenAI / Anthropic / Gemini / Ollama / Vercel AI SDK

The caller says what it wants (taskType: "triage"). The infrastructure decides how. No model name parameter. No provider parameter. Policy is deferred to config.

The proof: an SDK upgrade that did not hurt

The real test came during a major SDK version jump with breaking changes (maxTokens to maxOutputTokens, CoreMessage to ModelMessage, and more). Here is what the migration commit looked like:

2 files changed (the adapter and the agent runtime), plus 1 minor fix.
All 18 activity files unchanged.
All 10 agent files unchanged.
The final migration deleted more code than it added: 192 insertions, 688 deletions.

28 out of 31 files did not change, because they do not know the SDK exists. If a core dependency upgrade touches your business logic, your boundaries are wrong.

The part that surprised me: the same 7 operations, everywhere

I started this to isolate the SDK. Then I noticed the bigger problem. I was not calling LLMs in 21 different places. I was reimplementing the same seven cognitive operations with slight variations:

Capability	What you give it	What you get back
Classify	content + rubric	one label from an enum + reasoning
Score	content + rubric + axes	numeric ratings per axis
Draft	persona + situation	longer text in a chosen tone
Summarize	long content + length target	shorter content, key points kept
Extract	unstructured text + schema	a typed structured object
Plan	goal + constraints	an ordered list of steps
Analyze	evidence + question	recommendation with caveats

Five activities classified content with five different prompt structures. Nine drafted messages with nine different tone injections. Same operation, no shared implementation. When I improved one classification prompt, I had to remember to update four other places. I usually forgot.

You are not writing 47 prompts. You are writing 7 prompts, 47 times, with slightly different ingredients.

So I extracted them into capability factories. A factory takes the invariant parts (schema, rubric, model routing, observability hooks) and returns a function that takes only the varying part (the content):

import { createClassifier } from "@llm-ports/capabilities";
import { z } from "zod";

const IntentSchema = z.object({
  intent: z.enum(["question", "request", "complaint", "feedback", "other"]),
  urgency: z.enum(["low", "normal", "high"]),
  reasoning: z.string(),
});

export const classifyIntent = createClassifier({
  port: llm,                 // your provider-agnostic port
  schema: IntentSchema,
  schemaName: "user-intent",
  rubric: `
    question: asking for information
    request: wants something done
    complaint: reports a problem
    feedback: opinion only
    other: anything else
  `,
});

Then every call site, across all your files, is the same shape:

const result = await classifyIntent({ content: userMessage });
// { intent: "request", urgency: "high", reasoning: "..." }  fully typed

Improve the rubric once, and every classifier in the system gets better. Prompt engineering stops being scattered strings and becomes a reusable system asset.

llm-ports

I pulled this pattern out of my production system and shipped it as an open-source, MIT-licensed TypeScript library: llm-ports.

60 second setup

Configure providers in .env:

LLM_PROVIDER_FAST=anthropic|<model>|cost:50/day
LLM_PROVIDER_SMART=anthropic|<model>|cost:200/day
LLM_TASK_ROUTE_TRIAGE=fast,smart

Create the port once:

import { createRegistryFromEnv } from "@llm-ports/core";
import { createAnthropicAdapter } from "@llm-ports/adapter-anthropic";

export const llm = createRegistryFromEnv({
  adapters: {
    anthropic: createAnthropicAdapter({ apiKey: process.env.ANTHROPIC_API_KEY! }),
  },
}).getPort();

Use it anywhere, with no SDK imports:

const result = await llm.generateText({
  taskType: "triage",
  prompt: "Classify this email...",
});

The registry selects the right model for the task, enforces cost limits, falls back through the provider chain on budget exhaustion, and records usage, cost, and latency.

What you get

Multi-provider routing across OpenAI, Anthropic, Google Gemini, Ollama, and the Vercel AI SDK.
Fallback chains when a provider exceeds budget.
USD-based cost gating with hourly, daily, and monthly limits. Budget exhaustion is a typed exception, not a surprise invoice.
The 7 capability factories: createClassifier, createScorer, createDrafter, createSummarizer, createExtractor, createPlanner, createAnalyzer.
Validation recovery for structured output. If a model returns invalid JSON or a wrong enum, it auto-retries with a correction prompt. Bad output stops at the capability boundary instead of leaking downstream.
Tool-use safety primitives: destructive markers, confirmation-required actions, max output byte limits.
Observability hooks for cost, latency, quality, and outcomes.
No runtime dependency on LangChain or LlamaIndex. Core plus one adapter plus capabilities is a small install footprint, strict TypeScript throughout.

How it compares

Vercel AI SDK unifies provider calls. llm-ports adds the registry, fallback chains, USD cost gating, validation recovery, and capability factories on top. There is an adapter to migrate from it incrementally.
LiteLLM is a Python-first HTTP proxy. llm-ports is TypeScript and runs in-process, no extra network hop.
Portkey is a commercial hosted gateway. llm-ports is MIT and has no hosted dependency.
LangChain.js is a framework. llm-ports is a lightweight architecture and control layer, not a framework you build your whole app inside.

When to use it (and when not to)

Use it if you run 2+ providers (or might switch later), have 5+ call sites, keep getting bitten by SDK upgrades, or need cost control and centralized quality tracking.

Skip it if you have 1 or 2 LLM calls, you are just prototyping, or you want a full agent framework with a built-in memory and RAG layer.

Honest status

llm-ports is pre-release, currently at 0.1.0-alpha.5. The core architecture is stable with 250+ offline regression tests, but some adapter and agent paths are still being hardened (multi-turn agent in the Vercel adapter and retry-on-runtime-error both land in v0.2). The per-surface status is documented openly so you know what is solid before you adopt it.

Try it

npm install @llm-ports/core @llm-ports/adapter-anthropic @llm-ports/capabilities

npm: https://www.npmjs.com/package/@llm-ports/core
GitHub (7 runnable examples, including email triage and PDF extraction): https://github.com/baabakk/llm-ports
Docs: https://baabakk.github.io/llm-ports/

If the capability-factory pattern matches how you are building, I would genuinely like feedback in GitHub Discussions. What shapes are you reimplementing that are not on the list of seven? What knobs do the capabilities need that they do not have yet?

The LLM stops being a dependency you manage. It becomes infrastructure you configure. Once you make that shift, everything else gets simpler.

Based on two longer write-ups: Ports and Adapters for AI and The 7 LLM Capabilities Every Production AI System Reimplements.