DEV Community: David C Cavalcante

Static routing is a relic. #LangChain keeps you chained to manual configurations while costs mount. #ModelChain changes the paradigm. It routes prompts dynamically based on empirical cost, latency, and quality data. Drop it in. https://lnk.ua/XFl5MJBPl

David C Cavalcante — Mon, 01 Jun 2026 05:25:52 +0000

modelchain - measurable LLM router for Node, Edge & browser

Zero-dependency, drop-in router. Route one prompt across OpenAI, Anthropic, Gemini, or any OpenAI-compatible endpoint by cost, latency, and observed quality. Native streaming, tool calling, Vercel AI SDK adapter.

davccavalcante.github.io

ModelChain: Measurable LLM Router with Adaptive Model Selection, Real-Time Scoring, Budget Guards and Failover for Node.js, Edge and Browser

David C Cavalcante — Sat, 30 May 2026 22:01:10 +0000

ModelChain: Measurable LLM Router with Adaptive Model Selection, Real-Time Scoring, Budget Guards and Failover for Node.js, Edge and Browser

As a solo LLMOps engineer with over 25 years building production AI systems, I kept hitting the same limitation: when you have access to multiple LLM providers and models, choosing the right one for each request becomes fragile and outdated quickly.

Static if/else rules or fixed fallbacks do not survive real-world changes in pricing, latency, or model quality. Manual benchmarking is time-consuming and error-prone.

ModelChain (@takk/modelchain) was built to solve this.

The Problem

Developers and companies with keys for OpenAI, Anthropic, Gemini, Groq, and others waste time and money because they cannot dynamically route each prompt to the best available model based on current cost, observed latency, and actual response quality. Hard-coded choices quickly become suboptimal.

The Solution

ModelChain is a measurable, adaptive LLM router for Node.js, Edge runtimes, and browser. It selects the best model per request using seven routing strategies, scores every response in real time, feeds those scores back into future decisions, enforces hard budget guards, and includes per-model circuit breakers with automatic failover.

It normalises responses, tool calling, and streaming across providers while remaining zero-runtime-dependency and fully tree-shakable.

Core Features

Seven declarative routing strategies (cost-then-quality, cost-first, quality-first, etc.)
Six pluggable scorers (latency, token-budget, length-bound, regex-match, exact-match, schema-valid)
Native streaming over Web Streams with a unified CompletionChunk type
Normalised tool calling across OpenAI, Anthropic, and Gemini
Hard budget guard (per-request, per-task, daily ceilings) that throws before any network call
Per-model circuit breaker + full-jitter exponential backoff + automatic failover
EWMA health scoring that decays on failure and recovers on success
Thirteen in-process telemetry events (no external OpenTelemetry required)
Vercel AI SDK adapter (toVercelAILanguageModel)
CLI proxy, inspect, and bench modes
Six tree-shakeable entry points (core, providers, web, edge, ai-sdk, cli)
SLSA provenance on every release

Quickstart Examples

1. Basic Router Setup

import { createModelchain } from '@takk/modelchain';
import { openaiModel, anthropicModel, geminiModel } from '@takk/modelchain/providers';

const router = createModelchain({
  models: [
    openaiModel('gpt-4o-mini', {
      cost: { costPer1kInput: 0.00015, costPer1kOutput: 0.00060 },
      keys: process.env.OPENAI_API_KEY ?? '',
    }),
    anthropicModel('claude-3-5-haiku-latest', {
      cost: { costPer1kInput: 0.00080, costPer1kOutput: 0.00400 },
      keys: process.env.ANTHROPIC_API_KEY ?? '',
    }),
    geminiModel('gemini-2.0-flash', {
      cost: { costPer1kInput: 0.00010, costPer1kOutput: 0.00040 },
      keys: process.env.GEMINI_API_KEY ?? '',
    }),
  ],
  strategy: 'cost-then-quality',
  scoring: { built: ['latency', 'token-budget'] },
  budget: { perRequestUsd: 0.02, dailyUsd: 5 },
  telemetry: { enabled: true },
});

const response = await router.complete({
  prompt: 'Summarise X in 3 bullets.',
  maxTokens: 200,
});
console.log(response.text, response.finishReason, response.usage);

2. Streaming

for await (const chunk of router.stream({ prompt: 'Tell me a story.' })) {
  if (chunk.type === 'text-delta') process.stdout.write(chunk.delta);
  if (chunk.type === 'finish') console.log('\nDone:', chunk.finishReason, chunk.usage);
}

3. Vercel AI SDK Integration

import { generateText } from 'ai';
import { toVercelAILanguageModel } from '@takk/modelchain/ai-sdk';

const { text } = await generateText({
  model: toVercelAILanguageModel(router),
  prompt: 'Hello.',
});

4. Tool Calling (normalised)

const result = await router.complete({
  prompt: 'What is the weather in Tokyo?',
  tools: [ /* ToolDefinition shape */ ],
});

How It Works (Request Flow)

Select best model using chosen strategy and current health/scores
Pre-flight budget guard check
Dispatch request through normalised provider adapter
Classify response or error
Update EWMA health score and circuit breaker state
Score response quality and record for future routing
Emit telemetry events

All operations happen in-process with zero external dependencies.

Installation

pnpm add @takk/modelchain
# or
npm install @takk/modelchain
# or
yarn add @takk/modelchain
# or
bun add @takk/modelchain

Optional peer dependencies only if using richer typed adapters.

Why ModelChain Exists

ModelChain is the second building block (after KeyMesh) of a long-term family of high-reliability, open-source-first npm libraries for AI-native infrastructure that I plan to maintain through 2026–2030.

I built it because dynamic, measurable routing is the missing layer between raw LLM providers and production applications that care about cost, latency, quality, and reliability.

KeyMesh: Zero-Runtime-Dependency API Key Rotation, Circuit Breaker and Failover for Production LLM Applications in Node.js

David C Cavalcante — Sat, 30 May 2026 21:58:53 +0000

KeyMesh: Zero-Runtime-Dependency API Key Rotation, Circuit Breaker and Failover for Production LLM Applications in Node.js

As a solo LLMOps engineer with over 25 years of experience building production AI systems, I constantly faced the same critical failure point: API key rate limits and transient errors breaking LLM-powered applications.

KeyMesh was created to solve exactly this problem.

The Problem

When a single OpenAI, Anthropic or Gemini API key hits a 429 Too Many Requests (or any transient 5xx/408 error), most applications fail immediately for the user. Manual key rotation or on-call intervention becomes necessary. Existing gateway solutions add network hops, latency, and extra operational complexity.

I needed a solution that lives inside the application code itself.

The Solution

KeyMesh (@takk/keymesh) is a universal, zero-runtime-dependency Node.js library and CLI that provides intelligent API key rotation, per-key circuit breakers, smart retries, health scoring, and automatic failover.

It works as a drop-in replacement for official SDKs and supports any HTTP-based API.

KeyMesh is fully TypeScript-first, has 93% test coverage (145 tests), zero runtime dependencies, and ships with SLSA provenance for supply-chain security.

Core Features

Automatic key rotation using multiple selection strategies (round-robin, least-used, weighted, sequential-then-rotate, and custom)
Per-key circuit breaker with three states (closed, open, half-open)
Smart retry with AWS full-jitter exponential backoff and Retry-After support
Health scoring system (0-100) that decays on failure and recovers on success
In-process telemetry with 8 typed events (no external OpenTelemetry dependency)
Pluggable state backends (memory by default, file backend included; Redis/Postgres planned)
Auth-failure cooldown (401 errors disable key for 24 hours)
Official adapters for OpenAI, Anthropic, Gemini, and a generic HTTP adapter
CLI proxy mode for easy testing and non-Node.js environments

Quickstart Examples

1. OpenAI SDK Adapter

import { createKeymesh } from '@takk/keymesh';
import { openaiAdapter } from '@takk/keymesh/openai';

const client = createKeymesh({
  provider: openaiAdapter,
  keys: process.env.OPENAI_API_KEYS?.split(',') ?? [],
  strategy: 'least-used',
  circuitBreaker: { threshold: 3, cooldownMs: 30_000 },
  retry: { max: 5, baseMs: 200, jitter: true },
  telemetry: { enabled: true },
});

// Use exactly like the official OpenAI client
const response = await client.chat.completions.create({
  model: 'gpt-4.1',
  messages: [{ role: 'user', content: 'Hello.' }],
});

2. Generic HTTP Adapter (any API)

import { createKeymesh } from '@takk/keymesh';
import { httpAdapter } from '@takk/keymesh/http';

const tavily = createKeymesh({
  provider: httpAdapter({
    baseUrl: 'https://api.tavily.com',
    authHeader: (key) => ({ Authorization: `Bearer ${key}` }),
  }),
  keys: process.env.TAVILY_API_KEYS?.split(',') ?? [],
  strategy: 'round-robin',
});

const result = await tavily.post('/search', { query: 'AI infrastructure 2026' });

3. CLI Proxy Mode

OPENAI_API_KEYS=key1,key2,key3 npx @takk/keymesh start \
  --port 8787 \
  --adapter openai \
  --strategy round-robin

Then call it like a normal OpenAI endpoint on http://localhost:8787.

How It Works (Request Flow)

Pick key using selected strategy
Dispatch request through the provider adapter
Classify response/error
Update health score and circuit breaker state
Retry with backoff or rotate to next healthy key
Emit telemetry events

All keys remain hashed in state. Raw credentials are never logged or persisted.

Installation

pnpm add @takk/keymesh
# or
npm install @takk/keymesh
# or
yarn add @takk/keymesh
# or
bun add @takk/keymesh

Optional provider SDKs only if using the typed adapters.

Why KeyMesh Exists

I built KeyMesh because I got tired of production incidents caused by rate limits. It turns a common point of failure into silent, automatic self-healing.

It is the first piece of a larger family of high-reliability open-source libraries for the AI infrastructure stack that I plan to maintain long-term.

DEV Community: David C Cavalcante

Static routing is a relic. #LangChain keeps you chained to manual configurations while costs mount. #ModelChain changes the paradigm. It routes prompts dynamically based on empirical cost, latency, and quality data. Drop it in. https://lnk.ua/XFl5MJBPl

modelchain - measurable LLM router for Node, Edge & browser

ModelChain: Measurable LLM Router with Adaptive Model Selection, Real-Time Scoring, Budget Guards and Failover for Node.js, Edge and Browser

ModelChain: Measurable LLM Router with Adaptive Model Selection, Real-Time Scoring, Budget Guards and Failover for Node.js, Edge and Browser

The Problem

The Solution

Core Features

Quickstart Examples

1. Basic Router Setup

2. Streaming

3. Vercel AI SDK Integration

4. Tool Calling (normalised)

How It Works (Request Flow)

Installation

Why ModelChain Exists

Links

KeyMesh: Zero-Runtime-Dependency API Key Rotation, Circuit Breaker and Failover for Production LLM Applications in Node.js

KeyMesh: Zero-Runtime-Dependency API Key Rotation, Circuit Breaker and Failover for Production LLM Applications in Node.js

The Problem

The Solution

Core Features

Quickstart Examples

1. OpenAI SDK Adapter

2. Generic HTTP Adapter (any API)

3. CLI Proxy Mode

How It Works (Request Flow)

Installation

Why KeyMesh Exists

Links