Compass: Building a Real Multi-Model AI Inference Gateway

#ai #productivity #programming #dailybuild2026

How we treat models as infrastructure—and route each request like a map app chooses a road—in a **single Node.js app.

The problem with “just use GPT-4”

Most AI features hard-code one model. That works for demos until privacy, cost, latency, or outages force a better strategy.

Navigation apps do not always pick the shortest road. They weigh traffic, closures, and tolls. Compass applies the same idea to LLM inference:

Don’t ask which AI model is the best.

Ask which model is best for this request.

What we built

Compass is an inference control plane you can see:

Discover models from Ollama, LM Studio, OpenAI-compatible APIs, and OpenRouter
Route each prompt through a policy decision engine
Stream tokens with SSE (decision → steps → tokens → metrics)
Fail over when a provider errors
Compare models, edit rules, inspect analytics

One process: Next.js serves the UI and the gateway API. LangChain.js (ChatOpenAI) talks to every OpenAI-compatible backend. No Python sidecar.

Stack

Layer	Choice
App	Next.js 15 App Router
UI	React 19, TypeScript, Tailwind, Framer Motion, Recharts
Gateway	`frontend/src/server/*`
Model I/O	LangChain.js `@langchain/openai`
State (client)	Zustand
Persistence	`.compass-data/store.json`

CrewAI is Python multi-agent orchestration—the wrong layer for a router. LangChain.js is the right Node client for multi-backend chat.

Architecture

┌─────────────────────────────────────────────────────┐
│              Next.js (localhost:3000)               │
│   Browser  →  /api/*  →  discovery / policy / llm   │
└──────────────────────┬──────────────────────────────┘
                       │
        ┌──────────────┼──────────────┐
        ▼              ▼              ▼
     Ollama        LM Studio    OpenAI / OpenRouter

Pipeline

POST /api/chat/stream
  → PolicyEngine.decide(prompt)
  → SSE: decision + policy steps
  → LangChain ChatOpenAI.stream(...)
  → on error → alternate healthy model
  → SSE: metrics + done
  → persist history

How to run

npm install
npm run dev
# → http://localhost:3000

With Ollama or LM Studio running, models appear automatically. Optional keys in .env (see .env.example).

Code map

src/server/
  discovery.ts      # probe local + build cloud catalog
  policy-engine.ts  # IF/THEN + capability routing
  llm.ts            # LangChain streaming
  catalog.ts        # default policy pack
  store.ts          # history + rules
src/app/api/
  chat/stream/      # SSE
  models/ policies/ settings/ ...

LangChain client (sketch)

import { ChatOpenAI } from "@langchain/openai";
import { HumanMessage } from "@langchain/core/messages";

const chat = new ChatOpenAI({
  model: model.apiModel,
  apiKey,
  configuration: { baseURL }, // Ollama / LM Studio / OpenAI / OpenRouter
  streaming: true,
});

for await (const chunk of await chat.stream([new HumanMessage(prompt)])) {
  // stream tokens to the client
}