Ollama, vLLM & Private LLMs: Routing Your Own Models Next to Your Cloud Keys

You have cloud keys in your app, and a GPU box in the corner running vLLM or Ollama. Here's how SentioGate puts both behind one endpoint, with one request format, one stream format and one dashboard.

A question I get a lot: "I use the cloud for the hard tasks, but I have my own box for the easy stuff. Can one router handle both, without a separate integration for each?"

Yes, as long as your box speaks the OpenAI chat-completions API. vLLM, Ollama, LM Studio and most private inference appliances do.

The problem with "just call the API"

Most teams end up with something like this:

if task_difficulty > 0.7:
    response = openai_client.chat(...)      # cloud
else:
    response = ollama_client.generate(...)  # local

That gives you two SDKs, two error shapes, two streaming formats, no failover and no shared telemetry. Every new model means another branch, and every outage means a code change.

A router fixes the plumbing: one endpoint, one request format, one stream format, and one place to see what ran and what it cost.

Registering your endpoint

In SentioGate, your own endpoint is a custom provider. It's a provider_id, a base_url, an api_key, and the list of model IDs it serves.

curl -X POST https://api.sentiogate.dev/v1/custom-providers \
  -H "Authorization: Bearer $SENTIOGATE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "provider_id": "my-vllm",
    "display_name": "My vLLM",
    "base_url": "https://llm.example.com/v1",
    "api_key": "YOUR_ENDPOINT_KEY",
    "models": ["llama3.1:8b", "qwen2.5:14b"]
  }'

A few things to get right:

• Make it reachable. The hosted service calls your model from the edge, so base_url must be reachable from the internet. A URL like http://localhost:11434 or a LAN hostname won't work. Put the endpoint behind HTTPS.
• Protect it. An inference server on the open internet should require a key. vLLM supports --api-key, and for Ollama a small reverse proxy that checks a bearer token does the job.
• Put the credential in api_key. It's encrypted at rest with AES-256-GCM, like every other key in your Vault, and decrypted in memory only when SentioGate calls your endpoint.

Calling your models

Once registered, you can call a model explicitly as provider_id/model:

from ai_auto_router_client import AutoRouterClient

router = AutoRouterClient(api_key="sg_live_...")

resp = router.chat(
    model="my-vllm/qwen2.5:14b",
    messages=[{"role": "user", "content": "Summarise this ticket in one line: ..."}],
)
print(resp["choices"][0]["message"]["content"])

Or from JavaScript:

import { AutoRouter } from '@hushamdev/ai-auto-router-sdk';

const router = new AutoRouter({ apiKey: process.env.SENTIOGATE_API_KEY });
const res = await router.chat({
  model: 'my-vllm/qwen2.5:14b',
  messages: [{ role: 'user', content: 'Summarise this ticket in one line: ...' }],
});
console.log(res.choices[0].message.content);

Your models also become candidates for "model": "auto", right next to the cloud models your keys can reach.

How auto treats your models

With auto, SentioGate picks a model per request from everything you've enabled. It looks at whether a model is capable enough for the task, whether the conversation fits its context window, what it costs, and how fast and reliable it has been recently. That last part is measured, not configured: if your box slows down or starts erroring, it drops down the list, and requests fall back to the next candidate before any bytes reach your client.

You stay in control of the pool:

• Want your own model as the primary? Call it explicitly (my-vllm/...), or enable only your own models in the Vault so auto chooses among them.
• Want cloud and local side by side? Enable both, and auto picks per request across the whole pool, with fallback between providers.
• Want guardrails? Your policy (enabled models and spend caps) applies to every request, whichever provider serves it.

And you never have to guess what happened. Every response says which model was picked and why, in the router block, or in the X-SentioGate-Router header when streaming.

Everything else in the pipeline still applies

Your models go through the same request path as the cloud ones:

• Streaming: standard OpenAI-style SSE, so your front end doesn't care where the tokens came from.
• Fallback: on auto requests, a failed candidate moves on to the next one.
• Context compression: long chats are kept inside a token budget, exactly as for cloud models, and router.compression in the response shows what was saved.
• Telemetry: provider, model, latency and tokens show up in the same dashboard as everything else.

Streaming from your own model looks exactly like streaming from OpenAI:

for await (const chunk of router.streamChat({
  model: 'my-vllm/qwen2.5:14b',
  messages: [{ role: 'user', content: 'Write a haiku about GPUs' }],
})) {
  process.stdout.write(chunk.choices?.[0]?.delta?.content || '');
}

(Python: router.stream_chat(...).)

The hybrid pattern

The setup I'd recommend:

1. Cloud models for the tasks that need them: hard reasoning and long context.
2. Your own models, called explicitly for the workloads you know they handle well, such as classification, extraction, short summaries and internal tools.
3. auto wherever you want SentioGate to choose per request and fail over automatically.

You can mix all three in one app with one client, and pass cloud keys per request (provider_keys / providerKeys) if you'd rather not store them. Keys passed that way are used for that request only.

It's a simple mental model: explicit when you know what you want, auto when you want the router to decide, and one response format either way.

Want everything, router included, on your own infrastructure? The source isn't public yet, so get in touch.

Links

• SentioGate · Docs · Pricing
• Python SDK (ai-auto-router-client) · JS/TS SDK (@hushamdev/ai-auto-router-sdk)

Running a local box next to your cloud keys? Tell me in the comments what you serve on it. I'm curious which workloads people keep on their own hardware.