DEV Community

Ahmed Nafies
Ahmed Nafies

Posted on

Testing OmniRoute locally: verify your AI gateway actually works before you trust it

#ai #api #llm #testing

OmniRoute is a local AI gateway: it puts one OpenAI-compatible endpoint (http://localhost:20128/v1) in front of hundreds of providers, then routes requests between them with automatic fallback. Install it, point your coding tool at auto, and it "just works."

But "it starts" and "it works" are not the same thing. A gateway that answers Hello! but falls over on streaming, tool calls, or a provider outage is decorative, not functional. This article is a checklist for proving a fresh local install actually does its job before you wire a coding agent into it.

1. Install and boot it 

npm install -g omniroute
omniroute
Enter fullscreen mode Exit fullscreen mode

The server boots on port 20128, the dashboard opens at http://localhost:20128, and the API base URL is http://localhost:20128/v1.

If you use pnpm, add the native build flags: pnpm add -g omniroute@latest --allow-build=better-sqlite3 --allow-build=@swc/core.

A fresh install is zero-config: the free providers (OpenCode Free, Felo) are pre-wired into the auto combo, so a brand-new server answers out of the box with no API key.

2. Run local diagnostics (omniroute doctor)

OmniRoute ships a health checker that runs without starting the server:

omniroute doctor
omniroute doctor --json        # machine-readable output
omniroute doctor --no-liveness # skip live checks
Enter fullscreen mode Exit fullscreen mode

Treat this as the first gate. If doctor reports a broken runtime, native module, or DB issue, fix it before touching the network.

Other useful CLI probes:

omniroute status            # offline dashboard: version, DB, tools, config
omniroute providers list    # what's actually connected
omniroute providers validate
Enter fullscreen mode Exit fullscreen mode

3. Verify the model list

A connected provider should show up in /v1/models. If you've created an API key on the Endpoints page, query with it:

curl http://localhost:20128/v1/models \
  -H "Authorization: Bearer YOUR_KEY"
Enter fullscreen mode Exit fullscreen mode

A non-empty list means the gateway can see providers. But don't stop here — a model list is not a working request.

4. Smoke-test a real request

The fastest end-to-end proof:

curl http://localhost:20128/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"auto","messages":[{"role":"user","content":"Hello!"}]}'
Enter fullscreen mode Exit fullscreen mode

You can also call a specific free backend directly (e.g. oc/... for OpenCode Free) to isolate one provider, then graduate to auto and let the router pick.

Each response carries an X-OmniRoute-Decision header naming the strategy, provider, and latency that served it — use it to confirm routing is doing what you expect.

5. Test streaming

Many tools depend on streaming. Send the same request with "stream": true:

curl -N http://localhost:20128/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"auto","stream":true,"messages":[{"role":"user","content":"Count to ten."}]}'
Enter fullscreen mode Exit fullscreen mode

You should see data: chunks arriving incrementally rather than one blob. A gateway that only works in non-streaming mode will break most agent CLIs.

6. Test tool calling

Coding agents live on tool/function calling. Verify it round-trips:

curl http://localhost:20128/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "What is the weather in Lisbon?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }]
  }'
Enter fullscreen mode Exit fullscreen mode

A valid reply should include a tool_calls block with correctly structured arguments, not a plain-text answer pretending to be a tool call.

7. Test fallback (the whole point)

The reason you run a gateway is resilience. To prove it:

  1. Create a combo on the dashboard with two or more providers.
  2. Watch the X-OmniRoute-Decision header to confirm the primary provider is serving.
  3. Intentionally break the primary (remove its key, exhaust its quota, or disable it), then repeat the request.

The request should still succeed, now routed to the next provider — silently, with no error surfaced to your client. If it 500s instead, your fallback chain isn't wired the way you think.

8. Point a real tool at it

Once the API surface checks out, connect an actual agent:

Base URL: http://localhost:20128/v1
API Key:  [copy from the Endpoints page]
Model:    auto
Enter fullscreen mode Exit fullscreen mode

Or let OmniRoute write the config for you, per tool:

omniroute setup-codex      # ~/.codex/<name>.config.toml profiles
omniroute setup-claude     # ~/.claude/profiles/<name>/settings.json
omniroute setup-opencode   # opencode.json
omniroute setup-cursor     # prints Cursor's in-app steps
Enter fullscreen mode Exit fullscreen mode

Even simpler — launch a CLI through the gateway with no config written:

omniroute run claude   --model auto
omniroute run codex    --model auto
Enter fullscreen mode Exit fullscreen mode

Then do a real task (edit a file, run a tool) rather than a hello-world. A finished task proves the whole pipeline.

Verification checklist

Check Pass condition
omniroute doctor No failing checks
curl /v1/models Providers listed
Plain chat Valid completion + X-OmniRoute-Decision header
Streaming Incremental data: chunks
Tool calling Structured tool_calls in the response
Fallback Request survives a dead primary provider
Real agent task A coding CLI completes an actual edit

When a check fails

  • doctor complains about a native module — reinstall with the correct build flags for your package manager.
  • Model list empty but provider shows connected — verify the key on the Endpoints page is the one you're sending.
  • Works non-streaming, breaks streaming — check timeouts; streaming stalls are governed by STREAM_IDLE_TIMEOUT_MS.
  • Fallback doesn't trigger — confirm the broken provider actually fails with an error OmniRoute can act on (4xx/5xx), and that the combo has more than one step.

Ten minutes of this checklist beats an hour of debugging why your agent silently "can't reach the model" in the middle of real work.

Top comments (0)