diegosouzapw/OmniRoute is an MIT-licensed AI gateway that advertises one endpoint across many providers and models, with quota-aware fallback and token compression.
A fallback can keep an endpoint available while changing the behavior behind it. For agentic workloads, that is a semantic failover problem.
Assumptions
Consider one request routed through a primary model and a fallback model. Both expose an OpenAI-compatible interface, but they may differ in:
- system-role precedence;
- tool-call JSON shape;
- context limits;
- streaming event order;
- refusal behavior;
- tokenization and truncation.
HTTP success does not prove workflow equivalence.
Versioned route envelope
route_id: coding-agent-v3
primary: provider-a/model-x
fallbacks:
- provider-b/model-y
max_context_tokens: 32000
required_capabilities:
- system_message
- tool_calls
- streaming
fixture_revision: 9b77c41
Pin provider/model IDs and a fixture revision. A generic alias makes later failures hard to reconstruct.
Three fixtures
1. Tool-call shape
Ask for one deterministic tool call and validate exact required fields. Reject prose disguised as JSON.
2. Context boundary
Send a prompt just below the declared route limit, with a canary near the end. Verify both primary and fallback models preserve it or return an explicit context error. Silent truncation fails.
3. Streaming terminal state
Force provider failure after stream start. The gateway must not merge partial primary output with a fresh fallback response under one apparent completion. Emit an explicit retry or route-change event.
Event sequence
request accepted
-> primary selected
-> primary quota failure
-> fallback selected
-> response started
-> response completed
Every event needs request ID, route revision, provider/model, attempt number, and terminal status. Consumers should deduplicate by request plus attempt, not by text content.
Acceptance matrix
| Property | Primary | Fallback | Required |
|---|---|---|---|
| Valid tool schema | yes | yes | yes |
| System instruction honored | yes | yes | yes |
| Canary retained | yes | yes | yes |
| Stream has one terminal event | yes | yes | yes |
| Route identity observable | yes | yes | yes |
Availability is allowed to degrade. Semantics are not allowed to change silently.
Failure policy
If a fallback lacks a required capability, fail closed with a typed route_capability_unavailable error. Sending a lower-quality but syntactically successful answer may be worse than a visible outage when downstream tools can mutate state.
Limitations
I have not benchmarked OmniRoute, verified its provider count, or tested its compression claims. This protocol is derived from the repository's stated gateway and fallback role plus standard distributed-systems invariants. Confirm current configuration syntax in the project documentation.
A multi-provider gateway is reliable when failover remains observable, bounded, and semantically compatible—not merely when it returns status 200.
Top comments (0)