DEV Community

Sonam Gupta
Sonam Gupta

Posted on

Build Observable AI Model Fallbacks with Telnyx AI Gateway

#python #ai #llm #telnyx

An AI model fallback is only useful if your application can answer two questions afterward:

  1. Which model actually served the request?
  2. How many attempts did the gateway make?

A client-side retry loop can hide both answers. The application eventually receives a successful response, but the path to that response is difficult to reconstruct. SDK retries can also stack on top of gateway retries, creating extra requests and misleading attempt counts.

The gateway-fallbacks Python sample moves that resilience policy into Telnyx AI Gateway. It provisions a primary and fallback model, forces a timeout on the primary, and proves the resulting failover twice:

  • response headers show the model and attempt count on the live request
  • the spend ledger records the requested model, serving model, and attempts together

The architecture

The example separates management, inference, and observation:

Management plane                         Inference plane
api.telnyx.com/v2/llm_token_gateway     llm.telnyx.com/v1
Telnyx account API key                  scoped token key

POST /token_groups                      GET /models
POST /token_keys                        POST /chat/completions
GET  /spend/events                             |
GET  /usage/summary                            v
        |                               gateway retries and fallbacks
        +----------------------+----------------+
                               |
                               v
Observation: x-ltg-served-model, x-ltg-attempts,
             requested_model, model, attempts, cache_hit
Enter fullscreen mode Exit fullscreen mode

The management plane owns token groups, token keys, budgets, and usage reporting. It uses the Telnyx account API key.

The inference plane is OpenAI-compatible. It uses only the token key issued for the group, so application code does not need the account credential.

Put the fallback policy in the token group

The sample provisions two allowed model aliases and one fallback relationship:

allowed_models = ["Kimi-K3", "GLM-5.3-Flash"]
fallbacks = {
    "Kimi-K3": ["GLM-5.3-Flash"],
}
retries = {
    "max_attempts": 2,
    "retry_delay_ms": 250,
    "backoff": "exponential",
}

group = client.llm_token_gateway.token_groups.create(
    name="gateway-fallbacks-demo",
    allowed_models=allowed_models,
    fallbacks=fallbacks,
    retries=retries,
    max_budget=10,
    budget_duration="1d",
    idempotency_key=f"gateway-fallbacks-group-{uuid.uuid4()}",
).data
Enter fullscreen mode Exit fullscreen mode

The group constrains which models its key can call and defines how failures move through the chain. The sample validates the same policy limits locally before making the API request:

  • fallback-chain depth cannot exceed three
  • max_attempts must be between one and five
  • a chain cannot reference its own primary or contain duplicates
  • every fallback must appear in allowed_models
  • max_attempts × (1 + longest chain) cannot exceed eight

The gateway also enforces these rules. policy.py deliberately sends an invalid max_attempts=9 update and expects an HTTP 400 without mutating the group.

Keep retries in one layer

The gateway owns retries and model fallbacks, so the OpenAI client disables its own retry behavior:

llm = OpenAI(
    base_url="https://llm.telnyx.com/v1",
    api_key=token_key,
    max_retries=0,
)
Enter fullscreen mode Exit fullscreen mode

This matters because an SDK retry is a new request layered on top of the gateway's attempts. Disabling it keeps the response headers, ledger records, and billing aligned around one retry policy.

How the gateway treats failures

The sample distinguishes failures where no model work occurred from failures where processing may already have started.

Definite no-work failures such as pre-dispatch errors, 429, and 529 can retry the same model.

Uncertain failures such as timeouts, 408, 500, 502, 503, and 504 move to the next fallback because the original request may already have reached the provider.

A context-length rejection is terminal because sending the same oversized context to another attempt does not fix the input.

Force a model fallback for the demo

Waiting for a provider incident is not a useful test plan. The sample creates a controlled race instead.

inference.py sends a streaming request to Kimi-K3 with a one-second first-event deadline:

result = chat_completion(
    token_key,
    "Kimi-K3",
    build_forced_failure_prompt(),
    extra_headers={
        "x-ltg-request-timeout": "1000",
        "x-ltg-max-attempts": "1",
    },
    stream=True,
    max_tokens=64,
)
Enter fullscreen mode Exit fullscreen mode

The generated input is intentionally heavy, approximately 22,000 tokens, and includes a unique nonce so a repeated run cannot rely on provider KV caching to shorten prefill. The goal is for the primary's first event to miss the one-second deadline while GLM-5.3-Flash completes as the fallback.

Provider latency changes over time, so this is not presented as an absolute guarantee. The script tries up to three times and reports honestly if the primary answers within the deadline or if both attempts time out.

Verify the live request through headers

The client reads two headers from the raw response:

served_model = raw.headers.get("x-ltg-served-model")
attempts = raw.headers.get("x-ltg-attempts")
Enter fullscreen mode Exit fullscreen mode

A successful failover should show:

requested model:   Kimi-K3
served model:      GLM-5.3-Flash
x-ltg-attempts:    2 or more
Enter fullscreen mode Exit fullscreen mode

The SDK response model is useful, but the gateway headers make the routing decision explicit on the live path.

Verify the same event in the spend ledger

The next step reads the gateway spend events for the issued token key:

events = client.llm_token_gateway.spend.retrieve_events(
    start_date=start_date,
    end_date=end_date,
    token_key_id=token_key_id,
    page_number=1,
    page_size=50,
)

for event in events.data:
    print(
        event.requested_model,
        "->",
        event.model,
        "attempts:",
        event.attempts,
        "status:",
        event.status,
        "cache_hit:",
        event.cache_hit,
    )
Enter fullscreen mode Exit fullscreen mode

When requested_model and model differ, the ledger provides the durable counterpart to the response header. The sample uses a half-open UTC date range and snapshot pagination so the report remains reproducible as the event set changes.

GET /usage/summary adds totals and per-model rollups when the account has the llm_token_gateway.usage.read permission. A 403 from that endpoint is treated as a nonfatal warning because the spend-events ledger still contains the request-level proof.

Per-request controls, caching, and mutation safety

The gateway supports request-level overlays for attempts, delay, backoff, and timeout:

extra_headers = {
    "x-ltg-max-attempts": "1",
    "x-ltg-retry-delay": "1000",
    "x-ltg-backoff": "exponential",
}
Enter fullscreen mode Exit fullscreen mode

These controls adjust one request without rewriting the token-group policy.

The sample also updates cache_ttl to 60 seconds and sends the same prompt twice. Gateway caching is best effort, so the script does not invent a cache hit: it reports x-ltg-attempts: 0 when the repeat is served from cache and otherwise directs you to the ledger's cache_hit field.

Every update and delete follows the mutation-safety pattern:

  • retrieve a fresh ETag
  • send it through If-Match
  • include an Idempotency-Key

A stale ETag returns 412, preventing one process from overwriting another process's newer configuration.

Run the sample

Clone the repository and install the pinned dependencies:

git clone https://github.com/team-telnyx/telnyx-code-examples.git
cd telnyx-code-examples/gateway-fallbacks

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

cp .env.example .env
Enter fullscreen mode Exit fullscreen mode

Add your Telnyx account API key to .env:

TELNYX_API_KEY=your_telnyx_api_key_here
PRIMARY_MODEL=Kimi-K3
FALLBACK_MODEL=GLM-5.3-Flash
Enter fullscreen mode Exit fullscreen mode

Run the local smoke tests before making network requests:

python -m pytest smoke_test.py -v
Enter fullscreen mode Exit fullscreen mode

Then run the five-step demonstration:

# Creates the group and key. The key secret is printed once.
python provision.py

export GATEWAY_TOKEN_KEY="<paste the secret>"

python inference.py
python ledger.py
python policy.py
python teardown.py
Enter fullscreen mode Exit fullscreen mode

The live demonstration incurs real inference spend. The sample configures a maximum budget of USD 10 over one day, but you should review the policy before running it in your account.

provision.py stores only the group and key IDs in .gateway_state.json; it never writes the token secret to disk. teardown.py revokes the token key, deletes the group, verifies that the revoked key receives HTTP 401, and removes the local state file.

The reusable pattern

The useful pattern is not limited to these two models:

  1. Put model access, retry rules, fallbacks, and budgets in gateway policy.
  2. Give applications a scoped token key instead of the account credential.
  3. Disable overlapping client retries.
  4. Read routing evidence from the response headers.
  5. Confirm the same event in the spend ledger.

That turns model failover from an assumption into something operators can inspect during the request and audit afterward.

Resources

Top comments (0)