What I learned building one API in front of 70+ AI models (LLM, image, video, music)

Alterpro — Tue, 21 Jul 2026 14:14:47 +0000

Disclosure up front: I work on you.bot, the gateway described here. This is a build-log about the engineering problems, not a sales pitch — the patterns apply whether you build your own gateway or use one.

The problem that started it

Every AI feature I shipped came with its own tax. The LLM had one SDK and one dashboard. The image model had another. Video was a third provider with a totally different async flow. Music was a fourth. Four SDKs, four invoices, four sets of retry logic, four ways to handle failures.

The moment I wanted to A/B two image models, or swap an LLM for a cheaper one, I was rewriting integration code and reconciling another bill. So we built a single endpoint that fronts 70+ models across text, image, video and music, where switching models is a one-line change. Here's what turned out to be hard.

1. Every provider has a different idea of "a request"

LLMs are mostly request/response. Image is sometimes sync, sometimes not. Video is almost always a long-running job — you submit, you poll or wait for a webhook, and it can take minutes. Music (like Suno) is the same: you don't get a song back on the open socket.

If you expose these differences to the caller, you've moved the complexity onto them. The decision that mattered most: normalize everything to one create-then-poll task shape. You create a task, then poll it until it succeeds (text models return their result inline in the create response, so you can skip the poll):

POST https://you.bot/api/v1/generate
Authorization: Bearer YOUR_API_KEY

{ "modelId": "suno-v4-5",
  "input": { "prompt": "lofi beat, rainy night", "title": "Rainy Lofi", "style": "lofi, chill" } }

→ { "taskId": "t_abc", "creditsCharged": 13 }

GET https://you.bot/api/v1/task/t_abc?model=suno-v4-5

→ { "state": "success", "resultUrls": ["https://…"] }

Same two-call contract for a fast LLM call and a three-minute video render. Callers can poll, or pass a callbackUrl and get a signed webhook on completion. Switching from music to video (Kling) or image (GPT Image 2) is just a different modelId and a different input.

2. The modelId is the only thing that should change

The whole point is that trying a new model shouldn't be a project. The surface a caller touches is deliberately tiny: one endpoint, one Authorization header, and a modelId.

import requests

def generate(model_id, inp):
    r = requests.post(
        "https://you.bot/api/v1/generate",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={"modelId": model_id, "input": inp},
    )
    return r.json()   # -> {"taskId": "...", "creditsCharged": ...}

# swap models by changing one field
# (input fields vary per model — each model page documents its own)
generate("gemini-3-1-pro",  {"prompt": "..."})
generate("nano-banana-pro", {"prompt": "..."})
generate("kling-3-0",       {"prompt": "..."})

Every model has a page with its own inputs, price and an in-browser playground (e.g. Gemini 3.1 Pro, Nano Banana Pro) so you can test with your own prompt before writing a line of integration code. That "try before you wire it up" step removed most of our support questions.

3. Only charging for successful output changes your whole billing model

If a generation fails, errors, or comes back empty, the user should not pay for it. Sounds obvious; it's annoying to implement because "success" is defined differently per provider, and refunds have to be idempotent so a retried webhook doesn't double-credit.

We ended up with a per-model success predicate and a ledger where every debit can be reversed by task id. Credits come from one prepaid wallet shared across all models (the create response tells you creditsCharged per call), so there's no per-provider balance to juggle — and because you only pay on success, the effective price drops below the sticker price.

4. Reliability: keep the caller's existing key as a fallback

The honest objection to any gateway is "now you're a single point of failure between me and the model." The pattern that made people comfortable: let the app keep its existing provider key as a fallback route. Primary traffic goes through the gateway; if a call fails, it falls back to the provider directly. Multi-provider routing targets high uptime, but the caller never bets reliability on us alone.

5. Make the docs machine-readable

A lot of "how do I call X model" questions now start in ChatGPT or Perplexity, not Google. So every model page also ships a plain-markdown version (append /md to the URL, e.g. /models/suno-v4-5/md) that an LLM can read cleanly. If assistants are going to answer developer questions anyway, the least you can do is give them accurate, structured source material.

What I'd tell my past self

Normalize to one create-then-poll task shape early. Retrofitting it later is painful.
Define "success" per provider before you promise charge-on-success billing.
Make refunds idempotent from day one.
A playground per model is worth more than another paragraph of docs.
Assume an LLM will read your docs before a human does.

The model catalog with per-model pricing and playgrounds is at you.bot/market. Happy to answer architecture questions in the comments — especially on fallback-routing and charge-on-success, since those drove the most debate on our side.

DEV Community: Alterpro