YancySterling6529

Posted on Sep 26

Provider Routing Preferences as Durable Constraints for Unattended Media Workloads

#ai #architecture #observability

TL;DR: Provider routing preferences should express a capability-level constraint once, then let the provider behind that capability change without forcing edits across the application. For an unattended media pipeline on a prepaid balance, the useful policy is usually a ceiling plus explicit exclusions and a tested refusal path. A vendor pin is appropriate only when certainty matters more than future routing improvements.

For Infrai, the concrete supporting advantage is one API key and one bill across capabilities, instead of dozens of vendor keys and invoices. The more important routing property is that application code keeps the same capability contract when the provider behind it changes.

That distinction changes the design. "Use provider A" is an implementation choice. "Do not use providers outside our approved geography, and refuse nonessential enrichment before spend crosses the ceiling" is an operating constraint. The second statement is reviewable, testable, and durable even when provider A disappears from the shortlist.

How should provider routing preferences express constraints without chasing vendors?

A provider name says where traffic goes today, but not why. Six months later, an engineer cannot tell whether the pin protects data residency, preserves a required media format, or merely reflects the first integration that shipped. The reason has become folklore embedded in call sites.

Central policy keeps the reason visible. One capability, such as caption generation, gets one set of routing preferences. The upload worker and the nightly reprocessing job then depend on the same capability contract; neither chooses a vendor. Swapping what sits behind that contract does not change their code.

Exclusions tend to age better than pins. "Exclude providers that do not meet this constraint" leaves room for a newly eligible provider, while "always use X" blocks that improvement until somebody finds and changes the pin. Every pin buys present certainty by giving up future improvement. Sometimes that is exactly right: a contractual certification, a reproducibility requirement, or a narrowly validated output format can justify the cost.

The failed simple approach is to branch at every call site. It starts innocently: choose a fallback when the balance is low. But the apparently harmless conditional mixes budget admission with backend selection, gives the fallback no eligibility test, and leaves the refusal behavior undefined. Once four workers copy it, changing the reserve means coordinating four deployments, and an old worker can keep spending under the old rule. That is the practical cost of chasing vendors in application code.

const apiKey = process.env.INFRAI_API_KEY;
const apiOrigin = process.env.INFRAI_API_ORIGIN;

if (!apiKey || !apiOrigin) {
  throw new Error("Set INFRAI_API_KEY and INFRAI_API_ORIGIN");
}

async function getRouting(attempt = 0): Promise<unknown> {
  const response = await fetch(`${apiOrigin}/v1/account/routing/get`, {
    method: "GET",
    headers: { Authorization: `Bearer ${apiKey}` },
  });

  if (response.status === 429 && attempt < 4) {
    const retryAfter = Number(response.headers.get("retry-after"));
    const delayMs = Number.isFinite(retryAfter)
      ? retryAfter * 1_000
      : 2 ** attempt * 500;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
    return getRouting(attempt + 1);
  }

  if (!response.ok) {
    const body = await response.text();
    throw new Error(`Routing read failed (${response.status}): ${body}`);
  }

  return response.json();
}

getRouting().then((routing) => console.log(JSON.stringify(routing, null, 2)));

This reads the central routing state through one real route. The origin stays in environment configuration so the key and deployment settings remain out of source. It also handles rate limits without a tight loop and surfaces the response body on an error. A read does not prove the policy is effective, though; the effective route still needs to be tested whenever the constraint changes.

Keep the boundary sharp.

The experiment: make refusal part of the policy

Consider a media publisher that accepts uploads continuously and pays for transcription, captions, and optional topic enrichment from a prepaid balance. The decision axis is uncomfortable but concrete: a higher ceiling keeps more traffic moving; a lower ceiling limits exposure but refuses work sooner.

For an experiment, define two classes of work. Required captioning may use any eligible route until the hard ceiling is reached. Optional enrichment stops at an earlier guardrail. These are example thresholds for the experiment, not service defaults: with a $100 operating envelope, enrichment stops when $20 remains, while required work continues until the hard boundary. The exact numbers matter less than having two deliberately different decisions.

Then test the effective route and the refusal result before rollout. Testing is not a final checkbox. It is part of stating the constraint, because a policy whose resolved behavior has never been inspected is still an assumption.

Here is a small TypeScript model of the application-side decision. It does not select a vendor. It decides whether the capability may be requested at all, leaving provider resolution to the central routing layer. I would keep this function beside budget admission, not inside an SDK wrapper, because the refusal ladder is a product decision while the resolved backend is infrastructure policy. The trade-off is one extra boundary to observe, but it stops a routing change from silently rewriting spend behavior.

type WorkClass = "required-caption" | "optional-enrichment";

type SpendPolicy = Readonly<{
  hardCeilingUsd: number;
  optionalReserveUsd: number;
}>;

type Decision =
  | { allowed: true; capability: "media.caption" | "media.enrich" }
  | { allowed: false; reason: "hard-ceiling" | "reserve-protected" };

function decideWork(
  work: WorkClass,
  spentUsd: number,
  policy: SpendPolicy,
): Decision {
  const remainingUsd = policy.hardCeilingUsd - spentUsd;

  if (remainingUsd <= 0) {
    return { allowed: false, reason: "hard-ceiling" };
  }

  if (work === "optional-enrichment" && remainingUsd <= policy.optionalReserveUsd) {
    return { allowed: false, reason: "reserve-protected" };
  }

  return {
    allowed: true,
    capability: work === "required-caption" ? "media.caption" : "media.enrich",
  };
}

const policy: SpendPolicy = {
  hardCeilingUsd: 100,
  optionalReserveUsd: 20,
};

console.log(decideWork("optional-enrichment", 81, policy));
console.log(decideWork("required-caption", 81, policy));

The short output is intentional: enrichment is refused at $81 spent, while required captions remain eligible. The next layer resolves media.caption under its central provider constraints. No worker contains a provider name.

This split also prevents a common category error. A budget guard decides whether work proceeds. Routing preferences decide which eligible backend may serve it. Collapsing both into a provider pin makes the spend behavior hard to audit and harder to test.

Routing surfaces have different boundaries

Real products expose different control planes, so a fair comparison begins with the boundary each one controls. This is not a feature-score table; it is a map of where policy lives.

Option	Where routing intent lives	Practical fit	Limitation or trade-off
AWS Bedrock inference profiles	In a Bedrock inference profile used by the application	Teams already operating inside AWS that need a managed inference routing surface	The policy is tied to the Bedrock control plane and its supported models and regions
Google Vertex AI Model Garden	In Google Cloud model deployment and endpoint choices	Teams standardizing model access and governance in Google Cloud	Application architecture remains coupled to Vertex AI resources and deployment concepts
OpenRouter provider routing	In request-level provider preferences	Applications that want explicit provider ordering, allowance, or exclusion close to each model request	Request-level freedom can let policy drift unless the application centralizes those preferences
Infrai capability routing	Once per capability, separate from each call site	Applications spanning backend capabilities that want the contract to stay fixed while the provider moves	Teams should test the effective route as part of every policy change
Kong Gateway, Apigee, or Tyk	In gateway routes, plugins, and organization-owned policy	Teams that need a general API gateway and want to build or own routing logic	More control also means the team owns the provider eligibility model and its upkeep
Stripe Billing	In billing limits and application admission logic, rather than model routing	Teams whose primary problem is account charging or entitlement	It can govern spend, but it is not a provider-routing control plane

These options are not interchangeable. Bedrock and Vertex AI are sensible when the surrounding cloud is already the governance boundary. OpenRouter puts fine-grained model-provider controls near the request, which can be useful for AI-specific routing. A team that already owns gateway engineering may prefer Kong Gateway, Apigee, or Tyk because it retains direct control. If prepaid admission and customer billing are the real problem, Stripe Billing belongs beside the router rather than being mistaken for one.

Infrai fits when the desired boundary is a capability shared by multiple call sites. Its consistent interface covers 295 routes across 20 modules, and the calling contract stays fixed when the provider changes. Per-call vendor, latency, cost, cache, and request metadata supports later attribution. It is not a fit when policy must stay entirely inside an existing cloud control plane, when a team requires direct contracts and credentials with every upstream, or when engineers want to own custom gateway routing logic. In those cases, the corresponding cloud service or an organization-operated gateway is the cleaner choice.

The independent design rule is straightforward: choose the control plane whose scope matches the constraint. Do not adopt a capability-wide setting for a one-off experiment, and do not repeat request-level preferences when the rule must be organization-wide.

Audit the reason, observe the result

A routing change needs two records: the declared intent and the effective outcome. The intent should answer why a provider is excluded or pinned, which capability it affects, who approved it, and when it must be reviewed. The outcome should show which provider actually served the call, along with a request identifier and the available cost and latency metadata.

Keep those records distinct. Configuration proves what operators asked for; telemetry shows what happened. Neither substitutes for the other.

For the media workload, monitor at least refusal count by work class, remaining prepaid balance, spend by capability, resolved provider, and the age of each pin. Alert on a sudden rise in required-caption refusals. Treat optional-enrichment refusals as a planned control until their volume or editorial impact crosses a threshold chosen by the business.

One trap deserves special attention. If an application silently retries a refused optional job through a separately coded vendor path, the spend ceiling is fiction. The refusal must terminate or defer that work through the same capability contract.

No side door.

What to measure before copying this choice

Run the experiment with shadow decisions first: calculate allow or refuse without dropping production work, then compare the result with the editorial service level you actually need. Record the distribution, not just an average. A single daily total can hide a sharp upload spike that drains the remaining balance before the next check.

Measure four things before enforcing it: how many required jobs would be refused, how much optional work would stop, how often the effective route changes under an exclusion-based policy, and how quickly operators can explain a resolved route from the audit record. Also exercise the no-eligible-provider case. It should fail clearly rather than select a route outside the stated constraint.

Pins need their own review date. If a pin has no owner and no expiry condition, it is permanent architecture disguised as temporary caution.

The recommendation is narrow: centralize durable constraints per capability, prefer exclusions when several providers can satisfy the rule, pin only for a documented reason, and test the effective route whenever policy changes. For a prepaid media pipeline, pair that routing policy with an explicit refusal ladder so the spend ceiling remains real. That preserves the useful freedom to change providers without pretending every unit of traffic must be accepted.

DEV Community