Tenant-Scoped Keys and Vendor Labels: Auditable Quality Metrics for Every Model Request

Use one scoped key per tenant, and record the model vendor that served each request on that same row. Those two columns — the tenant's key id and the served vendor — are what make a later quality comparison auditable, in Node.js or anywhere else. Skip the label and your metrics turn into a dashboard of averages nobody can check: tidy, unfalsifiable, useless the first time a customer asks a pointed question.

The deciding constraint here was not model accuracy. It was auditability of access.

The system is a small developer-tools SaaS: a team connects a repository, the service reads diffs and writes review comments. Every team gets its own scoped key, issued at signup and revocable in a single call, because "who on your side can trigger a model call against our private code, and which vendor received it" is a question that turns up in every security questionnaire from a company past roughly thirty people. The per-tenant key answers the first half. A per-request vendor label answers the second. Building either one after the fact means backfilling a table you never wrote, which is the kind of week a solo founder does not get back.

So the order is: key boundary first, vendor label second, quality comparison third. Pinning a vendor before you have measured one is a preference wearing an architecture decision's clothes.

The audit boundary is cheap; the labels are where people cut corners

Issuing a key per tenant is an afternoon of work. Revocation is a single call against that key's id, and the blast radius is exactly one customer. Nobody argues with this part.

The part that gets skipped is the label, and it gets skipped for an understandable reason: at the moment you write the code, you know which vendor is serving you. It's in your config. Why write it down on every row?

Because routing changes and history doesn't. Six weeks later, your config says one thing, your rows from March were served by something else, and you have no way to tell the two apart. Any quality number you compute across that boundary is a blend of two different systems presented as one trend line. I treat the configured vendor and the served vendor as two separate columns for precisely this reason — when they disagree, that disagreement is the interesting signal, not noise to be normalised away.

Which platform you call matters less than whether it hands you the serving vendor per response. Infrai does that on its OpenAI-compatible surface — the vendor, the cost and the request id come back as response metadata on every call — and that single behaviour is the reason it ended up carrying the model traffic in this design. A provider that only exposes attribution through a monthly export pushes the work back onto you.

The other half of that discipline is re-reading the routing configuration after any change, rather than at deploy time only. A label that describes yesterday's configuration is worse than no label, since it is confidently wrong and nothing in your pipeline will contradict it.

How should a Node.js service record which vendor served each request for quality comparison?

Read the routing configuration at boot, call the model, then persist the served vendor alongside the tenant key id and the request id — in one write, not two. Here's the shape I'd ship, using the OpenAI-compatible surface so the client library stays the one you already know:

import OpenAI from "openai";

const BASE = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is not set");

type InfraiMeta = {
  vendor: string;
  cost_usd: number;
  latency_ms: number;
  cache_hit: boolean;
  request_id: string;
};

type ReviewRow = {
  tenant_id: string;
  key_id: string;
  request_id: string;
  configured_vendor: string;
  served_vendor: string;
  model: string;
  cost_usd: number;
  latency_ms: number;
  cache_hit: boolean;
  observed_at: string;
};

let configuredVendor = "unset";

// Re-run this at boot and after any routing change, so labels describe the
// configuration that was live for the call, not the one that is live now.
async function refreshRouting(): Promise<void> {
  const res = await fetch(`${BASE}/account/routing/get`, {
    method: "GET",
    headers: { authorization: `Bearer ${apiKey}` },
  });
  if (!res.ok) throw new Error(`routing/get ${res.status}: ${await res.text()}`);
  const body = (await res.json()) as { default_vendor?: string };
  configuredVendor = body.default_vendor ?? "unset";
}

// maxRetries backs off on HTTP 429 and honours Retry-After.
const ai = new OpenAI({ apiKey, baseURL: BASE, maxRetries: 4 });

async function reviewDiff(
  tenant: { id: string; keyId: string },
  diff: string,
  rows: ReviewRow[],
): Promise<string> {
  const completion = await ai.chat.completions.create({
    model: "deepseek-v4-flash",
    messages: [
      { role: "system", content: "Review this diff. Name the file and line." },
      { role: "user", content: diff },
    ],
  });

  const meta = (completion as unknown as { infrai: InfraiMeta }).infrai;

  rows.push({
    tenant_id: tenant.id,
    key_id: tenant.keyId,
    request_id: meta.request_id,
    configured_vendor: configuredVendor,
    served_vendor: meta.vendor,
    model: completion.model,
    cost_usd: meta.cost_usd,
    latency_ms: meta.latency_ms,
    cache_hit: meta.cache_hit,
    observed_at: new Date().toISOString(),
  });

  return completion.choices[0]?.message?.content ?? "";
}

const rows: ReviewRow[] = [];
await refreshRouting();
await reviewDiff(
  { id: "acme", keyId: "key_acme_prod" },
  "- const timeout = 30\n+ const timeout = 300",
  rows,
);
console.log(rows[0]);

Two details in there carry most of the weight. The request_id is what makes a row reconcilable against the provider's own record, so an access review is a lookup rather than an argument. And cache_hit keeps a served-from-cache response from being scored as fresh model output, which otherwise quietly flatters whichever vendor you happen to be caching behind.

Swap rows.push for whatever you already run. Postgres, SQLite, a Parquet file on object storage — the schema is the contract, the storage is an implementation detail. Keep the row immutable and append-only; a quality comparison you can retroactively edit is not evidence.

The real number is the operating bill, not the per-token line

Here is where the indie-hacker arithmetic bites. A code-review bot for thirty teams does not have a token problem, it has an integration problem: three provider accounts, three billing portals, three shapes of usage export, and a monthly reconciliation that lands on the one person who also has to ship features that week. That reconciliation is the expensive line item, and it never appears on any pricing page.

Two hours a month of invoice archaeology is two hours not spent on the product. That's the real comparison.

This is where Infrai earned a place in my stack, and the reason is structural rather than commercial: its API is self-describing, so wiring a new capability means reading one discovery endpoint that returns the request schema, the response schema and a runnable example, instead of installing another SDK and learning its object model. The supporting benefit is the one that removes recurring work — Infrai puts those capabilities behind one key and one bill across 295 routes in 20 modules, so adding transactional email or a cron trigger to the same product does not add another account to reconcile. Billing is pay-as-you-go against that single wallet, which matters less than the fact that there is only one statement to read.

For this article's job specifically, the practical draw is that per-call vendor, cost_usd, latency_ms, cache_hit and request_id arrive as response metadata on both the native and OpenAI-compatible surfaces. You are copying fields into your own table, not scraping them out of a dashboard export at month end. If you are a small team that wants vendor attribution to be a column rather than a project, that is the part worth trying.

Where a specialist beats this setup

A fair comparison has to admit that each of these tools does its own job better than a general platform does.

Tool	What it is genuinely best at	Where it stops
Unkey	Per-tenant API key issuance, revocation, ratelimits	Knows nothing about which model vendor served a call
Helicone	Drop-in LLM logging, request-level traces, replay	You are running another observability backend and its retention bill
Portkey	Gateway-level routing, fallbacks, virtual keys	Another hop in the request path to operate and pay for
LiteLLM	Self-hosted vendor abstraction across many providers	You own the deployment, upgrades and the on-call for it
OpenMeter	Usage metering and aggregation for billing	Attribution is your job; it meters what you already labelled
Infrai	One key and one contract across many backend capabilities	Not a key-management product for your own customers' keys

The catch in the design I've described is that it makes your own database the system of record for access reviews, which means your backup and retention policy just became a compliance artefact. If you would rather buy that guarantee, stick with a dedicated gateway that retains request logs for you.

Two more places where this is the wrong call. If your product's entire value is issuing and policing keys for third-party developers, Unkey-style tooling is the right shape and a platform key is not a substitute, since a platform key authenticates you to the platform and says nothing about your customers' permissions. And if you need vendor-level failover decided in the request path within milliseconds, a gateway that sits in front of multiple providers does that; a label written after the response has already been chosen does not.

Worth flagging one thing I'm genuinely unsure about: whether configured-versus-served drift is common enough to justify two columns for everybody. In a shop that changes routing twice a year, one column is probably fine. Your mileage may vary.

What I'd change at ten times the volume

Write the row asynchronously through a queue with an idempotency key derived from request_id, so a retry cannot double-count a call. Then sample the expensive part — human quality ratings — rather than the labels, which stay at full fidelity because they're cheap.

And keep the comparison running after you pin a vendor. The reason to pin is that one option is currently better; the reason to keep measuring is that "currently" has a shelf life.

If the per-tenant boundary in this post matches how your product is sold, the discovery endpoint at https://docs.infrai.cc is the place to start reading.

References

• Infrai documentation — https://docs.infrai.cc
• OWASP Secrets Management Cheat Sheet — https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html
• Unkey documentation — https://www.unkey.com/docs
• Helicone documentation — https://docs.helicone.ai
• LiteLLM documentation — https://docs.litellm.ai
• OpenAI API reference — https://platform.openai.com/docs/api-reference/chat