How to Benchmark a Unified LLM API: Node.js Invoice Extraction Under Latency Limits

RivenPulse5812 — Wed, 19 Aug 2026 02:16:26 +0000

Short answer: choose a unified LLM API for a Node.js invoice-extraction backend only after it passes the same field-level quality suite and latency budget as direct model access; one key and a tidy interface are useful, but they aren't acceptance criteria.

The deciding constraint is the gap between a plausible invoice and a usable support record. A response can look perfect while swapping invoice_number with purchase_order, dropping the currency, or turning an absent due date into a confident guess. Customer-support staff then inherit a fast mistake. I would rather make the gateway earn its place with a small repeatable test than compare provider feature grids.

This build log targets a simple backend that may send work to OpenAI, Claude, or Gemini through one credential. It doesn't rank those model families. The useful question is narrower: can the complete path meet the quality-versus-latency budget for the supplier documents the team actually receives in US and EU operations?

Measure that path first.

How should a Node.js backend benchmark a unified LLM API?

Start with an acceptance corpus, not a clever router. Twenty documents can expose basic contract mistakes, though they can't establish production accuracy. Keep the original text or OCR output, a reviewed expected record, and a few labels that describe why the document is hard. Useful labels for this job include multi-page, missing field, ambiguous date, comma decimal, and repeated totals. The labels are your debugging index — without them, an aggregate score hides the exact invoices that fail.

Don't let the model grade itself. For exact fields such as invoice number, currency code, and supplier ID, compare normalized values. For money, parse the value and currency separately before comparison. For dates, define which source date wins when an invoice contains issue, delivery, and due dates. Free-form supplier names may need an explicit alias table owned by the application. Every normalization rule belongs in version control because changing one changes the benchmark.

Use two budgets. The quality budget is a set of per-field thresholds plus a hard rule for critical omissions. The latency budget should include a percentile, not merely an average, because a handful of slow extractions can back up a support queue while the mean still looks respectable. Pick both limits from the workflow: how much correction the team accepts, and how long the caller can wait. I'm not sure a generic public benchmark can answer either question; resolving that uncertainty requires representative invoices and timings from the intended deployment path.

Keep the comparison controlled. Send identical input, schema instructions, sampling settings, and retry policy through each candidate path. Run enough warm and cold requests to see whether connection setup matters. Record the model alias returned by your own adapter, the gateway request ID, total duration, schema validity, and field results. A one-key gateway may simplify credential handling, but it still sits inside this measured path.

There is one easy trap here: “valid JSON” is not the same as “valid extraction.” These are distinct checks. A syntactically correct object with an invented due date passes the first and fails the second, so the evaluator needs both schema-level validation and expected-value assertions.

Implement the smallest useful evaluator

The application should own a narrow contract that doesn't leak a provider SDK into invoice logic. Put provider-specific request mapping behind an adapter process, then point this evaluator at that process with LLM_GATEWAY_URL. The URL is configuration rather than a hard-coded route, and MODEL_ALIAS is an application-controlled name rather than a guessed vendor model ID.

The fixture below is deliberately tiny. It proves the harness, not the model. Add reviewed fixtures before treating any score as evidence.

type Invoice = {
  invoice_number: string | null;
  supplier_name: string | null;
  currency: string | null;
  total: number | null;
  due_date: string | null;
};

type Fixture = {
  id: string;
  input: string;
  expected: Invoice;
};

const fixtures: Fixture[] = [
  {
    id: "missing-due-date",
    input: [
      "SUPPLIER: Northwind Parts",
      "INVOICE: NW-1042",
      "CURRENCY: USD",
      "TOTAL: 1840.50",
      "DUE DATE: not stated",
    ].join("\n"),
    expected: {
      invoice_number: "NW-1042",
      supplier_name: "Northwind Parts",
      currency: "USD",
      total: 1840.5,
      due_date: null,
    },
  },
  {
    id: "comma-decimal",
    input: [
      "SUPPLIER: Contoso Komponenten",
      "RECHNUNG: CK-7781",
      "CURRENCY: EUR",
      "TOTAL: 2.410,75",
      "DUE DATE: 2026-09-30",
    ].join("\n"),
    expected: {
      invoice_number: "CK-7781",
      supplier_name: "Contoso Komponenten",
      currency: "EUR",
      total: 2410.75,
      due_date: "2026-09-30",
    },
  },
];

Now make the runner strict. It rejects extra prose, checks every expected key, times the complete HTTP call, and prints newline-delimited JSON so a CI job can preserve each observation. The adapter contract is intentionally boring. Good. Boring contracts are easy to replace.

import { performance } from "node:perf_hooks";

type Invoice = {
  invoice_number: string | null;
  supplier_name: string | null;
  currency: string | null;
  total: number | null;
  due_date: string | null;
};

type Fixture = { id: string; input: string; expected: Invoice };
type AdapterResponse = {
  request_id: string;
  model_alias: string;
  output: unknown;
};

const endpoint = process.env.LLM_GATEWAY_URL;
const apiKey = process.env.LLM_GATEWAY_KEY;
const modelAlias = process.env.MODEL_ALIAS;

if (!endpoint || !apiKey || !modelAlias) {
  throw new Error("Set LLM_GATEWAY_URL, LLM_GATEWAY_KEY, and MODEL_ALIAS");
}

const keys: (keyof Invoice)[] = [
  "invoice_number",
  "supplier_name",
  "currency",
  "total",
  "due_date",
];

function isInvoice(value: unknown): value is Invoice {
  if (typeof value !== "object" || value === null || Array.isArray(value)) return false;
  const record = value as Record<string, unknown>;
  if (Object.keys(record).some((key) => !keys.includes(key as keyof Invoice))) return false;
  return keys.every((key) => {
    const field = record[key];
    return field === null || (key === "total" ? typeof field === "number" : typeof field === "string");
  });
}

function compare(actual: Invoice, expected: Invoice) {
  const fields = Object.fromEntries(
    keys.map((key) => [key, Object.is(actual[key], expected[key])]),
  ) as Record<keyof Invoice, boolean>;
  return { fields, passed: Object.values(fields).every(Boolean) };
}

async function evaluate(fixture: Fixture) {
  const started = performance.now();
  const response = await fetch(endpoint, {
    method: "POST",
    headers: {
      authorization: `Bearer ${apiKey}`,
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model_alias: modelAlias,
      task: "extract_supplier_invoice",
      input: fixture.input,
      output_fields: keys,
      null_when_absent: true,
    }),
  });
  const latencyMs = Math.round(performance.now() - started);

  if (!response.ok) {
    throw new Error(`Adapter request failed with status ${response.status}`);
  }

  const result = (await response.json()) as AdapterResponse;
  if (!isInvoice(result.output)) {
    return { fixture: fixture.id, latency_ms: latencyMs, schema_valid: false, passed: false };
  }

  return {
    fixture: fixture.id,
    request_id: result.request_id,
    model_alias: result.model_alias,
    latency_ms: latencyMs,
    schema_valid: true,
    ...compare(result.output, fixture.expected),
  };
}

for (const fixture of fixtures) {
  console.log(JSON.stringify(await evaluate(fixture)));
}

Run compiled JavaScript with the credential supplied by the deployment environment, not checked into the repository. The harness emits observations; it doesn't fabricate a pass threshold. Add a separate CI assertion only after the support owner and backend owner agree on per-field quality and percentile latency budgets.

This is also where DX becomes measurable. Count the configuration values, adapter branches, dependencies, and fixture changes needed to add a candidate path. “Simple” means less application-owned glue under the same acceptance behavior, not a shorter landing-page example.

What would change when the invoice workload grows?

The first change is isolation. Move extraction off the synchronous support request, store an immutable fixture version with each evaluation, and cap concurrency per model path. That separates user-facing latency from extraction throughput and makes a slow tail visible instead of burying it inside a web request.
Next, split observability into stages: document acquisition, text extraction, model call, validation, and persistence. Total duration alone can't tell you which stage moved. Record counts and timings without logging raw invoice text, credentials, or full model responses. Retention and access rules should be explicit for both US and EU deployments; a unified endpoint does not make data-location requirements disappear.
Retries need restraint. Retry only failures your adapter classifies as transient, use a bounded attempt count with backoff, and attach an idempotency key owned by the invoice job. Do not retry a structurally valid but incorrect extraction. That is a quality failure and belongs in review or in the next benchmark revision, not in a loop that spends more time producing the same answer.
Then broaden the corpus. Add scans, OCR noise, credit notes, duplicate invoice numbers across suppliers, tax-inclusive and tax-exclusive totals, and deliberately absent fields. Preserve failures as regression fixtures after review. Benchmark every meaningful prompt, adapter, or model-alias change against the frozen set before rollout. Averages can live on a dashboard, but the release gate should retain per-field failures and tail latency so the team can explain a rejection.

I would also keep the application contract smaller than any upstream API. OpenAI, Claude, and Gemini can remain replaceable targets behind the adapter, while the invoice service asks only for the fields it owns. OpenAI's embeddings guide is useful for understanding a different output primitive, but embeddings are not a substitute for extracting an auditable invoice record. A self-hosted gateway such as LiteLLM is evidence that gateway ownership can be brought in-house; doing so also makes its deployment, upgrades, policy, and on-call load your responsibility.

Trade-offs before choosing one key

One credential reduces secret sprawl in the invoice service, and a common HTTP boundary can avoid three SDKs in application code. The catch is concentration: the shared adapter becomes a dependency for every model path. Keep direct-path tests available for diagnosis, export request IDs, and make the adapter contract explicit enough that another implementation can satisfy it.

A unified gateway is not suitable when policy requires separate provider credentials under separate operational ownership, when the gateway cannot meet the required deployment boundary, or when a provider-specific capability is essential and cannot be represented by the common contract. In those cases, keep separate adapters behind the same application interface. Likewise, stick with direct provider access when the team values the newest provider-specific controls more than credential consolidation and is willing to maintain the extra glue.

Self-hosting isn't automatically simpler. It trades an external dependency for internal deployment and on-call work. A managed gateway trades some control for less infrastructure ownership. Direct integrations trade duplicated authentication and response mapping for immediate access to each provider's native surface. None wins without the invoice corpus, latency percentile, compliance boundary, and maintenance budget beside it.

The decision rule is plain: accept the least operationally expensive path that clears every critical field check, meets the agreed tail-latency limit, and preserves the required deployment boundary. If two paths pass, use configuration count and adapter maintenance as tie-breakers. If none pass, change the model path or workflow; don't lower a critical invoice check to rescue a one-key design.

DEV Community: RivenPulse5812

How to Benchmark a Unified LLM API: Node.js Invoice Extraction Under Latency Limits

How should a Node.js backend benchmark a unified LLM API?

Implement the smallest useful evaluator

What would change when the invoice workload grows?

Trade-offs before choosing one key

References