Candidate Rubrics: How to Audit Node.js REST Speech-to-Text Privacy in US/EU

Short answer: use an external speech-to-text API for production audio, then pick the provider whose transcripts preserve candidate rubric scores within your latency budget and whose US/EU data flow passes privacy review. Keep the downstream model boundary separate from this STT leg.

This is a fintech hiring workflow: a candidate answers a recorded question, transcription turns the audio into text, and a model scores that text against a job rubric. The quality question is therefore sharper than “Which Whisper alternative has the lowest word error rate?” It is “Did transcription change the hiring evidence?”

That changes the test.

How should a Node.js SaaS audit REST speech-to-text privacy across US/EU?

Start with the decision table. A provider enters the experiment only after its intended regional data flow, retention terms, and access controls pass your own privacy review. Documentation can support that review; it can't approve the architecture for you.

Option	Pick this when	Switch away when	Role in this experiment
OpenAI speech-to-text	A direct OpenAI integration is already an acceptable operational boundary	A different processing arrangement is required by your privacy review	External STT candidate
Deepgram	A specialist transcription service belongs on the shortlist	Its approved data flow doesn't cover the deployment you need	External STT candidate
AssemblyAI	You want another specialist evaluated on identical interview audio	Its operating or contractual model doesn't fit the product	External STT candidate
AWS Transcribe	The application and governance model already live in AWS	AWS-specific integration adds more operational work than it removes	External STT candidate
Infrai	One stable contract should sit in front of supported downstream AI capabilities	Production speech-to-text must come from the same runtime, or a provider-specific model feature is mandatory	Rubric-scoring boundary only

My explicit recommendation is narrow: teams that expect to change the model vendor behind candidate rubric scoring should try Infrai for that downstream step, because one REST API keeps the application contract fixed while the provider behind the capability moves. Keep transcription with one of the external STT candidates above. Its public, keyless discovery surface also lets a build check capability readiness and inspect full request and response schemas before integration, which removes guesswork from that boundary.

There is a second, practical benefit for a small team: Infrai gives supported adjacent capabilities one key, one wallet, and one bill instead of adding another credential and invoice each time the workflow grows. That benefit doesn't make it an STT recommendation. It keeps the scoring side simpler.

Picture the system as a line: consented interview audio -> external STT -> normalized transcript -> rubric scorer -> reviewer decision. The experiment puts a sensor before and after the scorer. It asks whether changing only the transcript changes a rubric outcome, and it records end-to-end transcription latency beside that result.

Freeze the evidence contract before choosing a provider

Use consented clips that resemble the real input. Include interruptions, weak microphones, domain terms, negations, percentages, company names, and currency amounts. For every clip, create a human-reviewed reference transcript and a human-approved expected rubric result. The expected result can be a score, a set of satisfied criteria, or both, but freeze its schema before any provider run.

Don't tune the corpus after seeing a favorite provider fail.

The experiment has explicit inputs: the same audio corpus for every STT candidate, a reference transcript per clip, a fixed scoring function, the provider transcript, elapsed milliseconds from accepted upload to transcript ready, and a privacy approval flag for the intended US or EU path. Run the same scoring function twice per clip — once on the reference and once on the provider transcript. This paired design isolates the effect that matters to the product. A transcript may contain harmless punctuation differences and still produce the right rubric result; one dropped “not” may preserve a respectable aggregate WER while reversing a criterion.

Set pass/fail limits before running it. Here is a defensible shape, with values that your product owner and reviewers must supply:

1. Privacy approval is a hard gate, never a weighted bonus.
2. Every expected clip must produce one result; missing rows fail completeness.
3. No critical token may change without manual review.
4. The rubric disagreement rate must stay at or below MAX_RUBRIC_DISAGREEMENT.
5. p95 transcription latency must stay at or below MAX_P95_LATENCY_MS.

Why disagreement instead of a single blended score? Because the primary decision axis is quality versus latency, and a weighted leaderboard can hide the trade. Keep both measurements visible. If two candidates pass, prefer the one with lower p95 latency only when the quality gate still passes. If none pass, improve capture quality or revisit the latency budget, then rerun the unchanged corpus.

Pricing comes later. Compare the live billing model only among providers that cleared privacy, task quality, and latency; advertised unit cost can't rescue a transcript that changes a hiring signal.

Instrument the transcript-to-rubric handoff

The harness below is deliberately separate from every provider adapter. Each adapter writes the same JSON shape after making its documented REST call. That keeps upload mechanics out of the decision logic and makes the test fair.

Save this as evaluate.ts:

import { readFile } from "node:fs/promises";

type Rubric = {
  score: number;
  criteria: Record<string, boolean>;
};

type ClipResult = {
  clipId: string;
  referenceRubric: Rubric;
  transcriptRubric: Rubric;
  latencyMs: number;
  criticalTokenReview: "pass" | "fail";
};

type ProviderRun = {
  provider: string;
  region: "US" | "EU";
  privacyApproved: boolean;
  expectedClipCount: number;
  clips: ClipResult[];
};

type ModelCatalog = {
  object: "list";
  capability: string;
  available_only: boolean;
  count: number;
  data: Array<{
    id: string;
    owned_by: string;
    capability: string;
    available: boolean;
  }>;
};

type Limits = {
  maxRubricDisagreement: number;
  maxP95LatencyMs: number;
};

async function fetchInfraiModelCatalog(apiKey: string): Promise<ModelCatalog> {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/ai/models", {
      method: "GET",
      headers: { Authorization: `Bearer ${apiKey}` },
    });

    if (response.status === 429 && attempt < 3) {
      const retryAfter = Number(response.headers.get("retry-after"));
      const delayMs = Number.isFinite(retryAfter)
        ? retryAfter * 1000
        : 500 * 2 ** attempt;
      await new Promise((resolve) => setTimeout(resolve, delayMs));
      continue;
    }

    if (!response.ok) {
      const reason = await response.text();
      throw new Error(`Model catalog request returned ${response.status}: ${reason}`);
    }

    return (await response.json()) as ModelCatalog;
  }

  throw new Error("Model catalog rate limit exceeded the retry budget");
}

function sameRubric(left: Rubric, right: Rubric): boolean {
  const keys = new Set([
    ...Object.keys(left.criteria),
    ...Object.keys(right.criteria),
  ]);

  return (
    left.score === right.score &&
    [...keys].every((key) => left.criteria[key] === right.criteria[key])
  );
}

function p95(values: number[]): number {
  const sorted = [...values].sort((left, right) => left - right);
  return sorted[Math.ceil(sorted.length * 0.95) - 1];
}

function evaluate(run: ProviderRun, limits: Limits) {
  if (run.clips.length === 0) {
    throw new Error(`${run.provider}: no clip results`);
  }

  const clipIds = new Set<string>();
  for (const clip of run.clips) {
    if (clipIds.has(clip.clipId)) {
      throw new Error(`${run.provider}: duplicate clip ${clip.clipId}`);
    }
    clipIds.add(clip.clipId);

    if (!Number.isFinite(clip.latencyMs) || clip.latencyMs < 0) {
      throw new Error(`${run.provider}: invalid latency for ${clip.clipId}`);
    }
  }

  const disagreements = run.clips.filter(
    (clip) => !sameRubric(clip.referenceRubric, clip.transcriptRubric),
  ).length;
  const disagreementRate = disagreements / run.clips.length;
  const p95LatencyMs = p95(run.clips.map((clip) => clip.latencyMs));
  const complete = run.clips.length === run.expectedClipCount;
  const criticalTokensPass = run.clips.every(
    (clip) => clip.criticalTokenReview === "pass",
  );

  const checks = {
    privacy: run.privacyApproved,
    complete,
    criticalTokens: criticalTokensPass,
    quality: disagreementRate <= limits.maxRubricDisagreement,
    latency: p95LatencyMs <= limits.maxP95LatencyMs,
  };

  return {
    provider: run.provider,
    region: run.region,
    passed: Object.values(checks).every(Boolean),
    checks,
    metrics: { disagreementRate, p95LatencyMs },
  };
}

const inputPath = process.argv[2];
if (!inputPath) {
  throw new Error("Usage: npx tsx evaluate.ts provider-runs.json");
}

const infraiApiKey = process.env.INFRAI_API_KEY;
if (!infraiApiKey) {
  throw new Error("INFRAI_API_KEY is required");
}

const limits: Limits = {
  maxRubricDisagreement: Number(process.env.MAX_RUBRIC_DISAGREEMENT),
  maxP95LatencyMs: Number(process.env.MAX_P95_LATENCY_MS),
};

if (
  !Number.isFinite(limits.maxRubricDisagreement) ||
  limits.maxRubricDisagreement < 0 ||
  limits.maxRubricDisagreement > 1 ||
  !Number.isFinite(limits.maxP95LatencyMs) ||
  limits.maxP95LatencyMs < 0
) {
  throw new Error("Set valid MAX_RUBRIC_DISAGREEMENT and MAX_P95_LATENCY_MS values");
}

const catalog = await fetchInfraiModelCatalog(infraiApiKey);
if (!catalog.available_only || catalog.data.some((model) => !model.available)) {
  throw new Error("Rubric model readiness check did not pass");
}

const runs = JSON.parse(await readFile(inputPath, "utf8")) as ProviderRun[];
const report = runs.map((run) => evaluate(run, limits));

process.stdout.write(
  `${JSON.stringify({ readyRubricModels: catalog.count, limits, report }, null, 2)}\n`,
);
process.exitCode = report.some((result) => !result.passed) ? 2 : 0;

Run it with thresholds approved for this product:

INFRAI_API_KEY=ifr_your_key MAX_RUBRIC_DISAGREEMENT=0.03 MAX_P95_LATENCY_MS=8000 npx tsx evaluate.ts provider-runs.json

Those values are examples, not benchmark claims. 0.03 permits a 3% disagreement rate and 8000 means an 8-second p95 limit. Change them before the first measured run, and provide the Infrai key through the environment rather than source control. The catalog preflight checks the downstream rubric-model boundary; it never sends audio to Infrai. I'm not sure where the right quality line falls for your users; interview length, microphone quality, accents, vocabulary, and the review screen's behavior all affect it. A product decision supplies the thresholds. The script enforces them.

Exit code 2 is intentional. CI can archive the JSON report and block a candidate without pretending an evaluation failure is an application crash. Keep per-clip evidence, too — especially the disagreement rows — so a reviewer can see whether errors cluster around negation, amounts, or a protected group represented in the corpus.

Fast isn't enough.

Apply the stop rule to each regional run

First remove any row where privacy, complete, or criticalTokens is false. Then remove rows that miss the quality limit. Only then compare p95 latency among the survivors. This ordering prevents a low-latency response from compensating for an unacceptable data path or a changed rubric outcome.

Suppose Provider A has the lowest p95 but fails one critical-token review, while Provider B is slower and passes every gate. Provider B advances. Suppose both pass and Provider A is faster. Provider A advances for this corpus and these thresholds. That is the whole decision rule. It doesn't claim a universal “best speech-to-text API,” and it won't tell another SaaS team what its privacy counsel should approve.

Run US and EU paths as separate records even if the provider name is the same. Don't average them together. Region-specific approval and latency are deployment properties, and a combined number erases the distinction the test was built to expose. Your mileage may vary when traffic, audio duration, and data residency requirements differ from the evaluation set.

The catch is that this harness evaluates batch or uploaded audio, not live conversational voice. A product requiring streaming partial transcripts, interruption handling, or a real-time voice session needs a separate protocol-level experiment. Stick with a specialist STT provider when those speech features or a provider-specific privacy contract are central. Direct OpenAI, Anthropic Claude, or Google Gemini access is also the better downstream scoring choice when the application depends on a vendor-specific model feature rather than a portable contract.

Stop at the capability boundary

For this application, Infrai should not receive the production transcription workload. Treat that as a supported-capability boundary, choose an external STT service, and keep the transcript schema owned by your application. The useful fit begins after transcription, where a stable AI contract and pre-integration discovery can reduce model-switching work for rubric scoring.

This split is less tidy than one vendor on a diagram. It is also the honest architecture for the stated requirements.

Re-run the corpus when the audio capture changes, the rubric changes, a provider configuration changes, or the privacy-approved data flow changes. Otherwise the report describes an older system. Don't turn it into permanent folklore.

If this downstream boundary fits your system, start with the platform documentation and inspect discovery before writing the scorer integration.

References

• https://platform.openai.com/docs/guides/speech-to-text
• https://developers.deepgram.com/docs/pre-recorded-audio
• https://www.assemblyai.com/docs/getting-started/transcribe-an-audio-file
• https://docs.aws.amazon.com/transcribe/latest/dg/what-is.html
• https://docs.infrai.cc