Private Clinical Speech Retrieval: One API Key for Multi-Model Answers

Use one API key at your application's boundary, but keep speech-to-text and answer generation as two explicit stages behind it. For a private clinical knowledge base, the best default is a thin credential broker plus a typed orchestration layer: transcribe once, retrieve only authorized material, then route the grounded prompt to a model under a measured latency budget. Don't make a model gateway pretend audio ingestion and text generation are the same workload.

That answer is less tidy than buying a magical universal key. It is also easier to test. A shared external credential is a DX choice; separate internal adapters are an architecture choice. Mixing those decisions makes provider changes leak into every CLI, SDK, and worker.

The deciding axis here is quality versus latency. A transcript that silently changes a medication name can poison retrieval, while a perfect answer that arrives after the clinician has moved on is still a failed interaction. Measure both stages independently.

Clinical evidence is the real control plane

The transcript is not the final artifact. It becomes a query over a private corpus containing policies, care pathways, and other controlled material. That creates three boundaries: audio to transcript, transcript to authorized evidence, and evidence to answer. Each boundary has a different failure mode and should emit its own timing and quality signals.

Start with access control. Resolve the caller's tenant, role, and corpus permissions before retrieval, and carry that scope through the request. The model should receive the smallest useful evidence set, not a broad dump of the knowledge base. Logs need the same discipline: identifiers, raw audio, transcript text, retrieved passages, and generated answers do not all belong in the same default log event.

Then separate accuracy questions. Speech recognition can lose negation, dosage units, abbreviations, or speaker boundaries. Retrieval can select a plausible but unauthorized or obsolete passage. Generation can produce a fluent answer that is not supported by the supplied evidence. One end-to-end score hides which component failed.

Latency is compositional too. Upload time, transcription, retrieval, first-token delay, and full generation each consume part of the budget. Record them separately. Otherwise a gateway dashboard may say generation is fast while the user waits on audio upload and indexing work that the dashboard never sees.

This is the concrete constraint that rules out a blind "one provider does everything" choice. The application's single key should authenticate one stable contract. Behind that contract, independently replaceable adapters can have different timeout, retry, residency, and retention policies. OpenAI, Claude, and Gemini may be candidates for the text stage, but brand coverage is not evidence that any one route meets a specific clinical workload. The benchmark corpus decides. Build that corpus before comparing a direct integration, a text-only gateway, or a broader API aggregator: use permission-safe audio that represents deployment conditions and contains terms whose corruption changes meaning; store expected transcript spans, allowed evidence IDs, and an adjudicated answer rubric; then capture transcript quality with a domain-aware review, retrieval recall against allowed evidence, citation validity, grounded-answer quality, and latency percentiles for each stage. Token counts and audio duration are useful operational inputs, but don't turn them into a synthetic score until the team has agreed on weights.

I'm not sure a universal quality threshold exists for clinical speech. Specialty, microphone conditions, accents, and the cost of a wrong answer change it. A useful acceptance test therefore starts with representative, permission-safe audio and an adjudicated set of expected citations rather than a generic leaderboard.

Put release gates ahead of provider selection

Keep protected material out of fixtures used in ordinary CI.

Signal	Why it matters	Reject when
Critical-term transcription	A small word error can reverse clinical meaning	A protected term, unit, or negation fails the review rubric
Authorized evidence recall	The answer cannot recover evidence it never receives	Expected in-scope passages are repeatedly missed
Citation validity	Users need inspectable support	A citation falls outside the retrieved authorized set
Time to first usable answer	Interactive work has a finite attention window	The agreed percentile exceeds the product budget
Cancellation completion	Abandoned work should stop	Downstream calls continue beyond the cancellation allowance

Don't benchmark only clean studio clips. Add background noise, interrupted speech, long pauses, multiple speakers when the workflow permits them, and terminology from the actual corpus. Do not invent an accuracy target from another team's blog post. The acceptance line comes from the harm model and product workflow. Error tests deserve equal weight: inject a client-visible 429 from an adapter and confirm that retry policy respects its declared retry timing rather than immediately amplifying load; reject malformed provider responses at the adapter boundary; test that a deadline stops retrieval and generation; and ensure an authorization failure cannot fall back to a broader corpus. These are deterministic checks, so they belong in CI. For quality versus latency, use a Pareto view rather than declaring one winner. A fast configuration that misses critical terms is dominated. A slower configuration may be suitable for review mode but not interactive use. Keep both only when the product genuinely exposes those modes and the operational complexity earns its keep.

How should one API key route speech-to-text and multi-model transcript summaries?

Keep the public contract boring. The client sends audio plus a knowledge-base scope to one application endpoint and authenticates with one application key. The service validates scope, calls a transcription adapter, retrieves allowed passages, and calls a text-model adapter selected by policy. Provider credentials never ship in the client.

That is one key for the developer consuming your API, not necessarily one secret for every upstream service. The distinction matters. Secret consolidation can reduce setup, but it can also couple incident response, quotas, and data policy across workloads that should remain isolated. A credential broker gives the SDK the small surface it wants without erasing the operational boundaries the backend needs.

The routing policy should use fields the system can defend: requested latency class, approved data region, context size, streaming need, and results from the team's evaluation set. Avoid routing on a vague label such as "best model." It cannot be tested.

A model gateway can normalize text-generation calls across providers. Its documentation is useful evidence for the exact providers, request fields, streaming behavior, and error semantics it currently exposes. It does not remove the need to verify whether speech transcription is part of the same contract. Treat that as a discovery question, not an assumption, and pin the behavior your adapter depends on.

There is a second trap. "Summarize the transcript" is underspecified for a private knowledge-base question. A summary compresses what was said; a grounded answer combines the question with retrieved evidence and should make its support inspectable. Keep those operations distinct in types and telemetry, even if the same text model can perform both.

Implement only the contract the evidence requires

This version has no provider-specific URL or SDK. It accepts adapters, which keeps the orchestration contract testable and stops vendor response shapes from reaching application code. The example returns citations with the answer because unsupported prose is not enough for this job.

type AudioInput = {
  bytes: Uint8Array;
  mediaType: string;
};

type Scope = {
  tenantId: string;
  corpusId: string;
  actorId: string;
};

type Transcript = {
  text: string;
  language?: string;
};

type Passage = {
  id: string;
  text: string;
  revision: string;
};

type Answer = {
  text: string;
  citationIds: string[];
};

type SpeechAdapter = {
  transcribe(audio: AudioInput, signal: AbortSignal): Promise<Transcript>;
};

type Retriever = {
  findAuthorized(
    query: string,
    scope: Scope,
    signal: AbortSignal,
  ): Promise<Passage[]>;
};

type ModelAdapter = {
  answerGrounded(
    input: {
      question: string;
      transcript: string;
      passages: Passage[];
    },
    signal: AbortSignal,
  ): Promise<Answer>;
};

type Dependencies = {
  speech: SpeechAdapter;
  retrieve: Retriever;
  modelFor: (latencyClass: "interactive" | "review") => ModelAdapter;
};

export async function answerFromClinicalAudio(
  deps: Dependencies,
  input: {
    audio: AudioInput;
    scope: Scope;
    latencyClass: "interactive" | "review";
    signal: AbortSignal;
  },
): Promise<Answer & { transcript: Transcript }> {
  const transcript = await deps.speech.transcribe(input.audio, input.signal);
  const passages = await deps.retrieve.findAuthorized(
    transcript.text,
    input.scope,
    input.signal,
  );

  const answer = await deps.modelFor(input.latencyClass).answerGrounded(
    {
      question: transcript.text,
      transcript: transcript.text,
      passages,
    },
    input.signal,
  );

  const allowed = new Set(passages.map((passage) => passage.id));
  if (answer.citationIds.some((id) => !allowed.has(id))) {
    throw new Error("MODEL_CITATION_OUT_OF_SCOPE");
  }

  return { ...answer, transcript };
}

Deliberately dull.

The important line is not the model selection. It is the citation membership check. That check does not prove the answer is clinically correct, but it prevents the response from claiming a source the retriever never authorized. A stricter implementation can require sentence-level citation coverage and reject stale passage revisions before displaying the answer. I've kept the explicit MODEL_CITATION_OUT_OF_SCOPE error because a typed, searchable failure is far more useful than letting an adapter return plausible prose — especially when the happy-path demo makes every option look interchangeable.

The AbortSignal also belongs in every adapter. A client cancellation or exhausted deadline should stop downstream work. Without propagation, a timed-out request can keep consuming transcription and generation capacity after its result has become useless.

For streaming text, Server-Sent Events are a reasonable fit when the server only needs to push updates to the browser. The browser's EventSource interface opens a persistent HTTP connection and receives named or default events. It is one-way, so use a different transport if the interaction requires bidirectional messages after the request begins. Never interpret streaming as permission to show an answer before its citations and authorization checks are ready.

Operations inherit the evidence boundary

At higher volume, split ingestion from answering. Normalize audio metadata, attach a retention class, and assign an idempotency key before expensive work begins. Cache only artifacts whose tenant, corpus revision, and policy scope are part of the cache key. A transcript reused across the wrong authorization boundary is not a performance optimization.

I would also version three contracts independently: transcription output, retrieval evidence, and grounded-answer schema. A provider swap then becomes an adapter rollout with shadow evaluation instead of an application-wide migration. Configuration stays centralized and typed. Config bloat is usually a sign that provider quirks have escaped their adapters.

Observability should follow the same boundaries. Emit a request correlation ID, adapter name, model identifier, stage duration, cancellation state, evidence revision, and coarse error class. Keep raw content out of routine metrics. Audit storage and product telemetry can have different retention and access rules; forcing both through one logging sink makes later controls harder.

The catch is added ownership. This approach is not suitable when a tiny team cannot operate authentication, policy enforcement, adapter conformance tests, and evaluation data. In that case, choose a managed contract that explicitly supports the required audio and text operations, data controls, and error semantics, then keep a narrow application adapter around it. A direct provider integration is also sensible when one approved model meets the workload and multi-model routing would only add configuration.

Stick with separate public credentials when clients must call upstream services directly, tenants bring their own accounts, or organizational policy requires isolated billing and revocation domains. One application key is a cleaner SDK surface, but it is not a law of nature.

The final decision is mechanical: pick the simplest contract that passes the clinical evaluation set inside the latency budget and preserves authorization at every boundary. Re-run that suite when a model, speech engine, prompt, retrieval index, or gateway behavior changes. Names rotate. Tests stay useful.

References

• https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
• https://openrouter.ai/docs

Further reading

The MDN Server-Sent Events guide covers browser connection behavior and event framing. The gateway documentation above is a useful example of the provider and request-contract details that should be verified during discovery.