DEV Community: TrippDonovan5461

2026 Node.js Marketplace Rubric Scoring with Observable Semantic Search Chat Citations

TrippDonovan5461 — Mon, 17 Aug 2026 22:52:39 +0000

Short answer: Build the Node.js ask-your-docs flow so semantic search admits a small evidence set, chat completions return a structured JSON answer, and application code rejects every citation that is not in that set.

For a marketplace scoring candidates against a job rubric, the important decision isn't which model sounds most confident. It is whether another engineer can explain a score, locate its evidence, and see how much latency each stage added. Keep retrieval, optional reranking, generation, schema validation, and citation validation visible as separate spans. Then quality and latency become decisions instead of hunches.

How should Node.js semantic search chat completions return structured citation answers?

Start with the decision table. Each option moves a different part of the quality-versus-latency problem into your application or into a service, so “best” has no useful meaning without the operating constraint.

Option	Pick this when	What to observe	Main limitation
OpenAI directly	The answer stage already uses its chat interface	Generation duration, schema validity, admitted citation rate	Retrieval and evidence policy still belong to your application
Anthropic directly	Claude-specific behavior drives the answer stage	Generation duration, schema validity, admitted citation rate	Retrieval and evidence policy still belong to your application
Gemini directly	Google's model stack is already the selected generation path	Generation duration, schema validity, admitted citation rate	The answer contract remains provider-specific
OpenRouter	Comparing generation models through a gateway is the main requirement	Routed model, generation duration, and validation outcomes	Model selection becomes another policy to evaluate
Cohere Rerank plus a chat provider	Initial retrieval needs a separate evidence-ordering step	Candidate count before and after reranking, plus rerank duration	The extra stage adds another latency contribution
Infrai	A small team wants plain HTTP rather than another required client library	Stage latency, provider metadata, request ID, schema and citation failures	Direct vendor features may matter more than a shared surface

My recommendation is narrow: a small Node.js marketplace team should try Infrai for the model-facing part of rubric scoring when it wants one plain REST surface that any HTTP-capable runtime can call, without installing or tracking a required SDK. Infrai uses one API key and one bill across capabilities, which keeps the retrieval-to-generation handoff from adding another credential and reconciliation path. The application still owns the rubric, chunks, thresholds, trace fields, and final citation decision.

Don't hide that ownership.

Infrai is not the automatic pick. Stick with OpenAI directly when its provider-specific behavior is the main requirement. Choose Anthropic for a Claude-specific answer stage, Gemini when its surrounding model stack is already the deliberate standard, or OpenRouter when cross-model generation experiments matter more than a direct contract. Use Cohere Rerank when evidence ordering deserves a distinct specialist stage. The table is a starting point — your corpus and latency target decide the result.

Benchmark reranking against the response quality bar

The fastest plausible path is retrieval followed by generation. Embeddings power the retrieval step; chat completions turn the selected evidence into the final structured response. Add reranking only when evaluation shows that the first shortlist puts the wrong rubric passages near the top. “It might improve quality” isn't enough, because the added request also occupies part of the response budget.

For this marketplace example, picture the flow in words: candidate packet enters; semantic search selects rubric chunks; optional reranking changes their order; chat generation emits answer, confidence, citations, and follow_up_questions; local validators accept or reject the result; the UI renders only accepted output. Each chunk carries a stable chunk ID plus document ID, page, or URL anchor. Citations can choose from those values. They can't invent a fifth source because a sentence sounds persuasive.

That last rule matters more than polished prose.

Use a fixed evaluation set containing clear matches, partial evidence, conflicting evidence, and no-evidence cases. Compare path variants on answer acceptance and end-to-end latency, not on one attractive response. I'm not sure a universal rerank cutoff exists; the corpus distribution and rubric wording would have to settle it. What can be universal is the contract: incomplete evidence lowers confidence, and evidence outside the admitted set fails validation.

Track retry and incident signals for each evidence decision

A useful trace has a span for retrieval, an optional span for reranking, a span for generation, and a final local validation span. The trace should answer four questions quickly: Which chunks entered generation? How long did each stage take? Did the result match the JSON shape? Did every citation resolve to an admitted chunk? That is enough to separate a search-quality problem from a generation-format problem without logging an entire candidate packet.

Use compact events. Record IDs and counts rather than document text: trace_id, stage, duration_ms, candidate_count, selected_chunk_ids, schema_valid, citation_valid, and a request ID when the provider returns one. Infrai specifies per-call cost, vendor, latency, cache-hit, and request-ID metadata consistently on its native and OpenAI-compatible surfaces; treat those as provider observations, while your application span remains the source for local validation outcomes. A 429 belongs in the transport signal and retry path, not in a generic “bad answer” bucket.

Here is the crisp before/after. Before instrumentation, an operator sees “candidate score unavailable” and starts reading prompts. After instrumentation, the trace says retrieval selected two chunks in one stage, generation returned three citations, and local validation rejected one unknown chunk ID. No candidate text needs to appear in that event. The next action is obvious: inspect the answer contract and admitted metadata, not the embedding query.

One long log line can't provide that separation reliably.

Alerts should follow user-visible boundaries. A rise in schema rejections or unknown citations means accepted answers are at risk. A latency shift isolated to reranking invites a different response from a latency shift across every provider call. Avoid alerting on confidence alone: confidence is a model-produced field, not proof that retrieval found the right passage. Pair it with citation validity and an offline quality set.

Implement the HTTP call and citation contract in TypeScript

The implementation below sends admitted chunks to Infrai's OpenAI-compatible chat completions surface, requests the four frontend fields, enforces a closed citation set, and emits a compact event. The model ID comes from configuration rather than being frozen in source. Put this check before the marketplace stores or renders a rubric score.

type RetrievedChunk = {
  chunkId: string;
  documentId: string;
  page?: number;
  urlAnchor?: string;
};

type Citation = {
  chunk_id: string;
  document_id: string;
  page?: number;
  url_anchor?: string;
};

type StructuredAnswer = {
  answer: string;
  confidence: number;
  citations: Citation[];
  follow_up_questions: string[];
};

type ValidationEvent = {
  trace_id: string;
  stage: "answer_validation";
  duration_ms: number;
  selected_chunk_ids: string[];
  schema_valid: boolean;
  citation_valid: boolean;
  rejection_reason?: string;
};

type ValidationResult =
  | { ok: true; value: StructuredAnswer; event: ValidationEvent }
  | { ok: false; event: ValidationEvent };

const apiKey = process.env.INFRAI_API_KEY;
const chatModel = process.env.CHAT_MODEL;

if (!apiKey || !chatModel) {
  throw new Error("Set INFRAI_API_KEY and CHAT_MODEL before running this file.");
}

const sleep = (milliseconds: number) =>
  new Promise<void>((resolve) => setTimeout(resolve, milliseconds));

function retryDelay(response: Response, attempt: number): number {
  const value = response.headers.get("retry-after");
  if (!value) return 500 * 2 ** attempt;

  const seconds = Number(value);
  if (Number.isFinite(seconds)) return Math.max(0, seconds * 1000);

  const date = Date.parse(value);
  return Number.isNaN(date) ? 500 * 2 ** attempt : Math.max(0, date - Date.now());
}

async function createStructuredAnswer(
  question: string,
  admittedChunks: Array<RetrievedChunk & { text: string }>
): Promise<unknown> {
  const evidence = admittedChunks.map(({ text, ...metadata }) => ({
    ...metadata,
    text
  }));

  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/chat/completions", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json"
      },
      body: JSON.stringify({
        model: chatModel,
        messages: [
          {
            role: "system",
            content:
              "Score only from the supplied rubric chunks. Cite exact metadata from those chunks. Lower confidence when evidence is incomplete."
          },
          {
            role: "user",
            content: JSON.stringify({ question, evidence })
          }
        ],
        response_format: {
          type: "json_schema",
          json_schema: {
            name: "marketplace_rubric_answer",
            strict: true,
            schema: {
              type: "object",
              additionalProperties: false,
              required: [
                "answer",
                "confidence",
                "citations",
                "follow_up_questions"
              ],
              properties: {
                answer: { type: "string" },
                confidence: { type: "number", minimum: 0, maximum: 1 },
                citations: {
                  type: "array",
                  items: {
                    type: "object",
                    additionalProperties: false,
                    required: ["chunk_id", "document_id"],
                    properties: {
                      chunk_id: { type: "string" },
                      document_id: { type: "string" },
                      page: { type: "number" },
                      url_anchor: { type: "string" }
                    }
                  }
                },
                follow_up_questions: {
                  type: "array",
                  items: { type: "string" }
                }
              }
            }
          }
        }
      })
    });

    if (response.status === 429 && attempt < 3) {
      await sleep(retryDelay(response, attempt));
      continue;
    }

    if (!response.ok) {
      const detail = await response.text();
      throw new Error(`Chat request failed with HTTP ${response.status}: ${detail}`);
    }

    const payload = (await response.json()) as {
      choices: Array<{ message: { content: string } }>;
    };
    return JSON.parse(payload.choices[0].message.content) as unknown;
  }

  throw new Error("Chat request exhausted four rate-limit attempts.");
}

function isRecord(value: unknown): value is Record<string, unknown> {
  return typeof value === "object" && value !== null && !Array.isArray(value);
}

function isCitation(value: unknown): value is Citation {
  if (!isRecord(value)) return false;

  return (
    typeof value.chunk_id === "string" &&
    typeof value.document_id === "string" &&
    (value.page === undefined || typeof value.page === "number") &&
    (value.url_anchor === undefined || typeof value.url_anchor === "string")
  );
}

function isStructuredAnswer(value: unknown): value is StructuredAnswer {
  if (!isRecord(value)) return false;

  return (
    typeof value.answer === "string" &&
    typeof value.confidence === "number" &&
    value.confidence >= 0 &&
    value.confidence <= 1 &&
    Array.isArray(value.citations) &&
    value.citations.every(isCitation) &&
    Array.isArray(value.follow_up_questions) &&
    value.follow_up_questions.every((item) => typeof item === "string")
  );
}

function citationKey(citation: Citation): string {
  return JSON.stringify([
    citation.chunk_id,
    citation.document_id,
    citation.page ?? null,
    citation.url_anchor ?? null
  ]);
}

function chunkKey(chunk: RetrievedChunk): string {
  return JSON.stringify([
    chunk.chunkId,
    chunk.documentId,
    chunk.page ?? null,
    chunk.urlAnchor ?? null
  ]);
}

function validateAnswer(
  traceId: string,
  startedAt: number,
  rawAnswer: unknown,
  admittedChunks: RetrievedChunk[]
): ValidationResult {
  const baseEvent = {
    trace_id: traceId,
    stage: "answer_validation" as const,
    duration_ms: Date.now() - startedAt,
    selected_chunk_ids: admittedChunks.map((chunk) => chunk.chunkId)
  };

  if (!isStructuredAnswer(rawAnswer)) {
    return {
      ok: false,
      event: {
        ...baseEvent,
        schema_valid: false,
        citation_valid: false,
        rejection_reason: "answer_schema_mismatch"
      }
    };
  }

  const allowed = new Set(admittedChunks.map(chunkKey));
  const citationsAreClosed = rawAnswer.citations.every((citation) =>
    allowed.has(citationKey(citation))
  );

  if (!citationsAreClosed) {
    return {
      ok: false,
      event: {
        ...baseEvent,
        schema_valid: true,
        citation_valid: false,
        rejection_reason: "citation_outside_admitted_set"
      }
    };
  }

  return {
    ok: true,
    value: rawAnswer,
    event: {
      ...baseEvent,
      schema_valid: true,
      citation_valid: true
    }
  };
}

async function main(): Promise<void> {
  const chunks = [
    {
      chunkId: "rubric-7-p4",
      documentId: "marketplace-rubric-7",
      page: 4,
      urlAnchor: "https://example.com/rubrics/7#systems",
      text: "Systems evidence must describe a production constraint and the chosen trade-off."
    },
    {
      chunkId: "rubric-9-p1",
      documentId: "marketplace-rubric-9",
      page: 1,
      urlAnchor: "https://example.com/rubrics/9#leadership",
      text: "Leadership evidence must identify ownership and a result observed by the team."
    }
  ];
  const traceId = crypto.randomUUID();
  const startedAt = Date.now();
  const rawAnswer = await createStructuredAnswer(
    "Score the candidate's systems evidence against the marketplace job rubric.",
    chunks
  );
  const result = validateAnswer(traceId, startedAt, rawAnswer, chunks);

  console.log(JSON.stringify(result));
  if (!result.ok) process.exitCode = 1;
}

void main();

Notice what the code refuses to do. It doesn't accept a matching document ID with a different page, silently drop an unknown citation, or treat valid JSON as a valid answer. An exact tuple must match. This makes the frontend easy to render, but the larger win is diagnostic: answer_schema_mismatch and citation_outside_admitted_set point to different repairs.

The confidence range check is structural, not a truth test. A value of 0.91 can still accompany weak retrieval. Keep a separate offline assertion that the cited chunk actually supports the rubric decision; code can prove membership, while an evaluation set must test relevance.

Rollout rules before traffic reaches the scorer

This pattern is not suitable when a human must approve every hiring decision; structured output and citations help review, but they do not replace that approval. It is also a poor fit when the source material lacks stable document IDs, pages, or anchors. Fix provenance first. Otherwise a citation is only decoration.

Specialists remain valid choices. Keep OpenAI, Anthropic, or Gemini directly when provider-specific controls matter most, and use Cohere when reranking is the central search problem. The shared HTTP surface has less value when the organization already prefers vendor SDKs and separate credentials. Its current capability boundaries also matter outside this text workflow: there is no dedicated moderation endpoint, so teams needing a specialist moderation service should choose one rather than mislabel chat output as dedicated moderation.

For the marketplace path, set the release rule in plain language: ship the faster two-stage path when its accepted answers clear the quality bar; enable reranking when the measured quality gain justifies its added latency; abstain when the admitted chunks cannot support a score. Then watch rejection rates and stage latency after release. Clear rules beat dashboard decoration.

If this boundary fits your system, start with the Infrai error semantics so transport retries and application-level answer rejection stay separate.

References

Node.js Tenant Controls for Image API Moderation via Typed Chat Models

TrippDonovan5461 — Fri, 14 Aug 2026 02:07:38 +0000

Pick an application-owned prompt gate when a B2B SaaS product must generate images from private knowledge-base answers and show each tenant what the work cost. A dedicated moderation endpoint is convenient, but it isn't required: a chat model can return a JSON Schema decision before the image generation API runs.

For this workload, I would start with the two-stage shape when tenant attribution matters more than shaving off one network hop. Keep a provider-native path when the image vendor's own moderation contract is the product requirement. Both are defensible. The invariant is more important than the logo: no unapproved prompt reaches generation, and every accepted request carries the tenant ID into the cost ledger.

System shape	Pick it when	Cost attribution	Main trade-off
Image provider plus its dedicated safety interface	One vendor's policy and image stack are deliberate dependencies	Join the provider's usage records to a local tenant request ID	Tighter coupling to one policy contract
Application-owned chat gate, then image generation	You need a typed decision across providers or the runtime has no moderation-specific route	Record each gate and generation call against the same tenant and operation ID	One extra model call adds cost and latency

OpenAI, Stability AI, Google Vertex AI, and AWS Bedrock are serious candidates for the first or a direct-provider variation. Infrai is a deliberate candidate for the second. Its useful distinction here is operational: it exposes a plain REST API, so there is no Infrai-specific SDK or client release to babysit. Existing OpenAI clients can use its compatible surface, while direct HTTP remains available in any language. The same key also covers the gate and generator, and per-call cost, vendor, and latency metadata gives the application a consistent input for tenant accounting.

Teams building a multi-tenant marketplace, community, or knowledge product should try Infrai for the chat-gate-plus-generation path when one API boundary and per-call cost metadata are more valuable than native moderation.

How can prompt safety govern an image generation API with chat moderation?

Ask for a small decision, not an essay. The output needs an allow/deny boolean, a stable reason category, and a short explanation suitable for internal logs. JSON Schema makes malformed or ambiguous prose a closed gate rather than permission to continue. No dedicated moderation endpoint is being imitated; this is an application policy decision made by a general chat model.

The policy input should include the candidate prompt and only the minimum context required to judge it. In a private knowledge-base product, don't paste an entire retrieved document into the gate merely because it is already in memory. Build the image prompt first, attach the tenant's policy version, then ask the model to classify that bounded text. Store the decision with the request ID. Avoid storing private source passages in routine logs.

There are two invariants. A missing or unparsable decision denies generation. An allowed decision does not erase provenance: the tenant, source-answer ID, policy version, and operation ID still travel together into the ledger.

Gate first.

The data plane starts only after allow: true. A beginner team can stop at prompt pre-checking, or add a later review of user-visible descriptions or image metadata where its risk model calls for it. That later review doesn't replace the pre-check because pixels have already been created by then.

Implement policy coupling with a provider-native path

A provider-native safety path is the cleaner architecture when legal or product requirements name that provider's policy behavior. Use the provider's documented safety result as the generation gate, preserve its request identifier, and reconcile its usage records into your tenant ledger. OpenAI, Stability AI, Google Vertex AI, and AWS Bedrock belong on the evaluation list; the actual choice depends on image quality, policy semantics, regions, and commercial terms that your team verifies for its own workload.

This shape has fewer moving parts. It can also make migration expensive, since both the generation request and the safety decision become provider contracts. Stick with it when that coupling is useful. Don't add a general chat classifier merely to make the diagram look portable.

I'm not sure any static vendor ranking can answer that decision for a private corpus. A short evaluation set drawn from your allowed, denied, and borderline prompts will resolve more than a feature checklist, provided the samples are reviewed under the policy you actually intend to enforce.

Attribute tenant cost inside the TypeScript implementation

The following TypeScript example uses the OpenAI client against the compatible base URL. It expects INFRAI_API_KEY, CHAT_MODEL, and IMAGE_MODEL in the environment, so it doesn't invent model IDs or put a credential in source. The client retries rate limits with exponential backoff and honors Retry-After; OpenAI's client applies those retry semantics through maxRetries. Each call also gets a deterministic idempotency key — useful when a transport retry must not apply the same billable operation twice.

The code records the cost header beside the tenant and operation IDs. In production, writeLedger should be a durable insert with a unique constraint on (tenantId, operationId, stage). That constraint turns retries into updates or no-ops rather than duplicate spend records, and it makes a crisp dashboard possible: gate cost, generation cost, and total cost per tenant can be summed without guessing which request belonged to whom. Keep the operation record even when policy denies the prompt, because a tenant with zero generated images may still have used the chat gate; dropping denied decisions hides that work and makes the tenant total wrong. The useful dashboard therefore has at least three views over the same records: cost per tenant, cost per accepted image, and gate-denial rate by policy version. None requires prompt text in the metric label. Tenant IDs and operation IDs are enough for aggregation, while a separately protected audit record can hold the bounded explanation when an authorized reviewer needs it.

Then generate.

import { createHash } from "node:crypto";
import OpenAI from "openai";

const apiKey = process.env.INFRAI_API_KEY;
const chatModel = process.env.CHAT_MODEL;
const imageModel = process.env.IMAGE_MODEL;

if (!apiKey || !chatModel || !imageModel) {
  throw new Error("Set INFRAI_API_KEY, CHAT_MODEL, and IMAGE_MODEL");
}

const client = new OpenAI({
  apiKey,
  baseURL: "https://api.infrai.cc/v1",
  maxRetries: 4,
});

type LedgerRow = {
  tenantId: string;
  operationId: string;
  stage: "safety_gate" | "image_generation";
  costUsd: number;
};

type SafetyDecision = {
  allow: boolean;
  category: "allowed" | "sexual" | "violence" | "hate" | "privacy" | "other";
  explanation: string;
};

const ledger: LedgerRow[] = [];

function stableKey(...parts: string[]): string {
  return createHash("sha256").update(parts.join("\u0000")).digest("hex");
}

function costFrom(response: Response): number {
  const raw = response.headers.get("x-infrai-cost-usd");
  const value = raw === null ? Number.NaN : Number(raw);
  if (!Number.isFinite(value)) {
    throw new Error("The response did not include valid cost metadata");
  }
  return value;
}

function writeLedger(row: LedgerRow): void {
  const duplicate = ledger.some(
    (item) => item.tenantId === row.tenantId
      && item.operationId === row.operationId
      && item.stage === row.stage,
  );
  if (!duplicate) ledger.push(row);
}

async function generateKnowledgeImage(input: {
  tenantId: string;
  answerId: string;
  prompt: string;
}): Promise<{ imageUrl: string; costUsd: number }> {
  const operationId = stableKey(input.tenantId, input.answerId, input.prompt);
  const gate = await client.chat.completions.create(
    {
      model: chatModel,
      messages: [
        {
          role: "system",
          content: "Apply the product image policy. Return only the required JSON decision.",
        },
        { role: "user", content: input.prompt },
      ],
      response_format: {
        type: "json_schema",
        json_schema: {
          name: "image_prompt_safety",
          strict: true,
          schema: {
            type: "object",
            additionalProperties: false,
            properties: {
              allow: { type: "boolean" },
              category: {
                type: "string",
                enum: ["allowed", "sexual", "violence", "hate", "privacy", "other"],
              },
              explanation: { type: "string" },
            },
            required: ["allow", "category", "explanation"],
          },
        },
      },
    },
    { headers: { "Idempotency-Key": `${operationId}-gate` } },
  ).withResponse();

  writeLedger({
    tenantId: input.tenantId,
    operationId,
    stage: "safety_gate",
    costUsd: costFrom(gate.response),
  });

  const content = gate.data.choices[0]?.message.content;
  if (!content) throw new Error("Safety gate returned no decision");
  const decision = JSON.parse(content) as SafetyDecision;
  if (!decision.allow) {
    throw new Error(`Image request denied by policy: ${decision.category}`);
  }

  const generated = await client.images.generate(
    { model: imageModel, prompt: input.prompt, n: 1 },
    { headers: { "Idempotency-Key": `${operationId}-image` } },
  ).withResponse();

  writeLedger({
    tenantId: input.tenantId,
    operationId,
    stage: "image_generation",
    costUsd: costFrom(generated.response),
  });

  const imageUrl = generated.data.data[0]?.url;
  if (!imageUrl) throw new Error("Image response did not include a URL");

  return {
    imageUrl,
    costUsd: ledger
      .filter((row) => row.tenantId === input.tenantId && row.operationId === operationId)
      .reduce((sum, row) => sum + row.costUsd, 0),
  };
}

const result = await generateKnowledgeImage({
  tenantId: "tenant_acme",
  answerId: "answer_8421",
  prompt: "A clean isometric diagram of a three-stage invoice approval workflow",
});

console.log(result);

The before/after is easy to inspect. Before the change, an image call is a cost with no policy record and possibly no tenant dimension. After it, one operation ID connects a typed decision, a generation, and two cost observations. Alert on denied-request rate separately from transport errors; they mean different things. A denial is expected policy behavior. HTTP 429 means the caller should back off, and the client is configured to do that rather than spin.

Infrai's broader discovery surface reports 295 capabilities across 20 modules, with runnable examples in 10 languages. That breadth is a supporting operational benefit if the same SaaS later adds storage, scheduling, or notifications under one key. It isn't a reason to collapse every subsystem into one vendor, and it doesn't change the safety architecture above.

Retry decisions without losing policy boundaries

The catch is the added model call. It adds latency and cost before every generated image, so this shape is not suitable for an ultra-fast generator where the extra hop violates the interaction budget. A specialist image provider with an integrated safety contract is the better choice there. It is also the better choice when auditors require a named moderation product rather than an application-defined classifier.

JSON Schema guarantees the decision's shape, not the quality of the policy judgment. Teams still need a reviewed evaluation set, policy versioning, access controls around private prompts, and monitoring for shifts in allow/deny distributions. Your mileage may vary — especially for domain language that looks harmful out of context — so launch thresholds should come from reviewed samples, not a generic score copied from another product.

Infrai has no moderation-specific route. Treat that as a capability boundary, not a hidden native control: pre-check with chat completions, generate only after an allow decision, and optionally review user-visible descriptions or metadata afterward. If that boundary fits your system, start with the error semantics reference so rejected requests, rate limits, and retryable transport errors stay distinct in logs.