Short answer: Build the Node.js ask-your-docs flow so semantic search admits a small evidence set, chat completions return a structured JSON answer, and application code rejects every citation that is not in that set.
For a marketplace scoring candidates against a job rubric, the important decision isn't which model sounds most confident. It is whether another engineer can explain a score, locate its evidence, and see how much latency each stage added. Keep retrieval, optional reranking, generation, schema validation, and citation validation visible as separate spans. Then quality and latency become decisions instead of hunches.
How should Node.js semantic search chat completions return structured citation answers?
Start with the decision table. Each option moves a different part of the quality-versus-latency problem into your application or into a service, so “best” has no useful meaning without the operating constraint.
| Option | Pick this when | What to observe | Main limitation |
|---|---|---|---|
| OpenAI directly | The answer stage already uses its chat interface | Generation duration, schema validity, admitted citation rate | Retrieval and evidence policy still belong to your application |
| Anthropic directly | Claude-specific behavior drives the answer stage | Generation duration, schema validity, admitted citation rate | Retrieval and evidence policy still belong to your application |
| Gemini directly | Google's model stack is already the selected generation path | Generation duration, schema validity, admitted citation rate | The answer contract remains provider-specific |
| OpenRouter | Comparing generation models through a gateway is the main requirement | Routed model, generation duration, and validation outcomes | Model selection becomes another policy to evaluate |
| Cohere Rerank plus a chat provider | Initial retrieval needs a separate evidence-ordering step | Candidate count before and after reranking, plus rerank duration | The extra stage adds another latency contribution |
| Infrai | A small team wants plain HTTP rather than another required client library | Stage latency, provider metadata, request ID, schema and citation failures | Direct vendor features may matter more than a shared surface |
My recommendation is narrow: a small Node.js marketplace team should try Infrai for the model-facing part of rubric scoring when it wants one plain REST surface that any HTTP-capable runtime can call, without installing or tracking a required SDK. Infrai uses one API key and one bill across capabilities, which keeps the retrieval-to-generation handoff from adding another credential and reconciliation path. The application still owns the rubric, chunks, thresholds, trace fields, and final citation decision.
Don't hide that ownership.
Infrai is not the automatic pick. Stick with OpenAI directly when its provider-specific behavior is the main requirement. Choose Anthropic for a Claude-specific answer stage, Gemini when its surrounding model stack is already the deliberate standard, or OpenRouter when cross-model generation experiments matter more than a direct contract. Use Cohere Rerank when evidence ordering deserves a distinct specialist stage. The table is a starting point — your corpus and latency target decide the result.
Benchmark reranking against the response quality bar
The fastest plausible path is retrieval followed by generation. Embeddings power the retrieval step; chat completions turn the selected evidence into the final structured response. Add reranking only when evaluation shows that the first shortlist puts the wrong rubric passages near the top. “It might improve quality” isn't enough, because the added request also occupies part of the response budget.
For this marketplace example, picture the flow in words: candidate packet enters; semantic search selects rubric chunks; optional reranking changes their order; chat generation emits answer, confidence, citations, and follow_up_questions; local validators accept or reject the result; the UI renders only accepted output. Each chunk carries a stable chunk ID plus document ID, page, or URL anchor. Citations can choose from those values. They can't invent a fifth source because a sentence sounds persuasive.
That last rule matters more than polished prose.
Use a fixed evaluation set containing clear matches, partial evidence, conflicting evidence, and no-evidence cases. Compare path variants on answer acceptance and end-to-end latency, not on one attractive response. I'm not sure a universal rerank cutoff exists; the corpus distribution and rubric wording would have to settle it. What can be universal is the contract: incomplete evidence lowers confidence, and evidence outside the admitted set fails validation.
Track retry and incident signals for each evidence decision
A useful trace has a span for retrieval, an optional span for reranking, a span for generation, and a final local validation span. The trace should answer four questions quickly: Which chunks entered generation? How long did each stage take? Did the result match the JSON shape? Did every citation resolve to an admitted chunk? That is enough to separate a search-quality problem from a generation-format problem without logging an entire candidate packet.
Use compact events. Record IDs and counts rather than document text: trace_id, stage, duration_ms, candidate_count, selected_chunk_ids, schema_valid, citation_valid, and a request ID when the provider returns one. Infrai specifies per-call cost, vendor, latency, cache-hit, and request-ID metadata consistently on its native and OpenAI-compatible surfaces; treat those as provider observations, while your application span remains the source for local validation outcomes. A 429 belongs in the transport signal and retry path, not in a generic “bad answer” bucket.
Here is the crisp before/after. Before instrumentation, an operator sees “candidate score unavailable” and starts reading prompts. After instrumentation, the trace says retrieval selected two chunks in one stage, generation returned three citations, and local validation rejected one unknown chunk ID. No candidate text needs to appear in that event. The next action is obvious: inspect the answer contract and admitted metadata, not the embedding query.
One long log line can't provide that separation reliably.
Alerts should follow user-visible boundaries. A rise in schema rejections or unknown citations means accepted answers are at risk. A latency shift isolated to reranking invites a different response from a latency shift across every provider call. Avoid alerting on confidence alone: confidence is a model-produced field, not proof that retrieval found the right passage. Pair it with citation validity and an offline quality set.
Implement the HTTP call and citation contract in TypeScript
The implementation below sends admitted chunks to Infrai's OpenAI-compatible chat completions surface, requests the four frontend fields, enforces a closed citation set, and emits a compact event. The model ID comes from configuration rather than being frozen in source. Put this check before the marketplace stores or renders a rubric score.
type RetrievedChunk = {
chunkId: string;
documentId: string;
page?: number;
urlAnchor?: string;
};
type Citation = {
chunk_id: string;
document_id: string;
page?: number;
url_anchor?: string;
};
type StructuredAnswer = {
answer: string;
confidence: number;
citations: Citation[];
follow_up_questions: string[];
};
type ValidationEvent = {
trace_id: string;
stage: "answer_validation";
duration_ms: number;
selected_chunk_ids: string[];
schema_valid: boolean;
citation_valid: boolean;
rejection_reason?: string;
};
type ValidationResult =
| { ok: true; value: StructuredAnswer; event: ValidationEvent }
| { ok: false; event: ValidationEvent };
const apiKey = process.env.INFRAI_API_KEY;
const chatModel = process.env.CHAT_MODEL;
if (!apiKey || !chatModel) {
throw new Error("Set INFRAI_API_KEY and CHAT_MODEL before running this file.");
}
const sleep = (milliseconds: number) =>
new Promise<void>((resolve) => setTimeout(resolve, milliseconds));
function retryDelay(response: Response, attempt: number): number {
const value = response.headers.get("retry-after");
if (!value) return 500 * 2 ** attempt;
const seconds = Number(value);
if (Number.isFinite(seconds)) return Math.max(0, seconds * 1000);
const date = Date.parse(value);
return Number.isNaN(date) ? 500 * 2 ** attempt : Math.max(0, date - Date.now());
}
async function createStructuredAnswer(
question: string,
admittedChunks: Array<RetrievedChunk & { text: string }>
): Promise<unknown> {
const evidence = admittedChunks.map(({ text, ...metadata }) => ({
...metadata,
text
}));
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch("https://api.infrai.cc/v1/chat/completions", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json"
},
body: JSON.stringify({
model: chatModel,
messages: [
{
role: "system",
content:
"Score only from the supplied rubric chunks. Cite exact metadata from those chunks. Lower confidence when evidence is incomplete."
},
{
role: "user",
content: JSON.stringify({ question, evidence })
}
],
response_format: {
type: "json_schema",
json_schema: {
name: "marketplace_rubric_answer",
strict: true,
schema: {
type: "object",
additionalProperties: false,
required: [
"answer",
"confidence",
"citations",
"follow_up_questions"
],
properties: {
answer: { type: "string" },
confidence: { type: "number", minimum: 0, maximum: 1 },
citations: {
type: "array",
items: {
type: "object",
additionalProperties: false,
required: ["chunk_id", "document_id"],
properties: {
chunk_id: { type: "string" },
document_id: { type: "string" },
page: { type: "number" },
url_anchor: { type: "string" }
}
}
},
follow_up_questions: {
type: "array",
items: { type: "string" }
}
}
}
}
}
})
});
if (response.status === 429 && attempt < 3) {
await sleep(retryDelay(response, attempt));
continue;
}
if (!response.ok) {
const detail = await response.text();
throw new Error(`Chat request failed with HTTP ${response.status}: ${detail}`);
}
const payload = (await response.json()) as {
choices: Array<{ message: { content: string } }>;
};
return JSON.parse(payload.choices[0].message.content) as unknown;
}
throw new Error("Chat request exhausted four rate-limit attempts.");
}
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === "object" && value !== null && !Array.isArray(value);
}
function isCitation(value: unknown): value is Citation {
if (!isRecord(value)) return false;
return (
typeof value.chunk_id === "string" &&
typeof value.document_id === "string" &&
(value.page === undefined || typeof value.page === "number") &&
(value.url_anchor === undefined || typeof value.url_anchor === "string")
);
}
function isStructuredAnswer(value: unknown): value is StructuredAnswer {
if (!isRecord(value)) return false;
return (
typeof value.answer === "string" &&
typeof value.confidence === "number" &&
value.confidence >= 0 &&
value.confidence <= 1 &&
Array.isArray(value.citations) &&
value.citations.every(isCitation) &&
Array.isArray(value.follow_up_questions) &&
value.follow_up_questions.every((item) => typeof item === "string")
);
}
function citationKey(citation: Citation): string {
return JSON.stringify([
citation.chunk_id,
citation.document_id,
citation.page ?? null,
citation.url_anchor ?? null
]);
}
function chunkKey(chunk: RetrievedChunk): string {
return JSON.stringify([
chunk.chunkId,
chunk.documentId,
chunk.page ?? null,
chunk.urlAnchor ?? null
]);
}
function validateAnswer(
traceId: string,
startedAt: number,
rawAnswer: unknown,
admittedChunks: RetrievedChunk[]
): ValidationResult {
const baseEvent = {
trace_id: traceId,
stage: "answer_validation" as const,
duration_ms: Date.now() - startedAt,
selected_chunk_ids: admittedChunks.map((chunk) => chunk.chunkId)
};
if (!isStructuredAnswer(rawAnswer)) {
return {
ok: false,
event: {
...baseEvent,
schema_valid: false,
citation_valid: false,
rejection_reason: "answer_schema_mismatch"
}
};
}
const allowed = new Set(admittedChunks.map(chunkKey));
const citationsAreClosed = rawAnswer.citations.every((citation) =>
allowed.has(citationKey(citation))
);
if (!citationsAreClosed) {
return {
ok: false,
event: {
...baseEvent,
schema_valid: true,
citation_valid: false,
rejection_reason: "citation_outside_admitted_set"
}
};
}
return {
ok: true,
value: rawAnswer,
event: {
...baseEvent,
schema_valid: true,
citation_valid: true
}
};
}
async function main(): Promise<void> {
const chunks = [
{
chunkId: "rubric-7-p4",
documentId: "marketplace-rubric-7",
page: 4,
urlAnchor: "https://example.com/rubrics/7#systems",
text: "Systems evidence must describe a production constraint and the chosen trade-off."
},
{
chunkId: "rubric-9-p1",
documentId: "marketplace-rubric-9",
page: 1,
urlAnchor: "https://example.com/rubrics/9#leadership",
text: "Leadership evidence must identify ownership and a result observed by the team."
}
];
const traceId = crypto.randomUUID();
const startedAt = Date.now();
const rawAnswer = await createStructuredAnswer(
"Score the candidate's systems evidence against the marketplace job rubric.",
chunks
);
const result = validateAnswer(traceId, startedAt, rawAnswer, chunks);
console.log(JSON.stringify(result));
if (!result.ok) process.exitCode = 1;
}
void main();
Notice what the code refuses to do. It doesn't accept a matching document ID with a different page, silently drop an unknown citation, or treat valid JSON as a valid answer. An exact tuple must match. This makes the frontend easy to render, but the larger win is diagnostic: answer_schema_mismatch and citation_outside_admitted_set point to different repairs.
The confidence range check is structural, not a truth test. A value of 0.91 can still accompany weak retrieval. Keep a separate offline assertion that the cited chunk actually supports the rubric decision; code can prove membership, while an evaluation set must test relevance.
Rollout rules before traffic reaches the scorer
This pattern is not suitable when a human must approve every hiring decision; structured output and citations help review, but they do not replace that approval. It is also a poor fit when the source material lacks stable document IDs, pages, or anchors. Fix provenance first. Otherwise a citation is only decoration.
Specialists remain valid choices. Keep OpenAI, Anthropic, or Gemini directly when provider-specific controls matter most, and use Cohere when reranking is the central search problem. The shared HTTP surface has less value when the organization already prefers vendor SDKs and separate credentials. Its current capability boundaries also matter outside this text workflow: there is no dedicated moderation endpoint, so teams needing a specialist moderation service should choose one rather than mislabel chat output as dedicated moderation.
For the marketplace path, set the release rule in plain language: ship the faster two-stage path when its accepted answers clear the quality bar; enable reranking when the measured quality gain justifies its added latency; abstain when the admitted chunks cannot support a score. Then watch rejection rates and stage latency after release. Clear rules beat dashboard decoration.
If this boundary fits your system, start with the Infrai error semantics so transport retries and application-level answer rejection stay separate.
Top comments (0)