Node.js Document Q&A: Validating Chat Completion Evidence with JSON Schema

#node #documentqa #jsonschema

Use generation only after retrieval has produced enough evidence, and accept the result only when every claim points back to evidence that was actually supplied. That is the decision rule. A JSON Schema can constrain the shape of a chat completion; it cannot make a weak search result true.

For an ask-your-docs feature, I would keep four boundaries visible: semantic search finds candidates, an optional reranker orders them, the model returns typed claims, and application code validates each citation before anything reaches the UI. The application, not the model, owns the final grounding decision.

How should Node.js semantic search feed JSON Schema chat completions?

Give the answerer small, addressable evidence units rather than a document-shaped wall of text. Each chunk needs a stable identifier, a display URL or path, and its text. The retrieval stage returns those records; the generation stage sees their identifiers verbatim; the validator rejects any identifier outside that exact set. This is deliberately boring.

Good.

The useful twist is to structure claims, not merely the whole answer. A single answer string followed by three citations leaves an awkward question: which source supports which sentence? Returning an array of { text, sourceIds } makes the rendering contract explicit and lets the application discard one unsupported claim without pretending the rest of the response is bad. It also makes an evaluation more precise, because a test can compare expected evidence with evidence attached to each claim.

The plain-language data flow is: normalize the question, retrieve a broad candidate set, optionally rerank it, select a context that fits the input budget, request schema-constrained claims from a chat completion adapter, parse the JSON, verify every source id, and either render the validated claims or abstain. Do not silently replace an invented id with the nearest real one. That would turn a detectable grounding failure into a polished false citation.

Put the evidence check in code

This example keeps transport behind ChatCompletion, so the grounding logic does not depend on an SDK or a guessed HTTP route. The same function can be tested with an in-memory adapter. It expects retrieval to happen elsewhere because vector indexes, embedding models, and keyword engines have different APIs; what should remain stable is the boundary between retrieved chunks and generated claims.

type Chunk = {
  id: string;
  url: string;
  text: string;
};

type Claim = {
  text: string;
  sourceIds: string[];
};

type CitedAnswer = {
  claims: Claim[];
  canAnswer: boolean;
};

type CompletionRequest = {
  system: string;
  user: string;
  schema: typeof CITED_ANSWER_SCHEMA;
};

type ChatCompletion = (request: CompletionRequest) => Promise<string>;

const CITED_ANSWER_SCHEMA = {
  type: "object",
  additionalProperties: false,
  required: ["claims", "canAnswer"],
  properties: {
    claims: {
      type: "array",
      maxItems: 6,
      items: {
        type: "object",
        additionalProperties: false,
        required: ["text", "sourceIds"],
        properties: {
          text: { type: "string", minLength: 1, maxLength: 500 },
          sourceIds: {
            type: "array",
            minItems: 1,
            maxItems: 3,
            uniqueItems: true,
            items: { type: "string" },
          },
        },
      },
    },
    canAnswer: { type: "boolean" },
  },
} as const;

function isCitedAnswer(value: unknown): value is CitedAnswer {
  if (!value || typeof value !== "object") return false;
  const candidate = value as Partial<CitedAnswer>;
  if (typeof candidate.canAnswer !== "boolean" || !Array.isArray(candidate.claims)) {
    return false;
  }

  return candidate.claims.every(
    (claim) =>
      claim &&
      typeof claim.text === "string" &&
      claim.text.length > 0 &&
      Array.isArray(claim.sourceIds) &&
      claim.sourceIds.length > 0 &&
      claim.sourceIds.every((id) => typeof id === "string"),
  );
}

export async function answerFromDocs(
  question: string,
  chunks: Chunk[],
  complete: ChatCompletion,
) {
  const allowedIds = new Set(chunks.map((chunk) => chunk.id));
  const context = chunks
    .map((chunk) => `<source id="${chunk.id}">\n${chunk.text}\n</source>`)
    .join("\n\n");

  const raw = await complete({
    schema: CITED_ANSWER_SCHEMA,
    system:
      "Use only the supplied sources. Return one claim per independently " +
      "supported statement. If the sources are insufficient, set canAnswer " +
      "to false and return no claims.",
    user: `Question: ${question}\n\nSources:\n${context}`,
  });

  let parsed: unknown;
  try {
    parsed = JSON.parse(raw);
  } catch {
    return { status: "abstained" as const, reason: "invalid-json" as const };
  }

  if (!isCitedAnswer(parsed) || !parsed.canAnswer || parsed.claims.length === 0) {
    return { status: "abstained" as const, reason: "insufficient-evidence" as const };
  }

  const hasUnknownSource = parsed.claims.some((claim) =>
    claim.sourceIds.some((id) => !allowedIds.has(id)),
  );
  if (hasUnknownSource) {
    return { status: "abstained" as const, reason: "unknown-source" as const };
  }

  const sources = new Map(chunks.map((chunk) => [chunk.id, chunk.url]));
  return {
    status: "answered" as const,
    claims: parsed.claims.map((claim) => ({
      text: claim.text,
      sources: claim.sourceIds.map((id) => ({ id, url: sources.get(id)! })),
    })),
  };
}

There are two validation layers here. The provider-side structured-output mechanism, when available, uses the schema to constrain decoding. isCitedAnswer still protects the application boundary, while the allowedIds check enforces a rule that generic schema validation cannot express: each returned id must belong to this request's retrieved set. In production, use a standards-compatible JSON Schema validator rather than expanding the handwritten shape check; the small guard above is present so the example's trust boundary is visible.

One detail deserves care. Source text is untrusted input — documents can contain instructions that look like prompts — so the system instruction defines the job and the source wrapper labels evidence as data. Delimiters reduce ambiguity, but they aren't a security boundary. Access control must happen before retrieval, and rendered claim text still needs the same escaping as any other user-facing content.

Retrieval quality comes before fluent output

Dense semantic search is useful when reader wording differs from documentation wording. It is weaker on exact tokens such as error codes, version strings, configuration keys, and product identifiers. Keyword retrieval has the opposite bias. A practical document Q&A system often gathers candidates from both, merges duplicate chunk ids, and applies a reranker only when the remaining ordering error justifies another model call. Reranking compares a query with candidate documents and returns relevance scores; its candidate count therefore belongs in both the latency budget and the test plan.

More context isn't automatically safer. Increasing the candidate count raises input size and can place marginal passages beside strong evidence, making the generation decision harder to audit. Start with the smallest selected set that preserves retrieval recall on a fixed evaluation collection, then change one stage at a time. I don't know what top-k will be right for a new corpus, and neither does a generic benchmark; the answer depends on chunk boundaries, query mix, document duplication, and how often the correct answer spans multiple passages.

Measure the pipeline at its boundaries. Retrieval evaluation should ask whether the expected chunk appears in the candidate set. Reranking evaluation should ask where it lands. Answer evaluation should inspect claim-to-source support and correct abstention, not just whether prose resembles a reference answer. A fluent paraphrase can score poorly on string similarity while being supported; a polished unsupported sentence can score well. Those are different failures, so one aggregate score hides the part that needs work.

Consider a concrete failure drill before choosing another prompt tweak. A question contains the exact configuration key session_ttl, while the relevant guide explains session expiration in prose and a migration note contains the literal key but describes an older version. Keyword search may rank the migration note first; dense search may favor the conceptual guide; a reranker may reorder both. The right candidate set can include both, but the context selector must retain their version metadata and must not splice sentences into a synthetic chunk with no durable address. Now suppose the answerer returns two claims: one describing the current behavior with source guide-17, and one naming the configuration key with sources guide-17 and migration-04. The JSON is perfectly shaped. The application still has work to do. It must confirm that both ids were supplied, that both records were authorized for this reader, that their document versions are current enough for the question, and that each URL still resolves. If migration-04 was excluded during context trimming, the second claim is rejected even if the id exists elsewhere in the index. If the completion adapter reports 429, retry policy may resubmit the unchanged request after bounded backoff; if parsing succeeds but the model returns migration-99, retrying generation blindly is the wrong response because the trust violation is already known. This drill separates retrieval, selection, transport, shape, evidence, freshness, and rendering. Calling all seven “answer quality” would make the dashboard tidy and the diagnosis useless.

Boundary	Useful check	Failure it exposes
Corpus to index	Stable chunk id and current document version	stale or broken citation targets
Query to candidates	Expected evidence appears within the candidate budget	retrieval miss
Candidates to context	Selected chunks retain the expected evidence	ranking or truncation loss
Context to claims	Every claim is supported by its attached chunks	groundedness failure
Claims to UI	Every source resolves and every string is escaped	rendering and access-control failure

Where this architecture is the wrong fit

The catch is operational lag. This design is not suitable when documents change faster than the index can be refreshed and old citation targets cannot remain addressable. In that case, stick with live structured lookups against the system of record, or show direct search results without a generated synthesis. A schema cannot repair stale evidence.

Skip semantic search for a small corpus dominated by exact names, codes, and filters; ordinary full-text search may be easier to inspect and cheaper to operate. Skip reranking when baseline retrieval already meets the acceptance threshold or when its extra hop breaks the latency target. And do not use this pattern as an authorization layer: retrieve only chunks the caller is allowed to read before any text reaches the completion context.

Multi-document reasoning needs a stricter claim model and stronger evaluation than a single-source FAQ. If a claim depends on two passages, preserve both ids and test the pair. If the corpus cannot support the answer, refuse.

Refusal is the feature.

Operate the contract, not the prompt

Before deployment, freeze a compact set of real questions with expected chunk ids, including exact identifiers, paraphrases, ambiguous questions, missing answers, and questions that require two sources. Run that set after changes to chunking, embeddings, keyword analyzers, reranking, or context selection. A prompt edit does not deserve a free pass either — it can change abstention behavior even when retrieval stays fixed.

In the request path, record timing and outcomes for each stage: retrieval, reranking, completion, JSON parsing, citation validation, and rendering. Track candidate count, selected context size, input and output tokens, abstention reason, and retry count. A 429 should follow bounded backoff outside the grounding function; malformed JSON or an unknown citation should become an abstention, not a blind completion retry that spends the same tokens on the same evidence. Keep document text and user questions out of routine logs unless the privacy policy explicitly permits them.

The final operational check is prose, not a giant checklist: confirm that index freshness is within its promised window, stored ids still resolve, unauthorized chunks cannot enter retrieval, evaluation recall has not regressed, claim citations survive validation, the refusal state renders usefully, and token plus latency ceilings are enforced. Then ship. The durable asset is the evidence contract around the model, because models, indexes, and providers can change without changing what the application is willing to trust.