Evidence Lifecycles for Docs: Node.js Chat Completions with JSON Schema

#node #jsonschema #semanticsearch

Short answer: treat an ask-your-docs response as a versioned evidence record, not as chat text that happens to contain links. Semantic search may propose passages and chat completions may draft prose, but a JSON Schema plus application-level citation checks must decide whether the Node.js service returns an answer or abstains.

No evidence, no answer.

This is an architecture decision record for that boundary. The decision is to preserve immutable source identity from ingestion through retrieval, emit a small typed envelope, and verify every cited excerpt against the exact document version supplied to generation. The model never gets to invent a URL, and fluency never overrides a failed evidence check.

What should a Node.js semantic search chat completion return for docs?

It should return one of two states: an answered response with one or more verified citations, or an explicit insufficient_evidence response with no claims. A third, accidental state -- confident prose beside an empty citation array -- is invalid. So is a citation whose identifier did not appear in the retrieved candidate set, even if the identifier resembles a real document.

The public contract needs only a few fields: status, answer text, and citations containing a stable source ID, document version, locator, and verbatim excerpt. Keep display URLs outside generation and join them from trusted application metadata after verification. That division matters because a URL is application data, not creative output.

JSON Schema enforces names, types, required properties, enums, and whether extra properties are permitted. It cannot, by itself, prove that an excerpt exists in a chunk or that insufficient_evidence has an empty answer. Those are relational rules, so the application must enforce them after schema validation. Don't blur the two checks. A payload can be valid JSON, satisfy its schema, and still lie about its evidence.

The storage identity deserves more attention than most examples give it. Array position is not an identity; reranking changes position. A bare URL is not a version; content at that URL can change. Use an immutable document version and chunk ID as the evidence key, then retain the normalized bytes or text that generation saw. If a later reader opens updated content, the system must still be able to explain the older answer against the older evidence rather than quietly citing the right location with the wrong contents.

Consider a policy page ingested as version v7, split into chunks c01 through c18, and later replaced by v8 at the same URL. A response generated from v7:c12 must keep that composite identity even if v8 moves the relevant sentence to c09, changes its wording, or removes it. Joining citations by URL would make the old response appear current; joining by chunk rank would be worse, because reranking can make c12 the first candidate in one request and the sixth in another. The verifier should look up v7:c12, compare the proposed excerpt with the stored normalized text, and let the presentation layer show that the evidence came from an older version. If retention policy requires deleting v7, then the system has lost the ability to verify that answer and should say so internally rather than silently rebinding it to v8. This is less glamorous than prompt tuning -- and much more consequential for an audit.

Invariants and failure boundaries

The critical path has five boundaries: ingest, retrieve, optionally rerank, generate, and verify. Ingest assigns stable identities, records versions, and applies one documented normalization policy. Retrieval returns candidates, not facts. A reranker reorders those candidates according to relevance; the Cohere overview describes this stage as taking a query and documents and returning relevance-ordered results [1]. Generation proposes an envelope. Verification decides whether that proposal may cross the public API boundary.

Name the failures because a single "answer quality" score hides where repairs belong. A retrieval miss means the supporting passage never entered the candidate set. A ranking inversion means it was retrieved but excluded from the generation budget. Citation drift means the quoted passage exists yet does not support the nearby claim. Identity drift means the citation resolves to a different document version. Schema escape means the shape is parseable but violates the response state machine. Authorization drift means a source was readable during retrieval but is no longer readable when the answer is displayed.

These failures require separate counters and separate tests. Log a request ID, corpus version, candidate count, context size, validation outcome, abstention reason, attempt count, and stage latency. Keep source text and generated prose out of metric labels; they create high-cardinality telemetry and can leak document contents. Diagnostic logs may retain redacted records under an explicit retention policy, but that is a policy decision, not a side effect of turning on debug output.

Retries need the same skepticism. A successful final attempt doesn't erase earlier throttling, so observe attempt-level outcomes and total retry delay independently from the final request result. Cap attempts, add jitter, and distinguish an exhausted dependency from insufficient corpus evidence. I'm not sure one universal retry budget exists -- latency targets and upstream limits differ -- so load tests and production traces have to settle that choice. Your mileage may vary.

Test the boundaries independently: retrieval recall against a labeled query set, ordering quality before and after reranking, schema rejection with malformed envelopes, unknown and duplicate source IDs, excerpts copied from the wrong version, Unicode normalization differences, revoked access, dependency timeouts, and exhausted retry budgets. Prompt guidance can make the requested output clearer [2], but a prompt remains input to a probabilistic operation. It isn't the enforcement layer.

Comparing enforcement options

The relevant choice is where an unsupported claim stops and what evidence remains inspectable afterward. Model brand is secondary to that boundary.

Option	Enforcement point	What it establishes	What it cannot establish	Appropriate use
Prompt-only JSON	Generation instruction	A likely output shape	Valid state or genuine evidence	Human-reviewed exploration
Schema-validated envelope	Decode boundary	Deterministic fields and types	Source membership or excerpt match	Low-risk internal workflows
Envelope plus evidence verification	Application after generation	Shape, state, source membership, and excerpt presence	Whether prose correctly interprets the excerpt	Auditable document Q&A
Extractive passages only	Retrieval output	Direct access to stored passages	Cross-passage synthesis	Narrow policy or record lookup

The decision here is the third row, with offline evaluation measuring retrieval recall separately from citation faithfulness. The catch is that excerpt presence is necessary but not sufficient: a sentence can quote a genuine passage and draw an unsupported conclusion from it. High-consequence domains therefore need a domain-specific support check or human review, with the review tied to corpus, prompt, and model versions.

Reranking is optional. It can help when first-stage retrieval already has acceptable recall but orders a noisy candidate set poorly, which matches the query-and-documents role described in the reranking overview [1]. It cannot recover a passage that retrieval never found. Stick with the simpler pipeline when a small corpus, tight latency budget, or measured evaluation shows no useful ranking gain; add a second ranking stage only after the failure data points there.

This design is also not suitable when source versions cannot be retained or access checks cannot be repeated at response time. Fix identity and authorization first. A perfectly typed citation to a revoked document remains a security failure.

Critical path and evidence verification

The schema below is represented as Python data so the state machine stays readable and the example does not depend on a commercial SDK. In a Node.js service, compile the equivalent JSON Schema once at process startup, validate every decoded completion, and then run the same membership and excerpt checks before serializing the public response.

ANSWER_SCHEMA = {
    "type": "object",
    "additionalProperties": False,
    "required": ["status", "answer", "citations"],
    "properties": {
        "status": {
            "type": "string",
            "enum": ["answered", "insufficient_evidence"],
        },
        "answer": {"type": "string"},
        "citations": {
            "type": "array",
            "items": {
                "type": "object",
                "additionalProperties": False,
                "required": [
                    "source_id",
                    "document_version",
                    "locator",
                    "excerpt",
                ],
                "properties": {
                    "source_id": {"type": "string", "minLength": 1},
                    "document_version": {"type": "string", "minLength": 1},
                    "locator": {"type": "string", "minLength": 1},
                    "excerpt": {"type": "string", "minLength": 1},
                },
            },
        },
    },
}


def evidence_key(item):
    return item["source_id"], item["document_version"]


def verify_answer(payload, candidate_chunks):
    candidates = {evidence_key(chunk): chunk for chunk in candidate_chunks}

    if payload["status"] == "insufficient_evidence":
        if payload["answer"].strip() or payload["citations"]:
            raise ValueError("abstention cannot contain claims or citations")
        return payload

    if not payload["answer"].strip() or not payload["citations"]:
        raise ValueError("answered responses require text and citations")

    seen = set()
    for citation in payload["citations"]:
        key = evidence_key(citation)
        if key in seen:
            raise ValueError("duplicate citation")
        seen.add(key)

        chunk = candidates.get(key)
        if chunk is None:
            raise ValueError("citation is outside the candidate set")
        if citation["locator"] != chunk["locator"]:
            raise ValueError("citation locator does not match stored metadata")
        if citation["excerpt"] not in chunk["text"]:
            raise ValueError("citation excerpt is absent from its source version")

    return payload

The generation request should contain only bounded candidate records: source ID, document version, locator, and chunk text. Apply the same Unicode normalization at ingest and verification. If whitespace is collapsed, either retain an offset map back to the displayed original or expose the stored normalized form; otherwise a supposedly verbatim excerpt may be impossible to highlight even though the verifier accepted it.

Validation failure is a controlled outcome, not permission to leak raw model output. Record a compact reason and follow the product contract: return insufficient_evidence when evidence is inadequate, or a typed internal dependency outcome when generation itself did not produce a valid candidate. Those states should not share a label, because corpus repair and dependency repair belong to different owners.

Small tests pay here.

Rejected option and its valid use case

Prompt-only JSON is rejected for production because it asks one probabilistic operation to guarantee syntax, state, source membership, and factual support. Longer instructions may reduce malformed responses, but they do not convert relational evidence rules into hard constraints. They also make failures harder to classify: was the corpus incomplete, was the relevant chunk cut from context, or did generation ignore a source identifier?

It still has a legitimate use. Use prompt-only output for disposable exploration over non-sensitive documents when a human inspects every response, no automation consumes the fields, and a wrong answer has negligible cost. An extractive-only interface is often preferable when synthesis adds little value and reviewers need exact passages. Neither option is an immature version of the verified envelope; each accepts a different failure boundary.

The final criterion is durability of evidence. A useful service can reproduce which versioned source text was considered, explain why it abstained, reject a citation absent from the request's candidate set, and recheck authorization before display. If those properties survive a model or retrieval change, experiments remain reversible. If they don't, polished prose is an unaudited write into somebody else's decision process.