Failure-Bounded Node.js Topic Classification with Semantic Search and LLM Evidence

GriffinHayes3461 — Fri, 07 Aug 2026 00:40:52 +0000

The operational constraint that changes this design is the consequence of a wrong topic: a removable search facet and a compliance-routing decision do not deserve the same pipeline. Short answer: for reviewable document tagging, use Node.js to coordinate immutable source revisions, versioned embeddings, broad semantic retrieval, a separate rerank step, and an LLM classifier constrained by a topic catalog; publish only validated decisions, and preserve enough evidence to replay each stage.

Do not collapse those stages into one call. Semantic search answers which evidence looks related, reranking changes the order of that evidence, and classification applies a labeling policy. A system that stores only the final tag cannot tell which claim failed.

Keep that boundary.

This architecture decision record assumes that topics change, source documents are revised, remote calls can be retried, and ambiguous documents may go to review. It does not assume that a similarity score is a probability or that one threshold travels safely between corpora. The right threshold isn't knowable from the interface alone; labeled evaluation data from the actual corpus is what would resolve it.

What should a Node.js semantic search, embeddings, rerank, and LLM classifier store?

Store the source revision and the evidence chain, not merely topic = "security". The durable record needs a digest of normalized content, a normalization revision, the embedding revision, the topic-catalog revision, retrieved candidate identifiers and scores, the reranker revision and new ordering, the classifier-policy revision, the validated response, and the final disposition. The source object stays canonical. Every derived item points back to the exact source revision that produced it.

Four invariants carry most of the load. A source revision has exactly one normalized-content digest under a named normalization revision. An embedding belongs to that digest and its embedding revision; mixing vectors created under incompatible revisions is a data error, not model nuance. A published label must exist in the catalog revision supplied to the classifier. Reprocessing writes another decision record and moves an active pointer only after validation, rather than overwriting the prior evidence.

That last rule matters during taxonomy edits. Suppose account-security is split into authentication and authorization. Re-embedding may be unnecessary when document content has not changed, but retrieval exemplars, reranking behavior, and classifier policy may all need a new revision. Keeping those identities separate allows targeted recomputation. A single pipeline_version string can still provide a convenient deployment boundary, but it shouldn't erase the component revisions needed to explain a transition.

The publication key should be derived from stable inputs such as source revision, normalized digest, catalog revision, and pipeline revision. Then two workers processing the same revision converge on one logical result. If publication is exposed through HTTP, RFC 9110's method semantics are relevant, but they don't make an application-specific operation safe by wishful thinking: the server-side conditional write and operation key must enforce the intended replay behavior.

Keep raw content out of routine logs. Traces can carry the document revision, stage revision, candidate IDs, duration, and disposition while sensitive text remains in its governed store. Evidence snippets, if retained, need the same deletion and access policy as the source because a small excerpt can still contain the material the storage policy was meant to protect.

Name the failure boundaries before choosing a classifier

The useful boundaries are source read, extraction, normalization, chunking, embedding, candidate retrieval, reranking, classification, schema validation, and publication. AI failed is not an actionable status. retrieval_empty, catalog_violation, and publication_conflict point to different owners and different recovery paths.

Extraction is an especially quiet failure. A loader can emit an empty string for a valid document ID, flatten a table in the wrong reading order, or retain navigation text that overwhelms the body. Imagine a 12,000-document import in which 317 records use content while the adapter reads body. The embedding stage may still accept an empty or boilerplate-only value unless the boundary rejects it. Counters can look healthy while recall decays. Validate the envelope before normalization, record empty-text counts by connector and revision, and retain a content digest so a corrected adapter triggers deliberate recomputation. Retrieval has a different failure shape. pgvector supports exact and approximate nearest-neighbor search and multiple distance functions, including cosine distance; that makes it a suitable implementation example, not a guarantee that the first neighbor represents the correct policy label. Index choice affects the candidate-generation boundary, while the classifier's job remains downstream. Preserve the initial candidate set before reranking so an evaluation can distinguish “the relevant exemplar was never retrieved” from “the reranker demoted it.” Then there is drift. Document length, language, template boilerplate, new terminology, and topic prevalence can change independently. Monitor acceptance, abstention, and review outcomes by those slices rather than trusting one aggregate accuracy number. A frozen evaluation set is useful for regression checks, but live sampling catches data shapes the frozen set doesn't contain. Don't log a confidence number without its scoring method and revision — 0.82 has no stable meaning by itself.

Network uncertainty also crosses the storage boundary. A client may lose the response after a successful write and retry. A second classification call can produce another valid ordering or label, even with identical input, so blindly repeating the whole pipeline turns transport uncertainty into state ambiguity. Persist stage completion under the operation key, resume from the last validated artifact, and make the final publication conditional. A duplicate attempt should receive a deterministic existing result or a clearly handled 409 conflict, never create an untraceable second canonical decision.

Small failures compound.

Observability should therefore answer three questions without opening the document: which revision ran, where it stopped, and whether any externally visible pointer changed. Track per-stage latency and outcome counts, candidate-set size, schema rejection, unknown-topic rejection, abstention, conditional-write conflict, and review resolution. Alerting on a rising empty-candidate rate is generally more diagnostic than alerting on a generic exception total.

Compare the paths by consequence, evidence, and operating cost

The decision is not “which model is smartest?” It is how much state and review the consequence justifies. This table forces the failure boundary into the choice.

Path	Appropriate use	Evidence to retain	Main limitation
Deterministic rules	Small, stable taxonomy with explicit language	Rule revision, matched terms, precedence	Paraphrase and overlapping rules become brittle
Embedding nearest label	Reversible suggestions and exploratory facets	Vector revision, metric, neighbors, scores	Proximity is not a policy decision
Retrieve, rerank, constrained LLM	Nuanced, changing topics with reviewable mistakes	Candidates before and after rerank, catalog, validated decision	Adds latency, retained state, and evaluation work
Supervised classifier	Stable classes with representative labeled data	Dataset revision, model revision, evaluation slices	Taxonomy and distribution changes require retraining and revalidation
Human adjudication	Rare ambiguity or high-consequence routing	Assignment, evidence shown, decision, reviewer policy	Throughput and consistency depend on process capacity

For a changing documentation corpus where topic tags help discovery, the staged retrieve-rerank-classify path is defensible because each claim can be evaluated independently and uncertain cases can abstain. The catch is operational ownership: it is not suitable when nobody maintains the catalog, reviews sampled errors, or governs retained evidence. In that environment, deterministic rules with an explicit unmatched state are more honest.

Stick with rules when vocabulary is controlled and precedence can be tested exhaustively. Prefer a supervised classifier when classes are stable and representative labeled examples actually exist. Send documents directly to human adjudication when a wrong label controls access, deletion, billing, or another hard-to-reverse action and the automated path has not been validated for that consequence. No component choice removes the need to name who accepts residual error.

Model operating cost as work units rather than transient prices: source bytes extracted, chunks embedded, candidate vectors examined, query-document pairs reranked, classifier input and output, review minutes, and retained evidence bytes. Digest-based caching avoids recomputing unchanged content. Revision keys prevent that cache from silently serving an artifact produced under an obsolete contract. This is dull accounting — and exactly what makes capacity reviews survive an implementation change.

Put the critical path behind replayable contracts

The deployed coordinator can be Node.js even though the contract example is Python. Language is secondary here; the important properties are injected stage boundaries, immutable inputs, explicit revisions, schema checks, and one conditional publication point. No external route is assumed.

from dataclasses import dataclass
from hashlib import sha256
from typing import Callable, Sequence


@dataclass(frozen=True)
class Candidate:
    topic: str
    evidence_id: str
    retrieval_score: float
    rerank_score: float | None = None


@dataclass(frozen=True)
class Decision:
    operation_key: str
    source_revision: str
    labels: tuple[str, ...]
    evidence: tuple[Candidate, ...]
    disposition: str


def decide_topics(
    *,
    source_revision: str,
    text: str,
    catalog_revision: str,
    pipeline_revision: str,
    allowed_topics: frozenset[str],
    embed: Callable[[str], Sequence[float]],
    retrieve: Callable[[Sequence[float], int], Sequence[Candidate]],
    rerank: Callable[[str, Sequence[Candidate]], Sequence[Candidate]],
    classify: Callable[[str, Sequence[Candidate]], Sequence[str]],
) -> Decision:
    normalized = " ".join(text.split())
    if not normalized:
        raise ValueError("document text is empty")

    digest = sha256(normalized.encode("utf-8")).hexdigest()
    operation_key = sha256(
        f"{source_revision}:{digest}:{catalog_revision}:{pipeline_revision}".encode()
    ).hexdigest()

    retrieved = tuple(retrieve(embed(normalized), 40))
    ranked = tuple(rerank(normalized, retrieved)[:8])
    labels = tuple(dict.fromkeys(classify(normalized, ranked)))
    unknown = set(labels) - allowed_topics
    if unknown:
        raise ValueError(f"unknown topics: {sorted(unknown)}")

    disposition = "accepted" if labels else "review"
    return Decision(
        operation_key=operation_key,
        source_revision=source_revision,
        labels=labels,
        evidence=ranked,
        disposition=disposition,
    )

The function intentionally does not publish. Its caller persists the retrieved set before reranking, persists the validated decision under operation_key, and conditionally advances the active pointer. Retry policy belongs at the boundary that knows whether an operation is idempotent and whether a prior result can be read back. RFC 9110 defines HTTP semantics for methods and retries; the domain still has to supply its own deduplication key and conditional state transition.

Deployment should run a candidate revision against held-out documents and a shadow sample, compare transitions by topic and data slice, inspect abstentions, and require an accountable approval before moving the active pointer. Rollback moves that pointer to a prior validated revision. It does not delete the newer evidence, because deletion would make the deployment impossible to audit. Retention limits can remove expired artifacts later under an explicit policy that also covers derived vectors and snippets.

Tests should target boundaries rather than only happy-path labels. Feed empty extracted text, duplicated workers, an unknown classifier label, a zero-candidate retrieval, reordered rerank results, and a retry after a simulated lost response. Property tests can assert that every published label belongs to its catalog and that identical stable inputs produce the same operation key. Evaluation tests, separately, measure whether the retrieved and final labels are useful; mixing data-integrity assertions with quality metrics makes both harder to interpret.

Record the rejected shortcut and its valid use case

The rejected option is to embed a document, select the nearest topic name, and overwrite the canonical record immediately. Topic names are thin policy descriptions, similarity is not calibrated confidence, and overwrite destroys the evidence needed to explain later drift. Adding an LLM after that lookup without recording candidates preserves the same architectural problem behind a more complicated call graph.

There is a valid narrow use case. The shortcut can generate removable, non-authoritative suggestions in an internal discovery interface when users can correct them, the original document remains canonical, and no access, retention, compliance, or financial action consumes the tag. Mark the output as a suggestion, attach its revision, and measure correction behavior. For everything with a harder consequence, keep the stages separate and preserve the decision trail.

The final criterion is deliberately plain: semantic retrieval finds candidate evidence, reranking prioritizes it, and an LLM classifier applies a catalog policy. Store and test them as three claims. If the system cannot show which claim produced a wrong topic, it is not ready to publish that topic as durable state.

References

RFC 9110, HTTP Semantics: https://www.rfc-editor.org/rfc/rfc9110
pgvector, vector similarity search for PostgreSQL: https://github.com/pgvector/pgvector