Gaming Catalog Enrichment: Structured JSON from Node.js Semantic Search

#python #semanticsearch #jsonschema

Short answer: for a gaming catalog built from messy descriptions, keep semantic search, chat completions, and citation validation behind a provider-neutral JSON contract; choose a provider only after it passes the same evidence and portability tests.

The constraint that changes the design is provider portability. A catalog team may switch embedding models, rerankers, or chat backends while the game cards, moderation queue, and import jobs stay put. If the first prototype lets one SDK shape the application data, that switch becomes a rewrite.

I care about the notebook-to-prod path here. A result that is easy to inspect in a Python notebook should also be easy to replay through an eval harness, log without leaking a full catalog, and move into a Node.js service later. The generated prose is the least stable part of the system. The evidence contract is the part worth keeping.

The experiment: classify before you generate

The tempting approach is to send each description straight to a chat completion and ask for a title, genre, platforms, and tags. It produces a nice demo. It also hides three different errors: the description may have been misunderstood, a field may have been invented, and a later provider may format the same answer differently.

My experiment starts with a small, explicit record. Imagine a game description such as: “A four-player couch racer with toy submarines, short rounds, and a campaign map. Works offline on the living-room console.” The catalog should extract claims, but it should not turn “short rounds” into a made-up runtime or infer a release date.

The first pass retrieves the description and nearby editorial rules. The second pass creates a structured candidate. A final validator checks that every cited evidence ID was actually supplied to the model and that every field has the agreed type. This makes the retrieval failure visible instead of letting fluent text cover it. For the submarine racer, that means a claim such as “supports offline play” can point to the exact description record, while an inferred release window is rejected or sent to review. I would preserve the raw evidence beside the candidate during the experiment, compare the selected IDs in a notebook, and only then decide how much of that metadata belongs in the production event. That small bit of extra plumbing pays off when an editor asks why a platform tag appeared or when a provider switch changes the ordering of retrieved records.

I once treated valid JSON as proof that the pipeline was healthy. It was a bad assumption. A response can parse perfectly while citing chunk-19, an ID that was never in the prompt. The parser was green; the catalog was not.

Keep it boring.

Here is the smallest contract I would put in an eval notebook. It is deliberately a provider-neutral interface, and the function name chat_completion represents an adapter owned by your application rather than a particular vendor.

import json
from typing import Any, Protocol


class ChatAdapter(Protocol):
    def chat_completion(
        self, *, messages: list[dict[str, str]], schema: dict[str, Any]
    ) -> str:
        """Return model text from the configured provider adapter."""


CATALOG_SCHEMA = {
    "type": "object",
    "required": ["title", "genres", "platforms", "claims"],
    "properties": {
        "title": {"type": ["string", "null"]},
        "genres": {"type": "array", "items": {"type": "string"}},
        "platforms": {"type": "array", "items": {"type": "string"}},
        "claims": {
            "type": "array",
            "items": {
                "type": "object",
                "required": ["text", "evidence_id"],
                "properties": {
                    "text": {"type": "string"},
                    "evidence_id": {"type": "string"},
                },
            },
        },
    },
}


def enrich_description(
    adapter: ChatAdapter, description: str, evidence: list[dict[str, str]]
) -> dict[str, Any]:
    evidence_text = "\n".join(
        f"[{item['id']}] {item['text']}" for item in evidence
    )
    messages = [
        {
            "role": "system",
            "content": (
                "Extract only claims supported by the evidence. Use null or an "
                "empty list when evidence is missing. Every claim needs one "
                "evidence_id from the supplied records."
            ),
        },
        {
            "role": "user",
            "content": f"Description:\n{description}\nEvidence:\n{evidence_text}",
        },
    ]
    raw = adapter.chat_completion(messages=messages, schema=CATALOG_SCHEMA)
    result = json.loads(raw)
    known_ids = {item["id"] for item in evidence}
    for claim in result.get("claims", []):
        if claim["evidence_id"] not in known_ids:
            raise ValueError("citation points outside the supplied evidence")
    return result

The schema is useful, but it is not a complete guarantee. Use a standard JSON Schema validator in the service as well; the example focuses on the boundary that most often gets missed, citation membership. In production I would also reject duplicate claims, cap the number of retrieved records, and retain the original description hash so a catalog editor can reproduce the decision.

The experiment is not “which model writes the best tag.” It is “which adapter preserves the same application contract while retrieval and generation change?” Measure that before copying the design into an importer.

How can Node.js semantic search produce a structured answer with citations?

Treat the Node.js layer as an orchestrator, even if the first implementation is Python. The search adapter returns records with id, text, source, and an optional page or field path. The chat adapter accepts messages plus a schema. The validator returns either a typed catalog object or a useful rejection. None of those interfaces needs to expose a provider-specific response object.

That split also clarifies what “ask your docs” means for a catalog team. Documentation search is not a magic memory lookup; it is a bounded evidence selection step. Semantic search finds descriptions that use different words from the incoming record. Reranking can improve ordering when several records are plausible, but it does not repair missing metadata. A citation should point to the source record, not to a sentence the model invented after the fact.

For chat completions, keep the prompt input small enough to replay. I log the number of candidates, selected evidence IDs, schema version, input token count, output token count, latency, and validation outcome. I do not log the entire customer catalog by default. Token cost is a real engineering constraint, and a huge context can make an eval set slower without making the answer more grounded.

There is a useful failure boundary here:

Failure	What the service should return	What to investigate
No relevant record	An empty or review-needed result	Chunking, indexing, and query wording
Relevant text, missing source ID	Reject the candidate	Ingestion metadata
Valid JSON, unknown citation	Reject before persistence	Adapter or prompt contract
Known citations, wrong genre	Flag the record	Retrieval precision and labels
Provider timeout or quota response	Retry within a bounded policy, then queue review	Capacity and backpressure

The last row is operational, not a reason to hide a failure inside a fallback answer. A catalog import should be able to pause one record and continue the batch with a visible status.

Where does provider portability actually break?

Portability breaks at the edges, not in the marketing phrase “supports JSON.” Providers differ in how they express schema constraints, how they count tokens, which embedding dimensions they return, how reranking scores are interpreted, and what retry metadata an adapter exposes. Those details matter when the same eval fixture must run against two backends.

Keep an internal capability matrix with facts your adapter can test: structured-output mode, maximum input size, embedding shape, reranking availability, streaming behavior, timeout semantics, and data-retention controls. Do not flatten every difference into a boolean called supports_ai. An adapter can expose a narrow common path and a clearly named optional capability; the application should not silently depend on the optional path.

The catch is that a portable contract can be too restrictive. If your team needs a provider-specific structured-output feature, private network boundary, or specialized reranker, a lowest-common-denominator interface may throw away a material benefit. Stick with the richer integration when that capability is central to the product, and isolate it behind a versioned adapter so the catalog domain still has one stable shape.

This is also where cost comparisons go wrong. A lower per-call number says little if the backend requires more retrieval calls, larger prompts, or manual review. Compare a fixed evaluation set with the same chunking, output limits, retry policy, and review threshold. Your mileage may vary; the right choice depends on the error cost of a wrong catalog attribute, not just the model bill.

What should the eval harness measure before production?

Start with labeled examples that include the ugly cases: contradictory platform names, a feature described as a joke, an old engine name used as a genre, and descriptions that contain no supported claim. For each record, keep acceptable values and acceptable evidence IDs. A reviewer should be able to tell why a result passed.

I would track retrieval recall, citation validity, schema validity, field precision, abstention quality, latency, and token use. Slice those metrics by description length and language before looking at one overall score. One overall score can hide the exact catalog segment that is failing.

Run the same fixtures through every adapter. A provider switch is complete only when the contract remains stable and any changed behavior is visible in the diff. If one backend returns a different title but the source claims and review status agree, that may be acceptable. If it adds a platform unsupported by the evidence, it is not.

Three words matter: measure before migrating.

I am not sure a reranker or an agent loop earns its operational cost for a small catalog. First establish that retrieval selects the right records and that the validator catches unsupported claims. Then test one added component at a time. That keeps notebook findings connected to production behavior instead of turning the system into a pile of opaque retries.