DEV Community

KenjiTanaka6849
KenjiTanaka6849

Posted on

Property Catalog Enrichment: 4 LLM API Schema Gates for Customer Support Chatbots

#llm #sre #chatbot

Short answer: the cheapest LLM API for a customer support chatbot is the one with the lowest cost per accepted catalog update, after schema validation and retry cost, not the one with the lowest advertised token rate.

For a property-management app, I would start with the least complex design that preserves that measure: one internal request type, one strict result type, a replay set made from messy catalog descriptions, and a validator before any model-written field reaches storage. Run the same corpus through GPT, Claude, Gemini, and any OpenAI-compatible candidate. Count only results that pass the same gates. This produces a decision tied to your data instead of a price table that will age before the deployment does.

I've been paged by missed jobs and duplicate deliveries while operating cron and queue infrastructure in production. The useful lesson wasn't specific to queues: success at the transport boundary is not success at the business boundary. A chatbot request can return successfully and still produce a category that doesn't exist, an invented amenity, or JSON that can't be decoded. Treating that response as completed work makes the catalog the dead-letter queue.

How should an LLM API handle customer support chatbot catalog enrichment?

Put a deterministic contract between the model and the catalog. Suppose a resident asks, "Does Harbor View allow cats, and is parking included?" The stored property description is Pet friendly. Reserved garage spaces may be leased separately. The chatbot needs a conversational answer, but the enrichment job behind it needs a much narrower artifact: recognized attributes, evidence copied from the source, and a disposition when the source doesn't support a value.

The contract should reject plausible guesses. parking_included: true is fluent and wrong; the description says that spaces may be leased separately. allows_cats: true is also too strong because "pet friendly" doesn't establish species. A useful result records both as unknown unless another authoritative field resolves them. This is where structured output correctness earns its keep: it turns uncertainty into data the application can route, rather than prose an operator has to notice.

I use four gates, in order:

  1. Syntax: the response decodes as exactly one JSON value, with no prose before or after it.
  2. Shape: required fields exist, unknown fields are rejected, and enums stay within the contract.
  3. Grounding: every asserted catalog value carries a source excerpt that appears in the input description.
  4. Commit safety: the write uses a stable idempotency key derived from the property, source revision, task, and contract version.

Stop early.

The third gate is intentionally stricter than ordinary JSON Schema. A schema can prove that parking_included is a boolean; it cannot prove that true follows from the description. Grounding needs application code or a separate verification pass. I'm not sure a single acceptance threshold transfers between portfolios, because lease language and local terminology vary. A labeled replay set from the actual portfolio would resolve that uncertainty.

A 4-gate contract is more portable than compatibility labels

GPT, Claude, and Gemini expose different native mechanisms for constrained results. OpenAI documents Structured Outputs with JSON Schema; Anthropic documents tool definitions with an input_schema; Google documents structured output using a response schema. Those mechanisms can all carry a small catalog contract, but their request envelopes, supported schema details, and response shapes are not interchangeable. An OpenAI-compatible label describes an API surface, not proof that every strict-schema behavior is identical. Test the behavior you depend on.

Compatibility must be proven.

Candidate surface Native mechanism to inspect Portability question
GPT Structured Outputs Does the chosen model and endpoint enforce the schema mode you need?
Claude Tool use with an input schema Can the adapter require the enrichment tool and normalize its result?
Gemini Structured output with a response schema Does the supported schema subset cover every constraint in the contract?
OpenAI-compatible API Provider-specific compatibility layer Are strict schema enforcement, usage fields, streaming, and errors actually compatible?

This isn't a vendor scorecard. Each surface changes over time, and model availability can differ by account or region. The comparison belongs in a versioned adapter test, not in assumptions scattered through business code. Keep the internal type deliberately boring: strings, enums, arrays with sensible bounds, and explicit nullable fields. If one candidate requires a feature that another cannot represent, decide whether that feature is a real product requirement before sacrificing portability.

The catch is that the common contract can leave model-specific accuracy on the table. A team committed to one provider, especially one using richer provider-specific tools or schema features, may be better served by its native types and SDK. Stick with the native integration when those features materially improve accepted results and migration is not an operating requirement. A thin HTTP adapter is more suitable when language neutrality, controlled failover, or routine candidate testing matters more than access to every proprietary option.

Reject bad enrichment before retrying it

Retries need classification. A timeout or rate limit can be eligible for a bounded retry with jitter. A locally detected contract violation should be recorded as schema_invalid; blindly sending the same request again can spend more money without changing the cause. A grounding failure should be unsupported_claim, and it should route to review or remain unknown. Never turn uncertainty into a default true merely to keep the pipeline moving.

The following Go path validates the response before returning an enrichment result. The endpoint and authentication are adapter configuration, so the domain layer does not depend on a vendor route. In production I would also cap response bytes, propagate a request deadline, and store only the minimum support transcript needed under the application's retention policy; they are omitted here to keep the gate itself readable.

package enrichment

import (
    "bytes"
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "net/http"
    "strings"
)

type Result struct {
    PropertyID string      `json:"property_id"`
    Revision   string      `json:"revision"`
    Attributes []Attribute `json:"attributes"`
}

type Attribute struct {
    Name       string `json:"name"`
    Value      string `json:"value"`
    Evidence   string `json:"evidence"`
    Disposition string `json:"disposition"`
}

type Client struct {
    Endpoint string
    APIKey   string
    HTTP     *http.Client
}

func (c Client) Enrich(req *http.Request, description string) (Result, error) {
    res, err := c.HTTP.Do(req)
    if err != nil {
        return Result{}, fmt.Errorf("send enrichment request: %w", err)
    }
    defer res.Body.Close()

    if res.StatusCode != http.StatusOK {
        return Result{}, fmt.Errorf("upstream status %d", res.StatusCode)
    }

    var out Result
    dec := json.NewDecoder(io.LimitReader(res.Body, 1<<20))
    dec.DisallowUnknownFields()
    if err := dec.Decode(&out); err != nil {
        return Result{}, fmt.Errorf("schema_invalid: %w", err)
    }
    if err := ensureEOF(dec); err != nil {
        return Result{}, err
    }
    if err := validate(out, description); err != nil {
        return Result{}, err
    }
    return out, nil
}

func ensureEOF(dec *json.Decoder) error {
    var extra json.RawMessage
    if err := dec.Decode(&extra); !errors.Is(err, io.EOF) {
        return errors.New("schema_invalid: trailing JSON value")
    }
    return nil
}

func validate(out Result, description string) error {
    if out.PropertyID == "" || out.Revision == "" {
        return errors.New("schema_invalid: missing identity")
    }
    for _, a := range out.Attributes {
        if a.Disposition != "asserted" && a.Disposition != "unknown" {
            return errors.New("schema_invalid: bad disposition")
        }
        if a.Disposition == "asserted" &&
            (a.Evidence == "" || !strings.Contains(description, a.Evidence)) {
            return errors.New("unsupported_claim: evidence not found")
        }
    }
    return nil
}

func NewRequest(endpoint, apiKey, idempotencyKey string, body []byte) (*http.Request, error) {
    req, err := http.NewRequest(http.MethodPost, endpoint, bytes.NewReader(body))
    if err != nil {
        return nil, err
    }
    req.Header.Set("Authorization", "Bearer "+apiKey)
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("Idempotency-Key", idempotencyKey)
    return req, nil
}
Enter fullscreen mode Exit fullscreen mode

The code does not assume that an idempotency header has universal provider semantics. The application still owns deduplication: persist the key with the accepted result under a unique constraint, and make redelivery return the prior outcome. That matters when a worker finishes the remote call and loses its lease before committing. The next worker sees the same job, and without a commit guard both can update the property or create duplicate review tasks. I first thought queue acknowledgment policy was the whole fix; it isn't. Acknowledgment controls delivery, while the database constraint controls the business effect.

There is another boundary: exact substring evidence is conservative and easy to audit, but it won't recognize paraphrases. It is suitable for high-precision fields whose source wording can be retained. It is not suitable when semantic inference is the actual job, such as mapping "five minutes from the station" into a distance band. For that case, use a reviewed ontology, labeled examples, and an evaluator that measures the permitted inference rather than weakening the evidence gate silently.

Measure accepted work, not quoted tokens

Build a replay harness around production-shaped but properly governed examples. Freeze the prompt, contract version, adapter version, and candidate model identifier for each run. For every description, record input and output usage, latency, finish reason, validation disposition, retry count, and whether a human accepted the proposed attributes. Keep the raw evidence needed for debugging subject to retention and access controls. The primary metric is accepted results per unit of spend; schema-pass rate and grounded-field precision explain why that number moved.

A cheap call that fails shape validation twice is three calls, zero accepted updates, and an operator interruption. A more expensive call that passes once can be cheaper at the workflow boundary. Prices can still be compared, but fetch current rates from each provider when the test runs and preserve the date and model identifier beside the result. Published rates, model catalogs, caching rules, and batch discounts change; an article cannot settle the cheapest option for every traffic shape.

Count accepted work.

Use two slices rather than one blended score. The first is ordinary traffic: short, clear descriptions with common fields. The second is the failure slice: negation, contradictory clauses, missing values, multilingual fragments, embedded instructions, and descriptions that mention amenities without saying the property has them. Report both. A candidate that wins the average while collapsing on negation may be a poor catalog writer, even if it remains useful for low-risk conversational drafting.

Streaming is another place where the user experience and the write path should diverge. Server-Sent Events are a standard browser mechanism for receiving a stream over an HTTP connection, and they can make an in-app answer feel responsive. Do not stream partial structured output into the catalog. Buffer the candidate result, validate the complete object, commit it once, then emit the accepted state to the UI. Fast fragments are display data, not durable facts.

Deploy it like a scheduled data migration

Shadow the adapter first: generate proposed attributes without writing them, then compare against labeled decisions. Next, canary by tenant or portfolio with a hard cap on concurrency and spend. Keep the previous contract reader during rollout so queued jobs created before a schema change remain decodable. A contract version belongs in the job payload and the idempotency key; otherwise a replay after a prompt change can look like the same operation while producing a different catalog mutation.

Alert on consequences, not merely request errors. Useful signals include queue age, oldest unprocessed revision, duplicate-key conflicts, schema-invalid rate, unsupported-claim rate, accepted updates, and review backlog. Page only on conditions that threaten the service objective, such as sustained queue age beyond the enrichment freshness target. A single malformed description belongs in a work queue with enough context to diagnose it, not in the middle of the night. This design is too heavy for a prototype that never writes durable fields, and it may be unnecessary when every generated value is reviewed before use. In those cases, a simple native integration and basic JSON decoding can be the honest choice. Once the chatbot can mutate a shared property catalog without mandatory review, though, schema enforcement, grounding, idempotency, and replay tests are the minimum control plane. Choose among APIs from the resulting evidence. The winner is the candidate that produces the required accepted work within your latency and operating constraints, and the adapter keeps that decision reversible.

References

Further reading

Top comments (0)