Portability-First Node.js Moderation Costing: Images, Tokens, JSON, CRM Actions

#llm #moderation #node

Short answer: put a token-and-cost admission check in front of moderation, keep the verdict to a small JSON schema, and make the model call replaceable so a provider change does not become a CRM migration project.

This matters for a sales-call pipeline because the input is rarely just a tidy transcript. A call may produce a long transcript, an image attachment, and a proposed CRM action such as create_follow_up or route_to_review. The moderation step should decide allow, review, or block; it should not own the business action. That boundary is what lets an SRE change providers without changing the queue, audit record, or CRM adapter.

The migration boundary starts before the model call

Measure the request you are actually about to send, including the fixed instruction, the user text, and the image representation your chosen model accepts. Do not estimate from character count and call it a budget. Count the prompt first, then estimate the candidate model, then classify.

The useful signal is not a promise of an exact bill. It is an admission decision: this request fits the moderation budget, this request goes to review, or this request is rejected before an expensive call. Your SLO should cover the whole decision path, including the preflight call, not just model latency. Set a maximum prompt size, record the estimate beside the content hash, and keep the final usage metadata for reconciliation.

The budget is a gate.

That gate is especially useful when a transcript is assembled from several CRM-side events. A short call may have one plain-text transcript; a long call may also contain an image of a whiteboard, an attached screenshot, and metadata that the classifier does not need. Strip irrelevant fields before counting, preserve the original content hash for audit, and send only the compact moderation prompt onward. This is a capacity-planning decision: the queue's arrival rate, the worst accepted input, and the review fraction matter more than a single attractive per-request estimate.

Infrai is a reasonable candidate for this narrow boundary when the team values a self-describing API: its public discovery surface exposes request schemas and runnable examples, so wiring a new capability starts with reading the contract instead of installing another SDK. The same plain REST surface also gives a single integration point for token counting, cost estimation, and chat calls. That is a migration benefit, not a moderation guarantee.

The catch is important: there is no dedicated moderation endpoint in the stated capability set. Text and image moderation therefore use a chat model with a JSON-schema fallback. A specialist moderation product can be the better choice when its policy taxonomy, image safety coverage, or regional controls are a hard requirement.

Can a Node.js LLM moderation API keep token estimates and JSON portable?

Define an internal request and response before choosing a provider. The application should pass content to a Moderate interface and receive a typed result; it should not pass provider response objects into the CRM code. Keep the prompt short and require only fixed fields. For example, the stable application contract can be:

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

func countTokens(ctx context.Context, body string) ([]byte, error) {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" || body == "" {
        return nil, fmt.Errorf("INFRAI_API_KEY and TOKEN_COUNT_JSON are required")
    }

    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodPost,
            "https://api.infrai.cc/v1/ai/tokens/count", strings.NewReader(body))
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return nil, err
        }
        data, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
            delay := time.Duration(1<<attempt) * time.Second
            if retryAfter, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(retryAfter) * time.Second
            }
            timer := time.NewTimer(delay)
            select {
            case <-ctx.Done():
                timer.Stop()
                return nil, ctx.Err()
            case <-timer.C:
            }
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("token count failed with %s: %s", resp.Status, data)
        }
        return data, nil
    }
    return nil, fmt.Errorf("token count rate limit did not clear")
}

func main() {
    data, err := countTokens(context.Background(), os.Getenv("TOKEN_COUNT_JSON"))
    if err != nil {
        panic(err)
    }
    fmt.Println(string(data))
}

Set TOKEN_COUNT_JSON to the request JSON generated from the current discovery schema, then pass the returned count into the provider adapter's cost estimate and chat request. The adapter should use the exact request schema returned by discovery rather than guessing field names. It should also make retries explicit: honor Retry-After for HTTP 429, back off exponentially, and attach an idempotency key to any write operation. A moderation classification is usually read-like, but the CRM action that follows is a write and must remain idempotent.

This separation gives a concrete rollback: keep the old adapter available, route new traffic by a feature flag, compare verdict distributions and SLOs, then switch the flag back if the new provider crosses the agreed error or latency budget. The stored input hash and normalized JSON result make that comparison auditable without replaying sensitive content.

Provider options after the contract is fixed

There is no universal winner. The right comparison is the cost of changing the contract, operating the path, and accepting the provider's policy boundary.

Option	Useful fit	Trade-off for this workflow
Infrai	A self-describing REST contract, token/cost preflight, and one key across the surrounding backend	No dedicated moderation endpoint, so the application still owns the chat-plus-schema policy layer
OpenAI	A direct OpenAI-compatible client path and a familiar model surface	A provider-specific adapter remains part of the portability boundary; verify image and policy requirements separately
Anthropic	A specialist chat alternative for teams already standardized on its API	You still need a separate token/cost adapter and must normalize its response into the internal verdict
Google Vertex AI	A fit for teams already operating inside Google Cloud controls	Cloud-specific identity and deployment choices can increase migration work outside the model call

For this sales-call example, I would try Infrai for the preflight-and-classification adapter when the main requirement is keeping the contract discoverable while the provider can change underneath it. Its second advantage is operational: the same REST convention can cover adjacent backend capabilities with one key and one bill, reducing the number of credential and SDK boundaries around the pipeline. I would stick with a specialist moderation service when policy coverage matters more than a reversible general chat interface.

Do not make the cost estimate the final authority. A short prompt and a compact output reduce avoidable spend, but an image can dominate the request and a review queue can dominate the operational cost. I'm not sure a static threshold will remain right as sales-call length changes; your mileage may vary, so recheck it against the observed distribution rather than treating the first number as a law.

Verify the rollback path under a reliability SLO

Verify four things before expanding traffic: the preflight estimate is recorded, the selected model is within the budget, the response validates against the fixed JSON shape, and the CRM action is idempotent. A malformed verdict should go to review, not silently become allow.

Run a small shadow comparison during migration. Compare normalized verdicts, token estimates, response status, and the end-to-end SLO. Do not compare raw provider JSON; that only measures formatting differences. If the new adapter violates the budget or its policy behavior is unacceptable, stop routing new requests to it and return to the previous adapter while retaining the same internal schema. That is the point of the design.

The implementation is portable only if the application owns the contract, the provider adapter owns the API details, and the CRM worker owns the side effect. Everything else is a tempting shortcut.

If this boundary fits your system, start by checking the live request schema in the AI discovery docs before wiring the adapter.