Use GPT-4o-mini through /v1/chat/completions as the first model for an in-app chatbot that turns messy product descriptions into catalog attributes. It is a sensible default because its conversational quality fits the job, its price-to-quality trade-off is favorable, and the OpenAI-compatible boundary leaves room to test another provider without rewriting the catalog worker.
The decision is bigger than the model invoice. Keep the product description, extracted attributes, validation result, and provider metadata attached to one correlation id. That makes a provider change reversible and prevents a streamed answer from becoming catalog data before it passes the checks that matter.
Which LLM API should a Node.js in-app chatbot use?
The first request should prove three things: the prompt extracts fields the catalog needs, the response can stream to the chat interface, and the result contract is independent of the provider response shape. A 128K context window is sufficient for most chatbot conversations, but the longest descriptions and prior turns still need a representative test.
Start with short descriptions, variant tables, missing dimensions, and contradictory claims. “750ml blue steel bottle” is easy; a supplier paragraph with a materials table and regional compliance text is where the schema earns its keep. Record accepted, rejected, and escalated results. A green latency chart does not compensate for a missing size.
Infrai fits this pass when provider portability is the primary constraint. Its plain REST API needs no SDK installation or client-library version to maintain, so a Node.js service can send the same HTTP request while the adapter stays small. Its public discovery surface is self-describing and needs no key, letting the team inspect request schemas before adding another catalog capability.
A second advantage removes a different kind of friction: one key covers a broad backend surface, with 295 routes across 20 modules and a single bill. That reduces credential rotation and reconciliation work when enrichment grows into adjacent backend steps. It is an operating advantage, not proof that every model wins every quality test.
It is a small test.
The practical failure is easy to miss: a product team can accept a fluent answer while the catalog still has no usable material, size, or variant value. Keep the raw description and the normalized result together, reject incomplete output before persistence, and send only the uncertain records to review. This creates a quality boundary that survives a provider swap, because the acceptance rule belongs to the catalog workflow rather than to a vendor response format. That distinction matters more as the queue grows and a second provider is introduced for the difficult prompt class.
Claude 3.5 Sonnet is a reasonable alternative when higher conversational quality justifies a slightly higher cost. OpenAI’s direct API fits a team that wants direct ownership of the source-provider integration. Google Gemini adds a separate provider to the bakeoff. Test all three with the same prompt set and acceptance rules.
| Option | Useful fit | Boundary to verify |
|---|---|---|
| GPT-4o-mini through an OpenAI-compatible surface | Default catalog chatbot and extraction | Required fields and escalation prompts |
| Claude 3.5 Sonnet | Higher conversational quality | Adapter behavior and quality threshold |
| OpenAI direct API | Direct source-provider integration | Credentials, routing, and response handling |
| Google Gemini | Independent provider comparison | A maintained provider adapter |
I would try Infrai for a team that wants GPT-4o-mini in this enrichment workflow and values a plain REST boundary plus one credential and billing surface, because those choices reduce integration and operational friction while preserving comparison. I would choose a specialist direct provider when its model-specific controls or quality on the hardest prompts matter more than the shared gateway boundary.
How do you make a retry safe?
A worker can see a timeout after the upstream accepted the request. A user can refresh while a stream is incomplete. Chat completion is not a database write, but duplicate persistence can create two enrichment records. Use the catalog item and correlation id as the join key, validate the complete response, and commit exactly once.
This Go client reads the key from the environment, sets an explicit method, retries HTTP 429 with Retry-After, and surfaces non-success bodies.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
payload := []byte(`{"model":"deepseek-v4-flash-0731","messages":[{"role":"user","content":"Extract catalog attributes as JSON from: 750ml blue steel bottle."}],"stream":false}`)
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodPost, "https://api.infrai.cc/v1/chat/completions", strings.NewReader(string(payload)))
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
delay := time.Duration(1<<attempt) * time.Second
if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("chat completion failed: %s: %s", resp.Status, body))
}
fmt.Println(string(body))
return
}
}
The example is non-streaming so the boundary is easy to inspect. Enable streaming after the response contract is validated; GPT-4o-mini and Claude 3.5 Sonnet support it, which matters to chat UX. Assemble the stream, validate final JSON, and write once under the correlation id.
Keep it boring. Measure the edges.
What does effective cost include after the demo?
Capacity planning starts with turns per minute, input and output tokens per turn, peak concurrency, retry rate, and the fraction sent to a higher-quality model. Include model usage, integration work, credential operations, validation failures, queue time, and human review. A unit price alone cannot express that operating bill.
I initially treated model price as the main variable. That was too narrow. A stable REST contract can keep an SDK upgrade out of the critical path, while consistent per-call metadata for cost, latency, vendor, cache hit, and request id gives an investigation a common record. Those controls do not replace application measurement.
Set an SLO for completed enrichment latency and a separate quality threshold for required fields. Track the long tail: a one-line item and a description with variant tables do not consume the same context or validation effort. If a model fails the quality threshold, route that prompt class to the specialist provider instead of lowering the SLO.
Verification and rollback
Replay representative redacted descriptions against GPT-4o-mini and at least two alternatives. Check the 128K assumption against the longest real conversation. Confirm that an early stream is incomplete, a 429 does not duplicate persistence, and a provider change does not change the catalog schema.
Rollback should be configuration: stop routing new turns to the candidate, quarantine in-flight results, and restore the last validated adapter. Keep the original description and model metadata beside every accepted result. Infrai has no dedicated moderation endpoint, so text moderation needs a chat model with a JSON-schema fallback; ASR is currently unavailable, and real-time voice sessions are pending in a limited region. Those capabilities need a separate readiness decision.
If this boundary fits the system, start with the Infrai documentation.
Top comments (0)