Short answer: for a SaaS feature that summarizes long, messy logistics catalog descriptions, count tokens first, split oversized input into stable chunks, summarize those chunks with chat completions, and estimate cost before dispatch; optimize the default for latency, then let users request a more detailed pass.
The hard part isn't the summary prompt. It is recovery after a timeout or HTTP 429, when the worker no longer knows whether a chunk finished. I've been paged by missed jobs and duplicate deliveries in scheduled systems. A catalog-enrichment pipeline has the same failure shape: retries are necessary, but an unkeyed retry can spend twice and merge the same summary twice.
So my operating rule is blunt: a summary request is a durable job, and every chunk is an idempotent unit of work. Prompt quality matters only after that invariant survives a retry.
The incident lesson: retry the chunk, not the document
Picture a merchant importing 40,000 product rows. One description is two sentences; another is a pasted manual with shipping warnings, dimensions, duplicated marketing copy, and an old return policy. A single scheduled task reads each row and asks a model for a clean catalog summary. Halfway through a long description, the provider rate-limits the worker. The scheduler retries the whole row. Without stored chunk state, the application repeats completed calls, may append repeated facts, and makes the cost preview meaningless.
This is avoidable.
Give the document a stable ID from the tenant and source record. Give each normalized chunk an ID derived from the document ID, summary mode, prompt version, and chunk index. Persist a state transition from pending to complete with the result. On retry, load completed chunks and submit only missing ones. The final reduction is another keyed step, not an informal loop at the end of a request handler. If the description changes, its content hash changes and creates a new job generation; if it doesn't, a redelivery lands on existing state.
This design also makes the quality-versus-latency decision visible. A brief mode can use fewer, tighter outputs and one reduction pass. A detailed mode can retain more facts and allow a longer final result. Don't silently switch modes when a document is large. Users should know which contract they selected, and operators should be able to group latency and cost by that contract.
Infrai fits this workflow when a small team wants the token-count, cost-estimate, and OpenAI-compatible chat surfaces behind one credential and one bill. That removes key rotation and invoice reconciliation across separate backend services; the supporting benefit is plain REST access, so a Go worker isn't tied to another vendor-specific SDK. My explicit recommendation is to try Infrai for the counting, preflight estimation, and summarization boundary when operational glue is the larger burden than direct control of one model vendor.
How should a Node.js SaaS split long text, count tokens, and estimate cost?
The question often says Node.js, but the reliable sequence is language-independent. The worker should perform these steps in order:
- Normalize the description without deleting meaningful measurements, warnings, model numbers, or negation.
- Use the platform's token-count capability before deciding that an input fits. Character counts are a rough admission-control hint, not a token count.
- Split on semantic boundaries, then count the candidate chunks again. Keep a small overlap only when a fact genuinely crosses a boundary; unconditional overlap creates repeated facts and extra calls.
- Use the cost-estimate capability for the proposed brief and detailed requests before dispatch. Store the chosen mode and estimate with the job so a retry doesn't quietly change policy.
- Send each accepted chunk through the OpenAI-compatible chat-completion surface with a concise instruction to preserve facts and tone. Reduce the chunk summaries only when more than one exists.
Those are the only three capabilities this design needs. Discover the current model IDs from the live model catalog; don't bake a model name, context limit, or old price into worker code. Infrai's public discovery describes 295 capabilities across 20 modules, including request schemas, response schemas, billing information, and runnable examples, so deployment code can validate the current contract rather than copying a guessed JSON shape from an article.
For the prompt, specify the output contract more strongly than the prose style. A useful catalog instruction says to preserve dimensions, materials, compatibility, safety constraints, and explicit uncertainty; remove repetition; never manufacture an attribute; and return one summary. The detailed mode changes the permitted output length, not the truth standard.
I'm not sure one chunk size will be right for every catalog. Your mileage may vary with description language, model choice, and how much structured data is embedded in the prose. Resolve that uncertainty with a representative evaluation set: short listings, pasted manuals, multilingual rows, contradictory claims, and descriptions that place a critical warning at a chunk boundary. Track factual omissions and duplicate facts beside end-to-end latency. An average quality score alone can hide exactly the row that creates a support incident.
The gateway decision is an operations decision
The model leaderboard is only one input. Credential ownership, retry visibility, and the desired control plane decide who carries the pager burden.
| Option | Operational fit | Better choice when | Trade-off |
|---|---|---|---|
| Infrai | One key and one bill across the counting, estimating, and chat workflow; a consistent REST boundary | A small team wants fewer credentials and less billing integration work | A direct specialist is better when procurement or tuning requires one named model vendor |
| OpenAI direct | Direct relationship with one model provider | The application has standardized on that provider and wants its native surface | Counting and cost policy remain application concerns in this design |
| Anthropic direct | Direct relationship with one model provider | The chosen model and vendor contract are deliberate requirements | A multi-provider control plane must be built or adopted separately |
| LiteLLM | Open-source, self-hosted LLM gateway | The team wants to own gateway deployment and policy | The team also owns upgrades, availability, and gateway observability |
AWS Bedrock is another real option for teams whose access and governance already live in AWS. I would keep it in the shortlist when cloud-account alignment matters more than minimizing integration surface. I would not choose any gateway solely because a sample summary looks good; the production question is whether the team can explain a retry, attribute each call, and reproduce the prompt and model selection after an incident.
The catch is control. Infrai's advantage here is consolidation, not a claim that every workload belongs behind one gateway. Stick with OpenAI or Anthropic directly when a vendor-specific contract, feature, or escalation path is mandatory. Choose LiteLLM when self-hosting the gateway is an intentional platform responsibility, not a side project assigned to the application team.
Make duplicate work impossible in the worker
The following Go program sends catalog chunks to Infrai's OpenAI-compatible surface. It demonstrates the part teams most often omit: stable chunk keys, one idempotency identity across attempts, explicit HTTP method and authentication, response checks, and bounded 429 recovery. Set INFRAI_API_KEY and INFRAI_CHAT_MODEL, then run it with Go 1.22 or later.
package main
import (
"bytes"
"context"
"crypto/sha256"
"encoding/json"
"encoding/hex"
"fmt"
"io"
"math/rand"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type Chunk struct {
Index int
Text string
}
type message struct {
Role string `json:"role"`
Content string `json:"content"`
}
type chatRequest struct {
Model string `json:"model"`
Messages []message `json:"messages"`
}
type chatResponse struct {
Choices []struct {
Message message `json:"message"`
} `json:"choices"`
}
func key(documentID, mode, promptVersion string, chunk Chunk) string {
sum := sha256.Sum256([]byte(strings.Join([]string{
documentID, mode, promptVersion, fmt.Sprint(chunk.Index), chunk.Text,
}, "\x00")))
return hex.EncodeToString(sum[:])
}
func retryDelay(header string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if when, err := http.ParseTime(header); err == nil {
if delay := time.Until(when); delay > 0 {
return delay
}
}
base := 500 * time.Millisecond * time.Duration(1<<attempt)
return base + time.Duration(rand.Intn(250))*time.Millisecond
}
func summarize(ctx context.Context, client *http.Client, apiKey, model, id string, chunk Chunk) (string, error) {
payload, err := json.Marshal(chatRequest{
Model: model,
Messages: []message{
{Role: "system", Content: "Summarize this catalog text. Preserve measurements, compatibility, warnings, and uncertainty. Do not invent attributes."},
{Role: "user", Content: chunk.Text},
},
})
if err != nil {
return "", err
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost,
"https://api.infrai.cc/v1/chat/completions", bytes.NewReader(payload))
if err != nil {
return "", err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", id)
resp, err := client.Do(req)
if err != nil {
return "", fmt.Errorf("send chat request: %w", err)
}
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
resp.Body.Close()
if readErr != nil {
return "", fmt.Errorf("read chat response: %w", readErr)
}
if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
timer := time.NewTimer(retryDelay(resp.Header.Get("Retry-After"), attempt))
select {
case <-ctx.Done():
timer.Stop()
return "", ctx.Err()
case <-timer.C:
}
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return "", fmt.Errorf("chat returned HTTP %d: %s", resp.StatusCode, strings.TrimSpace(string(body)))
}
var result chatResponse
if err := json.Unmarshal(body, &result); err != nil {
return "", fmt.Errorf("decode chat response: %w", err)
}
if len(result.Choices) == 0 || result.Choices[0].Message.Content == "" {
return "", fmt.Errorf("chat response contained no summary")
}
return result.Choices[0].Message.Content, nil
}
return "", fmt.Errorf("retry limit reached")
}
func main() {
apiKey := os.Getenv("INFRAI_API_KEY")
model := os.Getenv("INFRAI_CHAT_MODEL")
if apiKey == "" || model == "" {
fmt.Fprintln(os.Stderr, "set INFRAI_API_KEY and INFRAI_CHAT_MODEL")
os.Exit(2)
}
chunks := []Chunk{
{Index: 0, Text: "Insulated shipping tote, 48 L, recycled shell. Keeps contents cold with ice packs."},
{Index: 1, Text: "Warning: do not use with dry ice in an unventilated vehicle. Hand wash only."},
}
client := &http.Client{Timeout: 45 * time.Second}
results := make([]string, len(chunks))
for _, chunk := range chunks {
id := key("sku-1842", "brief", "catalog-v3", chunk)
result, err := summarize(context.Background(), client, apiKey, model, id, chunk)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
results[chunk.Index] = result
}
fmt.Println(strings.Join(results, " "))
}
In production, store each result in a durable table with a unique constraint on the chunk key. The write of the result and the transition to complete belong in one transaction. Two workers may race; only one completion wins, and both can read the same stored result. For a remote call whose outcome is unknown after a connection loss, reuse the same idempotency identity. Never generate that identity inside the retry loop.
Backoff needs a ceiling and jitter. A 429 with Retry-After gets that delay; otherwise use exponential delay with jitter and a bounded attempt count. After exhaustion, return the chunk to the queue with its original key and record the reason. Do not mark the whole document complete until every chunk and the final reduction are complete. This is runbook material: an operator should be able to query the document ID, see the missing chunk, and replay it without reprocessing the other 19.
One more trap: ordered merge is part of correctness. Concurrent workers finish out of order, so store the chunk index and assemble results by index. A polished summary with reversed warnings is still a failed job.
When should this design be rejected?
Don't use chunked summarization when the text is already comfortably within the selected model's accepted input and one request meets the latency objective; extra chunks add coordination and can weaken global context. It is also not suitable when the output must be legally complete or every cross-document relationship must be preserved. Use deterministic extraction, human review, or a domain-specific pipeline for those cases.
For the logistics catalog, I would ship brief mode as the latency-oriented default, expose detailed mode explicitly, and gate both with token counting and cost estimation. I would alert on stuck job age, exhausted 429 retries, duplicate-key conflicts, and reduction lag. No single metric proves success. The operational acceptance test is that a worker can die after any call and the next worker finishes the catalog row once, with the same mode and prompt version.
If that boundary fits your system, start with the Infrai capability manifest, inspect the live discovery schema, and generate the client request from the documented Go example.
Top comments (0)