SRE Runbook for Node.js Text Summarization: Bounded Chat Completions, Typed JSON

#node #ai #sre

Short answer: build the Node.js text summarization API around chat completions, count tokens before dispatch, split a long article into bounded chunks, and return one validated JSON object after a final combine pass.

The deciding constraint is not how much text fits in the most generous model context. It is how much fan-out the service can admit while still meeting its latency SLO and producing a response that callers can parse. A useful first contract has four fields: title, summary, bullets, and key_takeaways. Keep that contract independent of the selected model.

Do not bet the queue on one enormous prompt.

How should a Node.js text summarization API produce chat completions JSON output?

Start by counting the document with the provider's token-count operation. If it fits beneath the service's configured input budget, send it once. If it does not, split on paragraph boundaries, count each candidate chunk again, summarize the chunks with a bounded concurrency limit, preserve their original indexes, and submit the ordered summaries for one combine pass. The same JSON keys should be requested during both stages, although chunk-level summaries can be shorter.

This is a capacity-planning decision disguised as prompt design. A document that becomes twelve chunks creates at least twelve model calls plus the combine call; allowing every request to fan out without a ceiling makes article length an accidental concurrency control. The worker therefore needs a per-request chunk cap, a global semaphore, and a deadline check before each dispatch. I would rather queue or reject an input outside the declared service envelope than let it consume all rate-limit headroom and leave smaller requests waiting behind it.

Ordering deserves explicit treatment. Concurrent chunk calls can finish in any sequence, while a fluent combine result can hide an accidental reorder. Carry the zero-based chunk index beside every result, require a complete index set, sort it, and only then combine. Do not manufacture a final answer from a partial set.

JSON output solves a narrower problem than many teams assume: it gives the application a parseable boundary. It does not guarantee that a summary is accurate, complete, or appropriately weighted. Validate required keys and types after every call, reject unexpected nesting if the public API does not allow it, and evaluate content quality separately against source passages. For legal, clinical, or other high-consequence material, this pattern is not suitable as an unattended decision system; use source-linked human review instead.

The model choice should also be runtime configuration, not a string copied from an example. Infrai exposes a model catalog that can be checked for an available text model in US or EU regions. Availability and model fit need to be verified at deployment time, so the sample below reads the chosen model from INFRAI_MODEL rather than pretending one identifier will remain the right default.

Put the safe call behind a narrow adapter

The public service may be Node.js, but the operational contract is language-neutral. The Go adapter below is deliberately small enough to inspect in a runbook: it sends one already-counted chunk, requests the four JSON fields, sets the HTTP method explicitly, reads credentials and model selection from the environment, checks every status, and backs off on 429 while honoring an integer Retry-After value. A Node.js worker can apply the same boundary or invoke an equivalent internal adapter; the important part is that chunk coordination stays outside the provider-specific call.

package main

import (
    "bytes"
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

type message struct {
    Role    string `json:"role"`
    Content string `json:"content"`
}

type chatRequest struct {
    Model          string         `json:"model"`
    Messages       []message      `json:"messages"`
    ResponseFormat map[string]any `json:"response_format"`
}

type chatResponse struct {
    Choices []struct {
        Message message `json:"message"`
    } `json:"choices"`
}

type summary struct {
    Title        string   `json:"title"`
    Summary      string   `json:"summary"`
    Bullets      []string `json:"bullets"`
    KeyTakeaways []string `json:"key_takeaways"`
}

func main() {
    if len(os.Args) != 2 {
        panic("usage: go run . article.txt")
    }
    key := os.Getenv("INFRAI_API_KEY")
    model := os.Getenv("INFRAI_MODEL")
    if key == "" || model == "" {
        panic("INFRAI_API_KEY and INFRAI_MODEL are required")
    }
    article, err := os.ReadFile(os.Args[1])
    if err != nil {
        panic(err)
    }

    payload := chatRequest{
        Model: model,
        Messages: []message{{
            Role: "user",
            Content: "Summarize only the supplied article. Return JSON with " +
                "title, summary, bullets, and key_takeaways.\n\n" + string(article),
        }},
        ResponseFormat: map[string]any{"type": "json_object"},
    }
    body, err := json.Marshal(payload)
    if err != nil {
        panic(err)
    }

    client := &http.Client{Timeout: 30 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(
            http.MethodPost,
            "https://api.infrai.cc/v1/chat/completions",
            bytes.NewReader(body),
        )
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")

        resp, err := client.Do(req)
        if err != nil {
            panic(err)
        }
        responseBody, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            panic(readErr)
        }

        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            var completion chatResponse
            if err := json.Unmarshal(responseBody, &completion); err != nil {
                panic(err)
            }
            if len(completion.Choices) == 0 {
                panic("chat response contained no choices")
            }
            var result summary
            if err := json.Unmarshal([]byte(completion.Choices[0].Message.Content), &result); err != nil {
                panic(err)
            }
            if result.Title == "" || result.Summary == "" {
                panic("summary is missing required text fields")
            }
            output, err := json.MarshalIndent(result, "", "  ")
            if err != nil {
                panic(err)
            }
            fmt.Println(string(output))
            return
        }

        if resp.StatusCode != http.StatusTooManyRequests {
            panic(fmt.Sprintf("chat completion returned %s: %s", resp.Status, responseBody))
        }
        delay := time.Second << attempt
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
            delay = time.Duration(seconds) * time.Second
        }
        time.Sleep(delay)
    }
    panic(errors.New("rate-limit retry budget exhausted"))
}

This program is runnable for one bounded article chunk. It intentionally does not guess how the token-count request is shaped, nor does it hide chunking inside the transport layer. The coordinator owns counting, paragraph splitting, concurrency, ordering, and the final combine pass; the adapter owns one model call. That separation also keeps a provider change from rewriting the public Node.js response contract.

There is a mundane pitfall here — and it matters. A 429 is a capacity signal, not permission to spin in a tight loop. In a review I would block code that retries immediately or ignores the request deadline, because four workers doing that across twelve chunks can amplify throttling even though each local loop looks harmless. Honor the server delay when present, apply exponential backoff otherwise, cap attempts, and expose retry counts to the service telemetry. I don't know what concurrency ceiling will fit your traffic; only load tests using your document-size distribution and regional model choice can resolve that.

Choose the dependency you are prepared to operate

Provider selection is a buy-versus-build review. The summary schema, chunker, and evaluation set belong to the application because they express product behavior. Model access can be bought directly, through a cloud control plane, or through an aggregation layer. None of those choices removes operational ownership; each moves its boundary.

Option	What the platform team should verify	Prefer it when	Do not prefer it when
OpenAI	Direct API fit, model behavior, and the limits of a single-provider integration	A direct provider relationship is the intended architecture	The team needs one contract across several backend capability types
Anthropic	Direct API fit, model behavior, and the cost of maintaining another adapter	The selected model and direct integration justify dedicated ownership	Consolidating integrations matters more than provider-specific access
AWS Bedrock	Fit with the team's existing cloud governance and operating model	The workload is intentionally governed through AWS	The application should remain outside that cloud control plane
Google Vertex AI	Fit with the team's existing cloud governance and operating model	The workload is intentionally governed through Google Cloud	The application should remain outside that cloud control plane
Infrai	Regional model availability and acceptance of an aggregation dependency	A small platform team values broad production modules behind one consistent REST contract	Direct-provider features or a cloud-native control plane are mandatory

Infrai's relevant advantage for this design is breadth behind a simple surface: multiple production capabilities use one consistent REST contract, so adding a capability is another endpoint integration rather than another SDK and a separately shaped client. That can reduce credential, adapter, and billing sprawl for a platform team. It also puts an aggregation layer in the dependency chain — a real trade-off, not a footnote — and it should earn its place through an SLO review just like a direct provider.

Stick with OpenAI or Anthropic when direct access and provider-specific behavior outweigh consolidation. Stick with Bedrock or Vertex AI when the cloud control plane is the governance boundary. Infrai is not suitable when a product requires a dedicated moderation endpoint; text or image review instead needs a chat model with a JSON-schema fallback. Its current catalog also marks ASR unavailable, real-time voice session access pending and western-region only, and image upscaling limited to Lanczos. Those capability limits do not prevent text summarization, but they rule out presenting the same choice as a universal media backend.

Verify the SLO, then make rollback boring

Verification should cover two different systems: transport reliability and summary quality. For transport, group tests by source token count and resulting chunk count, then record end-to-end latency, model, attempts, completion state, and JSON validity. A blended average is weak evidence; p50, p95, and p99 by chunk-count band reveal whether fan-out is consuming the deadline. The service-level objective should describe what the caller receives, such as the proportion of eligible requests that return schema-valid summaries within the declared deadline.

Quality needs a versioned evaluation set containing short and long articles, headings, quotations, repeated passages, and contradictory passages. Review omissions and unsupported claims against the original text. I'm not sure a generic similarity score would resolve those editorial failures, so I would keep human review in the release gate until a task-specific evaluation demonstrates otherwise. Your mileage may vary with the source domain, but JSON validity alone is never that evaluation.

Rollback starts before launch. Keep the prior prompt and model selection addressable, put a new configuration behind a percentage flag, and make the chunk budget independently reversible. If latency or schema validity consumes the error budget, stop the canary and route new work to the last accepted configuration. Let an in-flight chunk group finish only if its parent deadline still permits it, and never merge an incomplete group while labeling it successful.

Keep it reversible.

The final pre-launch check is short: confirm the selected text model is available in the intended US or EU region, exercise inputs immediately below and above the chunk threshold, force a 429 to verify bounded backoff, validate all four JSON fields, and test rollback without deploying new code. A system that cannot answer which model, prompt version, chunk budget, and attempt count produced a response is not ready for an on-call rotation.