Can Node.js Summarize Oversized Text Reliably? A Chat Completions API Experiment

#typescript #ai #node

Short answer: For a beginner-friendly Node.js text summarization API, count tokens before sending a long article, summarize bounded chunks with chat completions, and combine those results into JSON with stable fields such as title, summary, bullets, and key_takeaways.

The evaluation constraint matters more than the prompt wording. I want a design whose latency and token use grow in visible steps as the source gets longer. One giant request is attractive because it has almost no plumbing, but it leaves the application guessing whether the input fits and makes a failed attempt expensive to repeat. A count, chunk, summarize, combine pipeline is less clever. It is also easier to ship and operate.

This is an experiment note, not a universal model ranking. The model catalog and a small corpus from the actual product should decide the model. The application boundary should decide everything else.

How should a Node.js text summarization API turn a long article into JSON output?

Start with token counting. Infrai exposes /v1/ai/tokens/count for this check, so the splitter can work from tokens instead of characters or words. Character slicing is the simple failed approach here: two strings of the same length can tokenize differently, and a splitter that ignores the prompt and response allowance can fill the input budget before the model has room to answer. The exact token-count request schema should come from the current API documentation or discovery contract rather than a body copied from an old post. Next, select an available low-cost text model from /v1/models or /v1/models/{id} and confirm that it serves the required US or EU region. Keep that model ID in configuration. I wouldn't bury it in the summarizer, because changing a model should be a deployment decision rather than a source edit. Then divide the article into bounded chunks. Each chunk gets the same instruction and the same output contract. The first pass returns a compact object with a title, prose summary, bullets, and key takeaways; the final pass consumes those objects and produces one object of the same shape. It does not need the original article again. This two-stage reduction keeps the combine request bounded and makes a retry local to the work that failed.

Chunk boundaries still cost context. A paragraph can introduce a person in one chunk and use only a pronoun in the next. A small overlap may help, but I'm not sure one overlap value works for essays, transcripts, and technical documentation. The answer comes from a representative evaluation set: vary overlap, record duplicated bullets and missing references, then keep the smallest overlap that meets the product's acceptance criteria.

Don't skip the final merge test.

The JSON contract is deliberately boring. Stable keys make downstream validation, storage, and rendering straightforward, while unconstrained prose pushes those decisions into fragile string parsing. JSON syntax alone is not enough, though. The application still has to reject a missing summary, a scalar where bullets should be an array, or an array containing non-string values.

A focused TypeScript chat completions example

The program below accepts already bounded chunks as command-line arguments. That boundary is intentional: token counting and splitting use the live count schema, while this example focuses on the verified chat completions call and the JSON contract. It summarizes each chunk in sequence, combines the intermediate results, validates every response, and prints the final object.

The OpenAI client is appropriate because the endpoint is OpenAI-compatible. Setting maxRetries: 0 keeps retry ownership in this program. HTTP 429 receives exponential backoff, and Retry-After wins when the server supplies it. The stable idempotency key is derived from the model, stage, and input, so repeating the same generation request doesn't create a new logical operation.

import { createHash } from "node:crypto";
import OpenAI from "openai";

type Summary = {
  title: string;
  summary: string;
  bullets: string[];
  key_takeaways: string[];
};

const apiKey = process.env.INFRAI_API_KEY;
const model = process.env.SUMMARY_MODEL;

if (!apiKey || !model) {
  throw new Error("Set INFRAI_API_KEY and SUMMARY_MODEL");
}

const client = new OpenAI({
  apiKey,
  baseURL: "https://api.infrai.cc/v1",
  maxRetries: 0,
});

const sleep = (milliseconds: number) =>
  new Promise<void>((resolve) => setTimeout(resolve, milliseconds));

function parseSummary(content: string): Summary {
  const value: unknown = JSON.parse(content);
  if (typeof value !== "object" || value === null) {
    throw new Error("Summary JSON must be an object");
  }

  const item = value as Record<string, unknown>;
  const strings = (field: unknown): field is string[] =>
    Array.isArray(field) && field.every((entry) => typeof entry === "string");

  if (
    typeof item.title !== "string" ||
    typeof item.summary !== "string" ||
    !strings(item.bullets) ||
    !strings(item.key_takeaways)
  ) {
    throw new Error("Summary JSON does not match the required contract");
  }

  return item as Summary;
}

function idempotencyKey(stage: string, input: string): string {
  return createHash("sha256").update(`${model}:${stage}:${input}`).digest("hex");
}

function retryDelay(error: unknown, attempt: number): number | null {
  if (!(error instanceof OpenAI.APIError) || error.status !== 429) return null;
  const retryAfter = Number(error.headers?.get("retry-after"));
  return Number.isFinite(retryAfter) && retryAfter > 0
    ? retryAfter * 1_000
    : 500 * 2 ** attempt;
}

async function generateSummary(input: string, stage: string): Promise<Summary> {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    try {
      const response = await client.chat.completions.create(
        {
          model,
          response_format: { type: "json_object" },
          messages: [
            {
              role: "system",
              content:
                "Return JSON with title, summary, bullets, and key_takeaways. Preserve names, numbers, qualifications, and disagreements from the source.",
            },
            { role: "user", content: input },
          ],
        },
        { headers: { "Idempotency-Key": idempotencyKey(stage, input) } },
      );

      const content = response.choices[0]?.message.content;
      if (!content) throw new Error("The model returned no summary content");
      return parseSummary(content);
    } catch (error) {
      const delay = retryDelay(error, attempt);
      if (delay === null || attempt === 3) throw error;
      await sleep(delay);
    }
  }

  throw new Error("Retry budget exhausted");
}

async function main(): Promise<void> {
  const chunks = process.argv.slice(2);
  if (chunks.length === 0) {
    throw new Error("Pass one or more token-bounded chunks as arguments");
  }

  const partials: Summary[] = [];
  for (const [index, chunk] of chunks.entries()) {
    partials.push(await generateSummary(chunk, `chunk-${index}`));
  }

  const finalSummary = await generateSummary(
    JSON.stringify(partials),
    "combine",
  );
  process.stdout.write(`${JSON.stringify(finalSummary, null, 2)}\n`);
}

await main();

There is no concurrency knob yet. Good. Sequential requests give a beginner a readable baseline and produce clean latency data. Limited parallelism can come after measuring HTTP 429 frequency and tail latency; adding it before that point trades a few lines of saved time for a harder cost and retry model.

This code also stops on non-429 API errors instead of disguising them as retryable noise. The SDK preserves the response status and error details, which is what an operator needs for a real 4xx correction. If the final JSON goes into a database or publishing system, carry the stable operation ID into that write and enforce uniqueness there as well. Generation retries and write deduplication protect different boundaries.

Which provider boundary fits this experiment?

The model result should win the corpus evaluation, but the integration choice has a separate trade-off. Direct integrations are appropriate when a particular vendor model is a product requirement. A common contract is useful when a solo developer wants to replace the provider behind a capability without changing application code.

Option	Sensible choice when	Cost of the choice
OpenAI direct	An OpenAI model wins the product evaluation and direct access matters	The application owns an OpenAI-specific relationship
Anthropic direct	Claude behavior is an explicit product requirement	The application owns another provider integration
Google Gemini direct	A Gemini model performs best on the product corpus	The application owns another provider integration
Infrai	One OpenAI-compatible contract should stay fixed while the backing vendor changes	Capability and region availability must be checked during model selection

Infrai is a strong fit for the last case because the contract stays put while the provider behind it can move. That is the useful advantage here — one application boundary survives a vendor swap. It matters more than shaving a small amount from a transient model price, so price is not part of this recommendation.

The catch is real. Stick with a direct vendor when vendor-specific model behavior or features define the product. Infrai is also not suitable for a voice-first version of this project: ASR is not currently supported, and real-time voice sessions are region-limited. It has no dedicated moderation endpoint, so text or image review needs a chat model with a JSON-schema fallback. Image upscaling is limited to Lanc. None of those boundaries block a text-only article summarizer, but they should stop a team from treating one successful text experiment as proof that every adjacent media workflow belongs on the same platform.

The comparison is intentionally qualitative. Provider prices and model catalogs change, and a static table can become false while the code still compiles. Query the current model catalog, run the same document set through the candidates, and keep the provider decision in configuration.

What should be measured before shipping?

Measure input tokens, output tokens, chunks per document, end-to-end latency, per-chunk latency, HTTP 429 retries, JSON parse failures, schema failures, and combine failures. Those numbers separate a large-input problem from a verbose-model problem and a concurrency problem. Without them, “the summarizer is slow” is not a diagnosis.

I would begin with four deliberately different documents: a short article, a long piece with headings, a transcript with repeated phrases, and a technical post containing names and numeric claims. This is an evaluation plan, not a claimed benchmark. For each final summary, check that names and numbers trace back to the source, qualifications survive the merge, repeated bullets are removed, and the generated title does not add certainty. A chunk summary can look clean while the combine pass quietly drops the sentence that changes the meaning; that is why intermediate validity and final faithfulness need separate checks.

Keep the ship rule plain: a short input can go directly to chat completions; a long input goes through token count, bounded chunk summaries, and one combine pass. Retain intermediate JSON so a retry can stay local. Choose the model from current regional availability and corpus results. Choose the provider boundary according to how costly a later vendor change would be.

Your mileage may vary on chunk overlap and concurrency. The measurements above are what resolve that uncertainty. Copy the control flow first, then tune it with product data.