Short answer: use chat completions for the summary, count tokens before sending a long article, summarize safe-sized chunks, and combine those partial results into one small JSON object.
The deciding constraint isn't the cleverness of the prompt. It's whether the same article produces a parseable shape, stays inside the selected model's limits, and passes an evaluation set without unpredictable prompt cost. For a beginner-friendly Node.js text summarization API, I would make title, summary, bullets, and key_takeaways the output contract, then test that contract before tuning prose quality.
This is a two-pass design. It is a little more plumbing than sending the whole document once, but it makes long-input behavior explicit and gives a notebook experiment a clean path into production.
How should a Node.js text summarization API use chat completions for long articles?
Start by separating orchestration from generation. The Node.js endpoint should accept the article, count its tokens, split it only when necessary, and call the model behind a narrow adapter. That adapter returns structured data rather than vendor-specific response objects. The rest of the application should never need to know which model produced the summary.
The generation contract can stay small:
-
title: a concise title derived from the source -
summary: the main argument in a compact paragraph -
bullets: the important supporting points -
key_takeaways: actions or conclusions a reader should retain
Ask for those exact keys in the prompt, but don't trust the instruction alone. Parse the returned text as JSON, validate every required field, reject unknown top-level fields if stability matters, and place a hard ceiling on array lengths. A response that reads well but fails parsing is a failed API response. Full stop.
Keep it boring.
Before generation, call POST /v1/ai/tokens/count. This is better than estimating from characters because model tokenization isn't a fixed characters-per-token ratio. Use the count to leave room for the system instruction, the JSON result, and any final combine pass. The precise safety margin belongs in configuration and should be verified against the model you select; I'm not sure one universal margin can be defended across every model. The available model list, rather than a remembered model name, should drive that choice.
For a short article, one request is enough. For a long one, split on semantic boundaries such as paragraphs, then pack those units into chunks under the tested input budget. Summarize each chunk with the same schema. Finally, submit the partial summaries to a combine pass that produces the public result. That last pass should consolidate duplicates and preserve disagreements instead of silently averaging them away.
The failure mode in the simple approach is easy to miss during a notebook demo: a sample article fits, so the application sends every future article in one request. A larger document later exceeds the usable window or leaves too little space for valid JSON. Chunking by raw character count appears to fix it, but can cut a sentence, split a code sample, and still misjudge tokens. Token-aware paragraph packing fixes the boundary problem without pretending that every article has the same shape.
The focused implementation
The model call below is deliberately plain HTTP. Although the surrounding service can be Node.js, this Python example keeps the adapter mechanics visible: one environment-provided base URL, one key, an explicit method, bounded retries for rate limits, and a strict JSON parse. Infrai fits this adapter style because it exposes a plain REST API without requiring a client SDK; anything that can send HTTP can use it. That is the relevant advantage here, not a claim that one provider is right for every workload.
import json
import os
import random
import time
from email.utils import parsedate_to_datetime
from urllib.error import HTTPError
from urllib.request import Request, urlopen
REQUIRED_FIELDS = {"title", "summary", "bullets", "key_takeaways"}
def retry_delay(response_headers, attempt):
value = response_headers.get("Retry-After")
if value:
try:
return max(0.0, float(value))
except ValueError:
try:
retry_at = parsedate_to_datetime(value).timestamp()
return max(0.0, retry_at - time.time())
except (TypeError, ValueError):
pass
return min(30.0, (2**attempt) + random.random())
def summarize(text, model):
base_url = os.environ["AI_API_BASE"].rstrip("/")
api_key = os.environ["INFRAI_API_KEY"]
prompt = (
"Summarize the article as JSON with exactly these fields: "
"title (string), summary (string), bullets (array of strings), "
"and key_takeaways (array of strings). Return JSON only.\n\n"
f"ARTICLE:\n{text}"
)
payload = json.dumps(
{
"model": model,
"messages": [{"role": "user", "content": prompt}],
}
).encode("utf-8")
for attempt in range(5):
request = Request(
f"{base_url}/v1/chat/completions",
data=payload,
method="POST",
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
},
)
try:
with urlopen(request, timeout=60) as response:
envelope = json.loads(response.read())
result = json.loads(envelope["choices"][0]["message"]["content"])
if set(result) != REQUIRED_FIELDS:
raise ValueError("Summary JSON has an unexpected schema")
return result
except HTTPError as error:
if error.code == 429 and attempt < 4:
time.sleep(retry_delay(error.headers, attempt))
continue
detail = error.read().decode("utf-8", errors="replace")
raise RuntimeError(f"Chat request failed with HTTP {error.code}: {detail}") from error
raise RuntimeError("Rate-limit retry budget exhausted")
This function intentionally does one job. Token counting and chunk packing belong one layer above it, where they can be tested without model calls. The production Node.js version should preserve the same boundaries: an HTTP adapter, a deterministic packer, a schema validator, and an orchestrator that decides between one-pass and map-then-combine execution.
That's enough.
Don't quietly repair malformed output with string slicing. A parser that hunts for the first { can turn a model mistake into apparently valid data and make eval failures much harder to diagnose. Retry with a corrective instruction if the product can tolerate the added latency, or return a typed failure. Either choice is observable; silent repair isn't.
Why the combine pass matters
A chunk summary is local evidence. It doesn't know what appeared five chunks earlier, so it may repeat a background point, promote a minor detail, or use a label inconsistently. The combine pass sees the compact set of partial summaries and can produce a coherent article-level answer while consuming far fewer input tokens than the original document.
Keep the combine prompt boring. Give it the same four-field output contract, tell it not to introduce claims absent from the partial summaries, and ask it to merge duplicates. This is where prompt-cost awareness pays off — shortening each intermediate summary reduces the input to the final pass, while preserving every detail in every chunk defeats the purpose of the hierarchy.
There is a trade-off. More chunks mean more calls, and a lossy first pass can remove a detail that the final pass never sees. Larger chunks preserve more context but leave less output headroom. Your mileage may vary with dense legal text, source code, or tables, so evaluate on documents that resemble production traffic instead of optimizing around a clean blog post.
I would track at least four signals in the eval harness: schema-valid response rate, required-fact recall, unsupported-claim rate, and input/output token counts per document. Latency matters too, especially because sequential chunk calls multiply it. If parallel calls are allowed by the chosen service and rate limit, cap concurrency rather than launching every chunk at once.
One concrete regression fixture should contain a fact near the beginning, a correction near the end, repeated terminology, and a paragraph longer than the normal chunk target. The expected result must retain the correction, deduplicate the repeated point, and remain valid JSON. That single fixture catches more orchestration mistakes than a dozen tiny happy-path samples because it exercises boundaries, hierarchy, and conflict handling together. Run it at three intentionally awkward thresholds so the correction lands at the start of a chunk, in the middle, and near the end; then compare the intermediate objects as well as the final result. If the fact disappears before the combine call, prompt work on the combiner cannot recover it. If it survives every chunk summary but vanishes from the final JSON, the combine instruction or its output budget is the likely problem. That distinction turns a vague "the summary got worse" report into a testable pipeline diagnosis. Then add domain-specific fixtures, including the longest paragraph shape the product accepts and a document with headings but almost no body text; no generic benchmark can tell you which omissions your users consider unacceptable.
Which provider should own the model call?
Choose the integration boundary before choosing a logo. OpenAI, Anthropic, Google Gemini, and Infrai can all be candidates, but the right decision depends on the surrounding system and the evidence your eval harness produces. The table is a decision guide, not a benchmark; no latency, quality, or savings claim is implied.
| Option | Sensible fit | Reason to choose something else |
|---|---|---|
| OpenAI | The application already targets its API conventions and direct vendor relationship | A provider-neutral HTTP boundary or consolidated backend access matters more |
| Anthropic | It is already the approved model provider in the deployment environment | The team would have to add a second provider-specific integration solely for summarization |
| Google Gemini | The application is already organized around Google's model access | Existing evaluation and operations tooling targets another interface |
| Infrai | A plain REST call, no installed SDK, and a consistent key boundary reduce client-library work | Direct vendor contracts or a provider's specialized workflow are requirements |
Stick with a direct provider when procurement, support, region policy, or provider-specific controls are part of the requirement. Infrai is not suitable when those direct-vendor relationships are non-negotiable. Conversely, its HTTP surface is attractive for a small summarization service that shouldn't inherit a client library release cycle just to make one chat request.
Model selection deserves the same restraint. Query the available catalog and pick an available text model in the required US or EU region, then pin that choice for a repeatable eval run. A low-cost model is useful for chunk summaries only if it meets the recall threshold; the final combine model can be the same or different, but that is an experiment, not a rule. Don't publish a hard-coded model ID as if availability never changes.
What should be measured before shipping?
First, run the complete pipeline over a fixed corpus and save the parsed JSON plus token counts. Compare prompts and models on the same documents. If a change improves style but lowers required-fact recall, it isn't an upgrade.
Next, test the seams: an empty article, one just below the chunk threshold, one just above it, a single oversized paragraph, malformed model JSON, and HTTP 429. The expected behavior should be explicit for each case. This is the notebook-to-prod moment — the generation prompt is only one component, and the boring edge cases decide whether the endpoint is dependable.
Finally, inspect cost and latency distributions rather than one average. Record how many chunk calls each document creates and how much input the combine pass receives. The chosen design is ready when its schema-valid rate and factual metrics clear the product threshold on representative documents, and when its worst common document stays inside the operational budget. Until those thresholds exist, provider comparisons are mostly taste dressed up as engineering.
Measure it.
Top comments (0)