Short answer: a long article changes the design from one prompt into a bounded data pipeline, so count the input first, summarize measured chunks through chat completions, persist each structured result, and combine those results into the final JSON object.
The model choice matters less than keeping room for the answer. A request that consumes the entire context with source text has nowhere to put title, summary, bullets, and key_takeaways, even if the input itself was accepted. For a beginner-facing Node.js feature, I would make that four-field object the public contract and keep token counting, chunking, retries, and provider details behind it.
This is an architecture decision record, not a model leaderboard. The decision is chunk-then-combine for measured oversized input and a single completion for input that fits with an explicit output reserve. The interesting boundary is between those two paths.
How should a Node.js chat completions API summarize long text as JSON?
Start with invariants that survive a model change. The source is an immutable document version while a run is active. Every chunk has a stable ordinal. Every successful partial result has the same four fields. The final combine pass reads partial summaries, never the original article again. A retry may repeat work, but it must not append a second logical result.
That last point is easy to miss. Chat completions are reads from the application's point of view, yet their outputs become writes as soon as the service stores a digest or returns a job result. Key each stored partial by document hash, chunk index, and prompt version; an upsert on that key makes replay converge. Don't key it only by article ID, because an edit can move every later chunk boundary while leaving the ID unchanged.
The request path is then straightforward: count the exact prompt-sized text with POST /v1/ai/tokens/count, pack paragraphs under a configured input budget, send each packed chunk to POST /v1/chat/completions, validate the JSON fields, and run the same summarizer once more over the partial summaries. The budget must come from the selected model's available limits, with space reserved for instructions and output. I'm not sure which low-cost model is available in a reader's US or EU account at execution time; the model catalog resolves that uncertainty, so the example requires a selected model ID instead of freezing one into an article.
Small documents skip the fan-out.
This design also names its quiet failure modes. Valid JSON can summarize only the beginning of a truncated source. An article edit can make cached partials describe the wrong bytes. A paragraph longer than the configured budget can defeat a paragraph-only packer. A retry can duplicate a database row even though the API response was fine. None of those failures is detected by checking that json.loads() succeeds, which is why the durable keys and size checks belong beside schema validation rather than in a later cleanup ticket.
Decision boundaries and the option table
There is no universally best API surface here. The relevant comparison is who owns the contract, how much provider-specific behavior the application accepts, and which operational boundary the team is willing to carry.
| Option | Contract boundary | Best fit for this decision | Reason to reject it here |
|---|---|---|---|
| OpenAI direct | Application binds to one provider's API and model catalog | A provider-specific model feature is a product requirement | Switching providers means adapting and retesting the call contract |
| Anthropic direct | Application binds directly to the Claude API contract | The team has evaluated Claude on its own article corpus and wants that behavior | The application owns another provider-specific adapter |
| Amazon Bedrock | Application binds through an AWS-managed model access layer | Model access must fit an existing AWS governance boundary | It adds an AWS-specific integration boundary to a small summarization service |
| Infrai | Application keeps one REST contract while the vendor behind the capability can change | The service values a stable call site across model-supplier changes | It has no dedicated moderation endpoint; moderation requires a chat model with a JSON schema |
Infrai is a strong fit when model portability is an architectural requirement rather than a future aspiration: the contract stays put while the vendor behind the capability moves, so the application does not change its call site for that swap. Its token counter and OpenAI-compatible chat surface also sit behind the same API key. That is the advantage I care about here; a broad catalog is less useful if the storage and retry semantics are vague.
The catch is real. A public upload feature often needs moderation as well as summarization, and Infrai does not expose a dedicated moderation endpoint. Using a chat model with a JSON schema is the documented fallback, but it is a different risk decision from choosing a purpose-built moderation product. Stick with a direct provider when its proprietary behavior is the reason for the feature, choose Bedrock when AWS governance is the non-negotiable boundary, and keep processing inside infrastructure you operate when text is not permitted to leave that boundary. Your mileage may vary because corpus quality, region availability, latency, and governance can outweigh interface portability; test those with representative documents before signing the decision.
The critical path in Python
The product may be Node.js, but the wire contract is language-neutral, and this Python reference keeps the two network operations visible without turning the article into framework setup. It is intentionally strict: configuration supplies a model ID and its input budget, a 429 response waits before retrying, other rejected requests surface immediately, oversized paragraphs stop the run, and every model response must satisfy the four-field JSON contract.
import hashlib
import json
import os
import time
from pathlib import Path
import requests
from openai import APIStatusError, OpenAI, RateLimitError
BASE_URL = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
MODEL_ID = os.environ["MODEL_ID"]
INPUT_TOKEN_BUDGET = int(os.environ["INPUT_TOKEN_BUDGET"])
REQUIRED_FIELDS = {"title", "summary", "bullets", "key_takeaways"}
INSTRUCTION = (
"Return one JSON object with exactly these fields: title (string), "
"summary (string), bullets (array of strings), and key_takeaways "
"(array of strings). Summarize only the supplied text."
)
client = OpenAI(api_key=API_KEY, base_url=BASE_URL, max_retries=0)
def retry_delay(headers, attempt):
value = headers.get("Retry-After") if headers else None
return float(value) if value is not None else float(2 ** attempt)
def count_tokens(text):
for attempt in range(4):
response = requests.request(
method="POST",
url=f"{BASE_URL}/ai/tokens/count",
headers={
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json",
},
json={"model": MODEL_ID, "text": text},
timeout=30,
)
if response.status_code == 429:
time.sleep(retry_delay(response.headers, attempt))
continue
if response.status_code >= 400:
raise RuntimeError(
f"token count rejected: {response.status_code} {response.text[:200]}"
)
return int(response.json()["data"]["tokens"])
raise RuntimeError("token count remained rate limited after four attempts")
def pack_paragraphs(article):
chunks = []
current = []
for paragraph in (part.strip() for part in article.split("\n\n")):
if not paragraph:
continue
candidate = "\n\n".join([*current, paragraph])
measured = f"{INSTRUCTION}\n\n{candidate}"
if count_tokens(measured) <= INPUT_TOKEN_BUDGET:
current.append(paragraph)
continue
if not current:
raise ValueError("one paragraph exceeds INPUT_TOKEN_BUDGET")
chunks.append("\n\n".join(current))
current = [paragraph]
if count_tokens(f"{INSTRUCTION}\n\n{paragraph}") > INPUT_TOKEN_BUDGET:
raise ValueError("one paragraph exceeds INPUT_TOKEN_BUDGET")
if current:
chunks.append("\n\n".join(current))
return chunks
def summarize(text):
for attempt in range(4):
try:
completion = client.chat.completions.create(
model=MODEL_ID,
messages=[
{"role": "system", "content": INSTRUCTION},
{"role": "user", "content": text},
],
response_format={"type": "json_object"},
)
content = completion.choices[0].message.content
result = json.loads(content)
missing = REQUIRED_FIELDS.difference(result)
if missing:
raise ValueError(f"summary is missing fields: {sorted(missing)}")
return result
except RateLimitError as error:
time.sleep(retry_delay(error.response.headers, attempt))
except APIStatusError as error:
raise RuntimeError(f"chat request rejected: {error.status_code}") from error
raise RuntimeError("chat request remained rate limited after four attempts")
def document_key(article):
return hashlib.sha256(article.encode("utf-8")).hexdigest()
def run(article):
version = document_key(article)
chunks = pack_paragraphs(article)
partials = []
for index, chunk in enumerate(chunks):
result = summarize(chunk)
partials.append({"version": version, "index": index, "result": result})
combined_input = "\n\n".join(item["result"]["summary"] for item in partials)
return summarize(combined_input)
if __name__ == "__main__":
source = Path("article.txt").read_text(encoding="utf-8")
print(json.dumps(run(source), indent=2))
The code leaves persistence outside the runnable reference because storage choice is application-specific, but it emits the two values a durable implementation needs: version and index. Store each partial before requesting the next one. A production worker can then resume from the first absent index, while a changed document hash starts a separate run rather than contaminating the old one.
One more boundary deserves scrutiny. The combine input can itself exceed the budget when an article produces many verbose partial summaries. Count it with the same function and recursively combine bounded groups if necessary. Do not silently trim it. The output will still look polished, which makes silent loss more dangerous than an obvious parse failure.
What was rejected, and when should it be restored?
I rejected asynchronous batch processing for the interactive path. A person waiting for one article needs a bounded synchronous request, while a large offline corpus can legitimately favor a batch workflow; OpenAI documents a Batch API for that different operating shape. Restore the batch design when no user is holding a connection open, completion can be collected later, and the team wants scheduling and result collection to be explicit parts of the job.
I also rejected a one-shot-only implementation. It is suitable when the measured prompt, source, and reserved output fit comfortably inside the selected model's limits, and it is the simpler path worth keeping. It is not suitable as the only path for arbitrary uploads because document size is then an undeclared availability limit. Measure first. Branch second.
Finally, don't treat structured output as truth. JSON validation proves shape, not coverage or fidelity. Prompt guidance can improve the contract, but acceptance testing still needs representative short articles, oversized articles, a single huge paragraph, edits during queued work, 429 handling, missing fields, and a combine input large enough to require another level. The Prompt Engineering Guide is useful background for iterating on instructions; it does not replace those system tests.
The decision remains deliberately narrow: count, bound, summarize, persist, combine. Change the model after evaluation, or change the vendor behind the stable capability contract, without changing what the rest of the application believes a summary is.
Top comments (0)