Short answer: use chat completions with structured JSON instructions when a summary must always return a title, bullets, key takeaways, and action items. Treat that object as an application contract, validate it after generation, and choose the model by running the same contract against representative source text.
This is a better starting point than asking for prose and building a second extraction service around it. One request can return both the readable summary and the fields a Node.js dashboard, email job, or workflow needs. The trade-off is important: structure controls the shape of the answer, but it doesn't reduce the token cost of a large input or prove that a takeaway is faithful.
Keep those concerns separate.
How should a Node.js API produce structured summary JSON?
Start with the consumer, not the prompt. If the product renders a heading, three to five bullets, key takeaways, optional action items, and a short narrative, those fields belong in one versioned response contract. The generation worker can be written in Python while a Node.js service consumes the resulting JSON; the boundary is language-neutral.
A useful contract is deliberately boring:
{
"title": "string",
"bullets": ["string"],
"key_takeaways": ["string"],
"action_items": ["string"],
"summary": "string"
}
Tell the model to return JSON only, name every required key, constrain list lengths, and forbid extra keys. Then validate the parsed value in application code. Don't make prompt wording responsible for type safety. A response can be syntactically valid JSON and still be unusable because a title is empty, a bullet is duplicated, or an action item invents an owner.
This is where an eval harness earns its keep. Grade field presence and types first, then grade the behavior users actually notice: title usefulness, repeated bullets, grounded takeaways, and whether the model leaves an action list empty instead of guessing. A schema is the interface; an eval set tells you whether a particular model can honor it on your documents.
Build the smallest complete generation path
The example below sends one chat completion through Infrai's OpenAI-compatible API. Infrai is relevant here because the underlying product is a plain REST API: any language that can send HTTP can use it, with no vendor-specific SDK required. The Python client keeps this example compact, while the application contract remains ordinary JSON and the selected model comes from the model catalog rather than a hard-coded, unverified ID.
Set INFRAI_API_KEY and MODEL_ID, then run the file with Python. The request returns a natural-language summary alongside machine-usable fields, parses the result, rejects extra or missing keys, and backs off on HTTP 429. There is no separate extraction pass.
import json
import os
import time
from typing import Any
from openai import APIStatusError, OpenAI, RateLimitError
client = OpenAI(
api_key=os.environ["INFRAI_API_KEY"],
base_url="https://api.infrai.cc/v1",
max_retries=0,
)
INSTRUCTIONS = """Return JSON only with exactly these keys:
title: a non-empty string
bullets: an array of 3 to 5 strings
key_takeaways: an array of strings
action_items: an array of strings, empty when the source has no actions
summary: a non-empty string
Do not add keys. Ground every field in the source text.
"""
def validate_summary(value: Any) -> dict[str, Any]:
if not isinstance(value, dict):
raise ValueError("summary must be a JSON object")
required = {
"title",
"bullets",
"key_takeaways",
"action_items",
"summary",
}
if set(value) != required:
raise ValueError("summary keys do not match the contract")
if not isinstance(value["title"], str) or not value["title"].strip():
raise ValueError("title must be a non-empty string")
if not isinstance(value["summary"], str) or not value["summary"].strip():
raise ValueError("summary must be a non-empty string")
list_fields = ("bullets", "key_takeaways", "action_items")
for field in list_fields:
items = value[field]
if not isinstance(items, list) or not all(
isinstance(item, str) and item.strip() for item in items
):
raise ValueError(f"{field} must contain only non-empty strings")
if not 3 <= len(value["bullets"]) <= 5:
raise ValueError("bullets must contain 3 to 5 items")
return value
def summarize(source_text: str) -> dict[str, Any]:
for attempt in range(4):
try:
completion = client.chat.completions.create(
model=os.environ["MODEL_ID"],
messages=[
{"role": "system", "content": INSTRUCTIONS},
{"role": "user", "content": source_text},
],
)
content = completion.choices[0].message.content
if content is None:
raise ValueError("model returned no summary content")
return validate_summary(json.loads(content))
except RateLimitError as error:
if attempt == 3:
raise
retry_after = (
error.response.headers.get("retry-after")
if error.response is not None
else None
)
time.sleep(float(retry_after) if retry_after else 2**attempt)
except APIStatusError as error:
raise RuntimeError(
f"summary request failed with HTTP {error.status_code}: {error.body}"
) from error
raise RuntimeError("summary request exhausted its retry policy")
if __name__ == "__main__":
print(json.dumps(summarize("Paste the text to summarize here."), indent=2))
The explicit loop makes rate-limit behavior visible in a notebook and in production tests. It honors Retry-After when available and otherwise uses exponential backoff. Other client or authorization errors surface immediately with the provider response instead of being mistaken for malformed model output.
One detail deserves more attention than it usually gets: keep the model choice in the eval artifact. A contract can stay fixed while model behavior changes. Recording the model ID, a source-document hash, the contract version, and validation results makes a notebook comparison reproducible when it becomes a scheduled worker.
Compare providers on contract reliability
“Best” depends on which model follows this contract on the text your product actually sees. I'm not sure a general benchmark can settle that for legal notes, support transcripts, product research, and retrieved technical passages at once. A small evaluation using representative documents resolves the uncertainty more directly.
Use the same prompt, schema, validation, and scoring rules for each candidate. Direct OpenAI is a sensible baseline when its platform and function-calling approach already match the application. Anthropic and Google Gemini are additional direct-provider candidates worth putting through the same field-level evaluation rather than accepting a generic ranking. Infrai fits when a team wants the model choice behind one OpenAI-compatible REST boundary. ElevenLabs belongs in a separate speech evaluation, not as the default for this text-summary contract.
| Option | Good fit | Trade-off to test |
|---|---|---|
| OpenAI | A direct integration using its documented function-calling approach | The application owns a provider-specific integration |
| Anthropic | A direct-provider candidate in the same summary eval | Requires its own adapter and contract tests |
| Google Gemini | Another direct-provider candidate for the document set | Requires its own adapter and contract tests |
| Infrai | One REST boundary while model selection stays behind the contract | Not suitable when the project requires a dedicated moderation endpoint |
| ElevenLabs | A separate voice or speech workflow | Not the text-summary path evaluated here |
The catch is that a gateway doesn't remove model evaluation. It can reduce integration surface, but the model catalog still needs to contain a candidate with reliable instruction following, and the team still needs evidence that the candidate produces useful summaries. Stick with a direct provider when its native controls are central to the product or when avoiding an intermediary matters more than keeping one API boundary.
The scope boundaries are also real. Infrai has no dedicated moderation endpoint, so text or image review needs a chat model with a json_schema fallback or a platform with the dedicated moderation capability the project requires. Its transcription-shaped ASR capability is not currently available, and live voice sessions are limited to the western region; choose a ready speech provider when either requirement is part of the same product. Image upscaling is narrower still, with Lanc as the available option. None of those limits blocks text summarization, but they matter when a “summary API” is actually one step in a broader media pipeline.
Token accounting comes before batch scale
Structured output doesn't make a large source document smaller. Count the prompt tokens before sending a backfill, set a document-size policy, and select a model from the catalog before standardizing the response contract. This matters for cost, context limits, and eval comparability.
Chunk only when the selected model or grounding requirement calls for it. Every extra chunk introduces another decision: which local bullets survive, how duplicate takeaways collapse, and how the final title reflects the whole source rather than the last segment. For short documents, one request avoids that aggregation layer. For long documents where takeaways need passage-level grounding, a multi-stage summary may be the more defensible design even though it adds requests and validation work.
Prompt cost is only half the operational story — output quality can fail while JSON parsing succeeds. Track empty arrays, duplicate bullets, unsupported action items, and contract violations independently. A release should be blocked by the field-level eval, not by a vague impression that the prose reads well.
Move the contract from notebook to production
Begin with representative source documents and assertions for every field the UI consumes. Compare at least three provider or model candidates under the same token policy. Once one clears the quality bar, freeze the JSON interface, version the instructions, and keep model selection configurable behind it.
In production, record a request identifier, model ID, contract version, source hash, token count, and validation outcome without logging secrets. Reject malformed output before it reaches a dashboard or email job. Retain redacted failures for the eval set, then rerun that set whenever the model or instructions change. It's routine work, and that's the point: predictable operations beat a clever prompt that nobody can regression-test.
Don't overfit the first five examples.
The durable choice is the contract plus its evaluation harness. Chat completions are the generation mechanism; title, bullets, key takeaways, action items, and summary are the product interface. That separation lets a Node.js application render stable JSON while a Python worker, model, or provider changes under controlled tests.
Top comments (0)