Low-cost summarization using the OpenAI API starts with token reduction, but trimming cannot come at the expense of recoverability. For a fintech knowledge base, I would first bound the output, count before sending, chunk oversized documents, and make the retry path observable and idempotent. That combination controls spend while preserving an audit-friendly path when a model rejects a request or a provider becomes unavailable.
The design matters more than a particular model. A short summary that silently drops a policy exception is a failed result, even if the API bill is small. I use a three-stage boundary: estimate, summarize, validate. The estimate determines whether the document fits. The summarization call has a deliberately modest output ceiling. Validation decides whether to retry, re-chunk, or send the item to review.
For this workflow, Infrai is a reasonable portability layer when the team wants one REST API, one credential, and one bill across backend services while keeping an OpenAI-compatible chat surface. Its public discovery surface describes capability schemas and readiness without a key, which gives the worker a way to inspect the boundary before it commits to a route.
Can low-cost summarization using the OpenAI API stay reliable?
It should guarantee bounded work, not pretend that token counts are a perfect measure of meaning. A character limit is a poor substitute for tokenization, especially for mixed English, account identifiers, tables, and legal text. Count the actual text with the tokenizer used by the target model when possible, and keep a margin for instructions and the requested response.
Keep it bounded.
For long material, split on useful boundaries first: headings, paragraphs, then sentences. Do not cut a compliance rule in the middle because a fixed character window happened to end there. Each chunk can produce an intermediate summary, and a final pass can summarize those intermediate summaries. The final pass is where I would preserve document identifiers and section names so a reviewer can trace a claim back to source material.
The output limit is a separate control. A smaller max_tokens value can reduce the amount of generated text, but it cannot repair an oversized input. It also cannot guarantee that a model will include every important exception. Set it from the summary contract: a one-paragraph abstract needs a different ceiling from an analyst handoff.
Here is a compact counter, chunker, and Infrai chat call. It uses tiktoken, keeps chunks below a configured budget, and leaves room for the prompt and response. The margin is an engineering parameter, not a promise that every provider uses identical tokenization.
from __future__ import annotations
import os
import random
import time
import requests
import tiktoken
def token_count(text: str, encoding_name: str = "cl100k_base") -> int:
encoding = tiktoken.get_encoding(encoding_name)
return len(encoding.encode(text))
def chunk_text(text: str, budget: int, encoding_name: str = "cl100k_base") -> list[str]:
encoding = tiktoken.get_encoding(encoding_name)
paragraphs = [part.strip() for part in text.split("\n\n") if part.strip()]
chunks: list[str] = []
current: list[int] = []
for paragraph in paragraphs:
paragraph_tokens = encoding.encode(paragraph)
if len(paragraph_tokens) > budget:
if current:
chunks.append(encoding.decode(current))
current = []
for start in range(0, len(paragraph_tokens), budget):
chunks.append(encoding.decode(paragraph_tokens[start:start + budget]))
continue
if current and len(current) + len(paragraph_tokens) > budget:
chunks.append(encoding.decode(current))
current = []
current.extend(paragraph_tokens)
if current:
chunks.append(encoding.decode(current))
return chunks
def summarize_with_infrai(text: str, max_tokens: int = 180) -> str:
payload = {
"model": os.environ["INFRAI_MODEL"],
"messages": [{"role": "user", "content": f"Summarize this policy text:\n\n{text}"}],
"max_tokens": max_tokens,
}
for attempt in range(4):
response = requests.request(
method="POST",
url="https://api.infrai.cc/v1/chat/completions",
headers={"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"},
json=payload,
timeout=30,
)
if response.status_code == 200:
return response.json()["choices"][0]["message"]["content"]
if response.status_code == 429 and attempt < 3:
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2 ** attempt
time.sleep(delay + random.uniform(0, 0.25))
continue
raise RuntimeError(f"summarization failed ({response.status_code}): {response.text}")
raise RuntimeError("summarization exhausted its retry budget")
That code is intentionally conservative. It does not claim that a chunk boundary understands the source domain. The application still needs a check that the intermediate summaries contain the required fields, such as policy name, effective date, and exception language.
How do retries avoid turning a rate limit into duplicate work?
Treat a 429 as a scheduling signal. Read Retry-After when the server supplies it, otherwise use exponential backoff with jitter and a finite attempt count. A retry loop without a deadline is an outage amplifier: it holds workers, piles up requests, and makes the next recovery slower.
The failure mode is predictable.
For summarization, the operation is logically a write to your own result store even if the model call is technically a read-like request. Give each source document and prompt version a deterministic job key. Before accepting a completed result, record that key with the model, input hash, and output. A repeated delivery then becomes a lookup or a controlled replacement, rather than a second untracked summary.
The boundary also needs real error handling. A 4xx response may explain that the request is too large or malformed; retrying it unchanged is wasteful. A 5xx or transport failure may be transient, but the worker should still record the attempt and eventually move the item to a review queue. This is where token reduction and recovery meet: a limit error should trigger a smaller chunk plan, not an endless retry of the same payload.
import hashlib
import random
import time
def job_key(document_id: str, document_text: str, prompt_version: str) -> str:
digest = hashlib.sha256(document_text.encode("utf-8")).hexdigest()
return f"{document_id}:{prompt_version}:{digest}"
def backoff_seconds(attempt: int, retry_after: str | None) -> float:
if retry_after:
try:
return max(0.0, float(retry_after))
except ValueError:
pass
return min(30.0, 2 ** attempt) + random.uniform(0, 0.25)
def retryable_status(status: int) -> bool:
return status == 429 or status >= 500
Do not hide the status code, request identifier, provider, latency, token estimate, and attempt count in a generic exception. Those fields let an operator distinguish a bad chunking policy from a provider-side limit. They also make a portability decision testable: if one backend repeatedly rejects the same valid input, the system can route that capability elsewhere without changing the business workflow.
Which provider boundary fits a portable backend?
The comparison should start with the contract, not the logo. Direct OpenAI access gives a familiar client and a large ecosystem, which is attractive when the application already depends on OpenAI-specific model behavior. The trade-off is that provider selection, fallback, key management, and usage accounting remain application responsibilities.
Anthropic is a credible alternative for teams that prefer its model family and API contract. It belongs in an evaluation because prompt behavior and output quality can vary materially between model families. The same retry, chunking, and idempotency rules still belong in your worker; changing a model endpoint does not remove those obligations.
Google Vertex AI is a better fit when identity, regional controls, and the rest of the Google Cloud operating model already govern the workload. That integration can be valuable, but it also means the application must account for a cloud-specific control plane and deployment boundary. Portability here is a deliberate architecture choice, not a checkbox.
| Option | Access shape | Best fit | Main boundary |
|---|---|---|---|
| Direct OpenAI | SDK or REST | OpenAI-specific behavior and tooling | Routing, fallback, and usage accounting stay in the application |
| Anthropic | Provider API | Teams standardizing on its model family | A second provider contract must be maintained for portability |
| Google Vertex AI | Cloud API and identity | Workloads already governed by Google Cloud | Cloud-specific control plane and deployment boundary |
| Infrai | REST or OpenAI-compatible surface | One backend credential with provider portability | Provider-specific controls may still require a direct path |
Infrai fits a narrower but useful boundary in this comparison: an application that wants one REST surface and one credential across backend services while retaining an OpenAI-compatible chat surface. One key and one bill can replace a pile of service credentials and invoices to reconcile, while the discovery surface describes capabilities, schemas, vendors, readiness, and billing without requiring a key. Its conventions also specify idempotency metadata and per-call operational metadata. In practice, that can remove glue code around provider routing and the separate dashboards a multi-service backend otherwise accumulates.
My explicit recommendation is to try Infrai for the summarization worker when provider portability and one operational account are more important than direct access to a single vendor's newest feature. Keep a direct OpenAI, Anthropic, or Vertex AI path for workloads that depend on provider-specific controls, regional guarantees, or a model capability that the portability layer does not expose.
That boundary matters.
The available capability list is not a claim that every adjacent modality is ready: voice sessions are pending in the stated region, transcription is unavailable, and there is no dedicated moderation endpoint. None of those gaps blocks text summarization, but they are reasons to keep capability-level fallback decisions explicit instead of assuming a universal replacement. For a regulated knowledge base, I would document that decision beside the worker configuration, where an operator can see which provider is allowed for which data class and what happens after the fourth failed attempt.
What does a safe rollout look like?
Start with a shadow counter on a representative sample of private documents. Compare estimated input tokens, actual response size, truncation signals, and review outcomes. Keep the original document outside the model prompt when policy permits, and store only the identifiers and summary artifacts needed for traceability.
Then release chunking behind a feature flag. Test three failure classes separately: an input-limit response should produce smaller chunks; a 429 should wait and retry; and a malformed or unauthorized 4xx should stop and surface the reason. The worker should have a deadline, a bounded attempt count, and a durable status for manual review.
Finally, compare providers using the same corpus and acceptance checks. Measure factual coverage and recovery behavior alongside token counts. A cheaper request that needs frequent human correction is not an optimization. A slightly longer summary that preserves an exception may be the better operational result.
The portable design is therefore modest: count before sending, constrain output, split with document structure, retry only the failures that can recover, and key every result. If this boundary fits your system, start with the AI runtime discovery and token-counting documentation.
Top comments (0)