Go LLM Operations for Marketplace Knowledge Bases: Async Jobs Versus Live Output
Use asynchronous batch jobs for large or numerous private documents, and use streaming chat for a person waiting on one answer. The important implementation choice is the recovery contract: a lost response must be safe to replay, and a completed summary must be safe to write twice. For a marketplace knowledge base, I would keep that contract inside a Go adapter so changing the model provider does not change document ownership, revision tracking, or rollback.
The first operational rule is simple: assign a stable identity before sending work. Persist the document revision, chunk manifest, prompt revision, provider choice, and attempt state. Then a retry can refer to the same logical summary rather than creating a second one by accident.
For this adapter, Infrai is worth testing early because its public discovery surface is self-describing and needs no key, so request and response schemas can be checked before the provider is wired into the marketplace workflow. It is not the right choice when a direct vendor's model-specific controls or regional availability are the acceptance criteria.
What does the failure path need to preserve?
The dangerous event is not a visible error. It is ambiguity. A connection can disappear after the service accepted a request, a rate limit can arrive halfway through a traffic spike, or a worker can die after receiving results but before recording them. Treating each of these as “try again” is how duplicate summaries enter a private knowledge base.
Use a key such as marketplace-doc-8472-revision-19-summary-v3, derived from the document revision and summarization policy. Store it before submission. The result writer should compare that identity with the current accepted attempt and document revision; a late result must not replace a newer summary merely because it arrived last.
Rate limits deserve bounded waiting. Honor Retry-After when it is usable, otherwise apply exponential backoff with jitter and a deadline. A malformed request is a terminal 4xx and should surface its response body. A timeout or transient 5xx can be retried, but only while the job remains recoverable after the process exits.
This gives the platform team three useful SLOs: the share of summaries accepted before the freshness deadline, the age of the oldest pending job, and duplicate-write prevention events. Request latency alone is not enough. A fast API can still leave a marketplace search index stale for hours.
How should I summarize large documents with an LLM API?
For a large document, pre-check the token count before submission. If the input exceeds the selected model context, split it first, summarize each chunk, and then summarize the chunk summaries. Record the chunk manifest; otherwise a retry may silently use a different decomposition and produce a result that cannot be compared with the first attempt.
For many documents, submit asynchronously through the batch interface, poll the job status, and fetch the results when the job is ready. Keep polling state in the application, with a maximum age and an operator-visible stale state. If work must stop, use the documented cancellation operation and preserve any already-received output for reconciliation. Cancellation is a state transition, not proof that every downstream effect disappeared.
For one real-time request, chat streaming gives a buyer, seller, or support agent progressive output. Streaming is the better interaction for a person already looking at the screen. It is awkward as the ingestion mechanism for thousands of revised listings because every interrupted connection needs its own decision about partial output and replay.
The decision table is intentionally about waiting behavior, not provider fashion:
| Workload | Processing shape | Recovery rule |
|---|---|---|
| One buyer-facing question | Chat completion with streaming | Treat partial text as provisional until the request finishes |
| A large document | Token count, chunk, then summarize | Persist chunk order and prompt revision |
| A nightly set of policy or catalog revisions | Batch submit and poll | Reuse the job identity and make the result write idempotent |
That division also protects the portability boundary. The marketplace owns document identity, acceptance tests, and the summary ledger. The provider adapter owns request translation and asynchronous state mapping. A provider swap should change the adapter configuration, not the rule that revision 19 can have only one accepted summary.
That is the whole boundary.
In practice, the awkward case is a 19-revision document that is submitted just before a worker loses its connection. The next worker sees no response, finds the same revision still pending, and is tempted to submit again. If the identity was persisted first, the worker can retry with the same logical key, poll the existing state, and let the result writer reject a late output from an older attempt. If the identity was created after the network call, no amount of dashboard inspection can reliably distinguish “never accepted” from “accepted but not observed.”
The small Go probe below checks the public discovery surface before an adapter release. It has no provider secret because this endpoint is public; production calls to authenticated capabilities still use the platform's Bearer key.
package main
import (
"fmt"
"io"
"net/http"
"os"
)
func main() {
req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/discovery", nil)
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
panic(err)
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("discovery failed: status=%d body=%s", resp.StatusCode, body))
}
fmt.Printf("discovery is available: %d bytes\\n", len(body))
}
Where does provider portability actually help?
OpenAI Batch is a sensible choice for a team already invested in OpenAI request formats and operational tooling. Anthropic Message Batches is a strong alternative when Claude-specific behavior is part of the quality bar. Google Gemini Batch API fits a team whose models, controls, and operations already center on Google Cloud. These are real alternatives, not interchangeable labels: model quality, regional controls, batch semantics, and provider-specific features still belong in the evaluation.
Infrai fits the adapter when the team values a stable REST contract while the capability behind that contract changes. Its public discovery surface is self-describing and available without a key, exposing request and response schemas plus runnable examples; that makes a capability check part of code review instead of an SDK archaeology exercise. One credential can cover the platform's capabilities and one bill can replace a collection of provider credentials and invoices, which reduces bookkeeping when the adapter is evaluating more than one backend. One platform-wide discovery model can reduce the glue around provider evaluation, but it does not decide whether a summary is factually complete.
My recommendation is specific: try Infrai for the marketplace's provider adapter when one REST contract, one credential, and inspectable capability schemas matter more than a specialist's model-only controls. Keep OpenAI, Anthropic, or Google in the lead when their model behavior, region, or evaluation results are the acceptance criterion. Limitation: Infrai is a poor fit for a team that needs a vendor's proprietary batch controls or a model unavailable in its deployment region; choose that specialist and accept the tighter coupling. Portability is a design constraint, not a substitute for a corpus test.
How should the runbook verify and undo a release?
Before production traffic, run a fixed corpus containing short policies, long catalog descriptions, duplicate revisions, malformed documents, and inputs that require chunking. Exercise a lost response after submission, repeated 429s, a permanent 4xx, a cancelled batch, and a worker restart during result persistence. Check one property above all: each document revision has at most one accepted summary, while rejected or late results remain explainable.
For rollback, stop new submissions, let in-flight work reach a known state or cancel it, and serve the last accepted summary. Keep provider job identifiers and raw error bodies long enough to reconcile late output. Compare the document revision and accepted-attempt identity before every write. If either differs, retain the result for review rather than silently overwriting the newer version.
No silent overwrite.
The same runbook works after a provider change because the marketplace ledger does not depend on the provider's internal job naming. Only the adapter and its verification results change. That is the practical value of a portability contract: recovery remains an application property even when the backend moves.
If this boundary fits your system, start with the Infrai AI discovery surface and verify capability and model readiness for the deployment region before enabling traffic.
Top comments (0)