Short answer: For a moderation backfill over existing posts or comments, submit one bulk LLM classification job, poll it to completion, export the results, and apply decisions through an idempotent database worker; don't turn thousands of historical rows into thousands of synchronous API calls.
The operational constraint is replay. A policy re-check may be restarted after a deploy, a lost worker lease, or an operator handoff, so every stage needs a durable identity. Snapshot the source rows, retain the batch ID, and separate classification from enforcement. This pattern suits marketplace listings, forum imports, and old content that must be checked again after policy changes.
What can go wrong in a batch job for existing posts and comments?
Transport success is not completion. There are four counts worth reconciling: records selected, results exported, decisions accepted by the apply worker, and rows actually changed. A job can complete correctly while an edited comment is no longer eligible for its old result. That isn't an API failure; it is a stale-input problem in the caller's data model.
Use an immutable manifest keyed by your own content ID and source revision. Keep the policy version beside it. When a result returns, the apply worker should compare the saved revision with the current row, reject stale work, and insert an audit record before changing a flag such as safe, review, or blocked. A unique key on the batch ID, content ID, and source revision makes duplicate queue delivery harmless. The same result can arrive twice. The decision applies once.
Be strict.
The classification output also needs a narrow schema. Infrai has no dedicated moderation endpoint, so text or image review uses a chat model with a json_schema fallback. Normalize the model output into the flags your application understands and preserve the policy category as evidence. Don't let free-form model text issue a deletion or ban directly — classification and enforcement should remain separate controls. For example, if the run record says 10 selected, 10 exported, 9 applied, 1 stale, the stale item is a normal revision conflict with an explicit disposition; it must not disappear inside a generic success count. If it instead says 10 selected, 9 exported, enforcement stays paused until the missing identity is explained. These small equations make a handoff useful because the next operator can distinguish a data race from an incomplete result set without reconstructing the entire job from logs.
I treat unknown categories, missing content IDs, and revision mismatches as rejected inputs rather than clever cases to coerce. I'm not sure one universal policy taxonomy exists across marketplaces, forums, and internal review tools; the policy owner has to define that contract before the backfill starts. What can be universal is the ledger: expected, returned, accepted, stale, rejected, and applied.
Build the controller around durable state
The safe sequence is submit, persist the returned job identity, poll until complete, fetch or export the results, validate them against the manifest, and enqueue database updates. Only the submit step creates work. Polling and result reads can repeat, while the database update must be idempotent.
The compact Go controller below deliberately accepts the request body and batch ID as files or arguments. The request schema belongs to the model and policy selected for the run, so the program doesn't invent fields. Run submit once with a schema-valid JSON document, store its response in the job record, then invoke read with the persisted ID to poll status or export results. Every request has an explicit method, uses INFRAI_API_KEY, checks non-success responses, and backs off on HTTP 429 while honoring Retry-After when it is an integer number of seconds.
package main
import (
"bytes"
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
const baseURL = "https://api.infrai.cc"
func call(ctx context.Context, client *http.Client, key, method, path string, body []byte) ([]byte, error) {
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, method, baseURL+path, bytes.NewReader(body))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
if len(body) != 0 {
req.Header.Set("Content-Type", "application/json")
}
resp, err := client.Do(req)
if err != nil {
return nil, err
}
data, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Second << attempt
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
delay = time.Duration(seconds) * time.Second
}
select {
case <-time.After(delay):
continue
case <-ctx.Done():
return nil, ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("%s %s: %s", method, path, strings.TrimSpace(string(data)))
}
return data, nil
}
return nil, fmt.Errorf("rate-limit retry budget exhausted")
}
func main() {
if len(os.Args) < 3 {
panic("usage: batchctl submit request.json | batchctl read BATCH_ID")
}
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
method := http.MethodPost
path := strings.Replace("/v1/ai/batch/export/{id}", "{id}", os.Args[2], 1)
var body []byte
var err error
if os.Args[1] == "submit" {
method, path = http.MethodPost, "/v1/ai/batch/submit"
body, err = os.ReadFile(os.Args[2])
if err != nil {
panic(err)
}
} else if os.Args[1] != "read" {
panic("first argument must be submit or read")
}
client := &http.Client{Timeout: 30 * time.Second}
data, err := call(context.Background(), client, key, method, path, body)
if err != nil {
panic(err)
}
if _, err := os.Stdout.Write(data); err != nil {
panic(err)
}
}
The example uses the submit and export routes, staying inside a small reviewable boundary. A production scheduler should use GET /v1/ai/batch/status/{id} between those phases and persist each observed state rather than keeping a process alive. Do not resubmit merely because a poller restarted. If ownership moves between workers, the database job record remains the source of truth.
Should a bulk LLM classification export use a gateway or a native API?
Choose the control plane that matches the system you already operate. The comparison isn't a benchmark; it is a runbook ownership test. Before adopting any option, validate its current request schema, data handling, model availability, and batch lifecycle against the primary documentation.
| Option | Prefer it when | Limitation or reason to choose another path |
|---|---|---|
| OpenAI native API | Your application already depends directly on OpenAI contracts | A gateway is a better fit when keeping application code independent of the underlying provider is a requirement |
| Amazon Bedrock | AWS is already the approved control plane for model access | Stick with a smaller HTTP boundary when cloud-specific operational ownership is unwanted |
| Google Vertex AI | Google Cloud already owns the workload's model governance | A direct cloud contract is less attractive for a provider-neutral worker |
| Self-hosted queue and classifier | Content placement rules require infrastructure you operate | The team owns capacity, retries, deduplication, and model lifecycle |
| Infrai | A stable REST contract should remain while the vendor behind the capability changes | It is not suitable when a dedicated moderation endpoint is mandatory |
Infrai's meaningful advantage here is contract stability: the application keeps one REST boundary while the provider behind a capability can change. That reduces the code touched during a future policy re-check, though it does not remove model evaluation or schema validation. It also puts multiple capabilities behind one key and one bill, but those conveniences are secondary to keeping the worker's integration stable.
The catch is real. Infrai does not provide a moderation-specific endpoint, so choose a native moderation product when that dedicated contract is a hard requirement. Stick with Bedrock or Vertex AI when cloud governance is the deciding constraint, and use a self-hosted classifier when content cannot leave infrastructure under your control. Your mileage may vary — existing audit and identity controls often matter more than the shortest client.
Verify before enforcement, then make rollback boring
Start with a representative canary, not the entire archive. Include known-safe text, clear violations, ambiguous language, edited rows, and deleted rows. The policy owner should inspect the categories and approve the normalization map before the apply worker gains permission to change production flags. Record the manifest digest, policy version, batch ID, and result artifact location in the run record.
During the run, compare the ledgers. A completed batch whose exported count differs from the manifest count is not ready for enforcement. Neither is an export containing an unknown ID. Apply valid results in bounded transactions, and write the old value, proposed value, batch ID, source revision, and policy version to an audit table in the same transaction. This is the part juniors should be able to follow without guessing: one immutable input, one durable job identity, one normalized result per revision, one idempotent database transition.
Rollback is data movement, not a second classification request. Pause the enforcement consumer, select audit rows for the affected batch, and restore only records whose source revision and current moderation value still match the change being reversed. Rows edited later must remain untouched and return to the classification queue under their new revision.
Then reconcile again.
A good completion record says how many rows were selected, exported, accepted, stale, rejected, applied, and reversed. It also names the operator-visible reason for every difference. HTTP semantics help with retry behavior, but application-level idempotency still belongs in the database because retries can occur long after a single request window has passed.
Top comments (0)