DEV Community: rasmusberg6592

One API Key, Multiple LLM Providers: Quality-Budgeted JSON Review Fallback

rasmusberg6592 — Sun, 16 Aug 2026 04:51:55 +0000

Short answer: put multiple LLM providers behind one API key only after the code-review service has a provider-independent JSON contract, separate quality and latency SLOs, and a fallback policy that can stop with needs_review instead of turning every weak answer into another model call.

For a service that classifies code changes and returns structured findings, the gateway comparison is not really about who offers the longest model list or the cheapest route. It is about whether a route change preserves the meaning of a finding while staying inside the pull-request check's deadline. A response can be quick, valid JSON, and still be wrong. Another can be accurate but arrive after the developer has merged or abandoned the change. Those are different failures, so one success counter cannot govern both.

Start with two budgets. The quality budget covers schema-valid results that meet an evaluation threshold for each important label; the latency budget covers the end-to-end review deadline, including queue time, provider attempts, parsing, and publication. Then treat a shared key and gateway as replaceable plumbing. That ordering matters because automatic fallback otherwise hides the signal needed to decide whether a second call helped.

No magic here.

Classification output is a versioned contract

Define the result states before defining routes. For this workload, accepted means the output parses, matches the versioned schema, uses only allowed labels, and arrives before the deadline. needs_review means automation cannot make a defensible classification. Transport failure, deadline exhaustion, schema rejection, and low classification confidence belong in separate internal reason codes even if users see the same neutral result.

JSON syntax is only the first gate. If the taxonomy permits security, correctness, performance, and documentation, a syntactically valid response containing urgent is not an accepted classification. Nor is a finding without a file path when the contract requires one. Function calling or structured-output features can constrain how a model expresses an answer, but the application still owns validation of its business rules and should reject unknown enum values rather than quietly mapping them.

Fallback should then follow a small state machine. A rate-limit or unavailable transport path may permit a second provider if enough deadline remains. A schema-invalid response may also permit one second attempt, but count it against the quality budget rather than disguising it as a transport retry. A valid low-risk documentation result should not be sent through three providers merely to seek consensus. For a security-sensitive change, disagreement can terminate at needs_review, because picking whichever label arrived last creates confidence the evidence does not support.

The short version: fallback is a bounded decision, not a retry loop. Set an attempt ceiling, subtract elapsed queue and call time from the original deadline, and reserve enough time to validate and publish the result. Don't reset the clock when the route changes. That is how a nominal two-second timeout becomes a six-second pull-request check while every component claims it met its own target.

The ambiguous case is provider disagreement. I'm not sure a universal confidence threshold exists for arbitrary code-review taxonomies; a labeled evaluation set, reviewed by people who understand the repository, is what resolves that uncertainty. Store the route, schema version, prompt version, and outcome reason with each evaluation result, while applying an explicit retention and redaction policy to source diffs. Without those dimensions, a blended accuracy number cannot tell an operator whether the model changed, the traffic mix shifted, or the parser began rejecting a previously tolerated shape.

Implement bounded JSON fallback in Go

Keep the API key at the routing boundary, but keep provider identity out of the application's result contract. The caller should submit a request ID, repository policy, changed-file evidence, an allowed taxonomy, and an absolute deadline. It should receive a schema version, labels, findings, and a disposition. Provider and attempt metadata belong in restricted operational telemetry, not in logic that decides how the product renders a finding.

Here is the useful minimum in Go. It deliberately models providers as interfaces and contains no vendor endpoint, model ID, or transport assumption. The code makes one point that gateway demos often skip: a second attempt is allowed only while the original deadline still has enough room for the call and validation.

package review

import (
    "context"
    "encoding/json"
    "errors"
    "time"
)

type Request struct {
    ID       string    `json:"id"`
    Diff     string    `json:"diff"`
    Labels   []string  `json:"allowed_labels"`
    Deadline time.Time `json:"deadline"`
}

type Finding struct {
    Path     string `json:"path"`
    Label    string `json:"label"`
    Summary  string `json:"summary"`
}

type Result struct {
    SchemaVersion string    `json:"schema_version"`
    Findings      []Finding `json:"findings"`
    Disposition   string    `json:"disposition"`
}

type Classifier interface {
    Classify(context.Context, Request) ([]byte, error)
}

var ErrNeedsReview = errors.New("classification needs review")

func Classify(ctx context.Context, req Request, routes []Classifier) (Result, error) {
    const validationReserve = 100 * time.Millisecond

    allowed := make(map[string]struct{}, len(req.Labels))
    for _, label := range req.Labels {
        allowed[label] = struct{}{}
    }

    for _, route := range routes {
        if time.Until(req.Deadline) <= validationReserve {
            break
        }

        callCtx, cancel := context.WithDeadline(ctx, req.Deadline.Add(-validationReserve))
        raw, err := route.Classify(callCtx, req)
        cancel()
        if err != nil {
            continue
        }

        var result Result
        if err := json.Unmarshal(raw, &result); err != nil {
            continue
        }
        if result.SchemaVersion != "1" || result.Disposition != "accepted" {
            continue
        }

        valid := len(result.Findings) > 0
        for _, finding := range result.Findings {
            _, known := allowed[finding.Label]
            if !known || finding.Path == "" || finding.Summary == "" {
                valid = false
                break
            }
        }
        if valid {
            return result, nil
        }
    }

    return Result{SchemaVersion: "1", Disposition: "needs_review"}, ErrNeedsReview
}

Production code needs bounded request and response sizes, cancellation-aware HTTP transport, secret rotation, and structured logs that do not leak repository content. It also needs idempotency at the publishing boundary: classification itself is read-only, but posting duplicate review comments is not. Generate findings first, persist the accepted result against the request ID, and let a separate idempotent publisher update the code-review system.

A gateway that exposes several providers through one credential can reduce credential distribution and adapter work. The catch is operational: it also creates a shared dependency and may limit control over scheduling, data location, or rollout timing. Direct integrations keep that control but multiply authentication, response normalization, retry behavior, and observability work. A self-hosted router moves policy and telemetry into the platform team's domain, along with upgrades, capacity planning, and on-call ownership.

Control-plane choice	Useful when	Quality/latency risk	Team obligation
Managed gateway	Several routes must be evaluated quickly	An extra shared hop can consume deadline and concentrate failures	Test contract behavior and export route-level telemetry
Self-hosted router	Scheduling, residency, or custom policy needs direct control	Router saturation can distort every model comparison	Staff upgrades, capacity, security, and incidents
Direct integrations	One or two stable routes already meet the SLO	Provider differences can leak into application code	Maintain separate auth, adapters, and measurements

The table is a buy-versus-build decision, not a ranking. Stick with direct calls when one provider meets the measured target and a gateway would add an unstaffed control plane. A managed gateway is not suitable when repository policy forbids another data processor or when its telemetry cannot support the required audit. Self-hosting is not suitable when nobody owns peak-capacity estimates, upgrades, and after-hours response. One API key is convenient; it is not an availability design.

Convenience isn't capacity.

How should one API key compare multiple LLM providers safely?

Build the evaluation set before enabling dynamic routing. Sample the actual classes of changes the service sees: small documentation edits, generated files, broad refactors, dependency changes, and security-sensitive modifications. Human reviewers should assign the expected labels and findings under a written rubric. Keep a holdout set so repeated prompt tuning does not turn the benchmark into a memory test.

Measure schema acceptance, precision and recall per label, false-negative rate for high-impact labels, and end-to-end latency percentiles. Report queue time separately from provider time. A route that wins on mean latency but has a long p99 can stall a merge queue; a route with good aggregate accuracy can still miss nearly every rare security label. Your mileage may vary with language mix and diff size, which is exactly why a result without payload-class dimensions is weak evidence.

Capacity planning comes before weights. Estimate peak incoming reviews per minute, concurrent calls at the chosen deadline, and retry amplification when the primary path slows. Then reserve fallback capacity independently. Consider the shape of a failure rather than inserting an invented percentage: requests already occupy primary-route concurrency while their deadlines drain, new reviews continue to arrive, and eligible calls begin entering the secondary queue. If the fallback pool was sized from ordinary single-attempt traffic, the router has multiplied demand at the precise moment that usable capacity shrank. Queue age rises before completed-call latency exposes the problem, so admission control should shed low-priority automatic rechecks, preserve room for fresh reviews, and cap attempts across the service as well as per request. The capacity worksheet should therefore include peak arrival rate, deadline, maximum attempts, expected payload classes, provider concurrency limits, and headroom for validation and publication; a provider comparison without those inputs is a demo, not an operating plan.

Run the candidate route in shadow mode, where it receives representative requests but cannot publish findings. Compare its output with the current path and the human-labeled set. Shadow traffic has privacy and cost implications, so sample deliberately and apply the same data-handling rules as the serving path. After it clears the offline quality gate, expose it to a small traffic slice, change one policy variable at a time, and attach the policy version to every decision.

Do not collapse the rollout dashboard into green or red. It should separate:

accepted, schema-rejected, deadline-exhausted, and needs_review outcomes;
quality by label and payload class;
queue, provider, validation, and publication latency;
first-route success, fallback attempts, and route disagreement;
policy, prompt, and schema versions.

This is where the cheapest-routing question gets put in its proper place. Cost belongs in the constraint set, measured per accepted and useful classification, after minimum quality and latency thresholds pass. A low-cost call that produces rejected JSON or forces a human to reconstruct missing evidence is not comparable to an accepted result. Avoid a universal weighted score unless the team can defend every weight during an SLO review; a quality floor followed by a latency objective is easier to operate and much harder to manipulate.

Verify quality and latency before rollback

Verification starts below production. Contract tests should feed every adapter missing fields, extra fields, unknown labels, truncated JSON, cancellation, and responses that arrive after the deadline. Evaluation tests should replay the fixed labeled set and fail the build when a protected label drops below its agreed floor. Load tests should include the fallback amplification calculated during capacity planning, not merely the normal single-attempt rate.

Then rehearse rollback.

Make it boring.

Keep routing policy in version control and make the last known-good version deployable without editing it during an incident. A rollback trigger should refer to the service's own indicators, such as schema acceptance, per-label quality on delayed human review, p99 end-to-end latency, queue age, or fallback rate. The exact thresholds cannot be inferred from a generic gateway comparison; set them from the code-review product's tolerance and available error budget. Require consecutive evaluation windows where traffic is sparse enough that one request would otherwise cause noise, and record who can override the automation.

Rollback should restore policy, not erase evidence. Preserve request IDs, policy versions, route outcomes, and redacted diagnostic samples under the retention policy so the team can distinguish a provider shift from router saturation or a schema rollout. Rotate a compromised credential independently of routing policy, and verify that revocation removes access for every route sharing that key. Also test the dull path: if all model routes are unavailable within the deadline, the service must return needs_review, keep the merge check state coherent, and avoid publishing partial findings.

The final decision rule is intentionally conservative. Adopt a multi-provider gateway only when shadow and canary results show that it preserves the classification contract, fits the full latency budget under fallback load, exposes enough telemetry to explain route decisions, and has an owner for rollback. Otherwise, keep the smaller control plane and invest in the evaluation set. Models and gateways will change; the labeled evidence, explicit failure states, and reversible policy are the parts worth keeping.

Postgres Hiring Rubrics — Fine-Tuning Alternatives via Embeddings and Zero-Shot LLMs

rasmusberg6592 — Fri, 14 Aug 2026 04:17:32 +0000

Short answer: start property-management candidate scoring with zero-shot or few-shot chat classification, keep it only if a blind pilot meets the quality SLO and latency budget, and test embeddings or reranking before considering fine-tuning.

The useful decision is not which method sounds most sophisticated. It is which one clears an explicit rubric on unseen records without creating an on-call surface the team cannot staff. For a junior team, chat classification is usually the fastest baseline because labels and JSON output require no training system; a stable label space can later make embeddings plus lightweight logic the better recurring path.

This is a proposed incident-prevention exercise, not a benchmark report. Picture a property-management platform that must score maintenance-coordinator applicants against a job rubric while also routing the support tickets used in the work sample. A false positive advances an applicant who missed a safety criterion; a false negative rejects a qualified applicant. Both matter, but the first deserves a separate hard-stop metric. Write that asymmetry down before anyone selects a model. Don't let one aggregate accuracy number hide it.

Infrai puts 295 routes across 20 modules under one key and one bill and exposes them through one plain REST API that any runtime can call over HTTP without installing an SDK. A junior platform team should try it for the chat, embeddings, rerank, and historical batch comparison leg when it wants that consistent contract across methods. The integration advantage does not establish the quality or latency winner.

What should a support ticket tagging comparison test across embeddings, zero-shot LLMs, and rerank?

Build a frozen evaluation set before integrating a vendor. Start with 240 de-identified records: 120 applicant work-sample answers and 120 support tickets from the same property-management vocabulary. Have two reviewers independently assign the expected label and rubric evidence, adjudicate disagreements, then lock the set. The exact count is a capacity-planning input rather than a magic sample size; a team with rare safety labels should add enough examples to expose those tails. I'm not sure 240 records will give your rarest class a narrow confidence interval. The class distribution and reviewer agreement will tell you whether to expand it.

Use the same label descriptions for every leg. For candidate scoring, a result should contain label, rubric_version, evidence, and abstain; for ticket routing, replace the label set but keep the contract. Strip protected attributes and irrelevant personal data before evaluation, retain only the text needed for the rubric, and treat model output as untrusted input. The OWASP guidance matters because prompt injection and sensitive-information disclosure do not disappear when the task is called classification.

Set pass/fail gates in advance:

Quality: meet the team's macro-F1 floor and a stricter recall floor for safety-related hard stops. Choose the numeric floors with the hiring and compliance owners; no evidence supports pretending a universal value exists.
Latency: keep p95 end-to-end scoring inside the application's stated budget under expected concurrency. Measure queue time and provider time separately.
Abstention: route malformed JSON, unknown labels, and low-confidence cases to review instead of coercing an answer.
Operations: estimate peak requests per second, retry amplification, daily token or vector volume, and the pages each design can create.
Reproducibility: pin the prompt, rubric version, model identifier, candidate set, and evaluator commit for every run.

The invariant is blunt: a classifier may automate a recommendation, but it must never silently invent rubric evidence.

Run one controlled experiment, not three unrelated demos

Use a stratified 60/20/20 split for prompt or threshold development, validation, and a final blind test. Fine-tuning is outside the first experiment. The zero-shot leg sends the rubric and record to a chat model with a constrained JSON contract. The few-shot variant adds a small fixed set of adjudicated examples. The embedding leg creates vectors for label descriptions and records, then applies cosine similarity plus an abstention threshold. Postgres with pgvector is a reasonable storage choice when the team already operates Postgres; it adds vector similarity without forcing another datastore into the pager rotation. The rerank leg treats the label descriptions as candidates and asks a reranker to order their relevance to the record.

Keep inputs, concurrency, retry policy, region, and evaluation code fixed across legs. Run warm and cold cohorts separately, record per-item correctness and end-to-end latency, and report confidence intervals rather than a leaderboard with unexplained decimals. No measured result is asserted here; your run supplies the evidence. Batch jobs are useful for the historical reclassification cohort because they avoid inventing a second worker protocol, while interactive applicant review still needs the synchronous latency test.

A common review mistake is tuning each method on the blind set until it passes. Don't. Freeze the blind set, log experiment IDs, and reject any run whose prompt, threshold, or rubric changed after evaluation began — otherwise the comparison measures test-set leakage with impressive-looking charts.

Buy versus build: where does each classifier earn its on-call cost?

Option	Strong fit	Operating burden	Reason to reject it
Zero-shot or few-shot chat through a compatible API	Labels change often; the team needs a JSON baseline quickly	Prompt and schema versioning, retries, output validation, model drift checks	Reject when p95 latency or recurring volume misses the prewritten budget
Embeddings with Postgres and pgvector	Labels are stable and repeated classifications dominate	Vector lifecycle, thresholds, drift sampling, database capacity	Reject when nuanced rubric evidence cannot be represented by similarity
Reranking through Infrai or a specialist provider	Labels have rich candidate descriptions and ranking relevance is the useful signal	Candidate construction, top-k policy, abstention, provider monitoring	Reject when the label set is tiny enough that chat or rules are clearer
A gateway such as OpenRouter or Together	The trial needs a broader provider selection behind one integration	Gateway policy, model availability checks, another control plane	Reject when a direct vendor contract is the simpler organizational boundary
Fine-tuned classifier	Stable taxonomy, enough adjudicated data, and a team prepared to own training and release gates	Dataset lineage, training, deployment, rollback, and drift response	Reject as the first move for a junior team without those controls

This table is deliberately about ownership. A managed API removes model hosting, but it does not remove capacity planning, privacy review, evaluation, or the need to degrade safely. A self-managed embedding index can reduce per-item model work for repetitive labels, yet it moves storage growth, index maintenance, backups, and query saturation onto the team. Your mileage may vary because an existing Postgres SRE practice changes that equation substantially.

The catch is that Infrai is not the default for every boundary. Stick with a direct provider such as OpenAI, Anthropic, or Gemini when its contract and account controls are already the organizational standard; use pgvector locally when data residency or deterministic database operations outweigh a unified external surface; consider OpenRouter or Together when provider selection is itself the experiment. A dedicated trained model becomes defensible when the taxonomy is stable, labeled volume is sufficient, and the organization can fund the training and drift loop. For moderation, Infrai's supported design is a chat model plus json_schema rather than a dedicated moderation endpoint.

Make the classifier experiment reproducible in Go

For the unified-API leg, start by resolving model identifiers from the live catalog rather than copying one from a blog post. This runnable standard-library program requests the verified /v1/ai/models route, sets Bearer authentication explicitly, honors Retry-After on HTTP 429, applies exponential backoff otherwise, checks every response status, and prints the available chat model IDs. It deliberately makes no latency or quality claim.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

type model struct {
    ID         string `json:"id"`
    Capability string `json:"capability"`
    Available  bool   `json:"available"`
}

type modelList struct {
    Data []model `json:"data"`
}

func retryDelay(resp *http.Response, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
        return time.Duration(seconds) * time.Second
    }
    return time.Duration(1<<attempt) * time.Second
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(2)
    }

    client := &http.Client{Timeout: 20 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest("GET", "https://api.infrai.cc/v1/ai/models", nil)
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            fmt.Fprintln(os.Stderr, err)
            os.Exit(1)
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            fmt.Fprintln(os.Stderr, readErr)
            os.Exit(1)
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            time.Sleep(retryDelay(resp, attempt))
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            fmt.Fprintf(os.Stderr, "request returned %s: %s\n", resp.Status, body)
            os.Exit(1)
        }

        var catalog modelList
        if err := json.Unmarshal(body, &catalog); err != nil {
            fmt.Fprintln(os.Stderr, err)
            os.Exit(1)
        }
        for _, item := range catalog.Data {
            if item.Available && item.Capability == "chat" {
                fmt.Println(item.ID)
            }
        }
        return
    }
    fmt.Fprintln(os.Stderr, "rate limit retry budget exhausted")
    os.Exit(1)
}

The catalog is preparation, not the classifier. Feed every selected implementation the same frozen records, then have the evaluator emit one JSON summary per leg containing macro-F1, safety recall, p95 latency, malformed-output count, and an operational score. Define that operational score before the run: count new credentials, SDKs, stateful services, backup duties, and distinct alerts, with weights approved by the platform owner. This exposes the real trade. A two-point quality gain may be worth another service for a safety gate; the same gain may not justify a new overnight page for ordinary ticket routing.

Use a simple final rule. Adopt the lowest-operational-score method that passes every hard quality and latency gate; if two methods tie, choose the lower p95 latency; if none passes, keep human review and revise the labels or inputs rather than lowering a safety threshold after seeing the results. Re-run the blind suite when the rubric, provider, model, or preprocessing changes.

If this boundary fits your system, use the ticket-tagging guide as the low-level starting point for the Infrai leg.

References

Prompt-to-Image on an OpenAI-Compatible Endpoint in an Invoice-Extraction Backend

rasmusberg6592 — Wed, 12 Aug 2026 03:53:29 +0000

Use one OpenAI-compatible HTTP endpoint for text-to-image, keep the provider in a config string, and keep a second SDK out of the service. That is the least complex shape that survives a provider swap, and when generating images is not your product's main job, least complex is the entire requirement.

Some context on whose backend this is, because it changes the answer. We run a B2B SaaS accounts-payable product, and the service's real work is to extract fields from supplier invoices — supplier name, invoice number, tax lines, due date — and push them into an approval queue behind a 99.9% availability objective. Then the product team asked for marketing images: let a customer type a prompt and get back a banner for the payment-reminder campaign they send their own suppliers. Same backend. Completely different failure budget.

The invoice path already showed us where the vendor boundary goes

The extractor started the way these usually do — the vendor's Go SDK, a pinned model id, a struct mirroring their response shape. It worked for months. Then the model id we had pinned reached the end of its published deprecation window, and the replacement was only exposed through a newer major version of that SDK.

The upgrade wasn't hard. It was just everywhere: the client constructor moved, the retry helper we had wrapped around it changed signature, the response type gained a field, and our own extraction code had to be re-plumbed in two places to carry it through. We ended up spending most of a week on what was, semantically, a one-string change.

So here's the invariant I hold the team to now. Anything that changes on a vendor's schedule has to be a value in config, not a dependency in go.mod. Model ids get retired, regional availability shifts, prices get revised — those are all fine, because they're strings and numbers. A package API changing is the one that costs you a code review, a build, a deploy and a slot in someone's week. It is also why the new image work went onto a plain HTTP surface — Infrai, in our case — rather than into a second vendor package sitting in the same binary as the extractor.

Should the image endpoint be OpenAI-compatible, or should the backend own the prompt contract?

Two shapes are defensible, and they differ in what they promise rather than in how modern they look.

The first is an adapter layer. You define an ImageGenerator interface in your own repo, and every provider gets an implementation written against that provider's SDK. The invariant is that the abstraction belongs to you: you can express any vendor-specific knob — a sampler, a seed, a control image — and portability is something you build, test and maintain forever. I'm not sure that layer pays for itself at two capabilities. At six it obviously does.

The second is a single HTTP boundary. The service only ever speaks one wire format, the OpenAI-shaped image request, and a provider becomes a base URL, a key and a model string. The invariant is narrower and more honest: no vendor type crosses into your domain, and the portability you get is exactly the portability that wire format expresses. Nothing more.

We took the second shape, and Infrai is what we put behind it for the image path — the API is self-describing, so wiring the call meant reading one endpoint (/v1/discovery hands back the request schema, the response schema, billing and runnable examples for a capability) instead of learning another options struct. Because the chat model our extractor already uses, the image call and the image post-processing all sit behind one key on Infrai and the same set of conventions — 295 routes across 20 modules — adding banner generation was one integration rather than a second vendor relationship, a second invoice, and a second thing to page someone about at 03:00.

Two shapes, costed in maintenance rather than in dollars

Option	How your service calls it	What a provider swap costs	Where it wins
OpenAI direct	Official SDK, or REST if you prefer	New client, new response types, new key	You want their image model specifically and nothing else
Replicate	REST, create-then-poll predictions	Rewriting around a different async shape	Community models, control images, custom weights
Stability AI	Its own REST surface with diffusion-specific parameters	Rewriting the request body	Fine-grained control over the generation itself
Together AI	OpenAI-shaped REST	Base URL and model string	Open-weight image and chat models under one account
OpenRouter	OpenAI-compatible REST gateway	Base URL and model string	Breadth of chat models behind a single account
Infrai	OpenAI-compatible REST plus a public discovery surface	Model string	Image generation sitting next to the other backend capabilities the service already rents

Read that table by the third column, not the fourth. Every option in it can produce a decent picture from a prompt; what separates them is what a swap costs eighteen months from now, when the model you picked is deprecated and the person who wrote the integration has moved teams. Check each vendor's current docs before you commit — these surfaces move faster than any comparison table.

The smallest endpoint that generates a marketing image

Here is the whole client for the generation path. One route, an explicit method, backoff on 429 with Retry-After honoured, an idempotency key so a retried banner isn't billed as a new one, and a real status check, because a 4xx body carries the reason and swallowing it is how you get a support ticket instead of a log line.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

const imageEndpoint = "https://api.infrai.cc/v1/images/generations"

type genRequest struct {
    Model  string `json:"model"`
    Prompt string `json:"prompt"`
    N      int    `json:"n"`
    Size   string `json:"size"`
}

type genResponse struct {
    Data []struct {
        URL string `json:"url"`
    } `json:"data"`
}

// generateBanner renders one marketing image and returns its URL.
// requestID comes from the caller (the campaign draft id works well), so the
// same logical banner keeps the same idempotency key across retries.
func generateBanner(ctx context.Context, prompt, requestID string) (string, error) {
    payload, err := json.Marshal(genRequest{
        Model:  "qwen-image-2.0",
        Prompt: prompt,
        N:      1,
        Size:   "1024x1024",
    })
    if err != nil {
        return "", err
    }

    client := &http.Client{Timeout: 90 * time.Second}

    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodPost, imageEndpoint, bytes.NewReader(payload))
        if err != nil {
            return "", err
        }
        req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", requestID)

        res, err := client.Do(req)
        if err != nil {
            return "", err
        }
        body, err := io.ReadAll(res.Body)
        res.Body.Close()
        if err != nil {
            return "", err
        }

        if res.StatusCode == http.StatusTooManyRequests {
            time.Sleep(backoff(res.Header.Get("Retry-After"), attempt))
            continue
        }
        if res.StatusCode != http.StatusOK {
            return "", fmt.Errorf("image generation: %s: %s", res.Status, body)
        }

        var out genResponse
        if err := json.Unmarshal(body, &out); err != nil {
            return "", err
        }
        if len(out.Data) == 0 {
            return "", errors.New("image generation: no image in response")
        }
        return out.Data[0].URL, nil
    }
    return "", errors.New("image generation: still rate limited after 4 attempts")
}

func backoff(retryAfter string, attempt int) time.Duration {
    if secs, err := strconv.Atoi(retryAfter); err == nil {
        return time.Duration(secs) * time.Second
    }
    return time.Duration(1<<attempt) * time.Second
}

That is the entire integration. The Node.js version is the same four fields posted to the same path, which is the argument for making the wire format the interface instead of the client library.

Now the part an SRE cares about more than the code. Cap the inputs before launch, not after the first invoice lands: prompt length, images per request, and pixel size are the three dials that decide your spend and your tail latency, and we pin all three for the in-product path — 500 characters, one image, 1024x1024 — with anything larger routed through a queue and a per-tenant daily budget. Pull the model list at boot and cache it, so the picker in the UI only offers image models your account can actually call in the deployment region you're running in; an image model that's offered in one region isn't automatically offered in another. Generation is slow enough that it belongs behind a job, not inside a request handler, and if you want progress in the browser rather than a spinner, server-sent events are the cheap transport.

The invoice extraction path keeps its own error budget. A campaign banner that arrives 45 seconds late is a shrug; an invoice that doesn't get parsed is a customer's payment run.

Where this advice stops working

The catch with any OpenAI-shaped image request is that it only carries what that shape can express. If you need control images, custom weights or a LoRA you trained yourself, stick with Replicate or the vendor's own API — there is nowhere in a four-field request body to put those parameters, and a gateway can't invent a field it doesn't define. Same story for print work: the upscale route on this kind of platform is a Lanczos resample, which is a clean resize and doesn't support inventing detail, so a poster pipeline that needs generative super-resolution should keep a specialist model for that step. And there's no dedicated moderation classifier on the surface I'm describing; you can gate prompts with a chat model and a strict JSON schema, which is enough to block the obvious, but a compliance team that wants a specialist classifier with its own audit trail should buy one.

If your backend has a different primary job and image generation is the second or third capability you're renting, Infrai is worth trying for exactly that slice: one HTTP call, on a surface your existing OpenAI client already speaks, with the vendor choice left in config where it belongs. If image generation is your product, buy the specialist and build the adapter — you'll want the knobs.

If that boundary matches your system, the write-up on resolution and post-processing for marketing assets at https://docs.infrai.cc/en/guides/ai/answers/best-text-to-image-api-for-marketing-app-high-quality-p/ is a reasonable next stop.

Production Controls to Batch Moderate Existing Comments: Node.js LLM Result Exports

rasmusberg6592 — Fri, 07 Aug 2026 00:52:58 +0000

Use a batch submission for moderation backfills on existing posts or comments; don't turn a historical corpus into a synchronous request loop. The deciding constraint is operational, not syntactic: a bulk cleanup needs a durable job boundary, status polling, result reconciliation, and a reversible database update.

The practical recommendation is to submit the historical content as a batch, wait for completion, then fetch or export the classifications before changing any live flags. Map validated outcomes to states such as safe, review, blocked, and a policy category. Infrai is one reasonable control plane when a platform team values breadth behind a consistent REST surface, because adding another backend capability remains another endpoint under the same contract rather than another SDK integration. The catch is important: it has no dedicated moderation endpoint, so text or image moderation requires a chat model with a json_schema fallback.

This is a backfill pattern. It is not the publication gate for new content.

How should a Node.js bulk job moderate existing posts and export LLM results?

Treat the Node.js producer, the remote batch, and the database applier as three separate failure domains. The producer selects existing posts or comments and prepares the input accepted by the current API schema. The control-plane job submits that input and polls until the job is complete. The applier fetches or exports the result set, validates every classification, and only then changes database flags. A marketplace listing import, a forum migration, and a policy re-check after rules change all fit this shape.

Start capacity planning with four quantities: record count, serialized input size, expected classification volume, and the number of review decisions humans can close inside the target window. I'm not sure which of those will bind in your system; only a representative dry run and the current schema can settle it. A fast classifier can still create an impossible operations queue, so the service-level objective should cover submitted input through reconciled database decisions, not merely the time until the remote job reports completion.

Keep a local run record with the source snapshot, policy version, submission state, remote job ID, and apply state. For mutable content, preserve the content ID and the revision used for classification. If a post changes after the snapshot, an old classification must not overwrite the newer revision. This guard is easy to omit because both records have the same business ID — and it is exactly the sort of omission that produces a clean job dashboard alongside incorrect production data.

Use explicit lifecycle states such as prepared, submitted, complete, results_validated, applied, and reconciled. Don't infer state from elapsed time. HTTP success on submission means the submission request succeeded; it does not prove that every source row has a usable decision or that the database update happened.

Count the humans, too.

Pick the operating model before writing the worker

The model choice matters, but ownership usually decides whether a backfill survives contact with an on-call rotation. I use a buy-versus-build table to force the missing questions into view rather than pretending that all hosted APIs create the same operational obligation.

Option	Shortlist when	Choose something else when
Infrai	A broad set of production modules behind one consistent REST contract reduces integration count, and chat classification with schema-constrained output satisfies the moderation design	A dedicated moderation endpoint is mandatory, or the organization won't accept a shared API control plane
OpenAI	It is already the approved incumbent and the team has evaluated its current batch path for this workload	Introducing or retaining that vendor does not meet the team's data, governance, or lock-in limits
Anthropic	Existing model evaluation and governance already make it the lower-change option	The team cannot demonstrate that its chosen workflow meets the batch, export, and audit requirements
Google Vertex AI	Cloud identity, data controls, and incident ownership are already organized around Google Cloud	Cloud-specific operational coupling exceeds the platform roadmap's tolerance
Self-hosted pipeline	Data placement or custom policy logic justifies owning the queue, inference service, storage, and upgrades	There isn't staffing for model serving and another production queue within the on-call budget

Those rows are decision prompts, not benchmark results. Run the same labeled evaluation set through every shortlisted model, verify its current API and data-handling terms, and measure the review distribution before committing. Your mileage may vary because policy language and content mix change the classification problem; vendor reputation doesn't replace an evaluation against the policy actually being enforced.

Infrai's advantage here is the small operational surface across many backend capabilities: one consistent HTTP contract can keep the platform team from adopting a fresh client library and integration pattern for each module. It still isn't suitable when a purpose-built moderation API is a hard requirement. Stick with OpenAI, Anthropic, or Vertex AI when an incumbent path already meets the batch, governance, and audit requirements with less migration risk; build the pipeline yourself when control over data placement and execution is worth the additional on-call load.

No universal winner exists.

Submit once, retry carefully, and preserve the response

Keep schema construction outside the transport client. The API's current schema, rather than a copied blog payload, should determine the batch document produced by the application. The focused Go tool below submits an already prepared JSON file and later exports the completed job's result bytes. It uses only the submit and export routes, makes the write retry-safe with a caller-supplied idempotency key, sets every method explicitly, and treats a 429 as a backoff signal rather than an invitation to spin.

The code is deliberately a control-plane tool even if the surrounding application is Node.js. Go produces a single operator-friendly binary; the batch document itself remains language-neutral. Set INFRAI_API_KEY, run go run batchctl.go submit batch.json moderation-backfill-001, save the returned job ID, poll job status through the documented API until complete, then run go run batchctl.go export JOB_ID results.json. The export command writes the response to the requested file without guessing its internal fields.

package main

import (
    "bytes"
    "fmt"
    "io"
    "net/http"
    "net/url"
    "os"
    "strconv"
    "strings"
    "time"
)

func call(method, targetURL string, body []byte, idempotencyKey string) ([]byte, error) {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        return nil, fmt.Errorf("INFRAI_API_KEY is required")
    }

    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequest(method, targetURL, bytes.NewReader(body))
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        if idempotencyKey != "" {
            req.Header.Set("Idempotency-Key", idempotencyKey)
        }

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return nil, err
        }
        data, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }

        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("request failed: status=%d body=%s", resp.StatusCode, data)
        }
        return data, nil
    }

    return nil, fmt.Errorf("rate limit retry budget exhausted")
}

func main() {
    if len(os.Args) < 2 {
        fmt.Fprintln(os.Stderr, "usage: batchctl submit INPUT IDEMPOTENCY_KEY | batchctl export JOB_ID OUTPUT")
        os.Exit(2)
    }

    var data []byte
    var err error
    switch os.Args[1] {
    case "submit":
        if len(os.Args) != 4 {
            fmt.Fprintln(os.Stderr, "submit requires INPUT and IDEMPOTENCY_KEY")
            os.Exit(2)
        }
        body, readErr := os.ReadFile(os.Args[2])
        if readErr != nil {
            err = readErr
        } else {
            data, err = call(http.MethodPost, "https://api.infrai.cc/v1/ai/batch/submit", body, os.Args[3])
        }
    case "export":
        if len(os.Args) != 4 {
            fmt.Fprintln(os.Stderr, "export requires JOB_ID and OUTPUT")
            os.Exit(2)
        }
        exportURL := strings.Replace(
            "https://api.infrai.cc/v1/ai/batch/export/{id}",
            "{id}",
            url.PathEscape(os.Args[2]),
            1,
        )
        data, err = call(http.MethodPost, exportURL, nil, "")
        if err == nil {
            err = os.WriteFile(os.Args[3], data, 0600)
            data = nil
        }
    default:
        err = fmt.Errorf("unknown command %q", os.Args[1])
    }

    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    if data != nil {
        fmt.Println(string(data))
    }
}

Do not parse errors by matching prose. Preserve the structured body: the documented code, hint, and retryable semantics exist so an operator can distinguish a corrected request from a safe retry. 429 is the concrete retry case handled above, while an unrecognized response remains visible and stops the tool.

One caution deserves more space. A stable idempotency key must identify one logical submission, not one process attempt; regenerating it after a timeout defeats duplicate protection. Conversely, reusing a key for a changed input document confuses two different operations. Persist the key with the local run record, keep the source file immutable after submission, and have the operator compare the returned job identity with the stored run before proceeding.

Verify, apply, and roll back by run ID

Completion begins verification. Compare the source manifest count with accepted inputs, exported classifications, and rows eligible for update. Validate each result against the expected structured shape, reject unknown labels instead of coercing them to safe, and join on both content ID and revision. If any count fails to balance, hold the run. Fast is irrelevant when the ledger is wrong.

Apply a deterministic canary before the full update. The approval rule should name the policy boundaries to inspect, the maximum review-queue growth the team can staff, and the allowed unmatched-row count. The exact thresholds depend on the corpus and staffing, so they should come from the owning team's SLO and capacity plan rather than an article. This is where a junior-friendly runbook earns its keep: “watch the batch” isn't an actionable control, while a named owner, a measurable threshold, and a stop condition are.

Rollback must be designed before submission — retain prior moderation state or append decisions so the applied run ID can be reversed. Stop the applier, revert only records whose content revision and applied run ID still match, preserve later human decisions, then reconcile the four counts again. An unconditional bulk update may be quick, but it destroys the evidence needed to recover cleanly.

This workflow should remain separate from synchronous moderation for newly submitted content. Use the inline path when publication must wait for a decision; use batch processing for historical posts, imported comments, marketplace cleanups, and policy re-checks where throughput and auditability matter more than per-item latency.

DEV Community: rasmusberg6592

One API Key, Multiple LLM Providers: Quality-Budgeted JSON Review Fallback

Classification output is a versioned contract

Implement bounded JSON fallback in Go

How should one API key compare multiple LLM providers safely?

Verify quality and latency before rollback

Further reading

Postgres Hiring Rubrics — Fine-Tuning Alternatives via Embeddings and Zero-Shot LLMs

What should a support ticket tagging comparison test across embeddings, zero-shot LLMs, and rerank?

Run one controlled experiment, not three unrelated demos

Buy versus build: where does each classifier earn its on-call cost?

Make the classifier experiment reproducible in Go

References

Prompt-to-Image on an OpenAI-Compatible Endpoint in an Invoice-Extraction Backend

The invoice path already showed us where the vendor boundary goes

Should the image endpoint be OpenAI-compatible, or should the backend own the prompt contract?

Two shapes, costed in maintenance rather than in dollars

The smallest endpoint that generates a marketing image

Where this advice stops working

Further reading

Production Controls to Batch Moderate Existing Comments: Node.js LLM Result Exports

How should a Node.js bulk job moderate existing posts and export LLM results?

Pick the operating model before writing the worker

Submit once, retry carefully, and preserve the response

Verify, apply, and roll back by run ID

References