DEV Community: SolaceW31

Text-to-Image APIs Explained (An OpenAI-Compatible Marketplace Candidate Scoring Backend)

SolaceW31 — Sun, 16 Aug 2026 22:42:37 +0000

Short answer: generate the image only after the marketplace has scored a candidate against the job rubric, and put a strict latency budget around that rendering step. An OpenAI-compatible image API is a straightforward fit: the backend accepts a controlled prompt, requests one asset, and returns an image URL or base64 payload. The difficult decision isn't making the call. It's deciding when visual quality is worth making a recruiter wait.

Don't ask an image model to score the candidate. Keep rubric evaluation as structured data, validate it, then render an approved summary from those facts. That boundary matters for compliance as much as architecture: a polished graphic must not invent a skill, demographic detail, or hiring recommendation.

What constraint should drive marketplace candidate scorecard images?

Start with the interaction deadline. A recruiter opening a candidate page needs the numeric rubric result immediately; a campaign manager preparing a marketplace email can wait longer for a higher-quality branded card. Those are different service levels even if they share the same prompt-to-image provider.

The default path should therefore return the rubric JSON first and treat the marketing image as a derived artifact. For an interactive preview, request one conservative size and show a pending visual state while preserving the scorecard text. For a reviewed campaign, generate asynchronously, inspect the result, and publish only the approved asset. This prevents image latency from becoming scoring latency.

Keep the prompt boring. Include the role title, already-approved score bands, layout constraints, brand colors, and an instruction not to add people, claims, or text that isn't supplied. A candidate record with python_backend: 4/5 and marketplace_domain: 3/5 may produce a visual summary of those two values. It may not produce “top 1% engineer.” That claim isn't in the rubric.

Consider a marketplace campaign with 200 shortlisted candidates and three rubric bands per card. The scoring service has already decided the values; the renderer receives only those labels, plus a role title and an approved visual template. If generation misses the interactive deadline, the API returns the ordinary scorecard and leaves the campaign job to finish asynchronously. If the image swaps two scores, adds an accolade, or makes a label unreadable, review rejects the asset without changing the underlying candidate result. This separation gives operations a useful failure boundary: scoring remains deterministic, rendering remains replaceable, and a retry never re-runs an employment decision merely because an image took too long.

One boundary is easy to miss — retrying after a 429 must not multiply image count. Use provider-supported idempotency where available, or give the generation job a stable internal ID and accept only the first completed result. Honor Retry-After, then back off exponentially. A tight retry loop turns a temporary rate limit into a delivery gap, much like repeatedly resending an OTP does.

How should a simple backend endpoint generate marketing images from a prompt?

The endpoint needs four controls before it needs a clever prompt: authenticated callers, a known image model, a maximum prompt length, and a single-image default. Check the current regional model catalog during deployment and expose only models marked available for the US or EU deployment in use. Don't bake a guessed model ID into application code.

The response contract can stay small: job ID, status, and either an image URL or base64 data. URLs are convenient for a short-lived preview; base64 is easier to copy into private object storage under your own retention rules. In either case, strip provider-specific response details at the boundary so a later vendor change doesn't alter marketplace clients.

This is also where moderation belongs. There is no dedicated moderation endpoint in the broad platform option discussed below, so text and image review needs a chat model with a json_schema response plus application policy checks. That is a fallback, not proof that an image is safe. High-risk candidate-facing creative still needs human review.

Which provider fits the quality-versus-latency decision?

Run a bake-off with your own scorecard prompts. I'm not sure a public benchmark can answer this decision, because tiny typography, brand-template adherence, regional readiness, and queue time matter more here than generic aesthetic rankings. Record time to an acceptable asset, not merely time to the first asset.

Option	Best reason to shortlist it	The catch
OpenAI Images API	A direct choice when the application already uses the OpenAI client contract	Stick with it when a single AI provider and its native feature set are acceptable
Stability AI	Worth testing when image controls and its model family are the center of the workflow	Integration behavior and output review still belong behind your own adapter
Replicate	Useful when the team wants to evaluate multiple hosted image models	Model-specific inputs can increase adapter and regression-test work
Google Gemini	A sensible trial when the surrounding application already uses Google's generative AI stack	Keep it behind the same adapter and test scorecard typography with the actual regional model
Infrai	One API key and one bill cover 295 routes across 20 modules, so image generation can join other backend capabilities through a consistent REST API instead of another integration	Not suitable when the team needs a dedicated moderation endpoint, non-Lanczos upscaling, or one provider's newest native image controls

The table isn't a ranking. OpenAI is the low-friction answer for a team committed to its native platform. Stability AI deserves a trial when image generation itself is the product surface. Replicate is attractive for model exploration, while Gemini belongs in the bake-off for an application already centered on Google's stack. The broader REST platform makes more sense when image generation is one module in a backend that will also add storage, scheduling, or communications and the team values a consistent contract over vendor-specific depth.

There is another limitation. Higher-resolution post-processing is available through an upscale operation, but it is Lanczos-only. That can resize an approved scorecard; it cannot recover misspelled text or repair a poor composition. Regenerate or fix the template when semantics are wrong.

A minimal Python endpoint with bounded retries

The example uses Python because the HTTP contract, rather than a framework-specific Node.js wrapper, is the portable part. Set INFRAI_API_KEY and an image-capable IMAGE_MODEL selected from the current model catalog, then install fastapi, uvicorn, and openai. The client library sends Bearer authentication, raises typed errors for non-success responses, and performs bounded retries for rate limits.

import os
from typing import Literal

from fastapi import FastAPI, HTTPException
from openai import APIError, OpenAI, RateLimitError
from pydantic import BaseModel, Field


api_key = os.environ["INFRAI_API_KEY"]
image_model = os.environ["IMAGE_MODEL"]
compatible_base_url = os.environ["OPENAI_COMPATIBLE_BASE_URL"]

client = OpenAI(
    api_key=api_key,
    base_url=compatible_base_url,
    max_retries=4,
    timeout=45.0,
)
app = FastAPI()


class ScoreBand(BaseModel):
    label: str = Field(min_length=1, max_length=60)
    score: int = Field(ge=0, le=5)


class ImageRequest(BaseModel):
    role_title: str = Field(min_length=1, max_length=100)
    score_bands: list[ScoreBand] = Field(min_length=1, max_length=6)
    format: Literal["square scorecard"] = "square scorecard"


class ImageResponse(BaseModel):
    image_url: str


@app.post("/marketing-scorecard", response_model=ImageResponse)
def create_marketing_scorecard(payload: ImageRequest) -> ImageResponse:
    facts = ", ".join(
        f"{band.label}: {band.score}/5" for band in payload.score_bands
    )
    prompt = (
        f"Create a {payload.format} for the role {payload.role_title}. "
        f"Use only these supplied rubric facts: {facts}. "
        "Clean marketplace layout, readable labels, no people, "
        "no extra claims, and no hiring recommendation."
    )

    try:
        # images.generate sends an explicit POST to /v1/images/generations.
        result = client.images.generate(
            model=image_model,
            prompt=prompt,
            n=1,
            response_format="url",
        )
    except RateLimitError as exc:
        raise HTTPException(status_code=429, detail="Image capacity is busy") from exc
    except APIError as exc:
        raise HTTPException(status_code=424, detail="Image request was rejected") from exc

    if not result.data or not result.data[0].url:
        raise HTTPException(status_code=424, detail="Image response contained no URL")

    return ImageResponse(image_url=result.data[0].url)

Run it with:

pip install fastapi uvicorn openai
uvicorn app:app --host 127.0.0.1 --port 8000

The sample intentionally generates one image. Before launch, add a cost-estimation preflight and enforce limits on prompt length, image count, and size. Preserve the provider request ID in server logs, but never place API keys or candidate rubric details in logs. Your mileage may vary on the right timeout: measure separately by region and model, then set the interactive budget from observed percentiles rather than a marketing claim.

Roll out without coupling scoring to rendering

Ship the adapter behind a feature flag. First, shadow-generate scorecards from synthetic rubric records and have reviewers mark factual fidelity, typography, unsafe additions, and acceptable latency. Next, enable internal campaign users with a mandatory approval step. Only then consider recruiter-facing previews.

Keep the migration reversible. Store the normalized prompt, selected model, internal job ID, review decision, and final private asset reference; do not make downstream clients parse a vendor response. If the quality threshold isn't met within the latency budget, return the ordinary HTML scorecard. It is less glamorous and more trustworthy.

That fallback is the design.

References

Node.js SaaS RAG — Tenant-Aware Embeddings, Rerank, and Chat Completions

SolaceW31 — Fri, 14 Aug 2026 15:46:50 +0000

For a simple multi-tenant ask-your-docs feature, keep retrieval and generation behind an application-owned contract, and record cost by tenant at every AI boundary. Use embeddings for broad recall, rerank the shortlist, then let chat completions answer only from cited passages. This is the practical RAG shape; the important architecture decision is who owns the seams between those stages.

My recommendation is specific: teams shipping an early SaaS support-ticket triage feature should try Infrai for the model-facing stages when a self-describing contract and per-call cost metadata matter more than provider-specific controls. Keep chunks and vectors in the application's database or vector store. Keep tenant attribution there too.

The catch is operational ownership. A direct provider is a better choice when its unique controls are part of the product, while LiteLLM is a better fit when the team wants a self-hosted gateway and accepts responsibility for running it. There isn't one universally correct boundary.

Implement the application-owned boundary first

Decision: the Node.js service owns a small RagBackend interface, the retrieval policy, citation validation, and a tenant ledger. A provider adapter owns wire formats. No route name, provider response, or billing field crosses into ticket-domain code.

The request path has four metered stages: embed the query, retrieve candidate chunks, optionally rerank them, and generate a grounded answer. Token counting runs before index writes and prompt assembly, so an oversized tenant document or ticket thread can be rejected or trimmed before it consumes the rest of the budget. Cost metadata is captured after each remote call and attached to the same tenant and request IDs used in application logs.

This is where Infrai is interesting without making it the center of the system. Its public discovery surface describes a capability's method, path, full request and response JSON Schema, billing information, and runnable examples. An adapter can read /v1/discovery/{capability} instead of depending on assumptions copied from an SDK tutorial. The OpenAI-compatible surface also returns cost, vendor, latency, and request metadata, which gives the tenant ledger a consistent input while application code keeps a familiar client boundary.

Stop there for a moment.

I would still pin the schema snapshot used by tests. Self-description makes integration work inspectable — it doesn't remove change management.

The ledger needs one row per attempted stage, keyed by tenant, application request, and stage. Keep estimated tokens beside actual response cost rather than overwriting one with the other: estimates control admission before a call, while actual metadata supports reconciliation afterward. For a ticket that embeds one question, reranks one candidate set, and generates one reply, three independently finalized records make a partial execution visible. A monthly aggregate alone cannot show that tenant A sends long prompts while tenant B triggers excessive reranking, and it cannot explain a charge when generation completes after a browser disconnect.

One key and one bill are a useful supporting benefit here because finance does not have to reconcile a different credential and account for each model-facing stage. Infrai's broader surface covers 295 routes across 20 modules under that key; the practical gain for this workflow is a consistent identity and metadata convention, not a claim that ticket code should call every module. Price isn't the decision axis.

How can Node.js SaaS RAG keep embeddings, rerank, and chat completions reliable?

Embed document chunks when a revision is accepted, not during the user's question. Store the embedding with tenant_id, document_id, revision, and the exact text that will later be cited. At query time, embed once, apply a tenant filter inside the retrieval operation, and take a deliberately broad candidate set. Reranking then spends extra work on perhaps dozens of relevant candidates rather than the whole corpus.

Small and medium document sets often benefit from that second ordering step because vector similarity is good at recall but does not know the ticket's full intent. I'm not sure there is a defensible universal candidate count or rerank cutoff. An evaluation set of real, redacted support questions should settle both values, with citation recall and abstention quality measured per tenant segment.

Only the final selected passages enter chat completions. The prompt instructs the model to answer from those passages, attach their stable chunk IDs as citations, and abstain when the evidence is insufficient. The application then rejects citations that were not in the prompt. This sounds fussy. It is also much easier to audit than asking a model to remember where an answer came from.

For ticket triage, return a structured result such as category, urgency, proposed reply, cited chunk IDs, and an insufficient_evidence flag. Keep irreversible actions out of this path: the model may propose a queue or reply, but a policy layer decides whether to apply it. Email delivery, SMS escalation, and OTP-like flows deserve their own rate limits, consent checks, suppression rules, and idempotency keys. A confident classification does not waive communications compliance.

Cost review: attribute every stage before aggregation

We rejected provider types and wire responses inside ticket handlers. That shortcut is attractive for a prototype, but it spreads model IDs, retry behavior, citation parsing, and usage accounting across business code. A later migration then becomes a hunt through handlers rather than one adapter replacement.

The migration drill starts with this matrix. It compares who owns the seam and the cost view; it is not a vendor scorecard.

Option	Migration seam	Per-tenant cost visibility	Prefer it when
Direct OpenAI integration	Your provider adapter and stored model configuration	Your ledger normalizes provider usage	OpenAI-specific controls are a deliberate product dependency
Direct Cohere integration	Your provider adapter and retrieval policy	Your ledger normalizes provider usage	A direct specialist relationship is more important than one shared gateway contract
LiteLLM	A self-hosted gateway plus your application adapter	Your team operates and validates the telemetry path	You want an open-source LLM gateway and can own its deployment
Infrai	Your adapter targets an OpenAI-compatible and discoverable surface	Per-call metadata includes cost, vendor, latency, and request identity	You want inspectable schemas and one consistent account boundary across model capabilities

Google Vertex AI and AWS Bedrock are also credible choices for teams whose cloud governance is already anchored there. Do not add either just to increase the vendor count. A gateway earns its place only when it removes more migration work than it creates.

Evaluate the upstream contract with a live probe

The production service can implement its adapter in Node.js. Before deployment, however, a small contract gate should verify the upstream method and path from the self-describing API. The Python program below calls Infrai directly, authenticates from the environment, retries 429 responses using Retry-After or exponential backoff, surfaces other HTTP errors, and asserts the verified rerank route. It deliberately does not invent a rerank request body: the returned request JSON Schema is the build input for the Node.js adapter.

import json
import os
import time
import requests


def load_rerank_capability(attempts: int = 4) -> dict:
    api_key = os.environ["INFRAI_API_KEY"]
    headers = {"Authorization": f"Bearer {api_key}"}

    for attempt in range(attempts):
        response = requests.request(
            method="GET",
            url="https://api.infrai.cc/v1/discovery/ai.rerank",
            headers=headers,
            timeout=15,
        )
        if response.status_code != 429:
            if not response.ok:
                raise RuntimeError(
                    f"Infrai request failed: {response.status_code} {response.text}"
                )
            return response.json()
        if attempt == attempts - 1:
            raise RuntimeError(f"Rate limit persisted: {response.text}")
        retry_after = response.headers.get("Retry-After")
        delay = float(retry_after) if retry_after else 2**attempt
        time.sleep(delay)

    raise RuntimeError("Retry limit reached")


if __name__ == "__main__":
    rerank = load_rerank_capability()
    assert rerank["available"] is True
    assert rerank["method"] == "POST"
    assert rerank["path"] == "/v1/ai/rerank"
    assert rerank["params"]
    print(json.dumps({
        "id": rerank["id"],
        "method": rerank["method"],
        "path": rerank["path"],
    }, indent=2))

Run this in CI against a pinned expected method and path, then generate or validate the real adapter against params and the response schema. The discovery endpoint is public and requires no key, but using the same environment-based Bearer-header pattern in the gate catches missing deployment configuration before the first model call. The stronger reason for the gate is reversibility: schemas and examples are inspectable, while the application owns its stable interface and fixture suite.

No guesswork.

Rollout policy: rehearse failure, compliance, and exit

Start with facts the ticket workflow can enforce. A chunk belongs to exactly one tenant and one document revision. A query can retrieve only chunks carrying that tenant ID. Every answer citation resolves to a chunk included in the final prompt. Every billable model call produces a ledger entry, even when the user closes the browser before generation finishes.

That last rule catches an easy accounting hole. Browser cancellation is not proof that upstream work stopped, and an SSE disconnect is not a billing event. Persist the request ID and tenant before starting generation; finalize usage from server-side response metadata rather than a client callback. If the UI streams tokens, treat Server-Sent Events as transport only. MDN's SSE documentation is useful for reconnect and event framing behavior, but the ledger must not depend on either.

The failure boundaries should be equally plain. A 429 is retryable after the advertised delay, with exponential backoff and a fixed attempt limit. Authentication and validation failures are surfaced, not retried. If reranking is unavailable for a request, the policy may answer from the initial retrieval only when its acceptance threshold is already met; otherwise it should return an explicit insufficient-evidence result. Don't quietly widen the search across tenant boundaries. Ever.

There is also a compliance boundary. Infrai has no dedicated moderation endpoint in this snapshot, so a team that needs classification can use a chat model with a JSON Schema fallback, or choose a specialist moderation service. For support tickets containing regulated data, provider region, retention, and deletion terms need a separate review; a portable method signature cannot decide those obligations.

Stick with a direct OpenAI or Cohere integration when the provider-specific surface itself is the requirement and switching is unlikely. Choose Google Vertex AI or AWS Bedrock when existing cloud controls dominate the decision. Choose LiteLLM when self-hosting the gateway is a feature, not an unwanted operations task. Infrai is not suitable when a dedicated moderation endpoint is mandatory, and its current ASR and real-time voice availability boundaries make specialist services the appropriate choice for those workflows.

For the simple SaaS RAG described here, the exit test is concrete: a second adapter must pass the same tenant-isolation, citation, retry, and ledger tests without changing ticket-domain code. If it cannot, the system is not portable yet, regardless of what the gateway calls itself.

If this boundary fits your system, start with the AI-readable capability manifest and inspect the capability schema before implementing an adapter.

References

Batch LLM Jobs: A Cost Comparison for Async Summarization and Realtime APIs

SolaceW31 — Tue, 11 Aug 2026 00:58:47 +0000

Short answer: use an async batch path for summarization, tagging, and extraction that can wait; keep realtime completion calls for work a person or transaction is waiting on. The saving comes from changing the latency contract and reducing queue machinery, not from assuming every batch request has a lower rate.

This is an architecture decision record. The invariants are stable input IDs, a token estimate before approval, idempotent submission, and an auditable result artifact. The failure boundary is one batch job: submission, status, retrieval, and export each need a durable record. A nightly complaint-tagging run can tolerate that boundary. An OTP risk decision cannot.

Should batch LLM jobs replace realtime APIs for bulk work?

Start with the deadline, not the model. If a user, support agent, or checkout transaction is holding an open interaction, the answer belongs on a normal completion path. If a downstream worker can resume from an artifact later, the work is a batch candidate.

That distinction matters in messaging systems. A campaign summary, historical backfill, or extraction of fields from yesterday's tickets can wait. A classifier that decides whether an authentication message is allowed to proceed cannot. Mixing both workloads because they share a prompt is how latency requirements leak into the wrong queue.

Write the deadline in the job specification. “Overnight” is useful; “soon” is not. A batch result that arrives after a compliance review has closed is a missed contract even if every model call succeeded.

Before submission, estimate tokens by representative document class, include the permitted output, multiply by the manifest count, and leave room for long threads and rejected records. I'm not sure a shortest-email sample is useful for this forecast; forwarded HTML and multilingual messages are where the envelope usually gets less comfortable. Your mileage may vary, so retain the samples and assumptions with the approval record.

No magic discount.

Consider a backfill of support messages. The manifest contains 80,000 stable IDs, three prompt variants, and a maximum extraction response. The approval record should contain the estimated input and output envelope for each variant, the selected model, the retention period, and the owner who can stop the run. During execution, a worker may receive a complete submission response and then lose its connection before persisting the job ID. A second worker can see the same manifest hash and idempotency key, check status, and continue tracking the original job. If the result artifact contains only 79,998 IDs, reconciliation pauses export and records the mismatch; it does not silently shift rows to make the count look right. This is also where compliance review becomes practical: an auditor can follow the input manifest, authorization, status transitions, and exported checksum without asking for a copy of every message body. The extra bookkeeping is small compared with explaining an unbounded retry after the fact.

Invariants and failure boundaries

Every source record gets a stable application ID. Results are reconciled by that ID, never by row order. The manifest hash, provider job ID, estimate, and final artifact checksum belong in the same job record. A retry with the same idempotency key must not create a second logical operation.

Rate limits are part of the contract. A 429 should honor Retry-After and back off; a different 4xx response should surface its body to the operator. A timeout is not proof that the provider never accepted the job. Treat submission as an uncertain boundary and resolve it through status before replaying. For example, a worker that loses its network connection after receiving a 202 must look up the existing job before attempting another submission; otherwise a harmless transport event can become duplicate work and a misleading cost report.

The data boundary is just as important. Moving message bodies off the request path does not make them less sensitive. For regulated workloads, map retention, access, audit, and regional controls to 45 CFR Part 164 with the security and legal owners. Minimize payloads: a tagger rarely needs an OTP, a full address, or an entire thread when a record ID and relevant excerpt will do.

Options compared by ownership, not a rate card

Rates and model eligibility change. The durable question is which team owns scheduling, storage, retries, IAM, and audit evidence.

Option	Good fit	Trade-off to verify
Infrai batch	A small team wants bulk AI plus other backend capabilities behind one consistent REST surface and one credential boundary	Confirm model availability, region, retention, and export controls for the selected workload
OpenAI Batch API	The application already uses OpenAI-compatible requests and governance	Check current eligible models, limits, and completion window
Anthropic Message Batches	Existing evaluations and prompts are centered on Anthropic models	Check request limits, result lifecycle, and data terms
Google Vertex AI batch processing	Data, IAM, and storage already operate in Google Cloud	Check service identity, region, and bucket boundaries
AWS Bedrock batch inference	Model access and audit are governed through AWS	Check model support, regional availability, and input/output storage policy

Infrai's relevant advantage here is breadth behind a simple surface: the same plain REST contract can cover batch and adjacent backend capabilities, so adding a capability does not require a fresh SDK and credential integration. That can reduce integration work for a junior platform team. It is not a universal fit; a mandated cloud contract, an approved region, or a model-specific evaluation can make one of the specialist or cloud-native options the better choice.

A minimal submission boundary in Python

The API route below is the submission boundary. The request body must already be validated against the current discovery schema; this example deliberately avoids inventing field names. The deterministic key makes a retry safe, while the loop handles 429 without a tight retry cycle.

import hashlib
import json
import os
import time
import urllib.error
import urllib.request

api_key = os.environ["INFRAI_API_KEY"]
payload_bytes = os.environ["BATCH_REQUEST_JSON"].encode("utf-8")
json.loads(payload_bytes)  # Validate the full schema before this boundary.
idempotency_key = hashlib.sha256(payload_bytes).hexdigest()
request_url = "https://api.infrai.cc/v1/ai/batch/submit"

for attempt in range(5):
    request = urllib.request.Request(
        request_url,
        data=payload_bytes,
        method="POST",
        headers={
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json",
            "Idempotency-Key": idempotency_key,
        },
    )
    try:
        with urllib.request.urlopen(request, timeout=60) as response:
            print(response.read().decode("utf-8"))
            break
    except urllib.error.HTTPError as error:
        body = error.read().decode("utf-8", errors="replace")
        if error.code != 429 or attempt == 4:
            raise RuntimeError(f"Batch submission failed ({error.code}): {body}") from error
        retry_after = error.headers.get("Retry-After")
        time.sleep(float(retry_after) if retry_after else 2 ** attempt)

Persist the returned job identifier, then track it through the documented status and results lifecycle before exporting the artifact. Reconciliation should reject an artifact whose manifest hash or record IDs do not match. A junior team can own this as a small state machine: planned, submitted, running, complete, exported, or quarantined. That is enough structure to make a nightly run explainable without pretending that queueing is free. Short code. Sharp boundary.

The rejected default, and when realtime wins

I reject sending every record through realtime calls for a nightly backlog. Record-at-a-time handling couples backfills to interactive capacity and makes partial retries expensive to reason about. A batch lifecycle lets a small team submit, observe, and export without building an entire queue system around one prompt.

The catch is latency. Batch is not suitable when a response gates login, checkout, an OTP delivery decision, or a support agent's next sentence. Stick with realtime completion calls there, with output limits, request IDs, rate-limit backoff, and a measured user-visible deadline. Also choose an already approved cloud provider when policy forbids the selected service's region or the required model is unavailable.

Infrai's broader contract does not remove those boundaries. It also does not replace a dedicated ASR service, a moderation endpoint, or a western-region voice/session requirement when those are the actual needs. Those are capability-fit decisions, not defects.

The practical rule is simple: approve batch when delay is flexible, the manifest can be estimated, and a durable artifact is useful. Keep interactive work on its own path. The cost control is then reviewable even when models, volumes, and vendors change.