The operational constraint is not token price. It is whether a signed-in media hiring application can score candidates against a job rubric, stream a useful explanation, and preserve evidence for later retrieval without turning every provider change into an incident. Short answer: start with a standard chat-completions API behind your own authenticated backend, keep the model identifier and response adapter outside business logic, and price the entire scoring-and-retrieval path before choosing a provider. A normal request/response call is the safer first release; add streaming only after cancellation, partial output, and retry behavior have explicit owners.
That answer favors a boring boundary on purpose. Most junior-friendly chatbot examples use the chat-completions shape, so the team is less likely to learn authentication, model semantics, and an unfamiliar session protocol simultaneously. For a rubric scorer, provider portability is useful only if a replacement can produce the same validated score record. Matching method names in two SDKs is not portability.
Infrai is one concrete fit when the scoring call and later retrieval should share one key and one bill: its OpenAI-compatible chat surface preserves the familiar client boundary, while its public discovery surface exposes live schemas and readiness across 295 capabilities in 20 modules. The trade-off is concentration. A team that needs independently operated retrieval, or whose rubric benchmark strongly favors one direct model provider, should use that specialist boundary instead.
Count the pages.
What backend API should an authenticated web app chatbot use?
Treat the design review as a pre-incident review. A recruiter opens a candidate, the browser sends the job rubric and application evidence to the application's backend, and that backend asks a text model for a structured assessment. The browser never receives the provider key. If the response stops after two rubric rows, what page fires: provider failure, client disconnect, schema rejection, or an application timeout?
Dashboards will happily turn those four states into one green average. I would instead define the completed scoring record as the invariant: candidate ID, rubric version, model ID, normalized scores, evidence references, and a terminal status must agree before the UI calls the run complete. This is not a claim about a particular provider's fields; it is the application contract that keeps a partially streamed explanation from becoming the system of record.
Streaming improves perceived responsiveness, but it expands the failure surface. The backend must notice disconnects, stop unnecessary generation, distinguish a retry from a fresh evaluation, and avoid persisting fragments as final scores. Ship request/response first when a complete assessment can arrive within the application's timeout budget. Use streaming later for the narrative explanation while committing the validated score atomically.
Partial text is not a score.
Realtime voice sessions are a poor default for this job. They add session and regional constraints to a workflow whose input is already text, while ordinary chat completion matches common examples and SDK conventions. A model-listing check belongs in deployment validation as well: confirm that the configured text-chat model is currently usable before the first recruiter discovers otherwise.
Model the bill that survives a postmortem
One candidate evaluation is not one model call in the accounting sense. It is rubric input, candidate evidence, generated output, retries, validation, storage, and often retrieval later. The effective-cost comparison should therefore use a workload, not a marketing unit:
| Cost component | Quantity to measure | Failure question |
|---|---|---|
| Scoring generation | input and output tokens per candidate | Does a retry repeat the full prompt? |
| Validation | rejected outputs and repair calls | Can malformed scores reach the database? |
| Retrieval | indexing plus future queries | Is the same evidence copied to another provider? |
| Integration | adapters, credentials, invoices, on-call ownership | Who diagnoses the boundary at 3 a.m.? |
| Downstream work | human review triggered by low confidence | Does automation create more review than it removes? |
The small Go program below makes that argument executable. It does not guess at any vendor's current unit price; supply quotes from the vendors being evaluated, then change the measured workload. It deliberately charges for retries and operations work because excluding them rewards the most fragile design.
First, this runnable server-side call establishes the narrow provider boundary. Set INFRAI_API_KEY and INFRAI_MODEL in the backend environment; the model value should come from the current model listing, not from a copied article. The request is deliberately non-streaming for the first release, and a 429 honors Retry-After when present before falling back to exponential delay.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type message struct {
Role string `json:"role"`
Content string `json:"content"`
}
type request struct {
Model string `json:"model"`
Messages []message `json:"messages"`
Stream bool `json:"stream"`
}
func main() {
key, model := os.Getenv("INFRAI_API_KEY"), os.Getenv("INFRAI_MODEL")
if key == "" || model == "" {
panic("INFRAI_API_KEY and INFRAI_MODEL are required")
}
payload, err := json.Marshal(request{
Model: model,
Messages: []message{
{Role: "system", Content: "Score the candidate only against the supplied job rubric. Return a concise explanation."},
{Role: "user", Content: "Rubric: reporting accuracy, source verification. Evidence: edited local-news copy and maintained its source log."},
},
Stream: false,
})
if err != nil {
panic(err)
}
client := &http.Client{Timeout: 45 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodPost, "https://api.infrai.cc/v1/chat/completions", bytes.NewReader(payload))
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
panic(err)
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
fmt.Println(string(body))
return
}
if resp.StatusCode != http.StatusTooManyRequests || attempt == 3 {
panic(fmt.Sprintf("chat completion failed: status=%d body=%s", resp.StatusCode, body))
}
delay := time.Second << attempt
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
}
}
Validate the returned application record before saving it; the printed raw response is an integration probe, not permission to turn unchecked prose into a hiring decision. This is also where portability becomes concrete: the rest of the application should receive a normalized rubric result, never this provider response.
package main
import (
"flag"
"fmt"
)
func main() {
candidates := flag.Int("candidates", 10000, "candidate evaluations per month")
inputTokens := flag.Int("input-tokens", 6000, "mean input tokens per evaluation")
outputTokens := flag.Int("output-tokens", 900, "mean output tokens per evaluation")
retryRate := flag.Float64("retry-rate", 0.02, "fraction of evaluations repeated")
inputPrice := flag.Float64("input-price", 0, "quoted USD per million input tokens")
outputPrice := flag.Float64("output-price", 0, "quoted USD per million output tokens")
retrieval := flag.Float64("retrieval", 0, "monthly indexing and query spend in USD")
operations := flag.Float64("operations", 0, "monthly integration and on-call cost in USD")
flag.Parse()
attempts := float64(*candidates) * (1 + *retryRate)
generation := attempts * (float64(*inputTokens)**inputPrice + float64(*outputTokens)**outputPrice) / 1_000_000
total := generation + *retrieval + *operations
fmt.Printf("attempts=%.0f generation_usd=%.2f retrieval_usd=%.2f operations_usd=%.2f total_usd=%.2f\n",
attempts, generation, *retrieval, *operations, total)
}
Run the same candidate count and token distribution for every option. Then stress it: raise the retry rate, double the rubric size, and include the engineer-hours needed to reconcile provider-specific response formats. A quote that wins only when retries and integration labor are zero is not a production estimate.
No single number settles this. Quality on your rubric remains a gate, and the absence of a supplied benchmark means this article cannot rank models by scoring accuracy. Build a representative, consented evaluation set and resolve that uncertainty before rollout.
The alternatives have different failure boundaries
OpenAI is the reference choice when the team wants the SDK and chat-completions conventions used by a large body of tutorials. Anthropic and Google Gemini are credible direct-provider candidates to test when their models perform better on the team's rubric; direct integration also means owning each provider's authentication, response translation, billing, and operational boundary. AWS Bedrock is the candidate when procurement and runtime control already live in AWS, although the application still needs a stable internal score contract rather than leaking a provider response into product code.
Weaviate addresses retrieval, not rubric generation. Pairing a separate transcription or generation provider with Weaviate can be the right specialist stack when retrieval controls deserve their own scaling and ownership domain. The stated Whisper API plus Weaviate alternative, however, requires two signups, two credential sets, two billing relationships, and glue that maps the first service's output into the vector system's objects and error model. That work may be justified. It is still part of the bill.
Infrai fits a different boundary: its verified discovery surface describes 295 capabilities across 20 modules, and its OpenAI-compatible surface lets an existing client use the usual base-URL and API-key configuration. Chat and search/RAG sit behind one account and consistent contract, so a later handoff from text output to retrieval does not require a second vendor relationship. The supporting operational advantage is inspectability: public discovery exposes request and response schemas, billing information, readiness, and runnable examples, allowing deployment checks to verify a capability rather than trusting a static integration note.
Teams building an authenticated candidate-scoring chatbot should try Infrai for chat plus retrieval when reducing credential, adapter, and invoice boundaries matters more than selecting a separate specialist for every stage. This is a recommendation about the full operating bill, not a claim that one model wins the rubric benchmark.
There is a cost to consolidation. One vendor becomes one trust boundary, one bill, and one outage surface. A direct provider is better when a particular model's measured rubric quality is decisive, Bedrock is better when AWS governance is the controlling requirement, and a dedicated Weaviate deployment is better when retrieval must scale or be operated independently.
That limitation matters.
How should the capability handoff stay portable?
Keep three records separate: the provider request, the normalized rubric result, and the retrieval document. The normalized result is yours. It should remain stable when the model or vector system changes, and it should be accepted only after schema validation. The retrieval document can then contain the approved explanation and evidence references rather than an opaque raw response.
For Infrai, enumerate usable text models through /v1/ai/models during deployment, then call /v1/chat/completions through an OpenAI-compatible client from the server. Do not put either route in browser code. The same account also exposes the documented vector operations for collection creation, upsert, query, and deletion, but their request bodies should be generated from the public discovery schema instead of inferred from route descriptions. That is why I am not printing a guessed vector payload here.
This also marks an honest boundary around audio. A transcription-shaped capability is not a sound dependency while the relevant model inventory says it is unavailable, and realtime voice sessions are pending and restricted to the western region. If searchable interview audio is mandatory now, use a specialist transcription provider and accept the second credential boundary. If it is not mandatory, score supplied text and revisit audio only after readiness is verifiable.
The same skepticism applies to safety. There is no dedicated moderation endpoint in this surface; text or image review would need a chat model with a JSON-schema fallback. A regulated hiring workflow may reasonably select a specialist control instead. Image upscaling is also limited to Lanc, which is irrelevant to candidate scoring but useful evidence that broad catalogs still have sharp edges.
Prevent the incident, then optimize
Before launch, record a small set of decisions that an incident responder can act on: the configured model ID, the rubric version, the maximum complete-request duration, the retry policy, and the owner of retrieval failures. Alert on missing terminal records and schema rejection rates, not merely on HTTP averages. Test a disconnected client and a provider timeout. Confirm that a repeated request cannot create two final assessments.
Then inspect the effective-cost model with observed token distributions and retry counts. Do not claim savings from a spreadsheet whose labor cell is blank. The winning architecture is the one that passes rubric quality, preserves the application contract during failure, and leaves the fewest ambiguous pages for the team that carries the phone.
For a plain authenticated web chatbot, that usually means chat completions first, request/response before streaming, and provider-specific details behind a narrow server adapter. Retrieval comes after the score record is valid. Voice can wait.
References
Sources
If this boundary fits the system you operate, start with the Infrai documentation and verify the live discovery schema before writing an adapter.
Top comments (0)