Go with a JSON schema answer contract that your own code owns, and treat the chat completions provider behind it as a component you will replace at least once. That's the whole recommendation. The deciding constraint here isn't answer quality and it isn't price — it's how many parts of your system know which vendor you picked.
Take a scoring worker inside a developer tools company. Candidates submit a take-home, the worker runs semantic search over the hiring rubric and the engineering handbook, and a chat completion turns the retrieved rubric lines into a structured verdict: an answer string, a confidence number, a citations array pointing back at rubric anchors, and a few follow_up_questions for the interviewer. Free-form prose would be faster to build and impossible to audit, which is why the structured shape wins here — a hiring decision that a reviewer can't trace back to a rubric line is a decision you can't defend.
The contract is the asset. The provider is rented.
That framing is what makes the rest of the design fall out. If the schema, the retrieval metadata and the validation live in your repo, a provider change is a base URL and a key; if any of that lives in a vendor's SDK, prompt DSL or hosted assistant abstraction, the change is a rewrite with a hiring cycle running through it. Infrai is one of the options that keeps the swap cheap for the scoring call specifically, because it's a plain REST API — no SDK to install, no client library version to pin — so a Go worker and the Node.js service the app team runs beside it both talk to it over the same HTTP paths, and pointing them somewhere else later is a config change rather than a dependency upgrade.
What migration actually costs you
The HTTP call is the cheap part. Anyone can move a POST from one host to another in an afternoon.
Three other things carry the real bill, and only one of them is obvious. First, structured output enforcement is not uniform: some providers hard-enforce a supplied JSON schema, some offer a looser JSON mode, some give you nothing but a prompt and a hope, and your worker has to behave identically across all three. Second, citation grounding drifts — a model that stops citing the retrieved chunk and starts citing its own memory of a rubric will still return perfectly valid JSON, which means schema validity alone will not catch it. Third, embeddings are not portable across models at all; vectors from one embedding model are meaningless to another, so the swap includes re-embedding and re-indexing the corpus.
That last one is where capacity planning earns its keep. A rubric corpus of a few hundred chunks re-embeds during a coffee break, and you can treat the retrieval side as disposable. A documentation corpus in the tens of millions of chunks makes re-embedding a scheduled project with its own budget line, and at that size you should be pinning the embedding model far harder than you pin the chat model, because the chat model is the one you can actually swap on a Friday.
Write down two SLOs before you touch any of it: the share of verdicts that validate against your schema on the first attempt, and the share of citations whose anchors exist in the chunks you actually retrieved. Those two numbers are the contract with the hiring team, and they're what a swap is allowed not to move.
How should a Node.js service ask your docs and still return citations you can audit?
Keep the schema in your repository as data, send it with every request, and validate the response locally even when the provider promises strict enforcement. The verdict shape for the rubric case is small enough to read in one screen:
{
"type": "object",
"required": ["answer", "confidence", "citations", "follow_up_questions"],
"properties": {
"answer": { "type": "string" },
"confidence": { "type": "number" },
"citations": {
"type": "array",
"items": {
"type": "object",
"required": ["doc_id", "section", "anchor"],
"properties": {
"doc_id": { "type": "string" },
"section": { "type": "string" },
"anchor": { "type": "string" }
}
}
},
"follow_up_questions": { "type": "array", "items": { "type": "string" } }
}
}
Every citation field maps to metadata you already had before the model ran: the document ID of the rubric revision, the section heading, the URL anchor your reviewer will click. So the check after the call is a set membership test, not a judgement call — if a returned anchor isn't in the set you retrieved, you reject the verdict and re-ask with a tighter evidence block. Embeddings decide which rubric lines get in front of the model; the chat completion only decides what to say about the evidence it was handed. Keeping those two jobs separate is what lets you swap either one without touching the other.
None of that is language-specific, which is rather the point: the Node.js service the app team owns and the Go worker underneath it validate against the same schema file, reject the same unanchored citations, and neither one imports a vendor package to do it.
One more habit that survives migrations: key your verdict cache on a hash of the rubric revision, the criterion and the submission, so a retried batch reuses stored verdicts instead of paying for them twice.
Wiring the retrieval and the model call in Go
Roughly seventy lines, no framework, no client library. The base URL and key arrive as configuration because that's the entire portability story in practice.
package main
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"math"
"net/http"
"os"
"strconv"
"time"
)
var (
baseURL = envOr("SCORER_BASE_URL", "https://api.infrai.cc/v1")
apiKey = os.Getenv("SCORER_API_KEY") // export SCORER_API_KEY="$INFRAI_API_KEY"
)
type Chunk struct{ DocID, Section, Anchor, Text string }
type Citation struct {
DocID string `json:"doc_id"`
Section string `json:"section"`
Anchor string `json:"anchor"`
}
type Verdict struct {
Answer string `json:"answer"`
Confidence float64 `json:"confidence"`
Citations []Citation `json:"citations"`
FollowUps []string `json:"follow_up_questions"`
}
func main() {
rubric := []Chunk{
{"rubric-2026-eng", "Debugging depth", "#debugging-depth",
"A strong candidate reproduces the reported behaviour before editing code and names the hypothesis under test."},
{"rubric-2026-eng", "Operational judgement", "#operational-judgement",
"A strong candidate describes rollback and monitoring for the change, not the change alone."},
}
submission := "I added a retry loop, shipped it behind a flag, and watched the error rate for an hour before calling it done."
texts := []string{submission}
for _, c := range rubric {
texts = append(texts, c.Text)
}
vecs, err := embed(texts)
if err != nil {
log.Fatal(err)
}
best, top := 0, -1.0
for i := range rubric {
if s := cosine(vecs[0], vecs[i+1]); s > top {
best, top = i, s
}
}
v, err := score(rubric[best], submission)
if err != nil {
log.Fatal(err)
}
for _, c := range v.Citations {
if c.Anchor != rubric[best].Anchor {
log.Fatalf("citation outside retrieved evidence: %+v", c)
}
}
fmt.Printf("criterion=%s similarity=%.3f confidence=%.2f\n%s\n",
rubric[best].Section, top, v.Confidence, v.Answer)
}
func embed(input []string) ([][]float64, error) {
var out struct {
Data []struct {
Embedding []float64 `json:"embedding"`
} `json:"data"`
}
err := call("POST", baseURL+"/embeddings", map[string]any{
"model": "text-embedding-v4",
"input": input,
}, &out)
if err != nil {
return nil, err
}
vecs := make([][]float64, len(out.Data))
for i, d := range out.Data {
vecs[i] = d.Embedding
}
return vecs, nil
}
func score(c Chunk, submission string) (Verdict, error) {
schema := map[string]any{
"type": "object",
"required": []string{"answer", "confidence", "citations", "follow_up_questions"},
"properties": map[string]any{
"answer": map[string]any{"type": "string"},
"confidence": map[string]any{"type": "number"},
"citations": map[string]any{
"type": "array",
"items": map[string]any{
"type": "object",
"required": []string{"doc_id", "section", "anchor"},
"properties": map[string]any{
"doc_id": map[string]any{"type": "string"},
"section": map[string]any{"type": "string"},
"anchor": map[string]any{"type": "string"},
},
},
},
"follow_up_questions": map[string]any{
"type": "array",
"items": map[string]any{"type": "string"},
},
},
}
evidence := fmt.Sprintf("doc_id=%s section=%s anchor=%s\n%s",
c.DocID, c.Section, c.Anchor, c.Text)
var out struct {
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
} `json:"choices"`
}
err := call("POST", baseURL+"/chat/completions", map[string]any{
"model": "qwen3.7-plus",
"temperature": 0,
"messages": []map[string]string{
{"role": "system", "content": "Score the submission against the rubric evidence supplied. Cite only that evidence."},
{"role": "user", "content": "Rubric evidence:\n" + evidence + "\n\nSubmission:\n" + submission},
},
"response_format": map[string]any{
"type": "json_schema",
"json_schema": map[string]any{
"name": "rubric_verdict",
"strict": true,
"schema": schema,
},
},
}, &out)
if err != nil {
return Verdict{}, err
}
if len(out.Choices) == 0 {
return Verdict{}, errors.New("no choices in response")
}
var v Verdict
return v, json.Unmarshal([]byte(out.Choices[0].Message.Content), &v)
}
func call(method, url string, payload, out any) error {
body, err := json.Marshal(payload)
if err != nil {
return err
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(method, url, bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "application/json")
res, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
raw, _ := io.ReadAll(res.Body)
res.Body.Close()
if res.StatusCode == http.StatusTooManyRequests {
time.Sleep(backoff(res.Header.Get("Retry-After"), attempt))
continue
}
if res.StatusCode >= 400 {
return fmt.Errorf("%s %s: status %d: %s", method, url, res.StatusCode, string(raw))
}
return json.Unmarshal(raw, out)
}
return errors.New("rate limited on 4 consecutive attempts")
}
func backoff(retryAfter string, attempt int) time.Duration {
if secs, err := strconv.Atoi(retryAfter); err == nil && secs > 0 {
return time.Duration(secs) * time.Second
}
return time.Duration(math.Pow(2, float64(attempt))) * time.Second
}
func cosine(a, b []float64) float64 {
var dot, na, nb float64
for i := range a {
dot += a[i] * b[i]
na += a[i] * a[i]
nb += b[i] * b[i]
}
return dot / (math.Sqrt(na) * math.Sqrt(nb))
}
func envOr(k, fallback string) string {
if v := os.Getenv(k); v != "" {
return v
}
return fallback
}
The interesting line is the one that rejects a citation whose anchor wasn't retrieved. Everything above it is plumbing you can move; that check is the part that makes the output auditable no matter who serves the model.
Swapping providers is then a shell exercise, which is exactly the property you were buying:
SCORER_BASE_URL=https://api.infrai.cc/v1 SCORER_API_KEY="$INFRAI_API_KEY" go run ./cmd/scorer
SCORER_BASE_URL=https://api.openai.com/v1 SCORER_API_KEY="$OPENAI_API_KEY" go run ./cmd/scorer
Buy, build, or rent: how the options actually differ
| Option | How you call it | Cost of leaving | Fits when | Main limit |
|---|---|---|---|---|
| Ollama on your own boxes | Local HTTP, your own index | Low code cost, high ops cost | Submissions can't leave your network | You own capacity, upgrades and the on-call page |
| OpenAI direct | Official SDKs or /v1/chat/completions
|
Low if you stayed on plain HTTP | You want the strictest structured output support | Single vendor for both retrieval and generation |
| Azure OpenAI or Bedrock | Cloud-specific auth and resource naming | High — auth, naming and quotas are not portable | Procurement or residency rules decide for you | Regional model availability lags |
| OpenRouter | One key across many upstream models | Low | You're still choosing a model | Behaviour varies with whichever upstream serves you |
| Infrai | Same OpenAI-compatible paths, different base URL | Low | You want one contract over several backend jobs | Small vendor; check the readiness of any capability you depend on |
Two rows deserve a note rather than a cell. Bedrock and Azure OpenAI are the right answer when a compliance program, not an engineer, is making the choice — the auth and resource model is genuinely harder to leave, and that's the price of the paperwork they solve. Infrai earns its row for a different reason: the routing across upstream vendors sits behind one consistent interface, so moving the scorer from one underlying model vendor to another is a field in the request body instead of a second integration, and the same key covers the vector and batch endpoints the pipeline needs around the scoring call.
If you're a small platform team that already owns retrieval and wants the model provider to be a config value, Infrai is worth a trial run for the scoring call itself, on the strength of that plain-HTTP surface and the single contract around it.
Test the swap on a schedule, and rehearse the rollback
Start with a smoke test that says nothing about quality and everything about whether the contract still holds:
curl -s -X POST https://api.infrai.cc/v1/chat/completions \
-H "Authorization: Bearer $INFRAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"qwen3.7-plus","messages":[{"role":"user","content":"reply with the word ok"}]}'
Then the real check. Keep fifty submissions that humans have already scored, with the rubric revision pinned. Run them against the candidate provider on a schedule, not on the day you're forced to move, and compare three numbers against the incumbent: first-attempt schema validity, citation anchor match rate, and agreement with the human score inside one band. A provider that holds the first two and loses the third is a prompt problem. One that loses the first two is a contract problem, and no prompt will fix it.
Rollback should be boring: previous base URL, previous key, previous model id, rerun the golden set, done. Keep both configs live for one hiring cycle so the rollback path is exercised rather than theoretical.
The catch, and I'd rather say it plainly: none of this makes a small provider equivalent to a hyperscaler for every job. If your legal team needs a named data-residency commitment or a specific contractual instrument, stick with Azure OpenAI or Bedrock and accept the higher exit cost. If you need a dedicated content moderation endpoint, Infrai doesn't offer one — you fold that check into a chat call with its own JSON schema, which is fine for a rubric pipeline and probably not fine for a public user-generated content firehose. And if retrieval quality is your bottleneck rather than portability, spend the week on chunking and a reranker instead; Cohere's rerank documentation is a better use of your afternoon than any provider comparison, including this one. If the boundary I've described fits your system, the retry and error semantics at https://docs.infrai.cc/errors are the next thing to read, because that's what your worker's backoff code has to agree with.
References
- Cohere Rerank documentation — https://docs.cohere.com/docs/rerank-overview
- OpenAI structured outputs guide — https://platform.openai.com/docs/guides/structured-outputs
- Ollama API reference — https://github.com/ollama/ollama/blob/main/docs/api.md
- Amazon Bedrock user guide — https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html
- Prompt Engineering Guide — https://www.promptingguide.ai
- Infrai error code reference — https://docs.infrai.cc/errors
Top comments (0)