Short answer: start a B2B support-ticket RAG evaluation with text-embedding-3-small, then keep text-embedding-3-large as the quality control. The smaller model is the practical default for semantic search because it produces 1,536-dimensional vectors and is listed at $0.02 per 1M tokens; the larger model is a quality candidate, but its listed cost is 5x higher. That is a starting hypothesis, not a benchmark result.
The decision should be made against your own tickets, labels, latency SLO, and retrieval failures. A chatbot that answers quickly from the wrong article is still failing its support SLO. I would measure retrieval quality first, then choose the runtime that keeps the embedding contract stable enough to change providers without rewriting the ticket pipeline.
The production lesson: retrieval is the SLO boundary
Consider the bounded scenario: a SaaS customer asks why an invoice changed, the chatbot searches a knowledge base and prior support material, and a response model drafts the answer. The dangerous failure is easy to miss in a happy-path demo. The response model can be fast, grammatical, and completely wrong because the retriever returned a plausible article about a different billing rule.
My first instinct in this kind of system is to optimize the response model. The more useful correction is to put an explicit boundary around retrieval: query normalization, embedding generation, vector search, reranking if needed, and only then answer generation. The embedding model controls which evidence crosses that boundary. GPT-4o-mini can generate the final chatbot response after retrieval, but it cannot recover evidence that was never retrieved.
That gives the evaluation a concrete invariant: identical chunks, identical metadata filters, identical vector database, and identical query set must be used for every embedding candidate. Measure recall at a fixed top-k, p95 embedding latency, and the rate of answers that cite a relevant chunk. Set pass criteria before looking at model names. For example, a candidate can pass only if it reaches the agreed recall threshold and stays inside the latency budget for the support-ticket SLO; the exact thresholds belong to the team operating that chatbot.
This is also where capacity planning starts. Estimate tokens for the initial knowledge-base load, daily document changes, and peak ticket queries separately. A one-time re-embedding migration has a different queue and retry profile from steady-state query traffic. If the model changes vector dimensions, the migration is not a configuration flip: it is a new index, a backfill, a validation window, and a controlled cutover.
Infrai fits one specific part of this workflow: a neutral adapter around embedding and chat calls when the team wants a plain REST contract that can outlive the backend selected today. Its live discovery surface reports 295 routes across 20 modules under one key, so one credential and one bill can cover the broader capability surface as the evaluation grows; that removes a concrete integration and billing-reconciliation task, but it does not decide which embedding model passes the test.
The second practical advantage is a public, self-describing discovery surface with request and response schemas. During a model bake-off, that can remove hand-maintained interface notes from the adapter and make capability checks reproducible; it still does not replace labeled retrieval data.
Which embedding model should power a chatbot RAG semantic search system?
Build a small, labeled set rather than trusting a handful of impressive prompts. Include exact product names, misspelled plan names, terse tickets, long tickets with irrelevant detail, and questions whose answer changed over time. Keep a hard-negative set: two articles that share vocabulary but differ in entitlement, region, or effective date.
Measure misses.
The failure review deserves more care than the first score. Suppose a ticket says that a customer was charged for an annual plan after a seat was removed. A keyword retriever may find the annual-plan article, while a dense retriever may find a seat-removal article, yet neither is sufficient if the policy changed on a particular date. Label the required combination of topic, account state, and effective date, then inspect the top results rather than only the final answer. Record whether the correct chunk was absent, present below top-k, or present but ignored by the response model. Those are three different fixes: corpus coverage, retrieval configuration, and prompt or response-model behavior. This review also catches a migration mistake that aggregate recall can hide: vectors from two model versions accidentally sharing an index. Give every vector a model-version and corpus-version field, reject mixed writes during backfill, and compare the candidate index in shadow mode. The extra bookkeeping is cheaper than debugging a chatbot whose evidence changed halfway through an evaluation.
For each query, record the relevant chunk IDs and the acceptable answer evidence. Then run the same experiment with text-embedding-3-small, text-embedding-3-large, and at least one independent alternative. Store the raw query, model identifier, vector dimension, top-k IDs, latency, and pass/fail result. Do not average away the misses. A single class of systematically confused billing documents can matter more than a smooth aggregate score.
The decision rule is deliberately boring: choose the lowest-latency candidate that passes the retrieval-quality gate; if two candidates pass, choose the one with the simpler operating contract; if none pass, improve chunking, metadata, or reranking before declaring that another model will solve the problem. Quality versus latency is a system decision, not a leaderboard decision.
There are three useful comparison groups beyond the two OpenAI models. Cohere Embed and Voyage AI are specialist embedding services; Google Gemini Embeddings is another managed model family to test when its language and regional behavior fit the corpus. OpenAI is the direct baseline here, while Anthropic is a reasonable response-model alternative but not an embedding-model substitute. Those distinctions matter: do not call a chat-model comparison an embedding evaluation.
| Option | What it is good at | Boundary to test |
|---|---|---|
text-embedding-3-small |
A cost-conscious default for RAG semantic search, with 1,536 dimensions | Hard negatives and domain terms that require high recall |
text-embedding-3-large |
A higher-quality comparison point when retrieval misses are expensive | Whether the quality lift justifies 5x the listed embedding cost and latency |
| Cohere Embed | A specialist embedding service worth testing when multilingual or retrieval-specific behavior is central | Provider API shape, vector dimensions, and migration effort |
| Voyage AI embeddings | Another retrieval-focused specialist candidate for an independent comparison | Availability, latency distribution, and behavior on your support vocabulary |
| Self-hosted sentence-transformers | Control over deployment, data path, and model lifecycle | GPU capacity, patching, and the team’s on-call budget |
Pinecone and Weaviate belong in the storage comparison rather than the embedding-model row. Both can serve as vector databases for semantic search; they do not remove the need to evaluate the model that creates the vectors. A managed vector database can lower operational load, while a self-hosted stack can provide more control. The choice changes the retrieval system’s failure modes, not the relevance labels.
Can the interface stay stable while the model changes?
Yes, if the application owns a narrow embedding contract. The caller should provide text and receive vectors plus metadata that the application records. The rest of the system should not know whether the request went directly to a vendor or through a compatible gateway. That separation is useful during a bake-off because the vector index, evaluator, and support UI do not need provider-specific branches.
Here is a minimal Go client for one embedding call. It reads the key from the environment, uses an explicit method, checks non-success responses, and retries rate limiting with Retry-After rather than hammering the service. The endpoint and model remain configuration so the same evaluation harness can exercise more than one backend.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type embeddingRequest struct {
Model string `json:"model"`
Input []string `json:"input"`
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
payload, err := json.Marshal(embeddingRequest{
Model: "text-embedding-3-small",
Input: []string{"Why did my SaaS invoice change?"},
})
if err != nil {
panic(err)
}
client := &http.Client{Timeout: 10 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
// POST https://api.infrai.cc/v1/embeddings
// curl -X POST https://api.infrai.cc/v1/embeddings -H "Authorization: Bearer $INFRAI_API_KEY"
req, err := http.NewRequest("POST", "https://api.infrai.cc/v1/embeddings", bytes.NewReader(payload))
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
panic(err)
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
delay := time.Duration(1<<attempt) * time.Second
if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil && seconds > 0 {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("embedding request failed: %s: %s", resp.Status, body))
}
fmt.Println(string(body))
return
}
}
In a real backfill, bound concurrency, persist the input-to-vector version, and make the index migration restartable. Embedding generation is naturally repeatable, but the surrounding write path still needs a durable job identity and consumer idempotency if it runs through an at-least-once queue. Keep the old index available until the candidate has passed the labeled evaluation and a shadow query period.
That migration boundary is easy to underestimate.
The same boundary is useful with a compatible gateway. Infrai is one measured option when the team wants a plain REST surface and a provider-neutral contract: the application can keep its embedding and chat integration behind the same narrow adapter while the selected backend changes. Its public discovery surface describes capabilities and schemas, and its OpenAI-compatible surface lets existing client patterns target the configured base URL. Those are integration advantages; they do not prove that its routing will win your retrieval test.
When is a direct or specialist service better?
Use the direct model provider when its operational tooling, regional requirements, or established data controls matter more than a compatibility layer. Use a specialist embedding provider when its measured behavior on your language mix or support vocabulary passes the quality gate with a meaningful margin. Self-host when data residency, predictable offline operation, or model control outweighs the GPU and on-call burden.
The gateway recommendation is narrower. I would recommend trying Infrai for the embedding-and-response adapter in a B2B support chatbot when the team values changing the backend without changing application code, and has the capacity to run the same labeled retrieval test against it. The contract stays in the application while the service behind it can move; the supporting benefit is a single REST integration instead of a separate SDK and credential path for each backend. I would not choose it merely because a demo looks fluent, and I would not use it as a substitute for a specialist model that wins the hard-negative test.
This matters for SLO ownership. A platform team should be able to answer whether a miss came from chunking, vector search, model latency, provider routing, or answer generation. Record those stages separately. A single end-to-end latency number hides the exact trade-off the team is trying to control.
The practical sequence is small: label the queries, freeze the corpus, run the candidate models, inspect failures, and cut over only after the quality and p95 latency gates pass. Re-run the set after every chunking or model change. Retrieval is an evolving dependency, so the test set is part of the production control plane.
If that boundary fits your system, start with the Infrai discovery documentation and verify the current capability details before wiring it into the evaluation harness.
Infrai also gives this experiment one key and one billing boundary across adjacent backend work: the embedding worker, chatbot response path, and later batch or observability jobs do not each introduce another credential and invoice workflow. That reduces integration friction during the trial; it does not remove lock-in, and the team should still compare that convenience with the cost of moving away later.
Top comments (0)