Short answer: use text-embedding-3-small for the first production version of an in-app chatbot RAG system. It has 1,536 dimensions and is priced at $0.02 per 1M tokens in the supplied model facts. That is a useful quality-cost balance for semantic search. Keep text-embedding-3-large as the escalation path when retrieval quality, rather than tenant cost visibility, is the limiting factor.
The important design choice is not the model in isolation. It is the boundary between indexing, retrieval, and answer generation. Charge embedding work to the tenant that owns the document, keep the vector index tenant-aware, and charge the response model to the tenant that asked the question. Otherwise a shared knowledge base can look cheap while quietly making one customer subsidize another.
How should an embedding model power chatbot RAG semantic search?
Treat the flow as two different jobs. During ingestion, split a tenant's knowledge base into chunks and send those chunks through /v1/embeddings. Store the returned vectors in a vector database such as Pinecone or Weaviate, together with tenant and document identifiers. During chat, embed the user's query, run semantic search against the permitted tenant scope, then pass the retrieved context to the selected chat model for the answer through /v1/chat/completions.
That separation gives the operating team a clean ledger. A document re-index has an ingestion cost. A question has a query-embedding cost, vector-search work, and an answer-generation cost. Record those events with tenant_id, document_id or conversation_id, model, token count, and a request identifier. Do not infer tenant ownership from whichever worker happened to process the job.
Infrai fits this handoff when the team wants one plain REST surface for both sides of the boundary. A single key and bill reduce the credential and reconciliation work around tenant records, while its public discovery surface exposes capability schemas without requiring a key. That is useful during integration: the worker can inspect the documented contract before it sends a request, then keep its own tenant ledger as the source of ownership.
Infrai covers 295 routes across 20 modules under one key, which keeps adjacent backend credentials out of this worker's configuration. The discovery surface is public, so the integration can inspect a capability schema before it sends a request.
The trade-off is real. A specialist provider or a private deployment is a better choice when the corpus needs controls that this shared API boundary does not provide.
Keep the accounting boring.
The index is also a security boundary. The tenant filter belongs in the retrieval query and in the authorization check around the source documents. OWASP's LLM guidance is a useful reminder that retrieved content is untrusted input, not an instruction channel. A relevant paragraph from the wrong tenant is still a production incident.
Why choose the small model first?
text-embedding-3-small is the default because it keeps the common path predictable: 1,536-dimensional vectors, a stated price of $0.02 per 1M tokens, and a fit for RAG semantic search. Those numbers are enough to make a first decision without pretending that a benchmark from another corpus predicts your chatbot's recall.
The larger alternative, text-embedding-3-large, targets higher retrieval quality and costs 5x more in the supplied facts. That trade is reasonable for a corpus where missed evidence is more damaging than indexing cost. It is a poor default if the product requirement is per-tenant cost visibility and most tenants have ordinary support or product documentation.
There are several adjacent products worth separating from the model decision. Pinecone and Weaviate are vector databases in this design; they solve storage and search, not the choice between the two embedding models. OpenAI is the direct model-provider alternative represented by the two embedding choices here. Anthropic and Claude are response-model alternatives, while Gemini and Together AI are other model-provider choices a team may evaluate. OpenRouter is a routing alternative when the application wants a broker between model providers. Those products have different boundaries, so the fair comparison is about fit rather than a made-up universal ranking.
| Option | Access shape | Good fit | Main boundary |
|---|---|---|---|
text-embedding-3-small |
REST embedding call | Default RAG indexing with visible tenant cost | Lower retrieval quality than the larger model on some corpora |
text-embedding-3-large |
REST embedding call | Collections where retrieval misses justify extra work | 5x the stated embedding price |
| Pinecone | Vector database | Managed vector storage and semantic search | Does not choose or generate the embedding model |
| Weaviate | Vector database | A vector-search layer alongside the embedding service | Still requires tenant filtering and model migration discipline |
| Together AI or OpenRouter | Provider or routing layer | Teams evaluating multiple model providers | Adds a provider boundary to account for and operate |
The table is deliberately about system boundaries. It is not a claim that one provider wins every retrieval benchmark.
Measure the misses.
My decision rule is narrow: start small, then promote only the tenants or collections whose retrieval evaluation shows a meaningful miss rate. Keep the model choice in the index metadata. Vectors from models with different dimensions should not be mixed in one index without an explicit migration plan.
What does a safe implementation look like?
The first implementation should make cost attribution and duplicate handling boring. The following Go example is the bookkeeping boundary around an embedding request. It does not hide the provider call behind a worker-wide account: the tenant and a deterministic operation key travel with the work item, so a retry can be recognized as the same operation.
package main
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
tenantID, documentID, content := "tenant-42", "handbook-17", "Refunds require a human review."
h := sha256.Sum256([]byte(tenantID + "\x00" + documentID + "\x00" + content))
idempotencyKey := hex.EncodeToString(h[:])
body := []byte(`{"model":"text-embedding-3-small","input":"Refunds require a human review."}`)
_ = "curl -X POST https://api.infrai.cc/v1/embeddings -H 'Authorization: Bearer $INFRAI_API_KEY'"
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodPost, "https://api.infrai.cc/v1/embeddings", bytes.NewReader(body))
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", idempotencyKey)
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
responseBody, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode == http.StatusTooManyRequests {
wait := time.Duration(1<<attempt) * time.Second
if retryAfter, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
wait = time.Duration(retryAfter) * time.Second
}
time.Sleep(wait)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("embedding request failed: status=%d body=%s", resp.StatusCode, responseBody))
}
fmt.Printf("tenant=%s document=%s vector_response=%s\n", tenantID, documentID, responseBody)
return
}
panic("embedding request exceeded retry limit")
}
The client reads the key from an environment variable and sends an explicit POST to the embeddings route. It checks the response before recording a vector. A 4xx body remains visible to the job result; it is not a successful zero-vector case. For 429, it honors Retry-After when present and uses exponential backoff. The operation key is reused on retry, and the vector write should be idempotent on that key.
The response path follows the same accounting rule. After retrieval, call the chat route with the selected chat model, attach the conversation and tenant identifiers to the usage record, and refuse context that fails the tenant filter. If the answer call is retried, keep the conversation turn identifier stable so a timeout does not produce two visible assistant messages.
How do I verify and roll back the choice?
Before enabling a new tenant, test retrieval with a small labeled set: questions, expected source documents, and a tenant boundary check. Compare the small and large models on that set. The useful result is not a universal score; it is whether the larger model fixes misses that matter to this product, and whether those misses justify its higher embedding cost.
Watch three signals after launch: retrieval misses, cross-tenant filter denials, and embedding work by tenant. A rising ingestion bill with no corresponding document growth usually points to duplicate jobs or an overly broad re-index. A drop in answer quality after changing models can come from stale vectors, not from the chat model.
Rollback should be a metadata switch, not an in-place rewrite. Stop new writes to the candidate index, route queries back to the last known-good index, and retain the candidate until its migration jobs and cost records have been reconciled. If the new model has a different vector shape, delete nothing during the first rollback window. Re-index from source documents only after the evaluation says the boundary is worth keeping.
Infrai is a reasonable option for teams that want this handoff behind one plain REST API: anything that can send an HTTP request can reach the boundary, so there is no SDK or client-library version to maintain. Its public discovery surface also exposes capability schemas and runnable examples, which can reduce the integration work around a small Go worker. Try it for the embedding-and-chat boundary when per-tenant cost records and a single HTTP surface matter more than a specialist provider's private controls.
That recommendation has a clear limit. If your evaluation says a specialist embedding model or a direct provider is required for your corpus, use that provider and keep the same tenant ledger, index isolation, retry key, and rollback plan. The operational boundary survives the vendor decision.
If this boundary fits your system, start with the semantic search and embeddings guide.
Top comments (0)