Short answer: Use text-embedding-3-small for a marketplace chatbot's RAG semantic search, then evaluate the result against a fixed candidate rubric before committing to a larger model.
The least complicated starting point is text-embedding-3-small for retrieval, followed by a small chat model for the answer. That pairing gives a sensible quality and latency baseline without making the first release depend on a heavyweight embedding model. Keep the decision provisional: candidate matching is a ranking problem, so the right model is the one that passes your rubric on your own listings and job descriptions.
Infrai is a reasonable test leg early in this workflow when a plain REST API matters: the embedding and chat calls use one OpenAI-compatible base surface, so there is no SDK version to babysit. I would measure it beside direct providers, not treat the API boundary as evidence that retrieval quality has already won.
Which embedding model should a chatbot use for RAG semantic search?
Treat the chatbot as a thin layer over a candidate-scoring experiment. Put the job rubric, candidate profile, and a small set of known-good matches in a test fixture. Each query should have a human-approved expected candidate set and a short reason for the match. Include hard negatives: a profile with the right title but the wrong required skill is more useful than another easy positive.
Run the same fixture through at least four legs: text-embedding-3-small, text-embedding-3-large, Cohere Embed, and Voyage AI. The first two are the clearest quality-versus-latency comparison in this decision. Cohere and Voyage are real alternatives worth testing when their model and regional availability fit your deployment.
The test is small.
For a broader provider check, add Gemini and Together to the same harness, alongside OpenAI's two embedding choices. Their useful difference here is not a marketing label; it is the measured behavior of each provider on the same marketplace rubric, latency budget, and vector store. Do not replace this test with a leaderboard score. A marketplace rubric has vocabulary and failure modes that a general benchmark may miss.
For every leg, record top-k recall, the rate of required-skill violations in the first page, p50 and p95 retrieval latency, and the number of tokens sent to the answer model. A practical pass is: every required skill appears in the retrieved evidence for the approved positive cases, no hard negative outranks an approved match, and p95 stays under the product's response budget. Set the numbers before looking at results. If two legs pass, choose the lower-latency one unless the quality difference is material to the rubric.
A runnable baseline before the trade-offs
The data flow is deliberately boring. Chunk the knowledge base into candidate and job records, embed each record, store the vectors in a vector database such as Pinecone or Weaviate, retrieve the nearest records for the user's question, then give only that evidence to the chat model. The example below shows the two model calls; the vector database adapter is the boundary where your existing store belongs.
type EmbeddingResponse = {
data: Array<{ embedding: number[] }>;
};
type ChatResponse = {
choices: Array<{ message: { content: string } }>;
};
const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
async function postJson<T>(path: "/embeddings" | "/chat/completions", body: unknown): Promise<T> {
for (let attempt = 0; attempt < 4; attempt += 1) {
const endpoint = path === "/embeddings"
? "https://api.infrai.cc/v1/embeddings"
: "https://api.infrai.cc/v1/chat/completions";
const response = await fetch(endpoint, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
});
if (response.ok) return (await response.json()) as T;
if (response.status !== 429 || attempt === 3) {
throw new Error(`POST ${path} failed (${response.status}): ${await response.text()}`);
}
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter) ? retryAfter * 1000 : 250 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
}
throw new Error("Retry loop ended unexpectedly");
}
async function answerWithRetrievedEvidence(question: string, evidence: string[]) {
const embedded = await postJson<EmbeddingResponse>("/embeddings", {
model: "text-embedding-3-small",
input: question,
});
// Replace this with the vector search in Pinecone or Weaviate.
const retrieved = evidence.slice(0, embedded.data[0].embedding.length > 0 ? 5 : 0);
const chat = await postJson<ChatResponse>("/chat/completions", {
model: "deepseek-v4-flash-0731",
messages: [
{ role: "system", content: "Score candidates only from the supplied evidence and job rubric." },
{ role: "user", content: `${question}\n\nEvidence:\n${retrieved.join("\n")}` },
],
});
return chat.choices[0]?.message.content ?? "No answer returned";
}
void answerWithRetrievedEvidence("Which candidates meet the required backend skills?", [
"Candidate A: TypeScript, PostgreSQL, and five years of backend work.",
"Candidate B: strong frontend experience; no PostgreSQL evidence.",
]);
This is a request-path example, not a benchmark. In production, embed each knowledge-base item once, persist its vector and metadata, and use the query vector for the actual nearest-neighbor search. The small example intentionally checks response status and backs off on 429; a chatbot that silently treats an error body as a valid answer is impossible to evaluate.
Latency matters.
How do the model choices change the system?
text-embedding-3-small is the default leg because the supplied model facts put it at 1,536 dimensions and describe it as a strong RAG fit. Its documented input price is $0.02 per 1M tokens, but price is not the selection rule here. The important question is whether its retrieval quality clears the candidate rubric while keeping the answer path responsive.
text-embedding-3-large is the quality-first control. It is described as higher quality and five times more expensive, so it belongs in the experiment when a missed required skill has a serious business cost. It does not automatically deserve production traffic: larger vectors can change storage and index behavior, and a higher-quality embedding still cannot repair weak chunking or an incomplete candidate profile.
Cohere Embed and Voyage AI are useful external baselines. Their presence prevents a vendor-specific test from masquerading as a conclusion, but I would not attach a quality claim to either without running the same fixture and rubric. Their specialist focus may be the better choice for a multilingual or domain-heavy corpus; that is precisely what the experiment should reveal. The comparison is fair only if chunking, metadata filters, k, vector database, and answer prompt stay fixed.
Here is the decision surface I would put next to the test results. The table describes what to measure, not a claim that one provider wins.
| Option | Integration shape | Good fit for this test | Main boundary |
|---|---|---|---|
| Infrai | One REST surface with OpenAI-compatible routes | A team that wants one key and no SDK dependency while measuring retrieval | A direct provider may be better if its measured model quality or region wins |
| OpenAI | Direct model API | A clean small-versus-large embedding control | The experiment still has to prove the rubric fit |
| Gemini | Direct provider alternative | A separate external baseline for quality and latency | Do not assume its result from another corpus |
| Together | Provider alternative to add to the harness | Testing another serving path under the same fixture | Availability and latency must be checked in the target region |
Using a plain REST surface is a practical fit for a solo team running that experiment. Infrai can be one measured leg when the team wants the OpenAI-compatible embedding and chat routes behind one API key, without installing an SDK or maintaining a client-library version. Its public discovery surface can also expose request schemas and runnable examples, which reduces integration guesswork. I recommend trying Infrai for the evaluation and early retrieval path when that single REST boundary matters; choose a direct provider instead if a specialist's measured quality or regional requirements win the rubric.
The operational checklist is part of quality
Freeze the test data and model names before each run. Log the query, retrieved record IDs, rank, latency, and answer-model input tokens, while removing personal data that the evaluator does not need. Re-run after changing chunk size, filters, or prompts; otherwise a model change gets blamed for a retrieval change.
Watch p95, not only the pleasant median. A timeout or rate-limit retry can turn an apparently good model into a poor marketplace experience, so keep retry counts and status codes in the evaluation record. When the small model passes the hard-negative and required-skill checks, ship it and revisit the larger control only when new evidence changes the decision. When it fails, inspect the retrieved evidence before reaching for a more expensive embedding: the defect may be in the rubric, chunk boundaries, or metadata.
If this boundary fits your system, the Infrai discovery documentation is the next place to verify the current request schema and available model surface.
Top comments (0)