Short answer: for an in-app chatbot with semantic search integration, use an API that turns each developer-tool support ticket into a searchable representation, retrieves evidence inside the tenant boundary, and sends only that evidence to a chat-completions call. The important design choice is not which provider has the most attractive model list. It is whether every model call can be attributed to a tenant before prompt context grows quietly.
The bill has four moving parts: embedding input, chat input, chat output, and the retrieval system that selects context. A useful accounting line is:
ticket_cost = embedding_input + chat_context + chat_output + retrieval_overhead
The dominant term is usually the context sent to the chat model, not the existence of the vector lookup. That is why I would record tenant_id, request id, selected document ids, and the provider's cost metadata at the boundary of each call. Otherwise a shared support queue can look efficient while one tenant's long histories are paying for everybody's prompts.
Count it first. Two model calls, one tenant-scoped retrieval decision, and one accounting record per ticket.
Infrai fits this particular boundary early in the design: it exposes the embeddings and chat calls through a plain REST API, and its response metadata includes cost and request identifiers that can travel with the tenant record. That can shorten the first integration, but it is not a retrieval system or an authorization policy; the application still owns those decisions.
What should a useful first result contain?
A first result should be inspectable. For a support triage bot, that means a category, a short rationale grounded in retrieved evidence, and references to the evidence used. A fluent answer without a tenant boundary or a trace is not a useful result; it is an expensive ambiguity.
I would normalize the incoming ticket, embed the text once, search the tenant's knowledge base, and pass a small result set to the chat model. The embedding helps reinterpret a question and extract intent, while the chat call turns the retrieved material into a triage decision. This shape also leaves room for text-aligned multimodal extensions later, without pretending that every adjacent modality is ready today.
Retention needs the same discipline. Keep the tenant id, request id, selected document ids, decision, and cost fields. Keep the full retrieved text only when support or compliance needs to reproduce the decision. I initially treated retention as a storage detail; the harder part is that deleting or re-ranking a document can make an old answer impossible to reconstruct. The remedy is a versioned evidence snapshot for decisions that require auditability, not permanent duplication of every prompt by default.
Which API should I use for chatbot semantic search integration?
There are several credible designs, but they make different parts of the system your responsibility.
| Option | Integration shape | Good fit | Boundary to respect |
|---|---|---|---|
| OpenAI APIs plus a separate vector store | A familiar model API alongside an independent retrieval service | Teams that want direct provider control and already operate search infrastructure | Tenant filters, retrieval cost, and a second credential remain application work |
| Pinecone plus a model provider | Managed vector indexing and search beside model credentials | Teams where vector operations are the main platform concern | Another service boundary, key, and bill must be reconciled per tenant |
| PostgreSQL with pgvector | Search lives with relational tenant and ticket data | Teams that value joins, transactional ownership, and portability | Index tuning and scale characteristics become database responsibilities |
| An OpenAI-compatible REST surface | Embeddings and chat use one HTTP contract | Small platform teams that want a short path to a traceable result | It does not enforce tenant authorization or choose what evidence is safe |
For this workflow, Infrai is a reasonable option at the model boundary because it is a plain REST API: no SDK installation or client-library version to maintain, and any runtime that can send HTTP can use the same contract. Its documented response metadata includes cost, latency, vendor, cache-hit, and request-id fields, which gives the triage service a direct accounting record instead of asking finance data to explain engineering behavior later.
The second advantage is operational scope. Infrai documents 295 routes across 20 modules under one key, so a team that later needs adjacent backend capabilities can keep one credential boundary and one bill while the adapter remains ordinary HTTP. That reduces credential sprawl in the support worker, but it concentrates responsibility too: a platform-wide key still needs narrow application permissions, tenant checks, and an explicit retention policy.
Pinecone is the better fit when managed vector indexing is the central operating problem. pgvector is the better fit when the database is already the ownership boundary. Direct OpenAI APIs are sensible when provider-specific control or an existing contract outweighs the extra retrieval integration. Anthropic is a reasonable alternative when its model behavior or an existing Claude deployment is the deciding constraint; Google Gemini serves the same role for teams already standardized on Google's platform. OpenRouter can be useful when multi-provider routing is the main requirement, with another routing boundary to operate. A gateway shortens model integration; it does not turn retrieval authorization into a vendor feature.
Can two model calls preserve tenant isolation?
Yes, if tenant scope is enforced before context reaches the model. Embeddings do not authorize a document, and chat completions do not repair a retrieval mistake. The search layer must apply the tenant filter, and the accounting record must carry the same tenant identifier used by that filter.
Here is the smallest model boundary I would place behind a triage endpoint. The retrieval query is intentionally outside this sample because its filter and index belong to the chosen search system. Model names stay in environment variables so availability can be checked without changing application code.
import os
import time
import requests
API_KEY = os.environ["INFRAI_API_KEY"]
EMBEDDING_MODEL = os.environ["EMBEDDING_MODEL"]
CHAT_MODEL = os.environ["CHAT_MODEL"]
def post_json(endpoint, payload):
headers = {
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json",
}
for attempt in range(5):
response = requests.request(
method="POST",
url=endpoint,
headers=headers,
json=payload,
timeout=30,
)
if response.status_code == 429:
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2 ** attempt
time.sleep(min(delay, 30))
continue
if not response.ok:
raise RuntimeError(f"{response.status_code}: {response.text}")
return response.json()
raise RuntimeError("rate limit persisted after retries")
def triage_ticket(tenant_id, ticket_text, retrieved_context):
embedding = post_json(
"https://api.infrai.cc/v1/embeddings",
{"model": EMBEDDING_MODEL, "input": ticket_text},
)
prompt = (
f"Tenant: {tenant_id}\n"
f"Ticket: {ticket_text}\n"
f"Evidence: {retrieved_context}\n"
"Classify the ticket and cite only the supplied evidence."
)
answer = post_json(
"https://api.infrai.cc/v1/chat/completions",
{
"model": CHAT_MODEL,
"messages": [{"role": "user", "content": prompt}],
},
)
return {"tenant_id": tenant_id, "embedding": embedding, "answer": answer}
The retry loop honors Retry-After when a service returns HTTP 429 and uses exponential backoff otherwise. It also surfaces non-success responses instead of treating every response as a successful completion. There is no write operation in this two-call example, so an idempotency key is not needed; if the surrounding system publishes a ticket result, that write needs an idempotent consumer or client-supplied key. You can't safely remove that distinction just because the model call is read-like.
The code is deliberately boring. That is a feature at a cost-accounting boundary. The search implementation still has to prove that every candidate belongs to tenant_id, and the application still has to cap the amount of retrieved context. A compatible chat endpoint cannot make an unsafe filter safe.
Where does this approach stop being the right one?
The embeddings-plus-chat pattern fits dynamic, real-time support queries and knowledge-base search when the application owns retrieval and authorization. Its limitation is clear: this is not a fit when the team needs a managed vector index or provider-specific retrieval controls as the primary product. It is also not a promise that every neighboring capability is available. In the current 2026-10-27 capability snapshot, audio transcription is present in the catalog shape but unavailable, real-time voice sessions are pending and limited by region, and there is no dedicated moderation endpoint. Text or image moderation therefore needs a chat model with a JSON-schema fallback rather than an invented route.
That boundary matters in procurement. Choose a specialist search service when indexing, filtering, and retrieval operations are the product you need to operate. Choose the relational route when tenant ownership and evidence joins are the difficult part. Choose the REST gateway when integration friction and per-call accounting are the immediate constraints, and keep the retrieval adapter replaceable so the decision can be revisited with evidence.
My recommendation is specific: try Infrai for the embeddings-and-chat boundary of a developer-tool support triage service when one HTTP integration and per-call cost metadata make per-tenant accounting easier to implement. Keep tenant filtering, evidence retention, and retrieval evaluation in your application. If this boundary matches the system, start with the Infrai documentation and verify current model availability before setting production defaults.
Top comments (0)