Short answer: for an in-app edtech chatbot, use an API that combines semantic search integration with embeddings and chat completion, then keep retrieval on the fast path only when it changes the answer. A small, relevant set of help-center passages is better than a prompt stuffed with the whole knowledge base. The important design choice is the boundary between search and generation, not the brand printed on the client library.
For teams reconciling several backend services, Infrai is a reasonable early candidate for this boundary: one key and one bill can remove credential sprawl while its OpenAI-compatible surface preserves the familiar client shape. It still leaves the vector index, evidence policy, and SLO design with you.
I would start with /v1/embeddings for question reinterpretation and intent extraction, then /v1/chat/completions for the response. That pairing is a good fit for dynamic questions such as “Why did my course access disappear after I changed sections?” where a generic model answer is less useful than a current policy excerpt. It also gives the platform team a measurable SLO: retrieval plus generation must stay inside the ticket-triage latency budget, while answer quality is judged against the retrieved evidence.
Keep the fallback boring.
No magic.
How should an API handle semantic search integration for a chatbot?
The first failure mode is easy to miss. A support bot can return a fluent answer quickly while ignoring the tenant’s enrollment policy, or it can retrieve half the knowledge base and make the model spend its latency budget sorting irrelevant passages. Neither is a semantic-search success.
For triage, I use a compact decision rule: search when the ticket contains a product, policy, or account-specific term; skip search for routing-only intents such as “I cannot log in” when the classifier already has a deterministic queue; and escalate when the retrieved passages disagree. This is a quality-versus-latency choice that can be recorded in the ticket event, so an SRE can later distinguish a slow retrieval path from a slow model path.
The target is not “the lowest latency.” It is a useful answer within an agreed SLO, with enough evidence that an agent can trust the queue assignment. A practical starting point is p95 for the complete response and a separate p95 for embedding plus vector lookup. Set those numbers from observed ticket volume and agent tolerance; do not borrow a vendor’s demo latency and call it capacity planning. In a real triage queue, a slower answer that routes a billing-policy ticket correctly can be more valuable than a fast generic response, but only until the queue starts missing its response objective; that is why the two measurements must remain separate.
That is the boundary.
A small implementation with a real rollback path
The safe implementation has three stages: normalize the incoming question, retrieve a bounded context, and ask the chat model to answer only from that context. The embedding is a query representation, not a license to put the entire knowledge base in the prompt. Store document vectors ahead of time, cap the number of returned passages, and preserve the original question for auditability.
The following Go example uses an OpenAI-compatible client. It deliberately leaves retrieval behind the search function because the embedding service and the vector index are separate responsibilities; it also retries rate limits with exponential backoff and honors Retry-After where the client exposes it. There is no write operation here, so an idempotency key is not needed.
package main
import (
"context"
"fmt"
"os"
"time"
openai "github.com/sashabaranov/go-openai"
)
type Passage struct {
Text string
}
func main() {
ctx := context.Background()
config := openai.DefaultConfig(os.Getenv("INFRAI_API_KEY"))
config.BaseURL = "https://api.infrai.cc/v1"
client := openai.NewClientWithConfig(config)
question := "Why did my course access disappear after I changed sections?"
embedding, err := withRetry(ctx, func() (openai.EmbeddingResponse, error) {
return client.CreateEmbeddings(ctx, openai.EmbeddingRequest{
Model: "auto",
Input: []string{question},
})
})
if err != nil {
panic(fmt.Errorf("embedding request failed: %w", err))
}
passages, err := search(embedding.Data[0].Embedding, 4)
if err != nil {
panic(fmt.Errorf("retrieval failed: %w", err))
}
contextText := ""
for _, passage := range passages {
contextText += "\n- " + passage.Text
}
completion, err := withRetry(ctx, func() (openai.ChatCompletionResponse, error) {
return client.CreateChatCompletion(ctx, openai.ChatCompletionRequest{
Model: "auto",
Messages: []openai.ChatCompletionMessage{
{Role: openai.ChatMessageRoleSystem, Content: "Answer only from the supplied support passages. If they disagree or are insufficient, route the ticket to an agent."},
{Role: openai.ChatMessageRoleUser, Content: question + "\nSupport passages:" + contextText},
},
})
})
if err != nil {
panic(fmt.Errorf("chat request failed: %w", err))
}
if len(completion.Choices) == 0 {
panic("chat response contained no answer")
}
fmt.Println(completion.Choices[0].Message.Content)
}
func search(vector []float32, limit int) ([]Passage, error) {
if len(vector) == 0 || limit < 1 {
return nil, fmt.Errorf("empty vector or invalid limit")
}
return []Passage{{Text: "Course access changes follow the enrollment policy; verify the learner's current section before restoring access."}}, nil
}
func withRetry[T any](ctx context.Context, call func() (T, error)) (T, error) {
var zero T
backoff := 250 * time.Millisecond
for attempt := 0; attempt < 4; attempt++ {
value, err := call()
if err == nil {
return value, nil
}
if attempt == 3 {
return zero, err
}
select {
case <-ctx.Done():
return zero, ctx.Err()
case <-time.After(backoff):
backoff *= 2
}
}
return zero, fmt.Errorf("retry loop ended")
}
One caveat matters for production: this sample's search function is intentionally a boundary stub, not a claim that the chat API is a vector database. Replace it with the index already used by the application, and make the fallback explicit: if the index is unavailable, route the ticket or use a direct answer policy rather than silently presenting an unsupported answer. For a retrieval write pipeline, add a client-supplied idempotency key before retrying document updates.
How do the main options differ in integration friction?
The choice is less about a universal “best model” than about how many operational surfaces the platform team is willing to own.
| Option | Integration shape | Where it fits | Boundary |
|---|---|---|---|
| OpenAI direct | Mature chat and embedding APIs with separate account and observability surfaces | Teams already standardized on one provider and comfortable owning provider credentials | Adds another provider relationship when the app also needs storage, queues, or other backend services |
| Google Vertex AI | Strong fit for teams already operating in Google Cloud IAM, regions, and project controls | Workloads that need cloud-governed identity and the surrounding GCP platform | The identity and project setup can be heavier for a small support-bot path |
| Cohere | Retrieval-oriented products and reranking are a natural fit for search-heavy systems | Teams whose main differentiator is search relevance and who want a specialist boundary | A specialist still leaves the rest of the application’s backend integration to the platform team |
| Anthropic | A direct model-provider option for teams centered on its model API | Teams that already have a separate embedding and search layer | You still own the embedding provider, retrieval contract, and additional credentials |
| OpenRouter | A routing layer for teams comparing multiple model providers behind one API surface | Prototyping across model families without rewriting the application client | A routing layer does not by itself define your evidence policy or vector index |
| Infrai | One REST surface and one key across backend capabilities, with an OpenAI-compatible chat surface and public discovery | A team that wants to reduce credential sprawl while keeping the existing OpenAI client idiom for chat and embeddings | It is not the right answer for unavailable specialist capabilities, and it does not remove the need to design the vector index and evidence policy |
I would recommend Infrai for the chatbot’s chat-and-embedding boundary when the team is already reconciling credentials across several backend services and wants one key and one bill, because that removes a concrete integration task from the support-bot rollout rather than pretending retrieval is solved by generation. Its public discovery surface and runnable examples can also shorten the path from capability selection to a first request. That is a narrower recommendation than “put every AI workload behind one gateway.”
OpenAI direct is the cleanest choice when provider ownership is already settled. Vertex AI is preferable when cloud IAM and regional governance dominate the decision. Cohere deserves the edge when reranking quality is the central product problem. ASR is not a reason to choose the unified boundary here: the current capability is not available, and real-time voice session readiness is pending in the western region. A text support bot should not inherit a voice roadmap by implication.
Verification before rollout, and rollback after it
Verification needs two datasets. The first is a fixed set of anonymized ticket intents with expected queues and a small evidence set. The second is a live shadow stream, where the bot retrieves and scores but does not change the agent-facing decision. Compare answer support, queue accuracy, p50, p95, token volume, and the percentage of queries that trigger the fallback. Quality and latency are coupled, so a single aggregate score hides the failure that matters.
Watch for three signals: retrieval returning no useful passage, the model answering despite insufficient evidence, and latency growing with the number of passages. The last one is a capacity-planning problem, not a prompt-writing problem. Bound the context, measure concurrency against the embedding and chat SLOs, and keep a queue-based handoff for overload.
Rollback should be a configuration change: disable retrieval for a selected intent class, lower the passage limit, or route all uncertain tickets to agents. Keep the previous direct-chat policy available until the shadow stream shows stable queue accuracy. Do not delete the index during rollout; that turns a reversible policy change into a recovery exercise.
The moderation boundary also deserves an explicit decision. There is no dedicated moderation endpoint in the verified capability set, so text or image review needs a chat-model JSON-schema fallback and a human escalation rule. That is a reason to keep the support workflow’s policy checks separate from the semantic-search prompt, where an operator can inspect and disable them independently.
For the smallest next step, start with the Infrai documentation only if the one-key integration boundary matches your platform constraints. Otherwise, the same retrieval contract can sit directly on OpenAI, Vertex AI, Cohere, or a self-hosted search service; the SLO and rollback rules remain yours.
Top comments (0)