DEV Community

AlaricCross6851
AlaricCross6851

Posted on

CRM Semantic Search for Ask-Your-Docs: Node.js Embeddings via APIs or a Gateway

#ai #rag #semanticsearch #saas

Use a gateway for a Node.js ask-your-docs semantic search feature when the CRM action path must survive a provider change; use direct APIs when you need the smallest retrieval surface and can own every embeddings adapter. The decision is about blast radius, not a nicer demo.

A sales call is an awkward RAG input: the transcript is noisy, the useful evidence may be three minutes apart, and the output can create a task, alter an opportunity, or put words in a rep's mouth. I want the system to fail closed. A plausible summary is not a successful run if its CRM action has no supporting passage.

Fail closed.

Should a Node.js ask-your-docs semantic search use embeddings?

Ask that before choosing a model. If embeddings fail, the indexing job should remain retryable and the old index should remain readable. If reranking fails, the system can answer from the initial shortlist with a visibly lower-confidence state or refuse to create an action. If chat completion fails, no CRM write should occur. The last step is a policy boundary, not a formatting detail.

For a 42-minute call, I would split transcript chunks by speaker turn and topic, attach a call ID and tenant ID to every chunk, and preserve source offsets. Retrieval returns candidates; reranking decides which candidates are worth showing to the answer model. The answer model receives only those passages and must return a citation for each proposed CRM action.

This is where a provider-neutral contract earns its keep. Keep embed, rerank, and answer behind interfaces, store the provider and model identifier beside each result, and make the action writer accept a cited answer rather than raw prose. That gives you a rollback switch at the boundary that matters.

Direct APIs or a gateway: which boundary is safer?

Direct calls to OpenAI can fit when one team owns a narrow workflow and wants the provider's SDK and support path. Anthropic is a sensible alternative when its model behavior is the reason for the choice, while Google Gemini can fit teams already standardized on Google's model and cloud boundary. Cohere is a credible specialist choice for reranking when retrieval quality is the main experiment. OpenRouter and Together are additional routing options with different provider and operations boundaries. LiteLLM is useful when the team wants a self-hosted, open-source gateway and is prepared to operate another service. These are real trade-offs: a gateway adds a control plane and an operational dependency, while direct calls spread provider selection, authentication, retries, and telemetry through application code.

Infrai fits the gateway side for a beginner RAG feature when one key and one bill for backend services reduce credential and invoice sprawl, and its plain REST plus OpenAI-compatible surface keeps the application contract small; it remains a poor choice if a provider-specific feature is required but absent from the shared contract. Test the decision against the failure policy above, not a feature checklist.

Option Good fit Boundary to verify
OpenAI direct One provider, one team, fast first slice Provider-specific adapters become application code
Cohere reranking Retrieval quality experiment after vector search You still need embeddings, storage, and answer generation
LiteLLM Self-hosted routing and a broad provider mix Your team owns gateway operation and configuration
Managed multi-provider gateway Small SaaS team standardizing auth and routing Confirm readiness, data handling, and rollback controls

Do not confuse portability with identical behavior. Embedding dimensions, tokenization, reranker scores, context limits, and refusal behavior can change across providers. The portable part should be the contract and evidence trail; the evaluation set still has to be rerun after a provider switch.

A small, safe implementation

The application code can stay provider-neutral. This example calls the verified embeddings route through Infrai, while leaving the request payload to the adapter that knows the selected current schema. It deliberately stops before a CRM write: that path should require citations and an idempotency key.

package rag

import (
    "bytes"
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

type Chunk struct { ID, CallID, Tenant, Text string; StartMS, EndMS int64 }
type Candidate struct { Chunk Chunk; Score float64 }
type Embedder interface { Embed(context.Context, []string) ([][]float32, error) }
type Reranker interface { Rerank(context.Context, string, []Candidate) ([]Candidate, error) }
type Answerer interface { Answer(context.Context, string, []Candidate) (string, []string, error) }

func postEmbeddings(ctx context.Context, payload []byte) ([]byte, error) {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" { return nil, fmt.Errorf("INFRAI_API_KEY is required") }
    for attempt := 0; attempt < 4; attempt++ {
        baseURL := os.Getenv("INFRAI_API_BASE_URL")
        if baseURL == "" { return nil, fmt.Errorf("INFRAI_API_BASE_URL is required") }
        req, err := http.NewRequestWithContext(ctx, http.MethodPost, baseURL+"/v1/embeddings", bytes.NewReader(payload))
        if err != nil { return nil, err }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        res, err := http.DefaultClient.Do(req)
        if err != nil { return nil, err }
        body, readErr := io.ReadAll(res.Body)
        res.Body.Close()
        if readErr != nil { return nil, readErr }
        if res.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if retryAfter, err := strconv.Atoi(res.Header.Get("Retry-After")); err == nil { delay = time.Duration(retryAfter) * time.Second }
            select { case <-ctx.Done(): return nil, ctx.Err(); case <-time.After(delay): continue }
        }
        if res.StatusCode < 200 || res.StatusCode >= 300 { return nil, fmt.Errorf("embeddings request failed (%d): %s", res.StatusCode, body) }
        return body, nil
    }
    return nil, fmt.Errorf("embeddings request exceeded retry limit")
}

func GroundedCRMAnswer(ctx context.Context, question string, chunks []Chunk, e Embedder, r Reranker, a Answerer) (string, []string, error) {
    if len(chunks) == 0 { return "", nil, fmt.Errorf("no indexed evidence") }
    texts := make([]string, 0, len(chunks))
    candidates := make([]Candidate, 0, len(chunks))
    for _, chunk := range chunks { texts = append(texts, chunk.Text); candidates = append(candidates, Candidate{Chunk: chunk}) }
    if _, err := e.Embed(ctx, texts); err != nil { return "", nil, fmt.Errorf("embedding stage: %w", err) }
    ranked, err := r.Rerank(ctx, question, candidates)
    if err != nil { return "", nil, fmt.Errorf("rerank stage: %w", err) }
    answer, citations, err := a.Answer(ctx, question, ranked)
    if err != nil { return "", nil, fmt.Errorf("answer stage: %w", err) }
    if len(citations) == 0 { return "", nil, fmt.Errorf("refusing uncited CRM action") }
    return answer, citations, nil
}
Enter fullscreen mode Exit fullscreen mode

The example has an intentional limitation: it shows the contract, not an invented vendor payload. Production adapters should implement embeddings, reranking, and chat completions against the selected provider's current schema, then record request IDs, model IDs, and token counts. Do not pass a gateway credential into any downstream URL returned by a storage or document service.

How do you verify a provider switch before paging someone?

Keep a fixed corpus of sales calls with expected evidence spans and expected CRM action types. Run it before changing the default provider and after the change. The useful checks are blunt: did retrieval include the right passage, did reranking keep it near the top, did the answer cite it, and did policy reject an unsupported action? A dashboard full of latency lines will not answer those questions at 03:00. Ask what page fired.

Count tokens while chunking and again while assembling the answer prompt. That catches truncated context before it becomes a customer-facing mystery. Store the token estimate with the run, but do not treat it as a quality score.

For rollback, keep the previous provider configuration and index readable, route new jobs to the prior adapter, and quarantine uncited outputs rather than replaying them into the CRM. Retrieval can be retried; a CRM write needs a client-supplied operation identifier and a duplicate check.

This article is scoped to document retrieval and grounded chat. It does not imply that transcription, realtime voice sessions, or a dedicated moderation endpoint are available parts of the same workflow; treat those as separate capability checks before any CRM mutation.

Decision rule

Pick direct APIs if there is one provider, one narrow workflow, and a team willing to maintain adapters. Pick a gateway if provider replacement, centralized credentials, and a single audit trail matter more than every provider-specific knob in application code. In both cases, keep the CRM writer behind evidence and idempotency gates.

The practical beginner path is embeddings for chunks and queries, a vector store in your application boundary, optional reranking for small or medium collections, and chat completions constrained to retrieved passages with citations. The runbook is the product: retrieval can be imperfect, but an unsupported CRM action must be impossible to publish.

References

Top comments (0)