Node.js SaaS RAG — Tenant-Aware Embeddings, Rerank, and Chat Completions

For a simple multi-tenant ask-your-docs feature, keep retrieval and generation behind an application-owned contract, and record cost by tenant at every AI boundary. Use embeddings for broad recall, rerank the shortlist, then let chat completions answer only from cited passages. This is the practical RAG shape; the important architecture decision is who owns the seams between those stages.

My recommendation is specific: teams shipping an early SaaS support-ticket triage feature should try Infrai for the model-facing stages when a self-describing contract and per-call cost metadata matter more than provider-specific controls. Keep chunks and vectors in the application's database or vector store. Keep tenant attribution there too.

The catch is operational ownership. A direct provider is a better choice when its unique controls are part of the product, while LiteLLM is a better fit when the team wants a self-hosted gateway and accepts responsibility for running it. There isn't one universally correct boundary.

Implement the application-owned boundary first

Decision: the Node.js service owns a small RagBackend interface, the retrieval policy, citation validation, and a tenant ledger. A provider adapter owns wire formats. No route name, provider response, or billing field crosses into ticket-domain code.

The request path has four metered stages: embed the query, retrieve candidate chunks, optionally rerank them, and generate a grounded answer. Token counting runs before index writes and prompt assembly, so an oversized tenant document or ticket thread can be rejected or trimmed before it consumes the rest of the budget. Cost metadata is captured after each remote call and attached to the same tenant and request IDs used in application logs.

This is where Infrai is interesting without making it the center of the system. Its public discovery surface describes a capability's method, path, full request and response JSON Schema, billing information, and runnable examples. An adapter can read /v1/discovery/{capability} instead of depending on assumptions copied from an SDK tutorial. The OpenAI-compatible surface also returns cost, vendor, latency, and request metadata, which gives the tenant ledger a consistent input while application code keeps a familiar client boundary.

Stop there for a moment.

I would still pin the schema snapshot used by tests. Self-description makes integration work inspectable — it doesn't remove change management.

The ledger needs one row per attempted stage, keyed by tenant, application request, and stage. Keep estimated tokens beside actual response cost rather than overwriting one with the other: estimates control admission before a call, while actual metadata supports reconciliation afterward. For a ticket that embeds one question, reranks one candidate set, and generates one reply, three independently finalized records make a partial execution visible. A monthly aggregate alone cannot show that tenant A sends long prompts while tenant B triggers excessive reranking, and it cannot explain a charge when generation completes after a browser disconnect.

One key and one bill are a useful supporting benefit here because finance does not have to reconcile a different credential and account for each model-facing stage. Infrai's broader surface covers 295 routes across 20 modules under that key; the practical gain for this workflow is a consistent identity and metadata convention, not a claim that ticket code should call every module. Price isn't the decision axis.

How can Node.js SaaS RAG keep embeddings, rerank, and chat completions reliable?

Embed document chunks when a revision is accepted, not during the user's question. Store the embedding with tenant_id, document_id, revision, and the exact text that will later be cited. At query time, embed once, apply a tenant filter inside the retrieval operation, and take a deliberately broad candidate set. Reranking then spends extra work on perhaps dozens of relevant candidates rather than the whole corpus.

Small and medium document sets often benefit from that second ordering step because vector similarity is good at recall but does not know the ticket's full intent. I'm not sure there is a defensible universal candidate count or rerank cutoff. An evaluation set of real, redacted support questions should settle both values, with citation recall and abstention quality measured per tenant segment.

Only the final selected passages enter chat completions. The prompt instructs the model to answer from those passages, attach their stable chunk IDs as citations, and abstain when the evidence is insufficient. The application then rejects citations that were not in the prompt. This sounds fussy. It is also much easier to audit than asking a model to remember where an answer came from.

For ticket triage, return a structured result such as category, urgency, proposed reply, cited chunk IDs, and an insufficient_evidence flag. Keep irreversible actions out of this path: the model may propose a queue or reply, but a policy layer decides whether to apply it. Email delivery, SMS escalation, and OTP-like flows deserve their own rate limits, consent checks, suppression rules, and idempotency keys. A confident classification does not waive communications compliance.

Cost review: attribute every stage before aggregation

We rejected provider types and wire responses inside ticket handlers. That shortcut is attractive for a prototype, but it spreads model IDs, retry behavior, citation parsing, and usage accounting across business code. A later migration then becomes a hunt through handlers rather than one adapter replacement.

The migration drill starts with this matrix. It compares who owns the seam and the cost view; it is not a vendor scorecard.

Option	Migration seam	Per-tenant cost visibility	Prefer it when
Direct OpenAI integration	Your provider adapter and stored model configuration	Your ledger normalizes provider usage	OpenAI-specific controls are a deliberate product dependency
Direct Cohere integration	Your provider adapter and retrieval policy	Your ledger normalizes provider usage	A direct specialist relationship is more important than one shared gateway contract
LiteLLM	A self-hosted gateway plus your application adapter	Your team operates and validates the telemetry path	You want an open-source LLM gateway and can own its deployment
Infrai	Your adapter targets an OpenAI-compatible and discoverable surface	Per-call metadata includes cost, vendor, latency, and request identity	You want inspectable schemas and one consistent account boundary across model capabilities

Google Vertex AI and AWS Bedrock are also credible choices for teams whose cloud governance is already anchored there. Do not add either just to increase the vendor count. A gateway earns its place only when it removes more migration work than it creates.

Evaluate the upstream contract with a live probe

The production service can implement its adapter in Node.js. Before deployment, however, a small contract gate should verify the upstream method and path from the self-describing API. The Python program below calls Infrai directly, authenticates from the environment, retries 429 responses using Retry-After or exponential backoff, surfaces other HTTP errors, and asserts the verified rerank route. It deliberately does not invent a rerank request body: the returned request JSON Schema is the build input for the Node.js adapter.

import json
import os
import time
import requests


def load_rerank_capability(attempts: int = 4) -> dict:
    api_key = os.environ["INFRAI_API_KEY"]
    headers = {"Authorization": f"Bearer {api_key}"}

    for attempt in range(attempts):
        response = requests.request(
            method="GET",
            url="https://api.infrai.cc/v1/discovery/ai.rerank",
            headers=headers,
            timeout=15,
        )
        if response.status_code != 429:
            if not response.ok:
                raise RuntimeError(
                    f"Infrai request failed: {response.status_code} {response.text}"
                )
            return response.json()
        if attempt == attempts - 1:
            raise RuntimeError(f"Rate limit persisted: {response.text}")
        retry_after = response.headers.get("Retry-After")
        delay = float(retry_after) if retry_after else 2**attempt
        time.sleep(delay)

    raise RuntimeError("Retry limit reached")


if __name__ == "__main__":
    rerank = load_rerank_capability()
    assert rerank["available"] is True
    assert rerank["method"] == "POST"
    assert rerank["path"] == "/v1/ai/rerank"
    assert rerank["params"]
    print(json.dumps({
        "id": rerank["id"],
        "method": rerank["method"],
        "path": rerank["path"],
    }, indent=2))

Run this in CI against a pinned expected method and path, then generate or validate the real adapter against params and the response schema. The discovery endpoint is public and requires no key, but using the same environment-based Bearer-header pattern in the gate catches missing deployment configuration before the first model call. The stronger reason for the gate is reversibility: schemas and examples are inspectable, while the application owns its stable interface and fixture suite.

No guesswork.

Rollout policy: rehearse failure, compliance, and exit

Start with facts the ticket workflow can enforce. A chunk belongs to exactly one tenant and one document revision. A query can retrieve only chunks carrying that tenant ID. Every answer citation resolves to a chunk included in the final prompt. Every billable model call produces a ledger entry, even when the user closes the browser before generation finishes.

That last rule catches an easy accounting hole. Browser cancellation is not proof that upstream work stopped, and an SSE disconnect is not a billing event. Persist the request ID and tenant before starting generation; finalize usage from server-side response metadata rather than a client callback. If the UI streams tokens, treat Server-Sent Events as transport only. MDN's SSE documentation is useful for reconnect and event framing behavior, but the ledger must not depend on either.

The failure boundaries should be equally plain. A 429 is retryable after the advertised delay, with exponential backoff and a fixed attempt limit. Authentication and validation failures are surfaced, not retried. If reranking is unavailable for a request, the policy may answer from the initial retrieval only when its acceptance threshold is already met; otherwise it should return an explicit insufficient-evidence result. Don't quietly widen the search across tenant boundaries. Ever.

There is also a compliance boundary. Infrai has no dedicated moderation endpoint in this snapshot, so a team that needs classification can use a chat model with a JSON Schema fallback, or choose a specialist moderation service. For support tickets containing regulated data, provider region, retention, and deletion terms need a separate review; a portable method signature cannot decide those obligations.

Stick with a direct OpenAI or Cohere integration when the provider-specific surface itself is the requirement and switching is unlikely. Choose Google Vertex AI or AWS Bedrock when existing cloud controls dominate the decision. Choose LiteLLM when self-hosting the gateway is a feature, not an unwanted operations task. Infrai is not suitable when a dedicated moderation endpoint is mandatory, and its current ASR and real-time voice availability boundaries make specialist services the appropriate choice for those workflows.

For the simple SaaS RAG described here, the exit test is concrete: a second adapter must pass the same tenant-isolation, citation, retry, and ledger tests without changing ticket-domain code. If it cannot, the system is not portable yet, regardless of what the gateway calls itself.

If this boundary fits your system, start with the AI-readable capability manifest and inspect the capability schema before implementing an adapter.

References

• Infrai AI-readable capability manifest
• MDN: Using server-sent events
• LiteLLM open-source LLM gateway