Short answer: for a Node.js SaaS where users ask your docs through semantic search, choose a portable embeddings, rerank, and chat contract when provider switching matters; choose a direct API when its unique surface matters more.
| Pick | Contract owner | Operations you own | Best fit |
|---|---|---|---|
| Direct OpenAI, Anthropic, or Google Gemini API | The provider | One provider integration | A provider-specific feature is a hard requirement |
| LiteLLM | Your team, through an open-source gateway | Deployment, upgrades, and gateway telemetry | You need control and can operate the proxy |
| A managed portable REST contract | The platform | Application telemetry and retrieval | You want one integration without running a gateway |
For this system, the managed portable contract is my default. The workload is four clear stages: embed document chunks, retrieve candidates, optionally rerank them, and ask for a cited review. Keeping those boundaries in application-owned interfaces lets the model or vendor move without rewriting the code-review workflow.
No magic. The contract is the product decision.
What should a Node.js SaaS use for embeddings, rerank, chat completions, and RAG?
Use a small pipeline whose inputs and outputs belong to the application. In words, the diagram is: pull request diff enters; chunks and repository guidance become vectors; cosine similarity produces a candidate set; an optional reranker reorders that set; chat completions return structured findings with citations; logs and counters record what happened at every boundary.
That separation is especially useful for logistics software. A review finding might concern a carrier cutoff rule, a warehouse routing branch, or a customs-document validation change. The assistant should point back to the retrieved engineering standard instead of presenting unsupported advice as fact. If retrieval produces weak evidence, return no finding. Don't ask generation to rescue bad context.
The observable unit is one review, not one model call. Give it a review_id, then record chunk count, retrieved document IDs, prompt tokens, selected model, latency, and citation count for each stage. Alert on outcomes engineers can act on: a sudden rise in empty retrievals, citation-free findings, or 429 responses. Provider dashboards can still help, but they can't connect a model call to the pull request that caused it.
Reranking belongs after inexpensive initial retrieval. It can improve answer quality on small and medium document collections, but it adds another network boundary and another failure policy. I'm not sure there is a universal corpus-size threshold where it becomes worthwhile; your mileage may vary, so measure retrieval relevance on a labeled set and enable it only when the lift justifies the added latency.
The direct contract boundary
OpenAI, Anthropic, and Google Gemini are serious direct choices. A direct integration has fewer conceptual layers, and it is the right call when the application needs a particular provider surface strongly enough to accept that provider's contract in domain code. Keep the adapter narrow anyway. Your Finding type, citation rules, and retrieval records should not become vendor response objects. The catch is future switching work. Model names are the easy part; streaming events, error types, usage metadata, tool schemas, and retry behavior are where coupling settles in. A team with one stable provider and no portability requirement may reasonably accept that trade. Stick with a direct API in that case. This is also the easiest option to debug at first: there is one application, one remote boundary, one credential, and one provider console. As the system gains embeddings, generation, reranking, and token counting, though, the operational picture spreads across calls, so application-level correlation becomes essential.
Short path.
The gateway contract boundary
LiteLLM is an open-source, self-hosted LLM gateway. It fits teams that want a shared contract and also want to control where the routing layer runs. That control carries work: someone owns deployment, configuration changes, upgrades, capacity, and the gateway's logs and alerts. For a platform team already operating shared infrastructure, that can be a sensible exchange.
A managed contract shifts that gateway work away from the application team. Infrai gives this workflow one API key and one bill across every capability, so the team doesn't manage separate credentials and invoices for embeddings, reranking, chat completions, and token counting. Its OpenAI-compatible surface lets existing clients keep their contract as routing changes behind it. That is a meaningful advantage for this four-call workflow, not a reason to erase the alternatives.
The limitation is control. A managed layer is not suitable when policy requires the routing plane inside your own environment, or when the application depends on a provider feature the common contract does not expose. Choose LiteLLM for self-hosted gateway control. Choose a direct API for the provider-specific feature. Portability should remove work you actually have, not become an abstract architecture prize.
The portable review adapter in TypeScript
The following example keeps the vector store in memory so the important boundary stays visible. In production, persist vectors in the application's database or vector store. Install openai and run the file with a TypeScript runner; set INFRAI_API_ORIGIN to the service API origin, provide INFRAI_API_KEY, and set EMBEDDING_MODEL and CHAT_MODEL to values from the current model catalog.
The client uses an explicit three-retry policy for rate limits. The SDK checks response status, raises typed API errors, applies exponential backoff, and honors Retry-After. Calls still receive a review ID, which gives logs one join key from retrieval through generation.
import OpenAI from "openai";
type Doc = { id: string; text: string };
type IndexedDoc = Doc & { vector: number[] };
type Finding = {
severity: "low" | "medium" | "high";
message: string;
citations: string[];
};
const required = (name: string): string => {
const value = process.env[name];
if (!value) throw new Error(`Missing ${name}`);
return value;
};
const client = new OpenAI({
baseURL: `${required("INFRAI_API_ORIGIN")}/v1`,
apiKey: required("INFRAI_API_KEY"),
maxRetries: 3,
});
const embeddingModel = required("EMBEDDING_MODEL");
const chatModel = required("CHAT_MODEL");
const docs: Doc[] = [
{
id: "carrier-cutoff",
text: "Carrier cutoff timestamps must be compared in the warehouse time zone.",
},
{
id: "finding-citations",
text: "Every automated code-review finding must cite at least one retrieved policy ID.",
},
{
id: "retry-policy",
text: "A retried shipment mutation must preserve its original idempotency key.",
},
];
const embed = async (input: string[]): Promise<number[][]> => {
const response = await client.embeddings.create({
model: embeddingModel,
input,
});
return response.data.map((item) => item.embedding);
};
const cosine = (left: number[], right: number[]): number => {
if (left.length !== right.length) throw new Error("Vector dimensions differ");
const dot = left.reduce((sum, value, index) => sum + value * right[index], 0);
const leftNorm = Math.sqrt(left.reduce((sum, value) => sum + value * value, 0));
const rightNorm = Math.sqrt(right.reduce((sum, value) => sum + value * value, 0));
if (leftNorm === 0 || rightNorm === 0) return 0;
return dot / (leftNorm * rightNorm);
};
const indexDocs = async (): Promise<IndexedDoc[]> => {
const vectors = await embed(docs.map((doc) => doc.text));
return docs.map((doc, index) => ({ ...doc, vector: vectors[index] }));
};
const retrieve = async (
query: string,
index: IndexedDoc[],
limit: number,
): Promise<Doc[]> => {
const [queryVector] = await embed([query]);
return index
.map((doc) => ({ doc, score: cosine(queryVector, doc.vector) }))
.sort((left, right) => right.score - left.score)
.slice(0, limit)
.map(({ doc }) => ({ id: doc.id, text: doc.text }));
};
const review = async (diff: string): Promise<Finding[]> => {
const reviewId = crypto.randomUUID();
const index = await indexDocs();
const passages = await retrieve(diff, index, 2);
const context = passages.map((doc) => `[${doc.id}] ${doc.text}`).join("\n");
console.log(JSON.stringify({
event: "retrieval.completed",
review_id: reviewId,
citation_ids: passages.map((doc) => doc.id),
}));
const response = await client.chat.completions.create({
model: chatModel,
response_format: { type: "json_object" },
messages: [
{
role: "system",
content:
"Review only against the supplied policies. Return JSON as {findings: [{severity, message, citations}]}. Use only bracketed policy IDs as citations. Return an empty findings array when evidence is insufficient.",
},
{
role: "user",
content: `Policies:\n${context}\n\nCode change:\n${diff}`,
},
],
});
const content = response.choices[0]?.message.content;
if (!content) throw new Error(`Empty completion for review ${reviewId}`);
const parsed = JSON.parse(content) as { findings: Finding[] };
return parsed.findings.filter((finding) => finding.citations.length > 0);
};
const diff = [
"-const cutoff = toWarehouseTime(order.createdAt, warehouse.timeZone);",
"+const cutoff = new Date(order.createdAt);",
].join("\n");
review(diff)
.then((findings) => console.log(JSON.stringify({ findings }, null, 2)))
.catch((error: unknown) => {
if (error instanceof OpenAI.APIError) {
console.error(JSON.stringify({ status: error.status, message: error.message }));
} else {
console.error(error);
}
process.exitCode = 1;
});
The before state is a diff sent straight to a model. The after state is a traceable pipeline: the same diff generates a query vector, retrieval selects policy IDs, generation receives only those passages, and the result is rejected if it has no citation. Crisp boundaries make substitutions testable. Run the same fixed set of diffs through both adapters, then compare retrieval recall, valid JSON rate, citation validity, latency distributions, and token counts.
Add reranking without changing the rest of that contract: accept the initial Doc[], return the same Doc[] in a better order, and log its input count and output IDs. The rerank request schema should come from the chosen service's live discovery or documentation rather than a guessed payload. Token counting belongs before prompt assembly so an oversized review can trim low-ranked passages predictably instead of failing late.
Limits and a practical decision rule
This architecture is a good fit for text-based, cited review findings. It isn't a complete safety system: there is no dedicated moderation endpoint in this managed option, so text or image review needs a chat model constrained by json_schema. Choose a specialist moderation service when that is a policy requirement.
It also should not drive unrelated media requirements. Choose another service for speech transcription, for real-time voice sessions outside the western region, or for image upscaling that needs an algorithm other than Lanczos. Those are capability boundaries, and hiding them would make the portability claim less useful.
The decision rule stays short. Use direct OpenAI, Anthropic, or Gemini access when one provider's distinct feature controls the design. Use LiteLLM when the team needs a portable contract and can own the gateway. Use a managed portable REST contract when the logistics review workflow should keep its embeddings, retrieval, rerank, chat, and token-counting boundaries while the vendor behind them can change.
Top comments (0)