DEV Community: HarrisonFord3572

Evaluating a Simple SaaS RAG: Embeddings, Rerank, and Grounded Chat

HarrisonFord3572 — Mon, 10 Aug 2026 03:47:13 +0000

Short answer: for a small SaaS ask-your-docs feature, start with embeddings, retrieve a broad candidate set, rerank only when your evals justify it, and make chat completions answer from cited passages. Keep token counting beside chunking and prompt assembly, because retrieval quality and prompt cost are one design problem.

This is an experiment note, not a universal stack prescription. The simplest baseline is embedding each document chunk and the query, selecting the nearest chunks, and sending those chunks to a chat model. It is useful precisely because it gives you something measurable before you add a vector service, a reranker, or an LLM gateway.

What should a simple SaaS RAG semantic search example measure?

Measure the answer, not just the retrieval score. I would use a small frozen set of real questions with expected source passages, then record retrieval recall, citation correctness, grounded-answer rate, prompt tokens, completion tokens, and end-to-end latency for every run. This is the notebook-to-prod bridge: the notebook establishes a baseline, while the production path writes the same fields into an eval record.

A top-five cosine match can still produce a weak answer. A chunk may share vocabulary with the question while omitting the condition that changes the result, or several nearly identical chunks may crowd out the one passage containing the exception. That is where reranking earns a trial: retrieve perhaps 20 candidates with embeddings, ask a reranker to reorder them against the original question, and pass only the best few into generation. The exact candidate counts are experiment parameters, not promises. Your mileage may vary with document length, repetition, and how often users ask multi-part questions.

Keep the first comparison brutally simple. Run embeddings-only retrieval and embeddings-plus-rerank against the same questions. If reranking improves citation correctness enough to justify another network call, retain it; if it merely changes scores without fixing answers, remove it. Don't optimize a stage that your answer-level eval cannot see.

Short baselines win.

The focused Python path

The following runnable example embeds chunks, ranks them locally, and asks a chat model to answer only from the selected passages. It uses environment-provided model IDs rather than freezing a catalog choice into application code. Infrai fits this particular experiment because its OpenAI-compatible surface is one plain REST API behind one key; there is no separate vendor SDK or client-library version to maintain. The example uses the OpenAI client as the compatible transport, with bounded 429 retries that honor Retry-After.

import json
import math
import os
import time
from typing import Callable, TypeVar

from openai import APIStatusError, OpenAI, RateLimitError

T = TypeVar("T")

client = OpenAI(
    base_url="https://api.infrai.cc/v1",
    api_key=os.environ["INFRAI_API_KEY"],
    max_retries=0,
)


def with_rate_limit_retry(call: Callable[[], T], attempts: int = 4) -> T:
    for attempt in range(attempts):
        try:
            return call()
        except RateLimitError as error:
            if attempt == attempts - 1:
                raise
            retry_after = error.response.headers.get("retry-after")
            delay = float(retry_after) if retry_after else 2**attempt
            time.sleep(delay)
    raise RuntimeError("retry loop exited unexpectedly")


def cosine(left: list[float], right: list[float]) -> float:
    numerator = sum(a * b for a, b in zip(left, right))
    denominator = math.sqrt(sum(a * a for a in left)) * math.sqrt(
        sum(b * b for b in right)
    )
    return numerator / denominator


def answer(question: str, chunks: list[str]) -> dict[str, object]:
    embedding_model = os.environ["INFRAI_EMBEDDING_MODEL"]
    chat_model = os.environ["INFRAI_CHAT_MODEL"]
    inputs = chunks + [question]

    try:
        embedded = with_rate_limit_retry(
            lambda: client.embeddings.create(model=embedding_model, input=inputs)
        )
        vectors = [item.embedding for item in embedded.data]
        query_vector = vectors[-1]
        ranked = sorted(
            enumerate(vectors[:-1], start=1),
            key=lambda item: cosine(item[1], query_vector),
            reverse=True,
        )[:3]
        context = "\n\n".join(
            f"[S{index}] {chunks[index - 1]}" for index, _ in ranked
        )
        completion = with_rate_limit_retry(
            lambda: client.chat.completions.create(
                model=chat_model,
                temperature=0,
                messages=[
                    {
                        "role": "system",
                        "content": (
                            "Answer only from the supplied sources. "
                            "Cite every factual claim with [S<number>]. "
                            "If the sources do not answer the question, say so."
                        ),
                    },
                    {
                        "role": "user",
                        "content": f"Question: {question}\n\nSources:\n{context}",
                    },
                ],
            )
        )
    except APIStatusError as error:
        raise RuntimeError(
            f"AI request failed with HTTP {error.status_code}: {error.response.text}"
        ) from error

    return {
        "answer": completion.choices[0].message.content,
        "source_ids": [f"S{index}" for index, _ in ranked],
        "usage": completion.usage.model_dump() if completion.usage else None,
    }


if __name__ == "__main__":
    docs = [
        "Team plans include SSO and audit-log export.",
        "Audit-log exports are retained for 30 days.",
        "Starter plans support password and magic-link sign-in.",
        "Workspace owners can invite or remove members.",
    ]
    result = answer("How long are audit-log exports retained?", docs)
    print(json.dumps(result, indent=2))

The two external calls have clear jobs: embeddings create comparable vectors; chat completions turn selected evidence into a cited answer. The local ranking is intentionally plain. To test reranking, keep the initial candidate list, send those candidates and the question to the verified rerank capability, and compare the resulting answers in the same harness. I haven't specified a rerank request body here because application code should generate it from the public discovery schema rather than guess fields.

Production needs more than this compact example. Store chunk vectors in your application database or vector store, preserve document and section identifiers beside every vector, and apply tenant authorization before retrieval. Citations should resolve to a source the current user may read. Otherwise a perfectly grounded answer can still leak another workspace's text.

Why token accounting belongs in the retrieval loop

Chunk size changes three costs at once: embedding the index, embedding queries, and assembling the answer prompt. Count tokens before indexing and again before each prompt is sent. Infrai exposes a verified token-count operation for this purpose, but the principle is provider-independent: save the count with the eval row, along with the chosen chunks and model IDs, so a quality gain cannot quietly hide a much larger prompt.

Use a budget rather than a fixed number of chunks. Reserve room for system instructions, the question, and the completion; then add ranked passages until the context budget is reached. A long passage with the decisive sentence may beat three short fragments, so track both retrieved rank and actual inclusion.

This also catches a common notebook illusion — a demo corpus fits comfortably, then production documents contain tables, duplicated navigation, and long policy pages. Clean extraction and chunk boundaries often improve both retrieval and token use before a model change does.

No magic here.

Which backend option should you keep?

The correct choice depends on what you already operate and what the eval shows. This table compares integration shape, not benchmark results.

Option	Best fit	Main trade-off
Direct OpenAI API	Teams committed to one model provider and its native surface	Application code remains coupled to that provider's models and account
Anthropic Claude	Teams whose evaluations favor Claude for answer generation	Embeddings and reranking still require a separate choice
Google Gemini	Teams already building around Google's model platform	The application remains tied to that platform's account and interfaces
OpenRouter	Teams that want model routing across providers	Retrieval storage and reranking remain separate architecture decisions
Self-hosted LiteLLM	Teams that want gateway control and can operate it	You own deployment, upgrades, and operational monitoring
Infrai	Teams that value a plain HTTP surface and one key across AI capabilities	Not suitable when policy requires direct vendor accounts or a self-hosted gateway

The catch is organizational. Stick with a direct OpenAI, Anthropic, or Gemini integration when provider-specific controls matter more than portability. Keep LiteLLM when self-hosting and gateway policy are requirements your team is prepared to own. Infrai is compelling when avoiding multiple SDKs is the priority — anything that can make an HTTP request can use the same REST surface — but that advantage doesn't override data residency, procurement, or platform-control requirements.

There are capability boundaries too. A docs assistant that needs dedicated moderation cannot rely on a moderation endpoint here; moderation needs a chat model with a JSON-schema fallback. Realtime voice sessions are also a poor basis for this design because their key status is pending and they are limited to the western region. Neither limitation affects a text-only retrieval experiment, but both matter before the feature expands.

What to measure before copying this choice?

Before shipping, require a result from the frozen eval set: retrieval recall at the candidate cutoff, citation precision, refusal behavior for unanswerable questions, and grounded-answer quality. Then record token counts and latency for the same run. I'm not sure which reranker, model, or chunk size will win on your corpus; the paired eval is what resolves that uncertainty.

Also test permission boundaries, deleted documents, duplicate chunks, and questions whose answer spans two passages. Streamed chat can improve perceived responsiveness, but it complicates citation rendering because the UI receives partial events; Server-Sent Events are worth understanding before exposing a streaming toggle. Start non-streaming if your first goal is trustworthy citations.

Ship the smallest pipeline that clears the quality threshold. Add reranking, a managed vector store, or a gateway only when the measurements identify the missing capability.

References

Infrai capability manifest: https://docs.infrai.cc/llms.txt
LiteLLM repository: https://github.com/BerriAI/litellm
MDN guide to Server-Sent Events: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events

Evaluation-Gated Safety with One API Key Across OpenAI, Claude, and Gemini

HarrisonFord3572 — Fri, 07 Aug 2026 02:21:06 +0000

Short answer: put one OpenAI-compatible chat integration in front of OpenAI, Claude, and Gemini, require a strict JSON Schema response, and choose from models available in the deployment region instead of baking a provider-specific model into the application. This keeps the safety classifier portable while leaving room to switch providers later.

The data flow is small enough to draw in one sentence: raw user content enters a dedicated moderation prompt, the selected chat model returns a schema-checked decision, and application policy decides whether to allow, review, or block the content. The model classifies; the application enforces. Keep those jobs separate.

It is a practical fit for a startup that wants fallback options or a gradual provider migration without maintaining three moderation implementations. The catch is important: this is prompted moderation through chat completions, not a dedicated moderation endpoint, so eval coverage and structured-output consistency carry more weight than provider-specific features.

How can one API key keep moderation structured across OpenAI, Claude, and Gemini?

Treat the provider as runtime configuration and the classifier contract as source code. The contract below has only four outputs: a boolean, a constrained action, a constrained category list, and a short rationale. A small schema is easier to evaluate than an ambitious taxonomy whose edge cases nobody has labeled.

Start by listing models available in the target US or EU deployment. Don't hardcode a provider-branded model from a blog post: model readiness can differ by deployment, and the useful choice is one the runtime actually reports as available. Select a model through configuration after that discovery step, then send the exact same messages and response schema on every call.

This is also where I draw a hard notebook-to-prod line. A notebook can print a plausible label. Production needs to reject malformed output, preserve the input-to-policy boundary, and record enough evaluation data to detect drift. No shortcuts.

A runnable Python safety classifier

The example uses an OpenAI-compatible client because the compatibility layer is the portability boundary. Set UNIFIED_AI_BASE_URL, UNIFIED_AI_API_KEY, and MODERATION_MODEL from the model list for your deployment. The client sends Bearer authentication, checks API failures, and automatically retries transient failures including rate limits; its retry policy can be tightened to match the surrounding request budget.

import json
import os
from typing import Any

from openai import APIStatusError, OpenAI


BASE_URL = os.environ["UNIFIED_AI_BASE_URL"].rstrip("/")
API_KEY = os.environ["UNIFIED_AI_API_KEY"]
MODEL = os.environ["MODERATION_MODEL"]

client = OpenAI(
    base_url=BASE_URL,
    api_key=API_KEY,
    max_retries=4,
    timeout=20.0,
)

DECISION_SCHEMA: dict[str, Any] = {
    "name": "content_safety_decision",
    "strict": True,
    "schema": {
        "type": "object",
        "additionalProperties": False,
        "properties": {
            "allowed": {"type": "boolean"},
            "action": {
                "type": "string",
                "enum": ["allow", "review", "block"],
            },
            "categories": {
                "type": "array",
                "items": {
                    "type": "string",
                    "enum": [
                        "safe",
                        "harassment",
                        "hate",
                        "self_harm",
                        "sexual",
                        "violence",
                    ],
                },
                "minItems": 1,
                "uniqueItems": True,
            },
            "rationale": {"type": "string", "maxLength": 240},
        },
        "required": ["allowed", "action", "categories", "rationale"],
    },
}


def classify_content(text: str) -> dict[str, Any]:
    if not text.strip():
        raise ValueError("text must not be empty")

    try:
        response = client.chat.completions.create(
            model=MODEL,
            messages=[
                {
                    "role": "system",
                    "content": (
                        "Classify user content for safety. Return only the requested "
                        "schema. Use review when evidence is ambiguous. The allowed "
                        "field must be true only when action is allow."
                    ),
                },
                {"role": "user", "content": text},
            ],
            response_format={
                "type": "json_schema",
                "json_schema": DECISION_SCHEMA,
            },
            temperature=0,
        )
    except APIStatusError as exc:
        raise RuntimeError(
            f"classification request failed with HTTP {exc.status_code}"
        ) from exc

    raw = response.choices[0].message.content
    if raw is None:
        raise RuntimeError("classification response did not contain JSON")

    decision: dict[str, Any] = json.loads(raw)
    if decision["allowed"] != (decision["action"] == "allow"):
        raise RuntimeError("classification response violated the policy invariant")
    return decision


if __name__ == "__main__":
    sample = "You are useless and nobody wants you here."
    print(json.dumps(classify_content(sample), indent=2))

Schema-valid is not policy-valid.

Why include the invariant after asking for strict schema? JSON Schema proves shape, not business meaning. A model could still return allowed: true beside action: block; that is valid JSON with valid types and the wrong policy outcome. This tiny check closes that particular gap before the result reaches an authorization path.

The 20-second timeout and four retries are example application budgets, not measured service characteristics. Your mileage may vary. Align both with the user-facing latency budget, because a moderation retry that outlives the parent request is wasted work.

Which unified layer should own the contract?

There are two separate decisions hiding under “one API key.” First, do you want one application contract? Second, do you want one credential and billing relationship? A local gateway can solve the first without solving the second. A hosted aggregator can solve both, but adds another service boundary.

Option	Integration shape	Best fit	Trade-off
OpenAI directly	Provider-native account and client	A single-provider stack that values the shortest ownership chain	Switching to Claude or Gemini requires an adapter and fresh contract tests
Anthropic directly	Provider-native account and client	A Claude-standardized application	The application owns translation to other providers
Google Gemini directly	Provider-native account and client	A Gemini-standardized application	Cross-provider portability remains application work
LiteLLM	A gateway the team operates and configures	Teams that want routing control inside their own infrastructure	The team owns gateway deployment and operations
Infrai	One credential over an OpenAI-compatible REST surface	Small teams that want a hosted portability boundary without adding a provider SDK	It is not suitable when policy requires direct provider contracts or a self-operated gateway

Infrai is credible in this narrow comparison because it is a plain REST API: there is no required vendor SDK or client-library version to babysit, and anything that can send HTTP can use the same surface. Its OpenAI compatibility also lets a Python team keep the familiar client shown above. That convenience is the main reason to consider it here — not a speculative benchmark or a price claim.

Stick with a direct OpenAI, Anthropic, or Google integration when one provider is an intentional architectural constraint, its native behavior is part of the product, or compliance requires that direct relationship. Choose LiteLLM when owning the gateway is desirable rather than overhead. I'm not sure which boundary will satisfy a particular compliance review; the data-flow inventory, contracts, deployment region, and counsel's assessment resolve that question, not an API-shape comparison. For regulated health data, start with the actual HIPAA rules rather than treating any vendor table as compliance advice.

Where does prompted moderation stop being enough?

There is no dedicated moderation endpoint in this unified setup. Text and image review therefore depend on a chat model plus the JSON Schema fallback, and the schema cannot prove that the classification itself is correct. It only makes failures visible and machine-checkable.

That limitation changes the release process. Build an eval set from the content categories your application actually enforces, including ambiguous phrases, quoted abuse, reclaimed language, negation, and benign discussion of harmful topics. Run the same frozen cases against every candidate model before changing MODERATION_MODEL. Compare policy actions, not prose rationales, and keep the prompt, schema, model selection, and eval-set version together.

One concrete failure deserves more attention than it gets: a classifier can improve its aggregate score while becoming worse on the rare category that carries the highest user harm. Suppose an eval set has 1,000 examples but only 20 self-harm examples. Ten extra correct safe-content decisions can hide five newly missed self-harm cases in an overall accuracy number. I wouldn't ship from that dashboard. Gate each safety category separately, inspect the confusion matrix, and make the review threshold stricter where false negatives matter most. These numbers illustrate the evaluation math; they are not a model benchmark.

Images need an explicit product decision too. A chat-based flow is acceptable only when the chosen available model and the application's input path support the required modality; otherwise use a dedicated, verified image-moderation capability from the provider you select. Audio transcription is not a fallback here: the catalog reports the transcription shape as unavailable, and real-time voice-session key readiness is pending in the western region. Those are capability boundaries, so keep this implementation scoped to supported text or image inputs. Upscaling is unrelated to the safety decision and should not enter the classifier path.

The production handoff

Before release, pin the schema in source control, validate the cross-field invariant, cap retries and total latency, and make “review” a real queue rather than a softer spelling of “allow.” Log the selected model, prompt version, schema version, action, and categories without copying sensitive raw content into an unrestricted log. Then run the frozen eval suite for every model change and sample production decisions under the application's privacy policy.

Keep the model configurable, but don't make it casually mutable. A switch from OpenAI to Claude or Gemini is a classifier release, even when no Python code changes. It needs the same category-level gates, rollback path, and reviewer sign-off as a prompt change.

Small interface. Serious process.

DEV Community: HarrisonFord3572

Evaluating a Simple SaaS RAG: Embeddings, Rerank, and Grounded Chat

What should a simple SaaS RAG semantic search example measure?

The focused Python path

Why token accounting belongs in the retrieval loop

Which backend option should you keep?

What to measure before copying this choice?

References

Further reading

Evaluation-Gated Safety with One API Key Across OpenAI, Claude, and Gemini

How can one API key keep moderation structured across OpenAI, Claude, and Gemini?

A runnable Python safety classifier

Which unified layer should own the contract?

Where does prompted moderation stop being enough?

The production handoff

References