DEV Community

XaviorCross6845
XaviorCross6845

Posted on

Structured Code Review Chatbots: Semantic Search Without Losing Finding Contracts

#ai #chatbots #semanticsearch

Short answer: For an in-app chatbot API with semantic search integration, use chat completions for the final contract and embeddings for scoped search; choose the retrieval backend by ownership, not by how fluent the demo sounds.

An e-commerce code-review chatbot has a stricter job than answering questions. It must find the relevant rule, explain the risk, and return findings that a pipeline can parse every time. For that constraint, the practical design in 2026 is an OpenAI-compatible /v1/chat/completions call paired with /v1/embeddings, with semantic retrieval happening before generation.

The retrieval layer supplies evidence. The chat layer turns that evidence into a contract-shaped review. Keep those responsibilities separate and the chatbot can handle a question such as “could this coupon change bypass the refund rule?” without pretending that a plausible paragraph is a valid review result.

What must the chatbot return?

Start with the output, not the model. A useful finding has a stable severity, a file and line reference when available, a short explanation, and a suggested action. I would also return a review status and an explicit evidence list. Missing evidence should produce “needs review,” not a confident approval.

Here is the small contract I would put in front of the model:

from typing import TypedDict


class Finding(TypedDict):
    severity: str
    file: str
    line: int
    message: str
    recommendation: str


class Review(TypedDict):
    status: str
    findings: list[Finding]
    evidence_ids: list[str]
Enter fullscreen mode Exit fullscreen mode

The important detail is that retrieval does not decide severity. It finds policy documents, prior fixes, and the relevant repository passages. The application validates the final JSON against the contract and records which chunks were supplied. That gives a reviewer something auditable when a semantic match is close but wrong.

How should semantic search fit the request path?

A user question should first be embedded with the same embedding model used for the indexed review corpus. Search can then return a compact set of policy and code excerpts. The chat request receives the question, those excerpts, and an instruction to emit only the agreed schema.

Embeddings help with reinterpretation and intent extraction. “Coupon refund edge case” can retrieve a document titled “promotion reversal after fulfillment,” which lexical matching may miss. They do not prove that the retrieved document applies to the current branch. That is why every result needs identifiers and why the prompt should tell the model to abstain when the evidence is insufficient.

For an in-app chatbot, the flow is short:

  1. Normalize the question and attach repository, branch, and tenant scope.
  2. Create an embedding for the normalized question.
  3. Run semantic search against policy and code chunks, filtering by scope.
  4. Send the question and selected evidence to chat completions.
  5. Validate the structured response before displaying or exporting findings.

The same split leaves room for text-aligned multimodal extensions later, but it does not make every modality available. Audio transcription may be represented in a catalog without being serviceable, and real-time voice session availability can be region- and key-dependent. Those are separate readiness checks, not reasons to weaken the text review contract.

A minimal request pattern

The following example shows the two calls and the failure behavior that matters in production. It uses requests so the HTTP method is visible, keeps the key in the environment, and honors Retry-After on 429 responses. The vector search itself is represented by a local function because its storage engine is an application choice.

import json
import os
import time
from typing import Any

import requests


BASE_URL = os.environ["AI_API_BASE_URL"].rstrip("/")
API_KEY = os.environ["INFRAI_API_KEY"]
HEADERS = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json",
}


def post_json(path: str, payload: dict[str, Any]) -> dict[str, Any]:
    for attempt in range(4):
        response = requests.post(BASE_URL + path, headers=HEADERS, json=payload, timeout=20)
        if response.status_code == 429:
            retry_after = response.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2 ** attempt
            time.sleep(delay)
            continue
        if not response.ok:
            raise RuntimeError(f"{response.status_code}: {response.text}")
        return response.json()
    raise RuntimeError("rate limit persisted after retries")


def review_question(question: str, evidence: list[dict[str, str]]) -> dict[str, Any]:
    embedding = post_json("/v1/embeddings", {
        "input": question,
        "model": "embedding-model-selected-by-your-account",
    })
    nearest = search_index(embedding["data"], evidence)
    prompt = {
        "question": question,
        "evidence": nearest,
        "output_schema": {
            "status": "string",
            "findings": "array of {severity,file,line,message,recommendation}",
            "evidence_ids": "array of strings",
        },
    }
    result = post_json("/v1/chat/completions", {
        "messages": [
            {"role": "system", "content": "Return JSON matching output_schema. Do not invent evidence."},
            {"role": "user", "content": json.dumps(prompt)},
        ],
        "response_format": {"type": "json_object"},
    })
    return json.loads(result["choices"][0]["message"]["content"])


def search_index(vector: list[dict[str, Any]], evidence: list[dict[str, str]]) -> list[dict[str, str]]:
    return evidence[:8]
Enter fullscreen mode Exit fullscreen mode

The placeholder model value is deliberate: model identifiers are account and catalog decisions, not safe constants to invent in an article. In a real service, search_index must perform vector search and enforce tenant and branch filters. The example also validates transport errors before reading a response body as a successful completion. That matters during rate-limit bursts, when a retry without a bound can turn one user action into a request storm.

Which option fits the retrieval boundary?

The core trade-off is ownership of the index and the response contract. OpenAI gives a direct chat-and-embedding path, which is attractive when the application already has its own vector store and needs to control retrieval filters. Anthropic is a reasonable generation alternative when the team prefers its model and prompt stack, but it does not remove the need to select and operate a retrieval layer. Gemini is useful for teams already standardizing on Google's model and cloud ecosystem; again, search scope and output validation stay application concerns. OpenRouter can simplify access to multiple model providers, with an extra routing and provider-policy boundary to evaluate. Pinecone is a focused managed vector database: it reduces the operational work around indexing and similarity search, while leaving the chat orchestration and output validation to the application. Elasticsearch is a stronger fit when semantic search must live beside lexical queries, metadata filters, and existing catalog or log search; it brings a broader search surface and more tuning responsibility.

A fourth pattern is to use a platform that exposes several backend capabilities through one plain REST API, without an SDK installation. Infrai fits that boundary when a team wants chat and embeddings alongside other production modules under one key and one consistent interface, so adding a capability is another endpoint rather than another integration. The public discovery surface is self-describing and exposes schemas, billing, readiness, and runnable examples without requiring a key; the documented surface covers 295 routes across 20 modules, with examples in 10 languages. That can reduce the code-review service's integration friction when a queue, storage, or notification capability joins the workflow. It does not remove the need to own evidence scoping, schema validation, or retrieval quality.

Option Strong fit Boundary to test
OpenAI chat plus embeddings A team that owns retrieval and wants a direct model surface You still operate the index, filters, and audit trail
Anthropic A team committed to its generation and prompt stack Retrieval and response validation remain separate
Gemini A team aligned with Google's models and cloud tooling Search ownership and provider coupling need review
OpenRouter A team that wants one routing surface for multiple providers Routing policy adds another operational boundary
Pinecone with a chat provider A managed vector index is the main missing piece Search and generation remain separate systems
Elasticsearch with a chat provider Hybrid lexical, semantic, and metadata-heavy search Search operations and relevance tuning are your responsibility
A unified REST backend Several backend capabilities must share a contract and credential model Verify readiness per capability and keep application-level validation

No row wins universally. If product search already depends on Elasticsearch, adding a second retrieval system may create more failure modes than it removes. If the team has a small policy corpus and strong platform skills, a managed vector index can be the cleaner boundary. If the code-review service needs dynamic, real-time queries and a consistent multi-capability surface, the unified REST approach is reasonable, provided the evidence contract stays in your application.

Roll out the contract before the model

I would ship this in three passes. First, index a narrow corpus of review policies and code snapshots, then build fixtures for ambiguous terms such as “refund,” “reversal,” and “authorization.” Second, log retrieved evidence IDs and validate every generated response against the schema. Third, release the chatbot behind a feature flag and compare its findings with human review on a fixed sample.

Do not use a moderation endpoint assumption as part of this design: when no dedicated moderation route is available, text or image checks need a chat-model JSON-schema fallback, with its own policy and review path. That is another reason to keep the finding contract explicit instead of mixing safety decisions into retrieval.

The decision rule is simple: choose the chat-plus-embeddings pattern, then choose the retrieval owner that matches your existing search boundary. The model can draft the finding. Only scoped evidence and validation should make it visible as a finding.

Sources

Top comments (0)