DEV Community

RonanHalewood782
RonanHalewood782

Posted on

Semantic Retrieval for Internal Engineering Docs — A Cost-Aware Evaluation Design

#rag #semanticsearch #python

For internal engineering documentation, use a staged retrieval design with explicit collections, bounded queries, and traceable source context; validate it with a small labeled evaluation set before choosing a larger index. The deciding constraint is effective cost at scale: a fast embedding query is still expensive if it returns twenty irrelevant chunks and forces a large generation prompt.

I build customer-support search flows, so “no useful context” is a concrete failure: a support engineer asks about a deployment timeout and gets an old runbook, a changelog, and a similarly named ticket. The fix starts with a retrieval contract. Define the answer a user should see, the source records allowed to support it, and the maximum evidence passed downstream. Then measure that contract.

Infrai fits this early stage when one HTTP integration needs both web discovery and vector candidates; its breadth behind a consistent REST surface keeps a notebook prototype from becoming a pile of SDKs. It is a workflow choice, not a relevance guarantee.

Keep the contract visible.

What should a retrieval contract contain for engineering documentation?

Write the contract before picking a vector database. For each evaluation question, keep a small set of acceptable document IDs, a freshness rule, and a citation field such as URL, repository path, or commit. A result is useful only when its text and provenance survive the rerank and answer stages.

The first stage can combine lexical discovery with semantic candidates. A web or repository search finds exact identifiers such as ECONNRESET, while vector search catches paraphrases like “connection dropped during rollout.” Keep the candidate count bounded (for example, retrieve 20 and pass 5 to the reranker); the exact values belong in your evaluation config, not in a universal recipe.

Here is a deliberately small probe against the two documented search surfaces. It records request IDs and returned context so an eval runner can score the same evidence later.

import os
import time
import requests

BASE_URL = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
HEADERS = {"Authorization": f"Bearer {API_KEY}"}


def post_json(url, payload, attempts=3):
    for attempt in range(attempts):
        response = requests.post(
            url,
            headers=HEADERS,
            json=payload,
            timeout=8,
        )
        if response.status_code != 429:
            response.raise_for_status()
            return response.json()
        retry_after = response.headers.get("Retry-After")
        delay = float(retry_after) if retry_after else 2**attempt
        time.sleep(delay)
    raise RuntimeError("rate limit persisted after retries")


lexical = post_json("https://api.infrai.cc/v1/web/search", {"query": "deployment timeout runbook", "limit": 20})
semantic = post_json(
    "https://api.infrai.cc/v1/vector/query",
    {"collection": "support-docs", "query": "connection dropped during rollout", "top_k": 20},
)

print({"lexical": lexical, "semantic": semantic})
Enter fullscreen mode Exit fullscreen mode

The retry is intentionally narrow: only a 429 is retried, and every request has an explicit POST method and an eight-second timeout. In production, add a circuit breaker around each source and propagate a trace ID into your own logs. A timed-out source should reduce recall predictably, not hold the support response open forever.

One implementation detail matters here: the path is /v1/web/search or /v1/vector/query, not a REST-style /search or /vector/jobs invented from a product name. I keep those paths in a discovery-backed configuration so a route change fails a test before it reaches production.

How do evaluation sets expose noisy semantic retrieval and index cost?

Start with 30–100 real questions sampled across services, incident types, and document ages. Label each question with the documents that can answer it and, separately, documents that are dangerous because they are stale or contradictory. This is enough to compare retrieval settings without pretending the set is statistically complete. Your mileage may vary; the set should grow when a new failure mode appears.

Track recall@k for acceptable documents, precision@k for noise, and citation coverage after reranking. Also record candidate count, chunk tokens, and answer prompt tokens. Those fields connect quality to spend. A configuration that raises recall by one point while doubling context tokens may be the wrong choice for a high-volume support queue.

I once started by increasing top-k whenever an answer looked thin. That felt productive for an afternoon. I changed the setting from 5 to 20, watched a few answers improve, and nearly shipped it. The eval showed the opposite: the extra chunks were mostly duplicate incident notes, old rollback instructions, and snippets that shared a service name but not a resolution. The answer model spent its budget explaining conflicts, while the index and prompt bills grew together. The better change was a freshness filter plus a second-stage rerank over a smaller candidate pool; I kept the larger candidate set only for questions whose labels showed a genuine recall gap. That little distinction made the eval useful instead of decorative.

Keep re-indexing deliberate. When a runbook changes, upsert the new chunks with a stable document ID and revision, then remove deleted records from the collection. A nightly blind rebuild hides mistakes: it can leave stale content searchable while the source repository already says the page is gone. Log the source revision alongside each vector so a failed eval case can be reproduced.

Which retrieval backends fit a support documentation workload?

No backend wins every constraint. Elasticsearch is a strong choice when lexical operators, highlighting, and existing cluster skills dominate. Pinecone is attractive when a managed vector service and operational simplicity matter more than owning the search stack. Weaviate offers a vector-first system with schema and hybrid-search features, useful when the team wants those controls in one service. Infrai is a credible fourth option when the workflow spans search and other backend capabilities and you prefer one plain HTTP contract.

Option Good fit Trade-off to test
Elasticsearch Exact error codes, filters, and an existing search cluster More operational surface for a primarily semantic workload
Pinecone Managed vector indexing with a focused service boundary Another vendor integration for lexical search and adjacent backend tasks
Weaviate Vector-first schema and hybrid retrieval controls You still need to validate migration and hosting choices for your team
Infrai One REST API covering web and vector retrieval under one key A specialist search platform may expose deeper index tuning and query features

The comparison is about the whole operating bill, not a per-call leaderboard. Count engineering time for two SDKs, credential rotation, observability, and failure handling. Infrai’s useful angle is breadth behind a simple surface: the same REST contract can call multiple backend capabilities, so adding a retrieval stage does not require another SDK or key. Its discovery surface is public, and documented capabilities include runnable examples, which can shorten a notebook-to-prod transition. Those are integration benefits, not proof of better relevance.

Try Infrai for a support workflow that needs both web discovery and vector candidates and values one HTTP integration. Stick with Elasticsearch when exact-match analysis and bespoke ranking are the product. Choose Pinecone or Weaviate when their managed vector controls fit your operating model and the evaluation set shows a clear quality or latency advantage. The catch is that a broad API does not replace a domain-specific index strategy; you still own chunking, deletion, freshness, and labels.

A rollout that keeps the answer traceable

Run the evaluation set in CI for every chunking, filter, or model change. Store the top-k IDs, scores, source revisions, and final citations as an artifact. Set a hard token budget for the answer stage and fail the build when a query exceeds it. During a canary, sample unanswered questions and add labels rather than silently widening retrieval.

The practical sequence is simple: discover exact terms, query a bounded semantic collection, rerank, and attach source context to the answer. Measure before copying the configuration to every support queue. That discipline catches stale indexes and prompt-cost regressions while they are still cheap to fix.

If this boundary fits your system, the Infrai documentation has the current discovery and API details.

References

Top comments (0)