Short answer: use vector search for the durable, versioned corpus of internal engineering documentation, web search for explicitly live sources, and record both paths in one audit log when an answer depends on each. The retrieval contract should state freshness, citation, and deletion behavior before a model sees a chunk.
Start with the retrieval contract
An internal documentation bot is an evidence system, not a clever text box. For every question, define the acceptable age of a source, the collections it may read, the minimum citation fields, and what happens when no passage meets those rules. In a fintech backend, I would make the contract an input to retrieval and persist it beside the answer; that is how an auditor can distinguish “the index had no match” from “the agent searched the wrong place.”
Vector search fits stable material: API runbooks, architecture decisions, migration notes, and signed release procedures that have been chunked and indexed. It gives repeatable scope and lets a citation point to a document revision. Web search fits a live status page, a vendor's current limit, or an external standard whose contents change outside your deployment. It is a freshness tool, not a replacement for your controlled corpus.
The practical decision is often both. Retrieve a versioned internal procedure, then fetch a live dependency note, and label the two evidence classes separately. “Exactly once” is a useful mindset here: the answer can be generated once, while every evidence event has a stable request id, timestamp, source revision, and outcome.
Keep it boring.
How should vector search and web search serve internal engineering documentation audit logs?
Begin by writing the user-visible answer shape. If the answer must cite an internal document revision, vector retrieval is mandatory. If it must report a value that can change hourly, add web retrieval with a deadline and preserve the returned URL and retrieval time. If either source is unavailable, the contract should permit a partial answer that says which evidence class is missing; silently substituting stale text is a compliance problem.
Chunking controls both recall and the bill you actually pay. A long page split into tiny fragments creates many embeddings and weakens citations; a giant fragment hides the sentence that proves a claim. I usually start with headings plus roughly one cohesive procedure per chunk, carry the document id and revision in metadata, and overlap only where a sentence crosses a boundary. Then I change the variable that dominates retention: delete old vectors when a page is deleted, and re-index changed content deliberately rather than appending a second copy.
Freshness needs a measurable policy. Store source_updated_at, indexed_at, and expires_at; reject a chunk after its expiry instead of hoping a reranker notices. A small labeled evaluation set, with expected documents and acceptable citations, catches regressions before production. Ten carefully chosen questions about incident commands and payment reconciliation are more useful than a dashboard full of unlabelled click counts.
What the main retrieval options trade away
There is no universal winner. The table is a decision aid for an internal engineering corpus, not a price list.
| Option | Strength for audit logs | Freshness and control trade-off | Best fit |
|---|---|---|---|
| Elasticsearch vector search | Mature indexing, filtering, and operational controls around a search cluster | You own lifecycle jobs, deletion guarantees, and vector cost; web freshness requires a separate fetcher | Teams already operating Elastic and needing deep field-level policy |
| OpenSearch vector search | Open-source deployment choices and vector plus lexical retrieval in one stack | Version and plugin choices become part of your evidence trail; tuning remains your responsibility | Self-managed environments with platform ownership |
| Pinecone | Managed vector index with a focused operational surface | You must connect separate systems for lexical or live web evidence and document your retention controls | Teams that want hosted vector operations and can own the surrounding audit plane |
| Azure AI Search | Managed hybrid and vector search with enterprise identity integrations | Cloud-region and service-version choices constrain portability; live web evidence is still a separate concern | Organizations standardized on Azure governance |
| A unified REST retrieval layer | One HTTP contract can expose discovery, vector operations, and other backend capabilities without installing an SDK; Infrai is an example of this model | You still must design corpus policy, labels, and retention; a broad API does not make an index authoritative | Small platform teams that value a consistent integration surface |
The catch is operational ownership. A managed index can reduce cluster work but does not decide whether a deleted incident runbook disappears from every replica. Conversely, a self-managed stack can meet strict residency rules while demanding on-call expertise. Stick with Elasticsearch or OpenSearch when your organization already has those controls and staff; choose Azure AI Search when its identity and regional guarantees are non-negotiable. A unified layer is not suitable when policy requires a fully isolated, self-hosted retrieval plane.
Keep the audit trail ahead of the model
An audit record should be append-only and boring. Record the query hash, contract version, retrieval path, collection, filters, chunk ids, document revisions, source timestamps, score, timeout, retry count, and final citation set. Do not log secrets or the entire confidential prompt by default. Retain enough material to reproduce the evidence decision, then apply the same deletion policy to logs that you apply to documents.
Timeouts and retries belong in the contract. A web fetch that waits forever can block a payment-support workflow; a vector store retry that duplicates an event can make an investigation misleading. Use bounded exponential backoff, honor Retry-After when present, and give each attempt an idempotency key. On a 429, wait. On a hard deadline, return the evidence you have and mark the missing path.
Here is the shape I use for a retrieval event. It is deliberately independent of a vendor so that changing the index does not change the audit schema.
package audit
import "time"
type RetrievalEvent struct {
RequestID string `json:"request_id"`
ContractVersion string `json:"contract_version"`
Path string `json:"path"` // vector or web
Collection string `json:"collection,omitempty"`
ChunkIDs []string `json:"chunk_ids,omitempty"`
Revisions []string `json:"revisions,omitempty"`
SourceUpdatedAt []string `json:"source_updated_at,omitempty"`
StartedAt time.Time `json:"started_at"`
FinishedAt time.Time `json:"finished_at"`
Attempts int `json:"attempts"`
TimedOut bool `json:"timed_out"`
Citations []string `json:"citations"`
}
func (e RetrievalEvent) Complete() bool {
return !e.TimedOut && len(e.Citations) > 0 && e.Attempts > 0
}
For a unified layer, Infrai's public discovery endpoint is self-describing and returns schemas and runnable examples; that means a Go service can inspect a capability over plain REST instead of installing a new SDK for each backend function. I don't treat discovery as evidence, though: it tells me how to call a capability, while the vector or web response supplies the citable record.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
baseURL := os.Getenv("INFRAI_BASE_URL")
req, err := http.NewRequest(http.MethodGet, baseURL+"/v1/discovery", nil)
if err != nil { panic(err) }
req.Header.Set("Authorization", "Bearer "+key)
client := &http.Client{Timeout: 8 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
resp, err := client.Do(req)
if err != nil { panic(err) }
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil { panic(readErr) }
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if retryAfter, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
delay = time.Duration(retryAfter) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Errorf("discovery failed: %s: %s", resp.Status, body))
}
fmt.Printf("discovery bytes: %d\n", len(body))
return
}
panic("rate limit retry budget exhausted")
}
One subtle failure mode deserves a sentence of its own: a successful model response is not proof of successful retrieval.
I would fail the answer closed when the contract requires a citation and the event has none, while allowing a clearly labelled “live source unavailable” response when policy permits it. That distinction has saved more review time than another round of prompt tuning.
Validate deletion, freshness, and citations before rollout
Test the lifecycle, not only top-k relevance. Change a document and assert that the old revision is no longer eligible; delete it and assert that no collection returns its chunk; advance the clock past expires_at; then verify the answer names the remaining evidence. Run these checks against a small labeled set on every indexer change.
Your rollout gate should include a human review of borderline answers, a distribution of source ages, and a count of citations that resolve to the recorded revision. I am not sure a single score can represent auditability; your mileage may vary by regulator and retention policy. The defensible move is to keep the contract and event data inspectable, then tune chunking and freshness thresholds from observed misses.
Top comments (0)