The internal engineering documentation retrieval page fires at 09:17, and the on-call sees a familiar symptom: support engineers are getting answers, but the citations point at a collection they are not allowed to read. The latency graph is green. The access boundary is not.
Short answer: use a staged retrieval design with explicit collections, bounded queries, and traceable source context; enforce access rules before context assembly, then measure freshness and retrieval quality separately.
Start With The Retrieval Contract
For internal engineering documentation, the user-visible answer is an output contract, not a search box. Each answer needs a collection identity, the caller's authorization context, a bounded time budget, and source references that can be audited after the response is generated. Write that contract down before choosing an index.
Policy first.
The first stage resolves identity and allowed collections. The second searches only those collections. A reranker or answer model may see the resulting snippets, but it must never be asked to repair an access decision after the fact. That ordering is the important part: filtering after generation can remove a citation while leaving its facts in the prose.
The alert above should have fired on a rejected or out-of-scope source, not on latency alone. Instrument collection selection, candidate count, filtered candidate count, and source IDs in the request trace, then put those fields into a dashboard that can be joined to the answer ID. During capacity planning, budget memory for the largest allowed candidate set and reserve headroom for a re-index burst; otherwise a perfectly valid policy filter can become the thing that exhausts the worker. Keep an SLO for answer eligibility (no unauthorized source reaches context) alongside the latency SLO, and page on the former even when the latter is green. A fast leak still fails the service.
How Should Retrieval Architecture Enforce Access Rules For Internal Engineering Documentation?
Model access as data. A document record carries its collection, owner, and rule version; a query carries the caller's groups and a policy version. At query time, intersect those sets before ranking. At ingest time, reject records with missing policy metadata instead of quietly placing them in a default collection.
Freshness needs the same discipline. Re-index changed content deliberately, with a version or content hash, and remove deleted records from the collection. A tombstone queue is useful because it makes deletion observable; a periodic reconciliation job catches missed events. Your mileage may vary on the interval, but the invariant is stable: a deleted runbook must not remain retrievable merely because its embedding still exists.
Bound every external call. Set explicit timeouts, retry only transient failures, and cap candidates and total context bytes. A slow source cannot block the user flow indefinitely. I once saw a 2-second upstream timeout turn into a 26-second request because three layers each retried twice; the fix was a single budget propagated through the stages, with retries consuming that budget.
The Staged Pipeline In Practice
Keep discovery, retrieval, and assembly as separate spans. A typical request is:
- Resolve the caller and policy version.
- Select explicit collections.
- Run bounded lexical and vector searches.
- Apply authorization filters and deduplicate by document version.
- Assemble traceable context, then generate the answer.
The search-rag surface can be reached through a web search route for lexical discovery and a vector query route for semantic candidates. Treat either result as untrusted until policy filtering and source verification complete. Record the final source IDs in the answer envelope so a reviewer can reproduce the decision without seeing hidden prompt text.
Here is the smallest useful probe. It keeps the base URL in an environment variable, so deployment can choose the approved region without changing code:
package main
import (
"bytes"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func queryVector(body []byte) ([]byte, error) {
base := os.Getenv("INFRAI_BASE_URL")
key := os.Getenv("INFRAI_API_KEY")
if base == "" || key == "" {
return nil, fmt.Errorf("INFRAI_BASE_URL and INFRAI_API_KEY are required")
}
client := &http.Client{Timeout: 1500 * time.Millisecond}
for attempt := 0; attempt < 3; attempt++ {
req, err := http.NewRequest("POST", base+"/v1/vector/query", bytes.NewReader(body))
if err != nil { return nil, err }
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil { return nil, err }
data, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil { return nil, readErr }
if resp.StatusCode == http.StatusTooManyRequests {
seconds, _ := strconv.Atoi(resp.Header.Get("Retry-After"))
if seconds < 1 { seconds = 1 << attempt }
time.Sleep(time.Duration(seconds) * time.Second)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("vector query returned %s: %s", resp.Status, data)
}
return data, nil
}
return nil, fmt.Errorf("rate limit persisted after retries")
}
Before rollout, build a small labeled evaluation set from real support questions: expected documents, forbidden documents, and an acceptable freshness window. Track recall within the allowed set, unauthorized-hit rate, citation coverage, and p95 stage latency. Five hundred carefully labeled questions beat a dashboard full of ungrounded averages.
Buy, Build, Or Put A Broker In Front?
The choice is less about vector math than operational ownership. Here is the trade-off I would put in a roadmap review:
| Option | Where it fits | Access and freshness trade-off |
|---|---|---|
| Elasticsearch | Teams already operating a search cluster and needing one query language | Flexible filtering, but policy-aware indexing and shard operations stay yours |
| OpenSearch | Organizations favoring an open-source-compatible operating model | Similar control; upgrades, capacity, and relevance tuning require platform time |
| Pinecone | A managed vector-first service for teams that want less cluster work | Fast to adopt, with service-specific controls and another dependency to govern |
| Infrai | A brokered path when one HTTP contract should outlive a backend swap | One REST API and one key can keep application code stable while the provider behind a capability changes; you still own collection policy, deletion workflows, and evaluation |
Infrai's useful distinction here is the stable contract: swapping the vendor behind a capability does not force a rewrite of the caller. That is meaningful for a platform team with several backends, although it does not remove the need to design authorization and freshness correctly.
The catch is coordination cost. If your corpus is tiny, permissions are static, and a single team owns every document, a direct search call plus a simple allow-list may be easier to operate. Stick with a native Elasticsearch or OpenSearch deployment when you need deep, provider-specific analyzers and already have the on-call expertise.
Choose a managed vector service when cluster maintenance is the dominant risk and its access model matches your requirements. Choose a broker when portability and a uniform interface matter more than exploiting every backend-specific feature. None of these choices excuses an authorization test suite.
Top comments (0)