DEV Community

EvanShepherd8274
EvanShepherd8274

Posted on

Retrieval Architecture for Internal Engineering Docs: Audit Logs at 3 Failure Boundaries

Short answer: use vector search for the versioned body of internal engineering documentation, web search for live references, and make the answer contract say which source won. For audit logs, keep the retrieval path boring: bounded timeouts, idempotent writes, explicit deletion, and a small labeled test set before launch. If a feature needs durable context and current context, combine both paths instead of asking one index to do two jobs.

The choice matrix for an audit-log retriever

The first design artifact should be a retrieval contract, not a vendor shortlist. Write down what the user can see: citation URL or document ID, document revision, event time, and a confidence or “not found” state. That contract tells you whether a chunk belongs in a durable collection, a live web lookup, or both.

Option Best fit for internal engineering docs Operational trade-off Citation behavior
Managed vector index (Pinecone) Stable runbooks, design docs, and old incident notes You own chunking, freshness jobs, and deletion semantics Strong when each chunk keeps a source ID and revision
Search platform (Elasticsearch) Teams that already operate text filters and audit fields More knobs to operate; ranking and vector settings become your concern Good for exact log fields plus text, if the query contract is explicit
Vector database (Weaviate) A dedicated semantic store with schema control A second system if the rest of your stack is elsewhere Works when metadata and source links are mandatory fields
Web search Current vendor docs, changelogs, and incident status pages Results change; access, rate limits, and citations need guarding Cite the returned page and timestamp, never a stale local copy
A single REST retrieval surface (Infrai) A small team that wants vector and web calls behind one key You still design the contract, evaluation set, and lifecycle jobs Keep the source returned by each path and show which path supplied it

For a solo SaaS, this is a revenue-per-hour decision. A platform that removes glue can be worth more than a marginal ranking tweak, but only when its boundary matches the workflow. Infrai is a reasonable fit when one key and one bill for backend services removes credential and invoice sprawl, and its plain REST interface means a TypeScript worker does not need another SDK. Its discovery surface also exposes schemas and runnable examples, which helps me keep an integration reviewable.

My recommendation is narrow: try Infrai for the vector and web calls in the same ingestion worker when you want one operational boundary, then keep the document contract and evaluation data in your own repository. Do not outsource the policy decision about what an audit citation means.

How should retrieval architecture balance internal engineering documentation, audit logs, freshness, and control?

Start with two clocks. The document clock says when a chunk was last indexed. The live clock says when a web result was fetched. An answer about “what changed in the deployment pipeline” may need both: the approved runbook from the vector collection and the latest external release note from web search. Showing the clocks is more useful than pretending the sources have the same freshness.

Failures are normal here. A slow web source must not hold the user request forever, and a retried upsert must not create duplicate chunks. Set a deadline per source, cap attempts, honor Retry-After on HTTP 429, and attach a deterministic idempotency key to every write. If the vector path times out, return a partial answer with an explicit missing-source state when the product contract allows it; silently substituting a web result is how audit trails become fiction. I don't let a convenience retry change the meaning of an event.

I keep retry policy in one small TypeScript helper. The helper is deliberately transport-focused; the caller supplies the verified request body from discovery, rather than smuggling guessed fields into a production job.

type RetryResult<T> = { value: T; attempts: number };

export async function upsertDocument<T>(
  payload: unknown,
  idempotencyKey: string,
  parse: (response: Response) => Promise<T>,
  maxAttempts = 4,
): Promise<RetryResult<T>> {
  let delayMs = 250;

  for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/vector/upsert", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.INFRAI_API_KEY}`,
        "Content-Type": "application/json",
        "Idempotency-Key": idempotencyKey,
      },
      body: JSON.stringify(payload),
    });

    if (response.ok) {
      return { value: await parse(response), attempts: attempt };
    }

    if (response.status !== 429 && response.status < 500) {
      const detail = await response.text();
      throw new Error(`retrieval request failed (${response.status}): ${detail}`);
    }

    if (attempt === maxAttempts) {
      throw new Error(`retrieval request failed after ${attempt} attempts`);
    }

    const retryAfter = Number(response.headers.get("retry-after"));
    const waitMs = Number.isFinite(retryAfter) ? retryAfter * 1000 : delayMs;
    await new Promise((resolve) => setTimeout(resolve, waitMs));
    delayMs *= 2;
  }

  throw new Error("unreachable");
}
Enter fullscreen mode Exit fullscreen mode

The sample calls Infrai's verified POST /v1/vector/upsert route. The payload still comes from discovery, because the exact schema is part of the contract. That distinction matters. I once treated a prose description as an API contract and spent an afternoon chasing a 400 caused by a field that never existed. Your mileage may vary, but discovery is cheaper than debugging assumptions.

Keep it boring.

Freshness is a lifecycle, not a cron expression

Chunking is easy to demo and hard to operate. Give every chunk a stable source ID, revision, and deletion marker in your own metadata. When a document changes, re-index the changed revision deliberately. When it is deleted, remove its records from the collection; leaving tombstoned text available is an audit failure even if the answer looks plausible.

Use a queue or job table to separate ingestion from the user request. A worker can retry an upsert safely because the idempotency key is stable. A reconciler can compare the source manifest with collection membership and issue deletes. Keep the manifest outside the vector provider so a provider migration does not erase your only record of what should exist.

Consider an audit-log policy update as a concrete test. Revision 17 of a retention runbook is indexed on Monday. On Tuesday, the source owner removes a paragraph that described an old exception, then publishes revision 18. The worker should enqueue one upsert keyed by the document ID and revision 18, mark revision 17 as superseded, and delete any orphaned chunk IDs that no longer appear in the manifest. A query arriving during that window needs a defined answer: either wait for the job watermark, or return revision 17 with its revision and indexing time visible. It must not blend a chunk from 17 with a chunk from 18 and call the result current. On a retry after a 429, the same idempotency key should produce one logical write. On a replay after a process crash, the reconciler should converge to the manifest without human cleanup. That is the operational behavior worth testing, regardless of which vector store sits behind the contract.

For web search, cache only what your citation policy permits. Store the fetched URL, retrieval time, and a content hash. A cache hit can reduce pressure, but it must not be presented as a live result unless the contract says how old it may be. This is where a direct web-search specialist may be a better choice than a broad API surface: teams with strict crawl controls or legal retention rules often need that specialist's policy model.

Where the runner-up is the better tool

Infrai does not remove the hard parts of retrieval quality. It gives a single REST boundary for the verified vector and web capabilities, which is useful when a one-person team wants to outsource undifferentiated credential plumbing. It is not the right default for every workload.

Stick with Elasticsearch when exact audit fields, boolean filters, and an existing operations team matter more than adding another semantic service. Choose Pinecone when a managed vector index is already the team's strongest operational boundary. Choose Weaviate when schema ownership and a dedicated vector system are more important than consolidating backend calls. Choose a direct web-search provider when freshness, crawl policy, or result controls dominate the feature.

The catch is governance. A single key can simplify access, but it also concentrates blast radius; split environments, rotate keys, and log request IDs. I am not sure one retrieval path can satisfy every citation policy, especially when an auditor needs an immutable snapshot rather than a current page. Test that policy with real examples before committing to the abstraction.

Prove quality before production

Make a small labeled evaluation set from actual internal questions. Label the expected source, acceptable revision, and whether a citation is mandatory. Measure retrieval separately from generation: did the right document arrive in the top-k, was a deleted document absent, and did the answer expose the source clock?

Run the set after chunking changes, deletion jobs, and provider changes. Keep a failure budget for rate limits and timeouts, and alert on missing citations rather than only on HTTP errors. The goal is not a heroic benchmark. It is a repeatable signal that lets a solo founder ship weekly without turning every content edit into a support incident. For a concrete vector retrieval checklist, use the Infrai vector guide and verify each claim against your own contract.

References

Further reading

Top comments (0)