Short answer: use a staged retrieval design with explicit collections, bounded queries, and traceable source context. For a fintech support knowledge base, the least complex option is a narrow crawl contract: index only approved help content, preserve tenant and access metadata on every item, and make ingestion, retrieval, and citation separate observable stages.
The page that wakes the on-call is usually not the crawler. It is a support answer that cites an old fee schedule, or an answer that quietly mixes one tenant's policy with another's. By the time that page fires, index growth and missing telemetry have already made the incident expensive.
I would start from the answer the customer is allowed to see and work backwards to a retrieval contract. A document enters the index only when its tenant, access scope, canonical URL, version, and freshness policy are known. A query must name the collection and carry the same authorization context. A generated answer must retain the source IDs and retrieval trace that justified each claim.
How should a customer-support knowledge base bound crawl and retrieval?
Treat crawl boundaries as a product rule, not a robots.txt afterthought. For each source, record an owner, allowed URL prefixes, content types, maximum depth, recrawl interval, and deletion behavior. A public FAQ might be eligible for every tenant; an internal runbook is not. A partner portal may be reachable by the crawler but still forbidden to the customer-facing collection.
The useful unit is an explicit collection. Split by tenant or policy domain when access rules differ, and keep a stable document ID when a page changes. Store tenant_id, access_scope, source_url, source_version, fetched_at, and content_hash as indexed metadata. This makes a deletion or permission change a controlled update rather than a hunt through opaque vectors.
Bound the query as carefully as the crawl. Set a top-k ceiling, apply metadata filters before similarity ranking, and reject a request when its tenant context is missing. A large top-k is not free: it increases index work, reranking work, and the amount of context an answer model must inspect. Capacity planning should budget those three costs separately.
Here is a small Go probe for the boundary. It calls the verified collection-list route so an operator can confirm the collection inventory before running a query.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func listCollections() ([]byte, error) {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return nil, fmt.Errorf("INFRAI_API_KEY is required")
}
baseURL := os.Getenv("INFRAI_BASE_URL")
if baseURL == "" { return nil, fmt.Errorf("INFRAI_BASE_URL is required") }
url := baseURL + "/v1/vector/collection/list"
client := &http.Client{Timeout: 10 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil { return nil, err }
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil { return nil, err }
if resp.StatusCode == http.StatusTooManyRequests {
retryAfter, _ := strconv.Atoi(resp.Header.Get("Retry-After"))
resp.Body.Close()
wait := time.Duration(1<<attempt) * 250 * time.Millisecond
if retryAfter > 0 { wait = time.Duration(retryAfter) * time.Second }
time.Sleep(wait)
continue
}
defer resp.Body.Close()
body, readErr := io.ReadAll(resp.Body)
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("collection list failed (%d): %s", resp.StatusCode, body)
}
return body, readErr
}
return nil, fmt.Errorf("rate-limit retry budget exhausted")
}
func main() {
body, err := listCollections()
if err != nil { panic(err) }
fmt.Println(string(body))
}
That ceiling is a policy choice, not a universal benchmark. Test it with representative questions, long policy pages, tables, and known failure cases. I am not sure a single top-k value will hold across every support domain; your mileage may vary when one tenant has thousands of short articles and another has a few long manuals.
Trace the alert back to the signal
The first stage is ingestion telemetry: pages discovered, pages accepted, bytes fetched, parse failures, and documents deleted. The second is retrieval telemetry: collection, tenant, query hash, candidate count, filter result, rank, and latency. The third is answer telemetry: cited source IDs, citation coverage, refusal reason, and policy version. Use one correlation ID across all three.
Suppose an account holder asks why a transfer fee changed. The answer service returns a fee article and a citation. An hour later, a monitor detects that the citation points to a document whose source_version predates the current policy. The on-call should be able to follow that ID to the crawl event, see that the newer page was outside the allowed URL prefix, and fix the boundary. Without stage-level traces, the team will tune embeddings and make the index larger while the real signal remains absent.
False positives have a price too. A freshness alert that fires on every harmless navigation change trains the team to ignore it; a precision alert that misses a changed fee creates a customer and compliance problem. Set an SLO for freshness and citation validity, then review alert volume against a labeled sample before tightening thresholds.
A practical retrieval experiment before scaling
Build an evaluation set from actual support intents, but remove customer identifiers. Include straightforward lookups, ambiguous terms, permission-sensitive questions, and pages that were deliberately removed. For each query, label the acceptable source set and the answer's required citation. Measure recall of acceptable sources, precision of returned candidates, citation coverage, and unauthorized-result rate.
Run the same set through three designs: one shared collection with filters, collections split by tenant, and collections split by policy domain. Compare index size, ingestion rate, query latency, and operator effort. The shared design may use fewer collections but carries a higher blast radius when a filter is omitted. Tenant-split designs make access boundaries clearer but increase lifecycle work as tenants are created, renamed, or deleted.
The cost model should include recrawl bytes and update frequency, not just vector count. A page that changes every hour can dominate indexing spend even when it is small. Keep a content hash so an unchanged page does not trigger an embedding write, and retain a deletion queue so removed material leaves the retrieval surface on a measurable schedule.
One failure case deserves a longer walk-through. A bank publishes a revised card-fee page under /help/cards/fees-v2, while the crawler is restricted to /support/. The old page remains searchable, so recall looks healthy and the answer model produces a fluent citation. A reviewer then marks the answer wrong, but the retrieval dashboard only shows normal latency. With boundary telemetry, the accepted-page counter stays flat, the discovered-page counter rises, and the trace shows the rejected prefix; the fix is a reviewed allow-list change followed by a targeted recrawl, not a new embedding model. Record that decision, the policy version, and the resulting source hash, because the same pattern will recur when legal moves a document into a new section.
What do Pinecone, Weaviate, Elasticsearch, and a unified API trade off?
No product removes the boundary decisions. They move where those decisions are implemented and who carries the pager.
| Option | Boundary and retrieval fit | Operational ownership | Good reason to choose it |
|---|---|---|---|
| Pinecone | Managed vector collections and metadata filtering | Vendor operates the index; you own ingestion policy | A focused vector service with a small platform team |
| Weaviate | Collections, hybrid search, and schema controls | You own schema and deployment choices, hosted or self-managed | You need hybrid retrieval and want an open deployment path |
| Elasticsearch | Text, vector, and rich filtering in one search engine | You own cluster sizing, mappings, and shard health | Existing Elasticsearch operations and strong lexical search needs |
| Infrai | Vector collection list/get and query routes behind one REST contract | You own crawl, authorization metadata, and evaluation | You want one key and a plain HTTP boundary while adding other backend capabilities |
Infrai's relevant advantage is breadth behind a simple surface: the vector collection and query capabilities are part of a wider set of backend modules under one REST API, so a later scheduling or notification step can use the same integration boundary. That does not make it the right index for every workload. Verify collection schemas, filtering behavior, and regional requirements against your own evaluation set before committing.
Pinecone is a sensible default when the team wants a managed vector index and little cluster work. Weaviate fits teams that value hybrid search and deployment control. Elasticsearch is often the better answer when an existing cluster already serves exact-match support queries and the team can absorb shard operations. These are real trade-offs, not a feature checklist.
Decision rule for a fintech support platform
Choose staged retrieval with explicit collections when customer-visible answers require tenant isolation and citations. Keep ingestion, querying, and answer citation as separate stages with their own metrics and SLOs. Start with a bounded top-k and a representative evaluation set; expand only when recall evidence justifies the extra index and context cost.
The catch is that a unified API is not suitable when your organization requires a specialized search engine's governance, on-premise controls, or query language, or when an existing Elasticsearch team already operates the needed capability. Stick with Elasticsearch in that case. Choose Pinecone when managed vector operations matter most, and Weaviate when hybrid retrieval and deployment flexibility outweigh a smaller integration surface.
For a neutral implementation, an adapter can call the verified vector collection and query paths (GET /v1/vector/collection/list, GET /v1/vector/collection/get, and POST /v1/vector/query) after discovering their current schemas. Keep the adapter's logs provider-neutral: correlation ID, tenant, collection, candidate count, request ID, and citation IDs. Do not log raw customer prompts by default.
The index is not the product. The contract around it is.
Then stop.
Top comments (0)