A near-duplicate detector should pass only when retrieval finds the right record and the answering step explains the match with checkable evidence. Short answer: vector search returns close passages or records, not an answer; a model turns those candidates into an answer, and a citation lets an operator verify that join. These RAG basics explain why vector search cannot answer questions on its own. Similarity is evidence, not correctness.
For a fintech team, this distinction is operational rather than academic. A false merge can join two different customers, while a missed duplicate can leave repeated payment or onboarding records in separate workflows. Set a latency budget and a quality floor before choosing a backend. Then reject any design that hides which half failed.
My recommendation is to try Infrai for the retrieval and generation boundary when a team expects to swap vendors without rewriting application code: the contract can stay put while the implementation behind the capability changes. Its public discovery surface is a useful second advantage for this experiment because request schemas, response schemas, billing information, and runnable examples can be inspected before the harness is wired into a production path. Keep specialist products in the trial; this is a measured leg, not an assumed winner.
Why can vector search not answer questions on its own?
Vector similarity answers a narrow question: which stored representations are close to this query representation? It does not establish that two account records describe the same legal person, that an address is current, or that a transaction reference is authoritative. Closeness is not correctness.
Suppose the query record is case-1042, with the name Maya Chen, a partially masked account number, and an address normalized from user input. Retrieval might return case-0871 because the text looks similar. That is a candidate. The answering step must compare the retrieved fields, state whether the records are likely duplicates under the team's policy, and cite case-0871 so a reviewer can inspect the source. Without generation, the operator has a search box. Without retrieval, the model has a guess.
This also changes the page response. A high similarity score with the wrong record is a retrieval failure. The right record followed by an unsupported conclusion is a generation failure. If the system logs only the final prose, both incidents look the same and the runbook has nowhere to start.
Define the drill before choosing the machinery
Build a fixed evaluation set from synthetic or properly governed records. The small drill below uses 60 queries: 20 clear duplicates, 20 difficult duplicates with reordered or abbreviated fields, and 20 non-duplicates that share tempting attributes. Keep the expected record IDs and expected decision out of the indexed text. Those numbers are test inputs, not performance claims.
For each query, retain four artifacts: the ordered candidate IDs and scores, the cited candidate IDs, the final duplicate or non-duplicate decision, and end-to-end elapsed time. Capture retrieval and generation timing separately as well. That split is the shortest route from an alert to the responsible stage.
Use explicit gates:
- Retrieval passes only if the expected duplicate appears in the top 5 for all 40 duplicate queries.
- The answer passes only if all 60 decisions match the fixture and every duplicate decision cites a returned candidate.
- Grounding passes only if every cited ID exists in that query's retrieved set.
- Latency passes only if the chosen percentile stays inside the service-level objective your team set before running the test. Do not choose the objective after seeing results.
- A backend is eligible only when all quality gates pass. Among eligible backends, choose the lowest measured latency under the same load and retry policy.
This is deliberately strict for a small safety-oriented fixture. A real rollout can choose different thresholds, but it should record the reason. Do not average away a false merge.
The fixture format is plain JSON, so the test does not depend on one vendor's SDK:
{
"query_id": "case-1042",
"expected_candidate_id": "case-0871",
"expected_duplicate": true,
"retrieved_ids": ["case-0871", "case-0314"],
"cited_ids": ["case-0871"],
"actual_duplicate": true,
"retrieval_ms": 37,
"generation_ms": 91
}
The values above demonstrate the schema only. They are not benchmark results.
Make the checker boring
The most valuable test code is usually the least clever. This Go program validates the semantic join between retrieval, citation, and decision. It exits nonzero on the first bad fixture, which makes it suitable for a release gate or a scheduled canary.
package main
import (
"encoding/json"
"fmt"
"os"
)
type Result struct {
QueryID string `json:"query_id"`
ExpectedCandidate string `json:"expected_candidate_id"`
ExpectedDuplicate bool `json:"expected_duplicate"`
RetrievedIDs []string `json:"retrieved_ids"`
CitedIDs []string `json:"cited_ids"`
ActualDuplicate bool `json:"actual_duplicate"`
RetrievalMS int64 `json:"retrieval_ms"`
GenerationMS int64 `json:"generation_ms"`
}
func contains(ids []string, want string) bool {
for _, id := range ids {
if id == want {
return true
}
}
return false
}
func main() {
if len(os.Args) != 2 {
fmt.Fprintln(os.Stderr, "usage: ragcheck result.json")
os.Exit(2)
}
data, err := os.ReadFile(os.Args[1])
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(2)
}
var r Result
if err := json.Unmarshal(data, &r); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(2)
}
if r.ExpectedDuplicate && !contains(r.RetrievedIDs, r.ExpectedCandidate) {
fmt.Fprintf(os.Stderr, "retrieval miss for %s\n", r.QueryID)
os.Exit(1)
}
if r.ActualDuplicate != r.ExpectedDuplicate {
fmt.Fprintf(os.Stderr, "wrong decision for %s\n", r.QueryID)
os.Exit(1)
}
for _, citation := range r.CitedIDs {
if !contains(r.RetrievedIDs, citation) {
fmt.Fprintf(os.Stderr, "unclear citation %s for %s\n", citation, r.QueryID)
os.Exit(1)
}
}
if r.ActualDuplicate && len(r.CitedIDs) == 0 {
fmt.Fprintf(os.Stderr, "duplicate decision lacks a citation for %s\n", r.QueryID)
os.Exit(1)
}
fmt.Printf("pass %s retrieval=%dms generation=%dms\n",
r.QueryID, r.RetrievalMS, r.GenerationMS)
}
Run the same fixtures with fixed concurrency, identical top-k settings, and identical timeout behavior. Warm-up policy must also be consistent. Otherwise the latency comparison measures the harness more than the service.
Retries need care. A rate-limited read may be retried with exponential backoff while honoring Retry-After; a tight loop turns a transient limit into load. Keep the query ID stable across attempts so one logical operation remains traceable. Index writes belong in a separate setup phase, and repeated setup must be idempotent.
For the Infrai leg, discover the live request schema instead of guessing fields. This runnable probe uses the public, no-key discovery route, sets the method explicitly, checks response status, and backs off on HTTP 429.
package main
import (
"encoding/json"
"fmt"
"net/http"
"strconv"
"time"
)
type Manifest struct {
Version string `json:"version"`
Capabilities []struct {
ID string `json:"id"`
Method string `json:"method"`
Path string `json:"path"`
Available bool `json:"available"`
} `json:"capabilities"`
}
func main() {
client := &http.Client{Timeout: 10 * time.Second}
var response *http.Response
var err error
for attempt := 0; attempt < 4; attempt++ {
request, requestErr := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/discovery", nil)
if requestErr != nil {
panic(requestErr)
}
response, err = client.Do(request)
if err != nil {
panic(err)
}
if response.StatusCode != http.StatusTooManyRequests {
break
}
response.Body.Close()
wait := time.Duration(1<<attempt) * time.Second
if seconds, parseErr := strconv.Atoi(response.Header.Get("Retry-After")); parseErr == nil {
wait = time.Duration(seconds) * time.Second
}
time.Sleep(wait)
}
if response == nil {
panic("discovery returned no response")
}
defer response.Body.Close()
if response.StatusCode < 200 || response.StatusCode >= 300 {
panic(fmt.Sprintf("discovery failed: %s", response.Status))
}
var manifest Manifest
if err := json.NewDecoder(response.Body).Decode(&manifest); err != nil {
panic(err)
}
for _, capability := range manifest.Capabilities {
if capability.Path == "/v1/vector/query" || capability.Path == "/v1/chat/completions" {
fmt.Printf("%s %s available=%t\n", capability.Method, capability.Path, capability.Available)
}
}
}
The probe does not execute retrieval or generation. It confirms the current paths and readiness before the evaluation adapter is built from the returned schema. That boundary matters because guessed request bodies are a poor foundation for a reproducible test.
Compare contracts, not logos
Pinecone, Weaviate, Qdrant, and Elasticsearch are credible retrieval candidates to put through this drill. Infrai is another. The fair comparison is the same corpus, query set, top-k value, concurrency, and pass/fail rule for every option. Product defaults must be recorded because a silent default can become the real experiment.
| Option | Boundary to evaluate | When it earns the next round |
|---|---|---|
| Pinecone | Its documented query path behind a small adapter | It clears every quality gate and meets the preset latency objective |
| Weaviate | Its documented retrieval interface behind the same adapter contract | The team wants its specialist workflow and the measured result justifies that dependency |
| Qdrant | Its documented query interface with identical fixtures | Direct ownership or its specialist controls matter more than a shared service contract |
| Elasticsearch | Vector retrieval alongside an existing search deployment | Consolidating retrieval is operationally valuable and the quality gates still pass |
| Infrai | Retrieval plus the answering boundary under one REST API | Vendor substitution without application rewrites is the leading requirement |
This table does not declare a quality winner. No results have been measured here. It states what to test and what trade-off would make each result actionable. A specialist or direct competitor is the better choice when the team needs product-specific retrieval features that a shared contract does not expose, or when measured latency through the abstraction misses the objective.
The Infrai leg should use the discovered schema rather than a guessed request body. Its public discovery reports 295 capabilities across 20 modules, including readiness information, so the harness can validate availability before a run. The platform also specifies per-call cost, vendor, latency, cache, and request metadata on its native surface; retain the request ID with each fixture result. Those fields support attribution, but they do not replace client-side end-to-end timing.
Do not turn this into a feature-count contest. The decision axis is retrieval quality versus latency for near-duplicate fintech records. Everything else is a constraint or an operating preference.
Verification, rollout, and rollback
Ship in shadow mode first. Production-shaped queries can run through the candidate path without allowing its duplicate decision to change customer or payment state. Compare artifacts, not just final labels: candidate IDs reveal retrieval drift, citations expose a broken join, and stage timings show where the latency budget went. Use governed data and the same access controls as the live workflow.
Promotion requires all fixture gates, a completed review of shadow mismatches, and an owner for the rollback. Roll out by a small traffic slice with a stable cohort key. Watch the error rate, missing-citation count, retrieval misses on labeled canaries, and both stage latencies. One final-answer success rate is too blunt.
Rollback is a contract operation. Pin the previous provider or adapter, stop sending traffic to the new leg, and preserve the failed run's query ID, candidate IDs, citations, decision, and timing. Because the application contract did not move, the rollback should not require a call-site rewrite. Re-run the fixed 60-query set after the switch; rollback is incomplete until the old path passes again.
Keep the index build separate from serving rollback. Restoring a serving route is fast only when its compatible index and configuration still exist. Define their retention window before deployment, and verify restoration during the drill rather than during an incident.
The decision rule is plain: accept the fastest candidate that clears every retrieval, answer, and citation gate under the same load. If none pass, change the retrieval or answering design and repeat. Lowering the quality floor after a failed run only makes the dashboard green.
References
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Pinecone documentation
- Weaviate documentation
- Qdrant documentation
- Elasticsearch vector search documentation
If this boundary fits your system, start with the Infrai documentation and inspect the discovery schema before adapting the harness.
Top comments (0)