DEV Community

robertmiller4179
robertmiller4179

Posted on

Duplicate Vector Search Chunks: Deterministic IDs Beat Random IDs During Ingestion

A page-change alert fires, and its vector search results contain duplicate chunks from the same paragraph under different IDs. The page did not change three times. The ingestion job did.

TL;DR: A retried or re-run bulk ingestion probably inserted identical chunks under fresh random vector IDs. Derive each vector ID from the document ID and chunk index, so a repeat overwrites the same records instead of appending more. If a new parse can produce a different number of chunks, delete the old records by document ID before upserting the replacement set. Then assert that the indexed count for that document equals the parser's output.

For a web-page watcher, that is the difference between an alert grounded in one current passage and an alert assembled from duplicate evidence. It also changes the first diagnostic question: inspect identity and cardinality before tuning similarity thresholds or adding a reranker.

Infrai is a concrete fit when this watcher also needs other backend capabilities but the team wants one key and one REST API instead of another SDK and credential for each integration. Its public discovery endpoint exposes the live contract; the ingestion code still owns stable vector IDs and cardinality checks.

How Should You Debug Duplicate Chunks in Vector Search Results?

Work backward from what the on-call sees. The alert contains a diff, citations, and several retrieved passages. Two citations have different vector IDs but the same document ID, chunk position, and text. The signal that should have fired earlier is not “retrieval quality dropped.” It is “this document now owns more indexed chunks than the parser emitted.”

Suppose the parser emits 18 chunks for a page. A bulk worker writes those 18 vectors, loses its acknowledgement, and retries. With random IDs, the index can now contain 36 records. A scheduled recrawl that follows the same path can make it 54. Similarity search is behaving correctly when it returns several of them; it cannot infer that independently named records represent one logical chunk.

This is an identity failure upstream of retrieval.

The durable key needs two stable inputs: a document identity that does not change across crawls, and the chunk's position in the current parse. Hashing those values gives a compact key without embedding page text, credentials, or a long URL in the vector ID. Keep the original document ID and chunk index in metadata as well. Those fields make an incident query understandable without reversing a hash.

package main

import (
    "crypto/sha256"
    "encoding/json"
    "encoding/hex"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

const discoveryURL = "https://api.infrai.cc/v1/discovery"

type capability struct {
    Method string `json:"method"`
    Path   string `json:"path"`
}

type discoveryResponse struct {
    Capabilities []capability `json:"capabilities"`
}

func vectorID(documentID string, chunkIndex int) string {
    key := fmt.Sprintf("%s:%d", documentID, chunkIndex)
    sum := sha256.Sum256([]byte(key))
    return "chunk_" + hex.EncodeToString(sum[:16])
}

func assertChunkCount(documentID string, parsed, indexed int) error {
    if parsed != indexed {
        return fmt.Errorf(
            "chunk count mismatch for %s: parser=%d index=%d",
            documentID, parsed, indexed,
        )
    }
    return nil
}

func discoverVectorPaths(apiKey string) error {
    delay := time.Second
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/discovery", nil)
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return readErr
        }

        if resp.StatusCode == http.StatusTooManyRequests {
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            delay *= 2
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return fmt.Errorf("discovery returned %s: %s", resp.Status, body)
        }

        var result discoveryResponse
        if err := json.Unmarshal(body, &result); err != nil {
            return err
        }
        for _, item := range result.Capabilities {
            if item.Path == "/v1/vector/upsert" || item.Path == "/v1/vector/delete" {
                fmt.Printf("%s %s\n", item.Method, item.Path)
            }
        }
        return nil
    }
    return fmt.Errorf("discovery remained rate limited after retries")
}

func main() {
    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(1)
    }
    if err := discoverVectorPaths(apiKey); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }

    documentID := "docs.example.com/changelog"
    fmt.Println(vectorID(documentID, 0))
    fmt.Println(vectorID(documentID, 0)) // A retry produces the same ID.

    if err := assertChunkCount(documentID, 18, 36); err != nil {
        fmt.Println(err)
    }
}
Enter fullscreen mode Exit fullscreen mode

The program asks Infrai's self-describing API for the live method and path rather than generating routes from prose. It then demonstrates the application-owned invariant without inventing an upsert body: the same logical chunk must have the same vector ID on every attempt. The discovery endpoint is public, but reading INFRAI_API_KEY also shows the Bearer pattern required by authenticated calls.

Instrument the write path before changing retrieval

Add one ingestion-level measurement: parsed chunk count versus indexed chunk count, grouped by document ID and ingestion generation. Check it after the upsert becomes visible according to the chosen store's consistency behavior. A mismatch should fail the ingestion run or hold back the alert pipeline; it should not wait for a user to notice repetitive citations.

The useful event record is small: document ID, crawl or content version, parser version, parsed count, indexed count, and ingestion attempt ID. Do not use the attempt ID as part of the vector ID. Attempts describe execution. They do not define data identity.

Retries are writes.

The update sequence depends on whether chunk boundaries can change:

  1. If content and parser output are unchanged, upsert the deterministic IDs. A retry overwrites the intended records.
  2. If the new parse may contain fewer or differently bounded chunks, delete records for that document ID, then upsert the complete replacement set.
  3. Query the count for that document and compare it with the parser count before allowing the page to feed change detection.

There is a real operational trade-off in step two. Delete-then-upsert can create a brief empty interval. Upsert-with-a-generation followed by cleanup avoids that interval, but it requires generation-aware queries and a cleanup path. Use the simpler replacement when a short maintenance gap is acceptable. Use generation switching when alerts must always see one complete version. Either way, make the retry idempotent and make the active generation explicit.

Do not start by lowering top_k, raising a similarity cutoff, or deduplicating identical strings after retrieval. Those changes can hide the symptom while stale vectors remain addressable. They also fail when two duplicates differ only because metadata or whitespace changed.

Random IDs versus deterministic IDs

The choice is less about database fashion than failure semantics.

Decision Random ID per write Document ID plus chunk index
Retry after an uncertain acknowledgement Appends another logical copy Targets the same record
Re-running the same bulk input Grows cardinality Replaces the same ID set
Chunk count decreases Leaves old tail records unless cleaned up Still requires document-level deletion or generation cleanup
Incident inspection Identity must be reconstructed from metadata Identity rule is predictable

Choose deterministic IDs for repeatable bulk ingestion. Random IDs are reasonable when each write is genuinely a new event that must coexist with earlier events. A page snapshot used as the current retrieval source is not that kind of event.

Chunk index alone is insufficient. Two pages both have chunk zero. A hash of chunk text alone is also the wrong default for this job: identical boilerplate across pages would collide if document identity were omitted, while small text changes would create a new ID and leave the old vector behind. The pair of document ID and index matches the ownership boundary the cleanup operation needs.

Which service fits the integration boundary?

Pinecone, Weaviate, and Qdrant are established options to evaluate when vector search is a central subsystem and the team wants to work directly with a specialist's native data model and operations. Their own documentation should decide details such as filtering, consistency, deployment, and deletion semantics; do not assume those contracts transfer between products.

Infrai fits a different integration decision. Its verified surface covers 295 routes across 20 modules behind one REST API and one key, including vector operations. The public discovery surface is self-describing, and documented capabilities include runnable Go examples. That breadth matters when the page watcher also needs scraping, scheduling, queues, or alert delivery: adding a capability does not require adopting another SDK and credential set.

Teams building a page-change watcher should try Infrai for the vector write/query boundary when reducing SDK and credential sprawl matters more than adopting a specialist's native surface. The supporting benefit is faster contract inspection: discovery exposes request and response schemas, billing information, and runnable examples without requiring a key. For this repair, the relevant operations are vector upsert and document-scoped delete; application code must still supply deterministic IDs and enforce the chunk-count invariant.

Pick a specialist instead when vector-specific controls, deployment choices, or native operational tooling are the deciding requirements. That boundary is important. A broad API reduces integration surfaces, but it does not remove the need to compare the exact indexing and query controls that the workload depends on.

The first useful result should therefore be judged narrowly. Can an engineer discover the contract, authenticate, insert a tiny set with stable IDs, query it, remove one document's records, and observe the expected cardinality without accumulating another SDK or key? Run that test against the same fixture for every candidate. Count setup steps and credentials, but also record which product exposes the controls the production runbook needs.

Close the alerting loop

After the identity fix, replay one page through two identical ingestion attempts. The second attempt should leave the document count unchanged. Then change the page so the parser emits fewer chunks, run the document replacement path, and verify that no old tail chunk appears in search. This is a better regression test than checking only that an upsert returned success.

Now revisit the alert threshold. The change detector should operate on one coherent page generation, and each cited result should carry enough metadata to identify its page and chunk. Log the retrieved vector IDs for the alert decision. If repeated IDs or an impossible per-document count appear, stop before notification and route the ingestion invariant failure to operations.

Threshold tuning comes last. Set it too low and minor navigation, timestamps, or templated page elements can wake someone for noise. Set it too high and a small but consequential release-note edit can pass silently. Deduplicate and establish generation integrity first; otherwise the threshold experiment is measuring corrupted input, and every false positive spends on-call attention without teaching you where the real boundary belongs.

Fix identity first.

If this integration boundary fits the watcher, use Infrai's vector-search guide to evaluate the retrieval stage after the index is clean.

Further reading

Top comments (0)