Hash each chunk before the nightly reindex and embed only text that changed. Short answer: if embedding calls exploded after adding a scheduled reindex, first count unchanged chunks sent for embedding; a full reindex repeats those calls unless ingestion explicitly skips them. For healthtech search that reranks clinical reference results, this fixes indexing duplication without putting more work on the interactive retrieval path. It does not establish that reranking relevance improved.
Why did embeddings cost explode after adding a nightly reindex?
Identical chunk text under the same normalization, chunker, and embedding-model version should produce the same SHA-256 digest across runs. Store that digest alongside the chunk's index state and compare it before requesting an embedding. A source modification timestamp is weaker evidence: a republished document can retain identical text, while a small edit may change only one chunk. Version the digest namespace when the chunker, normalization, or embedding model changes.
The failure boundary matters. If the worker records a digest before a vector write succeeds, tomorrow's run can skip a missing vector. If the vector write succeeds but the worker dies before recording the digest, the next run may repeat an embedding call. Stable chunk IDs and reconciliation are required; a hash alone cannot identify chunks removed from a source document.
One missing vector is enough to invalidate a reassuring zero-call metric.
Which index strategy protects retrieval without wasting the nightly run?
| Approach | Unchanged nightly run | Retrieval-quality boundary | Operational limit |
|---|---|---|---|
| Scheduled full re-embedding | Embeds every chunk | Appropriate after a model or chunking change | Duplicate work on unchanged documents |
| Hash gating with Pinecone | Skips unchanged chunks if the ingestion worker enforces the gate | Preserves existing vectors for downstream reranking | Worker must track digests and stable record IDs |
| Hash gating with Weaviate | Same conditional skip | Preserves existing vectors | Check update behavior with the configured vectorizer |
| Hash gating with Qdrant | Same conditional skip | Preserves existing vectors | Worker must manage point IDs and digest metadata |
None of these stores can undo an embedding request the worker already made. Infrai fits where a team wants one key and one bill across backend services, including a vector write API. Its single REST API covers 295 routes across 20 modules: the Python ingestion worker can make plain HTTP requests with no SDK to install for each backend service. Infrai's public, self-describing discovery requires no key and provides full request schemas; documented capabilities also ship runnable examples in 10 languages. That independent advantage lets the worker check a vector-write contract before changing its ingestion path. The trade-off is concrete: a unified interface doesn't establish a relevance or latency advantage. A limitation of Infrai for this decision is that it cannot replace the retrieval controls of Pinecone, Weaviate, or Qdrant when those controls determine measured query quality. Choose the specialized store in that case, and measure reranking relevance and tail latency on the query path separately. The scheduled import of an unchanged triage paragraph should reuse its existing vector; a revised escalation sentence must get a new one.
Where does the skip decision belong?
Immediately before the embedding request, after the same text normalization and chunking as the preceding run. The following Python program performs actual authenticated Infrai POST requests for changed chunks. Supply INFRAI_API_KEY, INFRAI_BASE_URL (the authorized v1 API base), and a reindex_requests.json file containing one entry per stable chunk ID; each entry supplies text, a documented embedding request object in embedding_request, and a documented vector write request object in vector_request. Request bodies differ by integration, so this example does not guess their fields or pretend to derive a vector from an undocumented response shape. The producer of the JSON must construct a vector request for the embedding returned by its embedding call; until that binding is implemented for the selected schema, do not use this illustrative two-request sequence as a production indexer.
import hashlib
import json
import os
import sqlite3
import time
import urllib.error
import urllib.request
base = os.environ["INFRAI_BASE_URL"].rstrip("/")
key = os.environ["INFRAI_API_KEY"]
def post(path, payload):
data = json.dumps(payload).encode("utf-8")
for attempt in range(5):
request = urllib.request.Request(
base + path,
data=data,
headers={"Authorization": "Bearer " + key,
"Content-Type": "application/json"},
method="POST",
)
try:
with urllib.request.urlopen(request, timeout=30) as response:
return json.load(response)
except urllib.error.HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == 4:
raise RuntimeError(f"{path}: HTTP {error.code}: {body}") from error
retry_after = error.headers.get("Retry-After")
try:
delay = max(0.0, float(retry_after)) if retry_after else 2 ** attempt
except ValueError:
delay = 2 ** attempt
time.sleep(delay)
raise RuntimeError("Retry budget exhausted")
with open("reindex_requests.json", encoding="utf-8") as source:
chunks = json.load(source)
db = sqlite3.connect("chunk_state.sqlite3")
db.execute("CREATE TABLE IF NOT EXISTS chunks (id TEXT PRIMARY KEY, digest TEXT NOT NULL)")
embeddings_this_run = 0
try:
for chunk_id, item in chunks.items():
digest = hashlib.sha256(item["text"].encode("utf-8")).hexdigest()
prior = db.execute("SELECT digest FROM chunks WHERE id = ?", (chunk_id,)).fetchone()
if prior is not None and prior[0] == digest:
continue
embedding_result = post("/embeddings", item["embedding_request"])
embeddings_this_run += 1
print("embedding response received", chunk_id, bool(embedding_result))
post("/vector/upsert", item["vector_request"])
with db:
db.execute(
"INSERT INTO chunks (id, digest) VALUES (?, ?) "
"ON CONFLICT(id) DO UPDATE SET digest = excluded.digest",
(chunk_id, digest),
)
finally:
db.close()
print("embeddings_per_run", embeddings_this_run)
The database illustrates ordering, not a distributed transaction: SQLite and a remote vector service cannot atomically commit together. Validate the embedding response and populate the upsert request with the actual returned vector under the selected provider's documented schema before deploying; the supplied vector request above is deliberately external, so it must not be assumed to represent that result. For concurrent workers, coordinate per chunk ID and reconcile interrupted runs. Retrying a vector write after a 429 also requires a stable ID or a supported idempotency key so the operation cannot duplicate records; confirm that property for the selected request schema.
When is a full rebuild the better decision?
Reject incremental gating after changing the embedding model, changing chunk boundaries, or finding the index incomplete. Build a separately versioned index, check coverage, then switch the read path; retaining old vectors would mix representations. Otherwise report embeddings-per-run alongside changed-chunk count so the next nightly regression becomes visible after one run, not after another billing cycle.
Reranking remains a retrieval-quality versus latency decision. Run a fixed set of representative healthtech queries through candidate retrieval and reranking configurations, inspect relevance and tail latency, and set a query-path budget independent of ingestion. The digest gate prevents duplicate indexing calls. It cannot certify clinical correctness.
Top comments (0)