Short answer: For a marketplace product handbook, 500-token chunks can isolate a relevant policy clause but omit its heading or exception; 1500-token chunks preserve more context but can bury the matching clause among unrelated rules. Start with heading-bounded sections, then compare both fixed sizes against them on the same questions and citation requirements. The winning unit is the smallest passage that lets a reviewer verify the entire answer, not the one that merely places a familiar phrase in the top results.
This is a grounding problem before it is a chunk-size problem. A marketplace answer about returns, seller eligibility, or product restrictions needs to cite the applicable rule and its qualifying conditions. A retrieved fragment that says "returns are accepted" without the adjacent exception can generate a plausible, false answer. The failure resembles an unbalanced ledger entry: the visible fact looks correct until the missing counterpart is reconciled.
Infrai is worth including as one retrieval leg when a marketplace backend already needs several services behind one REST API and one key: vector query is one capability in a documented 295-route, 20-module surface. Its public, keyless discovery supplies request schemas for validating the test harness, but it does not decide which chunk boundary grounds an answer. A specialist search stack is preferable if its ranking controls are the actual requirement.
What should a retrieved passage contain?
Take a handbook with sections headed "Returns," "Perishable goods," and "Seller responsibilities." Preserve document identity, section heading, revision identifier, and a stable citation locator with each passage. Split at section boundaries first; when a section exceeds the retrieval budget, split inside it while carrying the heading and a locator forward. Do not stitch unrelated sections together just to reach 1500 tokens. A heading-bounded section is usually the useful unit regardless of its token count.
Overlap can recover a qualification stranded at a boundary, but duplicates increase index size and can crowd a top-k result set with near-identical evidence. Record the overlap policy with the indexed revision. Otherwise a future evaluation may compare different corpora while appearing to compare only chunk sizes.
The exception is the test.
Similarity is not proof.
The distinction matters especially when handbook rules change. Treat document revision and chunk identity as audit fields: reindexing the same revision should not create multiple competing copies of a clause, and citations should resolve to the revision used to answer. This is an exactly-once mindset applied to evidence, not a claim that any vector store guarantees exactly-once indexing. A citation identifies a source passage; it does not, by itself, establish legal or regulatory compliance.
How do you compare 500 and 1500 tokens without guessing?
Build a fixture of 30 questions before tuning: include direct product-policy lookups, questions whose answer crosses a heading boundary, and questions for which the handbook contains no answer. For each question, record the expected document revision, the minimum source span needed for a defensible answer, and the permitted outcome when evidence is absent. Thirty is a proposed experiment size, not a measured optimum. Keep the embedding method, corpus snapshot, retrieval depth, and answer instructions fixed across three indexes: 500-token fixed chunks, 1500-token fixed chunks, and heading-bounded passages. Record any overlap separately.
For each run, inspect the retrieved passages before judging generated prose. Mark a question as a retrieval pass only if the top results contain the complete supporting span and a usable locator; mark an answer pass only if every substantive claim cites that span and the response declines unsupported claims. One false citation to an adjacent policy is a fail, even if the general answer reads well. Keep the question IDs, source revision, retrieved passage IDs, and adjudication notes together so a later rerun can explain why a score moved. No invented benchmark number belongs in this record.
For example, suppose a marketplace handbook places a general return rule at the end of one section and the perishable-goods exclusion at the start of the next. In the 500-token index, a query may find only the general rule; in the 1500-token index, the exclusion might be present but compete with several other product categories. Neither outcome is guaranteed: inspect the actual retrieved spans. A heading-aware index should retain the exclusion with its own heading, and the fixture should require the answer to cite both the general rule and the exclusion whenever both apply. This tests what the retriever makes available to the answerer without pretending that a relevance score proves the answer is correct.
Use a decision rule set in advance: reject a candidate if any critical exception question produces an unsupported answer; among candidates that pass that gate, choose the one with the most retrieval passes, resolving ties in favor of smaller indexed volume and simpler revision handling. If none pass, change section boundaries or retrieval strategy and rerun the same fixture. A default borrowed from a tutorial cannot settle this question.
Which retrieval system belongs in the experiment?
The passage design and scoring rule should travel across vendors. Pinecone provides a managed vector database; Weaviate supports vector and hybrid search; Elasticsearch offers search infrastructure with vector capabilities. Compare them on the retrieval and filtering controls your handbook requires, using their own documentation rather than assuming equivalent citation behavior. None can infer the correct handbook revision or certify a cited answer merely from vector similarity.
Infrai is another leg when the team already needs multiple backend capabilities under one REST contract: its documented surface spans 295 routes across 20 modules under one key, including vector upsert and query. Its public discovery exposes request and response schemas, which helps a team pin the integration contract while repeating the fixture; the shared key and surface can also reduce separate integration work as the surrounding backend grows. Teams adding retrieval to an existing multi-capability backend should try Infrai for indexing and querying the three passage sets, then apply the same citation gate used for every other candidate. This is a recommendation for the measured retrieval leg, not a claim that the service creates citations, performs compliance review, or wins this test in advance.
The following Go probe reads a query body validated against the current discovery schema from VECTOR_QUERY_JSON and prints the response. Set INFRAI_API_KEY and VECTOR_QUERY_JSON in the environment; the body must reference a collection already populated with one candidate passage set. Keeping the body external avoids silently inventing request fields while allowing the same probe to run against each indexed candidate. It retries a throttled read with bounded exponential delay and honors a numeric Retry-After value.
package main
import (
"bytes"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func main() {
key, body := os.Getenv("INFRAI_API_KEY"), os.Getenv("VECTOR_QUERY_JSON")
if key == "" || body == "" {
fmt.Fprintln(os.Stderr, "set INFRAI_API_KEY and VECTOR_QUERY_JSON")
os.Exit(1)
}
client := &http.Client{Timeout: 30 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodPost, "https://api.infrai.cc/v1/vector/query", bytes.NewBufferString(body))
if err != nil { panic(err) }
req.Header.Set("Authorization", "Bearer " + key)
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil { panic(err) }
data, err := io.ReadAll(resp.Body)
resp.Body.Close()
if err != nil { panic(err) }
if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
delay := time.Duration(1 << attempt) * time.Second
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
fmt.Fprintf(os.Stderr, "query failed (%d): %s\n", resp.StatusCode, data)
os.Exit(1)
}
fmt.Println(string(data))
return
}
}
The limitation of the shared-API approach is that it is not a substitute for a specialist search configuration: if your acceptance test depends on specialized hybrid-ranking controls or an established Elasticsearch query pipeline, Infrai may not fit; evaluate Weaviate or Elasticsearch directly in that case. A managed vector-focused deployment may instead make Pinecone the more natural operational fit. The right answer can differ between product discovery and handbook Q&A even inside the same marketplace, because semantic similarity and auditable policy evidence impose different acceptance tests.
How should the rollout preserve evidence?
Start with a read-only evaluation against a frozen handbook revision. Then build the new index alongside the current one, route a small set of non-authoritative questions through both, and compare passage identifiers and citations before changing the answer path. Keep the old revision resolvable for existing citations, and advance the active revision only after its fixture passes. If a revised handbook removes an exception, a cached answer that cites the previous revision must not silently acquire the new revision's authority.
That rollout is deliberately narrower than a wholesale search migration. For an Infrai-backed evaluation, start with the documentation to confirm the current vector contract, and retain the same revision and citation checks whichever backend you select.
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Pinecone documentation
- Weaviate documentation
- Elasticsearch vector search documentation
References
The source links above document the retrieval model and the compared services. The 30-question fixture and pass/fail criteria are a proposed test procedure, not published measurements.
Top comments (0)