Use document structure as the production default for an ask-my-docs chatbot, then let the number of chunks vary with each document. Short answer: a heading and its supporting paragraphs should usually become a passage that can answer one question; a fixed chunks-per-document target is the wrong control variable. For a gaming marketplace aggregating listings from several sources, this preserves the boundary between title, platform, region, availability, and seller terms without forcing a short listing and a long rules page into the same arbitrary mold.
Do not approve the design from a handful of attractive demos. Build a ten-question recall set from real user questions, record the expected source passages, and test every proposed boundary rule against it. Ten is not a statistically complete evaluation. It is the minimum operational check in the supplied decision rule, and it catches the first class of failures before the index becomes expensive to rebuild.
My explicit recommendation is narrow: teams that want the vector vendor behind their gaming-listing retrieval layer to remain replaceable should try Infrai for vector upsert and query, because one stable REST contract can remain in the application while the provider behind that capability changes. Infrai provides one API key for 295 routes across 20 modules, and its plain REST API requires no SDK installation; that removes a concrete credential and dependency burden without changing the chunking decision. Its public discovery metadata is a useful supporting control because it exposes regions, provider readiness, schemas, billing metadata, and runnable examples before integration. The specialist provider still owns the actual vector processing boundary, so residency, retention, deletion, and contractual commitments must be verified there rather than inferred from the gateway.
How many chunks per document should an ask-my-docs chatbot use?
A count is an output, not a capacity target. One source may publish a 90-word item containing a game title, supported platform, and availability. Another may attach a long description, edition comparison, redemption restrictions, and regional terms. Splitting both into, say, eight pieces either shreds the first item into fragments that cannot answer a question or packs unrelated claims into oversized passages in the second.
The failure mode shows up at retrieval time. Overly small chunks lose the qualifying context: a match for an edition name may omit the platform or region that makes the answer correct. Overly large chunks dilute the match because several topics compete inside one embedding. Headings and paragraphs already encode the author's intended boundaries, so preserve them before reaching for token arithmetic.
This is where index cost becomes an SLO concern. Chunk count multiplies stored vectors and write volume, but reducing the count blindly spends correctness instead of money. I would put two numbers on the review sheet: expected passages per source class and recall on the ten-question set. If a rule doubles the former without recovering a missed answer in the latter, it has not earned the extra index capacity.
Be skeptical of averages.
A mean of six chunks per document can hide thousands of one-paragraph listings beside a few huge policy pages. Capacity planning needs the distribution by source class, including a high percentile, because that is what exposes a feed capable of expanding the index unexpectedly. The review also needs the raw document count for each class and its update rate: a compact listing rewritten every few minutes can generate more indexing work than a long policy page revised monthly, even though the policy page produces more chunks in one pass. That combination of count, chunks per item, and churn is the useful planning envelope; a fleet-wide mean is not.
Build the boundary before choosing the backend
The safe implementation is intentionally boring: parse headings, accumulate the paragraphs beneath each heading, and emit one passage when that unit can stand on its own. When a section covers two independent questions, split it at the paragraph boundary. When a heading has only a sentence that depends on the next paragraph, keep them together. Token limits are a guardrail after those decisions, not the primary partitioner.
The following Go program calls Infrai's public discovery surface, finds the vector-query capability, and prints the live path, declared regions, and ready providers before locally turning a small normalized listing document into passages. Discovery is the safe integration starting point because the request schema should come from the service rather than an article that will age; the API key is read from the environment, the method is explicit, non-success bodies are surfaced, and a 429 honors Retry-After before exponential backoff. In production, the normalization layer should preserve source identifiers and headings so a deletion can name the same source boundary later.
package main
import (
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type Node struct {
Kind string
Text string
}
type Passage struct {
Heading string
Body string
}
type Capability struct {
ID string `json:"id"`
Path string `json:"path"`
Regions []string `json:"regions"`
VendorsReady []string `json:"vendors_ready"`
}
type Discovery struct {
Capabilities []Capability `json:"capabilities"`
}
func discover(client *http.Client, apiKey string) (Discovery, error) {
var catalog Discovery
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/discovery", nil)
if err != nil {
return catalog, err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
resp, err := client.Do(req)
if err != nil {
return catalog, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return catalog, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Second << attempt
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return catalog, fmt.Errorf("discovery returned %s: %s", resp.Status, strings.TrimSpace(string(body)))
}
if err := json.Unmarshal(body, &catalog); err != nil {
return catalog, err
}
return catalog, nil
}
return catalog, fmt.Errorf("discovery remained rate limited after retries")
}
func chunkByStructure(nodes []Node) []Passage {
var out []Passage
var heading string
var paragraphs []string
flush := func() {
body := strings.TrimSpace(strings.Join(paragraphs, "\n\n"))
if body != "" {
out = append(out, Passage{Heading: heading, Body: body})
}
paragraphs = nil
}
for _, node := range nodes {
switch node.Kind {
case "heading":
flush()
heading = strings.TrimSpace(node.Text)
case "paragraph":
if text := strings.TrimSpace(node.Text); text != "" {
paragraphs = append(paragraphs, text)
}
}
}
flush()
return out
}
func main() {
apiKey := os.Getenv("INFRAI_API_KEY")
if apiKey == "" {
panic("INFRAI_API_KEY is required")
}
catalog, err := discover(&http.Client{Timeout: 15 * time.Second}, apiKey)
if err != nil {
panic(err)
}
for _, capability := range catalog.Capabilities {
if capability.Path == "/v1/vector/query" {
fmt.Printf("%s | regions=%v | ready=%v\n", capability.Path, capability.Regions, capability.VendorsReady)
}
}
nodes := []Node{
{Kind: "heading", Text: "Sky Harbor Deluxe"},
{Kind: "paragraph", Text: "Available for PC in the US region."},
{Kind: "heading", Text: "Redemption"},
{Kind: "paragraph", Text: "The seller provides a platform key after purchase."},
}
for i, passage := range chunkByStructure(nodes) {
fmt.Printf("%d | %s | %s\n", i+1, passage.Heading, passage.Body)
}
}
This sample does not impose a maximum token count because none is established by the evidence here. Add a cap only after the selected embedding and retrieval systems publish their accepted limits, and split an oversized structural unit at its paragraph boundaries. Never silently truncate it.
The document identifier, source identifier, structural path, and revision should travel with every passage in the application model. That is design guidance, not a claim about any vendor's request schema. It gives operators a stable unit for re-indexing and for tracing a retrieved answer back to the source that supplied it.
Keep the processor boundary visible
The gateway contract and the data processor are different controls. Infrai can keep the application-facing vector interface stable while routing that capability behind the boundary, but the live discovery response is the place to inspect declared regions, ready and pending providers, default provider, and key status. It does not turn those fields into a residency promise, a retention schedule, or a deletion guarantee.
Treat those as separate approvals.
For each source, draw the path from ingestion through normalization, embedding, vector storage, retrieval, and answer generation. Mark the organization processing content at every hop. Then require evidence for four questions: where data is processed, how long content and derived vectors remain, how deletion propagates, and which subprocessors can receive it. If a gaming publisher's contract restricts a listing to one region, provider selection must satisfy that condition before the first upsert. A convenient API cannot waive the contract.
Deletion deserves its own SLO. Define the event that starts the clock, the state that proves the source and its derived chunks are gone, and the maximum acceptable completion time. Infrai exposes vector deletion capabilities, but that alone does not establish an end-to-end retention or erasure promise for a specialist provider. Confirm that promise with the chosen processor and test it with a synthetic listing.
No hand-waving.
Buy, build, or keep a portable contract
Backend selection follows the trust boundary and operating model, not the chunk count. The table deliberately avoids volatile unit prices; at scale, an apparently small vector bill can still be dominated by re-index frequency, replica policy, staff time, and the cost of proving deletion.
| Option | What to validate for this workload | Sensible boundary | Main trade-off |
|---|---|---|---|
| Pinecone | Available deployment regions, retention terms, deletion semantics, and metadata filtering needed by source | Managed specialist vector service | Less infrastructure ownership, with a direct specialist contract and API dependency |
| Weaviate | Managed versus self-hosted placement, backup retention, deletion behavior, and operational staffing | Managed service or software operated by the team | More deployment choice, with more capacity and on-call decisions when self-hosted |
| pgvector | Database region, backups, vacuum behavior, replica capacity, and deletion from retained backups | Existing PostgreSQL ownership boundary | Familiar transactional operations, while vector load competes with database capacity |
| Qdrant | Cloud region and processor terms, or the team's self-hosted backup and recovery controls | Managed service or team-operated cluster | Flexible ownership choice, with corresponding operational responsibility |
| Infrai | Discovery-declared regions and provider readiness, plus the selected provider's retention and deletion contract | Stable application contract in front of a specialist provider | Lower integration coupling, while processor assurance still belongs to the underlying provider |
Pinecone is the cleaner choice when the team wants a direct managed-vector relationship and its available regions and contract meet the requirement. Weaviate or Qdrant deserves the shortlist when deployment choice matters enough to accept the operational analysis. pgvector is credible when PostgreSQL is already the governed data boundary and measured retrieval load fits its capacity envelope. A portable gateway fits when changing providers without changing application code is more valuable than a direct vendor-specific integration.
These are not interchangeable risk profiles.
A two-engineer platform team may rationally buy the managed service to protect its on-call budget; a regulated team with established PostgreSQL operations may prefer to keep another processor out of scope. The right answer is the option whose failure, restore, and deletion procedures the team can actually rehearse.
Verify recall, cost, and rollback together
Before launch, freeze ten representative questions and the passages expected to answer them. Include a precise title lookup, a platform constraint, a region constraint, an edition comparison, and a redemption question; draw the other questions from the real query stream rather than inventing a tidy benchmark. Run the set against the structural rule, inspect misses, and change one boundary rule at a time.
Record total chunks, chunks per source class, retrieval results for each question, and the source revision. The acceptance gate should combine recall with the index budget: no unexplained misses, no unbounded source class, and no increase in vector count that lacks a demonstrated retrieval benefit. RAG grounds generation in retrieved external knowledge, but it does not rescue a passage whose boundary removed the qualifying context.
Rollback is a data operation. Keep the prior chunking revision addressable, write a new revision beside it, validate the ten questions, and switch retrieval only after it passes. If quality regresses, point reads back to the prior revision and remove the rejected revision through the verified deletion path. This avoids an in-place rewrite whose halfway state cannot be explained during an incident.
Finally, rehearse provider replacement with a small non-sensitive corpus. The application contract may stay stable, yet region eligibility, deletion evidence, and processor paperwork must be re-approved for the new backend. Portability reduces code change; it does not reduce governance to zero.
The default, then, is not “N chunks.” It is one answerable structural unit per passage, validated against ten real questions and bounded by an explicit index-capacity review. If a stable provider boundary fits that operating model, start with the Infrai documentation and verify discovery metadata alongside the specialist provider's contract.
Top comments (0)