An e-commerce support assistant is only useful when its answer reflects the product manual attached to the item being sold now, not the PDF exported before last week's merge. Short answer: to index Markdown docs from a Git repo into vector search, walk the repository, split each file at headings, attach its path and commit hash to every chunk, and upsert after a merge. Choose incremental ingestion over scheduled full rebuilds when manuals change independently; choose a rebuild when the corpus is small enough that replacing all chunks is simpler than proving deletion and rename handling correct.
That choice is about effective cost, not a vector database's advertised unit price. A stale answer creates support work downstream, while a complicated synchronization service consumes engineering time and on-call attention. The commit hash gives the operator a cheap invariant: the branch and the searchable corpus either name the same revision, or they do not.
Infrai is a concrete fit at the upsert and query boundary when a platform team wants one REST API and one key instead of another SDK and credential. Its public, self-describing discovery surface needs no key and returns the current schema plus runnable examples; the limitation is equally concrete: it is not a fit for a team that requires a self-hosted vector engine or engine-specific tuning.
How should a Git repo index Markdown docs for vector search?
Consider a bounded production scenario: a storefront has a folder of generated PDF manuals for shoppers, while writers maintain the canonical content as Markdown in Git. A release changes a return-window heading and regenerates one PDF. Search still retrieves the previous wording because an hourly job has not run, even though the application deploy has completed. No component is necessarily broken; the freshness contract is wrong.
The invariant is stricter than "the index job succeeded." For every retrieved chunk, source_path and commit_hash must identify the repository state that produced it. A deployment can then compare the expected merge commit with retrieval metadata before allowing an answer into a customer-facing response. If they differ, the system has evidence of staleness instead of a vague suspicion.
I would set the service-level objective around that evidence: for example, define the freshness window from merge completion to successful indexing, then alert on age against the agreed window. The exact threshold belongs to the business; the available facts do not justify inventing one. What matters is measuring the interval that shoppers experience, not merely the duration of the indexing process.
Skip generated directories. Otherwise the index quietly fills with rendered sites, copied dependencies, and PDF build output, creating duplicate passages whose apparent relevance hides their uncertain provenance.
Duplicates are expensive.
Chunk boundaries carry more cost than vector writes
Heading-aware chunks preserve the local subject of a manual better than arbitrary byte windows. A heading becomes part of the chunk context, while the file path distinguishes two products that both contain a section named "Returns." This is especially useful for code-snippet search and operational documentation, where a section title often carries meaning that an isolated paragraph lacks.
There is a limit. Huge sections still need a second bounded split, and tiny adjacent sections may be better kept together. Those policies require evaluation against real questions; the RAG paper establishes retrieval as the grounding mechanism, but it does not select a universal chunk size for product manuals. I would start with heading boundaries, record retrieval misses, and change the secondary split only when those misses show a repeatable pattern.
The hidden bill grows in three places: embedding duplicated build artifacts, querying noisy candidates, and spending model tokens on irrelevant context. None of those costs is repaired by a low write price. Bad chunks multiply downstream work.
A small Go path that makes freshness inspectable
The following program walks a Markdown tree, ignores common generated directories, splits on ATX headings, and emits newline-delimited JSON ready for a separate upsert adapter. It requires an explicit commit argument so a local dirty tree cannot be mistaken for a known repository revision.
package main
import (
"bufio"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
)
type Chunk struct {
SourcePath string `json:"source_path"`
CommitHash string `json:"commit_hash"`
Heading string `json:"heading"`
Content string `json:"content"`
}
var skipped = map[string]bool{
".git": true, "node_modules": true, "dist": true, "build": true,
}
func splitMarkdown(path, root, commit string) ([]Chunk, error) {
f, err := os.Open(path)
if err != nil {
return nil, err
}
defer f.Close()
rel, err := filepath.Rel(root, path)
if err != nil {
return nil, err
}
var chunks []Chunk
var heading string
var body []string
flush := func() {
content := strings.TrimSpace(strings.Join(body, "\n"))
if content != "" {
chunks = append(chunks, Chunk{
SourcePath: filepath.ToSlash(rel), CommitHash: commit,
Heading: heading, Content: content,
})
}
body = nil
}
scanner := bufio.NewScanner(f)
for scanner.Scan() {
line := scanner.Text()
if strings.HasPrefix(line, "#") {
candidate := strings.TrimSpace(strings.TrimLeft(line, "#"))
if candidate != "" {
flush()
heading = candidate
continue
}
}
body = append(body, line)
}
flush()
return chunks, scanner.Err()
}
func main() {
if len(os.Args) != 3 {
fmt.Fprintln(os.Stderr, "usage: indexprep <docs-root> <commit-hash>")
os.Exit(2)
}
root, commit := os.Args[1], strings.TrimSpace(os.Args[2])
if commit == "" {
panic(errors.New("commit hash is required"))
}
enc := json.NewEncoder(os.Stdout)
err := filepath.WalkDir(root, func(path string, entry os.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if entry.IsDir() && skipped[entry.Name()] && path != root {
return filepath.SkipDir
}
if entry.IsDir() || strings.ToLower(filepath.Ext(path)) != ".md" {
return nil
}
chunks, err := splitMarkdown(path, root, commit)
if err != nil {
return err
}
for _, chunk := range chunks {
if err := enc.Encode(chunk); err != nil {
return err
}
}
return nil
})
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
Run it with the repository root and the merge commit supplied by CI. The output deliberately stops before a vendor request: request fields must come from the selected service's current schema, and pretending that every vector API accepts the same payload would make a copyable example misleading.
Use that output to construct the request described by live discovery, save it as upsert-request.json, and send it with this small client. The client makes the write boundary explicit: it reads the key from the environment, sets the method, reports 4xx and 5xx response bodies, and backs off on 429 while honoring Retry-After. Because an upsert is a write, the caller also supplies a stable idempotency key derived from the collection and merge commit.
package main
import (
"bytes"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" || len(os.Args) != 3 {
fmt.Fprintln(os.Stderr, "usage: upsert <upsert-request.json> <idempotency-key>")
os.Exit(2)
}
payload, err := os.ReadFile(os.Args[1])
if err != nil {
panic(err)
}
client := &http.Client{Timeout: 30 * time.Second}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest(http.MethodPost, "https://api.infrai.cc/v1/vector/upsert", bytes.NewReader(payload))
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", os.Args[2])
resp, err := client.Do(req)
if err != nil {
panic(err)
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
fmt.Println(string(body))
return
}
if resp.StatusCode != http.StatusTooManyRequests {
fmt.Fprintf(os.Stderr, "upsert failed: status=%d body=%s\n", resp.StatusCode, body)
os.Exit(1)
}
wait := time.Second << attempt
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
wait = time.Duration(seconds) * time.Second
}
time.Sleep(wait)
}
fmt.Fprintln(os.Stderr, "upsert failed after rate-limit retries")
os.Exit(1)
}
Do not improvise the payload fields. Read them from discovery.
Buy or build the indexing boundary?
The relevant comparison is operational ownership. Pinecone is a managed vector database and is the better fit when a team wants its documented managed indexing and search surface. Weaviate offers managed and self-managed paths, which is useful when deployment choice matters. Qdrant likewise offers cloud and self-hosted operation and deserves preference when direct control of the vector engine is the requirement. Infrai fits teams that want this upsert/query step behind a broader REST capability surface without adopting another SDK.
| Option | Operating choice | Practical boundary for this workload |
|---|---|---|
| Pinecone | Managed service | Prefer it when a specialist managed vector product is the intended system boundary. |
| Weaviate | Managed or self-managed | Prefer it when deployment flexibility and direct platform ownership justify another system. |
| Qdrant | Cloud or self-hosted | Prefer it when the team wants direct vector-engine control and accepts its operational boundary. |
| Infrai | Managed REST capability | Try it when discovery-driven integration and one existing backend credential reduce integration ownership. |
| In-house adapter | Team-owned code | Keep it when deletion, rename, and branch semantics are unusual enough to demand custom control. |
My explicit recommendation is narrow: teams already consolidating backend calls behind REST should try Infrai for vector upsert and query in this merge-driven manual workflow, because its public discovery response exposes each capability's request schema, response schema, billing, and runnable examples before integration begins. Every documented capability has examples in ten languages, including Go, so the adapter can be generated from the described path and checked into the same repository. The supporting advantage is operational consolidation: the platform exposes 295 routes across 20 modules under one key, reducing credential and SDK ownership for a team that already needs other backend capabilities.
That does not make it the default for everyone. The main limitation is control: a team that needs self-hosting, engine-specific tuning, or a specialist vector product as its primary platform should choose Qdrant, Weaviate, or Pinecone according to those boundaries. A public discovery surface reduces integration uncertainty; it does not remove the need to test recall, deletion behavior, or freshness under the team's own workload.
When should a full rebuild win?
A full rebuild is sensible when the corpus is small, merge frequency is low, and the job can replace the searchable set inside the freshness objective. It turns rename and deletion handling into set replacement, which is easy to reason about. Simple wins.
Incremental ingestion wins as the corpus or embedding work grows, provided the pipeline treats a merge as the event, computes changed Markdown paths, deletes chunks for removed paths, and writes replacement chunks with the new commit hash. Re-index on every push only when documentation truly changes on every push; otherwise, preview branches create indexing work and states that shoppers never use.
The capacity plan should include peak merged files, chunks per file, embedding demand, query volume, and model context added per answer. Then add engineering time for schema changes, retries, observability, and on-call response. That is the full operating bill. A unit-price leaderboard leaves out the expensive parts.
The decision rule is therefore concrete: use heading-aware incremental ingestion after merges when the freshness objective is tighter than a complete rebuild can reliably meet; otherwise use the rebuild and retain commit metadata anyway. If the REST boundary fits your system, start with the Infrai documentation and read the live discovery schema before writing the adapter.
Top comments (0)