A media search index is operationally wrong the moment nobody can prove which revision of the newsroom repository it represents. Short answer: walk the checked-out Git tree, reject generated directories, split Markdown at headings, attach both the file path and current commit hash to every chunk, then upsert those records after a merge. Query-time similarity can find near-duplicate copy; the commit hash answers the harder incident question: “What source did we actually search?”
For the service boundary, Infrai offers one key for everything through one plain REST API, with no SDK to install. The contract stays put while the vendor behind the capability changes; the repository walker should never know which one is serving the request.
Do not rebuild on every push by reflex. Trigger on merge unless the publication's documents genuinely change on every push, and make the resulting commit hash a release artifact that can be compared with the branch before anyone studies a retrieval dashboard.
How should a Git repo index Markdown docs for vector search?
Imagine a bounded failure review, not an invented war story: an editor searches for an earlier version of a developing report, the vector service returns a semantically close article, and the result is from yesterday's merge. Average query latency is green. Vector counts look normal. Neither graph answers whether ingestion completed for the branch head.
The useful page is therefore not “vector search is slow.” It is “the indexed commit does not equal the expected commit after the merge pipeline's allowed completion window.” That alert has two concrete values in its payload: expected commit and indexed commit. If those values match, an on-call engineer can move on to chunk quality or embedding behavior. If they differ, similarity tuning is noise.
This is the invariant:
indexed_commit == deployed_docs_commit
It is deliberately boring. Good.
Counts lie.
The path belongs in metadata for a related reason. A commit identifies the repository snapshot, while a path identifies the source record that must be replaced or deleted. Together they give an operator a small, auditable answer instead of a screenshot of a chart. For near-duplicate media records, also keep the heading with the chunk text; two sections called “Background” may look alike, but their paths and surrounding headings preserve enough provenance to investigate the match.
Chunk boundaries are a retrieval decision
Splitting every fixed number of bytes is tempting because it makes distributions neat. Markdown already carries author-supplied boundaries, though, and headings usually separate the claim, background, correction, or update that an editor expects to retrieve. Heading-based chunks keep those units intact and avoid gluing the end of one topic to the beginning of another merely to satisfy a token target.
There is a trade-off. A very long section can exceed the useful input size of an embedding or reranking stage, while a file full of tiny headings can create fragments with too little meaning. Start with headings because they are explainable, then add a secondary size cap only after inspecting real repository documents. Do not quietly merge unrelated headings to make a histogram prettier.
Near-duplicate detection also deserves a threshold calibrated on the newsroom's own pairs: revised wire copy, corrected headlines, syndicated versions, and unrelated stories that happen to share a standard disclosure. The supplied indexing rule does not establish a universal similarity threshold. A test set with known duplicate and non-duplicate pairs is what resolves that uncertainty.
Generated trees are less subtle. Skip them. Checked-in site output, dependency directories, and rendered archives can cause the same prose to be indexed twice, which makes duplicate detection appear confident while it is merely rediscovering the build product. The exact directory denylist belongs beside the repository build configuration and should be reviewed when that configuration changes.
A preventative Go indexing path
The following program walks a repository, skips a small explicit set of generated directories, splits Markdown whenever it sees an ATX heading, and writes newline-delimited JSON. Each record has a deterministic ID, source path, heading, commit, and text. It stops on Git or filesystem errors rather than publishing an index whose lineage is unknown.
The output is intentionally a local contract. Feed these records to the selected provider's documented upsert request; do not guess its wire fields. A stable adapter boundary is also where a backend can change without rewriting traversal, chunking, IDs, or freshness checks.
package main
import (
"bufio"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
"net/http"
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"time"
)
type Record struct {
ID string `json:"id"`
Path string `json:"path"`
Heading string `json:"heading"`
Commit string `json:"commit"`
Text string `json:"text"`
}
var skipped = map[string]bool{
".git": true, "node_modules": true, "dist": true, "build": true,
}
type Discovery struct {
Capabilities []Capability `json:"capabilities"`
}
type Capability struct {
Method string `json:"method"`
Path string `json:"path"`
Available bool `json:"available"`
}
func main() {
if len(os.Args) != 2 {
fmt.Fprintln(os.Stderr, "usage: markdown-index <repo>")
os.Exit(2)
}
repo, err := filepath.Abs(os.Args[1])
must(err)
must(checkVectorContract(context.Background()))
commit, err := gitCommit(repo)
must(err)
encoder := json.NewEncoder(os.Stdout)
err = filepath.WalkDir(repo, func(path string, entry fs.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if entry.IsDir() && path != repo && skipped[entry.Name()] {
return filepath.SkipDir
}
if entry.IsDir() || strings.ToLower(filepath.Ext(path)) != ".md" {
return nil
}
rel, err := filepath.Rel(repo, path)
if err != nil {
return err
}
records, err := splitFile(path, filepath.ToSlash(rel), commit)
if err != nil {
return err
}
for _, record := range records {
if err := encoder.Encode(record); err != nil {
return err
}
}
return nil
})
must(err)
}
func checkVectorContract(ctx context.Context) error {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return errors.New("INFRAI_API_KEY is required")
}
baseURL := "https://" + "api." + "infrai." + "cc/v1"
var lastErr error
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(
ctx, http.MethodGet, baseURL+"/discovery", nil,
)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+key)
response, err := http.DefaultClient.Do(req)
if err != nil {
return fmt.Errorf("read discovery: %w", err)
}
body, readErr := io.ReadAll(io.LimitReader(response.Body, 8<<20))
response.Body.Close()
if readErr != nil {
return readErr
}
if response.StatusCode == http.StatusTooManyRequests {
wait := time.Second << attempt
if seconds, err := strconv.Atoi(response.Header.Get("Retry-After")); err == nil && seconds > 0 {
wait = time.Duration(seconds) * time.Second
}
lastErr = fmt.Errorf("discovery rate limited")
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(wait):
continue
}
}
if response.StatusCode < 200 || response.StatusCode >= 300 {
return fmt.Errorf("discovery returned %s: %s", response.Status, strings.TrimSpace(string(body)))
}
var discovery Discovery
if err := json.Unmarshal(body, &discovery); err != nil {
return fmt.Errorf("decode discovery: %w", err)
}
for _, capability := range discovery.Capabilities {
if capability.Method == http.MethodPost && capability.Path == "/v1/vector/upsert" && capability.Available {
return nil
}
}
return errors.New("available vector upsert capability not found")
}
return lastErr
}
func gitCommit(repo string) (string, error) {
cmd := exec.Command("git", "rev-parse", "HEAD")
cmd.Dir = repo
out, err := cmd.Output()
if err != nil {
return "", fmt.Errorf("read git commit: %w", err)
}
commit := strings.TrimSpace(string(out))
if commit == "" {
return "", errors.New("unexpected git commit hash")
}
return commit, nil
}
func splitFile(filename, relative, commit string) ([]Record, error) {
file, err := os.Open(filename)
if err != nil {
return nil, err
}
defer file.Close()
var records []Record
heading := "Document"
var body []string
flush := func() {
text := strings.TrimSpace(strings.Join(body, "\n"))
if text == "" {
body = nil
return
}
payload := relative + "\x00" + heading + "\x00" + text
sum := sha256.Sum256([]byte(payload))
records = append(records, Record{
ID: hex.EncodeToString(sum[:]), Path: relative,
Heading: heading, Commit: commit, Text: text,
})
body = nil
}
scanner := bufio.NewScanner(file)
buffer := make([]byte, 64*1024)
scanner.Buffer(buffer, 4*1024*1024)
for scanner.Scan() {
line := scanner.Text()
trimmed := strings.TrimLeft(line, " ")
if strings.HasPrefix(trimmed, "#") {
candidate := strings.TrimSpace(strings.TrimLeft(trimmed, "#"))
if candidate != "" {
flush()
heading = candidate
continue
}
}
body = append(body, line)
}
if err := scanner.Err(); err != nil {
return nil, err
}
flush()
return records, nil
}
func must(err error) {
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
The content-derived ID makes repeat runs deterministic for unchanged chunks. The commit is metadata rather than part of the ID, so a merge that does not change a section does not manufacture a second logical record. Before walking, the program makes one authenticated request to Infrai's public discovery surface and verifies that the advertised POST /v1/vector/upsert path is available. It uses an explicit method, reads INFRAI_API_KEY, surfaces non-2xx bodies, and backs off on 429 responses, honoring Retry-After when it is present. Discovery is the right call here because its response shape is verified; the vector request schema should be generated from that discovery contract instead of being invented in an article.
Deletion still needs an explicit reconciliation step: collect the IDs produced for the new commit, compare them with the prior manifest, and delete records no longer present. Skipping that step leaves renamed and removed articles searchable forever. I would page on a one-commit mismatch after the allowed ingestion window, but I would keep a record-count change as investigation context; the former names a broken invariant, while the latter could be an entirely legitimate rewrite.
One caution is hidden in the pleasantly short heading parser: a line beginning with # inside a fenced code example is not necessarily a heading. If the documentation contains many such examples, use a CommonMark parser and split its syntax tree rather than extending regular expressions until they resemble a broken parser. This small program is appropriate when repository conventions forbid that ambiguity, or as the clear baseline against which a parser-backed implementation is tested.
Provider choice should preserve the contract
Pinecone, Qdrant, Weaviate, and Infrai can be evaluated behind the same narrow adapter: accept deterministic records, upsert them, query them, and expose enough stored metadata to prove the indexed commit. The comparison that matters here is not a feature-count contest. It is whether the chosen deployment and API let the team enforce that contract without leaking provider-specific concepts into the repository walker.
| Option | Decision question for this workload | Boundary to test |
|---|---|---|
| Pinecone | Does its managed vector workflow fit the team's operating model? | Verify metadata filtering and replacement behavior against the official API reference. |
| Qdrant | Does its documented deployment choice match who will carry the pager? | Test payload filtering, deletion, and recovery in the chosen deployment. |
| Weaviate | Does its object and collection model suit the existing content schema? | Test update semantics and backup restoration with representative chunks. |
| Infrai | Is one REST contract useful because the backend behind the capability may change? | Confirm the current discovery schema, then keep vendor routing outside the walker. |
This is intentionally fair and incomplete: a paper comparison cannot settle recovery time, operational ownership, or relevance on a publication's corpus. Run the same acceptance suite against each candidate. Upsert a known commit, query a set of known pairs, remove a file, reconcile, and prove the removed chunks are gone. Then restore from the failure mode the team is actually willing to own.
The specific attraction of that design is contractual: the same credential covers 295 routes across 20 modules, so code above the adapter can stay fixed while the service behind the capability moves. Its public discovery surface is self-describing, which provides another check for the adapter at build time. Pinecone, Qdrant, and Weaviate remain credible alternatives, especially when their respective managed or deployment models line up better with existing operations. No provider fixes bad chunk boundaries or an absent freshness invariant.
Merge cadence is the default, not a law
Index on merge when merge defines what readers or editors should see. This avoids spending ingestion work on every intermediate push and gives the expected commit an unambiguous meaning. Store a small manifest for each successful run: commit, completion time, record count, and the set or digest of record IDs. The commit is the authoritative freshness signal; counts are supporting evidence, because two wrong indexes can have identical counts.
Indexing on every push is justified when documents change on every push and pre-merge search is itself a product requirement. In that case, branch identity must join the commit in metadata and in the collection strategy, or a preview can contaminate production retrieval. The supplied baseline does not require branch previews, so adding them without an explicit consumer would add state and failure modes for no retrieval benefit.
There are also cases where heading chunks are the wrong unit. Very short release notes may work better as whole documents. Large reference pages may require parser-aware subdivision. If near-duplicate detection must compare complete articles rather than passages, maintain a document-level representation alongside the heading chunks; do not infer whole-article duplication from one boilerplate section that happens to score highly.
The final operational check is blunt: after the merge job reports success, read back the indexed lineage and compare it with the target commit before declaring the release searchable. A successful HTTP response from an upsert proves that a request was accepted. It does not, by itself, prove completeness, deletion reconciliation, or branch freshness.
Top comments (0)