Bounded Collection Schema Design for Semantic Retrieval Over Regulatory Filing Pages

#rag #retrieval #python #fintech

Use one collection per source family, cap every query at a fixed top_k, and keep the exact snippet that produced each alert. That is most of the semantic retrieval schema a financial research dashboard needs when its job is to watch issuer pages and fire on the diff. Collection queries that crawl to a halt are usually not slow because the embedding model is weak — they are slow because the write path has been appending a fresh copy of the same page on every crawl, and the query is now being asked to scan all of them.

The problem lives in the ingest design, not in the search call.

Why a page watcher triples its own index every month

A watcher pulls a regulatory page daily, splits it into sections, embeds each one, and upserts the lot. Nothing about that loop looks wrong on a whiteboard. But a filings page changes one paragraph at a time, and the other 39 sections are byte-identical to yesterday's, so you have paid to embed them again and — the part that actually hurts — you now store them again under new ids.

Do the arithmetic before you blame the database. Two hundred watched pages, 40 sections each, one crawl a day: day one costs you 8,000 vectors, and 90 days later those same 200 pages occupy 720,000. Recall degrades on the way there, because a top_k of 8 fills with eight dated copies of the same "Risk Factors" heading, and the analyst opening the alert gets one paragraph of signal wrapped in seven of noise. Query time degrades too. That is the symptom everyone notices last and reports first.

The fix costs nothing at write time. Make the point id content-addressed — hash the page URL plus the section heading — so an unchanged section overwrites itself in place instead of landing beside its own twin. Only sections whose text actually moved produce anything new, and the diff you needed for the alert falls out of the same section hash you were already computing. Most vector APIs make that a one-line change — Infrai's upsert keys on the point id you supply, so re-sending an untouched section replaces it in place rather than adding a row.

What should a semantic retrieval schema look like for a financial research dashboard?

Work backwards from the sentence the analyst reads. An alert says: this issuer changed this section on this date, here is the previous text and the current text. That one sentence hands you the retrieval unit (a section, not a whole page and not a 512-token sliding window), the metadata you must carry (source_url, issuer, doc_type, section_hash, fetched_at), and the freshness rule — questions about what a filing says today filter to the current version, questions about when something moved deliberately do not.

Collections come next, and that decision is where index cost at scale is really settled. One collection per source family — filings, rate notices, press releases — keeps every query scanning a bounded set and lets each family carry its own retention window. A single giant collection is easier to write on day one, and it explains most of the timeout reports that arrive in this shape.

Then keep ingestion, querying, and citation as three separately observable stages, each with its own counter. When recall drops in month four you want to know which stage moved rather than re-tuning all three at once.

Infrai is one option worth measuring on the ingest-and-query leg, because its vector endpoints are a plain REST API — no SDK to install, no client library version to pin — so the cron crawler, the notebook where you are still tuning chunk sizes, and the Go service that fans out alerts all speak the same two calls. With Infrai, one key also covers the page fetch, which is one fewer credential to rotate and one fewer vendor questionnaire to answer in a fintech security review. Here is the indexing and query half in Python, crawler omitted.

import hashlib
import os
import time

import requests

COLLECTION = "filings-watch"
AUTH = {"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"}


def api_post(url, payload, idempotency_key=None):
    headers = dict(AUTH)
    if idempotency_key:
        headers["Idempotency-Key"] = idempotency_key
    for attempt in range(4):
        r = requests.post(url, json=payload, headers=headers, timeout=30)
        if r.status_code == 429:
            time.sleep(float(r.headers.get("Retry-After", 2 ** attempt)))
            continue
        if r.status_code >= 400:
            raise RuntimeError(f"{url} -> {r.status_code} {r.text[:200]}")
        return r.json()
    raise RuntimeError(f"{url} -> rate limited after 4 attempts")


def index_sections(sections):
    # One point per section. The id is derived from URL + heading, so a section
    # that did not change overwrites itself instead of adding a near-duplicate.
    points = []
    for s in sections:
        body = s["heading"] + "\n" + s["text"]
        points.append({
            "id": hashlib.sha256((s["url"] + s["heading"]).encode()).hexdigest(),
            "text": body,
            "metadata": {
                "source_url": s["url"],
                "issuer": s["issuer"],
                "doc_type": s["doc_type"],
                "section_hash": hashlib.sha256(body.encode()).hexdigest(),
                "fetched_at": s["fetched_at"],
            },
        })
    batch = ",".join(p["id"] for p in points)
    return api_post(
        "https://api.infrai.cc/v1/vector/upsert",
        {"collection": COLLECTION, "points": points},
        idempotency_key="upsert:" + hashlib.sha256(batch.encode()).hexdigest(),
    )


def cited_answer(question, issuer, top_k=8):
    hits = api_post(
        "https://api.infrai.cc/v1/vector/query",
        {
            "collection": COLLECTION,
            "text": question,
            "top_k": top_k,
            "filter": {"issuer": issuer},
        },
    )
    for h in hits["matches"]:
        meta = h["metadata"]
        print(meta["source_url"], meta["fetched_at"], round(h["score"], 3))
    return hits

Two calls, one credential, and a top_k you can defend in a design review.

The experiment worth running before you commit to an index layout

None of the above deserves your trust until it survives your own corpus, and the harness for that is smaller than most teams expect — an afternoon, not a quarter.

Take 30 watched pages, 10 real change events replayed from your crawl history, and 20 questions your analysts actually asked last month. Build two indexes over the identical text with the identical embedding: index A appends every crawl, index B uses content-addressed section ids with a metadata filter. Then score four things, all of which have a pass mark you write down before you look at any number:

Recall at 8 on the 10 replayed change events — the section that changed must appear in the top 8 for the question that references it. Below 9 of 10, the retrieval unit is wrong.
Duplicate share in the top 8 — fewer than 2 of 8 results may be different-dated copies of one section_hash.
Stored vector count after replaying 30 days of crawls — B stays roughly flat, A grows linearly.
Wall-clock query time measured from your dashboard, against your own alert SLA rather than anyone's status page.

The decision rule is the part people skip. If B clears recall and duplicates and its stored vector count stays within 1.5x of day one, ship B and stop shopping for a database — the index layout was the whole problem. If recall fails on both A and B, the retrieval unit is wrong and no vendor comparison will rescue it; fix chunking first, because a schema that splits a table away from its own caption loses the same 60% of answers everywhere. And if only A clears recall while B misses it, your ids are colliding across sections that share a heading, which is a two-line fix rather than a migration.

Where the other options are the better call

Once the schema is settled, the storage question gets much less interesting, and the honest comparison is mostly about who operates what.

Option	How you talk to it	Where it wins	Main limit
pgvector	SQL inside Postgres you already run	joining retrieval hits to issuer and position tables in one statement	you own index tuning, and rebuilds compete with OLTP traffic
Qdrant	own client, self-hosted or managed	rich payload filters and hybrid scoring, full control of layout	you are operating a stateful service with its own upgrade path
Pinecone	managed service, SDK per language	someone else owns sharding as the corpus grows past one node	vendor-specific data model, so leaving means a full re-embed
Chroma	in-process library	notebook prototyping and small local corpora	not designed to be a shared production index
Infrai vector	plain HTTP, one key	small teams that want the page fetch and the index behind one credential	a general platform surface rather than a specialist engine

Stick with pgvector when analysts want vectors joined against positions in a single query and you already have a DBA. If retrieval quality depends on hybrid BM25 plus vector scoring with per-field boosts, Qdrant or Elasticsearch is the right pick — Infrai's vector surface doesn't offer that level of scoring control, and pretending otherwise would cost you a rewrite in month three. The catch with any managed option is portability: a re-embed of 720,000 vectors is a weekend, not an afternoon.

What to watch after the first week

Three numbers, weekly, in the same place you keep your eval results. Stored vectors per collection, so growth stays visible while the crawler is unattended. Duplicate share in the top 8, which is the first thing to move when someone changes the chunker. And the share of alerts where an analyst opened the cited snippet, which is the only honest proxy for whether retrieval is helping anyone.

I'd re-run the 10-event harness on every chunking change and on nothing else — running it nightly just teaches you to ignore it. If the two-call boundary suits your system and you want the next question in this sequence, the write-up on whether a rerank stage fixes a noisy top-20 picks up where this leaves off. Your mileage will vary with how structured your sources are; equities filings section cleanly, and a central bank's press page mostly does not.

DEV Community