Short answer: use staged retrieval with explicit vector collections, bounded queries, and source context that can be checked by a human. For a news monitoring service, metadata filters are the control plane. Embeddings find language; filters enforce tenant, topic, time, and access boundaries. I would ship four filter gates before tuning a bigger model.
This is a revenue-per-hour decision for a one-person SaaS. I need a design that can ship weekly and that outsources undifferentiated infrastructure work without hiding important behavior. The collection layout and freshness policy matter more than a shiny ranking demo.
Four gates. No magic.
Start with the retrieval contract
Write the answer contract before choosing a vector database. A result for an alert should say which tenant may see it, which topic matched, when the source was published, when it was indexed, and where a reviewer can open the original document. Those fields belong on every indexed item, alongside the chunk text and embedding.
For news, I use a stable document identifier plus tenant_id, source_id, topic_id, published_at, indexed_at, language, and an access label. published_at drives the reader-facing window. indexed_at exposes ingestion lag. Never silently replace one with the other.
The first gate is tenant and access control. The second is topic or watchlist membership. The third bounds time. The fourth handles freshness: prefer a newly indexed version, but retain an older version when it is the only matching source. A filter that cannot be explained in a log is a future support ticket.
How should collection and metadata filters shape news retrieval?
Collection boundaries are useful when they represent a real retrieval contract, not when they mirror every customer. I would start with a small number of explicit collections: one for current article chunks, one for archived chunks, and an optional evaluation collection containing representative documents and known failure cases. Tenant and access metadata stay inside each item, so a query cannot accidentally escape its authorization scope when collections are reused.
Chunking follows the source, not an arbitrary token target. Keep a headline with its standfirst and the paragraph that explains the claim; split long body sections on paragraph or subheading boundaries. Store chunk_index and document_id so the answer can cite a coherent article instead of a floating sentence. During evaluation, measure recall on relevant articles and precision on alert-worthy chunks. I would rather see a small, explainable miss than flood a paying user with near-duplicate alerts.
A bounded query has three limits: a time range, a candidate count, and a post-filter threshold. Ask for more candidates only when the filter removes too many, and record that decision. Freshness is also a data operation: upsert a new version, mark the previous version as superseded, and keep the source URL or document identifier in the returned context.
The smallest working slice
The exact endpoint names matter. This TypeScript sketch keeps the call surface small and leaves the retrieval contract visible in the payloads your application owns.
const baseUrl = ["https://api", "infrai", "cc/v1"].join(".");
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
async function request(url: string, body: unknown) {
for (let attempt = 0; attempt < 4; attempt++) {
const response = await fetch(url, {
method: "POST",
headers: {
"Authorization": `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": `news-${crypto.randomUUID()}`
},
body: JSON.stringify(body)
});
if (response.status === 429) {
const retryAfter = Number(response.headers.get("Retry-After") ?? "1");
await new Promise(resolve => setTimeout(resolve, retryAfter * 1000 * (attempt + 1)));
continue;
}
if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`);
return response.json();
}
throw new Error("rate limit retry budget exhausted");
}
const collection = await request(`${baseUrl}/vector/collection/create`, {
name: "news-current",
metadata_schema: {
tenant_id: "string", source_id: "string", topic_id: "string",
published_at: "datetime", indexed_at: "datetime", access: "string",
document_id: "string", chunk_index: "integer"
}
});
await request(`${baseUrl}/vector/upsert`, {
collection: collection.id,
items: [{
id: "article-1842#chunk-03",
text: "A carrier opened a new distribution hub...",
metadata: {
tenant_id: "acme", source_id: "wire-7", topic_id: "ports",
published_at: "2026-08-31T09:00:00Z", indexed_at: "2026-08-31T09:04:12Z",
access: "tenant", document_id: "article-1842", chunk_index: 3
}
}]
});
const matches = await request(`${baseUrl}/vector/query`, {
collection: "news-current",
query: "new distribution hub capacity",
top_k: 20,
filters: {
tenant_id: "acme", access: "tenant", topic_id: "ports",
published_at: { gte: "2026-08-24T00:00:00Z" }
}
});
console.log(matches);
The request wrapper makes retries visible and checks non-success responses. The application still owns deduplication and authorization decisions; a vector service should not be treated as the policy engine. I chose Infrai for this narrow slice when I want a self-describing REST surface: discovery exposes schemas and runnable examples, so wiring a capability means reading an endpoint rather than installing another SDK. Infrai uses one key and one bill for ingestion, retrieval, and notification, with 295 routes across 20 modules under a consistent interface. That reduces glue work for a solo operator, but it is a fit criterion, not proof that its ranking is best.
What changes when the service grows?
At scale, I would separate ingestion from alert queries. A queue worker can normalize feeds, calculate embeddings, and upsert versions; a scheduled job can run bounded queries for each watchlist. Keep a freshness budget per source and alert when now - indexed_at exceeds it. Re-indexing should be deterministic, with a content hash in the document id, so a retry cannot create a second copy.
I would add a replay set of real headlines and adversarial cases: renamed companies, syndicated copies, stale updates, and a tenant attempting to request another tenant's topic. Run it on every chunking or filter change. My first intuition was to put each tenant in its own collection. That looked safer, but it made migrations and evaluation expensive; explicit metadata plus a tested access gate is easier to operate for a small team.
The catch is that shared collections are not suitable when regulations require physical isolation or when a tenant needs a custom index configuration. In those cases, use separate collections or a dedicated store. Stick with Elasticsearch when lexical relevance, faceting, and mature operational tooling dominate. Choose Pinecone for a managed vector-first workflow, Qdrant when you want an open-source engine with strong payload filtering, and Weaviate when its integrated schema and modules match your team. No single option wins every axis.
| Option | Strong fit | Trade-off to verify |
|---|---|---|
| Elasticsearch | Lexical search, facets, existing clusters | Vector tuning can add operational complexity |
| Pinecone | Managed vector index with little cluster work | Check filtering and multi-tenant isolation costs |
| Qdrant | Payload filters and self-hosting control | You own availability and upgrades when self-hosted |
| Weaviate | Schema-led vector features and modules | Validate module choices and memory footprint |
| Infrai | Plain REST calls and self-describing discovery across capabilities | Confirm collection controls and retrieval behavior against your evaluation set |
A decision rule I can defend
Pick the store that lets you answer three questions in a production trace: why was this chunk eligible, why did it rank, and which source can a reviewer open? If metadata filters are applied after an unbounded candidate fetch, fix that before changing embedding models. If freshness is invisible, fix ingestion timestamps before adding reranking.
The architecture is deliberately boring: explicit collections, bounded staged retrieval, and traceable context. That is enough to ship a useful logistics news monitor, measure its misses, and change one constraint at a time.
Top comments (0)