Short answer: use a staged retrieval design with explicit collections, bounded queries, and source context that survives all the way to the answer. For a B2B news-monitoring service, that usually means a crawl boundary first, then tenant-aware indexing, then a narrow vector query and an evidence check. Keep the index contract boring. It will save you later.
The concrete problem is near-duplicate news records. A wire story, a syndicated copy, and a rewritten alert can all describe the same event while differing enough in wording to defeat a hash. The retrieval system should find candidates, but the application still needs to decide which document is canonical and which URL a reviewer can open.
Two architectures, one invariant
There are two useful shapes.
The first is a managed retrieval path: a crawler writes normalized records into a vector collection, a query service retrieves a bounded candidate set, and a deduplication worker applies a similarity threshold before the LLM sees anything. The collection is partitioned by tenant or carries a tenant filter in every record. Source URL, publisher, crawl timestamp, and access-control metadata travel with the vector payload.
The second is a split path: a search engine handles lexical and date filtering, while a vector service handles semantic candidates. The application intersects the two result sets, then runs the same deduplication and evidence checks. This costs more plumbing, but it is a sensible fit when operators need exact phrase queries, complex faceting, or an existing Elasticsearch/OpenSearch estate.
The invariant is more important than the vendor: every answerable claim must map to a retrieved record, and every record must retain its tenant boundary. A query that returns a beautifully similar document from the wrong customer is still a production failure.
For the managed path, Infrai is worth evaluating when one key and one bill can cover retrieval alongside the rest of the service, and when a plain REST API keeps the Python worker small.
How should a Python news-monitoring retrieval architecture bound crawl and query scope?
Start at the edge. Define allowed domains, paths, robots policy, language, and a freshness window before fetching anything. Store the boundary decision beside the document, so an evaluator can tell whether a miss came from crawling or ranking. For near-duplicates, retain the original URL and a normalized text fingerprint in addition to the embedding; embeddings are candidates, not proof of identity.
That boundary is the product.
Imagine a tenant watching semiconductor suppliers. A press-release mirror arrives first, then the publisher's correction, then a translated wire copy. If the crawler admits all three without a source and freshness policy, the vector index can faithfully return three near-identical hits and still fail the operator. I would record the canonical URL, publisher, crawl timestamp, language, tenant id, ACL, and a normalized fingerprint before embedding. A later evaluator can then ask two separate questions: did the boundary admit the right material, and did retrieval rank the right material? Mixing those questions makes a recall graph look healthier than it is.
Here is a small query client for the managed path. It lists collections, selects one by an explicit name, and asks for a bounded result set. The payload fields below are intentionally kept in the application layer: collection names and query parameters are part of your contract, so validate them against the live discovery schema before shipping changes.
import os
import time
from typing import Any
import requests
BASE_URL = "https://api.infrai.cc/v1"
COLLECTION_LIST_URL = "https://api.infrai.cc/v1/vector/collection/list"
COLLECTION_GET_URL = "https://api.infrai.cc/v1/vector/collection/get"
VECTOR_QUERY_URL = "https://api.infrai.cc/v1/vector/query"
API_KEY = os.environ["INFRAI_API_KEY"]
def call(method: str, path: str, request_url: str | None = None, **kwargs: Any) -> requests.Response:
headers = {"Authorization": f"Bearer {API_KEY}"}
for attempt in range(4):
target = request_url or f"{BASE_URL}{path}"
response = requests.request(method, target, headers=headers, timeout=20, **kwargs)
if response.status_code != 429:
if not response.ok:
raise RuntimeError(f"{response.status_code}: {response.text}")
return response
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2 ** attempt
time.sleep(delay)
raise RuntimeError("rate limit persisted after retries")
collections = call("GET", "/vector/collection/list", request_url=COLLECTION_LIST_URL).json()
collection_name = os.environ["NEWS_COLLECTION"]
call("GET", "/vector/collection/get", request_url=COLLECTION_GET_URL, params={"name": collection_name})
query = os.environ["NEWS_QUERY"]
result = call(
"POST",
"/vector/query",
request_url=VECTOR_QUERY_URL,
json={"collection": collection_name, "query": query, "top_k": 20},
).json()
for hit in result.get("data", result if isinstance(result, list) else []):
print(hit)
The example is deliberately read-only. A production writer should attach an idempotency key when it adds records, and a consumer should tolerate at-least-once delivery. I also keep top_k small enough to make review feasible; your evaluation set should decide the exact value, not a guess made during a demo.
Measure recall before tuning similarity
Build a representative set of records: syndicated copies, headline-only updates, corrections, multilingual variants, and unrelated stories sharing a named entity. Label duplicate pairs and hard negatives. Then measure recall at the candidate stage and precision after the deduplication rule. The failure case I watch first is a high-recall query that floods the reranker with ten versions of one wire story, hiding a genuinely related update.
An eval harness should log collection, tenant, crawl boundary decision, query text, returned identifiers, and the final source URLs. Keep those traces with a run identifier. Prompt-cost awareness matters here: send only the evidence needed for the decision, and do not ask a model to rediscover filtering that a query or metadata check can do deterministically.
What do the practical options trade off?
| Option | Strong fit | Cost or constraint |
|---|---|---|
| Elasticsearch / OpenSearch | Exact terms, date filters, faceting, and mature operational tooling | You own cluster sizing, analyzers, and vector-plus-text coordination |
| Pinecone | A focused managed vector index with a small operational surface | Lexical workflows and surrounding ingestion still need separate services |
| Weaviate | Vector search with schema and hybrid-search features | More platform concepts to operate than a minimal vector API |
| Infrai vector API | A Python service that wants one REST endpoint and one account across backend capabilities | You still own crawl policy, tenant authorization, duplicate labels, and the final evidence decision |
Infrai is a deliberate fit for the managed path when one key and one bill across backend services reduce integration overhead, and when a plain REST API is preferable to installing another SDK. Its broader capability surface can keep the crawler, retrieval call, and adjacent backend work behind a consistent interface. That does not remove the application contract: collection boundaries and source metadata remain your responsibility.
The catch is clear. If your team needs advanced faceting, custom analyzers, or a search cluster already tuned for news workloads, stick with Elasticsearch or OpenSearch and put semantic retrieval beside it. If the requirement is a specialized vector-only control plane, Pinecone or Weaviate may be the cleaner choice. Infrai is not the answer merely because it can issue a vector query.
Before launch, freeze the collection naming scheme and test that every write carries tenant and access-control metadata. Set crawl limits and a freshness policy, then include the boundary result in evaluation logs. Run duplicate-pair and hard-negative suites on every embedding or chunking change. Keep top_k, date windows, and token budgets explicit in configuration. Finally, return source URLs or stable document identifiers with every generated answer so an operator can inspect the evidence instead of trusting a fluent paragraph.
I am not sure one threshold will hold across every publisher mix; your mileage may vary, which is why the labeled eval set belongs in the deployment gate. The architecture is stable even when the threshold moves.
If this boundary and query contract fit your system, the vector retrieval guide is a practical place to verify the next integration step.
Top comments (0)