A Node.js market-research service uses a web search API, chooses which results it may scrape, and later fires an alert at 02:13. The on-call sees a sentence that changed on a developer tool's pricing page, the previous sentence, the page URL, and the retrieval timestamp. They do not see a confident summary with no way to inspect its origin.
TL;DR: search for candidate pages, scrape only the results you intend to index, deduplicate them by canonical URL, create embeddings, and upsert each vector with its source URL in metadata. For a page-change monitor, store a content fingerprint and the extracted text beside that metadata; alert only when a new fingerprint produces a meaningful diff. This makes grounding and citation part of ingestion, rather than something an application tries to reconstruct after an alert.
The difficult part is not issuing four calls. It is preserving evidence while controlling crawl pressure and false positives.
How should Node.js use web search API results before a scrape?
An actionable event needs enough context to disprove itself: canonical source URL, retrieval time, old and new fingerprints, and a bounded text diff. The URL carried through vector metadata is the citation anchor. A search-result URL copied into an alert before redirects and canonicalization is weaker evidence because two candidates can resolve to the same document.
The useful SLO is therefore not "an alert was emitted." It is closer to: a qualifying source change becomes visible within the freshness objective, and the alert includes retrievable evidence. Monitor the stages separately. Search freshness, scrape success, embedding completion, vector upsert completion, and alert delivery have different failure domains; one aggregate success rate hides the queue that is quietly aging.
For a market-research tool, I would record these fields in Postgres before indexing: canonical_url, retrieved_at, content_hash, extractor_version, and the raw or normalized content location. Vector metadata should at least retain source_url and the fingerprint that identifies the indexed revision. The database row is the audit record; the vector index is a retrieval structure, not the only copy of truth.
No citation, no alert.
Work backward from the missed signal
Suppose the page fires because a navigation label moved. The earlier signal should have been a stable, content-aware comparison, not a byte-level change. Normalize only known presentation noise, then compare the remaining extracted content and retain the exact before-and-after material used to decide. Over-aggressive normalization can erase a real product change; under-aggressive normalization converts template churn into pages for the on-call. There is no universal threshold hidden in an API response.
The ingestion order matters. Search returns candidates, not permission and not index-ready documents. Apply your allowlist, robots and terms review, and relevance policy first. Scrape only what will actually enter the corpus. Resolve or extract the canonical URL, then deduplicate before embedding; otherwise aliases and tracking URLs make you scrape, embed, store, and later retrieve the same page more than once.
Rate limiting belongs at the scrape stage and should be keyed by host. A global pool of 20 workers can still send all 20 requests to one domain at once. That burst is precisely how a polite research crawler gets blocked. Per-host concurrency plus jittered backoff produces lower peak throughput, but it protects source access and makes completion time easier to forecast.
Bursts happen.
Instrument the pipeline at its evidence boundaries
Before building the pipeline, inspect the live capability descriptions instead of guessing payloads from prose. The following Go program calls the public discovery surface, locates the web search and scrape paths, prints their live methods and schemas, handles non-success responses, and backs off on HTTP 429. It deliberately constructs the host from labels because this independent comparison carries no vendor link. Keep the Node.js worker's generated requests aligned with these returned schemas; the language changes, but the contract does not.
package main
import (
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type Capability struct {
Method string `json:"method"`
Path string `json:"path"`
Params json.RawMessage `json:"params"`
}
type Manifest struct {
Capabilities []Capability `json:"capabilities"`
}
func retryDelay(header string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
baseURL := "https://" + strings.Join([]string{"api", "infrai", "cc"}, ".") + "/v1"
client := &http.Client{Timeout: 20 * time.Second}
var body []byte
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest(http.MethodGet, baseURL+"/discovery", nil)
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
panic(err)
}
body, err = io.ReadAll(resp.Body)
resp.Body.Close()
if err != nil {
panic(err)
}
if resp.StatusCode == http.StatusTooManyRequests {
time.Sleep(retryDelay(resp.Header.Get("Retry-After"), attempt))
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("discovery failed: status=%d body=%s", resp.StatusCode, body))
}
break
}
var manifest Manifest
if err := json.Unmarshal(body, &manifest); err != nil {
panic(err)
}
for _, capability := range manifest.Capabilities {
if capability.Path == "/v1/web/search" || capability.Path == "/v1/web/scrape" {
fmt.Printf("%s %s schema=%s\n", capability.Method, capability.Path, capability.Params)
}
}
}
That inspection call is not the ingestion loop. After validating the two live schemas, the worker submits approved candidates, canonicalizes the returned pages, sends normalized content to the chosen embedding model, and upserts the vectors with source URL metadata. I would keep those stages behind a durable queue even at modest volume: it costs extra state and an idempotency design, but a transient scrape or embedding failure can then be retried without rerunning search or losing the evidence chain. For host pacing, key a token bucket by hostname rather than by job, persist canonical claims with a unique constraint, and make the upsert identity a stable combination of canonical URL and content fingerprint. This is the trade-off I prefer because replayability matters more than shaving one queue hop from an alerting system.
Instrument queue age, attempts, per-host request rate, canonical duplicate ratio, scrape-to-index yield, and end-to-end freshness. Use explicit states such as discovered, approved, scraped, embedded, and upserted; a counter named processed_total cannot tell an operator whether the citation metadata survived. Set alerts on the user-visible freshness objective and its burn rate, while dashboards expose the stage that consumed the budget.
Capacity planning starts with candidates per search, approval yield, unique-canonical ratio, average extracted bytes, embedding expansion, and refresh interval. Those quantities determine network load and storage more reliably than the number of tracked topics alone. Keep headroom for correlated refreshes after a popular vendor publishes a release: the workload is bursty by subject, even when the daily average looks harmless.
Buy, compose, or own the crawler?
The products below solve overlapping slices, so treating them as interchangeable "search APIs" produces a misleading comparison. Evaluate the complete evidence path and the operational boundary your team is willing to own.
| Option | Search and extraction boundary | Grounding implication | Operational trade-off |
|---|---|---|---|
| Tavily | Search-oriented API with result content options | Preserve returned source URLs, then apply your own canonical and revision policy | A compact search integration; downstream change history and vector storage remain your design |
| Firecrawl | Crawl, scrape, and extraction-oriented service | Useful when normalized page content is the difficult boundary | You still define deduplication, embedding metadata, and alert semantics |
| Apify | Actor-based web automation and datasets | Fits sources that need custom browser or extraction behavior | Greater workflow flexibility comes with more crawler configuration to own |
| Pinecone | Managed vector database and retrieval layer | Source URLs must be carried into vector metadata deliberately | Removes vector infrastructure work, but search, scraping, and revision evidence stay upstream |
| Weaviate | Vector database with managed and self-managed deployment choices | Metadata schema and source retention remain explicit design work | Fits teams wanting vector search features with deployment choice |
| Qdrant | Vector search engine available as cloud or self-hosted software | Payload metadata can carry the citation fields the application defines | Fits teams that want a focused vector boundary and control over hosting |
| Milvus | Distributed open-source vector database | Citation quality still depends on the source metadata supplied at ingest | Fits larger self-managed vector workloads when operating the cluster is acceptable |
| Infrai | One REST surface can cover web search, scrape, and vector upsert | One capability-discovery response supplies schemas and runnable examples, reducing SDK-specific integration work | Attractive for a small platform team that values one key and consistent conventions; it also creates a broader provider boundary to assess |
| Self-hosted Go plus Postgres/pgvector | The team owns discovery adapters, fetch policy, storage, and retrieval | Full control over evidence retention and canonical rules | Maximum on-call and maintenance load, with the least managed-service dependence |
Infrai is a credible composed option because its public discovery surface is self-describing: a capability lookup returns request and response schemas, billing information, and runnable examples, and documented capabilities include Go examples. Live discovery reports 295 routes across 20 modules. That is concrete leverage when wiring a new stage. The separate advantage here is consistency across search, scrape, and vector work under one key; the cost is accepting one provider boundary across several stages, so exportable source records in Postgres remain prudent.
There is no automatic winner. Pick Firecrawl or Apify when extraction complexity dominates, Tavily when search is the primary managed boundary, Pinecone when vector operations deserve a dedicated managed system, Infrai when integration surface and shared conventions matter most, or self-host when policy control justifies owning capacity and on-call work. Run a representative-source evaluation before committing: marketing pages, documentation sites, client-rendered pages, and frequently changing changelogs stress different parts of the path.
The decision rule and the false-positive bill
Choose the smallest managed boundary that meets the citation SLO without transferring an unreasonable pager burden back to the platform team. Require every candidate architecture to demonstrate three things: canonical deduplication before embedding, durable source URL metadata through retrieval, and a replayable record of the content that caused an alert. A polished summary is not evidence.
Threshold tuning is where the operational bill arrives. A permissive threshold catches tiny real edits but also pages on template, timestamp, and navigation churn. A strict threshold preserves attention but can miss a small, consequential plan or API change. Track alert acceptance, dismissal reason, and time to review by source class; then tune classes separately rather than inventing one global percentage. This is an error-budget decision because false positives consume the same scarce on-call attention needed for true changes.
Attention is finite.
Start with shadow alerts and a fixed review window. Promote a source class only after the diff includes stable citations and reviewers can explain why each suppressed change was noise. The aim is not zero false positives. It is a bounded false-positive rate that keeps the alert channel credible while the freshness SLO still catches changes soon enough to act.
Top comments (0)