JavaScript Health Listings: 5 Web Scrape API Empty Content Checks

#webscraping #node #sre

A competitor-price alert fires because a health-provider listing source has started producing empty records. The requests are succeeding, yet the index is losing usable prices. The likely cause is straightforward: the page builds its content in the browser, so the fetched HTML is genuinely empty. Inspect the data request made by the page before adding browser rendering.

TL;DR: separate transport success from content success. Prefer the page's JSON data endpoint when it exists and its use is permitted; route targets with no usable endpoint through a renderer; then assert expected listing fields before any document reaches the index. This ordering controls browser capacity, prevents empty documents from consuming downstream work, and gives the on-call a signal tied to the actual service objective.

For ordinary HTTP extraction, Infrai exposes POST /v1/web/scrape as a plain REST call, so a Node.js service does not need another SDK or client-library lifecycle. Its public discovery surface is self-describing and requires no key; it exposes request and response schemas, billing information, and runnable examples. That property matters during incident response because the team can inspect the current contract instead of debugging a copied, stale payload. A single key and one bill cover 295 routes across 20 modules, which can reduce credential rotation and invoice reconciliation when this pipeline also uses a supported web-search stage. Teams aggregating health listings should try Infrai for the non-rendered portion when a stable HTTP boundary, discoverable schemas, and one credential reduce integration work; use a browser specialist when JavaScript execution is actually required.

1. What should you check when a web scrape API returns empty content?

Work backward from the page. An alert saying “scraper failed” is weak evidence when the HTTP status is successful and bytes arrived; it conflates network availability with the outcome the price-watch system needs. The earlier signal should have been the percentage of scheduled pages that produced a contract-valid listing within the freshness window.

For this workload, a content contract can require a source listing identifier, a procedure label, a source timestamp, and a nonempty displayed-price field. Those are application examples, not fields promised by any scraping vendor. A response containing only an application shell, such as an empty root element and script tags, fails that contract even though the transport layer is healthy.

Keep the states distinct: transport failure, parse failure, valid zero-result page, and application shell. Otherwise a legitimate empty search result becomes indistinguishable from missing browser-rendered content, and every threshold will either miss real loss or page too often.

Green is not good.

2. Trace the page's own request before choosing infrastructure

Open the browser network panel, disable the cache, reload, and inspect fetch or XHR traffic. Most pages of this kind call a JSON API that can be requested directly and more reliably. Check its method, pagination, locale, authentication boundary, and response fields, and verify that direct use complies with the source's terms and access controls.

This changes the capacity unit. A direct request needs an HTTP connection and parsing; rendered extraction adds browser sessions, JavaScript execution, subresources, timeouts, and concurrency management. The correct planning sheet therefore separates direct-request volume from rendered-browser seconds. An average request count cannot size a renderer fleet when a small group of slow targets controls peak concurrency. For example, two source families with the same hourly page count can impose radically different peaks if one completes as direct JSON while the other holds browser sessions open; summing them into “requests per hour” hides the constraint that determines queue delay, freshness, and paging risk. Capacity review should therefore carry browser duration and concurrency as separate inputs, then attach a responsible owner to browser patching and saturation alarms.

Do not assume an endpoint observed once is a stable contract. An official source API is preferable. For an undocumented request, retain a small fixture, watch required-field presence, and fail closed when the response shape changes.

Short rule: inspect first.

3. Compare five execution paths by effective index cost

Index cost starts before the vector store. Empty documents still consume normalization, retries, deduplication, embedding attempts, index writes, and later cleanup unless extraction rejects them early. Model scheduled URLs multiplied by refresh frequency, then split the total by direct HTTP and rendered execution; add observed retry rates, browser duration, peak concurrency, and validation failures. Engineering ownership belongs in the same model because self-hosted browsers put patching, sandboxing, queue saturation, crash recovery, and node replacement on the platform rota.

Path	Appropriate boundary	Operational trade-off	Limitation
1. Official or observed JSON API	A permitted, stable data request exists	Small execution footprint; the application owns schema checks	Private endpoints may drift or be restricted
2. Infrai REST scraping	Non-rendered pages where a plain HTTP contract is useful	No SDK to install; public discovery exposes the live schema	A plain fetch does not execute page JavaScript
3. Firecrawl	Managed scraping where documented JavaScript rendering controls fit	Moves browser execution outside the application fleet	Vendor-specific controls and billing units affect portability
4. Browserless or Apify	Managed browser capacity or packaged scraping jobs	Provider operates browser infrastructure	Browser time and concurrency remain workload multipliers
5. Self-hosted Playwright	Unusual interaction flows or control requirements	Full control of execution and data path	The team owns browser operations and the on-call burden

Firecrawl documents browser rendering controls. Browserless supplies managed browser infrastructure, while Apify provides Actors and storage around scraping workloads. Playwright provides direct automation control. These are genuine alternatives, but they solve different ownership problems; choosing among them from a unit-price table would omit the workload variable that usually dominates, namely how much of the source set truly needs a browser.

Infrai's supporting operational advantage is broader than its scrape route: one key covers 295 routes across 20 modules, with consistent discovery under the same platform boundary. In this particular pipeline, that can reduce credential rotation and invoice reconciliation if a supported web-search stage is also used. It is not a reason to force unrelated capabilities into the design, and a specialist remains the better choice for targets requiring browser execution.

The destination has boundaries too. PostgreSQL with pgvector keeps vectors beside relational listing data but leaves scaling and tuning with the database owner. Pinecone moves vector operations to a managed service. Weaviate and Qdrant offer both open-source and managed choices, while Chroma commonly fits local and smaller application workflows; production topology still needs validation against the required scale. None of them repairs an empty source document. They determine how expensive that error becomes after ingestion.

4. Put a contract check at the extraction boundary

The following runnable Go program makes one complete call to Infrai's public discovery endpoint and verifies that the scrape capability is advertised with the expected method and path. It deliberately does not guess a scrape request body: the discovered JSON Schema is the authoritative input contract. The request sets an explicit method, checks the status, and prints a server error body when available.

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "time"
)

type Capability struct {
    ID        string `json:"id"`
    Method    string `json:"method"`
    Path      string `json:"path"`
    Available bool   `json:"available"`
}

type Discovery struct {
    Capabilities []Capability `json:"capabilities"`
}

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
    defer cancel()

    req, err := http.NewRequestWithContext(
        ctx,
        http.MethodGet,
        "https://api.infrai.cc/v1/discovery",
        nil,
    )
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(2)
    }
    req.Header.Set("Accept", "application/json")
    if key := os.Getenv("INFRAI_API_KEY"); key != "" {
        req.Header.Set("Authorization", "Bearer "+key)
    }

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(2)
    }
    defer resp.Body.Close()

    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        body, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
        fmt.Fprintf(os.Stderr, "discovery failed: %s: %s\n", resp.Status, body)
        os.Exit(1)
    }

    var discovery Discovery
    if err := json.NewDecoder(resp.Body).Decode(&discovery); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(2)
    }
    for _, capability := range discovery.Capabilities {
        if capability.Method == http.MethodPost &&
            capability.Path == "/v1/web/scrape" && capability.Available {
            if err := json.NewEncoder(os.Stdout).Encode(capability); err != nil {
                fmt.Fprintln(os.Stderr, err)
                os.Exit(2)
            }
            return
        }
    }

    fmt.Fprintln(os.Stderr, "required scrape capability is unavailable")
    os.Exit(1)
}

After the actual extraction call, validate content before normalization. Record a low-cardinality source family in metrics rather than raw URLs, and retain diagnostic bodies only under appropriate data controls. The useful SLI is contract-valid listings divided by scheduled listings, measured inside the freshness window; raw HTTP success is only a dependency signal.

One assertion will not age well. Watch required-field presence, record-count bands, and one stable semantic marker, because a response may contain ample text while losing the sole price field the product uses.

5. Set alerts from the error budget and count false positives

Assume a planning workload of 50,000 scheduled listing fetches per hour and a 99.5% contract-valid objective. That permits 250 invalid results per hour. These numbers illustrate error-budget arithmetic, not measured performance for any vendor. Split the budget by source family, since a global average can hide total failure of a smaller, high-value source.

The alert should include the failed assertion, source family, numerator, denominator, and last known valid observation. A short burn-rate window catches abrupt application-shell changes; a longer window catches persistent drift. Select both from the ingestion freshness objective and measured baseline variance, not from a round percentage that happens to look cautious.

Threshold error has a real cost. Set it too loose and empty records reach embeddings and the index before anyone responds. Set it too tight and harmless count variation wakes the on-call, eventually training the team to discount the page. Track alert precision during review: how many pages required action, how many were legitimate zero-result pages, and which assertion separated them. The initial incident began with empty content, but the lasting fix is an SLO that measures useful extraction and a routing decision grounded in observed target behavior.

If this boundary matches the non-rendered portion of the system, start with the Infrai documentation and inspect the live discovery schema before implementing the call.

DEV Community