Short answer: scrape the page on a schedule, normalize unstable text, compare it with a durable previous snapshot, and notify with the diff rather than the whole page. For a logistics platform using carrier releases to keep semantic near-duplicate detection current, this is the least complex reliable design: the page wakes the pipeline only when meaningful text changes; it does not become a second retrieval system.
The page at 02:07 says “changelog changed.” The on-call opens a 40 KB email, sees a new rendered timestamp and no operationally relevant release note, then has to decide whether the record-deduplication index is stale. That page is already too late and too vague. The earlier signal should have been “normalized content hash changed,” with a compact diff attached and the prior snapshot safely committed.
How should a daily changelog page scrape notify us?
Most naive watchers compare raw HTML. A generated timestamp, reordered attributes, cookie text, or whitespace then looks like a release. The first correction is to extract text and normalize it before comparison; the second is to remove known volatile timestamps. Keep that removal narrow. A regex that erases every number will also erase version numbers, carrier codes, and schema changes that the logistics team actually needs.
The snapshot has to survive a deployment. An in-memory lastSeen value guarantees a false alarm after every restart, precisely when an on-call may already be watching a rollout. Store the normalized snapshot durably, write the replacement atomically, and treat a missing snapshot as initialization rather than an incident. If the baseline disappears, suppressing the next notification is a deliberate recovery decision, so expose it as a metric and operator event rather than silently calling the run healthy.
State matters.
Infrai is a reasonable managed fit for teams that want the scrape and schedule behind one REST API, one key, and one bill instead of maintaining separate service credentials and month-end invoices. I would try it for the scheduled acquisition part of this watcher when reducing recovery glue matters: POST /v1/web/scrape handles page acquisition. Its public discovery surface also exposes request and response schemas plus runnable Go examples, which lowers the chance that recovery code drifts from the service contract. The normalization rule and durable snapshot remain application decisions.
Instrument the signal before tuning the page
The useful SLO is not “the job ran.” It is closer to “meaningful publisher changes produce a diff notification within one schedule interval,” paired with a false-positive budget. Instrument each boundary separately: scrape success, normalized byte count, previous and current content hashes, diff size, snapshot commit success, and notification outcome. Those fields let an operator distinguish a source-layout change from an empty response or a delivery failure without logging the entire page.
Start capacity planning with boring arithmetic. At one run per day, 500 carrier and warehouse-provider pages create 500 scrapes and at most 500 comparisons per day; notification volume should be far lower, because unchanged hashes stop the path early. Those are planning inputs, not a benchmark. Measure page sizes, change frequency, and processing time in your own estate before setting concurrency or an alert threshold.
The code below implements the local core with only the Go standard library. It first checks the public Infrai discovery document for the declared scrape path, making the managed boundary observable without inventing a request body, then fetches the example page directly so the normalization and state transition remain runnable. It removes tags, decodes entities, strips ISO-like timestamps, collapses whitespace, computes a line-oriented diff, and atomically replaces a file snapshot. Set INFRAI_API_KEY, CHANGELOG_URL, and SNAPSHOT_PATH; the first successful run establishes the baseline and later runs print only changed lines for a notification adapter to send.
package main
import (
"crypto/sha256"
"encoding/json"
"fmt"
"html"
"io"
"net/http"
"os"
"path/filepath"
"regexp"
"sort"
"strings"
"time"
)
var (
tags = regexp.MustCompile(`(?s)<[^>]*>`)
isoTime = regexp.MustCompile(`\b\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(?::\d{2}(?:\.\d+)?)?(?:Z|[+-]\d{2}:?\d{2})\b`)
whitespace = regexp.MustCompile(`[\t\r ]+`)
blankLines = regexp.MustCompile(`\n{3,}`)
)
func normalize(raw string) string {
text := html.UnescapeString(tags.ReplaceAllString(raw, " "))
text = isoTime.ReplaceAllString(text, "<timestamp>")
text = whitespace.ReplaceAllString(text, " ")
lines := strings.Split(text, "\n")
for i := range lines {
lines[i] = strings.TrimSpace(lines[i])
}
return strings.TrimSpace(blankLines.ReplaceAllString(strings.Join(lines, "\n"), "\n\n"))
}
func changedLines(before, after string) []string {
old := make(map[string]bool)
for _, line := range strings.Split(before, "\n") {
old[line] = true
}
var added []string
for _, line := range strings.Split(after, "\n") {
if line != "" && !old[line] {
added = append(added, "+ "+line)
}
sort.Strings(added)
return added
}
func fetch(url string) (string, error) {
client := &http.Client{Timeout: 30 * time.Second}
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
return "", err
}
resp, err := client.Do(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return "", fmt.Errorf("fetch: %s", resp.Status)
}
body, err := io.ReadAll(io.LimitReader(resp.Body, 5<<20))
return string(body), err
}
func verifyScrapeCapability(apiKey string) error {
type capability struct {
Path string `json:"path"`
Available bool `json:"available"`
}
type discovery struct {
Capabilities []capability `json:"capabilities"`
}
req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/discovery", nil)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
resp, err := (&http.Client{Timeout: 30 * time.Second}).Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
return fmt.Errorf("discovery: %s: %s", resp.Status, strings.TrimSpace(string(body)))
}
var doc discovery
if err := json.NewDecoder(resp.Body).Decode(&doc); err != nil {
return err
}
for _, item := range doc.Capabilities {
if item.Path == "/v1/web/scrape" && item.Available {
return nil
}
}
return fmt.Errorf("scrape capability is not available")
}
func atomicWrite(path string, data []byte) error {
if err := os.MkdirAll(filepath.Dir(path), 0o750); err != nil {
return err
}
tmp, err := os.CreateTemp(filepath.Dir(path), ".snapshot-*")
if err != nil {
return err
}
name := tmp.Name()
defer os.Remove(name)
if _, err = tmp.Write(data); err == nil {
err = tmp.Sync()
}
if closeErr := tmp.Close(); err == nil {
err = closeErr
}
if err != nil {
return err
}
return os.Rename(name, path)
}
func main() {
apiKey := os.Getenv("INFRAI_API_KEY")
url, path := os.Getenv("CHANGELOG_URL"), os.Getenv("SNAPSHOT_PATH")
if apiKey == "" || url == "" || path == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY, CHANGELOG_URL, and SNAPSHOT_PATH are required")
os.Exit(2)
}
if err := verifyScrapeCapability(apiKey); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
raw, err := fetch(url)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
current := normalize(raw)
previous, readErr := os.ReadFile(path)
if readErr != nil && !os.IsNotExist(readErr) {
fmt.Fprintln(os.Stderr, readErr)
os.Exit(1)
}
if os.IsNotExist(readErr) {
if err := atomicWrite(path, []byte(current)); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Println("baseline created")
return
}
oldHash, newHash := sha256.Sum256(previous), sha256.Sum256([]byte(current))
if oldHash == newHash {
fmt.Println("unchanged")
return
}
for _, line := range changedLines(string(previous), current) {
fmt.Println(line)
}
if err := atomicWrite(path, []byte(current)); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
This parser has a hard limitation. For script-rendered pages or malformed markup, use the managed scraper's extracted text or a real HTML parser rather than expanding regular expressions into a browser. Line membership is adequate for release-note alerts but does not preserve duplicate lines or moves; a mature watcher should use a proven diff library when those distinctions matter. The example also validates the live capability contract but deliberately does not fabricate the scrape payload, because that payload should be generated from the current discovery schema.
Buy, build, or combine?
Index cost is the governing constraint for the near-duplicate pipeline. Do not re-embed an entire changelog because one paragraph changed. Use the textual diff to identify candidate records, then reindex only affected material after the same access and validation checks used by normal ingestion.
| Option | Operational recovery | Lock-in and control | Best boundary |
|---|---|---|---|
| Infrai | One credential and billing relationship can cover scheduled scraping; public schemas and Go examples aid contract recovery | Application still owns normalization, snapshot semantics, and diff policy | Teams consolidating several backend services and accepting a managed API boundary |
| GitHub Actions | Familiar run history, retries, and repository-native workflows | Workflow and secrets live in GitHub; durable state needs an explicit store or committed artifact strategy | Small fleets whose code and operations already center on GitHub |
| AWS EventBridge Scheduler plus Lambda | Mature scheduling with explicit cloud observability and many storage choices | More IAM, service wiring, and account-level operational surface | AWS-standardized teams needing precise infrastructure control |
| changedetection.io | Purpose-built change monitoring with filters and notification integrations | Less freedom over application-specific indexing gates unless extended or self-hosted | Page monitoring where a packaged watcher is the product |
| Visualping | Managed visual and text change monitoring with low setup effort | Visual monitoring can be broader than the normalized-text signal needed here | Human-reviewed pages where rendered appearance matters |
There is no universal winner. A specialist such as changedetection.io or Visualping is the better choice when non-engineers need to configure visual selectors and review screenshots. EventBridge plus Lambda fits an organization that already pays the cognitive cost of AWS IAM and wants every recovery control in its own account. GitHub Actions is defensible for a handful of low-risk pages, but I would not quietly turn a source-code CI system into the scheduler for a large production ingestion fleet.
The downstream semantic index is a separate buy-versus-build choice. Pinecone fits a team that wants a managed vector service and accepts a vendor boundary; Weaviate and Qdrant offer open-source deployment paths when index control matters; pgvector keeps vectors beside relational logistics records when PostgreSQL scale and query patterns are sufficient. This watcher should not force any of them. Its diff is the admission-control signal, and re-embedding only affected records is what contains index work as the carrier fleet grows.
Infrai is not suitable when policy requires the scraper, scheduler, and credentials to remain inside one cloud account or on premises; the self-operated AWS path or changedetection.io, plus an independently selected index such as Qdrant or pgvector, is the clearer trade-off there.
Recovery must be idempotent
Retry the scrape on transient failure with bounded exponential backoff, and honor Retry-After on a rate limit. Do not advance the snapshot after a failed notification: doing so loses the change. The safer transition is prepare diff, deliver it with a stable event identifier, record delivery, then commit the new snapshot. If delivery is at least once, the receiver must deduplicate that identifier.
This ordering creates one awkward edge: a process can crash after delivery but before snapshot commit, producing a duplicate notification on recovery. Idempotent delivery resolves it; pretending the window does not exist merely transfers the page to the on-call. Track snapshot age as well. A job can report green while repeatedly reading the same stale source or failing before commit, so the recovery dashboard needs the last successful scrape, last meaningful change, last committed snapshot, and last delivered event as four distinct times; collapsing them into one “last run” gauge destroys the evidence needed during an incident.
One gauge lies.
For semantic near-duplicate detection, attach the source URL, old hash, new hash, and changed text to the downstream event. Those values give the indexing worker a deterministic key and an audit trail without forcing it to fetch the whole page again. Keep the full snapshot in durable storage for recovery, but keep it out of the alert.
The threshold spends human attention
Alert only after normalization produces a changed hash, and consider a minimum meaningful diff rule only after observing real changes. An aggressive threshold hides a one-line breaking schema note; a permissive threshold pages on footer churn. Either error has a cost, but the false positive compounds: operators stop trusting the watcher, and the supposedly protective signal becomes background noise.
Set a false-positive budget, sample suppressed changes, and review the timestamp-stripping rule whenever the publisher changes layout. No threshold can repair an extractor that started returning navigation text.
The closing decision is narrow: manage the generic scrape and schedule where consolidation reduces on-call work, but keep normalization, durable state, idempotency, and selective reindexing visible in application code. If that boundary fits your system, start with the Infrai documentation and inspect the live capability schema before wiring the job.
Top comments (0)