Every organization that publishes advisories writes the same document twice: once when nothing happened, and once when something did. The intelligence is not in the document. It is in the delta between versions. Most monitoring pipelines miss this because they alert on new documents, not on changed wording.
The pipeline in five lines of logic:
- Poll the source (an advisory page, an embassy notice, an API status page, a terms page - anything versioned).
- Normalize the text: strip boilerplate (nav, footers, dates), collapse whitespace, lower-case.
- Compare against the last stored version with a fuzzy diff, not a hash. A hash change fires on a timestamp edit; you want the semantic delta.
- If the similarity is above 0.85 but below 1.0 - a real edit - emit the changed sentences, not the whole document.
- Store the new version as the baseline.
Why 0.85-1.0 is the interesting band. Below 0.85 similarity is a rewrite (a new notice, treat it as new). Above 1.0 is no change. In between is surgical wording change, the signature of an institution adjusting its position without wanting headlines: "monitored" → "monitored closely", "avoid travel to border regions" → "avoid all travel", "low probability" → "elevated probability". Analysts who read the full documents get the same information hours later, when the press notices.
Real examples of the delta being the event:
- Travel advisories: the level number changes at headlines speed; the text under the level changes first. The text diff is your hours-early warning.
- Status pages: "intermittent errors" → "degraded performance for a subset of users" is a public, timestamped admission ladder. Track the ladder per provider and you can price their incident honesty.
- ToS/privacy pages: a clause insertion (arbitration, data-sharing) is a governance event for a product, and product teams rarely announce it.
- Sanctions/entity lists: an entry edit (address, alias added) often precedes an entry addition - the alias edit is the paper trail of an evasion network being tied off.
Practical notes:
- Poll cadence follows the source class: advisories daily, status pages every 5 minutes, ToS weekly.
- Diff language-aware (a translated version of an advisory is not a change event; keep per-language baselines).
- Store the full version chain; the audit trail is the product. Being able to say "the wording changed from X to Y at time T" is the whole deliverable.
This is the core alerting pattern behind the monitoring sets in my Telegram & Web OSINT Bundle ($5) - the free sample brief shows a wording-diff timeline in the output format.
Runs free on GitHub Actions - no server, no paid APIs.
Top comments (1)
tr.ee/dev-to