Building a Website Change Monitoring System: From One URL to a Distributed Crawling Pipeline
Imagine someone gives us a URL:
https://example.com
and says:
Tell me if this website changes.
At first, this sounds almost trivial.
Fetch the page, save the HTML, come back later, fetch it again, compare the two versions.
For one page, that can work.
But almost immediately, questions start appearing.
What exactly are we monitoring?
Only the homepage?
Every page under the website?
How often should we check it?
Every two minutes?
Once per day?
Only once when an administrator requests it?
What happens when a harmless calendar or rotating banner changes?
And if we eventually monitor hundreds or thousands of websites, who decides what should be crawled, and who actually performs all that crawling?
Those questions are what turn a simple script into a monitoring system.
This article walks through that evolution one design decision at a time.
First question: what does “monitor this website” actually mean?
Suppose we start with:
https://example.com
If we fetch only that URL, we are monitoring only the homepage.
But the important page might actually be:
https://example.com/login
or:
https://example.com/payment
or:
https://example.com/products/important-product
So monitoring a website introduces our first configuration decision.
How deeply should we crawl?
Consider this website:
Homepage
│
├── About
├── Products
│ ├── Product A
│ └── Product B
│
└── Blog
├── Post 1
└── Post 2
If we monitor only the homepage, we might completely miss a change on Product A.
So we could tell the crawler:
Follow every internal link.
But that creates another problem.
Product A may contain links to documentation.
Documentation may contain hundreds of pages.
Those pages may link to thousands more.
Suddenly:
1 URL
becomes:
10 URLs
then:
100 URLs
then potentially:
10,000+ URLs
So we need a boundary.
That is what crawl depth gives us.
If the administrator chooses:
crawl depth = 0
we monitor only:
Homepage
With:
crawl depth = 1
we monitor:
Homepage
↓
direct internal links
With:
crawl depth = 2
we monitor:
Homepage
↓
Level 1 pages
↓
links discovered inside Level 1 pages
Conceptually:
Depth 0
example.com
|
v
Depth 1
/about
/products
/blog
|
v
Depth 2
/products/a
/products/b
/blog/post-1
/blog/post-2
Now the administrator can choose the monitoring surface instead of letting the crawler wander indefinitely.
This also tells us something important about our data model.
A website isn't just:
url
It needs configuration.
Something closer to:
Website
├── URL
├── Crawl depth
├── Monitoring frequency
├── Ignore rules
├── Notification preferences
└── Monitoring enabled/disabled
Now we are no longer building a crawler.
We are beginning to build a monitoring product.
Second question: how often should we check?
Suppose our baseline crawl finishes at:
10:00
When should we crawl again?
There is no universally correct answer.
For a low-risk informational website:
once per day
might be enough.
For something sensitive:
every 5 minutes
might be appropriate.
For another system:
every 2 minutes
might be required.
And sometimes continuous monitoring isn't even needed.
An administrator may simply want:
crawl once now
So frequency also belongs to the website's monitoring configuration.
For example:
example.com
crawl depth: 2
frequency: every 5 minutes
while:
another-site.com
crawl depth: 1
frequency: daily
This creates our next architectural requirement.
Something needs to keep asking:
Which websites are due to be checked now?
That responsibility belongs to a scheduler.
Conceptually:
Website configuration
|
v
Scheduler
|
v
Is next crawl due?
The scheduler doesn't need to know anything about HTML parsing or link discovery.
Its job is simply to decide when work should happen.
That separation becomes important later.
Third question: how do we avoid becoming the attacker?
Now imagine:
crawl depth = 3
frequency = 2 minutes
and the website contains thousands of pages.
A badly designed crawler could start firing hundreds of requests against the same server every two minutes.
Our monitoring system could accidentally create the exact availability problem it is supposed to help detect.
So monitoring frequency cannot mean:
Hit the website as aggressively as possible.
The crawler needs to behave responsibly.
A mature design should consider things such as:
per-domain concurrency limits
request delays
timeouts
maximum pages per crawl
retry backoff
429 handling
5xx handling
overlapping crawl prevention
For example:
example.com
maximum concurrent requests = 2
rather than letting dozens of workers hammer the same domain simultaneously.
And if the previous crawl is still running:
crawl #101 = running
we probably shouldn't blindly start:
crawl #102
just because the next two-minute interval arrived.
Already, the “simple crawler” has acquired scheduling and resource-control requirements.
Fourth question: what happens when hundreds of websites become due together?
Suppose we have 1,000 monitored websites.
At 10:00, hundreds become eligible for crawling.
One naive implementation would be:
Dashboard
|
v
Fetch website
|
v
Parse HTML
|
v
Save results
But now the application serving administrators is also responsible for long-running crawling work.
That creates several problems.
A slow website could tie up the main application.
A timeout could delay unrelated work.
A burst of scheduled crawls could overload the dashboard server.
And scaling the crawler would mean scaling the entire application.
So logically, we separate two responsibilities.
The main application manages:
customers
websites
configuration
monitoring settings
administration
The crawler handles:
network requests
HTML
link extraction
snapshots
change detection
In the system I worked on, the main application kept its structured application data in MySQL.
Crawler-oriented data was handled separately.
But now these two parts need a way to communicate.
This is where the queue appears
Instead of telling the crawler:
Crawl this website right now and make me wait until you're done,
the application can publish a job:
{
"websiteId": 842,
"url": "https://example.com",
"maxDepth": 2
}
into a message queue.
Now the architecture evolves naturally:
Dashboard
|
v
MySQL
|
v
Scheduler
|
v
Crawl Queue
|
v
Crawler Workers
This gives us a buffer.
If 500 crawl jobs appear suddenly, the main application doesn't need 500 crawlers immediately.
Jobs wait in the queue.
Crawler workers consume them according to available capacity.
If we later need more crawling throughput, we can add crawler workers without redesigning the dashboard.
The queue isn't there because queues are fashionable.
It appears because scheduled work and execution capacity are different problems.
Now the crawler finally receives a URL
Suppose the worker receives:
https://example.com
The first operation is straightforward.
Fetch the page.
Conceptually:
response = requests.get(
"https://example.com",
timeout=10
)
html = response.text
But this is the first crawl.
There is nothing to compare against yet.
So instead of detecting a change, this crawl establishes a baseline.
We store the page content.
In our crawler side, page snapshots and crawling data were stored in MongoDB.
Now the architecture has two distinct data concerns:
MySQL
↓
website/customer/configuration data
MongoDB
↓
crawler/page/snapshot-oriented data
That separation wasn't simply:
SQL good here, MongoDB good there.
The two sides represented different workloads.
The homepage is only the beginning
Once the homepage is downloaded, we inspect its internal links.
For example:
<a href="/about">About</a>
<a href="/products">Products</a>
<a href="/blog">Blog</a>
We extract:
/about
/products
/blog
convert them to full URLs where necessary, and keep only links belonging to the target website.
Then we need another important step:
deduplication.
Imagine the same page appears through several navigation paths:
Homepage → Products
Blog → Products
About → Products
Footer → Products
We don't want to crawl /products four times during the same crawl run.
So logically:
Extract links
|
v
Normalize URLs
|
v
Remove duplicates
|
v
Check crawl depth
|
v
Schedule inner pages
The inner crawler then processes those URLs.
Each inner page can discover more URLs.
So a queue item might carry:
{
"url": "https://example.com/products",
"depth": 1,
"maxDepth": 3
}
If:
depth < maxDepth
we extract more links.
When:
depth == maxDepth
we stop expanding.
That simple number prevents recursive discovery from turning into uncontrolled crawling.
URL deduplication is trickier than it looks
Consider:
https://example.com/about
https://example.com/about/
https://example.com/about#team
https://example.com/about?utm_source=email
Are those four separate pages?
Maybe.
But often they represent the same useful monitoring target.
So before deduplicating, a modern crawler may normalize URLs by handling:
fragments
tracking parameters
relative paths
trailing slashes
canonical URLs
host casing
Otherwise, the crawler may spend significant resources repeatedly monitoring effectively identical pages.
Now we finally reach the actual monitoring problem
After the first crawl, suppose we stored this page:
https://example.com/products
Five minutes later, according to its configured frequency, the scheduler queues the website again.
The crawler downloads the page again.
Now we have:
previous version
current version
We need to know:
Has anything changed?
In our implementation, one simple mechanism was an MD5 hash.
Conceptually:
HTML
|
v
MD5
|
v
hash
So baseline HTML produces:
A7F91...
and the next crawl produces another hash.
If:
old_hash == new_hash
the inputs are identical.
Nothing changed.
If:
old_hash != new_hash
something changed.
Very simple.
Very fast.
And also incomplete.
Because “different” does not mean “important”
Imagine this element:
<div class="calendar">
9 August
</div>
A day later it becomes:
<div class="calendar">
10 August
</div>
The HTML changed.
Therefore:
old MD5 != new MD5
The system alerts the administrator.
But there was no attack.
Nothing important happened.
Now imagine another page contains:
10:31:04
and one minute later:
10:32:04
Again:
hash changed
Other common examples include:
rotating headlines
advertisements
visitor counters
timestamps
calendars
live market values
random identifiers
dynamic widgets
This reveals the central weakness of pure hashing.
A hash can answer:
Are these two inputs identical?
It cannot answer:
Is this difference meaningful?
To MD5:
calendar date changed
and:
attacker replaced the homepage
are both simply:
different input
And that creates false positives.
Why false positives are dangerous
Imagine monitoring a security-sensitive website.
The administrator receives:
ALERT
ALERT
ALERT
ALERT
ALERT
throughout the day.
Most alerts are harmless calendar or headline changes.
Eventually the administrator begins ignoring them.
Now when an actual unexpected modification happens:
ALERT
it looks like everything else.
So false-positive handling isn't merely a convenience feature.
It directly affects whether the monitoring system remains useful.
Introducing the ignore list
Suppose we know this region changes constantly:
<div class="calendar">
10 August
</div>
The administrator can configure:
ignore .calendar
Similarly:
ignore .ticker
ignore #clock
ignore .rotating-banner
Now our comparison pipeline becomes:
Fetched HTML
|
v
Apply ignore rules
|
v
Remove known dynamic regions
|
v
Generate normalized content
|
v
MD5
|
v
Compare
Instead of hashing everything blindly, we're hashing the content we actually care about monitoring.
Conceptually:
def prepare_for_monitoring(html, ignored_selectors):
dom = parse_html(html)
for selector in ignored_selectors:
remove(dom, selector)
return serialize(dom)
then:
normalized_html = prepare_for_monitoring(
html,
[".calendar", ".ticker"]
)
page_hash = md5(normalized_html)
Now a calendar update doesn't automatically become a security alert.
This is where the system moves from:
change detection
toward:
meaningful change detection.
But what happens when a meaningful change is detected?
Suppose:
old_hash != new_hash
after ignore rules have been applied.
The crawler has discovered something.
Should it now send an email itself?
Send an SMS itself?
Update the dashboard itself?
It could.
But then the crawler would be responsible for:
HTTP crawling
HTML parsing
comparison
email
SMS
dashboard updates
That's too many responsibilities in one component.
So another boundary naturally appears.
The crawler emits a change event.
Change detected
|
v
Alert Queue
Then separate notification handlers can deliver it:
Alert Queue
|
+----------+----------+
| | |
v v v
Dashboard Email SMS
Now an SMS provider outage doesn't stop crawling.
Email delivery can retry independently.
And a new notification channel can be added later without rewriting the crawler.
Again, the queue isn't introduced because “event-driven architecture is cool.”
It appears because detecting something and notifying someone are separate reliability problems.
The architecture we ended up with
By following the requirements rather than starting from technologies, our simple URL checker evolved into something closer to this:
Administrator
|
v
Dashboard
|
v
MySQL
website/configuration
|
v
Scheduler
|
Is crawl due?
|
v
Crawl Queue
|
v
Python Crawlers
|
+----------+----------+
| |
v v
Link Discovery Page Snapshot
| |
v v
Normalize/Deduplicate MongoDB
|
v
Depth-controlled
inner crawling
|
v
Prepare HTML
apply ignore list
|
v
Generate MD5
|
v
Compare previous hash
/ \
/ \
unchanged changed
|
v
Alert Queue
|
+-----------+-----------+
| | |
v v v
Dashboard Email SMS
What began as:
download HTML
became scheduling, crawling, discovery, state, comparison, noise reduction, queueing, and notification.
One more problem: crawl politely
Let's return to:
frequency = 2 minutes
depth = 3
Suppose this website has 5,000 discoverable pages.
We absolutely don't want:
5,000 requests
fired aggressively against the target.
A responsible crawler needs controls.
Per-domain concurrency
Instead of:
100 workers → example.com
we might enforce:
maximum 2 concurrent requests → example.com
Crawl budgets
A website configuration could include:
max depth: 3
max pages per crawl: 1,000
request timeout: 10 seconds
per-domain concurrency: 2
Depth and page limit solve different problems.
Depth 1 could still contain 20,000 links.
Retry with backoff
If the target returns:
429 Too Many Requests
retrying immediately is the wrong response.
Instead:
1s
2s
4s
8s
16s
with jitter.
Don't overlap crawls blindly
If monitoring is configured every two minutes but one crawl takes four minutes:
10:00 crawl A starts
10:02 crawl B scheduled
the scheduler should probably detect:
website currently crawling
rather than blindly launching another run.
What I would change if I designed it today
The underlying problem is still interesting, but I wouldn't rebuild every detail exactly the same way.
Hash first, diff second
Hashing is still useful for a very fast first check.
hash same
|
v
stop
But if:
hash different
I would generate an actual structured diff.
Instead of:
Website changed.
the administrator might see:
- Payment destination: account A
+ Payment destination: account B
That's significantly more useful.
Compare the DOM, not only serialized HTML
HTML already contains structure.
So changes could be classified by region:
title changed
main content changed
form action changed
external script added
navigation changed
Then different changes could receive different severity levels.
For example:
Calendar changed INFO
Headline changed LOW
Form action changed HIGH
New external JavaScript HIGH
Large DOM replacement CRITICAL
Better normalization
Before comparison I would also normalize things such as:
whitespace
volatile generated IDs
known tracking parameters
timestamps
irrelevant markup differences
This would further reduce noise.
Explicit crawl-run tracking
Instead of only knowing whether a website is “currently crawling,” I would model crawl runs explicitly:
crawl_run_id
website_id
started_at
finished_at
pages_discovered
pages_processed
pages_failed
status
That makes operational debugging much easier.
Dead-letter queues
Some crawl jobs will repeatedly fail.
After controlled retries, they should move somewhere visible instead of looping forever.
normal queue
|
retries
|
v
dead-letter queue
Observability
For a serious deployment I'd monitor:
queue depth
crawl duration
pages/sec
HTTP error rate
retry count
change rate
false-positive rate
notification failures
per-domain request rate
Without those metrics, it's difficult to know whether the system is healthy or merely running.
And today, AI creates an interesting extra layer
The deterministic monitoring engine should remain deterministic.
I would not replace HTML comparison with an LLM.
But after a real change has already been detected, AI could help answer:
What does this change mean?
For example:
- <script src="/assets/app.js">
+ <script src="https://unknown-example.com/inject.js">
A semantic analysis stage might classify:
{
"type": "external_script_added",
"severity": "high",
"reason": "A previously unseen external JavaScript source was introduced."
}
So the architecture becomes:
Deterministic detection
|
v
Real change found
|
v
Generate structured diff
|
v
Rules / classifiers
|
v
Optional AI analysis
|
v
Severity + explanation
The important part is the ordering.
AI helps interpret the signal.
It doesn't replace the reliable mechanism that discovers the signal.
The interesting lesson
When we began, the requirement looked like:
Monitor a URL for changes.
But each real-world question forced another design decision.
Which pages matter?
→ crawl depth.
How frequently do they matter?
→ monitoring schedule.
What happens when hundreds become due?
→ queue + workers.
How do we revisit inner pages?
→ link extraction + deduplication + depth tracking.
How do we know something changed?
→ snapshots + hashes.
How do we avoid useless alerts?
→ ignore rules + normalization.
How do we notify reliably?
→ alert events + separate notification workers.
How do we avoid harming the monitored site?
→ crawl budgets, rate limits, concurrency control, and backoff.
That's what I find most interesting about systems like this.
The final architecture doesn't need to be invented on a whiteboard first.
It can emerge naturally from repeatedly asking:
What problem does our current simple solution fail to solve next?
Top comments (0)