DEV Community: Maz I

Building a Website Change Monitoring System: From One URL to a Distributed Crawling Pipeline

Maz I — Mon, 10 Aug 2026 14:27:40 +0000

Building a Website Change Monitoring System: From One URL to a Distributed Crawling Pipeline

Imagine someone gives us a URL:

https://example.com

and says:

Tell me if this website changes.

At first, this sounds almost trivial.

Fetch the page, save the HTML, come back later, fetch it again, compare the two versions.

For one page, that can work.

But almost immediately, questions start appearing.

What exactly are we monitoring?

Only the homepage?

Every page under the website?

How often should we check it?

Every two minutes?

Once per day?

Only once when an administrator requests it?

What happens when a harmless calendar or rotating banner changes?

And if we eventually monitor hundreds or thousands of websites, who decides what should be crawled, and who actually performs all that crawling?

Those questions are what turn a simple script into a monitoring system.

This article walks through that evolution one design decision at a time.

First question: what does “monitor this website” actually mean?

Suppose we start with:

https://example.com

If we fetch only that URL, we are monitoring only the homepage.

But the important page might actually be:

https://example.com/login

or:

https://example.com/payment

or:

https://example.com/products/important-product

So monitoring a website introduces our first configuration decision.

How deeply should we crawl?

Consider this website:

Homepage
│
├── About
├── Products
│   ├── Product A
│   └── Product B
│
└── Blog
    ├── Post 1
    └── Post 2

If we monitor only the homepage, we might completely miss a change on Product A.

So we could tell the crawler:

Follow every internal link.

But that creates another problem.

Product A may contain links to documentation.

Documentation may contain hundreds of pages.

Those pages may link to thousands more.

Suddenly:

1 URL

becomes:

10 URLs

then:

100 URLs

then potentially:

10,000+ URLs

So we need a boundary.

That is what crawl depth gives us.

If the administrator chooses:

crawl depth = 0

we monitor only:

Homepage

With:

crawl depth = 1

we monitor:

Homepage
    ↓
direct internal links

With:

crawl depth = 2

we monitor:

Homepage
    ↓
Level 1 pages
    ↓
links discovered inside Level 1 pages

Conceptually:

Depth 0

example.com
      |
      v

Depth 1

/about
/products
/blog
      |
      v

Depth 2

/products/a
/products/b
/blog/post-1
/blog/post-2

Now the administrator can choose the monitoring surface instead of letting the crawler wander indefinitely.

This also tells us something important about our data model.

A website isn't just:

url

It needs configuration.

Something closer to:

Website
├── URL
├── Crawl depth
├── Monitoring frequency
├── Ignore rules
├── Notification preferences
└── Monitoring enabled/disabled

Now we are no longer building a crawler.

We are beginning to build a monitoring product.

Second question: how often should we check?

Suppose our baseline crawl finishes at:

10:00

When should we crawl again?

There is no universally correct answer.

For a low-risk informational website:

once per day

might be enough.

For something sensitive:

every 5 minutes

might be appropriate.

For another system:

every 2 minutes

might be required.

And sometimes continuous monitoring isn't even needed.

An administrator may simply want:

crawl once now

So frequency also belongs to the website's monitoring configuration.

For example:

example.com

crawl depth: 2
frequency: every 5 minutes

while:

another-site.com

crawl depth: 1
frequency: daily

This creates our next architectural requirement.

Something needs to keep asking:

Which websites are due to be checked now?

That responsibility belongs to a scheduler.

Conceptually:

Website configuration
        |
        v
    Scheduler
        |
        v
Is next crawl due?

The scheduler doesn't need to know anything about HTML parsing or link discovery.

Its job is simply to decide when work should happen.

That separation becomes important later.

Third question: how do we avoid becoming the attacker?

Now imagine:

crawl depth = 3
frequency = 2 minutes

and the website contains thousands of pages.

A badly designed crawler could start firing hundreds of requests against the same server every two minutes.

Our monitoring system could accidentally create the exact availability problem it is supposed to help detect.

So monitoring frequency cannot mean:

Hit the website as aggressively as possible.

The crawler needs to behave responsibly.

A mature design should consider things such as:

per-domain concurrency limits
request delays
timeouts
maximum pages per crawl
retry backoff
429 handling
5xx handling
overlapping crawl prevention

For example:

example.com

maximum concurrent requests = 2

rather than letting dozens of workers hammer the same domain simultaneously.

And if the previous crawl is still running:

crawl #101 = running

we probably shouldn't blindly start:

crawl #102

just because the next two-minute interval arrived.

Already, the “simple crawler” has acquired scheduling and resource-control requirements.

Fourth question: what happens when hundreds of websites become due together?

Suppose we have 1,000 monitored websites.

At 10:00, hundreds become eligible for crawling.

One naive implementation would be:

Dashboard
   |
   v
Fetch website
   |
   v
Parse HTML
   |
   v
Save results

But now the application serving administrators is also responsible for long-running crawling work.

That creates several problems.

A slow website could tie up the main application.

A timeout could delay unrelated work.

A burst of scheduled crawls could overload the dashboard server.

And scaling the crawler would mean scaling the entire application.

So logically, we separate two responsibilities.

The main application manages:

customers
websites
configuration
monitoring settings
administration

The crawler handles:

network requests
HTML
link extraction
snapshots
change detection

In the system I worked on, the main application kept its structured application data in MySQL.

Crawler-oriented data was handled separately.

But now these two parts need a way to communicate.

This is where the queue appears

Instead of telling the crawler:

Crawl this website right now and make me wait until you're done,

the application can publish a job:

{
  "websiteId": 842,
  "url": "https://example.com",
  "maxDepth": 2
}

into a message queue.

Now the architecture evolves naturally:

Dashboard
   |
   v
 MySQL
   |
   v
Scheduler
   |
   v
Crawl Queue
   |
   v
Crawler Workers

This gives us a buffer.

If 500 crawl jobs appear suddenly, the main application doesn't need 500 crawlers immediately.

Jobs wait in the queue.

Crawler workers consume them according to available capacity.

If we later need more crawling throughput, we can add crawler workers without redesigning the dashboard.

The queue isn't there because queues are fashionable.

It appears because scheduled work and execution capacity are different problems.

Now the crawler finally receives a URL

Suppose the worker receives:

https://example.com

The first operation is straightforward.

Fetch the page.

Conceptually:

response = requests.get(
    "https://example.com",
    timeout=10
)

html = response.text

But this is the first crawl.

There is nothing to compare against yet.

So instead of detecting a change, this crawl establishes a baseline.

We store the page content.

In our crawler side, page snapshots and crawling data were stored in MongoDB.

Now the architecture has two distinct data concerns:

MySQL
↓
website/customer/configuration data

MongoDB
↓
crawler/page/snapshot-oriented data

That separation wasn't simply:

SQL good here, MongoDB good there.

The two sides represented different workloads.

The homepage is only the beginning

Once the homepage is downloaded, we inspect its internal links.

For example:

<a href="/about">About</a>
<a href="/products">Products</a>
<a href="/blog">Blog</a>

We extract:

/about
/products
/blog

convert them to full URLs where necessary, and keep only links belonging to the target website.

Then we need another important step:

deduplication.

Imagine the same page appears through several navigation paths:

Homepage → Products
Blog → Products
About → Products
Footer → Products

We don't want to crawl /products four times during the same crawl run.

So logically:

Extract links
      |
      v
Normalize URLs
      |
      v
Remove duplicates
      |
      v
Check crawl depth
      |
      v
Schedule inner pages

The inner crawler then processes those URLs.

Each inner page can discover more URLs.

So a queue item might carry:

{
  "url": "https://example.com/products",
  "depth": 1,
  "maxDepth": 3
}

If:

depth < maxDepth

we extract more links.

When:

depth == maxDepth

we stop expanding.

That simple number prevents recursive discovery from turning into uncontrolled crawling.

URL deduplication is trickier than it looks

Consider:

https://example.com/about
https://example.com/about/
https://example.com/about#team
https://example.com/about?utm_source=email

Are those four separate pages?

Maybe.

But often they represent the same useful monitoring target.

So before deduplicating, a modern crawler may normalize URLs by handling:

fragments
tracking parameters
relative paths
trailing slashes
canonical URLs
host casing

Otherwise, the crawler may spend significant resources repeatedly monitoring effectively identical pages.

Now we finally reach the actual monitoring problem

After the first crawl, suppose we stored this page:

https://example.com/products

Five minutes later, according to its configured frequency, the scheduler queues the website again.

The crawler downloads the page again.

Now we have:

previous version
current version

We need to know:

Has anything changed?

In our implementation, one simple mechanism was an MD5 hash.

Conceptually:

HTML
 |
 v
MD5
 |
 v
hash

So baseline HTML produces:

A7F91...

and the next crawl produces another hash.

If:

old_hash == new_hash

the inputs are identical.

Nothing changed.

If:

old_hash != new_hash

something changed.

Very simple.

Very fast.

And also incomplete.

Because “different” does not mean “important”

Imagine this element:

<div class="calendar">
    9 August
</div>

A day later it becomes:

<div class="calendar">
    10 August
</div>

The HTML changed.

Therefore:

old MD5 != new MD5

The system alerts the administrator.

But there was no attack.

Nothing important happened.

Now imagine another page contains:

10:31:04

and one minute later:

10:32:04

Again:

hash changed

Other common examples include:

rotating headlines
advertisements
visitor counters
timestamps
calendars
live market values
random identifiers
dynamic widgets

This reveals the central weakness of pure hashing.

A hash can answer:

Are these two inputs identical?

It cannot answer:

Is this difference meaningful?

To MD5:

calendar date changed

and:

attacker replaced the homepage

are both simply:

different input

And that creates false positives.

Why false positives are dangerous

Imagine monitoring a security-sensitive website.

The administrator receives:

ALERT
ALERT
ALERT
ALERT
ALERT

throughout the day.

Most alerts are harmless calendar or headline changes.

Eventually the administrator begins ignoring them.

Now when an actual unexpected modification happens:

ALERT

it looks like everything else.

So false-positive handling isn't merely a convenience feature.

It directly affects whether the monitoring system remains useful.

Introducing the ignore list

Suppose we know this region changes constantly:

<div class="calendar">
    10 August
</div>

The administrator can configure:

ignore .calendar

Similarly:

ignore .ticker
ignore #clock
ignore .rotating-banner

Now our comparison pipeline becomes:

Fetched HTML
     |
     v
Apply ignore rules
     |
     v
Remove known dynamic regions
     |
     v
Generate normalized content
     |
     v
MD5
     |
     v
Compare

Instead of hashing everything blindly, we're hashing the content we actually care about monitoring.

Conceptually:

def prepare_for_monitoring(html, ignored_selectors):
    dom = parse_html(html)

    for selector in ignored_selectors:
        remove(dom, selector)

    return serialize(dom)

then:

normalized_html = prepare_for_monitoring(
    html,
    [".calendar", ".ticker"]
)

page_hash = md5(normalized_html)

Now a calendar update doesn't automatically become a security alert.

This is where the system moves from:

change detection

toward:

meaningful change detection.

But what happens when a meaningful change is detected?

Suppose:

old_hash != new_hash

after ignore rules have been applied.

The crawler has discovered something.

Should it now send an email itself?

Send an SMS itself?

Update the dashboard itself?

It could.

But then the crawler would be responsible for:

HTTP crawling
HTML parsing
comparison
email
SMS
dashboard updates

That's too many responsibilities in one component.

So another boundary naturally appears.

The crawler emits a change event.

Change detected
      |
      v
Alert Queue

Then separate notification handlers can deliver it:

              Alert Queue
                  |
       +----------+----------+
       |          |          |
       v          v          v
   Dashboard    Email       SMS

Now an SMS provider outage doesn't stop crawling.

Email delivery can retry independently.

And a new notification channel can be added later without rewriting the crawler.

Again, the queue isn't introduced because “event-driven architecture is cool.”

It appears because detecting something and notifying someone are separate reliability problems.

The architecture we ended up with

By following the requirements rather than starting from technologies, our simple URL checker evolved into something closer to this:

                    Administrator
                         |
                         v
                    Dashboard
                         |
                         v
                       MySQL
             website/configuration
                         |
                         v
                     Scheduler
                         |
               Is crawl due?
                         |
                         v
                    Crawl Queue
                         |
                         v
                  Python Crawlers
                         |
              +----------+----------+
              |                     |
              v                     v
        Link Discovery          Page Snapshot
              |                     |
              v                     v
      Normalize/Deduplicate       MongoDB
              |
              v
        Depth-controlled
          inner crawling

                         |
                         v
                  Prepare HTML
                apply ignore list
                         |
                         v
                    Generate MD5
                         |
                         v
               Compare previous hash
                    /        \
                   /          \
              unchanged      changed
                                |
                                v
                           Alert Queue
                                |
                    +-----------+-----------+
                    |           |           |
                    v           v           v
                Dashboard      Email       SMS

What began as:

download HTML

became scheduling, crawling, discovery, state, comparison, noise reduction, queueing, and notification.

One more problem: crawl politely

Let's return to:

frequency = 2 minutes
depth = 3

Suppose this website has 5,000 discoverable pages.

We absolutely don't want:

5,000 requests

fired aggressively against the target.

A responsible crawler needs controls.

Per-domain concurrency

Instead of:

100 workers → example.com

we might enforce:

maximum 2 concurrent requests → example.com

Crawl budgets

A website configuration could include:

max depth: 3
max pages per crawl: 1,000
request timeout: 10 seconds
per-domain concurrency: 2

Depth and page limit solve different problems.

Depth 1 could still contain 20,000 links.

Retry with backoff

If the target returns:

429 Too Many Requests

retrying immediately is the wrong response.

Instead:

1s
2s
4s
8s
16s

with jitter.

Don't overlap crawls blindly

If monitoring is configured every two minutes but one crawl takes four minutes:

10:00 crawl A starts
10:02 crawl B scheduled

the scheduler should probably detect:

website currently crawling

rather than blindly launching another run.

What I would change if I designed it today

The underlying problem is still interesting, but I wouldn't rebuild every detail exactly the same way.

Hash first, diff second

Hashing is still useful for a very fast first check.

hash same
    |
    v
stop

But if:

hash different

I would generate an actual structured diff.

Instead of:

Website changed.

the administrator might see:

- Payment destination: account A
+ Payment destination: account B

That's significantly more useful.

Compare the DOM, not only serialized HTML

HTML already contains structure.

So changes could be classified by region:

title changed
main content changed
form action changed
external script added
navigation changed

Then different changes could receive different severity levels.

For example:

Calendar changed              INFO
Headline changed              LOW
Form action changed           HIGH
New external JavaScript       HIGH
Large DOM replacement         CRITICAL

Better normalization

Before comparison I would also normalize things such as:

whitespace
volatile generated IDs
known tracking parameters
timestamps
irrelevant markup differences

This would further reduce noise.

Explicit crawl-run tracking

Instead of only knowing whether a website is “currently crawling,” I would model crawl runs explicitly:

crawl_run_id
website_id
started_at
finished_at
pages_discovered
pages_processed
pages_failed
status

That makes operational debugging much easier.

Dead-letter queues

Some crawl jobs will repeatedly fail.

After controlled retries, they should move somewhere visible instead of looping forever.

normal queue
     |
   retries
     |
     v
dead-letter queue

Observability

For a serious deployment I'd monitor:

queue depth
crawl duration
pages/sec
HTTP error rate
retry count
change rate
false-positive rate
notification failures
per-domain request rate

Without those metrics, it's difficult to know whether the system is healthy or merely running.

And today, AI creates an interesting extra layer

The deterministic monitoring engine should remain deterministic.

I would not replace HTML comparison with an LLM.

But after a real change has already been detected, AI could help answer:

What does this change mean?

For example:

- <script src="/assets/app.js">
+ <script src="https://unknown-example.com/inject.js">

A semantic analysis stage might classify:

{
  "type": "external_script_added",
  "severity": "high",
  "reason": "A previously unseen external JavaScript source was introduced."
}

So the architecture becomes:

Deterministic detection
        |
        v
Real change found
        |
        v
Generate structured diff
        |
        v
Rules / classifiers
        |
        v
Optional AI analysis
        |
        v
Severity + explanation

The important part is the ordering.

AI helps interpret the signal.

It doesn't replace the reliable mechanism that discovers the signal.

The interesting lesson

When we began, the requirement looked like:

Monitor a URL for changes.

But each real-world question forced another design decision.

Which pages matter?

→ crawl depth.

How frequently do they matter?

→ monitoring schedule.

What happens when hundreds become due?

→ queue + workers.

How do we revisit inner pages?

→ link extraction + deduplication + depth tracking.

How do we know something changed?

→ snapshots + hashes.

How do we avoid useless alerts?

→ ignore rules + normalization.

How do we notify reliably?

→ alert events + separate notification workers.

How do we avoid harming the monitored site?

→ crawl budgets, rate limits, concurrency control, and backoff.

That's what I find most interesting about systems like this.

The final architecture doesn't need to be invented on a whiteboard first.

It can emerge naturally from repeatedly asking:

What problem does our current simple solution fail to solve next?

Why does PostgreSQL sometimes ignore an index you created?

Maz I — Sat, 08 Aug 2026 22:55:41 +0000

Why does PostgreSQL sometimes ignore an index you created?

A common assumption is:

“If a column has an index, PostgreSQL should use it.”

But PostgreSQL does not work that way.

An index is only one possible access path.

Before executing a query, PostgreSQL's query planner estimates the cost of different plans and chooses the one it believes will be cheapest.

That might be:

• Sequential Scan
• Index Scan
• Index Only Scan
• Bitmap Index Scan
• Parallel Sequential Scan

So sometimes PostgreSQL sees your perfectly valid index and deliberately decides:

“Scanning the table is cheaper.”

Consider this example:

CREATE INDEX idx_users_status
ON users(status);

And imagine the table contains 10 million users:

active   = 7,000,000
inactive = 2,900,000
banned   =   100,000

Now compare these queries:

SELECT *
FROM users
WHERE status = 'banned';

and:

SELECT *
FROM users
WHERE status = 'active';

Both use the indexed status column.

But PostgreSQL may choose very different execution plans.

For banned, only around 1% of rows match.

Using the index can make sense because PostgreSQL can locate a relatively small set of rows instead of scanning millions of unrelated rows.

For active, around 70% of the table matches.

Now using the index may mean:

Traverse the index.
Find millions of matching row locations.
Visit millions of table pages to retrieve those rows.

At that point, reading the table sequentially may simply cost less.

This is where selectivity becomes important.

A highly selective condition returns a small percentage of the table.

WHERE email = 'user@example.com'

That is usually a great candidate for an index.

A low-selectivity condition might match most of the table.

WHERE status = 'active'

An index may provide little advantage.

But how does PostgreSQL know how many rows are likely to match?

Statistics.

PostgreSQL collects information about the data distribution in a table.

The planner uses those statistics to estimate things such as:

• how many rows a condition will match
• common values
• value distribution
• number of distinct values
• relationships that affect selectivity

That is why stale statistics can result in poor plans.

You can refresh them with:

ANALYZE users;

And this brings us to one of the most useful tools for PostgreSQL performance work:

EXPLAIN ANALYZE
SELECT *
FROM users
WHERE status = 'banned';

EXPLAIN shows the plan PostgreSQL intends to use.

EXPLAIN ANALYZE actually executes the query and shows what really happened.

Two numbers I pay particular attention to are:

Estimated rows

What PostgreSQL thought would happen.

Actual rows

What really happened.

If the planner estimates:

rows = 100

but the query actually returns:

rows = 100,000

that difference is a clue.

The planner may be making decisions using an inaccurate picture of the data.

There is another important misconception:

More indexes do not automatically mean faster databases.

Every index has a cost.

When you:

INSERT
UPDATE
DELETE

PostgreSQL may also need to update the relevant indexes.

Indexes consume storage, add write overhead, and create additional structures the planner has to consider.

So the goal is not:

“Index every column used in a WHERE clause.”

The better approach is:

Identify a slow query.
Run EXPLAIN ANALYZE.
Understand the execution plan.
Check row estimates and actual rows.
Look at selectivity and data distribution.
Decide whether the query or index should change.
Measure again.

Composite indexes introduce another layer.

For example:

CREATE INDEX idx_orders_customer_status
ON orders(customer_id, status);

The order of columns matters because the index structure is organized around those columns.

And sometimes PostgreSQL can combine multiple indexes using bitmap scans rather than using one composite index.

There are also partial indexes:

CREATE INDEX idx_pending_orders
ON orders(created_at)
WHERE status = 'pending';

Instead of indexing every order, you can index only the subset important to a particular workload.

And covering indexes can sometimes allow PostgreSQL to answer a query without visiting the table heap at all.

The deeper I go into PostgreSQL performance, the more one idea stands out:

An index does not tell PostgreSQL what to do.

It gives the planner another option.

The planner still has to decide whether that option is actually cheaper.

So when PostgreSQL ignores an index, I would not immediately ask:

“Why isn't PostgreSQL using my index?”

I would first ask:

“What does PostgreSQL know about my data that makes another plan look cheaper?”

That question usually leads to a much more interesting investigation.