DEV Community: Maz I

Designing a Production Serverless API: Beyond API Gateway + Lambda

Maz I — Mon, 17 Aug 2026 07:00:00 +0000

Designing a Production Serverless API: Beyond API Gateway + Lambda

A serverless API can begin with a diagram that looks almost too easy:

Client
   ↓
API Gateway
   ↓
Lambda
   ↓
DynamoDB

And for many applications, this is a perfectly reasonable starting point.

But now imagine we are building an order API.

Traffic is unpredictable. Most of the day we receive only a few requests, but during promotions thousands of customers can arrive within a short period.

A customer sends:

POST /orders

with something like:

{
  "customerId": "C101",
  "items": [
    {
      "productId": "P10",
      "quantity": 2
    }
  ]
}

Our requirements initially sound straightforward:

Create the order.
Check inventory.
Take payment.
Send a confirmation email.

We could put everything inside one Lambda:

async function createOrder(order) {
  validate(order);

  await saveOrder(order);
  await reserveInventory(order);
  await chargePayment(order);
  await sendEmail(order);

  return {
    success: true
  };
}

It works.

Until it doesn't.

This article isn't about memorizing AWS services.

Instead, we'll start with that simple implementation and make the requirements progressively harder.

Every time the design breaks, we'll ask:

What problem do we need to solve next?

That is how our simple Lambda gradually becomes a production serverless architecture.

Why serverless in the first place?

Our traffic is unpredictable.

Running servers continuously for occasional spikes could mean maintaining capacity that sits mostly unused.

Lambda is attractive because AWS manages the underlying execution infrastructure and can increase concurrent execution as demand grows, within the applicable scaling and account limits.

So our first version is:

Customer
   ↓
API Gateway
   ↓
CreateOrder Lambda
   ↓
DynamoDB

DynamoDB also fits the serverless model well for workloads where its access-pattern-oriented data model makes sense.

But selecting serverless doesn't remove architecture.

We are about to discover why.

Problem #1: One Lambda is doing too much

Consider this sequence:

Save order        ✅
Reserve inventory ✅
Charge payment    ✅
Send email        ❌ timeout

Our Lambda throws an error.

From the client's perspective, the request failed.

So the client retries.

What should happen?

Should we:

create another order?
reserve inventory again?
charge the customer again?
attempt the email again?

We have mixed operations with very different failure characteristics inside one request.

Sending an email should not determine whether an already successful payment is considered successful.

So our first architectural improvement is decoupling.

Email doesn't belong in the critical request path

After payment succeeds, we don't need the customer to wait while an email provider processes a message.

Instead:

Payment succeeds
      ↓
     SQS
      ↓
Email Lambda
      ↓
Email provider

Now the payment workflow can finish while email processing happens independently.

Amazon SQS acts as a buffer between the producer and consumer.

This gives us an important principle:

A queue does not make the consumer faster. It allows producers and consumers to operate at different rates.

It also isolates failures.

If the email provider is temporarily unavailable, that does not need to break order creation.

But asynchronous systems introduce a new issue.

Lambda processing from SQS can result in messages being processed more than once, so AWS recommends making processing idempotent.

That brings us to one of the most important concepts in serverless systems.

Problem #2: Retries can duplicate business operations

Imagine the customer sends:

POST /orders

The server creates:

ORD-9001

but the response is lost because of a network timeout.

The customer retries.

Without protection, we might create:

ORD-9002

for exactly the same checkout.

Worse, the same problem can happen during payment.

Payment provider charges €100  ✅
             ↓
Lambda loses connection        ❌
             ↓
Lambda retries
             ↓
Charge another €100?           😬

A normal database rollback cannot solve this.

The payment provider exists outside our database transaction.

Idempotency

An operation is idempotent when repeating the same logical request does not create an additional side effect.

For order creation, the client might send:

Idempotency-Key: checkout-abc-123

On the first request:

checkout-abc-123
      ↓
Create ORD-9001

If that exact request is retried:

checkout-abc-123
      ↓
Already processed
      ↓
Return ORD-9001

Not:

Create ORD-9002

For payment, we want the same concept.

payment idempotency key = ORD-9001

If a network failure causes us to retry, the retry represents:

“Give me the result of the same payment operation.”

not:

“Perform a brand-new payment.”

This gives us a useful mental rule:

Retry makes systems resilient. Idempotency makes retries safe.

Problem #3: We now have a workflow

Our order process has grown:

Create order
     ↓
Check inventory
     ↓
Reserve inventory
     ↓
Process payment
     ↓
Update order
     ↓
Queue email

We could manually call Lambda A from Lambda B, Lambda B from Lambda C and keep adding flags to DynamoDB.

But now we have questions such as:

What if payment times out?
What if the card is declined?
Which errors should retry?
How long should we wait?
What happens after the last retry?
How do we know which step the order is currently in?

At this point we are no longer dealing with a simple chain of functions.

We are dealing with a workflow.

That is where AWS Step Functions becomes useful.

Step Functions lets us model workflows as state machines and orchestrate distributed applications and microservices.

Our order process can now be represented explicitly:

Create Order
     ↓
Reserve Inventory
     ↓
Process Payment
    /     \
   /       \
Success   Failure
  ↓          ↓
Update     Handle
Order      failure
  ↓
Queue Email
  ↓
Complete

Step Functions without the mystery

A few states are enough to understand most of our example.

Task

Do something.

ReserveInventory
ProcessPayment
UpdateOrder

Choice

Make a decision.

             Payment result
               /        \
              /          \
         SUCCESS        DECLINED
            ↓              ↓
        Continue       Failure path

A Choice state adds conditional branching to a Step Functions workflow.

Retry

Try a failed operation again.

But only when retrying makes sense.

Catch

If the operation cannot recover, route execution to another state.

Succeed / Fail

End the workflow intentionally as successful or failed.

The important part is not memorizing state names.

It's understanding that business flow is now visible outside our Lambda implementation.

Technical failure is not business failure

Consider payment.

A provider timeout might mean:

temporary network problem

That is potentially retryable.

But:

CARD_DECLINED

is not a temporary infrastructure failure.

Retrying the same declined card three times probably makes no sense.

So:

Payment
   |
   +-- TIMEOUT
   |      ↓
   |    Retry
   |
   +-- CARD_DECLINED
          ↓
       Do not retry
          ↓
       Ask customer
       for another method

Step Functions supports error-specific retry and catch behavior.

This gives us another useful distinction:

Temporary technical failure
→ Retry

Valid negative business result
→ Business workflow path

Not every undesirable result should become an exception.

So what is a DLQ?

Retry/Catch and a Dead-Letter Queue solve related but different problems.

Imagine SQS contains a message that repeatedly fails:

message
  ↓
attempt 1 ❌
attempt 2 ❌
attempt 3 ❌
attempt 4 ❌

Instead of allowing it to continually interfere with normal processing, it can eventually be moved to a dead-letter queue for investigation or later redrive.

So remember:

Retry
→ Try this operation again.

Catch
→ The operation did not recover.
  Which workflow path should run?

DLQ
→ This queued message repeatedly failed.
  Move it away from normal processing.

They are not interchangeable concepts.

Problem #4: We don't have one giant rollback anymore

Now suppose the workflow is:

Create Order       ✅
Reserve Inventory  ✅
Charge Payment     ❌

If this were one relational database transaction, we might think:

ROLLBACK;

But inventory, payment and shipping may be completely separate services.

There is no global database transaction covering everything.

So if payment permanently fails, we must undo the earlier successful business action:

Release inventory

This leads us to the Saga pattern.

Saga in simple language

A Saga breaks a distributed transaction into smaller local transactions.

For important forward actions, we define an appropriate compensating action.

For example:

Forward action          Compensation

Create order        →   Cancel order
Reserve inventory   →   Release inventory
Charge payment      →   Refund payment
Create shipment     →   Cancel shipment

Suppose:

Create Order       ✅
Reserve Inventory  ✅
Charge Payment     ✅
Create Shipment    ❌

We might compensate:

Shipment failed
      ↓
Refund payment
      ↓
Release inventory
      ↓
Cancel order

A compensation isn't necessarily a magical reversal of history.

A refund, for example, is another business operation that compensates for a successful charge.

That distinction matters.

There are two common Saga coordination styles.

Choreography

Services react to events:

OrderCreated
     ↓
Inventory service

InventoryReserved
     ↓
Payment service

PaymentFailed
     ↓
Inventory service releases stock

Orchestration

A central workflow explicitly coordinates the steps:

            Step Functions
                 |
        +--------+--------+
        |        |        |
     Order   Inventory  Payment

Our example is a natural fit for orchestration because the workflow has clear business states, retries and compensations.

Problem #5: Some work can happen in parallel

Payment has succeeded.

Now we need to:

Send confirmation email
Update loyalty points
Notify warehouse

These operations do not necessarily depend on one another.

Running:

Email
  ↓
Loyalty
  ↓
Warehouse

would unnecessarily serialize them.

Step Functions has a Parallel state for independent branches:

              Payment Success
                    ↓
                 Parallel
              /      |       \
             ↓       ↓        ↓
          Email   Loyalty   Warehouse
              \      |       /
               \     |      /
                 complete

A Parallel state starts its branches concurrently and waits for all branches to terminate before moving on.

But this introduces an architectural choice.

If the main workflow must know that all three operations completed, Parallel makes sense.

If these are independent side effects that the main workflow doesn't need to wait for, event-driven fan-out may be cleaner:

            PaymentSucceeded
                  ↓
              Event/Event Bus
             /       |       \
            ↓        ↓        ↓
         Email    Loyalty   Warehouse

So:

Parallel state means “do these things concurrently and my workflow cares about their completion.”

while event choreography often means:

“These consumers can react independently and the producer doesn't need to coordinate them.”

Map: parallel processing for collections

Parallel is useful when the branches perform different jobs.

What if an order contains 500 items and we need to check every item?

That's the same operation repeated over a collection.

A Step Functions Map state fits that problem.

500 order items
       ↓
      Map
       ↓
Check inventory for each item

For large workloads, Step Functions also provides Distributed Map; AWS recommends that mode for scenarios such as concurrency above 40 iterations or very large execution histories/datasets.

But this raises another question.

Should all 500 checks execute simultaneously?

Maybe not.

Problem #6: Scaling can become dangerous

This is one of the most important serverless lessons.

Imagine:

500 inventory checks

and the inventory dependency can safely handle:

10 concurrent operations

Running all 500 at once would simply move the bottleneck downstream.

The exact same principle appears in ordinary Node.js:

await Promise.all(hugeArray.map(processItem));

can create too much concurrency.

Different technology, same engineering rule:

Parallelism improves throughput until the dependency becomes the bottleneck.

Now apply that to Lambda itself.

Suppose our API normally receives:

20 requests/second

but a promotion suddenly sends thousands.

Lambda can scale concurrency.

That sounds excellent until every execution calls a payment provider that safely supports only 100 requests per second.

Thousands of requests
        ↓
      Lambda
  scales aggressively
        ↓
Payment Provider
     🔥🔥🔥

The Lambda tier can scale faster than the dependency behind it.

Reserved concurrency

Lambda supports reserved concurrency, which can reserve concurrency for a function while also placing an upper bound on that function's concurrency. AWS specifically notes it can help prevent overwhelming downstream resources such as database connections.

Conceptually:

Payment Lambda
maximum concurrency = 100
          ↓
Payment Provider

Now scaling has a guardrail.

But simply throttling thousands of requests isn't always a good customer experience.

For asynchronous workloads, SQS can absorb the burst:

5,000 jobs
     ↓
    SQS
====================
 jobs waiting safely
====================
     ↓
Lambda consumers
controlled throughput
     ↓
Dependency

SQS/Lambda event source mappings also support concurrency controls.

Again:

Automatic scaling is not automatically safe architecture.

Always ask:

Can Lambda scale?

        ↓ yes

Can everything behind Lambda
scale at the same rate?

Cold starts and provisioned concurrency

When Lambda needs a new execution environment, initialization occurs before the handler processes the request.

That extra initialization latency is commonly called a cold start.

A simplified comparison:

Warm

Request
  ↓
Existing environment
  ↓
handler()

versus:

Cold

Request
  ↓
Initialize environment
  ↓
Load runtime/code/dependencies
  ↓
handler()

This is also why reusable clients are often initialized outside the handler when appropriate:

const client = createClient();

export const handler = async (event) => {
  // reuse client
};

For latency-sensitive workloads, Lambda also offers provisioned concurrency, which keeps execution environments initialized ahead of demand.

Don't confuse the two concepts:

Reserved concurrency
→ controls/reserves concurrency capacity

Provisioned concurrency
→ keeps execution environments initialized
  for predictable startup latency

Problem #7: DynamoDB scales, but our key design still matters

A serverless database does not remove data-modeling decisions.

DynamoDB uses partition-key values to determine how items are distributed internally. Items sharing a partition-key value are grouped together, and when a sort key exists, they are ordered by sort-key value.

Suppose we choose:

PK = merchantId

Usually traffic is distributed:

Merchant A → moderate traffic
Merchant B → moderate traffic
Merchant C → moderate traffic

But during a huge campaign:

Merchant A → enormous traffic 🔥
Merchant B → small
Merchant C → small

Now one partition-key value becomes disproportionately busy.

AWS recommends designing partition keys with many distinct values and reasonably uniform activity.

This is why DynamoDB design begins with access patterns.

Partition key and sort key, simplified

A useful mental model:

Partition Key
→ Which logical group does this item belong to?

Sort Key
→ How do I identify/order items inside that group?

For example:

PK = MERCHANT#M123

SK = ORDER#2026-08-10#9001
SK = ORDER#2026-08-11#9002
SK = ORDER#2026-08-12#9003

Now:

PK = MERCHANT#M123

can retrieve that merchant's order collection.

The timestamp in the sort key can help support chronological/range-oriented access.

GSI: another access path

Suppose our base table is optimized for:

Get an order by orderId

but another requirement is:

Get orders by merchant and date

A Global Secondary Index (GSI) can use a different partition and sort key from the base table.

Conceptually:

Base table

PK = ORDER#9001

while:

GSI

PK = MERCHANT#M123
SK = 2026-08-10#ORDER#9001

Same underlying item.

Different access path.

The key lesson is:

DynamoDB schema design is strongly driven by how the application needs to retrieve data.

That is a different mental model from designing relational tables first and later adding indexes to optimize queries.

Eventual consistency: correct doesn't always mean immediately visible everywhere

Suppose payment succeeds:

ORDER#9001
status = PAID

Immediately afterward, another component reads through a GSI.

Could it briefly observe older data?

Yes.

DynamoDB table/LSI reads can use eventual or strong consistency, while GSI reads are eventually consistent.

Simplified:

Write:
status = PAID
      ↓
immediate eventual read
      ↓
possibly PENDING briefly
      ↓
later read
      ↓
PAID

This doesn't mean the write disappeared.

It reflects the consistency guarantee of that read path.

For an immediately authoritative order confirmation, we may prefer reading the exact order from the base table using the appropriate consistency requirement instead of assuming every secondary view has already converged.

Another distributed-systems lesson:

Correct state and immediate visibility of that state everywhere are different guarantees.

Problem #8: Every Lambda should not have permission to everything

Our architecture now contains several functions:

CreateOrder Lambda
Inventory Lambda
Payment Lambda
Email Lambda

The easiest IAM policy would be:

{
  "Effect": "Allow",
  "Action": "*",
  "Resource": "*"
}

It would also be a terrible production boundary.

A better architecture uses least privilege.

CreateOrder Lambda
→ write required order data
→ start the workflow

Payment Lambda
→ read/update payment state
→ access payment secret

Email Lambda
→ consume email messages
→ access email secret

Lambda uses execution roles to determine what AWS resources a function can access.

Each component should receive the permissions required for its responsibility, not unrestricted authority over the whole application.

For example:

Email Lambda

CAN:
✓ consume email queue
✓ read email-provider credentials

CANNOT:
✗ modify payment records
✗ read payment secrets
✗ delete orders

That reduces the blast radius if one component is compromised.

Secrets are not configuration strings in Git

This should never happen:

const PAYMENT_API_KEY = "live-secret-key";

inside source control.

Sensitive credentials can be stored in AWS Secrets Manager and accessed only by functions whose IAM role allows it.

Payment Lambda
      ↓
Secrets Manager
      ↓
payment-provider credentials

Meanwhile:

Email Lambda

does not need permission to read those credentials.

Authorization boundaries matter just as much as network/application boundaries.

And don't leak the secret through logs

This:

console.log({
  cardNumber,
  cvv,
  paymentToken
});

creates a new security problem.

Prefer operational context:

{
  "orderId": "ORD-9001",
  "paymentAttemptId": "PAY-72",
  "status": "FAILED",
  "errorType": "PROVIDER_TIMEOUT"
}

which takes us to our final production concern.

Problem #9: Order 9001 failed. Where?

Our order can now cross:

API Gateway
   ↓
Lambda
   ↓
DynamoDB
   ↓
Step Functions
   ↓
Inventory Lambda
   ↓
Payment Lambda
   ↓
SQS
   ↓
Email Lambda

A customer says:

“My order failed.”

Which component failed?

Without observability, debugging a distributed system becomes archaeology.

Logs: what happened?

Instead of:

console.log("payment failed");

prefer structured logs:

{
  "correlationId": "REQ-ABC-123",
  "orderId": "ORD-9001",
  "service": "payment",
  "status": "FAILED",
  "errorType": "PROVIDER_TIMEOUT"
}

AWS Powertools for Lambda includes utilities for structured logging, tracing and metrics.

Correlation IDs: connect the story

Carry an identifier across components:

REQ-ABC-123
     |
API Gateway
     |
CreateOrder
     |
Step Functions
     |
Payment
     |
Email

Now logs across several services can be correlated to the same business operation.

Metrics: is the system healthy?

Useful metrics might include:

Lambda error rate
Lambda duration
Lambda throttles
Concurrent executions

Payment failure rate

SQS queue depth
Oldest message age

Failed Step Functions executions
Workflow duration

Orders stuck in PAYMENT_PENDING

Notice the last one.

Observability isn't only:

technical metrics

It also needs:

business metrics

Your infrastructure could technically be healthy while thousands of orders remain stuck in an invalid business state.

Alarms: when should humans care?

Dashboards are useful.

But humans shouldn't have to stare at them continuously.

Examples:

Payment failures suddenly > normal threshold

SQS oldest message > 5 minutes

Lambda throttling increasing

Workflow failures spike

Those signals should trigger alarms and operational action.

Step Functions execution history: observe the business workflow

One advantage of explicit orchestration is that we can inspect a workflow execution:

CreateOrder        ✅
ReserveInventory   ✅
ProcessPayment     ❌
Retry              ❌
Retry              ✅
UpdateOrder        ✅
QueueEmail         ✅

Instead of reconstructing the entire process from Lambda-to-Lambda calls, we can see the state-machine execution itself.

Traces: where did the request spend time?

Suppose checkout completes but takes eight seconds.

Logs tell us that everything executed.

Metrics tell us latency increased.

Tracing helps answer:

API Gateway       40 ms
CreateOrder       90 ms
DynamoDB          12 ms
Inventory        140 ms
Payment         6.8 sec  ← bottleneck

Lambda integrates with AWS X-Ray, which can produce service maps and searchable traces for diagnosing errors and latency bottlenecks.

So a useful observability mental model is:

Logs
→ What happened?

Metrics
→ Is the system healthy overall?

Traces
→ Where did this request spend its time?

The architecture we ended up with

We started here:

API Gateway
    ↓
Lambda
    ↓
DynamoDB

After introducing real production requirements, we arrived at something closer to:

                         Client
                           |
                           v
                     API Gateway
                           |
                           v
                  CreateOrder Lambda
                           |
                           v
                       DynamoDB
                           |
                           v
                    Step Functions
                    /      |      \
                   /       |       \
                  v        v        v
             Inventory   Payment   Other Tasks
                 |          |
                 |      Retry / Catch
                 |          |
                 +---- Saga compensation
                            |
                        Success
                            |
                            v
                           SQS
                            |
                            v
                       Email Lambda

Surrounding that application:

IAM
→ least-privilege permissions

Secrets Manager
→ credentials

CloudWatch
→ logs, metrics, alarms

X-Ray / tracing
→ request path and latency

Correlation IDs
→ connect one order across components

The interesting part is that we didn't start by saying:

“Let's use API Gateway, Lambda, DynamoDB, SQS, Step Functions and ten other AWS services.”

Every component appeared because a requirement exposed a limitation in the simpler design.

We needed asynchronous side effects.

→ SQS.

Retries introduced duplicate operations.

→ Idempotency.

The process became a business workflow.

→ Step Functions.

Distributed operations couldn't share one rollback.

→ Saga and compensation.

Independent work didn't need serial execution.

→ Parallel and Map.

Automatic scaling threatened downstream systems.

→ Concurrency controls and queues.

DynamoDB needed multiple efficient read patterns.

→ PK/SK/GSI design.

Distributed reads did not always become visible simultaneously.

→ Consistency awareness.

More components increased security exposure.

→ Least privilege and secrets management.

More components made failures harder to diagnose.

→ Observability.

The serverless lesson I would keep

Serverless is often introduced as:

“You don't manage servers.”

That's true, but it is only the beginning.

You still have to reason about:

failure
retries
duplicates
idempotency
concurrency
downstream capacity
distributed transactions
compensation
data access patterns
consistency
permissions
security
observability

AWS can manage servers for us.

AWS cannot decide our business guarantees for us.

And perhaps the most useful architectural question throughout this entire example was not:

Which AWS service should I use?

It was:

What problem does my current simple design fail to solve next?

That question is what turns a collection of serverless services into an architecture.

Building a Website Change Monitoring System: From One URL to a Distributed Crawling Pipeline

Maz I — Mon, 10 Aug 2026 14:27:40 +0000

Building a Website Change Monitoring System: From One URL to a Distributed Crawling Pipeline

Imagine someone gives us a URL:

https://example.com

and says:

Tell me if this website changes.

At first, this sounds almost trivial.

Fetch the page, save the HTML, come back later, fetch it again, compare the two versions.

For one page, that can work.

But almost immediately, questions start appearing.

What exactly are we monitoring?

Only the homepage?

Every page under the website?

How often should we check it?

Every two minutes?

Once per day?

Only once when an administrator requests it?

What happens when a harmless calendar or rotating banner changes?

And if we eventually monitor hundreds or thousands of websites, who decides what should be crawled, and who actually performs all that crawling?

Those questions are what turn a simple script into a monitoring system.

This article walks through that evolution one design decision at a time.

First question: what does “monitor this website” actually mean?

Suppose we start with:

https://example.com

If we fetch only that URL, we are monitoring only the homepage.

But the important page might actually be:

https://example.com/login

or:

https://example.com/payment

or:

https://example.com/products/important-product

So monitoring a website introduces our first configuration decision.

How deeply should we crawl?

Consider this website:

Homepage
│
├── About
├── Products
│   ├── Product A
│   └── Product B
│
└── Blog
    ├── Post 1
    └── Post 2

If we monitor only the homepage, we might completely miss a change on Product A.

So we could tell the crawler:

Follow every internal link.

But that creates another problem.

Product A may contain links to documentation.

Documentation may contain hundreds of pages.

Those pages may link to thousands more.

Suddenly:

1 URL

becomes:

10 URLs

then:

100 URLs

then potentially:

10,000+ URLs

So we need a boundary.

That is what crawl depth gives us.

If the administrator chooses:

crawl depth = 0

we monitor only:

Homepage

With:

crawl depth = 1

we monitor:

Homepage
    ↓
direct internal links

With:

crawl depth = 2

we monitor:

Homepage
    ↓
Level 1 pages
    ↓
links discovered inside Level 1 pages

Conceptually:

Depth 0

example.com
      |
      v

Depth 1

/about
/products
/blog
      |
      v

Depth 2

/products/a
/products/b
/blog/post-1
/blog/post-2

Now the administrator can choose the monitoring surface instead of letting the crawler wander indefinitely.

This also tells us something important about our data model.

A website isn't just:

url

It needs configuration.

Something closer to:

Website
├── URL
├── Crawl depth
├── Monitoring frequency
├── Ignore rules
├── Notification preferences
└── Monitoring enabled/disabled

Now we are no longer building a crawler.

We are beginning to build a monitoring product.

Second question: how often should we check?

Suppose our baseline crawl finishes at:

10:00

When should we crawl again?

There is no universally correct answer.

For a low-risk informational website:

once per day

might be enough.

For something sensitive:

every 5 minutes

might be appropriate.

For another system:

every 2 minutes

might be required.

And sometimes continuous monitoring isn't even needed.

An administrator may simply want:

crawl once now

So frequency also belongs to the website's monitoring configuration.

For example:

example.com

crawl depth: 2
frequency: every 5 minutes

while:

another-site.com

crawl depth: 1
frequency: daily

This creates our next architectural requirement.

Something needs to keep asking:

Which websites are due to be checked now?

That responsibility belongs to a scheduler.

Conceptually:

Website configuration
        |
        v
    Scheduler
        |
        v
Is next crawl due?

The scheduler doesn't need to know anything about HTML parsing or link discovery.

Its job is simply to decide when work should happen.

That separation becomes important later.

Third question: how do we avoid becoming the attacker?

Now imagine:

crawl depth = 3
frequency = 2 minutes

and the website contains thousands of pages.

A badly designed crawler could start firing hundreds of requests against the same server every two minutes.

Our monitoring system could accidentally create the exact availability problem it is supposed to help detect.

So monitoring frequency cannot mean:

Hit the website as aggressively as possible.

The crawler needs to behave responsibly.

A mature design should consider things such as:

per-domain concurrency limits
request delays
timeouts
maximum pages per crawl
retry backoff
429 handling
5xx handling
overlapping crawl prevention

For example:

example.com

maximum concurrent requests = 2

rather than letting dozens of workers hammer the same domain simultaneously.

And if the previous crawl is still running:

crawl #101 = running

we probably shouldn't blindly start:

crawl #102

just because the next two-minute interval arrived.

Already, the “simple crawler” has acquired scheduling and resource-control requirements.

Fourth question: what happens when hundreds of websites become due together?

Suppose we have 1,000 monitored websites.

At 10:00, hundreds become eligible for crawling.

One naive implementation would be:

Dashboard
   |
   v
Fetch website
   |
   v
Parse HTML
   |
   v
Save results

But now the application serving administrators is also responsible for long-running crawling work.

That creates several problems.

A slow website could tie up the main application.

A timeout could delay unrelated work.

A burst of scheduled crawls could overload the dashboard server.

And scaling the crawler would mean scaling the entire application.

So logically, we separate two responsibilities.

The main application manages:

customers
websites
configuration
monitoring settings
administration

The crawler handles:

network requests
HTML
link extraction
snapshots
change detection

In the system I worked on, the main application kept its structured application data in MySQL.

Crawler-oriented data was handled separately.

But now these two parts need a way to communicate.

This is where the queue appears

Instead of telling the crawler:

Crawl this website right now and make me wait until you're done,

the application can publish a job:

{
  "websiteId": 842,
  "url": "https://example.com",
  "maxDepth": 2
}

into a message queue.

Now the architecture evolves naturally:

Dashboard
   |
   v
 MySQL
   |
   v
Scheduler
   |
   v
Crawl Queue
   |
   v
Crawler Workers

This gives us a buffer.

If 500 crawl jobs appear suddenly, the main application doesn't need 500 crawlers immediately.

Jobs wait in the queue.

Crawler workers consume them according to available capacity.

If we later need more crawling throughput, we can add crawler workers without redesigning the dashboard.

The queue isn't there because queues are fashionable.

It appears because scheduled work and execution capacity are different problems.

Now the crawler finally receives a URL

Suppose the worker receives:

https://example.com

The first operation is straightforward.

Fetch the page.

Conceptually:

response = requests.get(
    "https://example.com",
    timeout=10
)

html = response.text

But this is the first crawl.

There is nothing to compare against yet.

So instead of detecting a change, this crawl establishes a baseline.

We store the page content.

In our crawler side, page snapshots and crawling data were stored in MongoDB.

Now the architecture has two distinct data concerns:

MySQL
↓
website/customer/configuration data

MongoDB
↓
crawler/page/snapshot-oriented data

That separation wasn't simply:

SQL good here, MongoDB good there.

The two sides represented different workloads.

The homepage is only the beginning

Once the homepage is downloaded, we inspect its internal links.

For example:

<a href="/about">About</a>
<a href="/products">Products</a>
<a href="/blog">Blog</a>

We extract:

/about
/products
/blog

convert them to full URLs where necessary, and keep only links belonging to the target website.

Then we need another important step:

deduplication.

Imagine the same page appears through several navigation paths:

Homepage → Products
Blog → Products
About → Products
Footer → Products

We don't want to crawl /products four times during the same crawl run.

So logically:

Extract links
      |
      v
Normalize URLs
      |
      v
Remove duplicates
      |
      v
Check crawl depth
      |
      v
Schedule inner pages

The inner crawler then processes those URLs.

Each inner page can discover more URLs.

So a queue item might carry:

{
  "url": "https://example.com/products",
  "depth": 1,
  "maxDepth": 3
}

If:

depth < maxDepth

we extract more links.

When:

depth == maxDepth

we stop expanding.

That simple number prevents recursive discovery from turning into uncontrolled crawling.

URL deduplication is trickier than it looks

Consider:

https://example.com/about
https://example.com/about/
https://example.com/about#team
https://example.com/about?utm_source=email

Are those four separate pages?

Maybe.

But often they represent the same useful monitoring target.

So before deduplicating, a modern crawler may normalize URLs by handling:

fragments
tracking parameters
relative paths
trailing slashes
canonical URLs
host casing

Otherwise, the crawler may spend significant resources repeatedly monitoring effectively identical pages.

Now we finally reach the actual monitoring problem

After the first crawl, suppose we stored this page:

https://example.com/products

Five minutes later, according to its configured frequency, the scheduler queues the website again.

The crawler downloads the page again.

Now we have:

previous version
current version

We need to know:

Has anything changed?

In our implementation, one simple mechanism was an MD5 hash.

Conceptually:

HTML
 |
 v
MD5
 |
 v
hash

So baseline HTML produces:

A7F91...

and the next crawl produces another hash.

If:

old_hash == new_hash

the inputs are identical.

Nothing changed.

If:

old_hash != new_hash

something changed.

Very simple.

Very fast.

And also incomplete.

Because “different” does not mean “important”

Imagine this element:

<div class="calendar">
    9 August
</div>

A day later it becomes:

<div class="calendar">
    10 August
</div>

The HTML changed.

Therefore:

old MD5 != new MD5

The system alerts the administrator.

But there was no attack.

Nothing important happened.

Now imagine another page contains:

10:31:04

and one minute later:

10:32:04

Again:

hash changed

Other common examples include:

rotating headlines
advertisements
visitor counters
timestamps
calendars
live market values
random identifiers
dynamic widgets

This reveals the central weakness of pure hashing.

A hash can answer:

Are these two inputs identical?

It cannot answer:

Is this difference meaningful?

To MD5:

calendar date changed

and:

attacker replaced the homepage

are both simply:

different input

And that creates false positives.

Why false positives are dangerous

Imagine monitoring a security-sensitive website.

The administrator receives:

ALERT
ALERT
ALERT
ALERT
ALERT

throughout the day.

Most alerts are harmless calendar or headline changes.

Eventually the administrator begins ignoring them.

Now when an actual unexpected modification happens:

ALERT

it looks like everything else.

So false-positive handling isn't merely a convenience feature.

It directly affects whether the monitoring system remains useful.

Introducing the ignore list

Suppose we know this region changes constantly:

<div class="calendar">
    10 August
</div>

The administrator can configure:

ignore .calendar

Similarly:

ignore .ticker
ignore #clock
ignore .rotating-banner

Now our comparison pipeline becomes:

Fetched HTML
     |
     v
Apply ignore rules
     |
     v
Remove known dynamic regions
     |
     v
Generate normalized content
     |
     v
MD5
     |
     v
Compare

Instead of hashing everything blindly, we're hashing the content we actually care about monitoring.

Conceptually:

def prepare_for_monitoring(html, ignored_selectors):
    dom = parse_html(html)

    for selector in ignored_selectors:
        remove(dom, selector)

    return serialize(dom)

then:

normalized_html = prepare_for_monitoring(
    html,
    [".calendar", ".ticker"]
)

page_hash = md5(normalized_html)

Now a calendar update doesn't automatically become a security alert.

This is where the system moves from:

change detection

toward:

meaningful change detection.

But what happens when a meaningful change is detected?

Suppose:

old_hash != new_hash

after ignore rules have been applied.

The crawler has discovered something.

Should it now send an email itself?

Send an SMS itself?

Update the dashboard itself?

It could.

But then the crawler would be responsible for:

HTTP crawling
HTML parsing
comparison
email
SMS
dashboard updates

That's too many responsibilities in one component.

So another boundary naturally appears.

The crawler emits a change event.

Change detected
      |
      v
Alert Queue

Then separate notification handlers can deliver it:

              Alert Queue
                  |
       +----------+----------+
       |          |          |
       v          v          v
   Dashboard    Email       SMS

Now an SMS provider outage doesn't stop crawling.

Email delivery can retry independently.

And a new notification channel can be added later without rewriting the crawler.

Again, the queue isn't introduced because “event-driven architecture is cool.”

It appears because detecting something and notifying someone are separate reliability problems.

The architecture we ended up with

By following the requirements rather than starting from technologies, our simple URL checker evolved into something closer to this:

                    Administrator
                         |
                         v
                    Dashboard
                         |
                         v
                       MySQL
             website/configuration
                         |
                         v
                     Scheduler
                         |
               Is crawl due?
                         |
                         v
                    Crawl Queue
                         |
                         v
                  Python Crawlers
                         |
              +----------+----------+
              |                     |
              v                     v
        Link Discovery          Page Snapshot
              |                     |
              v                     v
      Normalize/Deduplicate       MongoDB
              |
              v
        Depth-controlled
          inner crawling

                         |
                         v
                  Prepare HTML
                apply ignore list
                         |
                         v
                    Generate MD5
                         |
                         v
               Compare previous hash
                    /        \
                   /          \
              unchanged      changed
                                |
                                v
                           Alert Queue
                                |
                    +-----------+-----------+
                    |           |           |
                    v           v           v
                Dashboard      Email       SMS

What began as:

download HTML

became scheduling, crawling, discovery, state, comparison, noise reduction, queueing, and notification.

One more problem: crawl politely

Let's return to:

frequency = 2 minutes
depth = 3

Suppose this website has 5,000 discoverable pages.

We absolutely don't want:

5,000 requests

fired aggressively against the target.

A responsible crawler needs controls.

Per-domain concurrency

Instead of:

100 workers → example.com

we might enforce:

maximum 2 concurrent requests → example.com

Crawl budgets

A website configuration could include:

max depth: 3
max pages per crawl: 1,000
request timeout: 10 seconds
per-domain concurrency: 2

Depth and page limit solve different problems.

Depth 1 could still contain 20,000 links.

Retry with backoff

If the target returns:

429 Too Many Requests

retrying immediately is the wrong response.

Instead:

1s
2s
4s
8s
16s

with jitter.

Don't overlap crawls blindly

If monitoring is configured every two minutes but one crawl takes four minutes:

10:00 crawl A starts
10:02 crawl B scheduled

the scheduler should probably detect:

website currently crawling

rather than blindly launching another run.

What I would change if I designed it today

The underlying problem is still interesting, but I wouldn't rebuild every detail exactly the same way.

Hash first, diff second

Hashing is still useful for a very fast first check.

hash same
    |
    v
stop

But if:

hash different

I would generate an actual structured diff.

Instead of:

Website changed.

the administrator might see:

- Payment destination: account A
+ Payment destination: account B

That's significantly more useful.

Compare the DOM, not only serialized HTML

HTML already contains structure.

So changes could be classified by region:

title changed
main content changed
form action changed
external script added
navigation changed

Then different changes could receive different severity levels.

For example:

Calendar changed              INFO
Headline changed              LOW
Form action changed           HIGH
New external JavaScript       HIGH
Large DOM replacement         CRITICAL

Better normalization

Before comparison I would also normalize things such as:

whitespace
volatile generated IDs
known tracking parameters
timestamps
irrelevant markup differences

This would further reduce noise.

Explicit crawl-run tracking

Instead of only knowing whether a website is “currently crawling,” I would model crawl runs explicitly:

crawl_run_id
website_id
started_at
finished_at
pages_discovered
pages_processed
pages_failed
status

That makes operational debugging much easier.

Dead-letter queues

Some crawl jobs will repeatedly fail.

After controlled retries, they should move somewhere visible instead of looping forever.

normal queue
     |
   retries
     |
     v
dead-letter queue

Observability

For a serious deployment I'd monitor:

queue depth
crawl duration
pages/sec
HTTP error rate
retry count
change rate
false-positive rate
notification failures
per-domain request rate

Without those metrics, it's difficult to know whether the system is healthy or merely running.

And today, AI creates an interesting extra layer

The deterministic monitoring engine should remain deterministic.

I would not replace HTML comparison with an LLM.

But after a real change has already been detected, AI could help answer:

What does this change mean?

For example:

- <script src="/assets/app.js">
+ <script src="https://unknown-example.com/inject.js">

A semantic analysis stage might classify:

{
  "type": "external_script_added",
  "severity": "high",
  "reason": "A previously unseen external JavaScript source was introduced."
}

So the architecture becomes:

Deterministic detection
        |
        v
Real change found
        |
        v
Generate structured diff
        |
        v
Rules / classifiers
        |
        v
Optional AI analysis
        |
        v
Severity + explanation

The important part is the ordering.

AI helps interpret the signal.

It doesn't replace the reliable mechanism that discovers the signal.

The interesting lesson

When we began, the requirement looked like:

Monitor a URL for changes.

But each real-world question forced another design decision.

Which pages matter?

→ crawl depth.

How frequently do they matter?

→ monitoring schedule.

What happens when hundreds become due?

→ queue + workers.

How do we revisit inner pages?

→ link extraction + deduplication + depth tracking.

How do we know something changed?

→ snapshots + hashes.

How do we avoid useless alerts?

→ ignore rules + normalization.

How do we notify reliably?

→ alert events + separate notification workers.

How do we avoid harming the monitored site?

→ crawl budgets, rate limits, concurrency control, and backoff.

That's what I find most interesting about systems like this.

The final architecture doesn't need to be invented on a whiteboard first.

It can emerge naturally from repeatedly asking:

What problem does our current simple solution fail to solve next?

Why does PostgreSQL sometimes ignore an index you created?

Maz I — Sat, 08 Aug 2026 22:55:41 +0000

Why does PostgreSQL sometimes ignore an index you created?

A common assumption is:

“If a column has an index, PostgreSQL should use it.”

But PostgreSQL does not work that way.

An index is only one possible access path.

Before executing a query, PostgreSQL's query planner estimates the cost of different plans and chooses the one it believes will be cheapest.

That might be:

• Sequential Scan
• Index Scan
• Index Only Scan
• Bitmap Index Scan
• Parallel Sequential Scan

So sometimes PostgreSQL sees your perfectly valid index and deliberately decides:

“Scanning the table is cheaper.”

Consider this example:

CREATE INDEX idx_users_status
ON users(status);

And imagine the table contains 10 million users:

active   = 7,000,000
inactive = 2,900,000
banned   =   100,000

Now compare these queries:

SELECT *
FROM users
WHERE status = 'banned';

and:

SELECT *
FROM users
WHERE status = 'active';

Both use the indexed status column.

But PostgreSQL may choose very different execution plans.

For banned, only around 1% of rows match.

Using the index can make sense because PostgreSQL can locate a relatively small set of rows instead of scanning millions of unrelated rows.

For active, around 70% of the table matches.

Now using the index may mean:

Traverse the index.
Find millions of matching row locations.
Visit millions of table pages to retrieve those rows.

At that point, reading the table sequentially may simply cost less.

This is where selectivity becomes important.

A highly selective condition returns a small percentage of the table.

WHERE email = 'user@example.com'

That is usually a great candidate for an index.

A low-selectivity condition might match most of the table.

WHERE status = 'active'

An index may provide little advantage.

But how does PostgreSQL know how many rows are likely to match?

Statistics.

PostgreSQL collects information about the data distribution in a table.

The planner uses those statistics to estimate things such as:

• how many rows a condition will match
• common values
• value distribution
• number of distinct values
• relationships that affect selectivity

That is why stale statistics can result in poor plans.

You can refresh them with:

ANALYZE users;

And this brings us to one of the most useful tools for PostgreSQL performance work:

EXPLAIN ANALYZE
SELECT *
FROM users
WHERE status = 'banned';

EXPLAIN shows the plan PostgreSQL intends to use.

EXPLAIN ANALYZE actually executes the query and shows what really happened.

Two numbers I pay particular attention to are:

Estimated rows

What PostgreSQL thought would happen.

Actual rows

What really happened.

If the planner estimates:

rows = 100

but the query actually returns:

rows = 100,000

that difference is a clue.

The planner may be making decisions using an inaccurate picture of the data.

There is another important misconception:

More indexes do not automatically mean faster databases.

Every index has a cost.

When you:

INSERT
UPDATE
DELETE

PostgreSQL may also need to update the relevant indexes.

Indexes consume storage, add write overhead, and create additional structures the planner has to consider.

So the goal is not:

“Index every column used in a WHERE clause.”

The better approach is:

Identify a slow query.
Run EXPLAIN ANALYZE.
Understand the execution plan.
Check row estimates and actual rows.
Look at selectivity and data distribution.
Decide whether the query or index should change.
Measure again.

Composite indexes introduce another layer.

For example:

CREATE INDEX idx_orders_customer_status
ON orders(customer_id, status);

The order of columns matters because the index structure is organized around those columns.

And sometimes PostgreSQL can combine multiple indexes using bitmap scans rather than using one composite index.

There are also partial indexes:

CREATE INDEX idx_pending_orders
ON orders(created_at)
WHERE status = 'pending';

Instead of indexing every order, you can index only the subset important to a particular workload.

And covering indexes can sometimes allow PostgreSQL to answer a query without visiting the table heap at all.

The deeper I go into PostgreSQL performance, the more one idea stands out:

An index does not tell PostgreSQL what to do.

It gives the planner another option.

The planner still has to decide whether that option is actually cheaper.

So when PostgreSQL ignores an index, I would not immediately ask:

“Why isn't PostgreSQL using my index?”

I would first ask:

“What does PostgreSQL know about my data that makes another plan look cheaper?”

That question usually leads to a much more interesting investigation.